...and abstract away the stream/cursor blocking/aborting functionality
so that demuxers can implement or ignore those methods as they see fit.
This is a step towards implementing a wrapper demuxer for MSE streams.
In order to free up memory when a video is paused for an extended
period, we add a new Suspended state to PlaybackManager which tells the
data providers to suspend. The data providers will handle this signal
by disposing of their entire decoded data queue and flushing their
decoder.
When initially creating a PlaybackManager, and when resuming to a
paused state, the delay before suspension will be much lower than when
pausing from any other state. This is intended to prevent media
elements from consuming memory for long when decoding the first frame
for display, as well as to allow the data providers to suspend much
more quickly after a seek while paused.
Currently, resuming playback doesn't display much of a delay on my
MacBook, though that may change once we completely tear down the
decoder in the suspended state. It may also be exacerbated by using
hardware decoders due more complex decoder initialization.
By sniffing specifically for MP4 and WebM, we were precluding
PlaybackManager from playing any other formats. Instead, use
MatroskaDemuxer if the media has a `matroska` or `webm` EBML doctype,
and fall back to FFmpeg for all others.
We'll need to limit the containers that FFmpeg is able to open at some
point, but for now, this allows us to play the formats we could before.
`IncrementallyPopulatedStream::Cursor` now tracks whether it's currently
blocked inside a wait for more bytes, allowing higher layers to
distinguish "no frames yet" from "decoder is idle".
Enter buffering when `DisplayingVideoSink` runs out of frames and the
associated `VideoDataProvider` is blocked waiting for data to arrive.
Exit buffering once decoding refills the frame queue.
For now, buffering behaves like paused, but it gives us an explicit
state to hook UI into.
When media data is fully buffered, we can just try Matroska first and
fall back to FFmpeg. With incremental fetching, that approach becomes
wasteful: we may repeatedly attempt demuxer construction before enough
bytes are available, and FFmpeg in particular tends to produce noisy
logs while probing partial input.
Add lightweight container sniffing for WebM and MP4 that operates on
`IncrementallyPopulatedStream::Cursor`,
`prepare_playback_from_media_data()` now blocks until there is enough
data to decide the container type, then constructs the appropriate
demuxer directly instead of probing both.
Co-authored-by: Zaggy1024 <Zaggy1024@gmail.com>
Refactor the FFmpeg and Matroska demuxers to consume data through
`IncrementallyPopulatedStream::Cursor` instead of a pointer to fully
buffered.
This change establishes a new rule: each track must be initialized with
its own cursor. Data providers now explicitly create a per-track context
via `Demuxer::create_context_for_track(track, cursor)`, and own pointer
to that cursor. In the upcoming changes, holding the cursor in the
provider would allow to signal "cancel blocking reads" so an
in-flight seek can fail immediately when a newer seek request arrives.
If we don't pause the updates, re-enabling the video track will show an
old frame briefly. Since a normal seek also pauses/resumes updates on
the active sinks, the display is guaranteed to be resumed.
Without removing the current block, we end up playing no audio until
its timestamp is reached, after we shift the next block forward if its
timestamp is less than the previous block's end time. This would mean
that re-enabling an audio track after a seek backwards would result in
a delay in its audio resuming.
Posting callbacks to the main thread is now predicated on whether the
event loop reference is alive, preventing a stack-use-after-return.
The data providers will also check if they've been requested to exit
before calling deferred_invoke, though this is not going to be the case
unless the media element gets GCed while the media is playing.
Demuxer creation and track+duration extraction are moved to a separate
thread so that the media data byte buffer is no longer accessed from the
main thread. This will be important once the buffer is populated
incrementally, as having the main thread both populate and read from the
same buffer could easily lead to deadlocks. Aside from that, moving
demuxer creation off the main thread helps to be more responsive.
`VideoDataProvider` and `AudioDataProvider` now accept the main thread
event loop pointer as they are constructed from the thread responsible
for demuxer creation.
This implementation allows:
- Accurate seeking to an exact timestamp
- Seeking to the keyframe before a timestamp
- Seeking to the keyframe after a timestamp
These three options will be used to satisfy the playback position
selection in the media element's seeking steps.
This time provider can later be swapped out for the AudioMixingSink
when it implements the MediaTimeProvider interface, so that frame
timing can be driven by audio when it is present.
This commit implements the functionality to play back audio through
PlaybackManager.
To decode the audio data, AudioDataProviders are created for each track
in the provided media data. These providers will fill their audio block
queue, then sit idle until their corresponding tracks are enabled.
In order to output the audio, one AudioMixingSink is created which
manages a PlaybackStream which requests audio blocks from multiple
AudioDataProviders and mixes them into one buffer with sample-perfect
precision.
With this commit, all PlaybackManager can do is autoplay a file from
start to finish, with no pausing or seeking functionality.
All audio playback functionality has been removed from HTMLMediaElement
and HTMLAudioElement in anticipation of PlaybackManager taking that
over, for both audio-only and audio/video.
Most users will only care about the total file duration, and shouldn't
be required to determine the file duration from multiple track
durations. To facilitate that, add a total_duration() function that
returns the demuxer's duration not associated to any particular track.
