The segments are parsed for the SourceBufferProcessor by the
WebMByteStreamParser. It parses the initialization segment to update
its internal set of tracks, then SourceBufferProcessor/SourceBuffer set
them up for playback. When a media segment is received, it also parses
as much of it as is available, returning all the coded frames found so
far. SourceBufferProcessor then tells TrackBufferDemuxer to remove any
overlapping frames and insert the new ones.
TrackBufferDemuxer implements the Demuxer interface in terms of the
coded frame store maintained by the SourceBufferProcessor. It returns
the frames in decode order when requested by a data provider. When a
is needed, it finds the keyframe prior to the target timestamp, and
checks that there are no gaps in data up to the target timestamp. If
there are any gaps, it blocks until the gaps are gone.
These steps are the best definition we have for how the ready state
should be set, and it seems to be reasonable to apply to plain file
playback as well.
Since our file demuxers are hardcoded to return the entire duration as
buffered, the ready state immediately progresses to HAVE_CURRENT_DATA.
This will probably change once we can check the demuxers for buffered
data.
d146adf made the fetch callbacks use the media element via weak
references. This caused the `error` event not to fire on media elements
that are detached from the document and go out of scope, if the GC got
to them before the fetch completed.
Instead of relying on weak references in the callbacks, we can stop the
ongoing fetch when the document becomes inactive to allow it to be GCed
after that point. By storing the FetchData on the media element, we're
able to resume the fetch where it left off if the document becomes
active again.
We could potentially figure out a way to make elements with no event
handlers and no parent stop their fetches in order to be GCed sooner,
but that is probably a bit fiddly, so may not be worth it for now.
Fixes a rare flake in WPT's `html/semantics/embedded-content/media-
elements/error-codes/error.html` test. A test to force the bug using
`Internals::gc()` has been added.
PlaybackManager's ref counting was only used to keep it alive in a few
callbacks. Instead, the callbacks can use weak references that can only
be used from the thread that the PlaybackManager was created on, to
ensure that the PlaybackManager can't be destroyed while being
accessed.
This ensures that:
- The PlaybackManager is destroyed immediately when it is reassigned
by HTMLMediaElement
- No callbacks are invoked after that point
This fixes the crash initially being addressed by #8081. The test from
that PR has been included as a regression test.
Instead of using a custom paintable to draw the controls for video and
audio elements, we build them out of plain old HTML elements within a
shadow root.
This required a few hacks in the previous commits in order to allow a
replaced element to host children within a shadow root, but it's
fairly self-contained.
A big benefit is that we can drive all the UI updates off of plain old
DOM events (except the play button overlay on videos, which uses the
video element representation), so we can test our media and input event
handling more thoroughly. :^)
The control bar visibility is now more similar to how other browsers
handle it. It will show upon hovering over the element, but if the
cursor is kept still for more than a second, it will hide again. While
dragging, the controls remain visible, and will then hide after the
mouse button is released.
The icons have been redesigned from scratch, and the mute icon now
visualizes the volume level along with indicating the mute state.
These are the same code, so we may as well move them up the chain. This
becomes useful in a later commit, where it will be used to rewrite
inline-flow to inline-block for layout of shadow DOM.
This should help avoid the footgun of forgetting to check for null on
m_fetch_controller. We had missed this check when firing off an error
due to an unsupported format in the PlaybackManager, so we could call
stop_fetch() on a null pointer if the download had completed already.
Publish new video frames to an ExternalContentSource, and switch
VideoPaintable from draw_scaled_immutable_bitmap to
draw_external_content.
Because DrawExternalContent reads the latest bitmap at replay time,
frame-only updates (no timeline or control change) now call
set_needs_display(InvalidateDisplayList::No) — skipping display list
rebuilds entirely. This addresses problem 2 from the previous commit.
Using the response's header over our own request's byte range is likely
to be a little more reliable for piecing the data together in the
stream, in case the server decides to give us a slightly different
range than we requested. The media element spec requires that the
Content-Range parses correctly anyway, so we should make use of the
values we get out of it.
This makes media playback able to start without having to wait for data
to sequentially download, especially when seeking the media to a
timestamp residing in data that hasn't loaded yet.
Initially, the HTMLMediaElement will request the file without range a
range request. Then, if the IncrementallyPopulatedStream finds that it
needs data that is not yet available, it will decide whether to wait
for that data to be received through the current request, or start a
new request that is closer to the required data.
In this commit, it assumes that the server will support range requests.
Refactor the FFmpeg and Matroska demuxers to consume data through
`IncrementallyPopulatedStream::Cursor` instead of a pointer to fully
buffered.
This change establishes a new rule: each track must be initialized with
its own cursor. Data providers now explicitly create a per-track context
via `Demuxer::create_context_for_track(track, cursor)`, and own pointer
to that cursor. In the upcoming changes, holding the cursor in the
provider would allow to signal "cancel blocking reads" so an
in-flight seek can fail immediately when a newer seek request arrives.
Demuxer creation and track+duration extraction are moved to a separate
thread so that the media data byte buffer is no longer accessed from the
main thread. This will be important once the buffer is populated
incrementally, as having the main thread both populate and read from the
same buffer could easily lead to deadlocks. Aside from that, moving
demuxer creation off the main thread helps to be more responsive.
`VideoDataProvider` and `AudioDataProvider` now accept the main thread
event loop pointer as they are constructed from the thread responsible
for demuxer creation.
Prevents observably calling Trusted Types, which can run arbitrary JS,
cause crashes due to use of MUST and allow arbitrary JS to modify
internal elements.
In cases where a script assigns `x = video.currentTime = y`, we are
expected to have a result of `x === y`, even if the video's duration
is less than y.
According to the spec, this happens because the official playback
position is set to `y` in this case, but since we are following
implementations in making `currentTime` immediately return the position
on the valid media timeline, we have to specifically return the
unchanged value from the setter.
See: https://github.com/whatwg/html/issues/11773
This allows playback to restart when playing is requested after the end
of playback was reached while loop was disabled, regardless of whether
loop is then subsequently enabled.
This matches other browsers' implementations, but differs from the spec
in how the ended attribute is handled.
See: https://github.com/whatwg/html/issues/11775
With this commit, all PlaybackManager can do is autoplay a file from
start to finish, with no pausing or seeking functionality.
All audio playback functionality has been removed from HTMLMediaElement
and HTMLAudioElement in anticipation of PlaybackManager taking that
over, for both audio-only and audio/video.
This allows the media paintable to be redrawn when the media element is
paused. We change the color of some components on hover, but if the
media was paused, that would not be rendered. This wasn't an issue while
the media is playing because the update to the timeline would take care
of redrawing the paintable.
Currently, this metadata is only provided on the insertion steps,
though I believe it would be useful to extend to the other cases
as well. This metadata can aid in making optimizations for these
steps by providing extra context into the type of change which
was made on the child.
This commit begins to implement the track processing model. When the
`src` attribute is updated, we now fetch the given source file.
Currently, we always fire an `error` event once fetching is completed,
as we don't support processing the fetched data.
Resulting in a massive rename across almost everywhere! Alongside the
namespace change, we now have the following names:
* JS::NonnullGCPtr -> GC::Ref
* JS::GCPtr -> GC::Ptr
* JS::HeapFunction -> GC::Function
* JS::CellImpl -> GC::Cell
* JS::Handle -> GC::Root
We currently have 2 virtual methods to inform DOM::Element subclasses
when an attribute has changed, one of which is spec-compliant. This
patch removes the non-compliant variant.
