Media element tasks are queued as element tasks. The element can move
to another document before the queued task runs.
Abort the task if the media element's current document is no longer
fully active. This preserves the media resource fetch invariant.
Add reduced crash coverage for adoption into an inactive document.
The allowed range for now is 0.0-64.0. If that turns out to be too wide
a range, we'll reduce it.
Chromium and Firefox are more restrictive, only allowing between a
minimum non-zero number and 16.0. They also allow 0.0.
The (invocation of) `on_unsupported_format_error` was already guarded
against this but it was possible for us to have moved on to the next
candidate in the interim.
Fixes a crash on https://chatgpt.com
Also, update the ready state when the progress event is fired.
Otherwise, we won't autoplay if 5 seconds of data aren't available in
on_metadata_parsed().
Represent WebIDL C++ types with a single CppType model that tracks
nullability, optional presence, and contained storage.
GC-like values now use GC::Ref/GC::Ptr directly, while containers choose
"plain", "Root", or "Conservative" container types depending on what
they contain. For example, sequence<Element> becomes a RootVector of
GC::Ref values, while sequence<SomeDictionary> becomes a
ConservativeVector only when the dictionary contains GC-like values.
This moves the generated bindings away from wrapping GC values in
GC::Root by default.
This has broad fallout as the types passed to interfaces for GC
objects changes almost fully across the board.
This ensures that we're only beginning a remote fetch when the document
is active, which the observer callbacks already assume. Combined with
the next commit, this will help ensure that the fix preventing the
VERIFY in the document_became_active callback from being hit works.
Similar to GC::Root<T>, make GC::RootVector<T> constructible without
explicitly passing a Heap.
This is implemented by having RootVectorBase use GC::Heap::the() for
heap-free construction.
Future compositor-process work needs to push video frame updates without
going through VideoFrameSource. That object cannot be shared through
IPC, so video display-list resources now use stable VideoFrameResourceId
values and the current frame is sent through explicit resource/update
commands.
Tear down the MediaControls (and its Core::Timer-driven hover handler)
during HTMLMediaElement finalization. Otherwise, between weak-clearing
and incremental sweep destroying the element, a queued hover timer
event can still fire and trip a VERIFY against an already-cleared
GC::Weak reference to a shadow tree node.
Previously we were inconsistent by generating code for enum definitions
but not generating code for dictionaries. With future changes to the
IDL generator to expose helpers to convert to and from IDL values
this produced circular depdendencies. To solve this problem, also
generate the dictionary definitions in bindings headers.
A video element should record video as video, not as generic external
bitmap content. Add VideoFrameSource and a dedicated display-list
command so the display-list player receives the current
Media::VideoFrame directly.
The Skia player can now upload YUV pixmaps from the frame when a GPU
context is available, without teaching the ordinary ImmutableBitmap
image cache about media formats. If GPU upload is unavailable, the
fallback explicitly converts the frame through YUVData::to_bitmap().
This gives video painting a clear extension point for future frame
backends, such as hardware frames or other planar formats, while
keeping bitmap drawing focused on immutable pixel snapshots.
Decoded video frames should own their planar YUV data and color space
directly. Keeping that storage behind ImmutableBitmap gave a
still-image abstraction media-specific behavior and made calls like
bitmap() potentially allocate and convert a whole video frame.
Move YUV ownership into Media::VideoFrame, where the lifetime naturally
follows media playback, and remove the YUV-backed mode from
ImmutableBitmap. This commit intentionally keeps the visible Web paint
path on ExternalContentSource by converting the current frame back to
an ImmutableBitmap where Web still expects one.
Callers that need pixels now ask the frame to convert explicitly. That
preserves behavior for canvas and bitmap consumers while making the
expensive YUV-to-pixel path visible at the call site instead of
hiding it behind ImmutableBitmap::bitmap().
The display queue used TimedImage even though the media pipeline is
selecting decoded video frames. That naming hid the real object being
handed from LibMedia to Web and kept the queue interface coupled to
bitmap-style painting.
Rename the wrapper to TimedVideoFrame and pass ref-counted VideoFrame
objects through the provider and display sink. Web still reads the
ImmutableBitmap from the frame for painting in this commit, so rendered
output and conversion behavior stay the same while the playback-facing
interfaces become frame-shaped.
This tightens the implementation of video element sizing to the spec by
implementing two spec concepts:
- The media resource's natural width and height, and
- The video element's natural width and height.
The element's natural dimensions change based on the representation,
which has many inputs, so update checks are triggered from many
locations.
The resize event is fired when the media resource's natural dimensions
change, and the layout is invalidated if the element's natural
dimensions change.
Tests for a few important resize triggers have been added.
This state will indicate to the media element that it's not guaranteed
to have a frame yet, for the purposes of determining the ready state.
JavaScript should be sure that video elements with a ready state of
HAVE_CURRENT_DATA or greater represent the current video frame already.
To allow the state to be exited if audio is disabled, audio tracks are
now only added to the buffering set on enable if the audio sink exists,
since without the sink starting the data provider, it will never be
removed.
This is a step towards making video ref tests.
This allows us to differentiate between having no data available yet,
having current data, and having future data. The main purpose of this
is to allow a new starting state to explicitly force HAVE_METADATA
instead of >= HAVE_CURRENT_DATA.
Note that the SeekingStateHandler returns Current instead of None. This
is deliberate, since the buffered ranges from the demuxer(s) can be
used to inform whether the possibly-current data is actually available
at the seek target.
A while ago, we removed the relayout upon rendering a new frame. In
doing so, it became possible for the layout to remain stale after the
video metadata had loaded, leaving the video drawn in a 0x0 box.
Otherwise, the PlaybackManager may get stuck waiting for enough data to
read the metadata and call on_metadata_parsed.
This is unfortunately difficult to test without direct control over the
fetching process, but it could cause flakes in tests that wait for
loadeddata.
Previously, the LibWeb bindings generator would output multiple per
interface files like Prototype/Constructor/Namespace/GlobalMixin
depending on the contents of that IDL file.
This complicates the build system as it means that it does not know
what files will be generated without knowledge of the contents of that
IDL file.
Instead, for each IDL file only generate a single Bindings/<IDLFile>.h
and Bindings/<IDLFile>.cpp.
This ensures that when we're switching from one video track to another,
we don't end up exiting buffering/seeking early due to no tracks being
enabled.
We could apparently update the ready state within time_marches_on()
called by set_ready_state(), which could result in triggering autoplay
twice for the same state transition.
The case of an unsupported format error wasn't covered for this,
meaning that it could crash if the fetch completed successfully after
the fetch was cancelled due to such an error.
A crash test is included for this issue, using an echo of a large
corrupted WebM file to ensure that the fetch completes after media
init.
Audio output on macOS was consuming Core Audio resources until the
PlaybackStream creation took well over the timeout for some tests.
This was observed in media-source-buffered.html, where it would time
out due to the long-running callback on the main thread to create the
PlaybackStream for AudioMixingSink.
However, the AudioUnit init should definitely not be blocking the main
thread, so I've added a FIXME there.
If we fire the error event synchronously within the on_error callback,
then we'll end up destroying the PlaybackManager inside its own
callback and crash. Instead, queue a task to execute the error steps.
This could happen with or without MSE, but I observed it occurring on
YouTube with MSE when we hit a decoding error, since they immediately
try another source when an error is reported.
The segments are parsed for the SourceBufferProcessor by the
WebMByteStreamParser. It parses the initialization segment to update
its internal set of tracks, then SourceBufferProcessor/SourceBuffer set
them up for playback. When a media segment is received, it also parses
as much of it as is available, returning all the coded frames found so
far. SourceBufferProcessor then tells TrackBufferDemuxer to remove any
overlapping frames and insert the new ones.
TrackBufferDemuxer implements the Demuxer interface in terms of the
coded frame store maintained by the SourceBufferProcessor. It returns
the frames in decode order when requested by a data provider. When a
is needed, it finds the keyframe prior to the target timestamp, and
checks that there are no gaps in data up to the target timestamp. If
there are any gaps, it blocks until the gaps are gone.
These steps are the best definition we have for how the ready state
should be set, and it seems to be reasonable to apply to plain file
playback as well.
Since our file demuxers are hardcoded to return the entire duration as
buffered, the ready state immediately progresses to HAVE_CURRENT_DATA.
This will probably change once we can check the demuxers for buffered
data.
Removing a display risks triggering callbacks on the playback manager
that may cause a recursive GC. This wasn't having any effect since the
playback manager became an OwnPtr.
...giving tracks a kind attribute, and renaming name to label.
Demuxers will need to determine the kind attribute, since the spec for
sourcing tracks requires us to select based on info we don't expose.
In `::spin_processing_tasks_with_source_until()`, we would first take a
set of tasks based on a filter, and then run them one by one. If there
was more than one task matched and put in that vector, they could
interfere with each other's runnability by making later tasks
permanently unrunnable.
The `::take_tasks_matching()` API is a footgun - remove it in favor of
an API that takes tasks one by one, performing the runnability check
just in time.
Since we know whether we're buffering from the PlaybackManager state,
let's use that to update the ready state. This ensures that when we set
the ready state to HAVE_CURRENT_DATA, we actually have a frame.
Setting the ready state to HAVE_ENOUGH_DATA should probably still be
further conditioned on the buffer size in IncrementallyPopulatedStream,
but this is still an improvement for now.
This workaround avoids a crash that occurs when various fetching
processes (in the media element) are quickly started and canceled.
Although a better solution would be to actually remove the body
callbacks when the fetch is stopped, this works for now.
Rename Document::set_needs_display() to set_needs_repaint() and make it
private. External callers must now go through Node/Paintable which
route the request to the document internally.
Fix one existing misuse in AnimationEffect that was calling
document-level set_needs_display() instead of routing through the
target element's paintable.
This is preparation for per-paintable display list command caching:
repaint requests must go through specific paintables so their cached
command lists can be invalidated.
d146adf made the fetch callbacks use the media element via weak
references. This caused the `error` event not to fire on media elements
that are detached from the document and go out of scope, if the GC got
to them before the fetch completed.
Instead of relying on weak references in the callbacks, we can stop the
ongoing fetch when the document becomes inactive to allow it to be GCed
after that point. By storing the FetchData on the media element, we're
able to resume the fetch where it left off if the document becomes
active again.
We could potentially figure out a way to make elements with no event
handlers and no parent stop their fetches in order to be GCed sooner,
but that is probably a bit fiddly, so may not be worth it for now.
Fixes a rare flake in WPT's `html/semantics/embedded-content/media-
elements/error-codes/error.html` test. A test to force the bug using
`Internals::gc()` has been added.
