Stowage/ladybird
2
0
Fork 0
mirror of https://github.com/LadybirdBrowser/ladybird.git synced 2025-11-07 16:51:01 +00:00
Code Issues Projects Releases Packages Wiki Activity
ladybird/Userland/Libraries/LibVideo/PlaybackManager.cpp

639 lines
25 KiB
C++
Raw Normal View History

LibVideo: Add PlaybackManager to load and decode videos This file will be the basis for abstracting away the out-of-thread or later out-of-process decoding from applications displaying videos. For now, the demuxer is hardcoded to be MatroskaParser, since that is all we support so far. The demuxer should later be selected based on the file header. The playback and decoding are currently all done on one thread using timers. The design of the code is such that adding threading should be trivial, at least based on an earlier version of the code. For now, though, it's better that this runs in one thread, as the multithreaded approach causes the Video Player to lock up permanently after a few frames are decoded.
2022-10-29 18:16:14 -05:00
/*
* Copyright (c) 2022, Gregory Bertilson <zaggy1024@gmail.com>
*
* SPDX-License-Identifier: BSD-2-Clause
*/
#include <AK/Format.h>
#include <LibCore/Timer.h>
LibVideo: Reorganize demuxer file hierarchy and rename Matroska files As new demuxers are added, this will get quite full of files, so it'll be good to have a separate folder for these. To avoid too many chained namespaces, the Containers subdirectory is not also a namespace, but the Matroska folder is for the sake of separating the multiple classes for parsed information entering the Video namespace.
2022-11-09 19:47:56 -06:00
#include <LibVideo/Containers/Matroska/MatroskaDemuxer.h>
LibVideo: Add PlaybackManager to load and decode videos This file will be the basis for abstracting away the out-of-thread or later out-of-process decoding from applications displaying videos. For now, the demuxer is hardcoded to be MatroskaParser, since that is all we support so far. The demuxer should later be selected based on the file header. The playback and decoding are currently all done on one thread using timers. The design of the code is such that adding threading should be trivial, at least based on an earlier version of the code. For now, though, it's better that this runs in one thread, as the multithreaded approach causes the Video Player to lock up permanently after a few frames are decoded.
2022-10-29 18:16:14 -05:00
#include <LibVideo/VP9/Decoder.h>
#include "PlaybackManager.h"
namespace Video {
LibVideo/PlaybackManager: Organize playback states into virtual classes Storing playback states in virtual classes allows the behavior to be much more clearly written. Each `PlaybackStateHandler` subclass can implement some event-handling functions to model their behavior, and has functions to change its parent PlaybackManager's state to any other state. This will allow expanding the functionality of playback in the future, for example to allow skipping a single frame forward/backward.
2023-02-06 01:24:33 -06:00
#define TRY_OR_FATAL_ERROR(expression) \
({ \
AK+Everywhere: Do not implicitly copy variables in TRY macros For example, consider cases where we want to propagate errors only in specific instances: auto result = read_data(); // something like ErrorOr<ByteBuffer> if (result.is_error() && result.error().code() != EINTR) continue; auto bytes = TRY(result); The TRY invocation will currently copy the byte buffer when the expression (in this case, just a local variable) is stored into _temporary_result. This patch binds the expression to a reference to prevent such copies. In less trival invocations (such as TRY(some_function()), this will incur only temporary lifetime extensions, i.e. no functional change.
2023-02-09 13:27:43 -05:00
auto&& _fatal_expression = (expression); \
LibVideo/PlaybackManager: Organize playback states into virtual classes Storing playback states in virtual classes allows the behavior to be much more clearly written. Each `PlaybackStateHandler` subclass can implement some event-handling functions to model their behavior, and has functions to change its parent PlaybackManager's state to any other state. This will allow expanding the functionality of playback in the future, for example to allow skipping a single frame forward/backward.
2023-02-06 01:24:33 -06:00
if (_fatal_expression.is_error()) { \
dispatch_fatal_error(_fatal_expression.release_error()); \
return; \
} \
static_assert(!::AK::Detail::IsLvalueReference<decltype(_fatal_expression.release_value())>, \
"Do not return a reference from a fallible expression"); \
_fatal_expression.release_value(); \
})
LibVideo: Remove hook to override LibVideo's playback timers This was added to allow Ladybird to override the timers with Qt timers.
2023-04-25 09:02:24 -04:00
DecoderErrorOr<NonnullOwnPtr<PlaybackManager>> PlaybackManager::from_file(StringView filename)
LibVideo: Add PlaybackManager to load and decode videos This file will be the basis for abstracting away the out-of-thread or later out-of-process decoding from applications displaying videos. For now, the demuxer is hardcoded to be MatroskaParser, since that is all we support so far. The demuxer should later be selected based on the file header. The playback and decoding are currently all done on one thread using timers. The design of the code is such that adding threading should be trivial, at least based on an earlier version of the code. For now, though, it's better that this runs in one thread, as the multithreaded approach causes the Video Player to lock up permanently after a few frames are decoded.
2022-10-29 18:16:14 -05:00
{
LibVideo: Add a factory to create a PlaybackManager from in-memory data The demuxer already has such a factory, so this just exposes the same factory in the PlaybackManager.
2023-04-08 21:22:15 -04:00
auto demuxer = TRY(Matroska::MatroskaDemuxer::from_file(filename));
LibVideo: Remove hook to override LibVideo's playback timers This was added to allow Ladybird to override the timers with Qt timers.
2023-04-25 09:02:24 -04:00
return create_with_demuxer(move(demuxer));
LibVideo: Add a factory to create a PlaybackManager from in-memory data The demuxer already has such a factory, so this just exposes the same factory in the PlaybackManager.
2023-04-08 21:22:15 -04:00
}
LibVideo: Add `PlaybackManager::from_mapped_file` This allows us to create a PlaybackManager from a file which has already been mapped, instead of passing a file name. This means that anyone who uses `PlaybackManager` can now use LibFSAC :)
2023-05-11 12:25:52 +01:00
DecoderErrorOr<NonnullOwnPtr<PlaybackManager>> PlaybackManager::from_mapped_file(NonnullRefPtr<Core::MappedFile> mapped_file)
{
auto demuxer = TRY(Matroska::MatroskaDemuxer::from_mapped_file(move(mapped_file)));
return create_with_demuxer(move(demuxer));
}
LibVideo: Remove hook to override LibVideo's playback timers This was added to allow Ladybird to override the timers with Qt timers.
2023-04-25 09:02:24 -04:00
DecoderErrorOr<NonnullOwnPtr<PlaybackManager>> PlaybackManager::from_data(ReadonlyBytes data)
LibVideo: Add a factory to create a PlaybackManager from in-memory data The demuxer already has such a factory, so this just exposes the same factory in the PlaybackManager.
2023-04-08 21:22:15 -04:00
{
auto demuxer = TRY(Matroska::MatroskaDemuxer::from_data(data));
LibVideo: Remove hook to override LibVideo's playback timers This was added to allow Ladybird to override the timers with Qt timers.
2023-04-25 09:02:24 -04:00
return create_with_demuxer(move(demuxer));
LibVideo: Add a factory to create a PlaybackManager from in-memory data The demuxer already has such a factory, so this just exposes the same factory in the PlaybackManager.
2023-04-08 21:22:15 -04:00
}
LibVideo: Remove hook to override LibVideo's playback timers This was added to allow Ladybird to override the timers with Qt timers.
2023-04-25 09:02:24 -04:00
DecoderErrorOr<NonnullOwnPtr<PlaybackManager>> PlaybackManager::create_with_demuxer(NonnullOwnPtr<Demuxer> demuxer)
LibVideo: Add a factory to create a PlaybackManager from in-memory data The demuxer already has such a factory, so this just exposes the same factory in the PlaybackManager.
2023-04-08 21:22:15 -04:00
{
LibVideo: Read Matroska lazily so that large files can start quickly The Demuxer class was changed to return errors for more functions so that all of the underlying reading can be done lazily. Other than that, the demuxer interface is unchanged, and only the underlying reader was modified. The MatroskaDocument class is no more, and MatroskaReader's getter functions replace it. Every MatroskaReader getter beyond the Segment element's position is parsed lazily from the file as needed. This means that all getter functions can return DecoderErrors which must be handled by callers.
2022-11-11 17:14:27 -06:00
auto video_tracks = TRY(demuxer->get_tracks_for_type(TrackType::Video));
LibVideo: Add PlaybackManager to load and decode videos This file will be the basis for abstracting away the out-of-thread or later out-of-process decoding from applications displaying videos. For now, the demuxer is hardcoded to be MatroskaParser, since that is all we support so far. The demuxer should later be selected based on the file header. The playback and decoding are currently all done on one thread using timers. The design of the code is such that adding threading should be trivial, at least based on an earlier version of the code. For now, though, it's better that this runs in one thread, as the multithreaded approach causes the Video Player to lock up permanently after a few frames are decoded.
2022-10-29 18:16:14 -05:00
if (video_tracks.is_empty())
return DecoderError::with_description(DecoderErrorCategory::Invalid, "No video track is present"sv);
auto track = video_tracks[0];
dbgln_if(PLAYBACK_MANAGER_DEBUG, "Selecting video track number {}", track.identifier());
LibVideo: Remove hook to override LibVideo's playback timers This was added to allow Ladybird to override the timers with Qt timers.
2023-04-25 09:02:24 -04:00
return make<PlaybackManager>(demuxer, track, make<VP9::Decoder>());
LibVideo: Add PlaybackManager to load and decode videos This file will be the basis for abstracting away the out-of-thread or later out-of-process decoding from applications displaying videos. For now, the demuxer is hardcoded to be MatroskaParser, since that is all we support so far. The demuxer should later be selected based on the file header. The playback and decoding are currently all done on one thread using timers. The design of the code is such that adding threading should be trivial, at least based on an earlier version of the code. For now, though, it's better that this runs in one thread, as the multithreaded approach causes the Video Player to lock up permanently after a few frames are decoded.
2022-10-29 18:16:14 -05:00
}
LibVideo: Remove hook to override LibVideo's playback timers This was added to allow Ladybird to override the timers with Qt timers.
2023-04-25 09:02:24 -04:00
PlaybackManager::PlaybackManager(NonnullOwnPtr<Demuxer>& demuxer, Track video_track, NonnullOwnPtr<VideoDecoder>&& decoder)
LibVideo+VideoPlayer: Convert playback event handler to callbacks To pass events from LibVideo's PlaybackManager to interested parties, we currently dispatch Core::Event objects that outside callers listen for. Dispatching events in this manner rely on a Core::EventLoop. In order to use PlaybackManager from LibWeb, change this mechanism to instead use a set of callbacks to inform callers of events.
2023-04-08 19:18:38 -04:00
: m_demuxer(move(demuxer))
LibVideo: Add PlaybackManager to load and decode videos This file will be the basis for abstracting away the out-of-thread or later out-of-process decoding from applications displaying videos. For now, the demuxer is hardcoded to be MatroskaParser, since that is all we support so far. The demuxer should later be selected based on the file header. The playback and decoding are currently all done on one thread using timers. The design of the code is such that adding threading should be trivial, at least based on an earlier version of the code. For now, though, it's better that this runs in one thread, as the multithreaded approach causes the Video Player to lock up permanently after a few frames are decoded.
2022-10-29 18:16:14 -05:00
, m_selected_video_track(video_track)
, m_decoder(move(decoder))
, m_frame_queue(make<VideoFrameQueue>())
LibVideo: Remove Starting playback state in favor of Seeking to zero
2023-04-12 00:42:14 -05:00
, m_playback_handler(make<SeekingStateHandler>(*this, false, Time::zero(), SeekMode::Fast))
LibVideo: Add PlaybackManager to load and decode videos This file will be the basis for abstracting away the out-of-thread or later out-of-process decoding from applications displaying videos. For now, the demuxer is hardcoded to be MatroskaParser, since that is all we support so far. The demuxer should later be selected based on the file header. The playback and decoding are currently all done on one thread using timers. The design of the code is such that adding threading should be trivial, at least based on an earlier version of the code. For now, though, it's better that this runs in one thread, as the multithreaded approach causes the Video Player to lock up permanently after a few frames are decoded.
2022-10-29 18:16:14 -05:00
{
LibVideo: Remove hook to override LibVideo's playback timers This was added to allow Ladybird to override the timers with Qt timers.
2023-04-25 09:02:24 -04:00
m_present_timer = Core::Timer::create_single_shot(0, [&] { timer_callback(); }).release_value_but_fixme_should_propagate_errors();
m_decode_timer = Core::Timer::create_single_shot(0, [&] { on_decode_timer(); }).release_value_but_fixme_should_propagate_errors();
LibVideo+VideoPlayer: Convert playback event handler to callbacks To pass events from LibVideo's PlaybackManager to interested parties, we currently dispatch Core::Event objects that outside callers listen for. Dispatching events in this manner rely on a Core::EventLoop. In order to use PlaybackManager from LibWeb, change this mechanism to instead use a set of callbacks to inform callers of events.
2023-04-08 19:18:38 -04:00
LibVideo/PlaybackManager: Organize playback states into virtual classes Storing playback states in virtual classes allows the behavior to be much more clearly written. Each `PlaybackStateHandler` subclass can implement some event-handling functions to model their behavior, and has functions to change its parent PlaybackManager's state to any other state. This will allow expanding the functionality of playback in the future, for example to allow skipping a single frame forward/backward.
2023-02-06 01:24:33 -06:00
TRY_OR_FATAL_ERROR(m_playback_handler->on_enter());
LibVideo: Add PlaybackManager to load and decode videos This file will be the basis for abstracting away the out-of-thread or later out-of-process decoding from applications displaying videos. For now, the demuxer is hardcoded to be MatroskaParser, since that is all we support so far. The demuxer should later be selected based on the file header. The playback and decoding are currently all done on one thread using timers. The design of the code is such that adding threading should be trivial, at least based on an earlier version of the code. For now, though, it's better that this runs in one thread, as the multithreaded approach causes the Video Player to lock up permanently after a few frames are decoded.
2022-10-29 18:16:14 -05:00
}
LibVideo: Remove hook to override LibVideo's playback timers This was added to allow Ladybird to override the timers with Qt timers.
2023-04-25 09:02:24 -04:00
PlaybackManager::~PlaybackManager() = default;
LibVideo: Add PlaybackManager to load and decode videos This file will be the basis for abstracting away the out-of-thread or later out-of-process decoding from applications displaying videos. For now, the demuxer is hardcoded to be MatroskaParser, since that is all we support so far. The demuxer should later be selected based on the file header. The playback and decoding are currently all done on one thread using timers. The design of the code is such that adding threading should be trivial, at least based on an earlier version of the code. For now, though, it's better that this runs in one thread, as the multithreaded approach causes the Video Player to lock up permanently after a few frames are decoded.
2022-10-29 18:16:14 -05:00
void PlaybackManager::resume_playback()
{
LibVideo/PlaybackManager: Organize playback states into virtual classes Storing playback states in virtual classes allows the behavior to be much more clearly written. Each `PlaybackStateHandler` subclass can implement some event-handling functions to model their behavior, and has functions to change its parent PlaybackManager's state to any other state. This will allow expanding the functionality of playback in the future, for example to allow skipping a single frame forward/backward.
2023-02-06 01:24:33 -06:00
dbgln_if(PLAYBACK_MANAGER_DEBUG, "Resuming playback.");
TRY_OR_FATAL_ERROR(m_playback_handler->play());
LibVideo: Add PlaybackManager to load and decode videos This file will be the basis for abstracting away the out-of-thread or later out-of-process decoding from applications displaying videos. For now, the demuxer is hardcoded to be MatroskaParser, since that is all we support so far. The demuxer should later be selected based on the file header. The playback and decoding are currently all done on one thread using timers. The design of the code is such that adding threading should be trivial, at least based on an earlier version of the code. For now, though, it's better that this runs in one thread, as the multithreaded approach causes the Video Player to lock up permanently after a few frames are decoded.
2022-10-29 18:16:14 -05:00
}
void PlaybackManager::pause_playback()
{
LibVideo/PlaybackManager: Organize playback states into virtual classes Storing playback states in virtual classes allows the behavior to be much more clearly written. Each `PlaybackStateHandler` subclass can implement some event-handling functions to model their behavior, and has functions to change its parent PlaybackManager's state to any other state. This will allow expanding the functionality of playback in the future, for example to allow skipping a single frame forward/backward.
2023-02-06 01:24:33 -06:00
dbgln_if(PLAYBACK_MANAGER_DEBUG, "Pausing playback.");
if (!m_playback_handler->is_playing())
warnln("Cannot pause.");
TRY_OR_FATAL_ERROR(m_playback_handler->pause());
LibVideo: Add PlaybackManager to load and decode videos This file will be the basis for abstracting away the out-of-thread or later out-of-process decoding from applications displaying videos. For now, the demuxer is hardcoded to be MatroskaParser, since that is all we support so far. The demuxer should later be selected based on the file header. The playback and decoding are currently all done on one thread using timers. The design of the code is such that adding threading should be trivial, at least based on an earlier version of the code. For now, though, it's better that this runs in one thread, as the multithreaded approach causes the Video Player to lock up permanently after a few frames are decoded.
2022-10-29 18:16:14 -05:00
}
Time PlaybackManager::current_playback_time()
{
LibVideo/PlaybackManager: Organize playback states into virtual classes Storing playback states in virtual classes allows the behavior to be much more clearly written. Each `PlaybackStateHandler` subclass can implement some event-handling functions to model their behavior, and has functions to change its parent PlaybackManager's state to any other state. This will allow expanding the functionality of playback in the future, for example to allow skipping a single frame forward/backward.
2023-02-06 01:24:33 -06:00
return m_playback_handler->current_time();
LibVideo: Add PlaybackManager to load and decode videos This file will be the basis for abstracting away the out-of-thread or later out-of-process decoding from applications displaying videos. For now, the demuxer is hardcoded to be MatroskaParser, since that is all we support so far. The demuxer should later be selected based on the file header. The playback and decoding are currently all done on one thread using timers. The design of the code is such that adding threading should be trivial, at least based on an earlier version of the code. For now, though, it's better that this runs in one thread, as the multithreaded approach causes the Video Player to lock up permanently after a few frames are decoded.
2022-10-29 18:16:14 -05:00
}
Time PlaybackManager::duration()
{
LibVideo: Read Matroska lazily so that large files can start quickly The Demuxer class was changed to return errors for more functions so that all of the underlying reading can be done lazily. Other than that, the demuxer interface is unchanged, and only the underlying reader was modified. The MatroskaDocument class is no more, and MatroskaReader's getter functions replace it. Every MatroskaReader getter beyond the Segment element's position is parsed lazily from the file as needed. This means that all getter functions can return DecoderErrors which must be handled by callers.
2022-11-11 17:14:27 -06:00
auto duration_result = m_demuxer->duration();
if (duration_result.is_error())
LibVideo: Deduplicate logic for dispatching video frame queue items Previously we had dispatch_decoder_error and on_decoder_error serving the same function, with one not handling the end of stream properly. There is also a new function to dispatch either an error or a frame to the owner of this playback manager, so that PlaybackStateHandlers don't have to duplicate this logic.
2023-02-12 00:44:16 -06:00
dispatch_decoder_error(duration_result.release_error());
LibVideo: Read Matroska lazily so that large files can start quickly The Demuxer class was changed to return errors for more functions so that all of the underlying reading can be done lazily. Other than that, the demuxer interface is unchanged, and only the underlying reader was modified. The MatroskaDocument class is no more, and MatroskaReader's getter functions replace it. Every MatroskaReader getter beyond the Segment element's position is parsed lazily from the file as needed. This means that all getter functions can return DecoderErrors which must be handled by callers.
2022-11-11 17:14:27 -06:00
return duration_result.release_value();
LibVideo: Add PlaybackManager to load and decode videos This file will be the basis for abstracting away the out-of-thread or later out-of-process decoding from applications displaying videos. For now, the demuxer is hardcoded to be MatroskaParser, since that is all we support so far. The demuxer should later be selected based on the file header. The playback and decoding are currently all done on one thread using timers. The design of the code is such that adding threading should be trivial, at least based on an earlier version of the code. For now, though, it's better that this runs in one thread, as the multithreaded approach causes the Video Player to lock up permanently after a few frames are decoded.
2022-10-29 18:16:14 -05:00
}
LibVideo/PlaybackManager: Organize playback states into virtual classes Storing playback states in virtual classes allows the behavior to be much more clearly written. Each `PlaybackStateHandler` subclass can implement some event-handling functions to model their behavior, and has functions to change its parent PlaybackManager's state to any other state. This will allow expanding the functionality of playback in the future, for example to allow skipping a single frame forward/backward.
2023-02-06 01:24:33 -06:00
void PlaybackManager::dispatch_fatal_error(Error error)
LibVideo: Handle corrupted video errors without spamming dialogs No longer will the video player explode with error dialogs that then lock the user out of closing them. To avoid issues where the playback state becomes invalid when an error occurs, I've made all decoder errors pass through the frame queue. This way, when a video is corrupted, there should be no chance that the playback state becomes invalid due to setting the state to Corrupted in the event handler while a presentation event is still pending. Or at least I think that was what caused some issues I was seeing :^) This system should be a lot more robust if any future errors need to be handled.
2022-11-09 06:52:37 -06:00
{
LibVideo/PlaybackManager: Organize playback states into virtual classes Storing playback states in virtual classes allows the behavior to be much more clearly written. Each `PlaybackStateHandler` subclass can implement some event-handling functions to model their behavior, and has functions to change its parent PlaybackManager's state to any other state. This will allow expanding the functionality of playback in the future, for example to allow skipping a single frame forward/backward.
2023-02-06 01:24:33 -06:00
dbgln_if(PLAYBACK_MANAGER_DEBUG, "Encountered fatal error: {}", error.string_literal());
// FIXME: For threading, this will have to use a pre-allocated event to send to the main loop
// to be able to gracefully handle OOM.
LibVideo+VideoPlayer: Convert playback event handler to callbacks To pass events from LibVideo's PlaybackManager to interested parties, we currently dispatch Core::Event objects that outside callers listen for. Dispatching events in this manner rely on a Core::EventLoop. In order to use PlaybackManager from LibWeb, change this mechanism to instead use a set of callbacks to inform callers of events.
2023-04-08 19:18:38 -04:00
if (on_fatal_playback_error)
on_fatal_playback_error(move(error));
LibVideo/PlaybackManager: Organize playback states into virtual classes Storing playback states in virtual classes allows the behavior to be much more clearly written. Each `PlaybackStateHandler` subclass can implement some event-handling functions to model their behavior, and has functions to change its parent PlaybackManager's state to any other state. This will allow expanding the functionality of playback in the future, for example to allow skipping a single frame forward/backward.
2023-02-06 01:24:33 -06:00
}
LibVideo: Don't crash when a decoder error is encountered while seeking When errors are encountered by PlaybackManager, it attempts to switch states to either Stopped or Corrupted. However, that causes it to set the last presentation media time to the current playback time while the last presentation time is unexpectedly negative because the seek never ended. Ending the seek before the state changes to Stopped or Corrupted prevents this situation from happening.
2022-11-27 00:45:14 -06:00
LibVideo: Deduplicate logic for dispatching video frame queue items Previously we had dispatch_decoder_error and on_decoder_error serving the same function, with one not handling the end of stream properly. There is also a new function to dispatch either an error or a frame to the owner of this playback manager, so that PlaybackStateHandlers don't have to duplicate this logic.
2023-02-12 00:44:16 -06:00
void PlaybackManager::dispatch_decoder_error(DecoderError error)
LibVideo/PlaybackManager: Organize playback states into virtual classes Storing playback states in virtual classes allows the behavior to be much more clearly written. Each `PlaybackStateHandler` subclass can implement some event-handling functions to model their behavior, and has functions to change its parent PlaybackManager's state to any other state. This will allow expanding the functionality of playback in the future, for example to allow skipping a single frame forward/backward.
2023-02-06 01:24:33 -06:00
{
LibVideo: Handle corrupted video errors without spamming dialogs No longer will the video player explode with error dialogs that then lock the user out of closing them. To avoid issues where the playback state becomes invalid when an error occurs, I've made all decoder errors pass through the frame queue. This way, when a video is corrupted, there should be no chance that the playback state becomes invalid due to setting the state to Corrupted in the event handler while a presentation event is still pending. Or at least I think that was what caused some issues I was seeing :^) This system should be a lot more robust if any future errors need to be handled.
2022-11-09 06:52:37 -06:00
switch (error.category()) {
case DecoderErrorCategory::EndOfStream:
LibVideo: Only print non-fatal playback errors when debug mode is on
2022-11-11 22:04:38 -06:00
dbgln_if(PLAYBACK_MANAGER_DEBUG, "{}", error.string_literal());
LibVideo/PlaybackManager: Organize playback states into virtual classes Storing playback states in virtual classes allows the behavior to be much more clearly written. Each `PlaybackStateHandler` subclass can implement some event-handling functions to model their behavior, and has functions to change its parent PlaybackManager's state to any other state. This will allow expanding the functionality of playback in the future, for example to allow skipping a single frame forward/backward.
2023-02-06 01:24:33 -06:00
TRY_OR_FATAL_ERROR(m_playback_handler->stop());
LibVideo: Handle corrupted video errors without spamming dialogs No longer will the video player explode with error dialogs that then lock the user out of closing them. To avoid issues where the playback state becomes invalid when an error occurs, I've made all decoder errors pass through the frame queue. This way, when a video is corrupted, there should be no chance that the playback state becomes invalid due to setting the state to Corrupted in the event handler while a presentation event is still pending. Or at least I think that was what caused some issues I was seeing :^) This system should be a lot more robust if any future errors need to be handled.
2022-11-09 06:52:37 -06:00
break;
default:
LibVideo: Only print non-fatal playback errors when debug mode is on
2022-11-11 22:04:38 -06:00
dbgln("Playback error encountered: {}", error.string_literal());
LibVideo/PlaybackManager: Organize playback states into virtual classes Storing playback states in virtual classes allows the behavior to be much more clearly written. Each `PlaybackStateHandler` subclass can implement some event-handling functions to model their behavior, and has functions to change its parent PlaybackManager's state to any other state. This will allow expanding the functionality of playback in the future, for example to allow skipping a single frame forward/backward.
2023-02-06 01:24:33 -06:00
TRY_OR_FATAL_ERROR(m_playback_handler->stop());
LibVideo+VideoPlayer: Convert playback event handler to callbacks To pass events from LibVideo's PlaybackManager to interested parties, we currently dispatch Core::Event objects that outside callers listen for. Dispatching events in this manner rely on a Core::EventLoop. In order to use PlaybackManager from LibWeb, change this mechanism to instead use a set of callbacks to inform callers of events.
2023-04-08 19:18:38 -04:00
if (on_decoder_error)
on_decoder_error(move(error));
LibVideo: Handle corrupted video errors without spamming dialogs No longer will the video player explode with error dialogs that then lock the user out of closing them. To avoid issues where the playback state becomes invalid when an error occurs, I've made all decoder errors pass through the frame queue. This way, when a video is corrupted, there should be no chance that the playback state becomes invalid due to setting the state to Corrupted in the event handler while a presentation event is still pending. Or at least I think that was what caused some issues I was seeing :^) This system should be a lot more robust if any future errors need to be handled.
2022-11-09 06:52:37 -06:00
break;
}
}
LibVideo: Deduplicate logic for dispatching video frame queue items Previously we had dispatch_decoder_error and on_decoder_error serving the same function, with one not handling the end of stream properly. There is also a new function to dispatch either an error or a frame to the owner of this playback manager, so that PlaybackStateHandlers don't have to duplicate this logic.
2023-02-12 00:44:16 -06:00
void PlaybackManager::dispatch_new_frame(RefPtr<Gfx::Bitmap> frame)
{
LibVideo+VideoPlayer: Convert playback event handler to callbacks To pass events from LibVideo's PlaybackManager to interested parties, we currently dispatch Core::Event objects that outside callers listen for. Dispatching events in this manner rely on a Core::EventLoop. In order to use PlaybackManager from LibWeb, change this mechanism to instead use a set of callbacks to inform callers of events.
2023-04-08 19:18:38 -04:00
if (on_video_frame)
on_video_frame(move(frame));
LibVideo: Deduplicate logic for dispatching video frame queue items Previously we had dispatch_decoder_error and on_decoder_error serving the same function, with one not handling the end of stream properly. There is also a new function to dispatch either an error or a frame to the owner of this playback manager, so that PlaybackStateHandlers don't have to duplicate this logic.
2023-02-12 00:44:16 -06:00
}
bool PlaybackManager::dispatch_frame_queue_item(FrameQueueItem&& item)
{
if (item.is_error()) {
dispatch_decoder_error(item.release_error());
return true;
}
LibVideo: Improve logging when PLAYBACK_MANAGER_DEBUG=on This adds a timestamp to the debug output when presenting a frame, so it can be clear the frame spacing between a presented frame and a state change. The seeking state will also now print when it early-exits if the seek point is within the current sample's duration.
2023-04-11 23:42:47 -05:00
dbgln_if(PLAYBACK_MANAGER_DEBUG, "Sent frame for presentation with timestamp {}ms", item.timestamp().to_milliseconds());
LibVideo: Deduplicate logic for dispatching video frame queue items Previously we had dispatch_decoder_error and on_decoder_error serving the same function, with one not handling the end of stream properly. There is also a new function to dispatch either an error or a frame to the owner of this playback manager, so that PlaybackStateHandlers don't have to duplicate this logic.
2023-02-12 00:44:16 -06:00
dispatch_new_frame(item.bitmap());
return false;
}
LibVideo/VideoPlayer: Dispatch state change events and update play icon The PlaybackStateChangeEvent wasn't connected up anymore, so the player wouldn't change icons when stopping playback due to reaching the end of the stream or encountering an error.
2023-02-12 02:25:25 -06:00
void PlaybackManager::dispatch_state_change()
{
LibVideo+VideoPlayer: Convert playback event handler to callbacks To pass events from LibVideo's PlaybackManager to interested parties, we currently dispatch Core::Event objects that outside callers listen for. Dispatching events in this manner rely on a Core::EventLoop. In order to use PlaybackManager from LibWeb, change this mechanism to instead use a set of callbacks to inform callers of events.
2023-04-08 19:18:38 -04:00
if (on_playback_state_change)
on_playback_state_change();
LibVideo/VideoPlayer: Dispatch state change events and update play icon The PlaybackStateChangeEvent wasn't connected up anymore, so the player wouldn't change icons when stopping playback due to reaching the end of the stream or encountering an error.
2023-02-12 02:25:25 -06:00
}
LibVideo/PlaybackManager: Organize playback states into virtual classes Storing playback states in virtual classes allows the behavior to be much more clearly written. Each `PlaybackStateHandler` subclass can implement some event-handling functions to model their behavior, and has functions to change its parent PlaybackManager's state to any other state. This will allow expanding the functionality of playback in the future, for example to allow skipping a single frame forward/backward.
2023-02-06 01:24:33 -06:00
void PlaybackManager::timer_callback()
LibVideo: Implement accurate seeking to inter frames in PlaybackManager This implements the PlaybackManager portion of seeking, so that it can seek to any frame in the video by dropping all preceding frames until it reaches the seek point. MatroskaDemuxer currently will only seek to the start of the video. That means that every seek has to drop all the frames until it comes across the target timestamp.
2022-11-14 00:54:21 -06:00
{
LibVideo/PlaybackManager: Organize playback states into virtual classes Storing playback states in virtual classes allows the behavior to be much more clearly written. Each `PlaybackStateHandler` subclass can implement some event-handling functions to model their behavior, and has functions to change its parent PlaybackManager's state to any other state. This will allow expanding the functionality of playback in the future, for example to allow skipping a single frame forward/backward.
2023-02-06 01:24:33 -06:00
TRY_OR_FATAL_ERROR(m_playback_handler->on_timer_callback());
LibVideo: Implement accurate seeking to inter frames in PlaybackManager This implements the PlaybackManager portion of seeking, so that it can seek to any frame in the video by dropping all preceding frames until it reaches the seek point. MatroskaDemuxer currently will only seek to the start of the video. That means that every seek has to drop all the frames until it comes across the target timestamp.
2022-11-14 00:54:21 -06:00
}
VideoPlayer/LibVideo: Seek accurately when scrolling in the seek bar Fast seeking does not work correctly when seeking in small increments, so it is necessary to use accurate seeking when using certain actions. The PlaybackManager has been changed to accept the seek mode as a parameter to `seek_to_timestamp` to facilitate this. This now means that it no longer has to track a seek mode preference.
2023-02-06 04:11:32 -06:00
void PlaybackManager::seek_to_timestamp(Time target_timestamp, SeekMode seek_mode)
LibVideo: Add PlaybackManager to load and decode videos This file will be the basis for abstracting away the out-of-thread or later out-of-process decoding from applications displaying videos. For now, the demuxer is hardcoded to be MatroskaParser, since that is all we support so far. The demuxer should later be selected based on the file header. The playback and decoding are currently all done on one thread using timers. The design of the code is such that adding threading should be trivial, at least based on an earlier version of the code. For now, though, it's better that this runs in one thread, as the multithreaded approach causes the Video Player to lock up permanently after a few frames are decoded.
2022-10-29 18:16:14 -05:00
{
VideoPlayer/LibVideo: Seek accurately when scrolling in the seek bar Fast seeking does not work correctly when seeking in small increments, so it is necessary to use accurate seeking when using certain actions. The PlaybackManager has been changed to accept the seek mode as a parameter to `seek_to_timestamp` to facilitate this. This now means that it no longer has to track a seek mode preference.
2023-02-06 04:11:32 -06:00
TRY_OR_FATAL_ERROR(m_playback_handler->seek(target_timestamp, seek_mode));
LibVideo: Add PlaybackManager to load and decode videos This file will be the basis for abstracting away the out-of-thread or later out-of-process decoding from applications displaying videos. For now, the demuxer is hardcoded to be MatroskaParser, since that is all we support so far. The demuxer should later be selected based on the file header. The playback and decoding are currently all done on one thread using timers. The design of the code is such that adding threading should be trivial, at least based on an earlier version of the code. For now, though, it's better that this runs in one thread, as the multithreaded approach causes the Video Player to lock up permanently after a few frames are decoded.
2022-10-29 18:16:14 -05:00
}
LibVideo: Pass the current sample to demuxers to lazily seek better In cases where the PlaybackManager's earliest buffered or displayed sample is closer to the seek target than the demuxer's chosen keyframe, we don't want to seek at all. To enable this, demuxers now receive an optional parameter with the earliest timestamp that the caller can still access. The demuxer in turn returns an optional to indicate when a seek was not needed, which allows PlaybackManager to avoid clearing its queue and re-decoding frames.
2023-02-06 01:25:02 -06:00
Optional<Time> PlaybackManager::seek_demuxer_to_most_recent_keyframe(Time timestamp, Optional<Time> earliest_available_sample)
LibVideo: Add PlaybackManager to load and decode videos This file will be the basis for abstracting away the out-of-thread or later out-of-process decoding from applications displaying videos. For now, the demuxer is hardcoded to be MatroskaParser, since that is all we support so far. The demuxer should later be selected based on the file header. The playback and decoding are currently all done on one thread using timers. The design of the code is such that adding threading should be trivial, at least based on an earlier version of the code. For now, though, it's better that this runs in one thread, as the multithreaded approach causes the Video Player to lock up permanently after a few frames are decoded.
2022-10-29 18:16:14 -05:00
{
VideoPlayer/LibVideo: Implement the UI functionality for seeking With these changes, the seek bar can be used, but only to seek to the start of the file. Seeking to anywhere else in the file will cause an error in the demuxer. The timestamp label that was previously invisible now has its text set according to either the playback or seek slider's position.
2022-11-11 22:26:19 -06:00
// FIXME: When the demuxer is getting samples off the main thread in the future, this needs to
// mutex so that seeking can't happen while that thread is getting a sample.
LibVideo: Pass the current sample to demuxers to lazily seek better In cases where the PlaybackManager's earliest buffered or displayed sample is closer to the seek target than the demuxer's chosen keyframe, we don't want to seek at all. To enable this, demuxers now receive an optional parameter with the earliest timestamp that the caller can still access. The demuxer in turn returns an optional to indicate when a seek was not needed, which allows PlaybackManager to avoid clearing its queue and re-decoding frames.
2023-02-06 01:25:02 -06:00
auto result = m_demuxer->seek_to_most_recent_keyframe(m_selected_video_track, timestamp, move(earliest_available_sample));
LibVideo: Handle corrupted video errors without spamming dialogs No longer will the video player explode with error dialogs that then lock the user out of closing them. To avoid issues where the playback state becomes invalid when an error occurs, I've made all decoder errors pass through the frame queue. This way, when a video is corrupted, there should be no chance that the playback state becomes invalid due to setting the state to Corrupted in the event handler while a presentation event is still pending. Or at least I think that was what caused some issues I was seeing :^) This system should be a lot more robust if any future errors need to be handled.
2022-11-09 06:52:37 -06:00
if (result.is_error())
LibVideo: Deduplicate logic for dispatching video frame queue items Previously we had dispatch_decoder_error and on_decoder_error serving the same function, with one not handling the end of stream properly. There is also a new function to dispatch either an error or a frame to the owner of this playback manager, so that PlaybackStateHandlers don't have to duplicate this logic.
2023-02-12 00:44:16 -06:00
dispatch_decoder_error(result.release_error());
LibVideo/PlaybackManager: Organize playback states into virtual classes Storing playback states in virtual classes allows the behavior to be much more clearly written. Each `PlaybackStateHandler` subclass can implement some event-handling functions to model their behavior, and has functions to change its parent PlaybackManager's state to any other state. This will allow expanding the functionality of playback in the future, for example to allow skipping a single frame forward/backward.
2023-02-06 01:24:33 -06:00
return result.release_value();
LibVideo: Add PlaybackManager to load and decode videos This file will be the basis for abstracting away the out-of-thread or later out-of-process decoding from applications displaying videos. For now, the demuxer is hardcoded to be MatroskaParser, since that is all we support so far. The demuxer should later be selected based on the file header. The playback and decoding are currently all done on one thread using timers. The design of the code is such that adding threading should be trivial, at least based on an earlier version of the code. For now, though, it's better that this runs in one thread, as the multithreaded approach causes the Video Player to lock up permanently after a few frames are decoded.
2022-10-29 18:16:14 -05:00
}
VideoPlayer/LibVideo: Implement the UI functionality for seeking With these changes, the seek bar can be used, but only to seek to the start of the file. Seeking to anywhere else in the file will cause an error in the demuxer. The timestamp label that was previously invisible now has its text set according to either the playback or seek slider's position.
2022-11-11 22:26:19 -06:00
void PlaybackManager::restart_playback()
{
seek_to_timestamp(Time::zero());
}
LibVideo/PlaybackManager: Organize playback states into virtual classes Storing playback states in virtual classes allows the behavior to be much more clearly written. Each `PlaybackStateHandler` subclass can implement some event-handling functions to model their behavior, and has functions to change its parent PlaybackManager's state to any other state. This will allow expanding the functionality of playback in the future, for example to allow skipping a single frame forward/backward.
2023-02-06 01:24:33 -06:00
void PlaybackManager::start_timer(int milliseconds)
{
m_present_timer->start(milliseconds);
}
LibVideo: Add PlaybackManager to load and decode videos This file will be the basis for abstracting away the out-of-thread or later out-of-process decoding from applications displaying videos. For now, the demuxer is hardcoded to be MatroskaParser, since that is all we support so far. The demuxer should later be selected based on the file header. The playback and decoding are currently all done on one thread using timers. The design of the code is such that adding threading should be trivial, at least based on an earlier version of the code. For now, though, it's better that this runs in one thread, as the multithreaded approach causes the Video Player to lock up permanently after a few frames are decoded.
2022-10-29 18:16:14 -05:00
bool PlaybackManager::decode_and_queue_one_sample()
{
LibVideo: Rewrite the video frame present function to be more readable The PlaybackManager::update_presented_frame function was getting out of hand and adding seeking was making it illegible. This rewrites it to be (hopefully) quite a bit more readable, and adds a few comments to help future readers of the code. In addition, some helpful debugging prints were added that should help debug any future issues with the player.
2022-11-14 00:48:31 -06:00
if (m_frame_queue->size() >= FRAME_BUFFER_COUNT) {
dbgln_if(PLAYBACK_MANAGER_DEBUG, "Frame queue is full, stopping");
LibVideo: Add PlaybackManager to load and decode videos This file will be the basis for abstracting away the out-of-thread or later out-of-process decoding from applications displaying videos. For now, the demuxer is hardcoded to be MatroskaParser, since that is all we support so far. The demuxer should later be selected based on the file header. The playback and decoding are currently all done on one thread using timers. The design of the code is such that adding threading should be trivial, at least based on an earlier version of the code. For now, though, it's better that this runs in one thread, as the multithreaded approach causes the Video Player to lock up permanently after a few frames are decoded.
2022-10-29 18:16:14 -05:00
return false;
LibVideo: Rewrite the video frame present function to be more readable The PlaybackManager::update_presented_frame function was getting out of hand and adding seeking was making it illegible. This rewrites it to be (hopefully) quite a bit more readable, and adds a few comments to help future readers of the code. In addition, some helpful debugging prints were added that should help debug any future issues with the player.
2022-11-14 00:48:31 -06:00
}
LibVideo: Add PlaybackManager to load and decode videos This file will be the basis for abstracting away the out-of-thread or later out-of-process decoding from applications displaying videos. For now, the demuxer is hardcoded to be MatroskaParser, since that is all we support so far. The demuxer should later be selected based on the file header. The playback and decoding are currently all done on one thread using timers. The design of the code is such that adding threading should be trivial, at least based on an earlier version of the code. For now, though, it's better that this runs in one thread, as the multithreaded approach causes the Video Player to lock up permanently after a few frames are decoded.
2022-10-29 18:16:14 -05:00
#if PLAYBACK_MANAGER_DEBUG
auto start_time = Time::now_monotonic();
#endif
LibVideo: Delay playback errors by their sample timestamps when we can If we encounter an error in the video decoder, we can set a timestamp for the error item in the queue so that it will display the error only when the frame that caused the error would have been displayed.
2023-02-12 00:50:30 -06:00
auto enqueue_error = [&](DecoderError&& error, Time timestamp) {
dbgln_if(PLAYBACK_MANAGER_DEBUG, "Enqueued decoder error: {}", error.string_literal());
m_frame_queue->enqueue(FrameQueueItem::error_marker(move(error), timestamp));
};
#define TRY_OR_ENQUEUE_ERROR(expression, timestamp) \
({ \
auto&& _temporary_result = (expression); \
if (_temporary_result.is_error()) { \
enqueue_error(_temporary_result.release_error(), (timestamp)); \
return false; \
} \
static_assert(!::AK::Detail::IsLvalueReference<decltype(_temporary_result.release_value())>, \
"Do not return a reference from a fallible expression"); \
_temporary_result.release_value(); \
LibVideo: Handle corrupted video errors without spamming dialogs No longer will the video player explode with error dialogs that then lock the user out of closing them. To avoid issues where the playback state becomes invalid when an error occurs, I've made all decoder errors pass through the frame queue. This way, when a video is corrupted, there should be no chance that the playback state becomes invalid due to setting the state to Corrupted in the event handler while a presentation event is still pending. Or at least I think that was what caused some issues I was seeing :^) This system should be a lot more robust if any future errors need to be handled.
2022-11-09 06:52:37 -06:00
})
LibVideo: Delay playback errors by their sample timestamps when we can If we encounter an error in the video decoder, we can set a timestamp for the error item in the queue so that it will display the error only when the frame that caused the error would have been displayed.
2023-02-12 00:50:30 -06:00
auto frame_sample = TRY_OR_ENQUEUE_ERROR(m_demuxer->get_next_video_sample_for_track(m_selected_video_track), Time::min());
LibVideo: Allow the VP9 decoder to queue multiple frames Frames will now be queued for retrieval by the user of the decoder. When the end of the current queue is reached, a DecoderError of category NeedsMoreInput will be emitted, allowing the caller to react by displaying what was previously retrieved for sending more samples.
2022-11-03 19:18:38 -05:00
OwnPtr<VideoFrame> decoded_frame = nullptr;
while (!decoded_frame) {
LibVideo: Delay playback errors by their sample timestamps when we can If we encounter an error in the video decoder, we can set a timestamp for the error item in the queue so that it will display the error only when the frame that caused the error would have been displayed.
2023-02-12 00:50:30 -06:00
TRY_OR_ENQUEUE_ERROR(m_decoder->receive_sample(frame_sample->data()), frame_sample->timestamp());
LibVideo: Allow the VP9 decoder to queue multiple frames Frames will now be queued for retrieval by the user of the decoder. When the end of the current queue is reached, a DecoderError of category NeedsMoreInput will be emitted, allowing the caller to react by displaying what was previously retrieved for sending more samples.
2022-11-03 19:18:38 -05:00
while (true) {
auto frame_result = m_decoder->get_decoded_frame();
if (frame_result.is_error()) {
if (frame_result.error().category() == DecoderErrorCategory::NeedsMoreInput)
break;
LibVideo: Add PlaybackManager to load and decode videos This file will be the basis for abstracting away the out-of-thread or later out-of-process decoding from applications displaying videos. For now, the demuxer is hardcoded to be MatroskaParser, since that is all we support so far. The demuxer should later be selected based on the file header. The playback and decoding are currently all done on one thread using timers. The design of the code is such that adding threading should be trivial, at least based on an earlier version of the code. For now, though, it's better that this runs in one thread, as the multithreaded approach causes the Video Player to lock up permanently after a few frames are decoded.
2022-10-29 18:16:14 -05:00
LibVideo: Delay playback errors by their sample timestamps when we can If we encounter an error in the video decoder, we can set a timestamp for the error item in the queue so that it will display the error only when the frame that caused the error would have been displayed.
2023-02-12 00:50:30 -06:00
enqueue_error(frame_result.release_error(), frame_sample->timestamp());
LibVideo: Allow the VP9 decoder to queue multiple frames Frames will now be queued for retrieval by the user of the decoder. When the end of the current queue is reached, a DecoderError of category NeedsMoreInput will be emitted, allowing the caller to react by displaying what was previously retrieved for sending more samples.
2022-11-03 19:18:38 -05:00
return false;
}
decoded_frame = frame_result.release_value();
VERIFY(decoded_frame);
}
}
LibVideo: Add PlaybackManager to load and decode videos This file will be the basis for abstracting away the out-of-thread or later out-of-process decoding from applications displaying videos. For now, the demuxer is hardcoded to be MatroskaParser, since that is all we support so far. The demuxer should later be selected based on the file header. The playback and decoding are currently all done on one thread using timers. The design of the code is such that adding threading should be trivial, at least based on an earlier version of the code. For now, though, it's better that this runs in one thread, as the multithreaded approach causes the Video Player to lock up permanently after a few frames are decoded.
2022-10-29 18:16:14 -05:00
auto& cicp = decoded_frame->cicp();
cicp.adopt_specified_values(frame_sample->container_cicp());
LibVideo: Rename "ColorRange" to "VideoFullRangeFlag" That matches the terminology used in ITU-T Rec. H.273, PNG's cICP chunk, and the ICC cicpTag. Also change the enum values to match the values in the spec -- 0 means "not full range" and 1 means "full range". (For now, keep the "Unspecified" entry around, and give it value 2. This value is not in the spec.) No intended behavior change.
2023-02-08 14:16:37 -05:00
cicp.default_code_points_if_unspecified({ ColorPrimaries::BT709, TransferCharacteristics::BT709, MatrixCoefficients::BT709, VideoFullRangeFlag::Studio });
LibVideo: Treat BT.601/709/2020 input transfer characteristics as sRGB I've realized that it probably makes more sense to change the input transfer characteristics to treat these as sRGB since color conversion in linear converted from BT.709 doesn't really make sense. If content creation applications expect media players to display BT.709 without conversions, this means they expect applications to treat it as sRGB, since that's what most displays use. That most likely also means they process it as sRGB internally, meaning we should do the same for our color primaries conversion.
2022-11-08 03:24:30 -06:00
// BT.601, BT.709 and BT.2020 have a similar transfer function to sRGB, so other applications
// (Chromium, VLC) forgo transfer characteristics conversion. We will emulate that behavior by
// handling those as sRGB instead, which causes no transfer function change in the output,
// unless display color management is later implemented.
switch (cicp.transfer_characteristics()) {
case TransferCharacteristics::BT601:
case TransferCharacteristics::BT709:
case TransferCharacteristics::BT2020BitDepth10:
case TransferCharacteristics::BT2020BitDepth12:
cicp.set_transfer_characteristics(TransferCharacteristics::SRGB);
break;
default:
break;
}
LibVideo: Add PlaybackManager to load and decode videos This file will be the basis for abstracting away the out-of-thread or later out-of-process decoding from applications displaying videos. For now, the demuxer is hardcoded to be MatroskaParser, since that is all we support so far. The demuxer should later be selected based on the file header. The playback and decoding are currently all done on one thread using timers. The design of the code is such that adding threading should be trivial, at least based on an earlier version of the code. For now, though, it's better that this runs in one thread, as the multithreaded approach causes the Video Player to lock up permanently after a few frames are decoded.
2022-10-29 18:16:14 -05:00
LibVideo: Delay playback errors by their sample timestamps when we can If we encounter an error in the video decoder, we can set a timestamp for the error item in the queue so that it will display the error only when the frame that caused the error would have been displayed.
2023-02-12 00:50:30 -06:00
auto bitmap = TRY_OR_ENQUEUE_ERROR(decoded_frame->to_bitmap(), frame_sample->timestamp());
LibVideo: Handle corrupted video errors without spamming dialogs No longer will the video player explode with error dialogs that then lock the user out of closing them. To avoid issues where the playback state becomes invalid when an error occurs, I've made all decoder errors pass through the frame queue. This way, when a video is corrupted, there should be no chance that the playback state becomes invalid due to setting the state to Corrupted in the event handler while a presentation event is still pending. Or at least I think that was what caused some issues I was seeing :^) This system should be a lot more robust if any future errors need to be handled.
2022-11-09 06:52:37 -06:00
m_frame_queue->enqueue(FrameQueueItem::frame(bitmap, frame_sample->timestamp()));
LibVideo: Add PlaybackManager to load and decode videos This file will be the basis for abstracting away the out-of-thread or later out-of-process decoding from applications displaying videos. For now, the demuxer is hardcoded to be MatroskaParser, since that is all we support so far. The demuxer should later be selected based on the file header. The playback and decoding are currently all done on one thread using timers. The design of the code is such that adding threading should be trivial, at least based on an earlier version of the code. For now, though, it's better that this runs in one thread, as the multithreaded approach causes the Video Player to lock up permanently after a few frames are decoded.
2022-10-29 18:16:14 -05:00
#if PLAYBACK_MANAGER_DEBUG
auto end_time = Time::now_monotonic();
LibVideo/PlaybackManager: Organize playback states into virtual classes Storing playback states in virtual classes allows the behavior to be much more clearly written. Each `PlaybackStateHandler` subclass can implement some event-handling functions to model their behavior, and has functions to change its parent PlaybackManager's state to any other state. This will allow expanding the functionality of playback in the future, for example to allow skipping a single frame forward/backward.
2023-02-06 01:24:33 -06:00
dbgln("Decoding sample at {}ms took {}ms, queue contains {} items", frame_sample->timestamp().to_milliseconds(), (end_time - start_time).to_milliseconds(), m_frame_queue->size());
LibVideo: Add PlaybackManager to load and decode videos This file will be the basis for abstracting away the out-of-thread or later out-of-process decoding from applications displaying videos. For now, the demuxer is hardcoded to be MatroskaParser, since that is all we support so far. The demuxer should later be selected based on the file header. The playback and decoding are currently all done on one thread using timers. The design of the code is such that adding threading should be trivial, at least based on an earlier version of the code. For now, though, it's better that this runs in one thread, as the multithreaded approach causes the Video Player to lock up permanently after a few frames are decoded.
2022-10-29 18:16:14 -05:00
#endif
return true;
}
void PlaybackManager::on_decode_timer()
{
LibVideo/PlaybackManager: Organize playback states into virtual classes Storing playback states in virtual classes allows the behavior to be much more clearly written. Each `PlaybackStateHandler` subclass can implement some event-handling functions to model their behavior, and has functions to change its parent PlaybackManager's state to any other state. This will allow expanding the functionality of playback in the future, for example to allow skipping a single frame forward/backward.
2023-02-06 01:24:33 -06:00
if (!decode_and_queue_one_sample()) {
// Note: When threading is implemented, this must be dispatched via an event loop.
TRY_OR_FATAL_ERROR(m_playback_handler->on_buffer_filled());
LibVideo: Add PlaybackManager to load and decode videos This file will be the basis for abstracting away the out-of-thread or later out-of-process decoding from applications displaying videos. For now, the demuxer is hardcoded to be MatroskaParser, since that is all we support so far. The demuxer should later be selected based on the file header. The playback and decoding are currently all done on one thread using timers. The design of the code is such that adding threading should be trivial, at least based on an earlier version of the code. For now, though, it's better that this runs in one thread, as the multithreaded approach causes the Video Player to lock up permanently after a few frames are decoded.
2022-10-29 18:16:14 -05:00
return;
}
// Continually decode until buffering is complete
LibVideo/PlaybackManager: Organize playback states into virtual classes Storing playback states in virtual classes allows the behavior to be much more clearly written. Each `PlaybackStateHandler` subclass can implement some event-handling functions to model their behavior, and has functions to change its parent PlaybackManager's state to any other state. This will allow expanding the functionality of playback in the future, for example to allow skipping a single frame forward/backward.
2023-02-06 01:24:33 -06:00
m_decode_timer->start(0);
}
Time PlaybackManager::PlaybackStateHandler::current_time() const
{
return m_manager.m_last_present_in_media_time;
}
ErrorOr<void> PlaybackManager::PlaybackStateHandler::seek(Time target_timestamp, SeekMode seek_mode)
{
return replace_handler_and_delete_this<SeekingStateHandler>(is_playing(), target_timestamp, seek_mode);
LibVideo: Add PlaybackManager to load and decode videos This file will be the basis for abstracting away the out-of-thread or later out-of-process decoding from applications displaying videos. For now, the demuxer is hardcoded to be MatroskaParser, since that is all we support so far. The demuxer should later be selected based on the file header. The playback and decoding are currently all done on one thread using timers. The design of the code is such that adding threading should be trivial, at least based on an earlier version of the code. For now, though, it's better that this runs in one thread, as the multithreaded approach causes the Video Player to lock up permanently after a few frames are decoded.
2022-10-29 18:16:14 -05:00
}
LibVideo/PlaybackManager: Organize playback states into virtual classes Storing playback states in virtual classes allows the behavior to be much more clearly written. Each `PlaybackStateHandler` subclass can implement some event-handling functions to model their behavior, and has functions to change its parent PlaybackManager's state to any other state. This will allow expanding the functionality of playback in the future, for example to allow skipping a single frame forward/backward.
2023-02-06 01:24:33 -06:00
ErrorOr<void> PlaybackManager::PlaybackStateHandler::stop()
{
return replace_handler_and_delete_this<StoppedStateHandler>();
}
template<class T, class... Args>
ErrorOr<void> PlaybackManager::PlaybackStateHandler::replace_handler_and_delete_this(Args... args)
{
auto temp_handler = TRY(adopt_nonnull_own_or_enomem<PlaybackStateHandler>(new (nothrow) T(m_manager, args...)));
m_manager.m_playback_handler.swap(temp_handler);
#if PLAYBACK_MANAGER_DEBUG
m_has_exited = true;
dbgln("Changing state from {} to {}", temp_handler->name(), m_manager.m_playback_handler->name());
#endif
TRY(m_manager.m_playback_handler->on_enter());
LibVideo: Dispatch PlaybackManager state changes after `on_enter()` Previously, the state change was dispatched before the new state that was adopted had been entered, causing it to have invalid state.
2023-04-12 00:21:43 -05:00
m_manager.dispatch_state_change();
LibVideo/PlaybackManager: Organize playback states into virtual classes Storing playback states in virtual classes allows the behavior to be much more clearly written. Each `PlaybackStateHandler` subclass can implement some event-handling functions to model their behavior, and has functions to change its parent PlaybackManager's state to any other state. This will allow expanding the functionality of playback in the future, for example to allow skipping a single frame forward/backward.
2023-02-06 01:24:33 -06:00
return {};
}
PlaybackManager& PlaybackManager::PlaybackStateHandler::manager() const
{
#if PLAYBACK_MANAGER_DEBUG
VERIFY(!m_has_exited);
#endif
return m_manager;
}
LibVideo: Create Resuming handler for seek/buffer handlers to inherit This allows the logic for keeping track of whether to resume to the paused or the playing state when exiting these states. The new StartingStateHandler also uses the class, since it can also be paused and unpaused while waiting for samples. The pause/play actions on the handlers inheriting from the resuming handler will also now notify the owner that the state has changed so that it can change icons, etc.
2023-02-12 02:36:06 -06:00
class PlaybackManager::ResumingStateHandler : public PlaybackManager::PlaybackStateHandler {
LibVideo: Add StartingStateHandler to prepare the first frames of video This new state handler will retrieve and display the first frame, while ensuring that playback can start as soon as possible by buffering two frames on top of the first frame for the PlayingStateHandler to set its next frame timer by.
2023-02-12 00:48:27 -06:00
public:
LibVideo: Create Resuming handler for seek/buffer handlers to inherit This allows the logic for keeping track of whether to resume to the paused or the playing state when exiting these states. The new StartingStateHandler also uses the class, since it can also be paused and unpaused while waiting for samples. The pause/play actions on the handlers inheriting from the resuming handler will also now notify the owner that the state has changed so that it can change icons, etc.
2023-02-12 02:36:06 -06:00
ResumingStateHandler(PlaybackManager& manager, bool playing)
LibVideo: Add StartingStateHandler to prepare the first frames of video This new state handler will retrieve and display the first frame, while ensuring that playback can start as soon as possible by buffering two frames on top of the first frame for the PlayingStateHandler to set its next frame timer by.
2023-02-12 00:48:27 -06:00
: PlaybackStateHandler(manager)
, m_playing(playing)
{
}
LibVideo: Create Resuming handler for seek/buffer handlers to inherit This allows the logic for keeping track of whether to resume to the paused or the playing state when exiting these states. The new StartingStateHandler also uses the class, since it can also be paused and unpaused while waiting for samples. The pause/play actions on the handlers inheriting from the resuming handler will also now notify the owner that the state has changed so that it can change icons, etc.
2023-02-12 02:36:06 -06:00
~ResumingStateHandler() override = default;
protected:
ErrorOr<void> assume_next_state()
{
if (!m_playing)
return replace_handler_and_delete_this<PausedStateHandler>();
return replace_handler_and_delete_this<PlayingStateHandler>();
}
ErrorOr<void> play() override
{
m_playing = true;
manager().dispatch_state_change();
return {};
}
LibVideo: Add a method to get the playback state from `PlaybackManager`
2023-04-12 01:01:29 -05:00
bool is_playing() const override { return m_playing; }
LibVideo: Create Resuming handler for seek/buffer handlers to inherit This allows the logic for keeping track of whether to resume to the paused or the playing state when exiting these states. The new StartingStateHandler also uses the class, since it can also be paused and unpaused while waiting for samples. The pause/play actions on the handlers inheriting from the resuming handler will also now notify the owner that the state has changed so that it can change icons, etc.
2023-02-12 02:36:06 -06:00
ErrorOr<void> pause() override
{
m_playing = false;
manager().dispatch_state_change();
return {};
}
LibVideo: Initialize primitive member variables in PlaybackManager Using uninitialized primitives is undefined behavior. In this case, all videos would autoplay on Serenity, but not play at all on Lagom, due to some m_playing booleans being uninitialized.
2023-04-09 09:39:48 -04:00
bool m_playing { false };
LibVideo: Create Resuming handler for seek/buffer handlers to inherit This allows the logic for keeping track of whether to resume to the paused or the playing state when exiting these states. The new StartingStateHandler also uses the class, since it can also be paused and unpaused while waiting for samples. The pause/play actions on the handlers inheriting from the resuming handler will also now notify the owner that the state has changed so that it can change icons, etc.
2023-02-12 02:36:06 -06:00
};
LibVideo/PlaybackManager: Organize playback states into virtual classes Storing playback states in virtual classes allows the behavior to be much more clearly written. Each `PlaybackStateHandler` subclass can implement some event-handling functions to model their behavior, and has functions to change its parent PlaybackManager's state to any other state. This will allow expanding the functionality of playback in the future, for example to allow skipping a single frame forward/backward.
2023-02-06 01:24:33 -06:00
class PlaybackManager::PlayingStateHandler : public PlaybackManager::PlaybackStateHandler {
public:
PlayingStateHandler(PlaybackManager& manager)
: PlaybackStateHandler(manager)
{
}
~PlayingStateHandler() override = default;
private:
ErrorOr<void> on_enter() override
{
m_last_present_in_real_time = Time::now_monotonic();
return on_timer_callback();
}
StringView name() override { return "Playing"sv; }
LibVideo: Add a method to get the playback state from `PlaybackManager`
2023-04-12 01:01:29 -05:00
bool is_playing() const override { return true; };
PlaybackState get_state() const override { return PlaybackState::Playing; }
LibVideo/PlaybackManager: Organize playback states into virtual classes Storing playback states in virtual classes allows the behavior to be much more clearly written. Each `PlaybackStateHandler` subclass can implement some event-handling functions to model their behavior, and has functions to change its parent PlaybackManager's state to any other state. This will allow expanding the functionality of playback in the future, for example to allow skipping a single frame forward/backward.
2023-02-06 01:24:33 -06:00
ErrorOr<void> pause() override
{
manager().m_last_present_in_media_time = current_time();
return replace_handler_and_delete_this<PausedStateHandler>();
}
ErrorOr<void> buffer() override
{
manager().m_last_present_in_media_time = current_time();
return replace_handler_and_delete_this<BufferingStateHandler>(true);
}
Time current_time() const override
{
return manager().m_last_present_in_media_time + (Time::now_monotonic() - m_last_present_in_real_time);
}
ErrorOr<void> on_timer_callback() override
{
LibVideo: Check whether it's time to present frames before doing so Previously, we assumed the timer was hitting at the correct time. This meant that if we changed states and the previous state handler had prepared a next frame, we would immediately display it without checking the timestamp.
2023-02-12 02:00:59 -06:00
auto set_presentation_timer = [&]() {
auto frame_time_ms = (manager().m_next_frame->timestamp() - current_time()).to_milliseconds();
VERIFY(frame_time_ms <= NumericLimits<int>::max());
dbgln_if(PLAYBACK_MANAGER_DEBUG, "Time until next frame is {}ms", frame_time_ms);
manager().start_timer(max(static_cast<int>(frame_time_ms), 0));
};
LibVideo: Add StartingStateHandler to prepare the first frames of video This new state handler will retrieve and display the first frame, while ensuring that playback can start as soon as possible by buffering two frames on top of the first frame for the PlayingStateHandler to set its next frame timer by.
2023-02-12 00:48:27 -06:00
if (manager().m_next_frame.has_value() && current_time() < manager().m_next_frame->timestamp()) {
LibVideo: Check whether it's time to present frames before doing so Previously, we assumed the timer was hitting at the correct time. This meant that if we changed states and the previous state handler had prepared a next frame, we would immediately display it without checking the timestamp.
2023-02-12 02:00:59 -06:00
dbgln_if(PLAYBACK_MANAGER_DEBUG, "Current time {}ms is too early to present the next frame at {}ms, delaying", current_time().to_milliseconds(), manager().m_next_frame->timestamp().to_milliseconds());
set_presentation_timer();
return {};
}
LibVideo/PlaybackManager: Organize playback states into virtual classes Storing playback states in virtual classes allows the behavior to be much more clearly written. Each `PlaybackStateHandler` subclass can implement some event-handling functions to model their behavior, and has functions to change its parent PlaybackManager's state to any other state. This will allow expanding the functionality of playback in the future, for example to allow skipping a single frame forward/backward.
2023-02-06 01:24:33 -06:00
Optional<FrameQueueItem> future_frame_item;
bool should_present_frame = false;
// Skip frames until we find a frame past the current playback time, and keep the one that precedes it to display.
while (!manager().m_frame_queue->is_empty()) {
future_frame_item.emplace(manager().m_frame_queue->dequeue());
manager().m_decode_timer->start(0);
LibVideo: Delay playback errors by their sample timestamps when we can If we encounter an error in the video decoder, we can set a timestamp for the error item in the queue so that it will display the error only when the frame that caused the error would have been displayed.
2023-02-12 00:50:30 -06:00
if (future_frame_item->timestamp() >= current_time() || future_frame_item->timestamp() == FrameQueueItem::no_timestamp) {
LibVideo/PlaybackManager: Organize playback states into virtual classes Storing playback states in virtual classes allows the behavior to be much more clearly written. Each `PlaybackStateHandler` subclass can implement some event-handling functions to model their behavior, and has functions to change its parent PlaybackManager's state to any other state. This will allow expanding the functionality of playback in the future, for example to allow skipping a single frame forward/backward.
2023-02-06 01:24:33 -06:00
dbgln_if(PLAYBACK_MANAGER_DEBUG, "Should present frame, future {} is error or after {}ms", future_frame_item->debug_string(), current_time().to_milliseconds());
should_present_frame = true;
break;
}
if (manager().m_next_frame.has_value()) {
dbgln_if(PLAYBACK_MANAGER_DEBUG, "At {}ms: Dropped {} in favor of {}", current_time().to_milliseconds(), manager().m_next_frame->debug_string(), future_frame_item->debug_string());
manager().m_skipped_frames++;
}
manager().m_next_frame.emplace(future_frame_item.release_value());
}
// If we don't have both of these items, we can't present, since we need to set a timer for
// the next frame. Check if we need to buffer based on the current state.
if (!manager().m_next_frame.has_value() || !future_frame_item.has_value()) {
#if PLAYBACK_MANAGER_DEBUG
StringBuilder debug_string_builder;
debug_string_builder.append("We don't have "sv);
if (!manager().m_next_frame.has_value()) {
debug_string_builder.append("a frame to present"sv);
if (!future_frame_item.has_value())
debug_string_builder.append(" or a future frame"sv);
} else {
debug_string_builder.append("a future frame"sv);
}
debug_string_builder.append(", checking for error and buffering"sv);
dbgln_if(PLAYBACK_MANAGER_DEBUG, debug_string_builder.to_deprecated_string());
#endif
if (future_frame_item.has_value()) {
if (future_frame_item->is_error()) {
LibVideo: Deduplicate logic for dispatching video frame queue items Previously we had dispatch_decoder_error and on_decoder_error serving the same function, with one not handling the end of stream properly. There is also a new function to dispatch either an error or a frame to the owner of this playback manager, so that PlaybackStateHandlers don't have to duplicate this logic.
2023-02-12 00:44:16 -06:00
manager().dispatch_decoder_error(future_frame_item.release_value().release_error());
LibVideo/PlaybackManager: Organize playback states into virtual classes Storing playback states in virtual classes allows the behavior to be much more clearly written. Each `PlaybackStateHandler` subclass can implement some event-handling functions to model their behavior, and has functions to change its parent PlaybackManager's state to any other state. This will allow expanding the functionality of playback in the future, for example to allow skipping a single frame forward/backward.
2023-02-06 01:24:33 -06:00
return {};
}
manager().m_next_frame.emplace(future_frame_item.release_value());
}
TRY(buffer());
return {};
}
// If we have a frame, send it for presentation.
if (should_present_frame) {
auto now = Time::now_monotonic();
manager().m_last_present_in_media_time += now - m_last_present_in_real_time;
m_last_present_in_real_time = now;
LibVideo: Deduplicate logic for dispatching video frame queue items Previously we had dispatch_decoder_error and on_decoder_error serving the same function, with one not handling the end of stream properly. There is also a new function to dispatch either an error or a frame to the owner of this playback manager, so that PlaybackStateHandlers don't have to duplicate this logic.
2023-02-12 00:44:16 -06:00
if (manager().dispatch_frame_queue_item(manager().m_next_frame.release_value()))
return {};
LibVideo/PlaybackManager: Organize playback states into virtual classes Storing playback states in virtual classes allows the behavior to be much more clearly written. Each `PlaybackStateHandler` subclass can implement some event-handling functions to model their behavior, and has functions to change its parent PlaybackManager's state to any other state. This will allow expanding the functionality of playback in the future, for example to allow skipping a single frame forward/backward.
2023-02-06 01:24:33 -06:00
}
// Now that we've presented the current frame, we can throw whatever error is next in queue.
// This way, we always display a frame before the stream ends, and should also show any frames
// we already had when a real error occurs.
if (future_frame_item->is_error()) {
LibVideo: Deduplicate logic for dispatching video frame queue items Previously we had dispatch_decoder_error and on_decoder_error serving the same function, with one not handling the end of stream properly. There is also a new function to dispatch either an error or a frame to the owner of this playback manager, so that PlaybackStateHandlers don't have to duplicate this logic.
2023-02-12 00:44:16 -06:00
manager().dispatch_decoder_error(future_frame_item.release_value().release_error());
LibVideo/PlaybackManager: Organize playback states into virtual classes Storing playback states in virtual classes allows the behavior to be much more clearly written. Each `PlaybackStateHandler` subclass can implement some event-handling functions to model their behavior, and has functions to change its parent PlaybackManager's state to any other state. This will allow expanding the functionality of playback in the future, for example to allow skipping a single frame forward/backward.
2023-02-06 01:24:33 -06:00
return {};
}
// The future frame item becomes the next one to present.
manager().m_next_frame.emplace(future_frame_item.release_value());
LibVideo: Check whether it's time to present frames before doing so Previously, we assumed the timer was hitting at the correct time. This meant that if we changed states and the previous state handler had prepared a next frame, we would immediately display it without checking the timestamp.
2023-02-12 02:00:59 -06:00
set_presentation_timer();
LibVideo/PlaybackManager: Organize playback states into virtual classes Storing playback states in virtual classes allows the behavior to be much more clearly written. Each `PlaybackStateHandler` subclass can implement some event-handling functions to model their behavior, and has functions to change its parent PlaybackManager's state to any other state. This will allow expanding the functionality of playback in the future, for example to allow skipping a single frame forward/backward.
2023-02-06 01:24:33 -06:00
return {};
}
Time m_last_present_in_real_time = Time::zero();
};
class PlaybackManager::PausedStateHandler : public PlaybackManager::PlaybackStateHandler {
public:
PausedStateHandler(PlaybackManager& manager)
: PlaybackStateHandler(manager)
{
}
~PausedStateHandler() override = default;
private:
StringView name() override { return "Paused"sv; }
ErrorOr<void> play() override
{
return replace_handler_and_delete_this<PlayingStateHandler>();
}
LibVideo: Add a method to get the playback state from `PlaybackManager`
2023-04-12 01:01:29 -05:00
bool is_playing() const override { return false; };
PlaybackState get_state() const override { return PlaybackState::Paused; }
LibVideo/PlaybackManager: Organize playback states into virtual classes Storing playback states in virtual classes allows the behavior to be much more clearly written. Each `PlaybackStateHandler` subclass can implement some event-handling functions to model their behavior, and has functions to change its parent PlaybackManager's state to any other state. This will allow expanding the functionality of playback in the future, for example to allow skipping a single frame forward/backward.
2023-02-06 01:24:33 -06:00
};
LibVideo: Create Resuming handler for seek/buffer handlers to inherit This allows the logic for keeping track of whether to resume to the paused or the playing state when exiting these states. The new StartingStateHandler also uses the class, since it can also be paused and unpaused while waiting for samples. The pause/play actions on the handlers inheriting from the resuming handler will also now notify the owner that the state has changed so that it can change icons, etc.
2023-02-12 02:36:06 -06:00
class PlaybackManager::BufferingStateHandler : public PlaybackManager::ResumingStateHandler {
using PlaybackManager::ResumingStateHandler::ResumingStateHandler;
LibVideo/PlaybackManager: Organize playback states into virtual classes Storing playback states in virtual classes allows the behavior to be much more clearly written. Each `PlaybackStateHandler` subclass can implement some event-handling functions to model their behavior, and has functions to change its parent PlaybackManager's state to any other state. This will allow expanding the functionality of playback in the future, for example to allow skipping a single frame forward/backward.
2023-02-06 01:24:33 -06:00
ErrorOr<void> on_enter() override
{
manager().m_decode_timer->start(0);
return {};
}
StringView name() override { return "Buffering"sv; }
ErrorOr<void> on_buffer_filled() override
{
LibVideo: Create Resuming handler for seek/buffer handlers to inherit This allows the logic for keeping track of whether to resume to the paused or the playing state when exiting these states. The new StartingStateHandler also uses the class, since it can also be paused and unpaused while waiting for samples. The pause/play actions on the handlers inheriting from the resuming handler will also now notify the owner that the state has changed so that it can change icons, etc.
2023-02-12 02:36:06 -06:00
return assume_next_state();
LibVideo/PlaybackManager: Organize playback states into virtual classes Storing playback states in virtual classes allows the behavior to be much more clearly written. Each `PlaybackStateHandler` subclass can implement some event-handling functions to model their behavior, and has functions to change its parent PlaybackManager's state to any other state. This will allow expanding the functionality of playback in the future, for example to allow skipping a single frame forward/backward.
2023-02-06 01:24:33 -06:00
}
LibVideo: Add a method to get the playback state from `PlaybackManager`
2023-04-12 01:01:29 -05:00
PlaybackState get_state() const override { return PlaybackState::Buffering; }
LibVideo/PlaybackManager: Organize playback states into virtual classes Storing playback states in virtual classes allows the behavior to be much more clearly written. Each `PlaybackStateHandler` subclass can implement some event-handling functions to model their behavior, and has functions to change its parent PlaybackManager's state to any other state. This will allow expanding the functionality of playback in the future, for example to allow skipping a single frame forward/backward.
2023-02-06 01:24:33 -06:00
};
LibVideo: Create Resuming handler for seek/buffer handlers to inherit This allows the logic for keeping track of whether to resume to the paused or the playing state when exiting these states. The new StartingStateHandler also uses the class, since it can also be paused and unpaused while waiting for samples. The pause/play actions on the handlers inheriting from the resuming handler will also now notify the owner that the state has changed so that it can change icons, etc.
2023-02-12 02:36:06 -06:00
class PlaybackManager::SeekingStateHandler : public PlaybackManager::ResumingStateHandler {
LibVideo/PlaybackManager: Organize playback states into virtual classes Storing playback states in virtual classes allows the behavior to be much more clearly written. Each `PlaybackStateHandler` subclass can implement some event-handling functions to model their behavior, and has functions to change its parent PlaybackManager's state to any other state. This will allow expanding the functionality of playback in the future, for example to allow skipping a single frame forward/backward.
2023-02-06 01:24:33 -06:00
public:
SeekingStateHandler(PlaybackManager& manager, bool playing, Time target_timestamp, SeekMode seek_mode)
LibVideo: Create Resuming handler for seek/buffer handlers to inherit This allows the logic for keeping track of whether to resume to the paused or the playing state when exiting these states. The new StartingStateHandler also uses the class, since it can also be paused and unpaused while waiting for samples. The pause/play actions on the handlers inheriting from the resuming handler will also now notify the owner that the state has changed so that it can change icons, etc.
2023-02-12 02:36:06 -06:00
: ResumingStateHandler(manager, playing)
LibVideo/PlaybackManager: Organize playback states into virtual classes Storing playback states in virtual classes allows the behavior to be much more clearly written. Each `PlaybackStateHandler` subclass can implement some event-handling functions to model their behavior, and has functions to change its parent PlaybackManager's state to any other state. This will allow expanding the functionality of playback in the future, for example to allow skipping a single frame forward/backward.
2023-02-06 01:24:33 -06:00
, m_target_timestamp(target_timestamp)
, m_seek_mode(seek_mode)
{
}
~SeekingStateHandler() override = default;
private:
ErrorOr<void> on_enter() override
{
LibVideo: Pass the current sample to demuxers to lazily seek better In cases where the PlaybackManager's earliest buffered or displayed sample is closer to the seek target than the demuxer's chosen keyframe, we don't want to seek at all. To enable this, demuxers now receive an optional parameter with the earliest timestamp that the caller can still access. The demuxer in turn returns an optional to indicate when a seek was not needed, which allows PlaybackManager to avoid clearing its queue and re-decoding frames.
2023-02-06 01:25:02 -06:00
auto earliest_available_sample = manager().m_last_present_in_media_time;
LibVideo: Delay playback errors by their sample timestamps when we can If we encounter an error in the video decoder, we can set a timestamp for the error item in the queue so that it will display the error only when the frame that caused the error would have been displayed.
2023-02-12 00:50:30 -06:00
if (manager().m_next_frame.has_value() && manager().m_next_frame->timestamp() != FrameQueueItem::no_timestamp) {
LibVideo: Pass the current sample to demuxers to lazily seek better In cases where the PlaybackManager's earliest buffered or displayed sample is closer to the seek target than the demuxer's chosen keyframe, we don't want to seek at all. To enable this, demuxers now receive an optional parameter with the earliest timestamp that the caller can still access. The demuxer in turn returns an optional to indicate when a seek was not needed, which allows PlaybackManager to avoid clearing its queue and re-decoding frames.
2023-02-06 01:25:02 -06:00
earliest_available_sample = min(earliest_available_sample, manager().m_next_frame->timestamp());
}
auto keyframe_timestamp = manager().seek_demuxer_to_most_recent_keyframe(m_target_timestamp, earliest_available_sample);
#if PLAYBACK_MANAGER_DEBUG
auto seek_mode_name = m_seek_mode == SeekMode::Accurate ? "Accurate"sv : "Fast"sv;
if (keyframe_timestamp.has_value()) {
dbgln("{} seeking to timestamp target {}ms, selected keyframe at {}ms", seek_mode_name, m_target_timestamp.to_milliseconds(), keyframe_timestamp->to_milliseconds());
} else {
LibVideo: Improve logging when PLAYBACK_MANAGER_DEBUG=on This adds a timestamp to the debug output when presenting a frame, so it can be clear the frame spacing between a presented frame and a state change. The seeking state will also now print when it early-exits if the seek point is within the current sample's duration.
2023-04-11 23:42:47 -05:00
dbgln("{} seeking to timestamp target {}ms, demuxer kept its iterator position after {}ms", seek_mode_name, m_target_timestamp.to_milliseconds(), earliest_available_sample.to_milliseconds());
LibVideo: Pass the current sample to demuxers to lazily seek better In cases where the PlaybackManager's earliest buffered or displayed sample is closer to the seek target than the demuxer's chosen keyframe, we don't want to seek at all. To enable this, demuxers now receive an optional parameter with the earliest timestamp that the caller can still access. The demuxer in turn returns an optional to indicate when a seek was not needed, which allows PlaybackManager to avoid clearing its queue and re-decoding frames.
2023-02-06 01:25:02 -06:00
}
#endif
LibVideo/PlaybackManager: Organize playback states into virtual classes Storing playback states in virtual classes allows the behavior to be much more clearly written. Each `PlaybackStateHandler` subclass can implement some event-handling functions to model their behavior, and has functions to change its parent PlaybackManager's state to any other state. This will allow expanding the functionality of playback in the future, for example to allow skipping a single frame forward/backward.
2023-02-06 01:24:33 -06:00
if (m_seek_mode == SeekMode::Fast) {
LibVideo: Always present a frame when not fast-exiting seek Previously, there was some leftover logic in `SeekingStateHandler` which would avoid presenting a frame if it didn't find a frame both before and after the target frame. However, this logic was unnecessary as the `on_enter()` function would check if it needed to present a frame and exit seeking if not. This allows seeking to succeed if the Seeking handler cannot find a frame before the one to be seeked to, which could occur if the first frame of a video is not at timestamp 0.
2023-04-12 00:24:05 -05:00
m_target_timestamp = keyframe_timestamp.value_or(manager().m_last_present_in_media_time);
LibVideo/PlaybackManager: Organize playback states into virtual classes Storing playback states in virtual classes allows the behavior to be much more clearly written. Each `PlaybackStateHandler` subclass can implement some event-handling functions to model their behavior, and has functions to change its parent PlaybackManager's state to any other state. This will allow expanding the functionality of playback in the future, for example to allow skipping a single frame forward/backward.
2023-02-06 01:24:33 -06:00
}
LibVideo: Pass the current sample to demuxers to lazily seek better In cases where the PlaybackManager's earliest buffered or displayed sample is closer to the seek target than the demuxer's chosen keyframe, we don't want to seek at all. To enable this, demuxers now receive an optional parameter with the earliest timestamp that the caller can still access. The demuxer in turn returns an optional to indicate when a seek was not needed, which allows PlaybackManager to avoid clearing its queue and re-decoding frames.
2023-02-06 01:25:02 -06:00
if (keyframe_timestamp.has_value()) {
LibVideo: Improve logging when PLAYBACK_MANAGER_DEBUG=on This adds a timestamp to the debug output when presenting a frame, so it can be clear the frame spacing between a presented frame and a state change. The seeking state will also now print when it early-exits if the seek point is within the current sample's duration.
2023-04-11 23:42:47 -05:00
dbgln_if(PLAYBACK_MANAGER_DEBUG, "Keyframe is nearer to the target than the current frames, clearing queue");
LibVideo/PlaybackManager: Organize playback states into virtual classes Storing playback states in virtual classes allows the behavior to be much more clearly written. Each `PlaybackStateHandler` subclass can implement some event-handling functions to model their behavior, and has functions to change its parent PlaybackManager's state to any other state. This will allow expanding the functionality of playback in the future, for example to allow skipping a single frame forward/backward.
2023-02-06 01:24:33 -06:00
manager().m_frame_queue->clear();
manager().m_next_frame.clear();
LibVideo: Always present a frame when not fast-exiting seek Previously, there was some leftover logic in `SeekingStateHandler` which would avoid presenting a frame if it didn't find a frame both before and after the target frame. However, this logic was unnecessary as the `on_enter()` function would check if it needed to present a frame and exit seeking if not. This allows seeking to succeed if the Seeking handler cannot find a frame before the one to be seeked to, which could occur if the first frame of a video is not at timestamp 0.
2023-04-12 00:24:05 -05:00
manager().m_last_present_in_media_time = keyframe_timestamp.value();
LibVideo: Pass the current sample to demuxers to lazily seek better In cases where the PlaybackManager's earliest buffered or displayed sample is closer to the seek target than the demuxer's chosen keyframe, we don't want to seek at all. To enable this, demuxers now receive an optional parameter with the earliest timestamp that the caller can still access. The demuxer in turn returns an optional to indicate when a seek was not needed, which allows PlaybackManager to avoid clearing its queue and re-decoding frames.
2023-02-06 01:25:02 -06:00
} else if (m_target_timestamp >= manager().m_last_present_in_media_time && manager().m_next_frame.has_value() && manager().m_next_frame.value().timestamp() > m_target_timestamp) {
LibVideo: Improve logging when PLAYBACK_MANAGER_DEBUG=on This adds a timestamp to the debug output when presenting a frame, so it can be clear the frame spacing between a presented frame and a state change. The seeking state will also now print when it early-exits if the seek point is within the current sample's duration.
2023-04-11 23:42:47 -05:00
dbgln_if(PLAYBACK_MANAGER_DEBUG, "Target timestamp is between the last presented frame and the next frame, exiting seek at {}ms", m_target_timestamp.to_milliseconds());
LibVideo/PlaybackManager: Organize playback states into virtual classes Storing playback states in virtual classes allows the behavior to be much more clearly written. Each `PlaybackStateHandler` subclass can implement some event-handling functions to model their behavior, and has functions to change its parent PlaybackManager's state to any other state. This will allow expanding the functionality of playback in the future, for example to allow skipping a single frame forward/backward.
2023-02-06 01:24:33 -06:00
manager().m_last_present_in_media_time = m_target_timestamp;
LibVideo: Create Resuming handler for seek/buffer handlers to inherit This allows the logic for keeping track of whether to resume to the paused or the playing state when exiting these states. The new StartingStateHandler also uses the class, since it can also be paused and unpaused while waiting for samples. The pause/play actions on the handlers inheriting from the resuming handler will also now notify the owner that the state has changed so that it can change icons, etc.
2023-02-12 02:36:06 -06:00
return assume_next_state();
LibVideo/PlaybackManager: Organize playback states into virtual classes Storing playback states in virtual classes allows the behavior to be much more clearly written. Each `PlaybackStateHandler` subclass can implement some event-handling functions to model their behavior, and has functions to change its parent PlaybackManager's state to any other state. This will allow expanding the functionality of playback in the future, for example to allow skipping a single frame forward/backward.
2023-02-06 01:24:33 -06:00
}
return skip_samples_until_timestamp();
}
ErrorOr<void> skip_samples_until_timestamp()
{
while (!manager().m_frame_queue->is_empty()) {
auto item = manager().m_frame_queue->dequeue();
manager().m_decode_timer->start(0);
LibVideo: Delay playback errors by their sample timestamps when we can If we encounter an error in the video decoder, we can set a timestamp for the error item in the queue so that it will display the error only when the frame that caused the error would have been displayed.
2023-02-12 00:50:30 -06:00
dbgln_if(PLAYBACK_MANAGER_DEBUG, "Dequeuing frame at {}ms and comparing to seek target {}ms", item.timestamp().to_milliseconds(), m_target_timestamp.to_milliseconds());
LibVideo: Always present a frame when not fast-exiting seek Previously, there was some leftover logic in `SeekingStateHandler` which would avoid presenting a frame if it didn't find a frame both before and after the target frame. However, this logic was unnecessary as the `on_enter()` function would check if it needed to present a frame and exit seeking if not. This allows seeking to succeed if the Seeking handler cannot find a frame before the one to be seeked to, which could occur if the first frame of a video is not at timestamp 0.
2023-04-12 00:24:05 -05:00
if (manager().m_next_frame.has_value() && (item.timestamp() > m_target_timestamp || item.timestamp() == FrameQueueItem::no_timestamp)) {
// If the frame we're presenting is later than the target timestamp, skip the timestamp forward to it.
if (manager().m_next_frame->timestamp() > m_target_timestamp) {
manager().m_last_present_in_media_time = manager().m_next_frame->timestamp();
} else {
LibVideo: Delay playback errors by their sample timestamps when we can If we encounter an error in the video decoder, we can set a timestamp for the error item in the queue so that it will display the error only when the frame that caused the error would have been displayed.
2023-02-12 00:50:30 -06:00
manager().m_last_present_in_media_time = m_target_timestamp;
LibVideo/PlaybackManager: Organize playback states into virtual classes Storing playback states in virtual classes allows the behavior to be much more clearly written. Each `PlaybackStateHandler` subclass can implement some event-handling functions to model their behavior, and has functions to change its parent PlaybackManager's state to any other state. This will allow expanding the functionality of playback in the future, for example to allow skipping a single frame forward/backward.
2023-02-06 01:24:33 -06:00
}
LibVideo: Delay playback errors by their sample timestamps when we can If we encounter an error in the video decoder, we can set a timestamp for the error item in the queue so that it will display the error only when the frame that caused the error would have been displayed.
2023-02-12 00:50:30 -06:00
LibVideo: Always present a frame when not fast-exiting seek Previously, there was some leftover logic in `SeekingStateHandler` which would avoid presenting a frame if it didn't find a frame both before and after the target frame. However, this logic was unnecessary as the `on_enter()` function would check if it needed to present a frame and exit seeking if not. This allows seeking to succeed if the Seeking handler cannot find a frame before the one to be seeked to, which could occur if the first frame of a video is not at timestamp 0.
2023-04-12 00:24:05 -05:00
if (manager().dispatch_frame_queue_item(manager().m_next_frame.release_value()))
return {};
LibVideo/PlaybackManager: Organize playback states into virtual classes Storing playback states in virtual classes allows the behavior to be much more clearly written. Each `PlaybackStateHandler` subclass can implement some event-handling functions to model their behavior, and has functions to change its parent PlaybackManager's state to any other state. This will allow expanding the functionality of playback in the future, for example to allow skipping a single frame forward/backward.
2023-02-06 01:24:33 -06:00
manager().m_next_frame.emplace(item);
LibVideo: Delay playback errors by their sample timestamps when we can If we encounter an error in the video decoder, we can set a timestamp for the error item in the queue so that it will display the error only when the frame that caused the error would have been displayed.
2023-02-12 00:50:30 -06:00
dbgln_if(PLAYBACK_MANAGER_DEBUG, "Exiting seek to {} state at {}ms", m_playing ? "Playing" : "Paused", manager().m_last_present_in_media_time.to_milliseconds());
LibVideo: Create Resuming handler for seek/buffer handlers to inherit This allows the logic for keeping track of whether to resume to the paused or the playing state when exiting these states. The new StartingStateHandler also uses the class, since it can also be paused and unpaused while waiting for samples. The pause/play actions on the handlers inheriting from the resuming handler will also now notify the owner that the state has changed so that it can change icons, etc.
2023-02-12 02:36:06 -06:00
return assume_next_state();
LibVideo: Delay playback errors by their sample timestamps when we can If we encounter an error in the video decoder, we can set a timestamp for the error item in the queue so that it will display the error only when the frame that caused the error would have been displayed.
2023-02-12 00:50:30 -06:00
}
manager().m_next_frame.emplace(item);
LibVideo/PlaybackManager: Organize playback states into virtual classes Storing playback states in virtual classes allows the behavior to be much more clearly written. Each `PlaybackStateHandler` subclass can implement some event-handling functions to model their behavior, and has functions to change its parent PlaybackManager's state to any other state. This will allow expanding the functionality of playback in the future, for example to allow skipping a single frame forward/backward.
2023-02-06 01:24:33 -06:00
}
dbgln_if(PLAYBACK_MANAGER_DEBUG, "Frame queue is empty while seeking, waiting for buffer fill.");
manager().m_decode_timer->start(0);
return {};
}
StringView name() override { return "Seeking"sv; }
ErrorOr<void> seek(Time target_timestamp, SeekMode seek_mode) override
{
m_target_timestamp = target_timestamp;
m_seek_mode = seek_mode;
return on_enter();
}
Time current_time() const override
{
return m_target_timestamp;
}
// We won't need this override when threaded, the queue can pause us in on_enter().
ErrorOr<void> on_buffer_filled() override
{
dbgln_if(PLAYBACK_MANAGER_DEBUG, "Buffer filled while seeking, dequeuing until timestamp.");
return skip_samples_until_timestamp();
}
LibVideo: Add a method to get the playback state from `PlaybackManager`
2023-04-12 01:01:29 -05:00
PlaybackState get_state() const override { return PlaybackState::Seeking; }
LibVideo/PlaybackManager: Organize playback states into virtual classes Storing playback states in virtual classes allows the behavior to be much more clearly written. Each `PlaybackStateHandler` subclass can implement some event-handling functions to model their behavior, and has functions to change its parent PlaybackManager's state to any other state. This will allow expanding the functionality of playback in the future, for example to allow skipping a single frame forward/backward.
2023-02-06 01:24:33 -06:00
Time m_target_timestamp { Time::zero() };
SeekMode m_seek_mode { SeekMode::Accurate };
};
class PlaybackManager::StoppedStateHandler : public PlaybackManager::PlaybackStateHandler {
public:
StoppedStateHandler(PlaybackManager& manager)
: PlaybackStateHandler(manager)
{
}
~StoppedStateHandler() override = default;
private:
ErrorOr<void> on_enter() override
{
return {};
}
StringView name() override { return "Stopped"sv; }
ErrorOr<void> play() override
{
LibVideo: Pass the current sample to demuxers to lazily seek better In cases where the PlaybackManager's earliest buffered or displayed sample is closer to the seek target than the demuxer's chosen keyframe, we don't want to seek at all. To enable this, demuxers now receive an optional parameter with the earliest timestamp that the caller can still access. The demuxer in turn returns an optional to indicate when a seek was not needed, which allows PlaybackManager to avoid clearing its queue and re-decoding frames.
2023-02-06 01:25:02 -06:00
manager().m_next_frame.clear();
manager().m_frame_queue->clear();
auto start_timestamp = manager().seek_demuxer_to_most_recent_keyframe(Time::zero());
VERIFY(start_timestamp.has_value());
manager().m_last_present_in_media_time = start_timestamp.release_value();
LibVideo: Remove Starting playback state in favor of Seeking to zero
2023-04-12 00:42:14 -05:00
return replace_handler_and_delete_this<SeekingStateHandler>(true, Time::zero(), SeekMode::Fast);
LibVideo/PlaybackManager: Organize playback states into virtual classes Storing playback states in virtual classes allows the behavior to be much more clearly written. Each `PlaybackStateHandler` subclass can implement some event-handling functions to model their behavior, and has functions to change its parent PlaybackManager's state to any other state. This will allow expanding the functionality of playback in the future, for example to allow skipping a single frame forward/backward.
2023-02-06 01:24:33 -06:00
}
LibVideo: Add a method to get the playback state from `PlaybackManager`
2023-04-12 01:01:29 -05:00
bool is_playing() const override { return false; };
PlaybackState get_state() const override { return PlaybackState::Stopped; }
LibVideo/PlaybackManager: Organize playback states into virtual classes Storing playback states in virtual classes allows the behavior to be much more clearly written. Each `PlaybackStateHandler` subclass can implement some event-handling functions to model their behavior, and has functions to change its parent PlaybackManager's state to any other state. This will allow expanding the functionality of playback in the future, for example to allow skipping a single frame forward/backward.
2023-02-06 01:24:33 -06:00
};
LibVideo: Add PlaybackManager to load and decode videos This file will be the basis for abstracting away the out-of-thread or later out-of-process decoding from applications displaying videos. For now, the demuxer is hardcoded to be MatroskaParser, since that is all we support so far. The demuxer should later be selected based on the file header. The playback and decoding are currently all done on one thread using timers. The design of the code is such that adding threading should be trivial, at least based on an earlier version of the code. For now, though, it's better that this runs in one thread, as the multithreaded approach causes the Video Player to lock up permanently after a few frames are decoded.
2022-10-29 18:16:14 -05:00
}
Reference in a new issue Copy permalink