Stowage/ladybird
2
0
Fork 0
mirror of https://github.com/LadybirdBrowser/ladybird.git synced 2025-11-07 08:40:59 +00:00
Code Issues Projects Releases Packages Wiki Activity
ladybird/Userland/Libraries/LibVideo/PlaybackManager.cpp

597 lines
23 KiB
C++
Raw Normal View History

LibVideo: Add PlaybackManager to load and decode videos This file will be the basis for abstracting away the out-of-thread or later out-of-process decoding from applications displaying videos. For now, the demuxer is hardcoded to be MatroskaParser, since that is all we support so far. The demuxer should later be selected based on the file header. The playback and decoding are currently all done on one thread using timers. The design of the code is such that adding threading should be trivial, at least based on an earlier version of the code. For now, though, it's better that this runs in one thread, as the multithreaded approach causes the Video Player to lock up permanently after a few frames are decoded.
2022-10-29 18:16:14 -05:00
/*
* Copyright (c) 2022, Gregory Bertilson <zaggy1024@gmail.com>
*
* SPDX-License-Identifier: BSD-2-Clause
*/
#include <AK/Format.h>
#include <LibCore/Timer.h>
LibVideo: Reorganize demuxer file hierarchy and rename Matroska files As new demuxers are added, this will get quite full of files, so it'll be good to have a separate folder for these. To avoid too many chained namespaces, the Containers subdirectory is not also a namespace, but the Matroska folder is for the sake of separating the multiple classes for parsed information entering the Video namespace.
2022-11-09 19:47:56 -06:00
#include <LibVideo/Containers/Matroska/MatroskaDemuxer.h>
LibVideo: Add PlaybackManager to load and decode videos This file will be the basis for abstracting away the out-of-thread or later out-of-process decoding from applications displaying videos. For now, the demuxer is hardcoded to be MatroskaParser, since that is all we support so far. The demuxer should later be selected based on the file header. The playback and decoding are currently all done on one thread using timers. The design of the code is such that adding threading should be trivial, at least based on an earlier version of the code. For now, though, it's better that this runs in one thread, as the multithreaded approach causes the Video Player to lock up permanently after a few frames are decoded.
2022-10-29 18:16:14 -05:00
#include <LibVideo/VP9/Decoder.h>
#include "PlaybackManager.h"
namespace Video {
LibVideo/PlaybackManager: Organize playback states into virtual classes Storing playback states in virtual classes allows the behavior to be much more clearly written. Each `PlaybackStateHandler` subclass can implement some event-handling functions to model their behavior, and has functions to change its parent PlaybackManager's state to any other state. This will allow expanding the functionality of playback in the future, for example to allow skipping a single frame forward/backward.
2023-02-06 01:24:33 -06:00
#define TRY_OR_FATAL_ERROR(expression) \
({ \
auto _fatal_expression = (expression); \
if (_fatal_expression.is_error()) { \
dispatch_fatal_error(_fatal_expression.release_error()); \
return; \
} \
static_assert(!::AK::Detail::IsLvalueReference<decltype(_fatal_expression.release_value())>, \
"Do not return a reference from a fallible expression"); \
_fatal_expression.release_value(); \
})
VideoPlayer: Make PlaybackManager use OwnPtr VideoPlayerWidget was keeping a reference to PlaybackManager when changing files, so the old and new managers would both send frames to be presented at the same time, causing it to flicker back and forth between the two videos. However, PlaybackManager no longer relies on event bubbling to pass events to its parent. By changing it to send events directly to an Object, it can avoid being ref counted, so that it will get destroyed with its containing object and stop sending events.
2022-11-11 07:55:20 -06:00
DecoderErrorOr<NonnullOwnPtr<PlaybackManager>> PlaybackManager::from_file(Core::Object& event_handler, StringView filename)
LibVideo: Add PlaybackManager to load and decode videos This file will be the basis for abstracting away the out-of-thread or later out-of-process decoding from applications displaying videos. For now, the demuxer is hardcoded to be MatroskaParser, since that is all we support so far. The demuxer should later be selected based on the file header. The playback and decoding are currently all done on one thread using timers. The design of the code is such that adding threading should be trivial, at least based on an earlier version of the code. For now, though, it's better that this runs in one thread, as the multithreaded approach causes the Video Player to lock up permanently after a few frames are decoded.
2022-10-29 18:16:14 -05:00
{
LibVideo: Reorganize demuxer file hierarchy and rename Matroska files As new demuxers are added, this will get quite full of files, so it'll be good to have a separate folder for these. To avoid too many chained namespaces, the Containers subdirectory is not also a namespace, but the Matroska folder is for the sake of separating the multiple classes for parsed information entering the Video namespace.
2022-11-09 19:47:56 -06:00
NonnullOwnPtr<Demuxer> demuxer = TRY(Matroska::MatroskaDemuxer::from_file(filename));
LibVideo: Read Matroska lazily so that large files can start quickly The Demuxer class was changed to return errors for more functions so that all of the underlying reading can be done lazily. Other than that, the demuxer interface is unchanged, and only the underlying reader was modified. The MatroskaDocument class is no more, and MatroskaReader's getter functions replace it. Every MatroskaReader getter beyond the Segment element's position is parsed lazily from the file as needed. This means that all getter functions can return DecoderErrors which must be handled by callers.
2022-11-11 17:14:27 -06:00
auto video_tracks = TRY(demuxer->get_tracks_for_type(TrackType::Video));
LibVideo: Add PlaybackManager to load and decode videos This file will be the basis for abstracting away the out-of-thread or later out-of-process decoding from applications displaying videos. For now, the demuxer is hardcoded to be MatroskaParser, since that is all we support so far. The demuxer should later be selected based on the file header. The playback and decoding are currently all done on one thread using timers. The design of the code is such that adding threading should be trivial, at least based on an earlier version of the code. For now, though, it's better that this runs in one thread, as the multithreaded approach causes the Video Player to lock up permanently after a few frames are decoded.
2022-10-29 18:16:14 -05:00
if (video_tracks.is_empty())
return DecoderError::with_description(DecoderErrorCategory::Invalid, "No video track is present"sv);
auto track = video_tracks[0];
dbgln_if(PLAYBACK_MANAGER_DEBUG, "Selecting video track number {}", track.identifier());
VideoPlayer: Make PlaybackManager use OwnPtr VideoPlayerWidget was keeping a reference to PlaybackManager when changing files, so the old and new managers would both send frames to be presented at the same time, causing it to flicker back and forth between the two videos. However, PlaybackManager no longer relies on event bubbling to pass events to its parent. By changing it to send events directly to an Object, it can avoid being ref counted, so that it will get destroyed with its containing object and stop sending events.
2022-11-11 07:55:20 -06:00
return make<PlaybackManager>(event_handler, demuxer, track, make<VP9::Decoder>());
LibVideo: Add PlaybackManager to load and decode videos This file will be the basis for abstracting away the out-of-thread or later out-of-process decoding from applications displaying videos. For now, the demuxer is hardcoded to be MatroskaParser, since that is all we support so far. The demuxer should later be selected based on the file header. The playback and decoding are currently all done on one thread using timers. The design of the code is such that adding threading should be trivial, at least based on an earlier version of the code. For now, though, it's better that this runs in one thread, as the multithreaded approach causes the Video Player to lock up permanently after a few frames are decoded.
2022-10-29 18:16:14 -05:00
}
VideoPlayer: Make PlaybackManager use OwnPtr VideoPlayerWidget was keeping a reference to PlaybackManager when changing files, so the old and new managers would both send frames to be presented at the same time, causing it to flicker back and forth between the two videos. However, PlaybackManager no longer relies on event bubbling to pass events to its parent. By changing it to send events directly to an Object, it can avoid being ref counted, so that it will get destroyed with its containing object and stop sending events.
2022-11-11 07:55:20 -06:00
PlaybackManager::PlaybackManager(Core::Object& event_handler, NonnullOwnPtr<Demuxer>& demuxer, Track video_track, NonnullOwnPtr<VideoDecoder>&& decoder)
: m_event_handler(event_handler)
LibVideo: Add PlaybackManager to load and decode videos This file will be the basis for abstracting away the out-of-thread or later out-of-process decoding from applications displaying videos. For now, the demuxer is hardcoded to be MatroskaParser, since that is all we support so far. The demuxer should later be selected based on the file header. The playback and decoding are currently all done on one thread using timers. The design of the code is such that adding threading should be trivial, at least based on an earlier version of the code. For now, though, it's better that this runs in one thread, as the multithreaded approach causes the Video Player to lock up permanently after a few frames are decoded.
2022-10-29 18:16:14 -05:00
, m_main_loop(Core::EventLoop::current())
, m_demuxer(move(demuxer))
, m_selected_video_track(video_track)
, m_decoder(move(decoder))
, m_frame_queue(make<VideoFrameQueue>())
LibVideo/PlaybackManager: Organize playback states into virtual classes Storing playback states in virtual classes allows the behavior to be much more clearly written. Each `PlaybackStateHandler` subclass can implement some event-handling functions to model their behavior, and has functions to change its parent PlaybackManager's state to any other state. This will allow expanding the functionality of playback in the future, for example to allow skipping a single frame forward/backward.
2023-02-06 01:24:33 -06:00
, m_playback_handler(make<StoppedStateHandler>(*this))
LibVideo: Add PlaybackManager to load and decode videos This file will be the basis for abstracting away the out-of-thread or later out-of-process decoding from applications displaying videos. For now, the demuxer is hardcoded to be MatroskaParser, since that is all we support so far. The demuxer should later be selected based on the file header. The playback and decoding are currently all done on one thread using timers. The design of the code is such that adding threading should be trivial, at least based on an earlier version of the code. For now, though, it's better that this runs in one thread, as the multithreaded approach causes the Video Player to lock up permanently after a few frames are decoded.
2022-10-29 18:16:14 -05:00
{
LibVideo/PlaybackManager: Organize playback states into virtual classes Storing playback states in virtual classes allows the behavior to be much more clearly written. Each `PlaybackStateHandler` subclass can implement some event-handling functions to model their behavior, and has functions to change its parent PlaybackManager's state to any other state. This will allow expanding the functionality of playback in the future, for example to allow skipping a single frame forward/backward.
2023-02-06 01:24:33 -06:00
m_present_timer = Core::Timer::create_single_shot(0, [&] { timer_callback(); }).release_value_but_fixme_should_propagate_errors();
Userland: Use Core::Timer::create_foo() factory functions where possible
2023-01-11 20:00:46 +00:00
m_decode_timer = Core::Timer::create_single_shot(0, [&] { on_decode_timer(); }).release_value_but_fixme_should_propagate_errors();
LibVideo/PlaybackManager: Organize playback states into virtual classes Storing playback states in virtual classes allows the behavior to be much more clearly written. Each `PlaybackStateHandler` subclass can implement some event-handling functions to model their behavior, and has functions to change its parent PlaybackManager's state to any other state. This will allow expanding the functionality of playback in the future, for example to allow skipping a single frame forward/backward.
2023-02-06 01:24:33 -06:00
TRY_OR_FATAL_ERROR(m_playback_handler->on_enter());
LibVideo: Add PlaybackManager to load and decode videos This file will be the basis for abstracting away the out-of-thread or later out-of-process decoding from applications displaying videos. For now, the demuxer is hardcoded to be MatroskaParser, since that is all we support so far. The demuxer should later be selected based on the file header. The playback and decoding are currently all done on one thread using timers. The design of the code is such that adding threading should be trivial, at least based on an earlier version of the code. For now, though, it's better that this runs in one thread, as the multithreaded approach causes the Video Player to lock up permanently after a few frames are decoded.
2022-10-29 18:16:14 -05:00
}
void PlaybackManager::resume_playback()
{
LibVideo/PlaybackManager: Organize playback states into virtual classes Storing playback states in virtual classes allows the behavior to be much more clearly written. Each `PlaybackStateHandler` subclass can implement some event-handling functions to model their behavior, and has functions to change its parent PlaybackManager's state to any other state. This will allow expanding the functionality of playback in the future, for example to allow skipping a single frame forward/backward.
2023-02-06 01:24:33 -06:00
dbgln_if(PLAYBACK_MANAGER_DEBUG, "Resuming playback.");
TRY_OR_FATAL_ERROR(m_playback_handler->play());
LibVideo: Add PlaybackManager to load and decode videos This file will be the basis for abstracting away the out-of-thread or later out-of-process decoding from applications displaying videos. For now, the demuxer is hardcoded to be MatroskaParser, since that is all we support so far. The demuxer should later be selected based on the file header. The playback and decoding are currently all done on one thread using timers. The design of the code is such that adding threading should be trivial, at least based on an earlier version of the code. For now, though, it's better that this runs in one thread, as the multithreaded approach causes the Video Player to lock up permanently after a few frames are decoded.
2022-10-29 18:16:14 -05:00
}
void PlaybackManager::pause_playback()
{
LibVideo/PlaybackManager: Organize playback states into virtual classes Storing playback states in virtual classes allows the behavior to be much more clearly written. Each `PlaybackStateHandler` subclass can implement some event-handling functions to model their behavior, and has functions to change its parent PlaybackManager's state to any other state. This will allow expanding the functionality of playback in the future, for example to allow skipping a single frame forward/backward.
2023-02-06 01:24:33 -06:00
dbgln_if(PLAYBACK_MANAGER_DEBUG, "Pausing playback.");
if (!m_playback_handler->is_playing())
warnln("Cannot pause.");
TRY_OR_FATAL_ERROR(m_playback_handler->pause());
LibVideo: Add PlaybackManager to load and decode videos This file will be the basis for abstracting away the out-of-thread or later out-of-process decoding from applications displaying videos. For now, the demuxer is hardcoded to be MatroskaParser, since that is all we support so far. The demuxer should later be selected based on the file header. The playback and decoding are currently all done on one thread using timers. The design of the code is such that adding threading should be trivial, at least based on an earlier version of the code. For now, though, it's better that this runs in one thread, as the multithreaded approach causes the Video Player to lock up permanently after a few frames are decoded.
2022-10-29 18:16:14 -05:00
}
Time PlaybackManager::current_playback_time()
{
LibVideo/PlaybackManager: Organize playback states into virtual classes Storing playback states in virtual classes allows the behavior to be much more clearly written. Each `PlaybackStateHandler` subclass can implement some event-handling functions to model their behavior, and has functions to change its parent PlaybackManager's state to any other state. This will allow expanding the functionality of playback in the future, for example to allow skipping a single frame forward/backward.
2023-02-06 01:24:33 -06:00
return m_playback_handler->current_time();
LibVideo: Add PlaybackManager to load and decode videos This file will be the basis for abstracting away the out-of-thread or later out-of-process decoding from applications displaying videos. For now, the demuxer is hardcoded to be MatroskaParser, since that is all we support so far. The demuxer should later be selected based on the file header. The playback and decoding are currently all done on one thread using timers. The design of the code is such that adding threading should be trivial, at least based on an earlier version of the code. For now, though, it's better that this runs in one thread, as the multithreaded approach causes the Video Player to lock up permanently after a few frames are decoded.
2022-10-29 18:16:14 -05:00
}
Time PlaybackManager::duration()
{
LibVideo: Read Matroska lazily so that large files can start quickly The Demuxer class was changed to return errors for more functions so that all of the underlying reading can be done lazily. Other than that, the demuxer interface is unchanged, and only the underlying reader was modified. The MatroskaDocument class is no more, and MatroskaReader's getter functions replace it. Every MatroskaReader getter beyond the Segment element's position is parsed lazily from the file as needed. This means that all getter functions can return DecoderErrors which must be handled by callers.
2022-11-11 17:14:27 -06:00
auto duration_result = m_demuxer->duration();
if (duration_result.is_error())
on_decoder_error(duration_result.release_error());
return duration_result.release_value();
LibVideo: Add PlaybackManager to load and decode videos This file will be the basis for abstracting away the out-of-thread or later out-of-process decoding from applications displaying videos. For now, the demuxer is hardcoded to be MatroskaParser, since that is all we support so far. The demuxer should later be selected based on the file header. The playback and decoding are currently all done on one thread using timers. The design of the code is such that adding threading should be trivial, at least based on an earlier version of the code. For now, though, it's better that this runs in one thread, as the multithreaded approach causes the Video Player to lock up permanently after a few frames are decoded.
2022-10-29 18:16:14 -05:00
}
LibVideo/PlaybackManager: Organize playback states into virtual classes Storing playback states in virtual classes allows the behavior to be much more clearly written. Each `PlaybackStateHandler` subclass can implement some event-handling functions to model their behavior, and has functions to change its parent PlaybackManager's state to any other state. This will allow expanding the functionality of playback in the future, for example to allow skipping a single frame forward/backward.
2023-02-06 01:24:33 -06:00
void PlaybackManager::dispatch_fatal_error(Error error)
LibVideo: Handle corrupted video errors without spamming dialogs No longer will the video player explode with error dialogs that then lock the user out of closing them. To avoid issues where the playback state becomes invalid when an error occurs, I've made all decoder errors pass through the frame queue. This way, when a video is corrupted, there should be no chance that the playback state becomes invalid due to setting the state to Corrupted in the event handler while a presentation event is still pending. Or at least I think that was what caused some issues I was seeing :^) This system should be a lot more robust if any future errors need to be handled.
2022-11-09 06:52:37 -06:00
{
LibVideo/PlaybackManager: Organize playback states into virtual classes Storing playback states in virtual classes allows the behavior to be much more clearly written. Each `PlaybackStateHandler` subclass can implement some event-handling functions to model their behavior, and has functions to change its parent PlaybackManager's state to any other state. This will allow expanding the functionality of playback in the future, for example to allow skipping a single frame forward/backward.
2023-02-06 01:24:33 -06:00
dbgln_if(PLAYBACK_MANAGER_DEBUG, "Encountered fatal error: {}", error.string_literal());
// FIXME: For threading, this will have to use a pre-allocated event to send to the main loop
// to be able to gracefully handle OOM.
VERIFY(&m_main_loop == &Core::EventLoop::current());
Everywhere: Remove needless copies of Error / ErrorOr instances Either take the underlying objects with release_* methods or move() the instances around.
2023-02-09 13:26:53 -05:00
FatalPlaybackErrorEvent event { move(error) };
LibVideo/PlaybackManager: Organize playback states into virtual classes Storing playback states in virtual classes allows the behavior to be much more clearly written. Each `PlaybackStateHandler` subclass can implement some event-handling functions to model their behavior, and has functions to change its parent PlaybackManager's state to any other state. This will allow expanding the functionality of playback in the future, for example to allow skipping a single frame forward/backward.
2023-02-06 01:24:33 -06:00
m_event_handler.dispatch_event(event);
}
LibVideo: Don't crash when a decoder error is encountered while seeking When errors are encountered by PlaybackManager, it attempts to switch states to either Stopped or Corrupted. However, that causes it to set the last presentation media time to the current playback time while the last presentation time is unexpectedly negative because the seek never ended. Ending the seek before the state changes to Stopped or Corrupted prevents this situation from happening.
2022-11-27 00:45:14 -06:00
LibVideo/PlaybackManager: Organize playback states into virtual classes Storing playback states in virtual classes allows the behavior to be much more clearly written. Each `PlaybackStateHandler` subclass can implement some event-handling functions to model their behavior, and has functions to change its parent PlaybackManager's state to any other state. This will allow expanding the functionality of playback in the future, for example to allow skipping a single frame forward/backward.
2023-02-06 01:24:33 -06:00
void PlaybackManager::on_decoder_error(DecoderError error)
{
LibVideo: Handle corrupted video errors without spamming dialogs No longer will the video player explode with error dialogs that then lock the user out of closing them. To avoid issues where the playback state becomes invalid when an error occurs, I've made all decoder errors pass through the frame queue. This way, when a video is corrupted, there should be no chance that the playback state becomes invalid due to setting the state to Corrupted in the event handler while a presentation event is still pending. Or at least I think that was what caused some issues I was seeing :^) This system should be a lot more robust if any future errors need to be handled.
2022-11-09 06:52:37 -06:00
switch (error.category()) {
case DecoderErrorCategory::EndOfStream:
LibVideo: Only print non-fatal playback errors when debug mode is on
2022-11-11 22:04:38 -06:00
dbgln_if(PLAYBACK_MANAGER_DEBUG, "{}", error.string_literal());
LibVideo/PlaybackManager: Organize playback states into virtual classes Storing playback states in virtual classes allows the behavior to be much more clearly written. Each `PlaybackStateHandler` subclass can implement some event-handling functions to model their behavior, and has functions to change its parent PlaybackManager's state to any other state. This will allow expanding the functionality of playback in the future, for example to allow skipping a single frame forward/backward.
2023-02-06 01:24:33 -06:00
TRY_OR_FATAL_ERROR(m_playback_handler->stop());
LibVideo: Handle corrupted video errors without spamming dialogs No longer will the video player explode with error dialogs that then lock the user out of closing them. To avoid issues where the playback state becomes invalid when an error occurs, I've made all decoder errors pass through the frame queue. This way, when a video is corrupted, there should be no chance that the playback state becomes invalid due to setting the state to Corrupted in the event handler while a presentation event is still pending. Or at least I think that was what caused some issues I was seeing :^) This system should be a lot more robust if any future errors need to be handled.
2022-11-09 06:52:37 -06:00
break;
default:
LibVideo: Only print non-fatal playback errors when debug mode is on
2022-11-11 22:04:38 -06:00
dbgln("Playback error encountered: {}", error.string_literal());
LibVideo/PlaybackManager: Organize playback states into virtual classes Storing playback states in virtual classes allows the behavior to be much more clearly written. Each `PlaybackStateHandler` subclass can implement some event-handling functions to model their behavior, and has functions to change its parent PlaybackManager's state to any other state. This will allow expanding the functionality of playback in the future, for example to allow skipping a single frame forward/backward.
2023-02-06 01:24:33 -06:00
TRY_OR_FATAL_ERROR(m_playback_handler->stop());
VideoPlayer: Make PlaybackManager use OwnPtr VideoPlayerWidget was keeping a reference to PlaybackManager when changing files, so the old and new managers would both send frames to be presented at the same time, causing it to flicker back and forth between the two videos. However, PlaybackManager no longer relies on event bubbling to pass events to its parent. By changing it to send events directly to an Object, it can avoid being ref counted, so that it will get destroyed with its containing object and stop sending events.
2022-11-11 07:55:20 -06:00
m_main_loop.post_event(m_event_handler, make<DecoderErrorEvent>(move(error)));
LibVideo: Handle corrupted video errors without spamming dialogs No longer will the video player explode with error dialogs that then lock the user out of closing them. To avoid issues where the playback state becomes invalid when an error occurs, I've made all decoder errors pass through the frame queue. This way, when a video is corrupted, there should be no chance that the playback state becomes invalid due to setting the state to Corrupted in the event handler while a presentation event is still pending. Or at least I think that was what caused some issues I was seeing :^) This system should be a lot more robust if any future errors need to be handled.
2022-11-09 06:52:37 -06:00
break;
}
}
LibVideo/PlaybackManager: Organize playback states into virtual classes Storing playback states in virtual classes allows the behavior to be much more clearly written. Each `PlaybackStateHandler` subclass can implement some event-handling functions to model their behavior, and has functions to change its parent PlaybackManager's state to any other state. This will allow expanding the functionality of playback in the future, for example to allow skipping a single frame forward/backward.
2023-02-06 01:24:33 -06:00
void PlaybackManager::timer_callback()
LibVideo: Implement accurate seeking to inter frames in PlaybackManager This implements the PlaybackManager portion of seeking, so that it can seek to any frame in the video by dropping all preceding frames until it reaches the seek point. MatroskaDemuxer currently will only seek to the start of the video. That means that every seek has to drop all the frames until it comes across the target timestamp.
2022-11-14 00:54:21 -06:00
{
LibVideo/PlaybackManager: Organize playback states into virtual classes Storing playback states in virtual classes allows the behavior to be much more clearly written. Each `PlaybackStateHandler` subclass can implement some event-handling functions to model their behavior, and has functions to change its parent PlaybackManager's state to any other state. This will allow expanding the functionality of playback in the future, for example to allow skipping a single frame forward/backward.
2023-02-06 01:24:33 -06:00
TRY_OR_FATAL_ERROR(m_playback_handler->on_timer_callback());
LibVideo: Implement accurate seeking to inter frames in PlaybackManager This implements the PlaybackManager portion of seeking, so that it can seek to any frame in the video by dropping all preceding frames until it reaches the seek point. MatroskaDemuxer currently will only seek to the start of the video. That means that every seek has to drop all the frames until it comes across the target timestamp.
2022-11-14 00:54:21 -06:00
}
VideoPlayer/LibVideo: Seek accurately when scrolling in the seek bar Fast seeking does not work correctly when seeking in small increments, so it is necessary to use accurate seeking when using certain actions. The PlaybackManager has been changed to accept the seek mode as a parameter to `seek_to_timestamp` to facilitate this. This now means that it no longer has to track a seek mode preference.
2023-02-06 04:11:32 -06:00
void PlaybackManager::seek_to_timestamp(Time target_timestamp, SeekMode seek_mode)
LibVideo: Add PlaybackManager to load and decode videos This file will be the basis for abstracting away the out-of-thread or later out-of-process decoding from applications displaying videos. For now, the demuxer is hardcoded to be MatroskaParser, since that is all we support so far. The demuxer should later be selected based on the file header. The playback and decoding are currently all done on one thread using timers. The design of the code is such that adding threading should be trivial, at least based on an earlier version of the code. For now, though, it's better that this runs in one thread, as the multithreaded approach causes the Video Player to lock up permanently after a few frames are decoded.
2022-10-29 18:16:14 -05:00
{
VideoPlayer/LibVideo: Seek accurately when scrolling in the seek bar Fast seeking does not work correctly when seeking in small increments, so it is necessary to use accurate seeking when using certain actions. The PlaybackManager has been changed to accept the seek mode as a parameter to `seek_to_timestamp` to facilitate this. This now means that it no longer has to track a seek mode preference.
2023-02-06 04:11:32 -06:00
TRY_OR_FATAL_ERROR(m_playback_handler->seek(target_timestamp, seek_mode));
LibVideo: Add PlaybackManager to load and decode videos This file will be the basis for abstracting away the out-of-thread or later out-of-process decoding from applications displaying videos. For now, the demuxer is hardcoded to be MatroskaParser, since that is all we support so far. The demuxer should later be selected based on the file header. The playback and decoding are currently all done on one thread using timers. The design of the code is such that adding threading should be trivial, at least based on an earlier version of the code. For now, though, it's better that this runs in one thread, as the multithreaded approach causes the Video Player to lock up permanently after a few frames are decoded.
2022-10-29 18:16:14 -05:00
}
LibVideo: Pass the current sample to demuxers to lazily seek better In cases where the PlaybackManager's earliest buffered or displayed sample is closer to the seek target than the demuxer's chosen keyframe, we don't want to seek at all. To enable this, demuxers now receive an optional parameter with the earliest timestamp that the caller can still access. The demuxer in turn returns an optional to indicate when a seek was not needed, which allows PlaybackManager to avoid clearing its queue and re-decoding frames.
2023-02-06 01:25:02 -06:00
Optional<Time> PlaybackManager::seek_demuxer_to_most_recent_keyframe(Time timestamp, Optional<Time> earliest_available_sample)
LibVideo: Add PlaybackManager to load and decode videos This file will be the basis for abstracting away the out-of-thread or later out-of-process decoding from applications displaying videos. For now, the demuxer is hardcoded to be MatroskaParser, since that is all we support so far. The demuxer should later be selected based on the file header. The playback and decoding are currently all done on one thread using timers. The design of the code is such that adding threading should be trivial, at least based on an earlier version of the code. For now, though, it's better that this runs in one thread, as the multithreaded approach causes the Video Player to lock up permanently after a few frames are decoded.
2022-10-29 18:16:14 -05:00
{
VideoPlayer/LibVideo: Implement the UI functionality for seeking With these changes, the seek bar can be used, but only to seek to the start of the file. Seeking to anywhere else in the file will cause an error in the demuxer. The timestamp label that was previously invisible now has its text set according to either the playback or seek slider's position.
2022-11-11 22:26:19 -06:00
// FIXME: When the demuxer is getting samples off the main thread in the future, this needs to
// mutex so that seeking can't happen while that thread is getting a sample.
LibVideo: Pass the current sample to demuxers to lazily seek better In cases where the PlaybackManager's earliest buffered or displayed sample is closer to the seek target than the demuxer's chosen keyframe, we don't want to seek at all. To enable this, demuxers now receive an optional parameter with the earliest timestamp that the caller can still access. The demuxer in turn returns an optional to indicate when a seek was not needed, which allows PlaybackManager to avoid clearing its queue and re-decoding frames.
2023-02-06 01:25:02 -06:00
auto result = m_demuxer->seek_to_most_recent_keyframe(m_selected_video_track, timestamp, move(earliest_available_sample));
LibVideo: Handle corrupted video errors without spamming dialogs No longer will the video player explode with error dialogs that then lock the user out of closing them. To avoid issues where the playback state becomes invalid when an error occurs, I've made all decoder errors pass through the frame queue. This way, when a video is corrupted, there should be no chance that the playback state becomes invalid due to setting the state to Corrupted in the event handler while a presentation event is still pending. Or at least I think that was what caused some issues I was seeing :^) This system should be a lot more robust if any future errors need to be handled.
2022-11-09 06:52:37 -06:00
if (result.is_error())
on_decoder_error(result.release_error());
LibVideo/PlaybackManager: Organize playback states into virtual classes Storing playback states in virtual classes allows the behavior to be much more clearly written. Each `PlaybackStateHandler` subclass can implement some event-handling functions to model their behavior, and has functions to change its parent PlaybackManager's state to any other state. This will allow expanding the functionality of playback in the future, for example to allow skipping a single frame forward/backward.
2023-02-06 01:24:33 -06:00
return result.release_value();
LibVideo: Add PlaybackManager to load and decode videos This file will be the basis for abstracting away the out-of-thread or later out-of-process decoding from applications displaying videos. For now, the demuxer is hardcoded to be MatroskaParser, since that is all we support so far. The demuxer should later be selected based on the file header. The playback and decoding are currently all done on one thread using timers. The design of the code is such that adding threading should be trivial, at least based on an earlier version of the code. For now, though, it's better that this runs in one thread, as the multithreaded approach causes the Video Player to lock up permanently after a few frames are decoded.
2022-10-29 18:16:14 -05:00
}
VideoPlayer/LibVideo: Implement the UI functionality for seeking With these changes, the seek bar can be used, but only to seek to the start of the file. Seeking to anywhere else in the file will cause an error in the demuxer. The timestamp label that was previously invisible now has its text set according to either the playback or seek slider's position.
2022-11-11 22:26:19 -06:00
void PlaybackManager::restart_playback()
{
seek_to_timestamp(Time::zero());
}
LibVideo/PlaybackManager: Organize playback states into virtual classes Storing playback states in virtual classes allows the behavior to be much more clearly written. Each `PlaybackStateHandler` subclass can implement some event-handling functions to model their behavior, and has functions to change its parent PlaybackManager's state to any other state. This will allow expanding the functionality of playback in the future, for example to allow skipping a single frame forward/backward.
2023-02-06 01:24:33 -06:00
void PlaybackManager::dispatch_decoder_error(DecoderError error)
LibVideo: Allow the VP9 decoder to queue multiple frames Frames will now be queued for retrieval by the user of the decoder. When the end of the current queue is reached, a DecoderError of category NeedsMoreInput will be emitted, allowing the caller to react by displaying what was previously retrieved for sending more samples.
2022-11-03 19:18:38 -05:00
{
VideoPlayer: Make PlaybackManager use OwnPtr VideoPlayerWidget was keeping a reference to PlaybackManager when changing files, so the old and new managers would both send frames to be presented at the same time, causing it to flicker back and forth between the two videos. However, PlaybackManager no longer relies on event bubbling to pass events to its parent. By changing it to send events directly to an Object, it can avoid being ref counted, so that it will get destroyed with its containing object and stop sending events.
2022-11-11 07:55:20 -06:00
m_main_loop.post_event(m_event_handler, make<DecoderErrorEvent>(error));
LibVideo: Allow the VP9 decoder to queue multiple frames Frames will now be queued for retrieval by the user of the decoder. When the end of the current queue is reached, a DecoderError of category NeedsMoreInput will be emitted, allowing the caller to react by displaying what was previously retrieved for sending more samples.
2022-11-03 19:18:38 -05:00
}
LibVideo/PlaybackManager: Organize playback states into virtual classes Storing playback states in virtual classes allows the behavior to be much more clearly written. Each `PlaybackStateHandler` subclass can implement some event-handling functions to model their behavior, and has functions to change its parent PlaybackManager's state to any other state. This will allow expanding the functionality of playback in the future, for example to allow skipping a single frame forward/backward.
2023-02-06 01:24:33 -06:00
void PlaybackManager::dispatch_new_frame(RefPtr<Gfx::Bitmap> frame)
{
m_main_loop.post_event(m_event_handler, make<VideoFramePresentEvent>(frame));
}
void PlaybackManager::start_timer(int milliseconds)
{
m_present_timer->start(milliseconds);
}
LibVideo: Add PlaybackManager to load and decode videos This file will be the basis for abstracting away the out-of-thread or later out-of-process decoding from applications displaying videos. For now, the demuxer is hardcoded to be MatroskaParser, since that is all we support so far. The demuxer should later be selected based on the file header. The playback and decoding are currently all done on one thread using timers. The design of the code is such that adding threading should be trivial, at least based on an earlier version of the code. For now, though, it's better that this runs in one thread, as the multithreaded approach causes the Video Player to lock up permanently after a few frames are decoded.
2022-10-29 18:16:14 -05:00
bool PlaybackManager::decode_and_queue_one_sample()
{
LibVideo: Rewrite the video frame present function to be more readable The PlaybackManager::update_presented_frame function was getting out of hand and adding seeking was making it illegible. This rewrites it to be (hopefully) quite a bit more readable, and adds a few comments to help future readers of the code. In addition, some helpful debugging prints were added that should help debug any future issues with the player.
2022-11-14 00:48:31 -06:00
if (m_frame_queue->size() >= FRAME_BUFFER_COUNT) {
dbgln_if(PLAYBACK_MANAGER_DEBUG, "Frame queue is full, stopping");
LibVideo: Add PlaybackManager to load and decode videos This file will be the basis for abstracting away the out-of-thread or later out-of-process decoding from applications displaying videos. For now, the demuxer is hardcoded to be MatroskaParser, since that is all we support so far. The demuxer should later be selected based on the file header. The playback and decoding are currently all done on one thread using timers. The design of the code is such that adding threading should be trivial, at least based on an earlier version of the code. For now, though, it's better that this runs in one thread, as the multithreaded approach causes the Video Player to lock up permanently after a few frames are decoded.
2022-10-29 18:16:14 -05:00
return false;
LibVideo: Rewrite the video frame present function to be more readable The PlaybackManager::update_presented_frame function was getting out of hand and adding seeking was making it illegible. This rewrites it to be (hopefully) quite a bit more readable, and adds a few comments to help future readers of the code. In addition, some helpful debugging prints were added that should help debug any future issues with the player.
2022-11-14 00:48:31 -06:00
}
LibVideo: Add PlaybackManager to load and decode videos This file will be the basis for abstracting away the out-of-thread or later out-of-process decoding from applications displaying videos. For now, the demuxer is hardcoded to be MatroskaParser, since that is all we support so far. The demuxer should later be selected based on the file header. The playback and decoding are currently all done on one thread using timers. The design of the code is such that adding threading should be trivial, at least based on an earlier version of the code. For now, though, it's better that this runs in one thread, as the multithreaded approach causes the Video Player to lock up permanently after a few frames are decoded.
2022-10-29 18:16:14 -05:00
#if PLAYBACK_MANAGER_DEBUG
auto start_time = Time::now_monotonic();
#endif
LibVideo: Only print non-fatal playback errors when debug mode is on
2022-11-11 22:04:38 -06:00
#define TRY_OR_ENQUEUE_ERROR(expression) \
({ \
auto _temporary_result = ((expression)); \
if (_temporary_result.is_error()) { \
dbgln_if(PLAYBACK_MANAGER_DEBUG, "Enqueued decoder error: {}", _temporary_result.error().string_literal()); \
m_frame_queue->enqueue(FrameQueueItem::error_marker(_temporary_result.release_error())); \
return false; \
} \
Everywhere: Fully qualify IsLvalueReference in TRY() macros If USING_AK_GLOBALLY is not defined, the name IsLvalueReference might not be available in the global namespace. Follow the pattern established in LibTest to fully qualify AK types in macros to avoid this problem.
2023-01-14 16:34:44 -07:00
static_assert(!::AK::Detail::IsLvalueReference<decltype(_temporary_result.release_value())>, \
AK+Everywhere: Disallow returning a reference from a fallible expression This will silently make a copy. Rather than masking this behavior, let's explicitly disallow it.
2023-01-13 15:48:09 -05:00
"Do not return a reference from a fallible expression"); \
LibVideo: Only print non-fatal playback errors when debug mode is on
2022-11-11 22:04:38 -06:00
_temporary_result.release_value(); \
LibVideo: Handle corrupted video errors without spamming dialogs No longer will the video player explode with error dialogs that then lock the user out of closing them. To avoid issues where the playback state becomes invalid when an error occurs, I've made all decoder errors pass through the frame queue. This way, when a video is corrupted, there should be no chance that the playback state becomes invalid due to setting the state to Corrupted in the event handler while a presentation event is still pending. Or at least I think that was what caused some issues I was seeing :^) This system should be a lot more robust if any future errors need to be handled.
2022-11-09 06:52:37 -06:00
})
auto frame_sample = TRY_OR_ENQUEUE_ERROR(m_demuxer->get_next_video_sample_for_track(m_selected_video_track));
LibVideo: Allow the VP9 decoder to queue multiple frames Frames will now be queued for retrieval by the user of the decoder. When the end of the current queue is reached, a DecoderError of category NeedsMoreInput will be emitted, allowing the caller to react by displaying what was previously retrieved for sending more samples.
2022-11-03 19:18:38 -05:00
OwnPtr<VideoFrame> decoded_frame = nullptr;
while (!decoded_frame) {
TRY_OR_ENQUEUE_ERROR(m_decoder->receive_sample(frame_sample->data()));
while (true) {
auto frame_result = m_decoder->get_decoded_frame();
if (frame_result.is_error()) {
if (frame_result.error().category() == DecoderErrorCategory::NeedsMoreInput)
break;
LibVideo: Add PlaybackManager to load and decode videos This file will be the basis for abstracting away the out-of-thread or later out-of-process decoding from applications displaying videos. For now, the demuxer is hardcoded to be MatroskaParser, since that is all we support so far. The demuxer should later be selected based on the file header. The playback and decoding are currently all done on one thread using timers. The design of the code is such that adding threading should be trivial, at least based on an earlier version of the code. For now, though, it's better that this runs in one thread, as the multithreaded approach causes the Video Player to lock up permanently after a few frames are decoded.
2022-10-29 18:16:14 -05:00
LibVideo/PlaybackManager: Organize playback states into virtual classes Storing playback states in virtual classes allows the behavior to be much more clearly written. Each `PlaybackStateHandler` subclass can implement some event-handling functions to model their behavior, and has functions to change its parent PlaybackManager's state to any other state. This will allow expanding the functionality of playback in the future, for example to allow skipping a single frame forward/backward.
2023-02-06 01:24:33 -06:00
dispatch_decoder_error(frame_result.release_error());
LibVideo: Allow the VP9 decoder to queue multiple frames Frames will now be queued for retrieval by the user of the decoder. When the end of the current queue is reached, a DecoderError of category NeedsMoreInput will be emitted, allowing the caller to react by displaying what was previously retrieved for sending more samples.
2022-11-03 19:18:38 -05:00
return false;
}
decoded_frame = frame_result.release_value();
VERIFY(decoded_frame);
}
}
LibVideo: Add PlaybackManager to load and decode videos This file will be the basis for abstracting away the out-of-thread or later out-of-process decoding from applications displaying videos. For now, the demuxer is hardcoded to be MatroskaParser, since that is all we support so far. The demuxer should later be selected based on the file header. The playback and decoding are currently all done on one thread using timers. The design of the code is such that adding threading should be trivial, at least based on an earlier version of the code. For now, though, it's better that this runs in one thread, as the multithreaded approach causes the Video Player to lock up permanently after a few frames are decoded.
2022-10-29 18:16:14 -05:00
auto& cicp = decoded_frame->cicp();
cicp.adopt_specified_values(frame_sample->container_cicp());
LibVideo: Rename "ColorRange" to "VideoFullRangeFlag" That matches the terminology used in ITU-T Rec. H.273, PNG's cICP chunk, and the ICC cicpTag. Also change the enum values to match the values in the spec -- 0 means "not full range" and 1 means "full range". (For now, keep the "Unspecified" entry around, and give it value 2. This value is not in the spec.) No intended behavior change.
2023-02-08 14:16:37 -05:00
cicp.default_code_points_if_unspecified({ ColorPrimaries::BT709, TransferCharacteristics::BT709, MatrixCoefficients::BT709, VideoFullRangeFlag::Studio });
LibVideo: Treat BT.601/709/2020 input transfer characteristics as sRGB I've realized that it probably makes more sense to change the input transfer characteristics to treat these as sRGB since color conversion in linear converted from BT.709 doesn't really make sense. If content creation applications expect media players to display BT.709 without conversions, this means they expect applications to treat it as sRGB, since that's what most displays use. That most likely also means they process it as sRGB internally, meaning we should do the same for our color primaries conversion.
2022-11-08 03:24:30 -06:00
// BT.601, BT.709 and BT.2020 have a similar transfer function to sRGB, so other applications
// (Chromium, VLC) forgo transfer characteristics conversion. We will emulate that behavior by
// handling those as sRGB instead, which causes no transfer function change in the output,
// unless display color management is later implemented.
switch (cicp.transfer_characteristics()) {
case TransferCharacteristics::BT601:
case TransferCharacteristics::BT709:
case TransferCharacteristics::BT2020BitDepth10:
case TransferCharacteristics::BT2020BitDepth12:
cicp.set_transfer_characteristics(TransferCharacteristics::SRGB);
break;
default:
break;
}
LibVideo: Add PlaybackManager to load and decode videos This file will be the basis for abstracting away the out-of-thread or later out-of-process decoding from applications displaying videos. For now, the demuxer is hardcoded to be MatroskaParser, since that is all we support so far. The demuxer should later be selected based on the file header. The playback and decoding are currently all done on one thread using timers. The design of the code is such that adding threading should be trivial, at least based on an earlier version of the code. For now, though, it's better that this runs in one thread, as the multithreaded approach causes the Video Player to lock up permanently after a few frames are decoded.
2022-10-29 18:16:14 -05:00
LibVideo: Handle corrupted video errors without spamming dialogs No longer will the video player explode with error dialogs that then lock the user out of closing them. To avoid issues where the playback state becomes invalid when an error occurs, I've made all decoder errors pass through the frame queue. This way, when a video is corrupted, there should be no chance that the playback state becomes invalid due to setting the state to Corrupted in the event handler while a presentation event is still pending. Or at least I think that was what caused some issues I was seeing :^) This system should be a lot more robust if any future errors need to be handled.
2022-11-09 06:52:37 -06:00
auto bitmap = TRY_OR_ENQUEUE_ERROR(decoded_frame->to_bitmap());
m_frame_queue->enqueue(FrameQueueItem::frame(bitmap, frame_sample->timestamp()));
LibVideo: Add PlaybackManager to load and decode videos This file will be the basis for abstracting away the out-of-thread or later out-of-process decoding from applications displaying videos. For now, the demuxer is hardcoded to be MatroskaParser, since that is all we support so far. The demuxer should later be selected based on the file header. The playback and decoding are currently all done on one thread using timers. The design of the code is such that adding threading should be trivial, at least based on an earlier version of the code. For now, though, it's better that this runs in one thread, as the multithreaded approach causes the Video Player to lock up permanently after a few frames are decoded.
2022-10-29 18:16:14 -05:00
#if PLAYBACK_MANAGER_DEBUG
auto end_time = Time::now_monotonic();
LibVideo/PlaybackManager: Organize playback states into virtual classes Storing playback states in virtual classes allows the behavior to be much more clearly written. Each `PlaybackStateHandler` subclass can implement some event-handling functions to model their behavior, and has functions to change its parent PlaybackManager's state to any other state. This will allow expanding the functionality of playback in the future, for example to allow skipping a single frame forward/backward.
2023-02-06 01:24:33 -06:00
dbgln("Decoding sample at {}ms took {}ms, queue contains {} items", frame_sample->timestamp().to_milliseconds(), (end_time - start_time).to_milliseconds(), m_frame_queue->size());
LibVideo: Add PlaybackManager to load and decode videos This file will be the basis for abstracting away the out-of-thread or later out-of-process decoding from applications displaying videos. For now, the demuxer is hardcoded to be MatroskaParser, since that is all we support so far. The demuxer should later be selected based on the file header. The playback and decoding are currently all done on one thread using timers. The design of the code is such that adding threading should be trivial, at least based on an earlier version of the code. For now, though, it's better that this runs in one thread, as the multithreaded approach causes the Video Player to lock up permanently after a few frames are decoded.
2022-10-29 18:16:14 -05:00
#endif
return true;
}
void PlaybackManager::on_decode_timer()
{
LibVideo/PlaybackManager: Organize playback states into virtual classes Storing playback states in virtual classes allows the behavior to be much more clearly written. Each `PlaybackStateHandler` subclass can implement some event-handling functions to model their behavior, and has functions to change its parent PlaybackManager's state to any other state. This will allow expanding the functionality of playback in the future, for example to allow skipping a single frame forward/backward.
2023-02-06 01:24:33 -06:00
if (!decode_and_queue_one_sample()) {
// Note: When threading is implemented, this must be dispatched via an event loop.
TRY_OR_FATAL_ERROR(m_playback_handler->on_buffer_filled());
LibVideo: Add PlaybackManager to load and decode videos This file will be the basis for abstracting away the out-of-thread or later out-of-process decoding from applications displaying videos. For now, the demuxer is hardcoded to be MatroskaParser, since that is all we support so far. The demuxer should later be selected based on the file header. The playback and decoding are currently all done on one thread using timers. The design of the code is such that adding threading should be trivial, at least based on an earlier version of the code. For now, though, it's better that this runs in one thread, as the multithreaded approach causes the Video Player to lock up permanently after a few frames are decoded.
2022-10-29 18:16:14 -05:00
return;
}
// Continually decode until buffering is complete
LibVideo/PlaybackManager: Organize playback states into virtual classes Storing playback states in virtual classes allows the behavior to be much more clearly written. Each `PlaybackStateHandler` subclass can implement some event-handling functions to model their behavior, and has functions to change its parent PlaybackManager's state to any other state. This will allow expanding the functionality of playback in the future, for example to allow skipping a single frame forward/backward.
2023-02-06 01:24:33 -06:00
m_decode_timer->start(0);
}
Time PlaybackManager::PlaybackStateHandler::current_time() const
{
return m_manager.m_last_present_in_media_time;
}
ErrorOr<void> PlaybackManager::PlaybackStateHandler::seek(Time target_timestamp, SeekMode seek_mode)
{
return replace_handler_and_delete_this<SeekingStateHandler>(is_playing(), target_timestamp, seek_mode);
LibVideo: Add PlaybackManager to load and decode videos This file will be the basis for abstracting away the out-of-thread or later out-of-process decoding from applications displaying videos. For now, the demuxer is hardcoded to be MatroskaParser, since that is all we support so far. The demuxer should later be selected based on the file header. The playback and decoding are currently all done on one thread using timers. The design of the code is such that adding threading should be trivial, at least based on an earlier version of the code. For now, though, it's better that this runs in one thread, as the multithreaded approach causes the Video Player to lock up permanently after a few frames are decoded.
2022-10-29 18:16:14 -05:00
}
LibVideo/PlaybackManager: Organize playback states into virtual classes Storing playback states in virtual classes allows the behavior to be much more clearly written. Each `PlaybackStateHandler` subclass can implement some event-handling functions to model their behavior, and has functions to change its parent PlaybackManager's state to any other state. This will allow expanding the functionality of playback in the future, for example to allow skipping a single frame forward/backward.
2023-02-06 01:24:33 -06:00
ErrorOr<void> PlaybackManager::PlaybackStateHandler::stop()
{
return replace_handler_and_delete_this<StoppedStateHandler>();
}
template<class T, class... Args>
ErrorOr<void> PlaybackManager::PlaybackStateHandler::replace_handler_and_delete_this(Args... args)
{
auto temp_handler = TRY(adopt_nonnull_own_or_enomem<PlaybackStateHandler>(new (nothrow) T(m_manager, args...)));
m_manager.m_playback_handler.swap(temp_handler);
#if PLAYBACK_MANAGER_DEBUG
m_has_exited = true;
dbgln("Changing state from {} to {}", temp_handler->name(), m_manager.m_playback_handler->name());
#endif
TRY(m_manager.m_playback_handler->on_enter());
return {};
}
PlaybackManager& PlaybackManager::PlaybackStateHandler::manager() const
{
#if PLAYBACK_MANAGER_DEBUG
VERIFY(!m_has_exited);
#endif
return m_manager;
}
class PlaybackManager::PlayingStateHandler : public PlaybackManager::PlaybackStateHandler {
public:
PlayingStateHandler(PlaybackManager& manager)
: PlaybackStateHandler(manager)
{
}
~PlayingStateHandler() override = default;
private:
ErrorOr<void> on_enter() override
{
m_last_present_in_real_time = Time::now_monotonic();
return on_timer_callback();
}
StringView name() override { return "Playing"sv; }
bool is_playing() override { return true; };
ErrorOr<void> pause() override
{
manager().m_last_present_in_media_time = current_time();
return replace_handler_and_delete_this<PausedStateHandler>();
}
ErrorOr<void> buffer() override
{
manager().m_last_present_in_media_time = current_time();
return replace_handler_and_delete_this<BufferingStateHandler>(true);
}
Time current_time() const override
{
return manager().m_last_present_in_media_time + (Time::now_monotonic() - m_last_present_in_real_time);
}
ErrorOr<void> on_timer_callback() override
{
Optional<FrameQueueItem> future_frame_item;
bool should_present_frame = false;
// Skip frames until we find a frame past the current playback time, and keep the one that precedes it to display.
while (!manager().m_frame_queue->is_empty()) {
future_frame_item.emplace(manager().m_frame_queue->dequeue());
manager().m_decode_timer->start(0);
if (future_frame_item->is_error() || future_frame_item->timestamp() >= current_time()) {
dbgln_if(PLAYBACK_MANAGER_DEBUG, "Should present frame, future {} is error or after {}ms", future_frame_item->debug_string(), current_time().to_milliseconds());
should_present_frame = true;
break;
}
if (manager().m_next_frame.has_value()) {
dbgln_if(PLAYBACK_MANAGER_DEBUG, "At {}ms: Dropped {} in favor of {}", current_time().to_milliseconds(), manager().m_next_frame->debug_string(), future_frame_item->debug_string());
manager().m_skipped_frames++;
}
manager().m_next_frame.emplace(future_frame_item.release_value());
}
// If we don't have both of these items, we can't present, since we need to set a timer for
// the next frame. Check if we need to buffer based on the current state.
if (!manager().m_next_frame.has_value() || !future_frame_item.has_value()) {
#if PLAYBACK_MANAGER_DEBUG
StringBuilder debug_string_builder;
debug_string_builder.append("We don't have "sv);
if (!manager().m_next_frame.has_value()) {
debug_string_builder.append("a frame to present"sv);
if (!future_frame_item.has_value())
debug_string_builder.append(" or a future frame"sv);
} else {
debug_string_builder.append("a future frame"sv);
}
debug_string_builder.append(", checking for error and buffering"sv);
dbgln_if(PLAYBACK_MANAGER_DEBUG, debug_string_builder.to_deprecated_string());
#endif
if (future_frame_item.has_value()) {
if (future_frame_item->is_error()) {
manager().on_decoder_error(future_frame_item.release_value().release_error());
return {};
}
manager().m_next_frame.emplace(future_frame_item.release_value());
}
TRY(buffer());
return {};
}
// If we have a frame, send it for presentation.
if (should_present_frame) {
auto now = Time::now_monotonic();
manager().m_last_present_in_media_time += now - m_last_present_in_real_time;
m_last_present_in_real_time = now;
manager().dispatch_new_frame(manager().m_next_frame.value().bitmap());
dbgln_if(PLAYBACK_MANAGER_DEBUG, "Sent frame for presentation");
}
// Now that we've presented the current frame, we can throw whatever error is next in queue.
// This way, we always display a frame before the stream ends, and should also show any frames
// we already had when a real error occurs.
if (future_frame_item->is_error()) {
manager().on_decoder_error(future_frame_item.release_value().release_error());
return {};
}
// The future frame item becomes the next one to present.
manager().m_next_frame.emplace(future_frame_item.release_value());
auto frame_time_ms = (manager().m_next_frame->timestamp() - current_time()).to_milliseconds();
VERIFY(frame_time_ms <= NumericLimits<int>::max());
dbgln_if(PLAYBACK_MANAGER_DEBUG, "Time until next frame is {}ms", frame_time_ms);
manager().start_timer(max(static_cast<int>(frame_time_ms), 0));
return {};
}
Time m_last_present_in_real_time = Time::zero();
};
class PlaybackManager::PausedStateHandler : public PlaybackManager::PlaybackStateHandler {
public:
PausedStateHandler(PlaybackManager& manager)
: PlaybackStateHandler(manager)
{
}
~PausedStateHandler() override = default;
private:
StringView name() override { return "Paused"sv; }
ErrorOr<void> play() override
{
return replace_handler_and_delete_this<PlayingStateHandler>();
}
bool is_playing() override { return false; };
};
class PlaybackManager::BufferingStateHandler : public PlaybackManager::PlaybackStateHandler {
public:
BufferingStateHandler(PlaybackManager& manager, bool playing)
: PlaybackStateHandler(manager)
, m_playing(playing)
{
}
~BufferingStateHandler() override = default;
private:
ErrorOr<void> on_enter() override
{
manager().m_decode_timer->start(0);
return {};
}
StringView name() override { return "Buffering"sv; }
ErrorOr<void> play() override
{
m_playing = true;
return {};
}
bool is_playing() override { return m_playing; };
ErrorOr<void> pause() override
{
m_playing = false;
return {};
}
ErrorOr<void> on_buffer_filled() override
{
if (m_playing)
return replace_handler_and_delete_this<PlayingStateHandler>();
return replace_handler_and_delete_this<PausedStateHandler>();
}
bool m_playing { false };
};
class PlaybackManager::SeekingStateHandler : public PlaybackManager::PlaybackStateHandler {
public:
SeekingStateHandler(PlaybackManager& manager, bool playing, Time target_timestamp, SeekMode seek_mode)
: PlaybackStateHandler(manager)
, m_playing(playing)
, m_target_timestamp(target_timestamp)
, m_seek_mode(seek_mode)
{
}
~SeekingStateHandler() override = default;
private:
ErrorOr<void> exit_seek()
{
if (m_playing)
return replace_handler_and_delete_this<PlayingStateHandler>();
return replace_handler_and_delete_this<PausedStateHandler>();
}
ErrorOr<void> on_enter() override
{
LibVideo: Pass the current sample to demuxers to lazily seek better In cases where the PlaybackManager's earliest buffered or displayed sample is closer to the seek target than the demuxer's chosen keyframe, we don't want to seek at all. To enable this, demuxers now receive an optional parameter with the earliest timestamp that the caller can still access. The demuxer in turn returns an optional to indicate when a seek was not needed, which allows PlaybackManager to avoid clearing its queue and re-decoding frames.
2023-02-06 01:25:02 -06:00
auto earliest_available_sample = manager().m_last_present_in_media_time;
if (manager().m_next_frame.has_value() && manager().m_next_frame->is_frame()) {
earliest_available_sample = min(earliest_available_sample, manager().m_next_frame->timestamp());
}
auto keyframe_timestamp = manager().seek_demuxer_to_most_recent_keyframe(m_target_timestamp, earliest_available_sample);
#if PLAYBACK_MANAGER_DEBUG
auto seek_mode_name = m_seek_mode == SeekMode::Accurate ? "Accurate"sv : "Fast"sv;
if (keyframe_timestamp.has_value()) {
dbgln("{} seeking to timestamp target {}ms, selected keyframe at {}ms", seek_mode_name, m_target_timestamp.to_milliseconds(), keyframe_timestamp->to_milliseconds());
} else {
dbgln("{} seeking to timestamp target {}ms, demuxer kept its iterator position", seek_mode_name, m_target_timestamp.to_milliseconds());
}
#endif
LibVideo/PlaybackManager: Organize playback states into virtual classes Storing playback states in virtual classes allows the behavior to be much more clearly written. Each `PlaybackStateHandler` subclass can implement some event-handling functions to model their behavior, and has functions to change its parent PlaybackManager's state to any other state. This will allow expanding the functionality of playback in the future, for example to allow skipping a single frame forward/backward.
2023-02-06 01:24:33 -06:00
if (m_seek_mode == SeekMode::Fast) {
LibVideo: Pass the current sample to demuxers to lazily seek better In cases where the PlaybackManager's earliest buffered or displayed sample is closer to the seek target than the demuxer's chosen keyframe, we don't want to seek at all. To enable this, demuxers now receive an optional parameter with the earliest timestamp that the caller can still access. The demuxer in turn returns an optional to indicate when a seek was not needed, which allows PlaybackManager to avoid clearing its queue and re-decoding frames.
2023-02-06 01:25:02 -06:00
m_target_timestamp = keyframe_timestamp.value_or(earliest_available_sample);
LibVideo/PlaybackManager: Organize playback states into virtual classes Storing playback states in virtual classes allows the behavior to be much more clearly written. Each `PlaybackStateHandler` subclass can implement some event-handling functions to model their behavior, and has functions to change its parent PlaybackManager's state to any other state. This will allow expanding the functionality of playback in the future, for example to allow skipping a single frame forward/backward.
2023-02-06 01:24:33 -06:00
}
LibVideo: Pass the current sample to demuxers to lazily seek better In cases where the PlaybackManager's earliest buffered or displayed sample is closer to the seek target than the demuxer's chosen keyframe, we don't want to seek at all. To enable this, demuxers now receive an optional parameter with the earliest timestamp that the caller can still access. The demuxer in turn returns an optional to indicate when a seek was not needed, which allows PlaybackManager to avoid clearing its queue and re-decoding frames.
2023-02-06 01:25:02 -06:00
if (keyframe_timestamp.has_value()) {
LibVideo/PlaybackManager: Organize playback states into virtual classes Storing playback states in virtual classes allows the behavior to be much more clearly written. Each `PlaybackStateHandler` subclass can implement some event-handling functions to model their behavior, and has functions to change its parent PlaybackManager's state to any other state. This will allow expanding the functionality of playback in the future, for example to allow skipping a single frame forward/backward.
2023-02-06 01:24:33 -06:00
dbgln_if(PLAYBACK_MANAGER_DEBUG, "Timestamp is earlier than current media time, clearing queue");
manager().m_frame_queue->clear();
manager().m_next_frame.clear();
LibVideo: Pass the current sample to demuxers to lazily seek better In cases where the PlaybackManager's earliest buffered or displayed sample is closer to the seek target than the demuxer's chosen keyframe, we don't want to seek at all. To enable this, demuxers now receive an optional parameter with the earliest timestamp that the caller can still access. The demuxer in turn returns an optional to indicate when a seek was not needed, which allows PlaybackManager to avoid clearing its queue and re-decoding frames.
2023-02-06 01:25:02 -06:00
} else if (m_target_timestamp >= manager().m_last_present_in_media_time && manager().m_next_frame.has_value() && manager().m_next_frame.value().timestamp() > m_target_timestamp) {
LibVideo/PlaybackManager: Organize playback states into virtual classes Storing playback states in virtual classes allows the behavior to be much more clearly written. Each `PlaybackStateHandler` subclass can implement some event-handling functions to model their behavior, and has functions to change its parent PlaybackManager's state to any other state. This will allow expanding the functionality of playback in the future, for example to allow skipping a single frame forward/backward.
2023-02-06 01:24:33 -06:00
manager().m_last_present_in_media_time = m_target_timestamp;
return exit_seek();
}
return skip_samples_until_timestamp();
}
ErrorOr<void> skip_samples_until_timestamp()
{
while (!manager().m_frame_queue->is_empty()) {
auto item = manager().m_frame_queue->dequeue();
manager().m_decode_timer->start(0);
if (item.is_error()) {
dbgln_if(PLAYBACK_MANAGER_DEBUG, "Encountered error while seeking: {}", item.error().description());
manager().on_decoder_error(item.release_error());
return {};
}
if (item.is_frame()) {
dbgln_if(PLAYBACK_MANAGER_DEBUG, "Dequeuing frame at {}ms and comparing to seek target {}ms", item.timestamp().to_milliseconds(), m_target_timestamp.to_milliseconds());
if (item.timestamp() > m_target_timestamp) {
// Fast seeking will result in an equal timestamp, so we can exit as soon as we see the next frame.
if (manager().m_next_frame.has_value()) {
manager().dispatch_new_frame(manager().m_next_frame.release_value().bitmap());
manager().m_last_present_in_media_time = m_target_timestamp;
}
manager().m_next_frame.emplace(item);
dbgln_if(PLAYBACK_MANAGER_DEBUG, "Exiting seek to {} state at {}ms", m_playing ? "Playing" : "Paused", manager().m_last_present_in_media_time.to_milliseconds());
return exit_seek();
}
manager().m_next_frame.emplace(item);
continue;
}
VERIFY_NOT_REACHED();
}
dbgln_if(PLAYBACK_MANAGER_DEBUG, "Frame queue is empty while seeking, waiting for buffer fill.");
manager().m_decode_timer->start(0);
return {};
}
StringView name() override { return "Seeking"sv; }
ErrorOr<void> play() override
{
m_playing = true;
return {};
}
bool is_playing() override { return m_playing; };
ErrorOr<void> pause() override
{
m_playing = false;
return {};
}
ErrorOr<void> seek(Time target_timestamp, SeekMode seek_mode) override
{
m_target_timestamp = target_timestamp;
m_seek_mode = seek_mode;
return on_enter();
}
Time current_time() const override
{
return m_target_timestamp;
}
// We won't need this override when threaded, the queue can pause us in on_enter().
ErrorOr<void> on_buffer_filled() override
{
dbgln_if(PLAYBACK_MANAGER_DEBUG, "Buffer filled while seeking, dequeuing until timestamp.");
return skip_samples_until_timestamp();
}
bool m_playing { false };
Time m_target_timestamp { Time::zero() };
SeekMode m_seek_mode { SeekMode::Accurate };
};
class PlaybackManager::StoppedStateHandler : public PlaybackManager::PlaybackStateHandler {
public:
StoppedStateHandler(PlaybackManager& manager)
: PlaybackStateHandler(manager)
{
}
~StoppedStateHandler() override = default;
private:
ErrorOr<void> on_enter() override
{
return {};
}
StringView name() override { return "Stopped"sv; }
ErrorOr<void> play() override
{
LibVideo: Pass the current sample to demuxers to lazily seek better In cases where the PlaybackManager's earliest buffered or displayed sample is closer to the seek target than the demuxer's chosen keyframe, we don't want to seek at all. To enable this, demuxers now receive an optional parameter with the earliest timestamp that the caller can still access. The demuxer in turn returns an optional to indicate when a seek was not needed, which allows PlaybackManager to avoid clearing its queue and re-decoding frames.
2023-02-06 01:25:02 -06:00
manager().m_next_frame.clear();
manager().m_frame_queue->clear();
auto start_timestamp = manager().seek_demuxer_to_most_recent_keyframe(Time::zero());
VERIFY(start_timestamp.has_value());
manager().m_last_present_in_media_time = start_timestamp.release_value();
LibVideo/PlaybackManager: Organize playback states into virtual classes Storing playback states in virtual classes allows the behavior to be much more clearly written. Each `PlaybackStateHandler` subclass can implement some event-handling functions to model their behavior, and has functions to change its parent PlaybackManager's state to any other state. This will allow expanding the functionality of playback in the future, for example to allow skipping a single frame forward/backward.
2023-02-06 01:24:33 -06:00
return replace_handler_and_delete_this<PlayingStateHandler>();
}
bool is_playing() override { return false; };
};
LibVideo: Add PlaybackManager to load and decode videos This file will be the basis for abstracting away the out-of-thread or later out-of-process decoding from applications displaying videos. For now, the demuxer is hardcoded to be MatroskaParser, since that is all we support so far. The demuxer should later be selected based on the file header. The playback and decoding are currently all done on one thread using timers. The design of the code is such that adding threading should be trivial, at least based on an earlier version of the code. For now, though, it's better that this runs in one thread, as the multithreaded approach causes the Video Player to lock up permanently after a few frames are decoded.
2022-10-29 18:16:14 -05:00
}
Reference in a new issue Copy permalink