Stowage/ladybird
2
0
Fork 0
mirror of https://github.com/LadybirdBrowser/ladybird.git synced 2026-04-18 18:00:31 +00:00
Code Issues Projects Releases Packages Wiki Activity
ladybird/Libraries/LibWeb/Fetch/Fetching/FetchedDataReceiver.cpp

202 lines
7.7 KiB
C++
Raw Permalink Normal View History

LibWeb: Support unbuffered fetch requests Supporting unbuffered fetches is actually part of the fetch spec in its HTTP-network-fetch algorithm. We had previously implemented this method in a very ad-hoc manner as a simple wrapper around ResourceLoader. This is still the case, but we now implement a good amount of these steps according to spec, using ResourceLoader's unbuffered API. The response data is forwarded through to the fetch response using streams. This will eventually let us remove the use of ResourceLoader's buffered API, as all responses should just be streamed this way. The streams spec then supplies ways to wait for completion, thus allowing fully buffered responses. However, we have more work to do to make the other parts of our fetch implementation (namely, Body::fully_read) use streams before we can do this.
2024-05-26 08:03:29 -04:00
/*
LibHTTP+LibWeb: Support the HTTP Vary response header We now partition the HTTP disk cache based on the Vary response header. If a cached response contains a Vary header, we look for each of the header names in the outgoing HTTP request. The outgoing request must match every header value in the original request for the cache entry to be used; otherwise, a new request will be issued, and a separate cache entry will be created. Note that we must now defer creating the disk cache file itself until we have received the response headers. The Vary key is computed from these headers, and affects the partitioned disk cache file name. There are further optimizations we can make here. If we have a Vary mismatch, we could find the best candidate cached response and issue a conditional HTTP request. The content server may then respond with an HTTP 304 if the mismatched request headers are actually okay. But for now, if we have a Vary mismatch, we issue an unconditional request as a purely correctness-oriented patch.
2025-12-28 10:23:56 -05:00
* Copyright (c) 2024-2026, Tim Flynn <trflynn89@ladybird.org>
LibWeb: Use unbuffered network requests for all Fetch requests Previously, unbuffered requests were only available as a special mode for EventSource. With this change, they are enabled by default, which means chunks can be read from the stream as soon as they arrive. This unlocks some interesting possibilities, such as starting to parse HTML documents before the entire response has been received (that, in turn, allows us to initiate subresource fetches earlier or begin executing scripts sooner), or start rendering videos before they are fully downloaded. Co-authored-by: Timothy Flynn <trflynn89@pm.me>
2025-04-17 07:58:24 -04:00
* Copyright (c) 2025, Aliaksandr Kalenik <kalenik.aliaksandr@gmail.com>
LibWeb: Support unbuffered fetch requests Supporting unbuffered fetches is actually part of the fetch spec in its HTTP-network-fetch algorithm. We had previously implemented this method in a very ad-hoc manner as a simple wrapper around ResourceLoader. This is still the case, but we now implement a good amount of these steps according to spec, using ResourceLoader's unbuffered API. The response data is forwarded through to the fetch response using streams. This will eventually let us remove the use of ResourceLoader's buffered API, as all responses should just be streamed this way. The streams spec then supplies ways to wait for completion, thus allowing fully buffered responses. However, we have more work to do to make the other parts of our fetch implementation (namely, Body::fully_read) use streams before we can do this.
2024-05-26 08:03:29 -04:00
*
* SPDX-License-Identifier: BSD-2-Clause
*/
LibGC+Everywhere: Factor out a LibGC from LibJS Resulting in a massive rename across almost everywhere! Alongside the namespace change, we now have the following names: * JS::NonnullGCPtr -> GC::Ref * JS::GCPtr -> GC::Ptr * JS::HeapFunction -> GC::Function * JS::CellImpl -> GC::Cell * JS::Handle -> GC::Root
2024-11-15 04:01:23 +13:00
#include <LibGC/Function.h>
LibHTTP+LibWeb: Store the in-memory HTTP cache without JS realms The in-memory HTTP Fetch cache currently keeps the realm which created each cache entry alive indefinitely. This patch migrates this cache to LibHTTP, to ensure it is completely unaware of any JS objects. Now that we are not interacting with Fetch response objects, we can no longer use Streams infrastructure to pipe the response body into the Fetch response. Fetch also ultimately creates the cache response once the HTTP response headers have arrived. So the LibHTTP cache will hold entries in a pending list until we have received the entire response body. Then it is moved to a completed list and may be used thereafter.
2025-12-21 08:58:55 -05:00
#include <LibHTTP/Cache/MemoryCache.h>
LibWeb: Support unbuffered fetch requests Supporting unbuffered fetches is actually part of the fetch spec in its HTTP-network-fetch algorithm. We had previously implemented this method in a very ad-hoc manner as a simple wrapper around ResourceLoader. This is still the case, but we now implement a good amount of these steps according to spec, using ResourceLoader's unbuffered API. The response data is forwarded through to the fetch response using streams. This will eventually let us remove the use of ResourceLoader's buffered API, as all responses should just be streamed this way. The streams spec then supplies ways to wait for completion, thus allowing fully buffered responses. However, we have more work to do to make the other parts of our fetch implementation (namely, Body::fully_read) use streams before we can do this.
2024-05-26 08:03:29 -04:00
#include <LibWeb/Bindings/ExceptionOrUtils.h>
#include <LibWeb/Fetch/Fetching/FetchedDataReceiver.h>
#include <LibWeb/Fetch/Infrastructure/FetchParams.h>
LibWeb: Support MIME type sniffing for streaming HTTP responses Previously, when loading a document, we would try to sniff the MIME type by reading from the response body's source. However, for streaming HTTP responses, the body source is Empty (the data comes through the stream instead), so we had no bytes to sniff. This caused pages like hypr.land (which sends no Content-Type header) to be misidentified as plain text instead of HTML, since the MIME sniffing algorithm would receive zero bytes and fall back to the default type. The fix captures the first bytes of the response body during fetch, storing them on the Body object. These bytes are the "resource header" defined by the MIME Sniffing spec - up to 1445 bytes, which is enough to identify any MIME type the spec can detect. Since bytes may arrive asynchronously during streaming, we use a callback mechanism: if bytes aren't ready yet when load_document() needs them, it registers a callback that fires once enough bytes have been captured (or the stream ends). The flow is: 1. FetchedDataReceiver receives network bytes, buffers them 2. When Body is created, buffered bytes are flushed to Body's sniff buffer, and subsequent bytes are appended as they arrive 3. Before calling load_document(), Navigable waits for sniff bytes 4. load_document() passes the bytes to MimeSniff::Resource::sniff()
2026-01-24 12:25:01 +01:00
#include <LibWeb/Fetch/Infrastructure/HTTP/Bodies.h>
LibHTTP+LibWeb: Support the HTTP Vary response header We now partition the HTTP disk cache based on the Vary response header. If a cached response contains a Vary header, we look for each of the header names in the outgoing HTTP request. The outgoing request must match every header value in the original request for the cache entry to be used; otherwise, a new request will be issued, and a separate cache entry will be created. Note that we must now defer creating the disk cache file itself until we have received the response headers. The Vary key is computed from these headers, and affects the partitioned disk cache file name. There are further optimizations we can make here. If we have a Vary mismatch, we could find the best candidate cached response and issue a conditional HTTP request. The content server may then respond with an HTTP 304 if the mismatched request headers are actually okay. But for now, if we have a Vary mismatch, we issue an unconditional request as a purely correctness-oriented patch.
2025-12-28 10:23:56 -05:00
#include <LibWeb/Fetch/Infrastructure/HTTP/Responses.h>
LibWeb: Support unbuffered fetch requests Supporting unbuffered fetches is actually part of the fetch spec in its HTTP-network-fetch algorithm. We had previously implemented this method in a very ad-hoc manner as a simple wrapper around ResourceLoader. This is still the case, but we now implement a good amount of these steps according to spec, using ResourceLoader's unbuffered API. The response data is forwarded through to the fetch response using streams. This will eventually let us remove the use of ResourceLoader's buffered API, as all responses should just be streamed this way. The streams spec then supplies ways to wait for completion, thus allowing fully buffered responses. However, we have more work to do to make the other parts of our fetch implementation (namely, Body::fully_read) use streams before we can do this.
2024-05-26 08:03:29 -04:00
#include <LibWeb/Fetch/Infrastructure/Task.h>
#include <LibWeb/HTML/Scripting/ExceptionReporter.h>
#include <LibWeb/HTML/Scripting/TemporaryExecutionContext.h>
LibWeb: Move ReadableStream AOs into their own file The main streams AO file has gotten very large, and is a bit difficult to navigate. In an effort to improve DX, this migrates ReadableStream AOs to their own file. And the helper classes used for the tee and pipe- to operations are also in their own files.
2025-04-17 15:47:53 -04:00
#include <LibWeb/Streams/ReadableStream.h>
LibWeb: Support unbuffered fetch requests Supporting unbuffered fetches is actually part of the fetch spec in its HTTP-network-fetch algorithm. We had previously implemented this method in a very ad-hoc manner as a simple wrapper around ResourceLoader. This is still the case, but we now implement a good amount of these steps according to spec, using ResourceLoader's unbuffered API. The response data is forwarded through to the fetch response using streams. This will eventually let us remove the use of ResourceLoader's buffered API, as all responses should just be streamed this way. The streams spec then supplies ways to wait for completion, thus allowing fully buffered responses. However, we have more work to do to make the other parts of our fetch implementation (namely, Body::fully_read) use streams before we can do this.
2024-05-26 08:03:29 -04:00
#include <LibWeb/WebIDL/Promise.h>
namespace Web::Fetch::Fetching {
LibGC+Everywhere: Factor out a LibGC from LibJS Resulting in a massive rename across almost everywhere! Alongside the namespace change, we now have the following names: * JS::NonnullGCPtr -> GC::Ref * JS::GCPtr -> GC::Ptr * JS::HeapFunction -> GC::Function * JS::CellImpl -> GC::Cell * JS::Handle -> GC::Root
2024-11-15 04:01:23 +13:00
GC_DEFINE_ALLOCATOR(FetchedDataReceiver);
LibWeb: Support unbuffered fetch requests Supporting unbuffered fetches is actually part of the fetch spec in its HTTP-network-fetch algorithm. We had previously implemented this method in a very ad-hoc manner as a simple wrapper around ResourceLoader. This is still the case, but we now implement a good amount of these steps according to spec, using ResourceLoader's unbuffered API. The response data is forwarded through to the fetch response using streams. This will eventually let us remove the use of ResourceLoader's buffered API, as all responses should just be streamed this way. The streams spec then supplies ways to wait for completion, thus allowing fully buffered responses. However, we have more work to do to make the other parts of our fetch implementation (namely, Body::fully_read) use streams before we can do this.
2024-05-26 08:03:29 -04:00
LibHTTP+LibWeb: Store the in-memory HTTP cache without JS realms The in-memory HTTP Fetch cache currently keeps the realm which created each cache entry alive indefinitely. This patch migrates this cache to LibHTTP, to ensure it is completely unaware of any JS objects. Now that we are not interacting with Fetch response objects, we can no longer use Streams infrastructure to pipe the response body into the Fetch response. Fetch also ultimately creates the cache response once the HTTP response headers have arrived. So the LibHTTP cache will hold entries in a pending list until we have received the entire response body. Then it is moved to a completed list and may be used thereafter.
2025-12-21 08:58:55 -05:00
FetchedDataReceiver::FetchedDataReceiver(GC::Ref<Infrastructure::FetchParams const> fetch_params, GC::Ref<Streams::ReadableStream> stream, RefPtr<HTTP::MemoryCache> http_cache)
LibWeb: Support unbuffered fetch requests Supporting unbuffered fetches is actually part of the fetch spec in its HTTP-network-fetch algorithm. We had previously implemented this method in a very ad-hoc manner as a simple wrapper around ResourceLoader. This is still the case, but we now implement a good amount of these steps according to spec, using ResourceLoader's unbuffered API. The response data is forwarded through to the fetch response using streams. This will eventually let us remove the use of ResourceLoader's buffered API, as all responses should just be streamed this way. The streams spec then supplies ways to wait for completion, thus allowing fully buffered responses. However, we have more work to do to make the other parts of our fetch implementation (namely, Body::fully_read) use streams before we can do this.
2024-05-26 08:03:29 -04:00
: m_fetch_params(fetch_params)
, m_stream(stream)
LibHTTP+LibWeb: Store the in-memory HTTP cache without JS realms The in-memory HTTP Fetch cache currently keeps the realm which created each cache entry alive indefinitely. This patch migrates this cache to LibHTTP, to ensure it is completely unaware of any JS objects. Now that we are not interacting with Fetch response objects, we can no longer use Streams infrastructure to pipe the response body into the Fetch response. Fetch also ultimately creates the cache response once the HTTP response headers have arrived. So the LibHTTP cache will hold entries in a pending list until we have received the entire response body. Then it is moved to a completed list and may be used thereafter.
2025-12-21 08:58:55 -05:00
, m_http_cache(move(http_cache))
LibWeb: Support unbuffered fetch requests Supporting unbuffered fetches is actually part of the fetch spec in its HTTP-network-fetch algorithm. We had previously implemented this method in a very ad-hoc manner as a simple wrapper around ResourceLoader. This is still the case, but we now implement a good amount of these steps according to spec, using ResourceLoader's unbuffered API. The response data is forwarded through to the fetch response using streams. This will eventually let us remove the use of ResourceLoader's buffered API, as all responses should just be streamed this way. The streams spec then supplies ways to wait for completion, thus allowing fully buffered responses. However, we have more work to do to make the other parts of our fetch implementation (namely, Body::fully_read) use streams before we can do this.
2024-05-26 08:03:29 -04:00
{
}
FetchedDataReceiver::~FetchedDataReceiver() = default;
LibWeb: Support MIME type sniffing for streaming HTTP responses Previously, when loading a document, we would try to sniff the MIME type by reading from the response body's source. However, for streaming HTTP responses, the body source is Empty (the data comes through the stream instead), so we had no bytes to sniff. This caused pages like hypr.land (which sends no Content-Type header) to be misidentified as plain text instead of HTML, since the MIME sniffing algorithm would receive zero bytes and fall back to the default type. The fix captures the first bytes of the response body during fetch, storing them on the Body object. These bytes are the "resource header" defined by the MIME Sniffing spec - up to 1445 bytes, which is enough to identify any MIME type the spec can detect. Since bytes may arrive asynchronously during streaming, we use a callback mechanism: if bytes aren't ready yet when load_document() needs them, it registers a callback that fires once enough bytes have been captured (or the stream ends). The flow is: 1. FetchedDataReceiver receives network bytes, buffers them 2. When Body is created, buffered bytes are flushed to Body's sniff buffer, and subsequent bytes are appended as they arrive 3. Before calling load_document(), Navigable waits for sniff bytes 4. load_document() passes the bytes to MimeSniff::Resource::sniff()
2026-01-24 12:25:01 +01:00
void FetchedDataReceiver::set_body(GC::Ref<Fetch::Infrastructure::Body> body)
{
m_body = body;
// Flush any bytes that were buffered before the body was set
if (!m_buffer.is_empty())
m_body->append_sniff_bytes(m_buffer);
LibWeb: Fix race in MIME sniff bytes when response completes early When a streaming HTTP response completes before set_body() is called on the FetchedDataReceiver, the sniff bytes would never be marked as complete, causing navigation to hang intermittently. The sequence that triggers this: 1. handle_network_bytes(data, Ongoing) runs, but m_body is null, so bytes only go into the FetchedDataReceiver's own m_buffer. 2. handle_network_bytes({}, Complete) runs, but m_body is still null, so the m_body->set_sniff_bytes_complete() call is skipped. 3. set_body(body) is called, which flushes m_buffer into the body via append_sniff_bytes(), but never marks them as complete. 4. populate_session_history_entry_document() calls sniff_bytes_if_available() on the body. Since the source is Empty (streaming) and m_sniff_bytes_complete is false, it returns no value. 5. The async path registers a callback via wait_for_sniff_bytes(), but since the stream already completed, no more data arrives, and the callback never fires. Navigation hangs. Fix this by checking the lifecycle state in set_body(). If we have already moved past the Receiving state, the Complete was already processed and we need to mark sniff bytes as complete now.
2026-03-08 08:29:33 +01:00
// If the stream already completed before the body was set,
// we missed the set_sniff_bytes_complete() call in handle_network_bytes.
if (m_lifecycle_state != LifecycleState::Receiving)
m_body->set_sniff_bytes_complete();
LibWeb: Support MIME type sniffing for streaming HTTP responses Previously, when loading a document, we would try to sniff the MIME type by reading from the response body's source. However, for streaming HTTP responses, the body source is Empty (the data comes through the stream instead), so we had no bytes to sniff. This caused pages like hypr.land (which sends no Content-Type header) to be misidentified as plain text instead of HTML, since the MIME sniffing algorithm would receive zero bytes and fall back to the default type. The fix captures the first bytes of the response body during fetch, storing them on the Body object. These bytes are the "resource header" defined by the MIME Sniffing spec - up to 1445 bytes, which is enough to identify any MIME type the spec can detect. Since bytes may arrive asynchronously during streaming, we use a callback mechanism: if bytes aren't ready yet when load_document() needs them, it registers a callback that fires once enough bytes have been captured (or the stream ends). The flow is: 1. FetchedDataReceiver receives network bytes, buffers them 2. When Body is created, buffered bytes are flushed to Body's sniff buffer, and subsequent bytes are appended as they arrive 3. Before calling load_document(), Navigable waits for sniff bytes 4. load_document() passes the bytes to MimeSniff::Resource::sniff()
2026-01-24 12:25:01 +01:00
}
LibWeb: Support unbuffered fetch requests Supporting unbuffered fetches is actually part of the fetch spec in its HTTP-network-fetch algorithm. We had previously implemented this method in a very ad-hoc manner as a simple wrapper around ResourceLoader. This is still the case, but we now implement a good amount of these steps according to spec, using ResourceLoader's unbuffered API. The response data is forwarded through to the fetch response using streams. This will eventually let us remove the use of ResourceLoader's buffered API, as all responses should just be streamed this way. The streams spec then supplies ways to wait for completion, thus allowing fully buffered responses. However, we have more work to do to make the other parts of our fetch implementation (namely, Body::fully_read) use streams before we can do this.
2024-05-26 08:03:29 -04:00
void FetchedDataReceiver::visit_edges(Visitor& visitor)
{
Base::visit_edges(visitor);
visitor.visit(m_fetch_params);
LibHTTP+LibWeb: Support the HTTP Vary response header We now partition the HTTP disk cache based on the Vary response header. If a cached response contains a Vary header, we look for each of the header names in the outgoing HTTP request. The outgoing request must match every header value in the original request for the cache entry to be used; otherwise, a new request will be issued, and a separate cache entry will be created. Note that we must now defer creating the disk cache file itself until we have received the response headers. The Vary key is computed from these headers, and affects the partitioned disk cache file name. There are further optimizations we can make here. If we have a Vary mismatch, we could find the best candidate cached response and issue a conditional HTTP request. The content server may then respond with an HTTP 304 if the mismatched request headers are actually okay. But for now, if we have a Vary mismatch, we issue an unconditional request as a purely correctness-oriented patch.
2025-12-28 10:23:56 -05:00
visitor.visit(m_response);
LibWeb: Support MIME type sniffing for streaming HTTP responses Previously, when loading a document, we would try to sniff the MIME type by reading from the response body's source. However, for streaming HTTP responses, the body source is Empty (the data comes through the stream instead), so we had no bytes to sniff. This caused pages like hypr.land (which sends no Content-Type header) to be misidentified as plain text instead of HTML, since the MIME sniffing algorithm would receive zero bytes and fall back to the default type. The fix captures the first bytes of the response body during fetch, storing them on the Body object. These bytes are the "resource header" defined by the MIME Sniffing spec - up to 1445 bytes, which is enough to identify any MIME type the spec can detect. Since bytes may arrive asynchronously during streaming, we use a callback mechanism: if bytes aren't ready yet when load_document() needs them, it registers a callback that fires once enough bytes have been captured (or the stream ends). The flow is: 1. FetchedDataReceiver receives network bytes, buffers them 2. When Body is created, buffered bytes are flushed to Body's sniff buffer, and subsequent bytes are appended as they arrive 3. Before calling load_document(), Navigable waits for sniff bytes 4. load_document() passes the bytes to MimeSniff::Resource::sniff()
2026-01-24 12:25:01 +01:00
visitor.visit(m_body);
LibWeb: Support unbuffered fetch requests Supporting unbuffered fetches is actually part of the fetch spec in its HTTP-network-fetch algorithm. We had previously implemented this method in a very ad-hoc manner as a simple wrapper around ResourceLoader. This is still the case, but we now implement a good amount of these steps according to spec, using ResourceLoader's unbuffered API. The response data is forwarded through to the fetch response using streams. This will eventually let us remove the use of ResourceLoader's buffered API, as all responses should just be streamed this way. The streams spec then supplies ways to wait for completion, thus allowing fully buffered responses. However, we have more work to do to make the other parts of our fetch implementation (namely, Body::fully_read) use streams before we can do this.
2024-05-26 08:03:29 -04:00
visitor.visit(m_stream);
visitor.visit(m_pending_promise);
}
LibGC+Everywhere: Factor out a LibGC from LibJS Resulting in a massive rename across almost everywhere! Alongside the namespace change, we now have the following names: * JS::NonnullGCPtr -> GC::Ref * JS::GCPtr -> GC::Ptr * JS::HeapFunction -> GC::Function * JS::CellImpl -> GC::Cell * JS::Handle -> GC::Root
2024-11-15 04:01:23 +13:00
void FetchedDataReceiver::set_pending_promise(GC::Ref<WebIDL::Promise> promise)
LibWeb: Support unbuffered fetch requests Supporting unbuffered fetches is actually part of the fetch spec in its HTTP-network-fetch algorithm. We had previously implemented this method in a very ad-hoc manner as a simple wrapper around ResourceLoader. This is still the case, but we now implement a good amount of these steps according to spec, using ResourceLoader's unbuffered API. The response data is forwarded through to the fetch response using streams. This will eventually let us remove the use of ResourceLoader's buffered API, as all responses should just be streamed this way. The streams spec then supplies ways to wait for completion, thus allowing fully buffered responses. However, we have more work to do to make the other parts of our fetch implementation (namely, Body::fully_read) use streams before we can do this.
2024-05-26 08:03:29 -04:00
{
LibWeb: Use unbuffered network requests for all Fetch requests Previously, unbuffered requests were only available as a special mode for EventSource. With this change, they are enabled by default, which means chunks can be read from the stream as soon as they arrive. This unlocks some interesting possibilities, such as starting to parse HTML documents before the entire response has been received (that, in turn, allows us to initiate subresource fetches earlier or begin executing scripts sooner), or start rendering videos before they are fully downloaded. Co-authored-by: Timothy Flynn <trflynn89@pm.me>
2025-04-17 07:58:24 -04:00
VERIFY(!m_pending_promise);
VERIFY(!m_has_unfulfilled_promise);
LibWeb: Support unbuffered fetch requests Supporting unbuffered fetches is actually part of the fetch spec in its HTTP-network-fetch algorithm. We had previously implemented this method in a very ad-hoc manner as a simple wrapper around ResourceLoader. This is still the case, but we now implement a good amount of these steps according to spec, using ResourceLoader's unbuffered API. The response data is forwarded through to the fetch response using streams. This will eventually let us remove the use of ResourceLoader's buffered API, as all responses should just be streamed this way. The streams spec then supplies ways to wait for completion, thus allowing fully buffered responses. However, we have more work to do to make the other parts of our fetch implementation (namely, Body::fully_read) use streams before we can do this.
2024-05-26 08:03:29 -04:00
m_pending_promise = promise;
LibWeb: Accumulate all network bytes in FetchedDataReceiver This will allow us to hand off the bytes to the HTTP memory cache.
2025-12-20 08:33:49 -05:00
if (!buffer_is_eof()) {
pull_bytes_into_stream();
LibWeb: Use unbuffered network requests for all Fetch requests Previously, unbuffered requests were only available as a special mode for EventSource. With this change, they are enabled by default, which means chunks can be read from the stream as soon as they arrive. This unlocks some interesting possibilities, such as starting to parse HTML documents before the entire response has been received (that, in turn, allows us to initiate subresource fetches earlier or begin executing scripts sooner), or start rendering videos before they are fully downloaded. Co-authored-by: Timothy Flynn <trflynn89@pm.me>
2025-04-17 07:58:24 -04:00
} else if (m_lifecycle_state == LifecycleState::ReadyToClose) {
close_stream();
LibWeb: Support unbuffered fetch requests Supporting unbuffered fetches is actually part of the fetch spec in its HTTP-network-fetch algorithm. We had previously implemented this method in a very ad-hoc manner as a simple wrapper around ResourceLoader. This is still the case, but we now implement a good amount of these steps according to spec, using ResourceLoader's unbuffered API. The response data is forwarded through to the fetch response using streams. This will eventually let us remove the use of ResourceLoader's buffered API, as all responses should just be streamed this way. The streams spec then supplies ways to wait for completion, thus allowing fully buffered responses. However, we have more work to do to make the other parts of our fetch implementation (namely, Body::fully_read) use streams before we can do this.
2024-05-26 08:03:29 -04:00
}
}
// This implements the parallel steps of the pullAlgorithm in HTTP-network-fetch.
LibWeb: Use unbuffered network requests for all Fetch requests Previously, unbuffered requests were only available as a special mode for EventSource. With this change, they are enabled by default, which means chunks can be read from the stream as soon as they arrive. This unlocks some interesting possibilities, such as starting to parse HTML documents before the entire response has been received (that, in turn, allows us to initiate subresource fetches earlier or begin executing scripts sooner), or start rendering videos before they are fully downloaded. Co-authored-by: Timothy Flynn <trflynn89@pm.me>
2025-04-17 07:58:24 -04:00
// https://fetch.spec.whatwg.org/#ref-for-in-parallel⑤
void FetchedDataReceiver::handle_network_bytes(ReadonlyBytes bytes, NetworkState state)
LibWeb: Support unbuffered fetch requests Supporting unbuffered fetches is actually part of the fetch spec in its HTTP-network-fetch algorithm. We had previously implemented this method in a very ad-hoc manner as a simple wrapper around ResourceLoader. This is still the case, but we now implement a good amount of these steps according to spec, using ResourceLoader's unbuffered API. The response data is forwarded through to the fetch response using streams. This will eventually let us remove the use of ResourceLoader's buffered API, as all responses should just be streamed this way. The streams spec then supplies ways to wait for completion, thus allowing fully buffered responses. However, we have more work to do to make the other parts of our fetch implementation (namely, Body::fully_read) use streams before we can do this.
2024-05-26 08:03:29 -04:00
{
LibWeb: Use unbuffered network requests for all Fetch requests Previously, unbuffered requests were only available as a special mode for EventSource. With this change, they are enabled by default, which means chunks can be read from the stream as soon as they arrive. This unlocks some interesting possibilities, such as starting to parse HTML documents before the entire response has been received (that, in turn, allows us to initiate subresource fetches earlier or begin executing scripts sooner), or start rendering videos before they are fully downloaded. Co-authored-by: Timothy Flynn <trflynn89@pm.me>
2025-04-17 07:58:24 -04:00
VERIFY(m_lifecycle_state == LifecycleState::Receiving);
if (state == NetworkState::Complete) {
VERIFY(bytes.is_empty());
m_lifecycle_state = LifecycleState::CompletePending;
LibWeb: Support MIME type sniffing for streaming HTTP responses Previously, when loading a document, we would try to sniff the MIME type by reading from the response body's source. However, for streaming HTTP responses, the body source is Empty (the data comes through the stream instead), so we had no bytes to sniff. This caused pages like hypr.land (which sends no Content-Type header) to be misidentified as plain text instead of HTML, since the MIME sniffing algorithm would receive zero bytes and fall back to the default type. The fix captures the first bytes of the response body during fetch, storing them on the Body object. These bytes are the "resource header" defined by the MIME Sniffing spec - up to 1445 bytes, which is enough to identify any MIME type the spec can detect. Since bytes may arrive asynchronously during streaming, we use a callback mechanism: if bytes aren't ready yet when load_document() needs them, it registers a callback that fires once enough bytes have been captured (or the stream ends). The flow is: 1. FetchedDataReceiver receives network bytes, buffers them 2. When Body is created, buffered bytes are flushed to Body's sniff buffer, and subsequent bytes are appended as they arrive 3. Before calling load_document(), Navigable waits for sniff bytes 4. load_document() passes the bytes to MimeSniff::Resource::sniff()
2026-01-24 12:25:01 +01:00
// Mark sniff bytes as complete when the stream ends
if (m_body)
m_body->set_sniff_bytes_complete();
LibWeb: Use unbuffered network requests for all Fetch requests Previously, unbuffered requests were only available as a special mode for EventSource. With this change, they are enabled by default, which means chunks can be read from the stream as soon as they arrive. This unlocks some interesting possibilities, such as starting to parse HTML documents before the entire response has been received (that, in turn, allows us to initiate subresource fetches earlier or begin executing scripts sooner), or start rendering videos before they are fully downloaded. Co-authored-by: Timothy Flynn <trflynn89@pm.me>
2025-04-17 07:58:24 -04:00
}
LibWeb: Support unbuffered fetch requests Supporting unbuffered fetches is actually part of the fetch spec in its HTTP-network-fetch algorithm. We had previously implemented this method in a very ad-hoc manner as a simple wrapper around ResourceLoader. This is still the case, but we now implement a good amount of these steps according to spec, using ResourceLoader's unbuffered API. The response data is forwarded through to the fetch response using streams. This will eventually let us remove the use of ResourceLoader's buffered API, as all responses should just be streamed this way. The streams spec then supplies ways to wait for completion, thus allowing fully buffered responses. However, we have more work to do to make the other parts of our fetch implementation (namely, Body::fully_read) use streams before we can do this.
2024-05-26 08:03:29 -04:00
LibWeb: Support MIME type sniffing for streaming HTTP responses Previously, when loading a document, we would try to sniff the MIME type by reading from the response body's source. However, for streaming HTTP responses, the body source is Empty (the data comes through the stream instead), so we had no bytes to sniff. This caused pages like hypr.land (which sends no Content-Type header) to be misidentified as plain text instead of HTML, since the MIME sniffing algorithm would receive zero bytes and fall back to the default type. The fix captures the first bytes of the response body during fetch, storing them on the Body object. These bytes are the "resource header" defined by the MIME Sniffing spec - up to 1445 bytes, which is enough to identify any MIME type the spec can detect. Since bytes may arrive asynchronously during streaming, we use a callback mechanism: if bytes aren't ready yet when load_document() needs them, it registers a callback that fires once enough bytes have been captured (or the stream ends). The flow is: 1. FetchedDataReceiver receives network bytes, buffers them 2. When Body is created, buffered bytes are flushed to Body's sniff buffer, and subsequent bytes are appended as they arrive 3. Before calling load_document(), Navigable waits for sniff bytes 4. load_document() passes the bytes to MimeSniff::Resource::sniff()
2026-01-24 12:25:01 +01:00
if (state == NetworkState::Ongoing) {
LibWeb: Accumulate all network bytes in FetchedDataReceiver This will allow us to hand off the bytes to the HTTP memory cache.
2025-12-20 08:33:49 -05:00
m_buffer.append(bytes);
LibWeb: Support MIME type sniffing for streaming HTTP responses Previously, when loading a document, we would try to sniff the MIME type by reading from the response body's source. However, for streaming HTTP responses, the body source is Empty (the data comes through the stream instead), so we had no bytes to sniff. This caused pages like hypr.land (which sends no Content-Type header) to be misidentified as plain text instead of HTML, since the MIME sniffing algorithm would receive zero bytes and fall back to the default type. The fix captures the first bytes of the response body during fetch, storing them on the Body object. These bytes are the "resource header" defined by the MIME Sniffing spec - up to 1445 bytes, which is enough to identify any MIME type the spec can detect. Since bytes may arrive asynchronously during streaming, we use a callback mechanism: if bytes aren't ready yet when load_document() needs them, it registers a callback that fires once enough bytes have been captured (or the stream ends). The flow is: 1. FetchedDataReceiver receives network bytes, buffers them 2. When Body is created, buffered bytes are flushed to Body's sniff buffer, and subsequent bytes are appended as they arrive 3. Before calling load_document(), Navigable waits for sniff bytes 4. load_document() passes the bytes to MimeSniff::Resource::sniff()
2026-01-24 12:25:01 +01:00
// Capture bytes for MIME sniffing
if (m_body)
m_body->append_sniff_bytes(bytes);
}
LibWeb: Accumulate all network bytes in FetchedDataReceiver This will allow us to hand off the bytes to the HTTP memory cache.
2025-12-20 08:33:49 -05:00
LibWeb: Support unbuffered fetch requests Supporting unbuffered fetches is actually part of the fetch spec in its HTTP-network-fetch algorithm. We had previously implemented this method in a very ad-hoc manner as a simple wrapper around ResourceLoader. This is still the case, but we now implement a good amount of these steps according to spec, using ResourceLoader's unbuffered API. The response data is forwarded through to the fetch response using streams. This will eventually let us remove the use of ResourceLoader's buffered API, as all responses should just be streamed this way. The streams spec then supplies ways to wait for completion, thus allowing fully buffered responses. However, we have more work to do to make the other parts of our fetch implementation (namely, Body::fully_read) use streams before we can do this.
2024-05-26 08:03:29 -04:00
if (!m_pending_promise) {
LibWeb: Accumulate all network bytes in FetchedDataReceiver This will allow us to hand off the bytes to the HTTP memory cache.
2025-12-20 08:33:49 -05:00
if (m_lifecycle_state == LifecycleState::CompletePending && buffer_is_eof() && !m_has_unfulfilled_promise)
LibWeb: Use unbuffered network requests for all Fetch requests Previously, unbuffered requests were only available as a special mode for EventSource. With this change, they are enabled by default, which means chunks can be read from the stream as soon as they arrive. This unlocks some interesting possibilities, such as starting to parse HTML documents before the entire response has been received (that, in turn, allows us to initiate subresource fetches earlier or begin executing scripts sooner), or start rendering videos before they are fully downloaded. Co-authored-by: Timothy Flynn <trflynn89@pm.me>
2025-04-17 07:58:24 -04:00
m_lifecycle_state = LifecycleState::ReadyToClose;
LibWeb: Support unbuffered fetch requests Supporting unbuffered fetches is actually part of the fetch spec in its HTTP-network-fetch algorithm. We had previously implemented this method in a very ad-hoc manner as a simple wrapper around ResourceLoader. This is still the case, but we now implement a good amount of these steps according to spec, using ResourceLoader's unbuffered API. The response data is forwarded through to the fetch response using streams. This will eventually let us remove the use of ResourceLoader's buffered API, as all responses should just be streamed this way. The streams spec then supplies ways to wait for completion, thus allowing fully buffered responses. However, we have more work to do to make the other parts of our fetch implementation (namely, Body::fully_read) use streams before we can do this.
2024-05-26 08:03:29 -04:00
return;
}
LibWeb: Use unbuffered network requests for all Fetch requests Previously, unbuffered requests were only available as a special mode for EventSource. With this change, they are enabled by default, which means chunks can be read from the stream as soon as they arrive. This unlocks some interesting possibilities, such as starting to parse HTML documents before the entire response has been received (that, in turn, allows us to initiate subresource fetches earlier or begin executing scripts sooner), or start rendering videos before they are fully downloaded. Co-authored-by: Timothy Flynn <trflynn89@pm.me>
2025-04-17 07:58:24 -04:00
// 1. If one or more bytes have been transmitted from response’s message body, then:
if (!bytes.is_empty()) {
// 1. Let bytes be the transmitted bytes.
// FIXME: 2. Let codings be the result of extracting header list values given `Content-Encoding` and response’s header list.
// FIXME: 3. Increase response’s body info’s encoded size by bytes’s length.
// FIXME: 4. Set bytes to the result of handling content codings given codings and bytes.
// FIXME: 5. Increase response’s body info’s decoded size by bytes’s length.
// FIXME: 6. If bytes is failure, then terminate fetchParams’s controller.
// 7. Append bytes to buffer.
LibWeb: Accumulate all network bytes in FetchedDataReceiver This will allow us to hand off the bytes to the HTTP memory cache.
2025-12-20 08:33:49 -05:00
pull_bytes_into_stream();
LibWeb: Use unbuffered network requests for all Fetch requests Previously, unbuffered requests were only available as a special mode for EventSource. With this change, they are enabled by default, which means chunks can be read from the stream as soon as they arrive. This unlocks some interesting possibilities, such as starting to parse HTML documents before the entire response has been received (that, in turn, allows us to initiate subresource fetches earlier or begin executing scripts sooner), or start rendering videos before they are fully downloaded. Co-authored-by: Timothy Flynn <trflynn89@pm.me>
2025-04-17 07:58:24 -04:00
// FIXME: 8. If the size of buffer is larger than an upper limit chosen by the user agent, ask the user agent
// to suspend the ongoing fetch.
return;
}
// 2. Otherwise, if the bytes transmission for response’s message body is done normally and stream is readable,
// then close stream, and abort these in-parallel steps.
if (m_stream->is_readable()) {
VERIFY(m_lifecycle_state == LifecycleState::CompletePending);
close_stream();
}
}
// This implements the parallel steps of the pullAlgorithm in HTTP-network-fetch.
// https://fetch.spec.whatwg.org/#ref-for-in-parallelâ‘£
LibWeb: Accumulate all network bytes in FetchedDataReceiver This will allow us to hand off the bytes to the HTTP memory cache.
2025-12-20 08:33:49 -05:00
void FetchedDataReceiver::pull_bytes_into_stream()
LibWeb: Use unbuffered network requests for all Fetch requests Previously, unbuffered requests were only available as a special mode for EventSource. With this change, they are enabled by default, which means chunks can be read from the stream as soon as they arrive. This unlocks some interesting possibilities, such as starting to parse HTML documents before the entire response has been received (that, in turn, allows us to initiate subresource fetches earlier or begin executing scripts sooner), or start rendering videos before they are fully downloaded. Co-authored-by: Timothy Flynn <trflynn89@pm.me>
2025-04-17 07:58:24 -04:00
{
LibWeb: Accumulate all network bytes in FetchedDataReceiver This will allow us to hand off the bytes to the HTTP memory cache.
2025-12-20 08:33:49 -05:00
VERIFY(m_lifecycle_state == LifecycleState::Receiving || m_lifecycle_state == LifecycleState::CompletePending);
LibWeb: Use unbuffered network requests for all Fetch requests Previously, unbuffered requests were only available as a special mode for EventSource. With this change, they are enabled by default, which means chunks can be read from the stream as soon as they arrive. This unlocks some interesting possibilities, such as starting to parse HTML documents before the entire response has been received (that, in turn, allows us to initiate subresource fetches earlier or begin executing scripts sooner), or start rendering videos before they are fully downloaded. Co-authored-by: Timothy Flynn <trflynn89@pm.me>
2025-04-17 07:58:24 -04:00
// FIXME: 1. If the size of buffer is smaller than a lower limit chosen by the user agent and the ongoing fetch
// is suspended, resume the fetch.
// 2. Wait until buffer is not empty.
LibWeb: Accumulate all network bytes in FetchedDataReceiver This will allow us to hand off the bytes to the HTTP memory cache.
2025-12-20 08:33:49 -05:00
// NB: It would be nice to avoid a copy here, but ReadableStream::pull_from_bytes currently requires an allocated
// ByteBuffer to create a JS::ArrayBuffer.
auto bytes = copy_unpulled_bytes();
LibWeb: Use unbuffered network requests for all Fetch requests Previously, unbuffered requests were only available as a special mode for EventSource. With this change, they are enabled by default, which means chunks can be read from the stream as soon as they arrive. This unlocks some interesting possibilities, such as starting to parse HTML documents before the entire response has been received (that, in turn, allows us to initiate subresource fetches earlier or begin executing scripts sooner), or start rendering videos before they are fully downloaded. Co-authored-by: Timothy Flynn <trflynn89@pm.me>
2025-04-17 07:58:24 -04:00
VERIFY(!bytes.is_empty());
LibWeb: Support unbuffered fetch requests Supporting unbuffered fetches is actually part of the fetch spec in its HTTP-network-fetch algorithm. We had previously implemented this method in a very ad-hoc manner as a simple wrapper around ResourceLoader. This is still the case, but we now implement a good amount of these steps according to spec, using ResourceLoader's unbuffered API. The response data is forwarded through to the fetch response using streams. This will eventually let us remove the use of ResourceLoader's buffered API, as all responses should just be streamed this way. The streams spec then supplies ways to wait for completion, thus allowing fully buffered responses. However, we have more work to do to make the other parts of our fetch implementation (namely, Body::fully_read) use streams before we can do this.
2024-05-26 08:03:29 -04:00
// 3. Queue a fetch task to run the following steps, with fetchParams’s task destination.
LibWeb: Use unbuffered network requests for all Fetch requests Previously, unbuffered requests were only available as a special mode for EventSource. With this change, they are enabled by default, which means chunks can be read from the stream as soon as they arrive. This unlocks some interesting possibilities, such as starting to parse HTML documents before the entire response has been received (that, in turn, allows us to initiate subresource fetches earlier or begin executing scripts sooner), or start rendering videos before they are fully downloaded. Co-authored-by: Timothy Flynn <trflynn89@pm.me>
2025-04-17 07:58:24 -04:00
VERIFY(!m_has_unfulfilled_promise);
m_has_unfulfilled_promise = true;
LibWeb: Accumulate all network bytes in FetchedDataReceiver This will allow us to hand off the bytes to the HTTP memory cache.
2025-12-20 08:33:49 -05:00
LibWeb: Support unbuffered fetch requests Supporting unbuffered fetches is actually part of the fetch spec in its HTTP-network-fetch algorithm. We had previously implemented this method in a very ad-hoc manner as a simple wrapper around ResourceLoader. This is still the case, but we now implement a good amount of these steps according to spec, using ResourceLoader's unbuffered API. The response data is forwarded through to the fetch response using streams. This will eventually let us remove the use of ResourceLoader's buffered API, as all responses should just be streamed this way. The streams spec then supplies ways to wait for completion, thus allowing fully buffered responses. However, we have more work to do to make the other parts of our fetch implementation (namely, Body::fully_read) use streams before we can do this.
2024-05-26 08:03:29 -04:00
Infrastructure::queue_fetch_task(
m_fetch_params->controller(),
LibWeb: Add "parallel queue" and allow it as fetch task destination Note that it's not actually executing tasks in parallel, it's still throwing them on the HTML event loop task queue, each with its own unique task source. This makes our fetch implementation a lot more robust when HTTP caching is enabled, and you can now click links on https://terminal.shop/ without hitting TODO assertions in fetch.
2025-07-16 12:29:26 +02:00
m_fetch_params->task_destination(),
LibWeb: Use unbuffered network requests for all Fetch requests Previously, unbuffered requests were only available as a special mode for EventSource. With this change, they are enabled by default, which means chunks can be read from the stream as soon as they arrive. This unlocks some interesting possibilities, such as starting to parse HTML documents before the entire response has been received (that, in turn, allows us to initiate subresource fetches earlier or begin executing scripts sooner), or start rendering videos before they are fully downloaded. Co-authored-by: Timothy Flynn <trflynn89@pm.me>
2025-04-17 07:58:24 -04:00
GC::create_function(heap(), [this, bytes = move(bytes), pending_promise = m_pending_promise]() mutable {
m_has_unfulfilled_promise = false;
VERIFY(m_lifecycle_state == LifecycleState::Receiving || m_lifecycle_state == LifecycleState::CompletePending);
LibWeb: Make TemporaryExecutionContext take a Realm&
2024-10-24 20:39:18 +13:00
HTML::TemporaryExecutionContext execution_context { m_stream->realm(), HTML::TemporaryExecutionContext::CallbacksEnabled::Yes };
LibWeb: Support unbuffered fetch requests Supporting unbuffered fetches is actually part of the fetch spec in its HTTP-network-fetch algorithm. We had previously implemented this method in a very ad-hoc manner as a simple wrapper around ResourceLoader. This is still the case, but we now implement a good amount of these steps according to spec, using ResourceLoader's unbuffered API. The response data is forwarded through to the fetch response using streams. This will eventually let us remove the use of ResourceLoader's buffered API, as all responses should just be streamed this way. The streams spec then supplies ways to wait for completion, thus allowing fully buffered responses. However, we have more work to do to make the other parts of our fetch implementation (namely, Body::fully_read) use streams before we can do this.
2024-05-26 08:03:29 -04:00
// 1. Pull from bytes buffer into stream.
LibWeb/Streams: Move ReadableStream functions out of AbstractOperations These are not defined in the abstract operations section of the spec and are the publically exported Stream APIs exposed on ReadableStream.
2024-12-08 17:28:44 +13:00
if (auto result = m_stream->pull_from_bytes(move(bytes)); result.is_error()) {
LibWeb: Remove `dom_` from `dom_exception_to_throw_completion` We're not converting `WebIDL::DOMException`, but `WebIDL::Exception` instead.
2024-11-04 14:37:27 +01:00
auto throw_completion = Bindings::exception_to_throw_completion(m_stream->vm(), result.release_error());
LibWeb: Support unbuffered fetch requests Supporting unbuffered fetches is actually part of the fetch spec in its HTTP-network-fetch algorithm. We had previously implemented this method in a very ad-hoc manner as a simple wrapper around ResourceLoader. This is still the case, but we now implement a good amount of these steps according to spec, using ResourceLoader's unbuffered API. The response data is forwarded through to the fetch response using streams. This will eventually let us remove the use of ResourceLoader's buffered API, as all responses should just be streamed this way. The streams spec then supplies ways to wait for completion, thus allowing fully buffered responses. However, we have more work to do to make the other parts of our fetch implementation (namely, Body::fully_read) use streams before we can do this.
2024-05-26 08:03:29 -04:00
dbgln("FetchedDataReceiver: Stream error pulling bytes");
HTML::report_exception(throw_completion, m_stream->realm());
return;
}
// 2. If stream is errored, then terminate fetchParams’s controller.
if (m_stream->is_errored())
m_fetch_params->controller()->terminate();
// 3. Resolve promise with undefined.
LibWeb: Use unbuffered network requests for all Fetch requests Previously, unbuffered requests were only available as a special mode for EventSource. With this change, they are enabled by default, which means chunks can be read from the stream as soon as they arrive. This unlocks some interesting possibilities, such as starting to parse HTML documents before the entire response has been received (that, in turn, allows us to initiate subresource fetches earlier or begin executing scripts sooner), or start rendering videos before they are fully downloaded. Co-authored-by: Timothy Flynn <trflynn89@pm.me>
2025-04-17 07:58:24 -04:00
WebIDL::resolve_promise(m_stream->realm(), *pending_promise, JS::js_undefined());
LibWeb: Accumulate all network bytes in FetchedDataReceiver This will allow us to hand off the bytes to the HTTP memory cache.
2025-12-20 08:33:49 -05:00
if (m_lifecycle_state == LifecycleState::CompletePending && buffer_is_eof())
LibWeb: Use unbuffered network requests for all Fetch requests Previously, unbuffered requests were only available as a special mode for EventSource. With this change, they are enabled by default, which means chunks can be read from the stream as soon as they arrive. This unlocks some interesting possibilities, such as starting to parse HTML documents before the entire response has been received (that, in turn, allows us to initiate subresource fetches earlier or begin executing scripts sooner), or start rendering videos before they are fully downloaded. Co-authored-by: Timothy Flynn <trflynn89@pm.me>
2025-04-17 07:58:24 -04:00
m_lifecycle_state = LifecycleState::ReadyToClose;
LibWeb: Support unbuffered fetch requests Supporting unbuffered fetches is actually part of the fetch spec in its HTTP-network-fetch algorithm. We had previously implemented this method in a very ad-hoc manner as a simple wrapper around ResourceLoader. This is still the case, but we now implement a good amount of these steps according to spec, using ResourceLoader's unbuffered API. The response data is forwarded through to the fetch response using streams. This will eventually let us remove the use of ResourceLoader's buffered API, as all responses should just be streamed this way. The streams spec then supplies ways to wait for completion, thus allowing fully buffered responses. However, we have more work to do to make the other parts of our fetch implementation (namely, Body::fully_read) use streams before we can do this.
2024-05-26 08:03:29 -04:00
}));
LibWeb: Use unbuffered network requests for all Fetch requests Previously, unbuffered requests were only available as a special mode for EventSource. With this change, they are enabled by default, which means chunks can be read from the stream as soon as they arrive. This unlocks some interesting possibilities, such as starting to parse HTML documents before the entire response has been received (that, in turn, allows us to initiate subresource fetches earlier or begin executing scripts sooner), or start rendering videos before they are fully downloaded. Co-authored-by: Timothy Flynn <trflynn89@pm.me>
2025-04-17 07:58:24 -04:00
m_pending_promise = {};
}
void FetchedDataReceiver::close_stream()
{
VERIFY(m_has_unfulfilled_promise == 0);
LibWeb: Accumulate all network bytes in FetchedDataReceiver This will allow us to hand off the bytes to the HTTP memory cache.
2025-12-20 08:33:49 -05:00
VERIFY(buffer_is_eof());
LibWeb: Use unbuffered network requests for all Fetch requests Previously, unbuffered requests were only available as a special mode for EventSource. With this change, they are enabled by default, which means chunks can be read from the stream as soon as they arrive. This unlocks some interesting possibilities, such as starting to parse HTML documents before the entire response has been received (that, in turn, allows us to initiate subresource fetches earlier or begin executing scripts sooner), or start rendering videos before they are fully downloaded. Co-authored-by: Timothy Flynn <trflynn89@pm.me>
2025-04-17 07:58:24 -04:00
WebIDL::resolve_promise(m_stream->realm(), *m_pending_promise, JS::js_undefined());
m_pending_promise = {};
m_lifecycle_state = LifecycleState::Closed;
m_stream->close();
LibHTTP+LibWeb: Store the in-memory HTTP cache without JS realms The in-memory HTTP Fetch cache currently keeps the realm which created each cache entry alive indefinitely. This patch migrates this cache to LibHTTP, to ensure it is completely unaware of any JS objects. Now that we are not interacting with Fetch response objects, we can no longer use Streams infrastructure to pipe the response body into the Fetch response. Fetch also ultimately creates the cache response once the HTTP response headers have arrived. So the LibHTTP cache will hold entries in a pending list until we have received the entire response body. Then it is moved to a completed list and may be used thereafter.
2025-12-21 08:58:55 -05:00
if (m_http_cache) {
LibHTTP+LibWeb: Support the HTTP Vary response header We now partition the HTTP disk cache based on the Vary response header. If a cached response contains a Vary header, we look for each of the header names in the outgoing HTTP request. The outgoing request must match every header value in the original request for the cache entry to be used; otherwise, a new request will be issued, and a separate cache entry will be created. Note that we must now defer creating the disk cache file itself until we have received the response headers. The Vary key is computed from these headers, and affects the partitioned disk cache file name. There are further optimizations we can make here. If we have a Vary mismatch, we could find the best candidate cached response and issue a conditional HTTP request. The content server may then respond with an HTTP 304 if the mismatched request headers are actually okay. But for now, if we have a Vary mismatch, we issue an unconditional request as a purely correctness-oriented patch.
2025-12-28 10:23:56 -05:00
auto request = m_fetch_params->request();
if (m_response && request->cache_mode() != HTTP::CacheMode::NoStore)
m_http_cache->finalize_entry(request->current_url(), request->method(), request->header_list(), m_response->status(), m_response->header_list(), move(m_buffer));
LibHTTP+LibWeb: Store the in-memory HTTP cache without JS realms The in-memory HTTP Fetch cache currently keeps the realm which created each cache entry alive indefinitely. This patch migrates this cache to LibHTTP, to ensure it is completely unaware of any JS objects. Now that we are not interacting with Fetch response objects, we can no longer use Streams infrastructure to pipe the response body into the Fetch response. Fetch also ultimately creates the cache response once the HTTP response headers have arrived. So the LibHTTP cache will hold entries in a pending list until we have received the entire response body. Then it is moved to a completed list and may be used thereafter.
2025-12-21 08:58:55 -05:00
m_http_cache.clear();
}
LibWeb: Support unbuffered fetch requests Supporting unbuffered fetches is actually part of the fetch spec in its HTTP-network-fetch algorithm. We had previously implemented this method in a very ad-hoc manner as a simple wrapper around ResourceLoader. This is still the case, but we now implement a good amount of these steps according to spec, using ResourceLoader's unbuffered API. The response data is forwarded through to the fetch response using streams. This will eventually let us remove the use of ResourceLoader's buffered API, as all responses should just be streamed this way. The streams spec then supplies ways to wait for completion, thus allowing fully buffered responses. However, we have more work to do to make the other parts of our fetch implementation (namely, Body::fully_read) use streams before we can do this.
2024-05-26 08:03:29 -04:00
}
LibWeb: Accumulate all network bytes in FetchedDataReceiver This will allow us to hand off the bytes to the HTTP memory cache.
2025-12-20 08:33:49 -05:00
ByteBuffer FetchedDataReceiver::copy_unpulled_bytes()
{
auto bytes = MUST(m_buffer.slice(m_pulled_bytes, m_buffer.size() - m_pulled_bytes));
m_pulled_bytes += bytes.size();
return bytes;
}
LibWeb: Support unbuffered fetch requests Supporting unbuffered fetches is actually part of the fetch spec in its HTTP-network-fetch algorithm. We had previously implemented this method in a very ad-hoc manner as a simple wrapper around ResourceLoader. This is still the case, but we now implement a good amount of these steps according to spec, using ResourceLoader's unbuffered API. The response data is forwarded through to the fetch response using streams. This will eventually let us remove the use of ResourceLoader's buffered API, as all responses should just be streamed this way. The streams spec then supplies ways to wait for completion, thus allowing fully buffered responses. However, we have more work to do to make the other parts of our fetch implementation (namely, Body::fully_read) use streams before we can do this.
2024-05-26 08:03:29 -04:00
}
Reference in a new issue Copy permalink