There are a couple of remaining RFC 9111 methods in LibWeb's Fetch, but
these are currently directly tied to the way we store GC-allocated HTTP
response objects. So de-coupling that is left as a future exercise.
The end goal here is for LibHTTP to be the home of our RFC 9111 (HTTP
caching) implementation. We currently have one implementation in LibWeb
for our in-memory cache and another in RequestServer for our disk cache.
The implementations both largely revolve around interacting with HTTP
headers. But in LibWeb, we are using Fetch's header infra, and in RS we
are using are home-grown header infra from LibHTTP.
So to give these a common denominator, this patch replaces the LibHTTP
implementation with Fetch's infra. Our existing LibHTTP implementation
was not particularly compliant with any spec, so this at least gives us
a standards-based common implementation.
This migration also required moving a handful of other Fetch AOs over
to LibHTTP. (It turns out these AOs were all from the Fetch/Infra/HTTP
folder, so perhaps it makes sense for LibHTTP to be the implementation
of that entire set of facilities.)
An upcoming commit will migrate the contents of Headers.h/cpp to LibHTTP
for use outside of LibWeb. These CORS and MIME helpers depend on other
LibWeb facilities, however, so they cannot be moved.
Fixes a regression from commit:
f675cfe90f
It is not sufficient to only check if the builder is empty, as we will
then drop empty header values (when the first found value is empty).
This is tested in WPT by /cors/origin.htm, but that requires an HTTP
server.
The spec declares these as a byte sequence, which we then implemented as
a ByteBuffer. This has become pretty awkward to deal with, as evidenced
by the plethora of `MUST(ByteBuffer::copy(...))` and `.bytes()` calls
everywhere inside Fetch. We would then treat the bytes as a string
anyways by wrapping them in StringView everywhere.
We now store these as a ByteString. This is more comfortable to deal
with, and we no longer need to continually copy underlying storage (as
ByteString is ref-counted).
This work is largely preparatory for an upcoming HTTP header refactor.
Generally just define things in the order they are declared (will make a
change to use ByteString in this file a bit easier to follow). Also make
a couple of free functions be class methods on Header / HeaderList.
There might be a race between read_all_bytes and stream population.
If document load reads stream before it is populated, the stream will
be empty and might lead to hang in SessionHistoryTraversalQueue which
is expecting a promise to be resolved on document load.
This race can occur when stream population and document source are set
very close to each other. For example, when a newly generated blob is
set as the source of an iframe.
- navigation/multiple-navigable-cross-document-navigation.html has been
modified to trigger this race.
Disallow calling `StringBase::bytes()` on temporaries to avoid returning
`ReadonlyBytes` that outlive the underlying string.
With this change, we catch a real UAF:
`load_result.data = maybe_response.release_value().bytes();`
All other updated call sites were already safe, they just needed to use
an intermediate named variable to satisfy the new lvalue-only
requirement.
Previously, unbuffered requests were only available as a special mode
for EventSource. With this change, they are enabled by default, which
means chunks can be read from the stream as soon as they arrive.
This unlocks some interesting possibilities, such as starting to parse
HTML documents before the entire response has been received (that, in
turn, allows us to initiate subresource fetches earlier or begin
executing scripts sooner), or start rendering videos before they are
fully downloaded.
Co-authored-by: Timothy Flynn <trflynn89@pm.me>
Our HTTP disk cache is currently manually tested against various sites.
This patch adds some tests to cover various scenarios, including non-
cacheable responses, expired responses, and revalidation.
In order to ensure we hit the disk cache in RequestServer, we must
disable the in-memory cache in WebContent.
HTMLLinkElement is the final user of Resource/ResourceClient (used for
preloads and icons). This ports these link types to use fetch according
to the spec.
Preloads were particularly goofy because they would be stored in the
ResourceLoader's ad-hoc cache. But this cache was never consulted for
organic loads, thus were never used. There is more work to be done to
use these preloads within fetch, but for now they at least are stored
in fetch's HTTP cache for re-use.
Global Privacy Control aims to be a replacement for Do Not Track. DNT
ended up not being a great solution, as it wasn't enforced by law. This
actually resulted in the DNT header serving as an extra fingerprinting
data point.
GPC is becoming enforced by law in USA states such as California and
Colorado. CA is further working on a bill which requires that browsers
implement such an opt-out preference signal (OOPS):
https://cppa.ca.gov/announcements/2025/20250911.html
This patch replaces DNT with GPC and hooks up the associated settings.
According to RFC 2046, the BNF of the form data body is:
multipart-body := [preamble CRLF]
dash-boundary transport-padding CRLF
body-part *encapsulation
close-delimiter transport-padding
[CRLF epilogue]
Where "epilogue" is any text that "may be ignored or discarded". So we
should stop parsing the body once we encounter the terminating delimiter
("--").
Note that our parsing function is from an attempt to standardize the
grammar in the spec: https://andreubotella.github.io/multipart-form-data
This proposal hasn't been updated in ~4 years, and the fetch spec still
does not have a formal definition of the body string.
The Request constructor’s mode validation threw
"Mode must not be 'navigate"
missing the closing quote. Add the trailing quote so the error reads:
"Mode must not be 'navigate'".
Previously we depended on an associated document on the ESO to get to
the page, but Workers do not have documents. However, we can simply get
to the page with `principal_host_defined_page`, removing the issue.
This first pass only applies to the following two cases:
- Public functions returning a view type into an object they own
- Public ctors storing a view type
This catches a grand total of one (1) issue, which is fixed in
the previous commit.
This required some changes in LibURL & LibIPC since it has its own
definition of an BlobURLEntry. For now, we don't have a concrete usage
of MediaSource in LibURL so it is defined as an empty struct.
This removes one FIXME in an idl file.
This is an editoral change from the fetch spec. Since we already defined
the stream before it being used this only re-numbers the spec steps.
It also corrects a minor typo ('followings' to 'following') which was
corrected in the same editoral spec change.
Note that it's not actually executing tasks in parallel, it's still
throwing them on the HTML event loop task queue, each with its own
unique task source.
This makes our fetch implementation a lot more robust when HTTP caching
is enabled, and you can now click links on https://terminal.shop/
without hitting TODO assertions in fetch.
Which has an optmization if both size of the string being passed
through are FlyStrings, which actually ends up being the case
in some places during selector matching comparing attribute names.
Instead of maintaining more overloads of
Infra::is_ascii_case_insensitive_match, switch
everything over to equals_ignoring_ascii_case instead.
We currently store Web::Fetch::Infrastructure::Response objects in the
HTTP cache. They are associated with their original realm, but when we
use a cached response, we clone it into the target realm. For example,
two <iframe> objects loading the same HTML will be in different realms.
When we clone the response, we must use the target realm throughout the
entire cloning process. We neglected to do this for the cloned response
body stream, which is cloned via teeing. The result was the the stream
for the "cloned" response was created in the original realm, causing
issues down the line when reading from that stream tried to handle read
promises on behalf of the original realm. There are protections in place
to prevent this from happening, and the cached response read would never
complete.
