The end goal here is for LibHTTP to be the home of our RFC 9111 (HTTP
caching) implementation. We currently have one implementation in LibWeb
for our in-memory cache and another in RequestServer for our disk cache.
The implementations both largely revolve around interacting with HTTP
headers. But in LibWeb, we are using Fetch's header infra, and in RS we
are using are home-grown header infra from LibHTTP.
So to give these a common denominator, this patch replaces the LibHTTP
implementation with Fetch's infra. Our existing LibHTTP implementation
was not particularly compliant with any spec, so this at least gives us
a standards-based common implementation.
This migration also required moving a handful of other Fetch AOs over
to LibHTTP. (It turns out these AOs were all from the Fetch/Infra/HTTP
folder, so perhaps it makes sense for LibHTTP to be the implementation
of that entire set of facilities.)
The only thing in HTTPResponse being used is reason_phrase_for_code,
which is just a static helper method. Move it to its own file and remove
HTTPResponse.
This is just one less thing to have to port to an upcoming HTTP header
refactor.
Disallow calling `StringBase::bytes()` on temporaries to avoid returning
`ReadonlyBytes` that outlive the underlying string.
With this change, we catch a real UAF:
`load_result.data = maybe_response.release_value().bytes();`
All other updated call sites were already safe, they just needed to use
an intermediate named variable to satisfy the new lvalue-only
requirement.
Previously, unbuffered requests were only available as a special mode
for EventSource. With this change, they are enabled by default, which
means chunks can be read from the stream as soon as they arrive.
This unlocks some interesting possibilities, such as starting to parse
HTML documents before the entire response has been received (that, in
turn, allows us to initiate subresource fetches earlier or begin
executing scripts sooner), or start rendering videos before they are
fully downloaded.
Co-authored-by: Timothy Flynn <trflynn89@pm.me>
JavaScript module requests (in a non-worker context) always have CORS
enabled. However, CORS requests are only allowed for same-origin or
HTTP/S requests. This patch extends this to allow resource:// requests
from opaque origins (e.g. about: URLs).
We must also set the Access-Control-Allow-Origin header to "null" to
ensure that the response is accepted by the CORS checks. This does not
affect requesting resource:// URLs from resource:// URLs as those are
same-origin and skip CORS checks.
This ultimately enables requesting resource:// JS modules from the
about:settings page.
This commit:
- Prevents path traversal via the about: scheme
- Prevents loading about:inspector
- Requires about: URIs to be opaque paths
- Prevents crashes with invalid percent encoded paths
We only need a Page for file:// urls. At some point we probably
needed it for other kinds of requests, but the current functionality
doesn't need to store the Page pointer on the ResourceLoader.
Resulting in a massive rename across almost everywhere! Alongside the
namespace change, we now have the following names:
* JS::NonnullGCPtr -> GC::Ref
* JS::GCPtr -> GC::Ptr
* JS::HeapFunction -> GC::Function
* JS::CellImpl -> GC::Cell
* JS::Handle -> GC::Root
This patch adds a global (per-process) filter list to LibWeb that is
used to filter all outgoing resource load requests.
Basically we check the URL against a list of filter patterns and if
it's a match for any one of them, we immediately fail the load.
The filter list is a simple text file:
~/.config/BrowserContentFilters.txt
It's one filter per line and they are simple glob filters for now,
with implicit asterisks (*) at the start and end of the line.
This patchset makes ProtocolServer stream the downloads to its client
(LibProtocol), and as such changes the download API; a possible
download lifecycle could be as such:
notation = client->server:'>', server->client:'<', pipe activity:'*'
```
> StartDownload(GET, url, headers, {})
< Response(0, fd 8)
* {data, 1024b}
< HeadersBecameAvailable(0, response_headers, 200)
< DownloadProgress(0, 4K, 1024)
* {data, 1024b}
* {data, 1024b}
< DownloadProgress(0, 4K, 2048)
* {data, 1024b}
< DownloadProgress(0, 4K, 1024)
< DownloadFinished(0, true, 4K)
```
Since managing the received file descriptor is a pain, LibProtocol
implements `Download::stream_into(OutputStream)`, which can be used to
stream the download into any given output stream (be it a file, or
memory, or writing stuff with a delay, etc.).
Also, as some of the users of this API require all the downloaded data
upfront, LibProtocol also implements `set_should_buffer_all_input()`,
which causes the download instance to buffer all the data until the
download is complete, and to call the `on_buffered_download_finish`
hook.
Instead of everyone overriding save_to() and set_property() and doing
a pretty asymmetric job of implementing the various properties, let's
add a bit of structure here.
Object properties are now represented by a Core::Property. Properties
are registered with a getter and setter (optional) in constructors.
I've added some convenience macros for creating and registering
properties, but this does still feel a bit bulky. We'll have to
iterate on this and see where it goes.
This function did a const_cast internally which made the call side look
"safe". This method is removed completely and call sites are replaced
with ByteBuffer::wrap(const_cast<void*>(data), size) which makes the
behaviour obvious.
Clients now receive HTTP status codes like 200, 404, etc.
Note that a 404 with content is still considered a "successful"
download from ProtocolServer's perspective. It's up to the client
to interpret the status code.
I'm not sure if this is the best API, but it'll work for now.
This patch adds ImageResource as a subclass of Resource. This new class
also keeps a Gfx::ImageDecoder so that we can share decoded bitmaps
between all clients of an image resource inside LibWeb.
With this, we now share both encoded and decoded data for images. :^)
I had to change how the purgeable-volatile flag is updated to keep the
volatile-images-outside-the-visible-viewport optimization working.
HTMLImageElement now inherits from ImageResourceClient (a subclass of
ResourceClient with additional image-specific stuff) and informs its
ImageResource about whether it's inside the viewport or outside.
This is pretty awesome! :^)
This patch introduces a caching mechanism in ResourceLoader. It's keyed
on a LoadRequest object which is what you provide to load_resource()
when you want to load a resource.
We currently never prune the cache, so resources will stay in there
forever. This is obviously not gonna stay that way, but we're just
getting started here. :^)
This should drastically reduce the number of requests when loading
some sites (like Twitter) that reuse the same images over and over.
A Resource represents a resource that we're loading, have loaded or
will soon load. Basically, it's a downloadable resource that can be
shared by multiple clients.
A typical usecase is multiple <img> elements with the same src.
In a future patch, we will try to make sure that those <img> elements
get the same Resource if possible. This will reduce network usage,
memory usage, and CPU usage. :^)
For now, this first patch simply introduces the mechanism.
You get a Resource by calling ResourceLoader::load_resource().
To get notified about changes to a Resource's load status, you inherit
from ResourceClient and implement the callbacks you're interested in.
This patch turns HTMLImageElement into a ResourceClient.
