Stowage/ladybird
2
0
Fork 0
mirror of https://github.com/LadybirdBrowser/ladybird.git synced 2026-06-19 00:01:59 +00:00
Code Issues Projects Releases Packages Wiki Activity
ladybird/Libraries/LibWeb/HTML/Parser/HTMLTokenizer.cpp

300 lines
9.8 KiB
C++
Raw Normal View History

LibWeb: Begin work on a spec-compliant HTML parser In order to actually view the web as it is, we're gonna need a proper HTML parser. So let's build one! This patch introduces the Web::HTMLTokenizer class, which currently operates on a StringView input stream where it fetches (ASCII only atm) codepoints and tokenizes acccording to the HTML spec tokenization algo. The tokenizer state machine looks a bit weird but is written in a way that tries to mimic the spec as closely as possible, in order to make development easier and bugs less likely. This initial version is far from finished, but it can parse a trivial document with a DOCTYPE and open/close tags. :^)
2020-05-22 21:46:13 +02:00
/*
Meta: Update my e-mail address everywhere
2024-10-04 13:19:50 +02:00
* Copyright (c) 2020, Andreas Kling <andreas@ladybird.org>
LibWeb: Implement state switch for "[CDATA[" in HTML parser
2022-02-15 18:52:45 +00:00
* Copyright (c) 2022, Linus Groh <linusg@serenityos.org>
LibWeb: Begin work on a spec-compliant HTML parser In order to actually view the web as it is, we're gonna need a proper HTML parser. So let's build one! This patch introduces the Web::HTMLTokenizer class, which currently operates on a StringView input stream where it fetches (ASCII only atm) codepoints and tokenizes acccording to the HTML spec tokenization algo. The tokenizer state machine looks a bit weird but is written in a way that tries to mimic the spec as closely as possible, in order to make development easier and bugs less likely. This initial version is far from finished, but it can parse a trivial document with a DOCTYPE and open/close tags. :^)
2020-05-22 21:46:13 +02:00
*
Everything: Move to SPDX license identifiers in all files. SPDX License Identifiers are a more compact / standardized way of representing file license information. See: https://spdx.dev/resources/use/#identifiers This was done with the `ambr` search and replace tool. ambr --no-parent-ignore --key-from-file --rep-from-file key.txt rep.txt *
2021-04-22 01:24:48 -07:00
* SPDX-License-Identifier: BSD-2-Clause
LibWeb: Begin work on a spec-compliant HTML parser In order to actually view the web as it is, we're gonna need a proper HTML parser. So let's build one! This patch introduces the Web::HTMLTokenizer class, which currently operates on a StringView input stream where it fetches (ASCII only atm) codepoints and tokenizes acccording to the HTML spec tokenization algo. The tokenizer state machine looks a bit weird but is written in a way that tries to mimic the spec as closely as possible, in order to make development easier and bugs less likely. This initial version is far from finished, but it can parse a trivial document with a DOCTYPE and open/close tags. :^)
2020-05-22 21:46:13 +02:00
*/
Everywhere: Replace a bundle of dbg with dbgln. These changes are arbitrarily divided into multiple commits to make it easier to find potentially introduced bugs with git bisect.
2021-01-17 16:57:17 +01:00
#include <AK/Debug.h>
AK+LibWeb: Centralize FFI helper functions This commit creates a central FFIHelpers.h header which implements common conversions from FFI.
2026-05-27 20:08:22 +02:00
#include <AK/FFIHelpers.h>
LibWeb: Replace the HTML tokenizer with Rust Replace the C++ HTML tokenizer with a Rust implementation behind the existing HTMLTokenizer API. Keep the parser-facing integration points for streaming input, insertion points, document.write(), EOF insertion, parser aborts, speculative parser input, and last start tag tracking. The generated FFI handle stays an implementation detail of HTMLTokenizer, so callers keep a single tokenizer class. Preserve duplicate attributes through FFI so C++ token normalization can record the duplicate-attribute signal used by CSP nonce checks. Keep bulk tag-name and attribute scans capped at the active insertion point so streamed parser input is spliced at the right offset. Use generated DAFSA tables for named character references and intern common tag and attribute names to reduce FFI marshalling overhead. This also fixes attribute name source positions, nested old insertion points, and aborted fast-path handling. TestHTMLTokenizer covers duplicate attributes and insertion points in fast tag-name, attribute-name, and quoted-value scans. A CSP text test covers duplicate nonce attributes on parser-created script elements. The tokenizer dump fixtures still match, TestHTMLTokenizer passes, and the full release test-web run passes with 6981 tests and 226 skipped.
2026-05-15 15:13:43 +02:00
#include <AK/FlyString.h>
Meta: Enable exit-time destructor warnings for libraries Enable -Wexit-time-destructors for all in-tree library targets and update process-lifetime library statics so they no longer register exit-time destructors. Long-lived caches, lookup tables, singleton registries, and generated constants now use NeverDestroyed or leaked references where the data is intended to live until process exit. Update LibWeb, LibLine, and the binding generators so regenerated sources follow the same rule instead of reintroducing destructed statics.
2026-06-04 10:40:32 +02:00
#include <AK/NeverDestroyed.h>
LibWeb: Normalize decoded HTML string parsing Preserve leading BOMs when parsing already-decoded HTML strings, since those strings do not go through the encoded byte decoder path. Decoded markup from JS strings can also contain WTF-8 for lone surrogate code units. Keep the common scalar UTF-8 path to a single validation and copy, but replace surrogates before handing bytes to the Rust tokenizer. Add text coverage for DOMParser and innerHTML string parsing, including leading BOMs, text and attributes, lone high and low surrogates, and a valid surrogate pair.
2026-05-24 09:07:05 +02:00
#include <AK/StringBuilder.h>
#include <AK/Utf8View.h>
LibWeb: Replace the HTML tokenizer with Rust Replace the C++ HTML tokenizer with a Rust implementation behind the existing HTMLTokenizer API. Keep the parser-facing integration points for streaming input, insertion points, document.write(), EOF insertion, parser aborts, speculative parser input, and last start tag tracking. The generated FFI handle stays an implementation detail of HTMLTokenizer, so callers keep a single tokenizer class. Preserve duplicate attributes through FFI so C++ token normalization can record the duplicate-attribute signal used by CSP nonce checks. Keep bulk tag-name and attribute scans capped at the active insertion point so streamed parser input is spliced at the right offset. Use generated DAFSA tables for named character references and intern common tag and attribute names to reduce FFI marshalling overhead. This also fixes attribute name source positions, nested old insertion points, and aborted fast-path handling. TestHTMLTokenizer covers duplicate attributes and insertion points in fast tag-name, attribute-name, and quoted-value scans. A CSP text test covers duplicate nonce attributes on parser-created script elements. The tokenizer dump fixtures still match, TestHTMLTokenizer passes, and the full release test-web run passes with 6981 tests and 226 skipped.
2026-05-15 15:13:43 +02:00
#include <AK/Vector.h>
LibWeb: Plumb content encoding into the new HTML parser We still don't handle non-ASCII input correctly, but at least now we'll convert e.g ISO-8859-1 to UTF-8 before starting to tokenize. This patch also makes "view source" work with the new parser. :^)
2020-05-28 12:35:19 +02:00
#include <LibTextCodec/Decoder.h>
LibWeb: Move the HTML parser into HTML/Parser/
2020-07-28 19:18:23 +02:00
#include <LibWeb/HTML/Parser/HTMLToken.h>
#include <LibWeb/HTML/Parser/HTMLTokenizer.h>
LibWeb: Replace the HTML tokenizer with Rust Replace the C++ HTML tokenizer with a Rust implementation behind the existing HTMLTokenizer API. Keep the parser-facing integration points for streaming input, insertion points, document.write(), EOF insertion, parser aborts, speculative parser input, and last start tag tracking. The generated FFI handle stays an implementation detail of HTMLTokenizer, so callers keep a single tokenizer class. Preserve duplicate attributes through FFI so C++ token normalization can record the duplicate-attribute signal used by CSP nonce checks. Keep bulk tag-name and attribute scans capped at the active insertion point so streamed parser input is spliced at the right offset. Use generated DAFSA tables for named character references and intern common tag and attribute names to reduce FFI marshalling overhead. This also fixes attribute name source positions, nested old insertion points, and aborted fast-path handling. TestHTMLTokenizer covers duplicate attributes and insertion points in fast tag-name, attribute-name, and quoted-value scans. A CSP text test covers duplicate nonce attributes on parser-created script elements. The tokenizer dump fixtures still match, TestHTMLTokenizer passes, and the full release test-web run passes with 6981 tests and 226 skipped.
2026-05-15 15:13:43 +02:00
#include <LibWeb/HTMLTokenizerRustFFI.h>
LibWeb: Begin work on a spec-compliant HTML parser In order to actually view the web as it is, we're gonna need a proper HTML parser. So let's build one! This patch introduces the Web::HTMLTokenizer class, which currently operates on a StringView input stream where it fetches (ASCII only atm) codepoints and tokenizes acccording to the HTML spec tokenization algo. The tokenizer state machine looks a bit weird but is written in a way that tries to mimic the spec as closely as possible, in order to make development easier and bugs less likely. This initial version is far from finished, but it can parse a trivial document with a DOCTYPE and open/close tags. :^)
2020-05-22 21:46:13 +02:00
LibWeb: Move HTML classes into the Web::HTML namespace
2020-07-28 18:20:36 +02:00
namespace Web::HTML {
LibWeb: Replace the HTML tokenizer with Rust Replace the C++ HTML tokenizer with a Rust implementation behind the existing HTMLTokenizer API. Keep the parser-facing integration points for streaming input, insertion points, document.write(), EOF insertion, parser aborts, speculative parser input, and last start tag tracking. The generated FFI handle stays an implementation detail of HTMLTokenizer, so callers keep a single tokenizer class. Preserve duplicate attributes through FFI so C++ token normalization can record the duplicate-attribute signal used by CSP nonce checks. Keep bulk tag-name and attribute scans capped at the active insertion point so streamed parser input is spliced at the right offset. Use generated DAFSA tables for named character references and intern common tag and attribute names to reduce FFI marshalling overhead. This also fixes attribute name source positions, nested old insertion points, and aborted fast-path handling. TestHTMLTokenizer covers duplicate attributes and insertion points in fast tag-name, attribute-name, and quoted-value scans. A CSP text test covers duplicate nonce attributes on parser-created script elements. The tokenizer dump fixtures still match, TestHTMLTokenizer passes, and the full release test-web run passes with 6981 tests and 226 skipped.
2026-05-15 15:13:43 +02:00
static Vector<u32> code_points_from_string(String const& string)
{
Vector<u32> code_points;
code_points.ensure_capacity(string.bytes().size());
for (auto code_point : string.code_points())
code_points.append(code_point);
return code_points;
}
LibWeb: Begin work on a spec-compliant HTML parser In order to actually view the web as it is, we're gonna need a proper HTML parser. So let's build one! This patch introduces the Web::HTMLTokenizer class, which currently operates on a StringView input stream where it fetches (ASCII only atm) codepoints and tokenizes acccording to the HTML spec tokenization algo. The tokenizer state machine looks a bit weird but is written in a way that tries to mimic the spec as closely as possible, in order to make development easier and bugs less likely. This initial version is far from finished, but it can parse a trivial document with a DOCTYPE and open/close tags. :^)
2020-05-22 21:46:13 +02:00
LibWeb: Replace the HTML tokenizer with Rust Replace the C++ HTML tokenizer with a Rust implementation behind the existing HTMLTokenizer API. Keep the parser-facing integration points for streaming input, insertion points, document.write(), EOF insertion, parser aborts, speculative parser input, and last start tag tracking. The generated FFI handle stays an implementation detail of HTMLTokenizer, so callers keep a single tokenizer class. Preserve duplicate attributes through FFI so C++ token normalization can record the duplicate-attribute signal used by CSP nonce checks. Keep bulk tag-name and attribute scans capped at the active insertion point so streamed parser input is spliced at the right offset. Use generated DAFSA tables for named character references and intern common tag and attribute names to reduce FFI marshalling overhead. This also fixes attribute name source positions, nested old insertion points, and aborted fast-path handling. TestHTMLTokenizer covers duplicate attributes and insertion points in fast tag-name, attribute-name, and quoted-value scans. A CSP text test covers duplicate nonce attributes on parser-created script elements. The tokenizer dump fixtures still match, TestHTMLTokenizer passes, and the full release test-web run passes with 6981 tests and 226 skipped.
2026-05-15 15:13:43 +02:00
static RustFfiTokenizerHandle* create_tokenizer_from_utf8(StringView utf8_bytes)
LibWeb: Pause tokenizer at a CR right before the insertion point HTML newline normalization collapses CRLF into a single LF, so next_code_point() needs one code point of lookahead at a CR to decide whether the CR stands alone or is the first half of a CRLF pair. When the tokenizer is paused at the insertion point and the next code point to consume is a CR sitting one position before it, that lookahead has not been written yet. Previously the tokenizer consumed the CR and emitted it as LF, so a subsequent document.write() that began with LF surfaced as a second LF instead of being absorbed into the original CRLF pair. Stop one code point earlier in this case and wait for the next write to arrive. This makes four html5lib write_single WPT tests pass.
2026-04-27 20:38:28 +02:00
{
LibWeb: Replace the HTML tokenizer with Rust Replace the C++ HTML tokenizer with a Rust implementation behind the existing HTMLTokenizer API. Keep the parser-facing integration points for streaming input, insertion points, document.write(), EOF insertion, parser aborts, speculative parser input, and last start tag tracking. The generated FFI handle stays an implementation detail of HTMLTokenizer, so callers keep a single tokenizer class. Preserve duplicate attributes through FFI so C++ token normalization can record the duplicate-attribute signal used by CSP nonce checks. Keep bulk tag-name and attribute scans capped at the active insertion point so streamed parser input is spliced at the right offset. Use generated DAFSA tables for named character references and intern common tag and attribute names to reduce FFI marshalling overhead. This also fixes attribute name source positions, nested old insertion points, and aborted fast-path handling. TestHTMLTokenizer covers duplicate attributes and insertion points in fast tag-name, attribute-name, and quoted-value scans. A CSP text test covers duplicate nonce attributes on parser-created script elements. The tokenizer dump fixtures still match, TestHTMLTokenizer passes, and the full release test-web run passes with 6981 tests and 226 skipped.
2026-05-15 15:13:43 +02:00
auto* bytes = reinterpret_cast<u8 const*>(utf8_bytes.characters_without_null_termination());
if (bytes == nullptr)
bytes = reinterpret_cast<u8 const*>("");
return rust_html_tokenizer_create_from_utf8(bytes, utf8_bytes.length());
}
LibWeb: Track input stream closed state in HTMLTokenizer Add an explicit "input stream closed" flag plus the streaming-input API (append_to_input_stream, close_input_stream, is_input_stream_closed) to let a future incremental driver feed bytes as they arrive. Rewrite should_pause_before_next_input_character so the tokenizer pauses when the buffer is exhausted but more bytes may still arrive, including the case where a chunk ends in CR (CRLF normalization needs one code point of lookahead). Existing call sites are unaffected: the StringView constructor immediately marks the input stream closed, and insert_eof() now also closes the stream so document.close() drives the same exit path.
2026-04-28 18:04:17 +02:00
LibWeb: Normalize decoded HTML string parsing Preserve leading BOMs when parsing already-decoded HTML strings, since those strings do not go through the encoded byte decoder path. Decoded markup from JS strings can also contain WTF-8 for lone surrogate code units. Keep the common scalar UTF-8 path to a single validation and copy, but replace surrogates before handing bytes to the Rust tokenizer. Add text coverage for DOMParser and innerHTML string parsing, including leading BOMs, text and attributes, lone high and low surrogates, and a valid surrogate pair.
2026-05-24 09:07:05 +02:00
static String decoded_string_for_utf8_tokenizer(StringView input)
{
Utf8View utf8_view { input };
if (utf8_view.validate(AllowLonelySurrogates::No))
return String::from_utf8_without_validation(input.bytes());
// Decoded strings may come from WTF-16 JS strings. Rust's UTF-8 path
// requires scalar-value UTF-8, so replace lone surrogates but keep BOMs.
VERIFY(utf8_view.validate());
StringBuilder builder(input.length());
for (auto code_point : utf8_view)
builder.append_code_point(is_unicode_surrogate(code_point) ? AK::UnicodeUtils::REPLACEMENT_CODE_POINT : code_point);
return builder.to_string_without_validation();
}
LibWeb: Replace the HTML tokenizer with Rust Replace the C++ HTML tokenizer with a Rust implementation behind the existing HTMLTokenizer API. Keep the parser-facing integration points for streaming input, insertion points, document.write(), EOF insertion, parser aborts, speculative parser input, and last start tag tracking. The generated FFI handle stays an implementation detail of HTMLTokenizer, so callers keep a single tokenizer class. Preserve duplicate attributes through FFI so C++ token normalization can record the duplicate-attribute signal used by CSP nonce checks. Keep bulk tag-name and attribute scans capped at the active insertion point so streamed parser input is spliced at the right offset. Use generated DAFSA tables for named character references and intern common tag and attribute names to reduce FFI marshalling overhead. This also fixes attribute name source positions, nested old insertion points, and aborted fast-path handling. TestHTMLTokenizer covers duplicate attributes and insertion points in fast tag-name, attribute-name, and quoted-value scans. A CSP text test covers duplicate nonce attributes on parser-created script elements. The tokenizer dump fixtures still match, TestHTMLTokenizer passes, and the full release test-web run passes with 6981 tests and 226 skipped.
2026-05-15 15:13:43 +02:00
static Vector<FlyString> build_interned_name_table(size_t count, void (*fetch)(uint16_t, uint8_t const**, size_t*))
{
Vector<FlyString> table;
// Slot 0 is unused (id 0 means "not interned"); store an empty FlyString there.
table.append(FlyString {});
table.ensure_capacity(count + 1);
for (size_t i = 0; i < count; ++i) {
uint8_t const* ptr = nullptr;
size_t len = 0;
fetch(static_cast<uint16_t>(i + 1), &ptr, &len);
if (ptr == nullptr || len == 0) {
table.append(FlyString {});
continue;
LibWeb: Track input stream closed state in HTMLTokenizer Add an explicit "input stream closed" flag plus the streaming-input API (append_to_input_stream, close_input_stream, is_input_stream_closed) to let a future incremental driver feed bytes as they arrive. Rewrite should_pause_before_next_input_character so the tokenizer pauses when the buffer is exhausted but more bytes may still arrive, including the case where a chunk ends in CR (CRLF normalization needs one code point of lookahead). Existing call sites are unaffected: the StringView constructor immediately marks the input stream closed, and insert_eof() now also closes the stream so document.close() drives the same exit path.
2026-04-28 18:04:17 +02:00
}
LibWeb: Replace the HTML tokenizer with Rust Replace the C++ HTML tokenizer with a Rust implementation behind the existing HTMLTokenizer API. Keep the parser-facing integration points for streaming input, insertion points, document.write(), EOF insertion, parser aborts, speculative parser input, and last start tag tracking. The generated FFI handle stays an implementation detail of HTMLTokenizer, so callers keep a single tokenizer class. Preserve duplicate attributes through FFI so C++ token normalization can record the duplicate-attribute signal used by CSP nonce checks. Keep bulk tag-name and attribute scans capped at the active insertion point so streamed parser input is spliced at the right offset. Use generated DAFSA tables for named character references and intern common tag and attribute names to reduce FFI marshalling overhead. This also fixes attribute name source positions, nested old insertion points, and aborted fast-path handling. TestHTMLTokenizer covers duplicate attributes and insertion points in fast tag-name, attribute-name, and quoted-value scans. A CSP text test covers duplicate nonce attributes on parser-created script elements. The tokenizer dump fixtures still match, TestHTMLTokenizer passes, and the full release test-web run passes with 6981 tests and 226 skipped.
2026-05-15 15:13:43 +02:00
table.append(MUST(FlyString::from_utf8(StringView { ptr, len })));
LibWeb: Track input stream closed state in HTMLTokenizer Add an explicit "input stream closed" flag plus the streaming-input API (append_to_input_stream, close_input_stream, is_input_stream_closed) to let a future incremental driver feed bytes as they arrive. Rewrite should_pause_before_next_input_character so the tokenizer pauses when the buffer is exhausted but more bytes may still arrive, including the case where a chunk ends in CR (CRLF normalization needs one code point of lookahead). Existing call sites are unaffected: the StringView constructor immediately marks the input stream closed, and insert_eof() now also closes the stream so document.close() drives the same exit path.
2026-04-28 18:04:17 +02:00
}
LibWeb: Replace the HTML tokenizer with Rust Replace the C++ HTML tokenizer with a Rust implementation behind the existing HTMLTokenizer API. Keep the parser-facing integration points for streaming input, insertion points, document.write(), EOF insertion, parser aborts, speculative parser input, and last start tag tracking. The generated FFI handle stays an implementation detail of HTMLTokenizer, so callers keep a single tokenizer class. Preserve duplicate attributes through FFI so C++ token normalization can record the duplicate-attribute signal used by CSP nonce checks. Keep bulk tag-name and attribute scans capped at the active insertion point so streamed parser input is spliced at the right offset. Use generated DAFSA tables for named character references and intern common tag and attribute names to reduce FFI marshalling overhead. This also fixes attribute name source positions, nested old insertion points, and aborted fast-path handling. TestHTMLTokenizer covers duplicate attributes and insertion points in fast tag-name, attribute-name, and quoted-value scans. A CSP text test covers duplicate nonce attributes on parser-created script elements. The tokenizer dump fixtures still match, TestHTMLTokenizer passes, and the full release test-web run passes with 6981 tests and 226 skipped.
2026-05-15 15:13:43 +02:00
return table;
LibWeb: Pause tokenizer at a CR right before the insertion point HTML newline normalization collapses CRLF into a single LF, so next_code_point() needs one code point of lookahead at a CR to decide whether the CR stands alone or is the first half of a CRLF pair. When the tokenizer is paused at the insertion point and the next code point to consume is a CR sitting one position before it, that lookahead has not been written yet. Previously the tokenizer consumed the CR and emitted it as LF, so a subsequent document.write() that began with LF surfaced as a second LF instead of being absorbed into the original CRLF pair. Stop one code point earlier in this case and wait for the next write to arrive. This makes four html5lib write_single WPT tests pass.
2026-04-27 20:38:28 +02:00
}
LibWeb: Replace the HTML tokenizer with Rust Replace the C++ HTML tokenizer with a Rust implementation behind the existing HTMLTokenizer API. Keep the parser-facing integration points for streaming input, insertion points, document.write(), EOF insertion, parser aborts, speculative parser input, and last start tag tracking. The generated FFI handle stays an implementation detail of HTMLTokenizer, so callers keep a single tokenizer class. Preserve duplicate attributes through FFI so C++ token normalization can record the duplicate-attribute signal used by CSP nonce checks. Keep bulk tag-name and attribute scans capped at the active insertion point so streamed parser input is spliced at the right offset. Use generated DAFSA tables for named character references and intern common tag and attribute names to reduce FFI marshalling overhead. This also fixes attribute name source positions, nested old insertion points, and aborted fast-path handling. TestHTMLTokenizer covers duplicate attributes and insertion points in fast tag-name, attribute-name, and quoted-value scans. A CSP text test covers duplicate nonce attributes on parser-created script elements. The tokenizer dump fixtures still match, TestHTMLTokenizer passes, and the full release test-web run passes with 6981 tests and 226 skipped.
2026-05-15 15:13:43 +02:00
static FlyString const& interned_rust_tag_name(uint16_t id)
LibWeb: Allow tokenizer to run out of characters mid-tokenization Add can_run_out_of_characters() and use it in the NamedCharacterReference state and consume_next_if_match() so that an open input stream gets the same code-point-at-a-time treatment as an active document.write insertion point. Without this, a network chunk that ends partway through a named character reference or a multi-character match would make the tokenizer commit to a "no match" decision before the remaining bytes arrive. No behavior change for existing callers: the new helper still returns false once the input stream is closed (which the StringView constructor sets immediately).
2026-04-28 18:05:24 +02:00
{
Meta: Enable exit-time destructor warnings for libraries Enable -Wexit-time-destructors for all in-tree library targets and update process-lifetime library statics so they no longer register exit-time destructors. Long-lived caches, lookup tables, singleton registries, and generated constants now use NeverDestroyed or leaked references where the data is intended to live until process exit. Update LibWeb, LibLine, and the binding generators so regenerated sources follow the same rule instead of reintroducing destructed statics.
2026-06-04 10:40:32 +02:00
static NeverDestroyed<Vector<FlyString>> table { build_interned_name_table(
LibWeb: Replace the HTML tokenizer with Rust Replace the C++ HTML tokenizer with a Rust implementation behind the existing HTMLTokenizer API. Keep the parser-facing integration points for streaming input, insertion points, document.write(), EOF insertion, parser aborts, speculative parser input, and last start tag tracking. The generated FFI handle stays an implementation detail of HTMLTokenizer, so callers keep a single tokenizer class. Preserve duplicate attributes through FFI so C++ token normalization can record the duplicate-attribute signal used by CSP nonce checks. Keep bulk tag-name and attribute scans capped at the active insertion point so streamed parser input is spliced at the right offset. Use generated DAFSA tables for named character references and intern common tag and attribute names to reduce FFI marshalling overhead. This also fixes attribute name source positions, nested old insertion points, and aborted fast-path handling. TestHTMLTokenizer covers duplicate attributes and insertion points in fast tag-name, attribute-name, and quoted-value scans. A CSP text test covers duplicate nonce attributes on parser-created script elements. The tokenizer dump fixtures still match, TestHTMLTokenizer passes, and the full release test-web run passes with 6981 tests and 226 skipped.
2026-05-15 15:13:43 +02:00
rust_html_tokenizer_interned_tag_name_count(),
Meta: Enable exit-time destructor warnings for libraries Enable -Wexit-time-destructors for all in-tree library targets and update process-lifetime library statics so they no longer register exit-time destructors. Long-lived caches, lookup tables, singleton registries, and generated constants now use NeverDestroyed or leaked references where the data is intended to live until process exit. Update LibWeb, LibLine, and the binding generators so regenerated sources follow the same rule instead of reintroducing destructed statics.
2026-06-04 10:40:32 +02:00
rust_html_tokenizer_interned_tag_name) };
if (id == 0 || id >= table->size())
return (*table)[0];
return (*table)[id];
LibWeb: Allow tokenizer to run out of characters mid-tokenization Add can_run_out_of_characters() and use it in the NamedCharacterReference state and consume_next_if_match() so that an open input stream gets the same code-point-at-a-time treatment as an active document.write insertion point. Without this, a network chunk that ends partway through a named character reference or a multi-character match would make the tokenizer commit to a "no match" decision before the remaining bytes arrive. No behavior change for existing callers: the new helper still returns false once the input stream is closed (which the StringView constructor sets immediately).
2026-04-28 18:05:24 +02:00
}
LibWeb: Replace the HTML tokenizer with Rust Replace the C++ HTML tokenizer with a Rust implementation behind the existing HTMLTokenizer API. Keep the parser-facing integration points for streaming input, insertion points, document.write(), EOF insertion, parser aborts, speculative parser input, and last start tag tracking. The generated FFI handle stays an implementation detail of HTMLTokenizer, so callers keep a single tokenizer class. Preserve duplicate attributes through FFI so C++ token normalization can record the duplicate-attribute signal used by CSP nonce checks. Keep bulk tag-name and attribute scans capped at the active insertion point so streamed parser input is spliced at the right offset. Use generated DAFSA tables for named character references and intern common tag and attribute names to reduce FFI marshalling overhead. This also fixes attribute name source positions, nested old insertion points, and aborted fast-path handling. TestHTMLTokenizer covers duplicate attributes and insertion points in fast tag-name, attribute-name, and quoted-value scans. A CSP text test covers duplicate nonce attributes on parser-created script elements. The tokenizer dump fixtures still match, TestHTMLTokenizer passes, and the full release test-web run passes with 6981 tests and 226 skipped.
2026-05-15 15:13:43 +02:00
static FlyString const& interned_rust_attr_name(uint16_t id)
LibWeb: Begin work on a spec-compliant HTML parser In order to actually view the web as it is, we're gonna need a proper HTML parser. So let's build one! This patch introduces the Web::HTMLTokenizer class, which currently operates on a StringView input stream where it fetches (ASCII only atm) codepoints and tokenizes acccording to the HTML spec tokenization algo. The tokenizer state machine looks a bit weird but is written in a way that tries to mimic the spec as closely as possible, in order to make development easier and bugs less likely. This initial version is far from finished, but it can parse a trivial document with a DOCTYPE and open/close tags. :^)
2020-05-22 21:46:13 +02:00
{
Meta: Enable exit-time destructor warnings for libraries Enable -Wexit-time-destructors for all in-tree library targets and update process-lifetime library statics so they no longer register exit-time destructors. Long-lived caches, lookup tables, singleton registries, and generated constants now use NeverDestroyed or leaked references where the data is intended to live until process exit. Update LibWeb, LibLine, and the binding generators so regenerated sources follow the same rule instead of reintroducing destructed statics.
2026-06-04 10:40:32 +02:00
static NeverDestroyed<Vector<FlyString>> table { build_interned_name_table(
LibWeb: Replace the HTML tokenizer with Rust Replace the C++ HTML tokenizer with a Rust implementation behind the existing HTMLTokenizer API. Keep the parser-facing integration points for streaming input, insertion points, document.write(), EOF insertion, parser aborts, speculative parser input, and last start tag tracking. The generated FFI handle stays an implementation detail of HTMLTokenizer, so callers keep a single tokenizer class. Preserve duplicate attributes through FFI so C++ token normalization can record the duplicate-attribute signal used by CSP nonce checks. Keep bulk tag-name and attribute scans capped at the active insertion point so streamed parser input is spliced at the right offset. Use generated DAFSA tables for named character references and intern common tag and attribute names to reduce FFI marshalling overhead. This also fixes attribute name source positions, nested old insertion points, and aborted fast-path handling. TestHTMLTokenizer covers duplicate attributes and insertion points in fast tag-name, attribute-name, and quoted-value scans. A CSP text test covers duplicate nonce attributes on parser-created script elements. The tokenizer dump fixtures still match, TestHTMLTokenizer passes, and the full release test-web run passes with 6981 tests and 226 skipped.
2026-05-15 15:13:43 +02:00
rust_html_tokenizer_interned_attr_name_count(),
Meta: Enable exit-time destructor warnings for libraries Enable -Wexit-time-destructors for all in-tree library targets and update process-lifetime library statics so they no longer register exit-time destructors. Long-lived caches, lookup tables, singleton registries, and generated constants now use NeverDestroyed or leaked references where the data is intended to live until process exit. Update LibWeb, LibLine, and the binding generators so regenerated sources follow the same rule instead of reintroducing destructed statics.
2026-06-04 10:40:32 +02:00
rust_html_tokenizer_interned_attr_name) };
if (id == 0 || id >= table->size())
return (*table)[0];
return (*table)[id];
LibWeb: Begin work on a spec-compliant HTML parser In order to actually view the web as it is, we're gonna need a proper HTML parser. So let's build one! This patch introduces the Web::HTMLTokenizer class, which currently operates on a StringView input stream where it fetches (ASCII only atm) codepoints and tokenizes acccording to the HTML spec tokenization algo. The tokenizer state machine looks a bit weird but is written in a way that tries to mimic the spec as closely as possible, in order to make development easier and bugs less likely. This initial version is far from finished, but it can parse a trivial document with a DOCTYPE and open/close tags. :^)
2020-05-22 21:46:13 +02:00
}
LibWeb: Replace the HTML tokenizer with Rust Replace the C++ HTML tokenizer with a Rust implementation behind the existing HTMLTokenizer API. Keep the parser-facing integration points for streaming input, insertion points, document.write(), EOF insertion, parser aborts, speculative parser input, and last start tag tracking. The generated FFI handle stays an implementation detail of HTMLTokenizer, so callers keep a single tokenizer class. Preserve duplicate attributes through FFI so C++ token normalization can record the duplicate-attribute signal used by CSP nonce checks. Keep bulk tag-name and attribute scans capped at the active insertion point so streamed parser input is spliced at the right offset. Use generated DAFSA tables for named character references and intern common tag and attribute names to reduce FFI marshalling overhead. This also fixes attribute name source positions, nested old insertion points, and aborted fast-path handling. TestHTMLTokenizer covers duplicate attributes and insertion points in fast tag-name, attribute-name, and quoted-value scans. A CSP text test covers duplicate nonce attributes on parser-created script elements. The tokenizer dump fixtures still match, TestHTMLTokenizer passes, and the full release test-web run passes with 6981 tests and 226 skipped.
2026-05-15 15:13:43 +02:00
HTMLTokenizer::HTMLTokenizer()
LibWeb: Add position tracking information to HTML tokens
2021-05-20 23:11:41 +04:30
{
LibWeb: Replace the HTML tokenizer with Rust Replace the C++ HTML tokenizer with a Rust implementation behind the existing HTMLTokenizer API. Keep the parser-facing integration points for streaming input, insertion points, document.write(), EOF insertion, parser aborts, speculative parser input, and last start tag tracking. The generated FFI handle stays an implementation detail of HTMLTokenizer, so callers keep a single tokenizer class. Preserve duplicate attributes through FFI so C++ token normalization can record the duplicate-attribute signal used by CSP nonce checks. Keep bulk tag-name and attribute scans capped at the active insertion point so streamed parser input is spliced at the right offset. Use generated DAFSA tables for named character references and intern common tag and attribute names to reduce FFI marshalling overhead. This also fixes attribute name source positions, nested old insertion points, and aborted fast-path handling. TestHTMLTokenizer covers duplicate attributes and insertion points in fast tag-name, attribute-name, and quoted-value scans. A CSP text test covers duplicate nonce attributes on parser-created script elements. The tokenizer dump fixtures still match, TestHTMLTokenizer passes, and the full release test-web run passes with 6981 tests and 226 skipped.
2026-05-15 15:13:43 +02:00
m_tokenizer = create_tokenizer_from_utf8({});
rust_html_tokenizer_set_input_stream_closed(m_tokenizer, false);
LibWeb: Add position tracking information to HTML tokens
2021-05-20 23:11:41 +04:30
}
LibWeb: Replace the HTML tokenizer with Rust Replace the C++ HTML tokenizer with a Rust implementation behind the existing HTMLTokenizer API. Keep the parser-facing integration points for streaming input, insertion points, document.write(), EOF insertion, parser aborts, speculative parser input, and last start tag tracking. The generated FFI handle stays an implementation detail of HTMLTokenizer, so callers keep a single tokenizer class. Preserve duplicate attributes through FFI so C++ token normalization can record the duplicate-attribute signal used by CSP nonce checks. Keep bulk tag-name and attribute scans capped at the active insertion point so streamed parser input is spliced at the right offset. Use generated DAFSA tables for named character references and intern common tag and attribute names to reduce FFI marshalling overhead. This also fixes attribute name source positions, nested old insertion points, and aborted fast-path handling. TestHTMLTokenizer covers duplicate attributes and insertion points in fast tag-name, attribute-name, and quoted-value scans. A CSP text test covers duplicate nonce attributes on parser-created script elements. The tokenizer dump fixtures still match, TestHTMLTokenizer passes, and the full release test-web run passes with 6981 tests and 226 skipped.
2026-05-15 15:13:43 +02:00
HTMLTokenizer::~HTMLTokenizer()
LibWeb: Begin work on a spec-compliant HTML parser In order to actually view the web as it is, we're gonna need a proper HTML parser. So let's build one! This patch introduces the Web::HTMLTokenizer class, which currently operates on a StringView input stream where it fetches (ASCII only atm) codepoints and tokenizes acccording to the HTML spec tokenization algo. The tokenizer state machine looks a bit weird but is written in a way that tries to mimic the spec as closely as possible, in order to make development easier and bugs less likely. This initial version is far from finished, but it can parse a trivial document with a DOCTYPE and open/close tags. :^)
2020-05-22 21:46:13 +02:00
{
LibWeb: Replace the HTML tokenizer with Rust Replace the C++ HTML tokenizer with a Rust implementation behind the existing HTMLTokenizer API. Keep the parser-facing integration points for streaming input, insertion points, document.write(), EOF insertion, parser aborts, speculative parser input, and last start tag tracking. The generated FFI handle stays an implementation detail of HTMLTokenizer, so callers keep a single tokenizer class. Preserve duplicate attributes through FFI so C++ token normalization can record the duplicate-attribute signal used by CSP nonce checks. Keep bulk tag-name and attribute scans capped at the active insertion point so streamed parser input is spliced at the right offset. Use generated DAFSA tables for named character references and intern common tag and attribute names to reduce FFI marshalling overhead. This also fixes attribute name source positions, nested old insertion points, and aborted fast-path handling. TestHTMLTokenizer covers duplicate attributes and insertion points in fast tag-name, attribute-name, and quoted-value scans. A CSP text test covers duplicate nonce attributes on parser-created script elements. The tokenizer dump fixtures still match, TestHTMLTokenizer passes, and the full release test-web run passes with 6981 tests and 226 skipped.
2026-05-15 15:13:43 +02:00
if (m_tokenizer)
rust_html_tokenizer_destroy(m_tokenizer);
LibWeb: Begin work on a spec-compliant HTML parser In order to actually view the web as it is, we're gonna need a proper HTML parser. So let's build one! This patch introduces the Web::HTMLTokenizer class, which currently operates on a StringView input stream where it fetches (ASCII only atm) codepoints and tokenizes acccording to the HTML spec tokenization algo. The tokenizer state machine looks a bit weird but is written in a way that tries to mimic the spec as closely as possible, in order to make development easier and bugs less likely. This initial version is far from finished, but it can parse a trivial document with a DOCTYPE and open/close tags. :^)
2020-05-22 21:46:13 +02:00
}
LibWeb: Normalize decoded HTML string parsing Preserve leading BOMs when parsing already-decoded HTML strings, since those strings do not go through the encoded byte decoder path. Decoded markup from JS strings can also contain WTF-8 for lone surrogate code units. Keep the common scalar UTF-8 path to a single validation and copy, but replace surrogates before handing bytes to the Rust tokenizer. Add text coverage for DOMParser and innerHTML string parsing, including leading BOMs, text and attributes, lone high and low surrogates, and a valid surrogate pair.
2026-05-24 09:07:05 +02:00
HTMLTokenizer::HTMLTokenizer(StringView input, ByteString const& encoding, InputType input_type)
LibWeb: Be more forgiving when adding source positions in HTMLTokenizer This patch changes HTMLTokenizer::nth_last_position to not fail if the requested position is not available. Rather, it will just return (0-0). While this is not the correct solution, it prevents the tokenizer from crashing just because it cannot find a source position. This should only affect SyntaxHighlighter.
2021-06-04 11:31:43 +02:00
{
LibWeb: Normalize decoded HTML string parsing Preserve leading BOMs when parsing already-decoded HTML strings, since those strings do not go through the encoded byte decoder path. Decoded markup from JS strings can also contain WTF-8 for lone surrogate code units. Keep the common scalar UTF-8 path to a single validation and copy, but replace surrogates before handing bytes to the Rust tokenizer. Add text coverage for DOMParser and innerHTML string parsing, including leading BOMs, text and attributes, lone high and low surrogates, and a valid surrogate pair.
2026-05-24 09:07:05 +02:00
if (input_type == InputType::EncodedBytes) {
auto decoder = TextCodec::decoder_for(encoding);
VERIFY(decoder.has_value());
m_source = MUST(decoder->to_utf8(input));
} else {
m_source = decoded_string_for_utf8_tokenizer(input);
}
LibWeb: Replace the HTML tokenizer with Rust Replace the C++ HTML tokenizer with a Rust implementation behind the existing HTMLTokenizer API. Keep the parser-facing integration points for streaming input, insertion points, document.write(), EOF insertion, parser aborts, speculative parser input, and last start tag tracking. The generated FFI handle stays an implementation detail of HTMLTokenizer, so callers keep a single tokenizer class. Preserve duplicate attributes through FFI so C++ token normalization can record the duplicate-attribute signal used by CSP nonce checks. Keep bulk tag-name and attribute scans capped at the active insertion point so streamed parser input is spliced at the right offset. Use generated DAFSA tables for named character references and intern common tag and attribute names to reduce FFI marshalling overhead. This also fixes attribute name source positions, nested old insertion points, and aborted fast-path handling. TestHTMLTokenizer covers duplicate attributes and insertion points in fast tag-name, attribute-name, and quoted-value scans. A CSP text test covers duplicate nonce attributes on parser-created script elements. The tokenizer dump fixtures still match, TestHTMLTokenizer passes, and the full release test-web run passes with 6981 tests and 226 skipped.
2026-05-15 15:13:43 +02:00
m_input_stream_closed = true;
m_tokenizer = create_tokenizer_from_utf8(m_source.bytes_as_string_view());
LibWeb: Be more forgiving when adding source positions in HTMLTokenizer This patch changes HTMLTokenizer::nth_last_position to not fail if the requested position is not available. Rather, it will just return (0-0). While this is not the correct solution, it prevents the tokenizer from crashing just because it cannot find a source position. This should only affect SyntaxHighlighter.
2021-06-04 11:31:43 +02:00
}
LibWeb: Stop parsing after `document.write` at the insertion point If a call to `document.write` inserts an incomplete HTML tag, e.g.: document.write("<p"); we would previously continue parsing the document until we reached a closing angle bracket. However, the spec states we should stop once we reach the new insertion point.
2024-02-18 12:45:53 -05:00
Optional<HTMLToken> HTMLTokenizer::next_token(StopAtInsertionPoint stop_at_insertion_point)
LibWeb: Begin work on a spec-compliant HTML parser In order to actually view the web as it is, we're gonna need a proper HTML parser. So let's build one! This patch introduces the Web::HTMLTokenizer class, which currently operates on a StringView input stream where it fetches (ASCII only atm) codepoints and tokenizes acccording to the HTML spec tokenization algo. The tokenizer state machine looks a bit weird but is written in a way that tries to mimic the spec as closely as possible, in order to make development easier and bugs less likely. This initial version is far from finished, but it can parse a trivial document with a DOCTYPE and open/close tags. :^)
2020-05-22 21:46:13 +02:00
{
LibWeb: Replace the HTML tokenizer with Rust Replace the C++ HTML tokenizer with a Rust implementation behind the existing HTMLTokenizer API. Keep the parser-facing integration points for streaming input, insertion points, document.write(), EOF insertion, parser aborts, speculative parser input, and last start tag tracking. The generated FFI handle stays an implementation detail of HTMLTokenizer, so callers keep a single tokenizer class. Preserve duplicate attributes through FFI so C++ token normalization can record the duplicate-attribute signal used by CSP nonce checks. Keep bulk tag-name and attribute scans capped at the active insertion point so streamed parser input is spliced at the right offset. Use generated DAFSA tables for named character references and intern common tag and attribute names to reduce FFI marshalling overhead. This also fixes attribute name source positions, nested old insertion points, and aborted fast-path handling. TestHTMLTokenizer covers duplicate attributes and insertion points in fast tag-name, attribute-name, and quoted-value scans. A CSP text test covers duplicate nonce attributes on parser-created script elements. The tokenizer dump fixtures still match, TestHTMLTokenizer passes, and the full release test-web run passes with 6981 tests and 226 skipped.
2026-05-15 15:13:43 +02:00
RustFfiToken ffi;
bool stop = stop_at_insertion_point == StopAtInsertionPoint::Yes;
LibWeb: Remove the C++ HTML tree builder Delete the old C++ tree-construction implementation and helper classes that became unused once the Rust parser is unconditional. Remove the C++ stack of open elements, active formatting elements, speculative mock element, and tree-builder-only token storage. Keep the C++ parser entry points that still own LibWeb DOM integration, encoding detection, tokenizer bridging, incremental parsing, and the speculative parser support used by resource discovery.
2026-05-16 13:47:49 +02:00
bool cdata_allowed = false;
LibWeb: Replace the HTML tokenizer with Rust Replace the C++ HTML tokenizer with a Rust implementation behind the existing HTMLTokenizer API. Keep the parser-facing integration points for streaming input, insertion points, document.write(), EOF insertion, parser aborts, speculative parser input, and last start tag tracking. The generated FFI handle stays an implementation detail of HTMLTokenizer, so callers keep a single tokenizer class. Preserve duplicate attributes through FFI so C++ token normalization can record the duplicate-attribute signal used by CSP nonce checks. Keep bulk tag-name and attribute scans capped at the active insertion point so streamed parser input is spliced at the right offset. Use generated DAFSA tables for named character references and intern common tag and attribute names to reduce FFI marshalling overhead. This also fixes attribute name source positions, nested old insertion points, and aborted fast-path handling. TestHTMLTokenizer covers duplicate attributes and insertion points in fast tag-name, attribute-name, and quoted-value scans. A CSP text test covers duplicate nonce attributes on parser-created script elements. The tokenizer dump fixtures still match, TestHTMLTokenizer passes, and the full release test-web run passes with 6981 tests and 226 skipped.
2026-05-15 15:13:43 +02:00
if (!rust_html_tokenizer_next_token(m_tokenizer, &ffi, stop, cdata_allowed))
LibWeb: Implement aborting the HTML parser This is roughly on-spec, although I had to invent a simple "aborted" state for the tokenizer.
2022-09-20 21:08:14 +02:00
return {};
LibWeb: Replace the HTML tokenizer with Rust Replace the C++ HTML tokenizer with a Rust implementation behind the existing HTMLTokenizer API. Keep the parser-facing integration points for streaming input, insertion points, document.write(), EOF insertion, parser aborts, speculative parser input, and last start tag tracking. The generated FFI handle stays an implementation detail of HTMLTokenizer, so callers keep a single tokenizer class. Preserve duplicate attributes through FFI so C++ token normalization can record the duplicate-attribute signal used by CSP nonce checks. Keep bulk tag-name and attribute scans capped at the active insertion point so streamed parser input is spliced at the right offset. Use generated DAFSA tables for named character references and intern common tag and attribute names to reduce FFI marshalling overhead. This also fixes attribute name source positions, nested old insertion points, and aborted fast-path handling. TestHTMLTokenizer covers duplicate attributes and insertion points in fast tag-name, attribute-name, and quoted-value scans. A CSP text test covers duplicate nonce attributes on parser-created script elements. The tokenizer dump fixtures still match, TestHTMLTokenizer passes, and the full release test-web run passes with 6981 tests and 226 skipped.
2026-05-15 15:13:43 +02:00
HTMLToken::Type type;
switch (ffi.token_type) {
case 1:
type = HTMLToken::Type::DOCTYPE;
break;
case 2:
type = HTMLToken::Type::StartTag;
break;
case 3:
type = HTMLToken::Type::EndTag;
break;
case 4:
type = HTMLToken::Type::Comment;
break;
case 5:
type = HTMLToken::Type::Character;
break;
case 6:
type = HTMLToken::Type::EndOfFile;
break;
default:
VERIFY_NOT_REACHED();
}
LibWeb: Implement all CDATA tokenizer states Even though we haven't implemented any switches to these states yet, we may as well have them ready for when we do implement the switches.
2020-06-12 23:43:06 +01:00
LibWeb: Replace the HTML tokenizer with Rust Replace the C++ HTML tokenizer with a Rust implementation behind the existing HTMLTokenizer API. Keep the parser-facing integration points for streaming input, insertion points, document.write(), EOF insertion, parser aborts, speculative parser input, and last start tag tracking. The generated FFI handle stays an implementation detail of HTMLTokenizer, so callers keep a single tokenizer class. Preserve duplicate attributes through FFI so C++ token normalization can record the duplicate-attribute signal used by CSP nonce checks. Keep bulk tag-name and attribute scans capped at the active insertion point so streamed parser input is spliced at the right offset. Use generated DAFSA tables for named character references and intern common tag and attribute names to reduce FFI marshalling overhead. This also fixes attribute name source positions, nested old insertion points, and aborted fast-path handling. TestHTMLTokenizer covers duplicate attributes and insertion points in fast tag-name, attribute-name, and quoted-value scans. A CSP text test covers duplicate nonce attributes on parser-created script elements. The tokenizer dump fixtures still match, TestHTMLTokenizer passes, and the full release test-web run passes with 6981 tests and 226 skipped.
2026-05-15 15:13:43 +02:00
HTMLToken token { type };
token.set_start_position({}, { ffi.start_line, ffi.start_column });
token.set_end_position({}, { ffi.end_line, ffi.end_column });
switch (type) {
case HTMLToken::Type::Character:
token.set_code_point(ffi.code_point);
break;
case HTMLToken::Type::StartTag:
case HTMLToken::Type::EndTag: {
if (ffi.tag_name_id != 0)
token.set_tag_name(interned_rust_tag_name(ffi.tag_name_id));
else
token.set_tag_name(MUST(FlyString::from_utf8(ffi_string_view(ffi.tag_name_ptr, ffi.tag_name_len))));
token.set_self_closing(ffi.self_closing);
for (size_t i = 0; i < ffi.attributes_len; ++i) {
auto const& ffi_attribute = ffi.attributes_ptr[i];
HTMLToken::Attribute attribute;
if (ffi_attribute.name_id != 0)
attribute.local_name = interned_rust_attr_name(ffi_attribute.name_id);
else
attribute.local_name = MUST(FlyString::from_utf8(ffi_string_view(ffi_attribute.name_ptr, ffi_attribute.name_len)));
attribute.value = MUST(String::from_utf8(ffi_string_view(ffi_attribute.value_ptr, ffi_attribute.value_len)));
attribute.name_start_position = { ffi_attribute.name_start_line, ffi_attribute.name_start_column };
attribute.name_end_position = { ffi_attribute.name_end_line, ffi_attribute.name_end_column };
attribute.value_start_position = { ffi_attribute.value_start_line, ffi_attribute.value_start_column };
attribute.value_end_position = { ffi_attribute.value_end_line, ffi_attribute.value_end_column };
token.add_attribute(move(attribute));
LibWeb: Begin work on a spec-compliant HTML parser In order to actually view the web as it is, we're gonna need a proper HTML parser. So let's build one! This patch introduces the Web::HTMLTokenizer class, which currently operates on a StringView input stream where it fetches (ASCII only atm) codepoints and tokenizes acccording to the HTML spec tokenization algo. The tokenizer state machine looks a bit weird but is written in a way that tries to mimic the spec as closely as possible, in order to make development easier and bugs less likely. This initial version is far from finished, but it can parse a trivial document with a DOCTYPE and open/close tags. :^)
2020-05-22 21:46:13 +02:00
}
LibWeb: Replace the HTML tokenizer with Rust Replace the C++ HTML tokenizer with a Rust implementation behind the existing HTMLTokenizer API. Keep the parser-facing integration points for streaming input, insertion points, document.write(), EOF insertion, parser aborts, speculative parser input, and last start tag tracking. The generated FFI handle stays an implementation detail of HTMLTokenizer, so callers keep a single tokenizer class. Preserve duplicate attributes through FFI so C++ token normalization can record the duplicate-attribute signal used by CSP nonce checks. Keep bulk tag-name and attribute scans capped at the active insertion point so streamed parser input is spliced at the right offset. Use generated DAFSA tables for named character references and intern common tag and attribute names to reduce FFI marshalling overhead. This also fixes attribute name source positions, nested old insertion points, and aborted fast-path handling. TestHTMLTokenizer covers duplicate attributes and insertion points in fast tag-name, attribute-name, and quoted-value scans. A CSP text test covers duplicate nonce attributes on parser-created script elements. The tokenizer dump fixtures still match, TestHTMLTokenizer passes, and the full release test-web run passes with 6981 tests and 226 skipped.
2026-05-15 15:13:43 +02:00
token.normalize_attributes();
LibWeb: Complete Rust HTML tree construction Finish the Rust implementation of the spec tree-construction algorithms needed by the LibWeb test suite. Add the remaining table modes, foster parenting, scope helpers, adoption agency handling, ruby/list/form and select cases, frameset state, foreign-content edge cases, and parser host callbacks. Preserve behavior that depends on the C++ DOM integration, including parser-created custom element reactions, fragment quirks mode, arbitrary fragment namespaces, template fragment mode, fragment form ownership, MathML annotation-xml boundaries, contextual fragment scripts, parser script source positions, document.close() parser state, void-element insertion, and duplicate attribute tracking. Add focused tests for the parser edge cases that are easy to regress at the boundary between the Rust tree builder and the C++ DOM host.
2026-05-15 21:56:35 +02:00
if (ffi.had_duplicate_attribute)
token.set_had_duplicate_attribute({});
LibWeb: Replace the HTML tokenizer with Rust Replace the C++ HTML tokenizer with a Rust implementation behind the existing HTMLTokenizer API. Keep the parser-facing integration points for streaming input, insertion points, document.write(), EOF insertion, parser aborts, speculative parser input, and last start tag tracking. The generated FFI handle stays an implementation detail of HTMLTokenizer, so callers keep a single tokenizer class. Preserve duplicate attributes through FFI so C++ token normalization can record the duplicate-attribute signal used by CSP nonce checks. Keep bulk tag-name and attribute scans capped at the active insertion point so streamed parser input is spliced at the right offset. Use generated DAFSA tables for named character references and intern common tag and attribute names to reduce FFI marshalling overhead. This also fixes attribute name source positions, nested old insertion points, and aborted fast-path handling. TestHTMLTokenizer covers duplicate attributes and insertion points in fast tag-name, attribute-name, and quoted-value scans. A CSP text test covers duplicate nonce attributes on parser-created script elements. The tokenizer dump fixtures still match, TestHTMLTokenizer passes, and the full release test-web run passes with 6981 tests and 226 skipped.
2026-05-15 15:13:43 +02:00
break;
LibWeb: Begin work on a spec-compliant HTML parser In order to actually view the web as it is, we're gonna need a proper HTML parser. So let's build one! This patch introduces the Web::HTMLTokenizer class, which currently operates on a StringView input stream where it fetches (ASCII only atm) codepoints and tokenizes acccording to the HTML spec tokenization algo. The tokenizer state machine looks a bit weird but is written in a way that tries to mimic the spec as closely as possible, in order to make development easier and bugs less likely. This initial version is far from finished, but it can parse a trivial document with a DOCTYPE and open/close tags. :^)
2020-05-22 21:46:13 +02:00
}
LibWeb: Replace the HTML tokenizer with Rust Replace the C++ HTML tokenizer with a Rust implementation behind the existing HTMLTokenizer API. Keep the parser-facing integration points for streaming input, insertion points, document.write(), EOF insertion, parser aborts, speculative parser input, and last start tag tracking. The generated FFI handle stays an implementation detail of HTMLTokenizer, so callers keep a single tokenizer class. Preserve duplicate attributes through FFI so C++ token normalization can record the duplicate-attribute signal used by CSP nonce checks. Keep bulk tag-name and attribute scans capped at the active insertion point so streamed parser input is spliced at the right offset. Use generated DAFSA tables for named character references and intern common tag and attribute names to reduce FFI marshalling overhead. This also fixes attribute name source positions, nested old insertion points, and aborted fast-path handling. TestHTMLTokenizer covers duplicate attributes and insertion points in fast tag-name, attribute-name, and quoted-value scans. A CSP text test covers duplicate nonce attributes on parser-created script elements. The tokenizer dump fixtures still match, TestHTMLTokenizer passes, and the full release test-web run passes with 6981 tests and 226 skipped.
2026-05-15 15:13:43 +02:00
case HTMLToken::Type::Comment:
token.set_comment(MUST(String::from_utf8(ffi_string_view(ffi.comment_ptr, ffi.comment_len))));
break;
case HTMLToken::Type::DOCTYPE: {
auto& doctype = token.ensure_doctype_data();
if (!ffi.missing_name) {
doctype.name = MUST(String::from_utf8(ffi_string_view(ffi.doctype_name_ptr, ffi.doctype_name_len)));
doctype.missing_name = false;
LibWeb: Teach more of HTMLTokenizer to stop at the insertion point In particular, input character lookahead now knows how to stop at the insertion point marker if needed. This makes it possible to do amazing things like having document.write() insert doctypes one character at a time.
2024-11-24 10:18:17 +01:00
}
LibWeb: Replace the HTML tokenizer with Rust Replace the C++ HTML tokenizer with a Rust implementation behind the existing HTMLTokenizer API. Keep the parser-facing integration points for streaming input, insertion points, document.write(), EOF insertion, parser aborts, speculative parser input, and last start tag tracking. The generated FFI handle stays an implementation detail of HTMLTokenizer, so callers keep a single tokenizer class. Preserve duplicate attributes through FFI so C++ token normalization can record the duplicate-attribute signal used by CSP nonce checks. Keep bulk tag-name and attribute scans capped at the active insertion point so streamed parser input is spliced at the right offset. Use generated DAFSA tables for named character references and intern common tag and attribute names to reduce FFI marshalling overhead. This also fixes attribute name source positions, nested old insertion points, and aborted fast-path handling. TestHTMLTokenizer covers duplicate attributes and insertion points in fast tag-name, attribute-name, and quoted-value scans. A CSP text test covers duplicate nonce attributes on parser-created script elements. The tokenizer dump fixtures still match, TestHTMLTokenizer passes, and the full release test-web run passes with 6981 tests and 226 skipped.
2026-05-15 15:13:43 +02:00
if (!ffi.missing_public_id) {
doctype.public_identifier = MUST(String::from_utf8(ffi_string_view(ffi.public_id_ptr, ffi.public_id_len)));
doctype.missing_public_identifier = false;
LibWeb: Checking for "DOCTYPE" should be case insensitive in tokenizer
2020-05-25 19:22:23 +02:00
}
LibWeb: Replace the HTML tokenizer with Rust Replace the C++ HTML tokenizer with a Rust implementation behind the existing HTMLTokenizer API. Keep the parser-facing integration points for streaming input, insertion points, document.write(), EOF insertion, parser aborts, speculative parser input, and last start tag tracking. The generated FFI handle stays an implementation detail of HTMLTokenizer, so callers keep a single tokenizer class. Preserve duplicate attributes through FFI so C++ token normalization can record the duplicate-attribute signal used by CSP nonce checks. Keep bulk tag-name and attribute scans capped at the active insertion point so streamed parser input is spliced at the right offset. Use generated DAFSA tables for named character references and intern common tag and attribute names to reduce FFI marshalling overhead. This also fixes attribute name source positions, nested old insertion points, and aborted fast-path handling. TestHTMLTokenizer covers duplicate attributes and insertion points in fast tag-name, attribute-name, and quoted-value scans. A CSP text test covers duplicate nonce attributes on parser-created script elements. The tokenizer dump fixtures still match, TestHTMLTokenizer passes, and the full release test-web run passes with 6981 tests and 226 skipped.
2026-05-15 15:13:43 +02:00
if (!ffi.missing_system_id) {
doctype.system_identifier = MUST(String::from_utf8(ffi_string_view(ffi.system_id_ptr, ffi.system_id_len)));
doctype.missing_system_identifier = false;
}
doctype.force_quirks = ffi.force_quirks;
break;
}
case HTMLToken::Type::EndOfFile:
break;
case HTMLToken::Type::Invalid:
VERIFY_NOT_REACHED();
LibWeb: Begin work on a spec-compliant HTML parser In order to actually view the web as it is, we're gonna need a proper HTML parser. So let's build one! This patch introduces the Web::HTMLTokenizer class, which currently operates on a StringView input stream where it fetches (ASCII only atm) codepoints and tokenizes acccording to the HTML spec tokenization algo. The tokenizer state machine looks a bit weird but is written in a way that tries to mimic the spec as closely as possible, in order to make development easier and bugs less likely. This initial version is far from finished, but it can parse a trivial document with a DOCTYPE and open/close tags. :^)
2020-05-22 21:46:13 +02:00
}
LibWeb: Replace the HTML tokenizer with Rust Replace the C++ HTML tokenizer with a Rust implementation behind the existing HTMLTokenizer API. Keep the parser-facing integration points for streaming input, insertion points, document.write(), EOF insertion, parser aborts, speculative parser input, and last start tag tracking. The generated FFI handle stays an implementation detail of HTMLTokenizer, so callers keep a single tokenizer class. Preserve duplicate attributes through FFI so C++ token normalization can record the duplicate-attribute signal used by CSP nonce checks. Keep bulk tag-name and attribute scans capped at the active insertion point so streamed parser input is spliced at the right offset. Use generated DAFSA tables for named character references and intern common tag and attribute names to reduce FFI marshalling overhead. This also fixes attribute name source positions, nested old insertion points, and aborted fast-path handling. TestHTMLTokenizer covers duplicate attributes and insertion points in fast tag-name, attribute-name, and quoted-value scans. A CSP text test covers duplicate nonce attributes on parser-created script elements. The tokenizer dump fixtures still match, TestHTMLTokenizer passes, and the full release test-web run passes with 6981 tests and 226 skipped.
2026-05-15 15:13:43 +02:00
return token;
LibWeb: Begin work on a spec-compliant HTML parser In order to actually view the web as it is, we're gonna need a proper HTML parser. So let's build one! This patch introduces the Web::HTMLTokenizer class, which currently operates on a StringView input stream where it fetches (ASCII only atm) codepoints and tokenizes acccording to the HTML spec tokenization algo. The tokenizer state machine looks a bit weird but is written in a way that tries to mimic the spec as closely as possible, in order to make development easier and bugs less likely. This initial version is far from finished, but it can parse a trivial document with a DOCTYPE and open/close tags. :^)
2020-05-22 21:46:13 +02:00
}
LibWeb: Throw out decoded UTF-32 data in HTMLTokenizer after parser runs This ends up saving quite a bit of memory on many pages, since UTF-32 uses 4 bytes per code points. As an example, it reduces the footprint on https://gymgrossisten.com/ by 2 MiB.
2025-10-23 21:45:00 +02:00
void HTMLTokenizer::parser_did_run(Badge<HTMLParser>)
{
LibWeb: Replace the HTML tokenizer with Rust Replace the C++ HTML tokenizer with a Rust implementation behind the existing HTMLTokenizer API. Keep the parser-facing integration points for streaming input, insertion points, document.write(), EOF insertion, parser aborts, speculative parser input, and last start tag tracking. The generated FFI handle stays an implementation detail of HTMLTokenizer, so callers keep a single tokenizer class. Preserve duplicate attributes through FFI so C++ token normalization can record the duplicate-attribute signal used by CSP nonce checks. Keep bulk tag-name and attribute scans capped at the active insertion point so streamed parser input is spliced at the right offset. Use generated DAFSA tables for named character references and intern common tag and attribute names to reduce FFI marshalling overhead. This also fixes attribute name source positions, nested old insertion points, and aborted fast-path handling. TestHTMLTokenizer covers duplicate attributes and insertion points in fast tag-name, attribute-name, and quoted-value scans. A CSP text test covers duplicate nonce attributes on parser-created script elements. The tokenizer dump fixtures still match, TestHTMLTokenizer passes, and the full release test-web run passes with 6981 tests and 226 skipped.
2026-05-15 15:13:43 +02:00
rust_html_tokenizer_parser_did_run(m_tokenizer);
LibWeb: Throw out decoded UTF-32 data in HTMLTokenizer after parser runs This ends up saving quite a bit of memory on many pages, since UTF-32 uses 4 bytes per code points. As an example, it reduces the footprint on https://gymgrossisten.com/ by 2 MiB.
2025-10-23 21:45:00 +02:00
}
LibWeb: Implement the speculative HTML parser When the HTML parser blocks on a synchronous external script, run a separate tokenizer over the unparsed input and issue speculative fetches for the resources it finds (script src, link rel=stylesheet|preload, img src), with <base href> tracking and template/foreign-content skipping. Also fills in the previously-stubbed "consume a preloaded resource" algorithm and the document's "map of preloaded resources", so that <link rel="preload"> followed by a matching consumer deduplicates to a single fetch.
2026-04-26 03:21:39 +02:00
String HTMLTokenizer::unparsed_input() const
{
LibWeb: Replace the HTML tokenizer with Rust Replace the C++ HTML tokenizer with a Rust implementation behind the existing HTMLTokenizer API. Keep the parser-facing integration points for streaming input, insertion points, document.write(), EOF insertion, parser aborts, speculative parser input, and last start tag tracking. The generated FFI handle stays an implementation detail of HTMLTokenizer, so callers keep a single tokenizer class. Preserve duplicate attributes through FFI so C++ token normalization can record the duplicate-attribute signal used by CSP nonce checks. Keep bulk tag-name and attribute scans capped at the active insertion point so streamed parser input is spliced at the right offset. Use generated DAFSA tables for named character references and intern common tag and attribute names to reduce FFI marshalling overhead. This also fixes attribute name source positions, nested old insertion points, and aborted fast-path handling. TestHTMLTokenizer covers duplicate attributes and insertion points in fast tag-name, attribute-name, and quoted-value scans. A CSP text test covers duplicate nonce attributes on parser-created script elements. The tokenizer dump fixtures still match, TestHTMLTokenizer passes, and the full release test-web run passes with 6981 tests and 226 skipped.
2026-05-15 15:13:43 +02:00
uint8_t const* ptr = nullptr;
size_t len = 0;
rust_html_tokenizer_unparsed_input(m_tokenizer, &ptr, &len);
return MUST(String::from_utf8(ffi_string_view(ptr, len)));
LibWeb: Implement the speculative HTML parser When the HTML parser blocks on a synchronous external script, run a separate tokenizer over the unparsed input and issue speculative fetches for the resources it finds (script src, link rel=stylesheet|preload, img src), with <base href> tracking and template/foreign-content skipping. Also fills in the previously-stubbed "consume a preloaded resource" algorithm and the document's "map of preloaded resources", so that <link rel="preload"> followed by a matching consumer deduplicates to a single fetch.
2026-04-26 03:21:39 +02:00
}
LibWeb: Track input stream closed state in HTMLTokenizer Add an explicit "input stream closed" flag plus the streaming-input API (append_to_input_stream, close_input_stream, is_input_stream_closed) to let a future incremental driver feed bytes as they arrive. Rewrite should_pause_before_next_input_character so the tokenizer pauses when the buffer is exhausted but more bytes may still arrive, including the case where a chunk ends in CR (CRLF normalization needs one code point of lookahead). Existing call sites are unaffected: the StringView constructor immediately marks the input stream closed, and insert_eof() now also closes the stream so document.close() drives the same exit path.
2026-04-28 18:04:17 +02:00
void HTMLTokenizer::append_to_input_stream(StringView input)
{
LibWeb: Replace the HTML tokenizer with Rust Replace the C++ HTML tokenizer with a Rust implementation behind the existing HTMLTokenizer API. Keep the parser-facing integration points for streaming input, insertion points, document.write(), EOF insertion, parser aborts, speculative parser input, and last start tag tracking. The generated FFI handle stays an implementation detail of HTMLTokenizer, so callers keep a single tokenizer class. Preserve duplicate attributes through FFI so C++ token normalization can record the duplicate-attribute signal used by CSP nonce checks. Keep bulk tag-name and attribute scans capped at the active insertion point so streamed parser input is spliced at the right offset. Use generated DAFSA tables for named character references and intern common tag and attribute names to reduce FFI marshalling overhead. This also fixes attribute name source positions, nested old insertion points, and aborted fast-path handling. TestHTMLTokenizer covers duplicate attributes and insertion points in fast tag-name, attribute-name, and quoted-value scans. A CSP text test covers duplicate nonce attributes on parser-created script elements. The tokenizer dump fixtures still match, TestHTMLTokenizer passes, and the full release test-web run passes with 6981 tests and 226 skipped.
2026-05-15 15:13:43 +02:00
if (input.is_empty())
return;
auto utf8_input = MUST(String::from_utf8(input));
auto code_points = code_points_from_string(utf8_input);
rust_html_tokenizer_append_input(m_tokenizer, code_points.data(), code_points.size());
LibWeb: Track input stream closed state in HTMLTokenizer Add an explicit "input stream closed" flag plus the streaming-input API (append_to_input_stream, close_input_stream, is_input_stream_closed) to let a future incremental driver feed bytes as they arrive. Rewrite should_pause_before_next_input_character so the tokenizer pauses when the buffer is exhausted but more bytes may still arrive, including the case where a chunk ends in CR (CRLF normalization needs one code point of lookahead). Existing call sites are unaffected: the StringView constructor immediately marks the input stream closed, and insert_eof() now also closes the stream so document.close() drives the same exit path.
2026-04-28 18:04:17 +02:00
}
void HTMLTokenizer::close_input_stream()
{
m_input_stream_closed = true;
LibWeb: Replace the HTML tokenizer with Rust Replace the C++ HTML tokenizer with a Rust implementation behind the existing HTMLTokenizer API. Keep the parser-facing integration points for streaming input, insertion points, document.write(), EOF insertion, parser aborts, speculative parser input, and last start tag tracking. The generated FFI handle stays an implementation detail of HTMLTokenizer, so callers keep a single tokenizer class. Preserve duplicate attributes through FFI so C++ token normalization can record the duplicate-attribute signal used by CSP nonce checks. Keep bulk tag-name and attribute scans capped at the active insertion point so streamed parser input is spliced at the right offset. Use generated DAFSA tables for named character references and intern common tag and attribute names to reduce FFI marshalling overhead. This also fixes attribute name source positions, nested old insertion points, and aborted fast-path handling. TestHTMLTokenizer covers duplicate attributes and insertion points in fast tag-name, attribute-name, and quoted-value scans. A CSP text test covers duplicate nonce attributes on parser-created script elements. The tokenizer dump fixtures still match, TestHTMLTokenizer passes, and the full release test-web run passes with 6981 tests and 226 skipped.
2026-05-15 15:13:43 +02:00
rust_html_tokenizer_set_input_stream_closed(m_tokenizer, true);
LibWeb: Track input stream closed state in HTMLTokenizer Add an explicit "input stream closed" flag plus the streaming-input API (append_to_input_stream, close_input_stream, is_input_stream_closed) to let a future incremental driver feed bytes as they arrive. Rewrite should_pause_before_next_input_character so the tokenizer pauses when the buffer is exhausted but more bytes may still arrive, including the case where a chunk ends in CR (CRLF normalization needs one code point of lookahead). Existing call sites are unaffected: the StringView constructor immediately marks the input stream closed, and insert_eof() now also closes the stream so document.close() drives the same exit path.
2026-04-28 18:04:17 +02:00
}
LibWeb: Make Document::run_the_document_write_steps take a StringView Which flows on down into HTMLTokenizer::insert_input_at_insertion_point.
2023-09-12 23:16:10 +12:00
void HTMLTokenizer::insert_input_at_insertion_point(StringView input)
LibWeb: Add basic support for dynamic markup insertion This implements basic support for dynamic markup insertion, adding * Document::open() * Document::write(Vector<String> const&) * Document::writeln(Vector<String> const&) * Document::close() The HTMLParser is modified to make it possible to create a script-created parser which initially only contains a HTMLTokenizer without any data. Aditionally the HTMLParser::run method gains an overload which does not modify the Document and does not run HTMLParser::the_end() so that we can reenter the parser at a later time. Furthermore all FIXMEs that consern the insertion point are implemented wich is defined in the HTMLTokenizer. Additionally the following member-variables of the HTMLParser are now exposed by getter funcions: * m_tokenizer * m_aborted * m_script_nesting_level The HTMLTokenizer is modified so that it contains an insertion point which keeps track of where the next input from the Document::write functions will be inserted. The insertion point is implemented as the charakter offset into m_decoded_input and a boolean describing if the insertion point is defined. Functions to update, check and {re}store the insertion point are also added. The function HTMLTokenizer::insert_eof is added to tell a script-created parser that document::close was called and HTMLParser::the_end() should be called. Lastly an explicit default constructor is added to HTMLTokenizer to create a empty HTMLTokenizer into which data can be inserted.
2022-02-19 15:58:21 +01:00
{
LibWeb: Replace the HTML tokenizer with Rust Replace the C++ HTML tokenizer with a Rust implementation behind the existing HTMLTokenizer API. Keep the parser-facing integration points for streaming input, insertion points, document.write(), EOF insertion, parser aborts, speculative parser input, and last start tag tracking. The generated FFI handle stays an implementation detail of HTMLTokenizer, so callers keep a single tokenizer class. Preserve duplicate attributes through FFI so C++ token normalization can record the duplicate-attribute signal used by CSP nonce checks. Keep bulk tag-name and attribute scans capped at the active insertion point so streamed parser input is spliced at the right offset. Use generated DAFSA tables for named character references and intern common tag and attribute names to reduce FFI marshalling overhead. This also fixes attribute name source positions, nested old insertion points, and aborted fast-path handling. TestHTMLTokenizer covers duplicate attributes and insertion points in fast tag-name, attribute-name, and quoted-value scans. A CSP text test covers duplicate nonce attributes on parser-created script elements. The tokenizer dump fixtures still match, TestHTMLTokenizer passes, and the full release test-web run passes with 6981 tests and 226 skipped.
2026-05-15 15:13:43 +02:00
auto utf8_input = MUST(String::from_utf8(input));
auto code_points = code_points_from_string(utf8_input);
rust_html_tokenizer_insert_input(m_tokenizer, code_points.data(), code_points.size());
LibWeb: Add basic support for dynamic markup insertion This implements basic support for dynamic markup insertion, adding * Document::open() * Document::write(Vector<String> const&) * Document::writeln(Vector<String> const&) * Document::close() The HTMLParser is modified to make it possible to create a script-created parser which initially only contains a HTMLTokenizer without any data. Aditionally the HTMLParser::run method gains an overload which does not modify the Document and does not run HTMLParser::the_end() so that we can reenter the parser at a later time. Furthermore all FIXMEs that consern the insertion point are implemented wich is defined in the HTMLTokenizer. Additionally the following member-variables of the HTMLParser are now exposed by getter funcions: * m_tokenizer * m_aborted * m_script_nesting_level The HTMLTokenizer is modified so that it contains an insertion point which keeps track of where the next input from the Document::write functions will be inserted. The insertion point is implemented as the charakter offset into m_decoded_input and a boolean describing if the insertion point is defined. Functions to update, check and {re}store the insertion point are also added. The function HTMLTokenizer::insert_eof is added to tell a script-created parser that document::close was called and HTMLParser::the_end() should be called. Lastly an explicit default constructor is added to HTMLTokenizer to create a empty HTMLTokenizer into which data can be inserted.
2022-02-19 15:58:21 +01:00
}
void HTMLTokenizer::insert_eof()
{
LibWeb: Track input stream closed state in HTMLTokenizer Add an explicit "input stream closed" flag plus the streaming-input API (append_to_input_stream, close_input_stream, is_input_stream_closed) to let a future incremental driver feed bytes as they arrive. Rewrite should_pause_before_next_input_character so the tokenizer pauses when the buffer is exhausted but more bytes may still arrive, including the case where a chunk ends in CR (CRLF normalization needs one code point of lookahead). Existing call sites are unaffected: the StringView constructor immediately marks the input stream closed, and insert_eof() now also closes the stream so document.close() drives the same exit path.
2026-04-28 18:04:17 +02:00
close_input_stream();
LibWeb: Replace the HTML tokenizer with Rust Replace the C++ HTML tokenizer with a Rust implementation behind the existing HTMLTokenizer API. Keep the parser-facing integration points for streaming input, insertion points, document.write(), EOF insertion, parser aborts, speculative parser input, and last start tag tracking. The generated FFI handle stays an implementation detail of HTMLTokenizer, so callers keep a single tokenizer class. Preserve duplicate attributes through FFI so C++ token normalization can record the duplicate-attribute signal used by CSP nonce checks. Keep bulk tag-name and attribute scans capped at the active insertion point so streamed parser input is spliced at the right offset. Use generated DAFSA tables for named character references and intern common tag and attribute names to reduce FFI marshalling overhead. This also fixes attribute name source positions, nested old insertion points, and aborted fast-path handling. TestHTMLTokenizer covers duplicate attributes and insertion points in fast tag-name, attribute-name, and quoted-value scans. A CSP text test covers duplicate nonce attributes on parser-created script elements. The tokenizer dump fixtures still match, TestHTMLTokenizer passes, and the full release test-web run passes with 6981 tests and 226 skipped.
2026-05-15 15:13:43 +02:00
rust_html_tokenizer_insert_eof(m_tokenizer);
LibWeb: Add basic support for dynamic markup insertion This implements basic support for dynamic markup insertion, adding * Document::open() * Document::write(Vector<String> const&) * Document::writeln(Vector<String> const&) * Document::close() The HTMLParser is modified to make it possible to create a script-created parser which initially only contains a HTMLTokenizer without any data. Aditionally the HTMLParser::run method gains an overload which does not modify the Document and does not run HTMLParser::the_end() so that we can reenter the parser at a later time. Furthermore all FIXMEs that consern the insertion point are implemented wich is defined in the HTMLTokenizer. Additionally the following member-variables of the HTMLParser are now exposed by getter funcions: * m_tokenizer * m_aborted * m_script_nesting_level The HTMLTokenizer is modified so that it contains an insertion point which keeps track of where the next input from the Document::write functions will be inserted. The insertion point is implemented as the charakter offset into m_decoded_input and a boolean describing if the insertion point is defined. Functions to update, check and {re}store the insertion point are also added. The function HTMLTokenizer::insert_eof is added to tell a script-created parser that document::close was called and HTMLParser::the_end() should be called. Lastly an explicit default constructor is added to HTMLTokenizer to create a empty HTMLTokenizer into which data can be inserted.
2022-02-19 15:58:21 +01:00
}
LibWeb: Replace the HTML tokenizer with Rust Replace the C++ HTML tokenizer with a Rust implementation behind the existing HTMLTokenizer API. Keep the parser-facing integration points for streaming input, insertion points, document.write(), EOF insertion, parser aborts, speculative parser input, and last start tag tracking. The generated FFI handle stays an implementation detail of HTMLTokenizer, so callers keep a single tokenizer class. Preserve duplicate attributes through FFI so C++ token normalization can record the duplicate-attribute signal used by CSP nonce checks. Keep bulk tag-name and attribute scans capped at the active insertion point so streamed parser input is spliced at the right offset. Use generated DAFSA tables for named character references and intern common tag and attribute names to reduce FFI marshalling overhead. This also fixes attribute name source positions, nested old insertion points, and aborted fast-path handling. TestHTMLTokenizer covers duplicate attributes and insertion points in fast tag-name, attribute-name, and quoted-value scans. A CSP text test covers duplicate nonce attributes on parser-created script elements. The tokenizer dump fixtures still match, TestHTMLTokenizer passes, and the full release test-web run passes with 6981 tests and 226 skipped.
2026-05-15 15:13:43 +02:00
bool HTMLTokenizer::is_insertion_point_defined() const
LibWeb: More work on the HTML parser and tokenizer The parser can now switch the state of the tokenizer! Very webby. :^)
2020-05-24 20:24:43 +02:00
{
LibWeb: Replace the HTML tokenizer with Rust Replace the C++ HTML tokenizer with a Rust implementation behind the existing HTMLTokenizer API. Keep the parser-facing integration points for streaming input, insertion points, document.write(), EOF insertion, parser aborts, speculative parser input, and last start tag tracking. The generated FFI handle stays an implementation detail of HTMLTokenizer, so callers keep a single tokenizer class. Preserve duplicate attributes through FFI so C++ token normalization can record the duplicate-attribute signal used by CSP nonce checks. Keep bulk tag-name and attribute scans capped at the active insertion point so streamed parser input is spliced at the right offset. Use generated DAFSA tables for named character references and intern common tag and attribute names to reduce FFI marshalling overhead. This also fixes attribute name source positions, nested old insertion points, and aborted fast-path handling. TestHTMLTokenizer covers duplicate attributes and insertion points in fast tag-name, attribute-name, and quoted-value scans. A CSP text test covers duplicate nonce attributes on parser-created script elements. The tokenizer dump fixtures still match, TestHTMLTokenizer passes, and the full release test-web run passes with 6981 tests and 226 skipped.
2026-05-15 15:13:43 +02:00
return rust_html_tokenizer_is_insertion_point_defined(m_tokenizer);
LibWeb: More work on the HTML parser and tokenizer The parser can now switch the state of the tokenizer! Very webby. :^)
2020-05-24 20:24:43 +02:00
}
LibWeb: Replace the HTML tokenizer with Rust Replace the C++ HTML tokenizer with a Rust implementation behind the existing HTMLTokenizer API. Keep the parser-facing integration points for streaming input, insertion points, document.write(), EOF insertion, parser aborts, speculative parser input, and last start tag tracking. The generated FFI handle stays an implementation detail of HTMLTokenizer, so callers keep a single tokenizer class. Preserve duplicate attributes through FFI so C++ token normalization can record the duplicate-attribute signal used by CSP nonce checks. Keep bulk tag-name and attribute scans capped at the active insertion point so streamed parser input is spliced at the right offset. Use generated DAFSA tables for named character references and intern common tag and attribute names to reduce FFI marshalling overhead. This also fixes attribute name source positions, nested old insertion points, and aborted fast-path handling. TestHTMLTokenizer covers duplicate attributes and insertion points in fast tag-name, attribute-name, and quoted-value scans. A CSP text test covers duplicate nonce attributes on parser-created script elements. The tokenizer dump fixtures still match, TestHTMLTokenizer passes, and the full release test-web run passes with 6981 tests and 226 skipped.
2026-05-15 15:13:43 +02:00
bool HTMLTokenizer::is_insertion_point_reached()
LibWeb: More work on the HTML parser and tokenizer The parser can now switch the state of the tokenizer! Very webby. :^)
2020-05-24 20:24:43 +02:00
{
LibWeb: Replace the HTML tokenizer with Rust Replace the C++ HTML tokenizer with a Rust implementation behind the existing HTMLTokenizer API. Keep the parser-facing integration points for streaming input, insertion points, document.write(), EOF insertion, parser aborts, speculative parser input, and last start tag tracking. The generated FFI handle stays an implementation detail of HTMLTokenizer, so callers keep a single tokenizer class. Preserve duplicate attributes through FFI so C++ token normalization can record the duplicate-attribute signal used by CSP nonce checks. Keep bulk tag-name and attribute scans capped at the active insertion point so streamed parser input is spliced at the right offset. Use generated DAFSA tables for named character references and intern common tag and attribute names to reduce FFI marshalling overhead. This also fixes attribute name source positions, nested old insertion points, and aborted fast-path handling. TestHTMLTokenizer covers duplicate attributes and insertion points in fast tag-name, attribute-name, and quoted-value scans. A CSP text test covers duplicate nonce attributes on parser-created script elements. The tokenizer dump fixtures still match, TestHTMLTokenizer passes, and the full release test-web run passes with 6981 tests and 226 skipped.
2026-05-15 15:13:43 +02:00
return rust_html_tokenizer_is_insertion_point_reached(m_tokenizer);
}
LibWeb: Set consistent positions for the start and end of HTML tags To illustrate the previous behavior, consider these tags and their start and end positions (shown inclusively below): Start tag: End tag: <span> </span> ^ start ^ start ^end ^end The start position of a tag is the first ASCII-alpha code point after the opening brace. The start position of a close tag is the slash just before the first ASCII-alpha code point. And the end position of both is the closing brace. So the opening brace is not included in the emitted tag, but the closing brace is. And the end tag including the slash is an oddity that had to be worked around in its only use case (syntax highlighting). We now consistently exclude the braces from the emitted tag, and also exclude the slash from the end tag, so that it does not need to be accounted for in syntax highlighting. That is, we now have: Start tag: End tag: <span> </span> ^ start ^ start ^end ^end The tokenizer unit test has been extended to test these positions.
2023-08-24 17:01:19 -04:00
LibWeb: Replace the HTML tokenizer with Rust Replace the C++ HTML tokenizer with a Rust implementation behind the existing HTMLTokenizer API. Keep the parser-facing integration points for streaming input, insertion points, document.write(), EOF insertion, parser aborts, speculative parser input, and last start tag tracking. The generated FFI handle stays an implementation detail of HTMLTokenizer, so callers keep a single tokenizer class. Preserve duplicate attributes through FFI so C++ token normalization can record the duplicate-attribute signal used by CSP nonce checks. Keep bulk tag-name and attribute scans capped at the active insertion point so streamed parser input is spliced at the right offset. Use generated DAFSA tables for named character references and intern common tag and attribute names to reduce FFI marshalling overhead. This also fixes attribute name source positions, nested old insertion points, and aborted fast-path handling. TestHTMLTokenizer covers duplicate attributes and insertion points in fast tag-name, attribute-name, and quoted-value scans. A CSP text test covers duplicate nonce attributes on parser-created script elements. The tokenizer dump fixtures still match, TestHTMLTokenizer passes, and the full release test-web run passes with 6981 tests and 226 skipped.
2026-05-15 15:13:43 +02:00
void HTMLTokenizer::undefine_insertion_point()
{
rust_html_tokenizer_undefine_insertion_point(m_tokenizer);
}
LibWeb: Deduplicate attributes when emitting start and end tags The HTML tokenizer specification says that we're supposed to do this when leaving the Attribute name or when emitting the token, as appropriate. Hopefully 'as appropriate' can mean only when emitting the token, as that's the easiest place to insert this logic without complicating the tokenizer any more.
2024-09-30 17:52:30 -06:00
LibWeb: Replace the HTML tokenizer with Rust Replace the C++ HTML tokenizer with a Rust implementation behind the existing HTMLTokenizer API. Keep the parser-facing integration points for streaming input, insertion points, document.write(), EOF insertion, parser aborts, speculative parser input, and last start tag tracking. The generated FFI handle stays an implementation detail of HTMLTokenizer, so callers keep a single tokenizer class. Preserve duplicate attributes through FFI so C++ token normalization can record the duplicate-attribute signal used by CSP nonce checks. Keep bulk tag-name and attribute scans capped at the active insertion point so streamed parser input is spliced at the right offset. Use generated DAFSA tables for named character references and intern common tag and attribute names to reduce FFI marshalling overhead. This also fixes attribute name source positions, nested old insertion points, and aborted fast-path handling. TestHTMLTokenizer covers duplicate attributes and insertion points in fast tag-name, attribute-name, and quoted-value scans. A CSP text test covers duplicate nonce attributes on parser-created script elements. The tokenizer dump fixtures still match, TestHTMLTokenizer passes, and the full release test-web run passes with 6981 tests and 226 skipped.
2026-05-15 15:13:43 +02:00
void HTMLTokenizer::store_insertion_point()
{
rust_html_tokenizer_store_insertion_point(m_tokenizer);
LibWeb: More work on the HTML parser and tokenizer The parser can now switch the state of the tokenizer! Very webby. :^)
2020-05-24 20:24:43 +02:00
}
LibWeb: Replace the HTML tokenizer with Rust Replace the C++ HTML tokenizer with a Rust implementation behind the existing HTMLTokenizer API. Keep the parser-facing integration points for streaming input, insertion points, document.write(), EOF insertion, parser aborts, speculative parser input, and last start tag tracking. The generated FFI handle stays an implementation detail of HTMLTokenizer, so callers keep a single tokenizer class. Preserve duplicate attributes through FFI so C++ token normalization can record the duplicate-attribute signal used by CSP nonce checks. Keep bulk tag-name and attribute scans capped at the active insertion point so streamed parser input is spliced at the right offset. Use generated DAFSA tables for named character references and intern common tag and attribute names to reduce FFI marshalling overhead. This also fixes attribute name source positions, nested old insertion points, and aborted fast-path handling. TestHTMLTokenizer covers duplicate attributes and insertion points in fast tag-name, attribute-name, and quoted-value scans. A CSP text test covers duplicate nonce attributes on parser-created script elements. The tokenizer dump fixtures still match, TestHTMLTokenizer passes, and the full release test-web run passes with 6981 tests and 226 skipped.
2026-05-15 15:13:43 +02:00
void HTMLTokenizer::restore_insertion_point()
LibWeb: More work on the HTML parser and tokenizer The parser can now switch the state of the tokenizer! Very webby. :^)
2020-05-24 20:24:43 +02:00
{
LibWeb: Replace the HTML tokenizer with Rust Replace the C++ HTML tokenizer with a Rust implementation behind the existing HTMLTokenizer API. Keep the parser-facing integration points for streaming input, insertion points, document.write(), EOF insertion, parser aborts, speculative parser input, and last start tag tracking. The generated FFI handle stays an implementation detail of HTMLTokenizer, so callers keep a single tokenizer class. Preserve duplicate attributes through FFI so C++ token normalization can record the duplicate-attribute signal used by CSP nonce checks. Keep bulk tag-name and attribute scans capped at the active insertion point so streamed parser input is spliced at the right offset. Use generated DAFSA tables for named character references and intern common tag and attribute names to reduce FFI marshalling overhead. This also fixes attribute name source positions, nested old insertion points, and aborted fast-path handling. TestHTMLTokenizer covers duplicate attributes and insertion points in fast tag-name, attribute-name, and quoted-value scans. A CSP text test covers duplicate nonce attributes on parser-created script elements. The tokenizer dump fixtures still match, TestHTMLTokenizer passes, and the full release test-web run passes with 6981 tests and 226 skipped.
2026-05-15 15:13:43 +02:00
rust_html_tokenizer_restore_insertion_point(m_tokenizer);
LibWeb: More work on the HTML parser and tokenizer The parser can now switch the state of the tokenizer! Very webby. :^)
2020-05-24 20:24:43 +02:00
}
LibWeb: Implement partial support for numeric character references
2020-05-27 16:16:23 +02:00
LibWeb: Replace the HTML tokenizer with Rust Replace the C++ HTML tokenizer with a Rust implementation behind the existing HTMLTokenizer API. Keep the parser-facing integration points for streaming input, insertion points, document.write(), EOF insertion, parser aborts, speculative parser input, and last start tag tracking. The generated FFI handle stays an implementation detail of HTMLTokenizer, so callers keep a single tokenizer class. Preserve duplicate attributes through FFI so C++ token normalization can record the duplicate-attribute signal used by CSP nonce checks. Keep bulk tag-name and attribute scans capped at the active insertion point so streamed parser input is spliced at the right offset. Use generated DAFSA tables for named character references and intern common tag and attribute names to reduce FFI marshalling overhead. This also fixes attribute name source positions, nested old insertion points, and aborted fast-path handling. TestHTMLTokenizer covers duplicate attributes and insertion points in fast tag-name, attribute-name, and quoted-value scans. A CSP text test covers duplicate nonce attributes on parser-created script elements. The tokenizer dump fixtures still match, TestHTMLTokenizer passes, and the full release test-web run passes with 6981 tests and 226 skipped.
2026-05-15 15:13:43 +02:00
void HTMLTokenizer::update_insertion_point()
LibWeb: Implement partial support for numeric character references
2020-05-27 16:16:23 +02:00
{
LibWeb: Replace the HTML tokenizer with Rust Replace the C++ HTML tokenizer with a Rust implementation behind the existing HTMLTokenizer API. Keep the parser-facing integration points for streaming input, insertion points, document.write(), EOF insertion, parser aborts, speculative parser input, and last start tag tracking. The generated FFI handle stays an implementation detail of HTMLTokenizer, so callers keep a single tokenizer class. Preserve duplicate attributes through FFI so C++ token normalization can record the duplicate-attribute signal used by CSP nonce checks. Keep bulk tag-name and attribute scans capped at the active insertion point so streamed parser input is spliced at the right offset. Use generated DAFSA tables for named character references and intern common tag and attribute names to reduce FFI marshalling overhead. This also fixes attribute name source positions, nested old insertion points, and aborted fast-path handling. TestHTMLTokenizer covers duplicate attributes and insertion points in fast tag-name, attribute-name, and quoted-value scans. A CSP text test covers duplicate nonce attributes on parser-created script elements. The tokenizer dump fixtures still match, TestHTMLTokenizer passes, and the full release test-web run passes with 6981 tests and 226 skipped.
2026-05-15 15:13:43 +02:00
rust_html_tokenizer_update_insertion_point(m_tokenizer);
LibWeb: Implement partial support for numeric character references
2020-05-27 16:16:23 +02:00
}
LibWeb: Replace the HTML tokenizer with Rust Replace the C++ HTML tokenizer with a Rust implementation behind the existing HTMLTokenizer API. Keep the parser-facing integration points for streaming input, insertion points, document.write(), EOF insertion, parser aborts, speculative parser input, and last start tag tracking. The generated FFI handle stays an implementation detail of HTMLTokenizer, so callers keep a single tokenizer class. Preserve duplicate attributes through FFI so C++ token normalization can record the duplicate-attribute signal used by CSP nonce checks. Keep bulk tag-name and attribute scans capped at the active insertion point so streamed parser input is spliced at the right offset. Use generated DAFSA tables for named character references and intern common tag and attribute names to reduce FFI marshalling overhead. This also fixes attribute name source positions, nested old insertion points, and aborted fast-path handling. TestHTMLTokenizer covers duplicate attributes and insertion points in fast tag-name, attribute-name, and quoted-value scans. A CSP text test covers duplicate nonce attributes on parser-created script elements. The tokenizer dump fixtures still match, TestHTMLTokenizer passes, and the full release test-web run passes with 6981 tests and 226 skipped.
2026-05-15 15:13:43 +02:00
void HTMLTokenizer::abort()
LibWeb: Add position tracking information to HTML tokens
2021-05-20 23:11:41 +04:30
{
LibWeb: Replace the HTML tokenizer with Rust Replace the C++ HTML tokenizer with a Rust implementation behind the existing HTMLTokenizer API. Keep the parser-facing integration points for streaming input, insertion points, document.write(), EOF insertion, parser aborts, speculative parser input, and last start tag tracking. The generated FFI handle stays an implementation detail of HTMLTokenizer, so callers keep a single tokenizer class. Preserve duplicate attributes through FFI so C++ token normalization can record the duplicate-attribute signal used by CSP nonce checks. Keep bulk tag-name and attribute scans capped at the active insertion point so streamed parser input is spliced at the right offset. Use generated DAFSA tables for named character references and intern common tag and attribute names to reduce FFI marshalling overhead. This also fixes attribute name source positions, nested old insertion points, and aborted fast-path handling. TestHTMLTokenizer covers duplicate attributes and insertion points in fast tag-name, attribute-name, and quoted-value scans. A CSP text test covers duplicate nonce attributes on parser-created script elements. The tokenizer dump fixtures still match, TestHTMLTokenizer passes, and the full release test-web run passes with 6981 tests and 226 skipped.
2026-05-15 15:13:43 +02:00
rust_html_tokenizer_abort(m_tokenizer);
}
void HTMLTokenizer::switch_to(State new_state)
LibWeb: Remove StringBuilders from HTMLToken::m_doctype
2021-05-23 08:20:03 +02:00
{
LibWeb: Replace the HTML tokenizer with Rust Replace the C++ HTML tokenizer with a Rust implementation behind the existing HTMLTokenizer API. Keep the parser-facing integration points for streaming input, insertion points, document.write(), EOF insertion, parser aborts, speculative parser input, and last start tag tracking. The generated FFI handle stays an implementation detail of HTMLTokenizer, so callers keep a single tokenizer class. Preserve duplicate attributes through FFI so C++ token normalization can record the duplicate-attribute signal used by CSP nonce checks. Keep bulk tag-name and attribute scans capped at the active insertion point so streamed parser input is spliced at the right offset. Use generated DAFSA tables for named character references and intern common tag and attribute names to reduce FFI marshalling overhead. This also fixes attribute name source positions, nested old insertion points, and aborted fast-path handling. TestHTMLTokenizer covers duplicate attributes and insertion points in fast tag-name, attribute-name, and quoted-value scans. A CSP text test covers duplicate nonce attributes on parser-created script elements. The tokenizer dump fixtures still match, TestHTMLTokenizer passes, and the full release test-web run passes with 6981 tests and 226 skipped.
2026-05-15 15:13:43 +02:00
dbgln_if(TOKENIZER_TRACE_DEBUG, "[{}] Switch to {}", state_name(m_state), state_name(new_state));
m_state = new_state;
rust_html_tokenizer_switch_state(m_tokenizer, static_cast<uint8_t>(new_state));
LibWeb: Remove StringBuilders from HTMLToken::m_doctype
2021-05-23 08:20:03 +02:00
}
LibWeb: Begin work on a spec-compliant HTML parser In order to actually view the web as it is, we're gonna need a proper HTML parser. So let's build one! This patch introduces the Web::HTMLTokenizer class, which currently operates on a StringView input stream where it fetches (ASCII only atm) codepoints and tokenizes acccording to the HTML spec tokenization algo. The tokenizer state machine looks a bit weird but is written in a way that tries to mimic the spec as closely as possible, in order to make development easier and bugs less likely. This initial version is far from finished, but it can parse a trivial document with a DOCTYPE and open/close tags. :^)
2020-05-22 21:46:13 +02:00
}
Reference in a new issue Copy permalink