Stowage/ladybird
2
0
Fork 0
mirror of https://github.com/LadybirdBrowser/ladybird.git synced 2025-10-20 08:03:21 +00:00
Code Issues Projects Releases Packages Wiki Activity
ladybird/Libraries/LibWeb/HTML/Parser/HTMLTokenizer.cpp

2994 lines
116 KiB
C++
Raw Normal View History

LibWeb: Begin work on a spec-compliant HTML parser In order to actually view the web as it is, we're gonna need a proper HTML parser. So let's build one! This patch introduces the Web::HTMLTokenizer class, which currently operates on a StringView input stream where it fetches (ASCII only atm) codepoints and tokenizes acccording to the HTML spec tokenization algo. The tokenizer state machine looks a bit weird but is written in a way that tries to mimic the spec as closely as possible, in order to make development easier and bugs less likely. This initial version is far from finished, but it can parse a trivial document with a DOCTYPE and open/close tags. :^)
2020-05-22 21:46:13 +02:00
/*
Meta: Update my e-mail address everywhere
2024-10-04 13:19:50 +02:00
* Copyright (c) 2020, Andreas Kling <andreas@ladybird.org>
LibWeb: Implement state switch for "[CDATA[" in HTML parser
2022-02-15 18:52:45 +00:00
* Copyright (c) 2022, Linus Groh <linusg@serenityos.org>
LibWeb: Begin work on a spec-compliant HTML parser In order to actually view the web as it is, we're gonna need a proper HTML parser. So let's build one! This patch introduces the Web::HTMLTokenizer class, which currently operates on a StringView input stream where it fetches (ASCII only atm) codepoints and tokenizes acccording to the HTML spec tokenization algo. The tokenizer state machine looks a bit weird but is written in a way that tries to mimic the spec as closely as possible, in order to make development easier and bugs less likely. This initial version is far from finished, but it can parse a trivial document with a DOCTYPE and open/close tags. :^)
2020-05-22 21:46:13 +02:00
*
Everything: Move to SPDX license identifiers in all files. SPDX License Identifiers are a more compact / standardized way of representing file license information. See: https://spdx.dev/resources/use/#identifiers This was done with the `ambr` search and replace tool. ambr --no-parent-ignore --key-from-file --rep-from-file key.txt rep.txt *
2021-04-22 01:24:48 -07:00
* SPDX-License-Identifier: BSD-2-Clause
LibWeb: Begin work on a spec-compliant HTML parser In order to actually view the web as it is, we're gonna need a proper HTML parser. So let's build one! This patch introduces the Web::HTMLTokenizer class, which currently operates on a StringView input stream where it fetches (ASCII only atm) codepoints and tokenizes acccording to the HTML spec tokenization algo. The tokenizer state machine looks a bit weird but is written in a way that tries to mimic the spec as closely as possible, in order to make development easier and bugs less likely. This initial version is far from finished, but it can parse a trivial document with a DOCTYPE and open/close tags. :^)
2020-05-22 21:46:13 +02:00
*/
Everywhere: Replace ctype.h to avoid narrowing conversions This replaces ctype.h with CharacterType.h everywhere I could find issues with narrowing conversions. While using it will probably make sense almost everywhere in the future, the most critical places should have been addressed.
2021-06-01 21:18:08 +02:00
#include <AK/CharacterTypes.h>
Everywhere: Replace a bundle of dbg with dbgln. These changes are arbitrarily divided into multiple commits to make it easier to find potentially introduced bugs with git bisect.
2021-01-17 16:57:17 +01:00
#include <AK/Debug.h>
LibWeb: Don't include Layout/Node.h from DOM/Element.h This required moving the CSS::StyleProperty destruct out of line.
2023-05-08 07:51:03 +02:00
#include <AK/GenericShorthands.h>
LibWeb: Utilize SourceLocation for HTMLTokenizer logging
2021-04-24 20:22:30 -07:00
#include <AK/SourceLocation.h>
LibWeb: Plumb content encoding into the new HTML parser We still don't handle non-ASCII input correctly, but at least now we'll convert e.g ISO-8859-1 to UTF-8 before starting to tokenize. This patch also makes "view source" work with the new parser. :^)
2020-05-28 12:35:19 +02:00
#include <LibTextCodec/Decoder.h>
LibWeb: Move the HTML parser into HTML/Parser/
2020-07-28 19:18:23 +02:00
#include <LibWeb/HTML/Parser/Entities.h>
LibWeb: Implement state switch for "[CDATA[" in HTML parser
2022-02-15 18:52:45 +00:00
#include <LibWeb/HTML/Parser/HTMLParser.h>
LibWeb: Move the HTML parser into HTML/Parser/
2020-07-28 19:18:23 +02:00
#include <LibWeb/HTML/Parser/HTMLToken.h>
#include <LibWeb/HTML/Parser/HTMLTokenizer.h>
LibWeb: Implement state switch for "[CDATA[" in HTML parser
2022-02-15 18:52:45 +00:00
#include <LibWeb/Namespace.h>
LibWeb: In the HTML tokenizer, pretty up ON_WHITESPACE a tiny bit No behavior change.
2020-10-09 11:25:10 -04:00
#include <string.h>
LibWeb: Begin work on a spec-compliant HTML parser In order to actually view the web as it is, we're gonna need a proper HTML parser. So let's build one! This patch introduces the Web::HTMLTokenizer class, which currently operates on a StringView input stream where it fetches (ASCII only atm) codepoints and tokenizes acccording to the HTML spec tokenization algo. The tokenizer state machine looks a bit weird but is written in a way that tries to mimic the spec as closely as possible, in order to make development easier and bugs less likely. This initial version is far from finished, but it can parse a trivial document with a DOCTYPE and open/close tags. :^)
2020-05-22 21:46:13 +02:00
LibWeb: Move HTML classes into the Web::HTML namespace
2020-07-28 18:20:36 +02:00
namespace Web::HTML {
LibWeb: Start building the tree building part of the new HTML parser This patch adds a new HTMLDocumentParser class. It keeps a tokenizer object internally and feeds itself with one token at a time from it. The names and idioms in this class are expressed as closely to the actual HTML parsing spec as possible, to make development as easy and bug free as possible. :^) This is going to become pretty large, but it's pretty cool!
2020-05-24 00:14:23 +02:00
#pragma GCC diagnostic ignored "-Wunused-label"
LibWeb: Implement partial support for numeric character references
2020-05-27 16:16:23 +02:00
#define CONSUME_NEXT_INPUT_CHARACTER \
LibWeb: Teach more of HTMLTokenizer to stop at the insertion point In particular, input character lookahead now knows how to stop at the insertion point marker if needed. This makes it possible to do amazing things like having document.write() insert doctypes one character at a time.
2024-11-24 10:18:17 +01:00
current_input_character = next_code_point(stop_at_insertion_point);
LibWeb: Implement partial support for numeric character references
2020-05-27 16:16:23 +02:00
LibWeb: Remove StringBuilders from HTMLToken::m_doctype
2021-05-23 08:20:03 +02:00
#define SWITCH_TO(new_state) \
do { \
VERIFY(m_current_builder.is_empty()); \
SWITCH_TO_WITH_UNCLEAN_BUILDER(new_state); \
} while (0)
LibWeb: Make HTML tokenizer stop at insertion point after state switch If we reach the insertion point at the same time as we switch to another tokenizer state, we have to bail immediately without proceeding with the next code point. Otherwise we'd fetch the next token, get an EOF marker, and then proceed as if we're at the end of the input stream, even though more data may be coming (with more calls to document.write()..)
2024-11-23 18:30:01 +01:00
#define SWITCH_TO_WITH_UNCLEAN_BUILDER(new_state) \
do { \
will_switch_to(State::new_state); \
m_state = State::new_state; \
if (stop_at_insertion_point == StopAtInsertionPoint::Yes && is_insertion_point_reached()) \
return {}; \
CONSUME_NEXT_INPUT_CHARACTER; \
goto new_state; \
LibWeb: A whole bunch of work towards spec-compliant <script> elements This is still very unfinished, but there's at least a skeleton of code.
2020-05-24 22:00:46 +02:00
} while (0)
LibWeb: Begin work on a spec-compliant HTML parser In order to actually view the web as it is, we're gonna need a proper HTML parser. So let's build one! This patch introduces the Web::HTMLTokenizer class, which currently operates on a StringView input stream where it fetches (ASCII only atm) codepoints and tokenizes acccording to the HTML spec tokenization algo. The tokenizer state machine looks a bit weird but is written in a way that tries to mimic the spec as closely as possible, in order to make development easier and bugs less likely. This initial version is far from finished, but it can parse a trivial document with a DOCTYPE and open/close tags. :^)
2020-05-22 21:46:13 +02:00
LibWeb: A whole bunch of work towards spec-compliant <script> elements This is still very unfinished, but there's at least a skeleton of code.
2020-05-24 22:00:46 +02:00
#define RECONSUME_IN(new_state) \
do { \
will_reconsume_in(State::new_state); \
m_state = State::new_state; \
goto new_state; \
} while (0)
LibWeb: Begin work on a spec-compliant HTML parser In order to actually view the web as it is, we're gonna need a proper HTML parser. So let's build one! This patch introduces the Web::HTMLTokenizer class, which currently operates on a StringView input stream where it fetches (ASCII only atm) codepoints and tokenizes acccording to the HTML spec tokenization algo. The tokenizer state machine looks a bit weird but is written in a way that tries to mimic the spec as closely as possible, in order to make development easier and bugs less likely. This initial version is far from finished, but it can parse a trivial document with a DOCTYPE and open/close tags. :^)
2020-05-22 21:46:13 +02:00
LibWeb: Implement partial support for numeric character references
2020-05-27 16:16:23 +02:00
#define SWITCH_TO_RETURN_STATE \
do { \
will_switch_to(m_return_state); \
m_state = m_return_state; \
goto _StartOfFunction; \
} while (0)
LibWeb: Add position tracking information to HTML tokens
2021-05-20 23:11:41 +04:30
#define RECONSUME_IN_RETURN_STATE \
do { \
will_reconsume_in(m_return_state); \
m_state = m_return_state; \
if (current_input_character.has_value()) \
LibWeb: Let HTMLTokenizer walk over code points instead of UTF-8 Instead of using UTF-8 iterators to traverse the HTMLTokenizer input stream one code point at a time, we now do a one-shot conversion up front from the input encoding to a Vector<u32> of Unicode code points. This simplifies the tokenizer logic somewhat, and ends up being faster as well, so win-win. 1.02x speedup on Speedometer 2.1
2025-05-10 11:28:35 +02:00
restore_to(m_prev_offset); \
LibWeb: Add position tracking information to HTML tokens
2021-05-20 23:11:41 +04:30
goto _StartOfFunction; \
LibWeb: Implement partial support for numeric character references
2020-05-27 16:16:23 +02:00
} while (0)
LibWeb: Use move() when enqueuing tokens in HTMLTokenizer We're not using the current token anymore once it's enqueued so let's use move() when enqueuing the tokens.
2021-05-23 12:39:00 +02:00
#define SWITCH_TO_AND_EMIT_CURRENT_TOKEN(new_state) \
do { \
LibWeb: VERIFY an empty builder when emitting tokens in HTMLTokenizer
2021-07-15 00:34:54 +02:00
VERIFY(m_current_builder.is_empty()); \
LibWeb: Use move() when enqueuing tokens in HTMLTokenizer We're not using the current token anymore once it's enqueued so let's use move() when enqueuing the tokens.
2021-05-23 12:39:00 +02:00
will_switch_to(State::new_state); \
m_state = State::new_state; \
will_emit(m_current_token); \
m_queued_tokens.enqueue(move(m_current_token)); \
return m_queued_tokens.dequeue(); \
LibWeb: Allow HTML tokenizer to emit more than one token Tokens are now put on a queue when emitted, and we always pop from that queue when returning from next_token().
2020-05-26 15:50:05 +02:00
} while (0)
Unicode: Try s/codepoint/code_point/g again This time, without trailing 's'. Ran: git grep -l 'codepoint' | xargs sed -ie 's/codepoint/code_point/g
2020-08-05 16:31:20 -04:00
#define EMIT_CHARACTER_AND_RECONSUME_IN(code_point, new_state) \
Meta+LibHTTP through LibWeb: Make clang-format-10 clean
2020-09-18 09:49:51 +02:00
do { \
Unicode: Try s/codepoint/code_point/g again This time, without trailing 's'. Ran: git grep -l 'codepoint' | xargs sed -ie 's/codepoint/code_point/g
2020-08-05 16:31:20 -04:00
m_queued_tokens.enqueue(HTMLToken::make_character(code_point)); \
Meta+LibHTTP through LibWeb: Make clang-format-10 clean
2020-09-18 09:49:51 +02:00
will_reconsume_in(State::new_state); \
m_state = State::new_state; \
goto new_state; \
LibWeb: A whole bunch of work towards spec-compliant <script> elements This is still very unfinished, but there's at least a skeleton of code.
2020-05-24 22:00:46 +02:00
} while (0)
LibWeb: Start building the tree building part of the new HTML parser This patch adds a new HTMLDocumentParser class. It keeps a tokenizer object internally and feeds itself with one token at a time from it. The names and idioms in this class are expressed as closely to the actual HTML parsing spec as possible, to make development as easy and bug free as possible. :^) This is going to become pretty large, but it's pretty cool!
2020-05-24 00:14:23 +02:00
LibWeb: Use getter and setter for Character type HTMLTokens While storing the code point in a UTF-8 encoded String in horrendously inefficient, this problem will be addressed at a later stage.
2021-07-14 23:33:12 +02:00
#define FLUSH_CODEPOINTS_CONSUMED_AS_A_CHARACTER_REFERENCE \
do { \
for (auto code_point : m_temporary_buffer) { \
if (consumed_as_part_of_an_attribute()) { \
m_current_builder.append_code_point(code_point); \
} else { \
create_new_token(HTMLToken::Type::Character); \
m_current_token.set_code_point(code_point); \
m_queued_tokens.enqueue(move(m_current_token)); \
} \
} \
LibWeb: Implement partial support for numeric character references
2020-05-27 16:16:23 +02:00
} while (0)
LibWeb: Fix numeric character reference at EOF leaking its last digit Previously, if the NumericCharacterReferenceEnd state was reached when current_input_character was None, then the DONT_CONSUME_NEXT_INPUT_CHARACTER macro would restore back before the EOF, and allow the next state (after the SWITCH_TO_RETURN_STATE) to proceed with the last digit of the numeric character reference. For example, with something like `&#1111`, before this commit the output would incorrectly be `<code point with the value 1111>1` instead of just `<code point with the value 1111>`. Instead of putting the `if (current_input_character.has_value())` check inside NumericCharacterReferenceEnd directly, it was instead added to DONT_CONSUME_NEXT_INPUT_CHARACTER, because all usages of the macro benefit from this check, even if the other existing usage sites don't exhibit any bugs without it: - In MarkupDeclarationOpen, if the current_input_character is EOF, then the previous character is always `!`, so restoring and then checking forward for strings like `--`, `DOCTYPE`, etc won't match and the BogusComment state will run one extra time (once for `!` and once for EOF) with no practical consequences. With the `has_value()` check, BogusComment will only run once with EOF. - In AfterDOCTYPEName, ConsumeNextResult::RanOutOfCharacters can only occur when stopping at the insertion point, and because of how the code is structured, it is guaranteed that current_input_character is either `P` or `S`, so the `has_value()` check is irrelevant.
2024-12-20 06:05:37 -08:00
#define DONT_CONSUME_NEXT_INPUT_CHARACTER \
do { \
if (current_input_character.has_value()) \
LibWeb: Let HTMLTokenizer walk over code points instead of UTF-8 Instead of using UTF-8 iterators to traverse the HTMLTokenizer input stream one code point at a time, we now do a one-shot conversion up front from the input encoding to a Vector<u32> of Unicode code points. This simplifies the tokenizer logic somewhat, and ends up being faster as well, so win-win. 1.02x speedup on Speedometer 2.1
2025-05-10 11:28:35 +02:00
restore_to(m_prev_offset); \
LibWeb: Make the new HTML parser parse input as UTF-8 We already convert the input to UTF-8 before starting the tokenizer, so all this patch had to do was switch the tokenizer to use an Utf8View for its input (and to emit 32-bit codepoints.)
2020-06-04 21:06:54 +02:00
} while (0)
LibWeb: Begin work on a spec-compliant HTML parser In order to actually view the web as it is, we're gonna need a proper HTML parser. So let's build one! This patch introduces the Web::HTMLTokenizer class, which currently operates on a StringView input stream where it fetches (ASCII only atm) codepoints and tokenizes acccording to the HTML spec tokenization algo. The tokenizer state machine looks a bit weird but is written in a way that tries to mimic the spec as closely as possible, in order to make development easier and bugs less likely. This initial version is far from finished, but it can parse a trivial document with a DOCTYPE and open/close tags. :^)
2020-05-22 21:46:13 +02:00
Unicode: Try s/codepoint/code_point/g again This time, without trailing 's'. Ran: git grep -l 'codepoint' | xargs sed -ie 's/codepoint/code_point/g
2020-08-05 16:31:20 -04:00
#define ON(code_point) \
if (current_input_character.has_value() && current_input_character.value() == code_point)
LibWeb: Begin work on a spec-compliant HTML parser In order to actually view the web as it is, we're gonna need a proper HTML parser. So let's build one! This patch introduces the Web::HTMLTokenizer class, which currently operates on a StringView input stream where it fetches (ASCII only atm) codepoints and tokenizes acccording to the HTML spec tokenization algo. The tokenizer state machine looks a bit weird but is written in a way that tries to mimic the spec as closely as possible, in order to make development easier and bugs less likely. This initial version is far from finished, but it can parse a trivial document with a DOCTYPE and open/close tags. :^)
2020-05-22 21:46:13 +02:00
#define ON_EOF \
if (!current_input_character.has_value())
#define ON_ASCII_ALPHA \
Everywhere: Replace ctype.h to avoid narrowing conversions This replaces ctype.h with CharacterType.h everywhere I could find issues with narrowing conversions. While using it will probably make sense almost everywhere in the future, the most critical places should have been addressed.
2021-06-01 21:18:08 +02:00
if (current_input_character.has_value() && is_ascii_alpha(current_input_character.value()))
LibWeb: Begin work on a spec-compliant HTML parser In order to actually view the web as it is, we're gonna need a proper HTML parser. So let's build one! This patch introduces the Web::HTMLTokenizer class, which currently operates on a StringView input stream where it fetches (ASCII only atm) codepoints and tokenizes acccording to the HTML spec tokenization algo. The tokenizer state machine looks a bit weird but is written in a way that tries to mimic the spec as closely as possible, in order to make development easier and bugs less likely. This initial version is far from finished, but it can parse a trivial document with a DOCTYPE and open/close tags. :^)
2020-05-22 21:46:13 +02:00
LibWeb: Implement partial support for numeric character references
2020-05-27 16:16:23 +02:00
#define ON_ASCII_ALPHANUMERIC \
Everywhere: Replace ctype.h to avoid narrowing conversions This replaces ctype.h with CharacterType.h everywhere I could find issues with narrowing conversions. While using it will probably make sense almost everywhere in the future, the most critical places should have been addressed.
2021-06-01 21:18:08 +02:00
if (current_input_character.has_value() && is_ascii_alphanumeric(current_input_character.value()))
LibWeb: Implement partial support for numeric character references
2020-05-27 16:16:23 +02:00
LibWeb: Implement a bit more of DOCTYPE tokenization
2020-05-23 19:56:07 +02:00
#define ON_ASCII_UPPER_ALPHA \
Everywhere: Replace ctype.h to avoid narrowing conversions This replaces ctype.h with CharacterType.h everywhere I could find issues with narrowing conversions. While using it will probably make sense almost everywhere in the future, the most critical places should have been addressed.
2021-06-01 21:18:08 +02:00
if (current_input_character.has_value() && is_ascii_upper_alpha(current_input_character.value()))
LibWeb: Implement a bit more of DOCTYPE tokenization
2020-05-23 19:56:07 +02:00
LibWeb: More work on the HTML parser and tokenizer The parser can now switch the state of the tokenizer! Very webby. :^)
2020-05-24 20:24:43 +02:00
#define ON_ASCII_LOWER_ALPHA \
Everywhere: Replace ctype.h to avoid narrowing conversions This replaces ctype.h with CharacterType.h everywhere I could find issues with narrowing conversions. While using it will probably make sense almost everywhere in the future, the most critical places should have been addressed.
2021-06-01 21:18:08 +02:00
if (current_input_character.has_value() && is_ascii_lower_alpha(current_input_character.value()))
LibWeb: More work on the HTML parser and tokenizer The parser can now switch the state of the tokenizer! Very webby. :^)
2020-05-24 20:24:43 +02:00
LibWeb: Implement partial support for numeric character references
2020-05-27 16:16:23 +02:00
#define ON_ASCII_DIGIT \
Everywhere: Replace ctype.h to avoid narrowing conversions This replaces ctype.h with CharacterType.h everywhere I could find issues with narrowing conversions. While using it will probably make sense almost everywhere in the future, the most critical places should have been addressed.
2021-06-01 21:18:08 +02:00
if (current_input_character.has_value() && is_ascii_digit(current_input_character.value()))
LibWeb: Implement partial support for numeric character references
2020-05-27 16:16:23 +02:00
#define ON_ASCII_HEX_DIGIT \
Everywhere: Replace ctype.h to avoid narrowing conversions This replaces ctype.h with CharacterType.h everywhere I could find issues with narrowing conversions. While using it will probably make sense almost everywhere in the future, the most critical places should have been addressed.
2021-06-01 21:18:08 +02:00
if (current_input_character.has_value() && is_ascii_hex_digit(current_input_character.value()))
LibWeb: Implement partial support for numeric character references
2020-05-27 16:16:23 +02:00
LibWeb: Begin work on a spec-compliant HTML parser In order to actually view the web as it is, we're gonna need a proper HTML parser. So let's build one! This patch introduces the Web::HTMLTokenizer class, which currently operates on a StringView input stream where it fetches (ASCII only atm) codepoints and tokenizes acccording to the HTML spec tokenization algo. The tokenizer state machine looks a bit weird but is written in a way that tries to mimic the spec as closely as possible, in order to make development easier and bugs less likely. This initial version is far from finished, but it can parse a trivial document with a DOCTYPE and open/close tags. :^)
2020-05-22 21:46:13 +02:00
#define ON_WHITESPACE \
LibWeb: Make ON_WHITESPACE less heavy in HTML tokenizer Once we know that the current code point is an ASCII character, we can just check if it's one of the HTML whitespace characters. Before this patch, we were using the generic StringView::contains(u32) path that splats a code point into a StringBuilder and then searches for it with memmem(). This reduces time spent in the HTML tokenizer from 16% to 6% when loading the ECMA-262 spec.
2022-11-05 00:20:15 +01:00
if (current_input_character.has_value() && is_ascii(*current_input_character) && first_is_one_of(static_cast<char>(*current_input_character), '\t', '\n', '\f', ' '))
LibWeb: Begin work on a spec-compliant HTML parser In order to actually view the web as it is, we're gonna need a proper HTML parser. So let's build one! This patch introduces the Web::HTMLTokenizer class, which currently operates on a StringView input stream where it fetches (ASCII only atm) codepoints and tokenizes acccording to the HTML spec tokenization algo. The tokenizer state machine looks a bit weird but is written in a way that tries to mimic the spec as closely as possible, in order to make development easier and bugs less likely. This initial version is far from finished, but it can parse a trivial document with a DOCTYPE and open/close tags. :^)
2020-05-22 21:46:13 +02:00
#define ANYTHING_ELSE if (1)
LibWeb: Use move() when enqueuing tokens in HTMLTokenizer We're not using the current token anymore once it's enqueued so let's use move() when enqueuing the tokens.
2021-05-23 12:39:00 +02:00
#define EMIT_EOF \
do { \
if (m_has_emitted_eof) \
return {}; \
m_has_emitted_eof = true; \
create_new_token(HTMLToken::Type::EndOfFile); \
will_emit(m_current_token); \
m_queued_tokens.enqueue(move(m_current_token)); \
return m_queued_tokens.dequeue(); \
LibWeb: A whole bunch of work towards spec-compliant <script> elements This is still very unfinished, but there's at least a skeleton of code.
2020-05-24 22:00:46 +02:00
} while (0)
LibWeb: More work on the HTML parser and tokenizer The parser can now switch the state of the tokenizer! Very webby. :^)
2020-05-24 20:24:43 +02:00
LibWeb: Emit the current token before EOF on invalid comments The spec for each of these state: -> EOF: This is an eof-in-comment parse error. Emit the current comment token. Emit an end-of-file token. We were neglecting to emit the current comment token before emitting an EOF token. Note the existing EMIT_CURRENT_TOKEN macro was unused.
2024-03-23 10:11:29 -04:00
#define EMIT_CURRENT_TOKEN_FOLLOWED_BY_EOF \
LibWeb: Use move() when enqueuing tokens in HTMLTokenizer We're not using the current token anymore once it's enqueued so let's use move() when enqueuing the tokens.
2021-05-23 12:39:00 +02:00
do { \
LibWeb: VERIFY an empty builder when emitting tokens in HTMLTokenizer
2021-07-15 00:34:54 +02:00
VERIFY(m_current_builder.is_empty()); \
LibWeb: Use move() when enqueuing tokens in HTMLTokenizer We're not using the current token anymore once it's enqueued so let's use move() when enqueuing the tokens.
2021-05-23 12:39:00 +02:00
will_emit(m_current_token); \
m_queued_tokens.enqueue(move(m_current_token)); \
LibWeb: Emit the current token before EOF on invalid comments The spec for each of these state: -> EOF: This is an eof-in-comment parse error. Emit the current comment token. Emit an end-of-file token. We were neglecting to emit the current comment token before emitting an EOF token. Note the existing EMIT_CURRENT_TOKEN macro was unused.
2024-03-23 10:11:29 -04:00
\
m_has_emitted_eof = true; \
create_new_token(HTMLToken::Type::EndOfFile); \
will_emit(m_current_token); \
m_queued_tokens.enqueue(move(m_current_token)); \
\
LibWeb: Use move() when enqueuing tokens in HTMLTokenizer We're not using the current token anymore once it's enqueued so let's use move() when enqueuing the tokens.
2021-05-23 12:39:00 +02:00
return m_queued_tokens.dequeue(); \
LibWeb: A whole bunch of work towards spec-compliant <script> elements This is still very unfinished, but there's at least a skeleton of code.
2020-05-24 22:00:46 +02:00
} while (0)
LibWeb: Start building the tree building part of the new HTML parser This patch adds a new HTMLDocumentParser class. It keeps a tokenizer object internally and feeds itself with one token at a time from it. The names and idioms in this class are expressed as closely to the actual HTML parsing spec as possible, to make development as easy and bug free as possible. :^) This is going to become pretty large, but it's pretty cool!
2020-05-24 00:14:23 +02:00
LibWeb: Use getter and setter for Character type HTMLTokens While storing the code point in a UTF-8 encoded String in horrendously inefficient, this problem will be addressed at a later stage.
2021-07-14 23:33:12 +02:00
#define EMIT_CHARACTER(code_point) \
do { \
create_new_token(HTMLToken::Type::Character); \
m_current_token.set_code_point(code_point); \
m_queued_tokens.enqueue(move(m_current_token)); \
return m_queued_tokens.dequeue(); \
LibWeb: A whole bunch of work towards spec-compliant <script> elements This is still very unfinished, but there's at least a skeleton of code.
2020-05-24 22:00:46 +02:00
} while (0)
LibWeb: Begin work on a spec-compliant HTML parser In order to actually view the web as it is, we're gonna need a proper HTML parser. So let's build one! This patch introduces the Web::HTMLTokenizer class, which currently operates on a StringView input stream where it fetches (ASCII only atm) codepoints and tokenizes acccording to the HTML spec tokenization algo. The tokenizer state machine looks a bit weird but is written in a way that tries to mimic the spec as closely as possible, in order to make development easier and bugs less likely. This initial version is far from finished, but it can parse a trivial document with a DOCTYPE and open/close tags. :^)
2020-05-22 21:46:13 +02:00
LibWeb: More work on the HTML parser and tokenizer The parser can now switch the state of the tokenizer! Very webby. :^)
2020-05-24 20:24:43 +02:00
#define EMIT_CURRENT_CHARACTER \
EMIT_CHARACTER(current_input_character.value());
Unicode: Try s/codepoint/code_point/g again This time, without trailing 's'. Ran: git grep -l 'codepoint' | xargs sed -ie 's/codepoint/code_point/g
2020-08-05 16:31:20 -04:00
#define SWITCH_TO_AND_EMIT_CHARACTER(code_point, new_state) \
Meta+LibHTTP through LibWeb: Make clang-format-10 clean
2020-09-18 09:49:51 +02:00
do { \
will_switch_to(State::new_state); \
m_state = State::new_state; \
Unicode: Try s/codepoint/code_point/g again This time, without trailing 's'. Ran: git grep -l 'codepoint' | xargs sed -ie 's/codepoint/code_point/g
2020-08-05 16:31:20 -04:00
EMIT_CHARACTER(code_point); \
LibWeb: Fully implement all script tokenizer states Also fixes RAWTEXTLessThanSign having a separate emit and reconsume.
2020-06-06 07:06:46 +01:00
} while (0)
#define SWITCH_TO_AND_EMIT_CURRENT_CHARACTER(new_state) \
SWITCH_TO_AND_EMIT_CHARACTER(current_input_character.value(), new_state)
LibWeb: Begin work on a spec-compliant HTML parser In order to actually view the web as it is, we're gonna need a proper HTML parser. So let's build one! This patch introduces the Web::HTMLTokenizer class, which currently operates on a StringView input stream where it fetches (ASCII only atm) codepoints and tokenizes acccording to the HTML spec tokenization algo. The tokenizer state machine looks a bit weird but is written in a way that tries to mimic the spec as closely as possible, in order to make development easier and bugs less likely. This initial version is far from finished, but it can parse a trivial document with a DOCTYPE and open/close tags. :^)
2020-05-22 21:46:13 +02:00
#define BEGIN_STATE(state) \
state: \
LibWeb: Make BEGIN_STATE and END_STATE include some {{{ and }}} This makes it a compile error to omit the END_STATE. Also add some more missing END_STATE's exposed by this (nice!) Thanks to @predmond for suggesting the multi-pair trick! :^)
2020-05-23 14:04:53 +02:00
case State::state: { \
{ \
{
LibWeb: Begin work on a spec-compliant HTML parser In order to actually view the web as it is, we're gonna need a proper HTML parser. So let's build one! This patch introduces the Web::HTMLTokenizer class, which currently operates on a StringView input stream where it fetches (ASCII only atm) codepoints and tokenizes acccording to the HTML spec tokenization algo. The tokenizer state machine looks a bit weird but is written in a way that tries to mimic the spec as closely as possible, in order to make development easier and bugs less likely. This initial version is far from finished, but it can parse a trivial document with a DOCTYPE and open/close tags. :^)
2020-05-22 21:46:13 +02:00
#define END_STATE \
Everywhere: Rename ASSERT => VERIFY (...and ASSERT_NOT_REACHED => VERIFY_NOT_REACHED) Since all of these checks are done in release builds as well, let's rename them to VERIFY to prevent confusion, as everyone is used to assertions being compiled out in release. We can introduce a new ASSERT macro that is specifically for debug checks, but I'm doing this wholesale conversion first since we've accumulated thousands of these already, and it's not immediately obvious which ones are suitable for ASSERT.
2021-02-23 20:42:32 +01:00
VERIFY_NOT_REACHED(); \
LibWeb: Make BEGIN_STATE and END_STATE include some {{{ and }}} This makes it a compile error to omit the END_STATE. Also add some more missing END_STATE's exposed by this (nice!) Thanks to @predmond for suggesting the multi-pair trick! :^)
2020-05-23 14:04:53 +02:00
break; \
} \
} \
}
LibWeb: Begin work on a spec-compliant HTML parser In order to actually view the web as it is, we're gonna need a proper HTML parser. So let's build one! This patch introduces the Web::HTMLTokenizer class, which currently operates on a StringView input stream where it fetches (ASCII only atm) codepoints and tokenizes acccording to the HTML spec tokenization algo. The tokenizer state machine looks a bit weird but is written in a way that tries to mimic the spec as closely as possible, in order to make development easier and bugs less likely. This initial version is far from finished, but it can parse a trivial document with a DOCTYPE and open/close tags. :^)
2020-05-22 21:46:13 +02:00
LibWeb: Change HTMLTokenizer.{cpp,h} to east const style
2021-07-12 12:44:21 +02:00
static inline void log_parse_error(SourceLocation const& location = SourceLocation::current())
LibWeb: Utilize SourceLocation for HTMLTokenizer logging
2021-04-24 20:22:30 -07:00
{
dbgln_if(TOKENIZER_TRACE_DEBUG, "Parse error (tokenization) {}", location);
}
LibWeb: Teach more of HTMLTokenizer to stop at the insertion point In particular, input character lookahead now knows how to stop at the insertion point marker if needed. This makes it possible to do amazing things like having document.write() insert doctypes one character at a time.
2024-11-24 10:18:17 +01:00
Optional<u32> HTMLTokenizer::next_code_point(StopAtInsertionPoint stop_at_insertion_point)
LibWeb: Begin work on a spec-compliant HTML parser In order to actually view the web as it is, we're gonna need a proper HTML parser. So let's build one! This patch introduces the Web::HTMLTokenizer class, which currently operates on a StringView input stream where it fetches (ASCII only atm) codepoints and tokenizes acccording to the HTML spec tokenization algo. The tokenizer state machine looks a bit weird but is written in a way that tries to mimic the spec as closely as possible, in order to make development easier and bugs less likely. This initial version is far from finished, but it can parse a trivial document with a DOCTYPE and open/close tags. :^)
2020-05-22 21:46:13 +02:00
{
LibWeb: Let HTMLTokenizer walk over code points instead of UTF-8 Instead of using UTF-8 iterators to traverse the HTMLTokenizer input stream one code point at a time, we now do a one-shot conversion up front from the input encoding to a Vector<u32> of Unicode code points. This simplifies the tokenizer logic somewhat, and ends up being faster as well, so win-win. 1.02x speedup on Speedometer 2.1
2025-05-10 11:28:35 +02:00
if (m_current_offset >= static_cast<ssize_t>(m_decoded_input.size()))
LibWeb: Begin work on a spec-compliant HTML parser In order to actually view the web as it is, we're gonna need a proper HTML parser. So let's build one! This patch introduces the Web::HTMLTokenizer class, which currently operates on a StringView input stream where it fetches (ASCII only atm) codepoints and tokenizes acccording to the HTML spec tokenization algo. The tokenizer state machine looks a bit weird but is written in a way that tries to mimic the spec as closely as possible, in order to make development easier and bugs less likely. This initial version is far from finished, but it can parse a trivial document with a DOCTYPE and open/close tags. :^)
2020-05-22 21:46:13 +02:00
return {};
LibWeb: Implement tokenization newline preprocessing Newline normalization will replace \r and \r\n with \n. The spec specifically states > Before the tokenization stage, the input stream must be preprocessed > by normalizing newlines. wheras this is implemented the processing during the tokenization itself. This should still exhibit the same behaviour, while keeping the tokenization logic in the same place.
2022-02-18 22:46:28 +00:00
u32 code_point;
// https://html.spec.whatwg.org/multipage/parsing.html#preprocessing-the-input-stream:tokenization
// https://infra.spec.whatwg.org/#normalize-newlines
LibWeb: Teach more of HTMLTokenizer to stop at the insertion point In particular, input character lookahead now knows how to stop at the insertion point marker if needed. This makes it possible to do amazing things like having document.write() insert doctypes one character at a time.
2024-11-24 10:18:17 +01:00
if (peek_code_point(0, stop_at_insertion_point).value_or(0) == '\r' && peek_code_point(1, stop_at_insertion_point).value_or(0) == '\n') {
LibWeb: Implement tokenization newline preprocessing Newline normalization will replace \r and \r\n with \n. The spec specifically states > Before the tokenization stage, the input stream must be preprocessed > by normalizing newlines. wheras this is implemented the processing during the tokenization itself. This should still exhibit the same behaviour, while keeping the tokenization logic in the same place.
2022-02-18 22:46:28 +00:00
// replace every U+000D CR U+000A LF code point pair with a single U+000A LF code point,
skip(2);
code_point = '\n';
LibWeb: Teach more of HTMLTokenizer to stop at the insertion point In particular, input character lookahead now knows how to stop at the insertion point marker if needed. This makes it possible to do amazing things like having document.write() insert doctypes one character at a time.
2024-11-24 10:18:17 +01:00
} else if (peek_code_point(0, stop_at_insertion_point).value_or(0) == '\r') {
LibWeb: Implement tokenization newline preprocessing Newline normalization will replace \r and \r\n with \n. The spec specifically states > Before the tokenization stage, the input stream must be preprocessed > by normalizing newlines. wheras this is implemented the processing during the tokenization itself. This should still exhibit the same behaviour, while keeping the tokenization logic in the same place.
2022-02-18 22:46:28 +00:00
// replace every remaining U+000D CR code point with a U+000A LF code point.
skip(1);
code_point = '\n';
} else {
skip(1);
LibWeb: Let HTMLTokenizer walk over code points instead of UTF-8 Instead of using UTF-8 iterators to traverse the HTMLTokenizer input stream one code point at a time, we now do a one-shot conversion up front from the input encoding to a Vector<u32> of Unicode code points. This simplifies the tokenizer logic somewhat, and ends up being faster as well, so win-win. 1.02x speedup on Speedometer 2.1
2025-05-10 11:28:35 +02:00
code_point = m_decoded_input[m_prev_offset];
LibWeb: Implement tokenization newline preprocessing Newline normalization will replace \r and \r\n with \n. The spec specifically states > Before the tokenization stage, the input stream must be preprocessed > by normalizing newlines. wheras this is implemented the processing during the tokenization itself. This should still exhibit the same behaviour, while keeping the tokenization logic in the same place.
2022-02-18 22:46:28 +00:00
}
dbgln_if(TOKENIZER_TRACE_DEBUG, "(Tokenizer) Next code_point: {}", code_point);
return code_point;
LibWeb: Begin work on a spec-compliant HTML parser In order to actually view the web as it is, we're gonna need a proper HTML parser. So let's build one! This patch introduces the Web::HTMLTokenizer class, which currently operates on a StringView input stream where it fetches (ASCII only atm) codepoints and tokenizes acccording to the HTML spec tokenization algo. The tokenizer state machine looks a bit weird but is written in a way that tries to mimic the spec as closely as possible, in order to make development easier and bugs less likely. This initial version is far from finished, but it can parse a trivial document with a DOCTYPE and open/close tags. :^)
2020-05-22 21:46:13 +02:00
}
LibWeb: Add position tracking information to HTML tokens
2021-05-20 23:11:41 +04:30
void HTMLTokenizer::skip(size_t count)
{
LibWeb: Fix Array OOBs in the HTMLTokenizer Accessing last() if there are no elements makes WebContent crash :^)
2022-06-02 01:03:44 +02:00
if (!m_source_positions.is_empty())
m_source_positions.append(m_source_positions.last());
LibWeb: Add position tracking information to HTML tokens
2021-05-20 23:11:41 +04:30
for (size_t i = 0; i < count; ++i) {
LibWeb: Let HTMLTokenizer walk over code points instead of UTF-8 Instead of using UTF-8 iterators to traverse the HTMLTokenizer input stream one code point at a time, we now do a one-shot conversion up front from the input encoding to a Vector<u32> of Unicode code points. This simplifies the tokenizer logic somewhat, and ends up being faster as well, so win-win. 1.02x speedup on Speedometer 2.1
2025-05-10 11:28:35 +02:00
m_prev_offset = m_current_offset;
auto code_point = m_decoded_input[m_current_offset];
LibWeb: Fix Array OOBs in the HTMLTokenizer Accessing last() if there are no elements makes WebContent crash :^)
2022-06-02 01:03:44 +02:00
if (!m_source_positions.is_empty()) {
if (code_point == '\n') {
m_source_positions.last().column = 0;
m_source_positions.last().line++;
} else {
m_source_positions.last().column++;
}
LibWeb: Add position tracking information to HTML tokens
2021-05-20 23:11:41 +04:30
}
LibWeb: Let HTMLTokenizer walk over code points instead of UTF-8 Instead of using UTF-8 iterators to traverse the HTMLTokenizer input stream one code point at a time, we now do a one-shot conversion up front from the input encoding to a Vector<u32> of Unicode code points. This simplifies the tokenizer logic somewhat, and ends up being faster as well, so win-win. 1.02x speedup on Speedometer 2.1
2025-05-10 11:28:35 +02:00
++m_current_offset;
LibWeb: Add position tracking information to HTML tokens
2021-05-20 23:11:41 +04:30
}
}
LibWeb: Let HTMLTokenizer walk over code points instead of UTF-8 Instead of using UTF-8 iterators to traverse the HTMLTokenizer input stream one code point at a time, we now do a one-shot conversion up front from the input encoding to a Vector<u32> of Unicode code points. This simplifies the tokenizer logic somewhat, and ends up being faster as well, so win-win. 1.02x speedup on Speedometer 2.1
2025-05-10 11:28:35 +02:00
Optional<u32> HTMLTokenizer::peek_code_point(ssize_t offset, StopAtInsertionPoint stop_at_insertion_point) const
LibWeb: Begin work on a spec-compliant HTML parser In order to actually view the web as it is, we're gonna need a proper HTML parser. So let's build one! This patch introduces the Web::HTMLTokenizer class, which currently operates on a StringView input stream where it fetches (ASCII only atm) codepoints and tokenizes acccording to the HTML spec tokenization algo. The tokenizer state machine looks a bit weird but is written in a way that tries to mimic the spec as closely as possible, in order to make development easier and bugs less likely. This initial version is far from finished, but it can parse a trivial document with a DOCTYPE and open/close tags. :^)
2020-05-22 21:46:13 +02:00
{
LibWeb: Let HTMLTokenizer walk over code points instead of UTF-8 Instead of using UTF-8 iterators to traverse the HTMLTokenizer input stream one code point at a time, we now do a one-shot conversion up front from the input encoding to a Vector<u32> of Unicode code points. This simplifies the tokenizer logic somewhat, and ends up being faster as well, so win-win. 1.02x speedup on Speedometer 2.1
2025-05-10 11:28:35 +02:00
auto it = m_current_offset + offset;
if (it >= static_cast<ssize_t>(m_decoded_input.size()))
return {};
LibWeb: Teach more of HTMLTokenizer to stop at the insertion point In particular, input character lookahead now knows how to stop at the insertion point marker if needed. This makes it possible to do amazing things like having document.write() insert doctypes one character at a time.
2024-11-24 10:18:17 +01:00
if (stop_at_insertion_point == StopAtInsertionPoint::Yes
&& m_insertion_point.defined
LibWeb: Let HTMLTokenizer walk over code points instead of UTF-8 Instead of using UTF-8 iterators to traverse the HTMLTokenizer input stream one code point at a time, we now do a one-shot conversion up front from the input encoding to a Vector<u32> of Unicode code points. This simplifies the tokenizer logic somewhat, and ends up being faster as well, so win-win. 1.02x speedup on Speedometer 2.1
2025-05-10 11:28:35 +02:00
&& it >= m_insertion_point.position) {
LibWeb: Teach more of HTMLTokenizer to stop at the insertion point In particular, input character lookahead now knows how to stop at the insertion point marker if needed. This makes it possible to do amazing things like having document.write() insert doctypes one character at a time.
2024-11-24 10:18:17 +01:00
return {};
}
LibWeb: Let HTMLTokenizer walk over code points instead of UTF-8 Instead of using UTF-8 iterators to traverse the HTMLTokenizer input stream one code point at a time, we now do a one-shot conversion up front from the input encoding to a Vector<u32> of Unicode code points. This simplifies the tokenizer logic somewhat, and ends up being faster as well, so win-win. 1.02x speedup on Speedometer 2.1
2025-05-10 11:28:35 +02:00
return m_decoded_input[it];
LibWeb: Begin work on a spec-compliant HTML parser In order to actually view the web as it is, we're gonna need a proper HTML parser. So let's build one! This patch introduces the Web::HTMLTokenizer class, which currently operates on a StringView input stream where it fetches (ASCII only atm) codepoints and tokenizes acccording to the HTML spec tokenization algo. The tokenizer state machine looks a bit weird but is written in a way that tries to mimic the spec as closely as possible, in order to make development easier and bugs less likely. This initial version is far from finished, but it can parse a trivial document with a DOCTYPE and open/close tags. :^)
2020-05-22 21:46:13 +02:00
}
LibWeb: Be more forgiving when adding source positions in HTMLTokenizer This patch changes HTMLTokenizer::nth_last_position to not fail if the requested position is not available. Rather, it will just return (0-0). While this is not the correct solution, it prevents the tokenizer from crashing just because it cannot find a source position. This should only affect SyntaxHighlighter.
2021-06-04 11:31:43 +02:00
HTMLToken::Position HTMLTokenizer::nth_last_position(size_t n)
{
if (n + 1 > m_source_positions.size()) {
dbgln_if(TOKENIZER_TRACE_DEBUG, "(Tokenizer::nth_last_position) Invalid position requested: {}th-last of {}. Returning (0-0).", n, m_source_positions.size());
return HTMLToken::Position { 0, 0 };
};
return m_source_positions.at(m_source_positions.size() - 1 - n);
}
LibWeb: Stop parsing after `document.write` at the insertion point If a call to `document.write` inserts an incomplete HTML tag, e.g.: document.write("<p"); we would previously continue parsing the document until we reached a closing angle bracket. However, the spec states we should stop once we reach the new insertion point.
2024-02-18 12:45:53 -05:00
Optional<HTMLToken> HTMLTokenizer::next_token(StopAtInsertionPoint stop_at_insertion_point)
LibWeb: Begin work on a spec-compliant HTML parser In order to actually view the web as it is, we're gonna need a proper HTML parser. So let's build one! This patch introduces the Web::HTMLTokenizer class, which currently operates on a StringView input stream where it fetches (ASCII only atm) codepoints and tokenizes acccording to the HTML spec tokenization algo. The tokenizer state machine looks a bit weird but is written in a way that tries to mimic the spec as closely as possible, in order to make development easier and bugs less likely. This initial version is far from finished, but it can parse a trivial document with a DOCTYPE and open/close tags. :^)
2020-05-22 21:46:13 +02:00
{
LibWeb: Fix Array OOBs in the HTMLTokenizer Accessing last() if there are no elements makes WebContent crash :^)
2022-06-02 01:03:44 +02:00
if (!m_source_positions.is_empty()) {
LibWeb: Add a super basic HTML syntax highlighter This can currently highlight tag names and attribute names/values.
2021-05-20 23:15:33 +04:30
auto last_position = m_source_positions.last();
LibWeb: Use Vector::clear_with_capacity() in HTMLTokenizer This avoids constantly reallocating the Vector<HTMLToken>.
2022-02-18 23:49:51 +01:00
m_source_positions.clear_with_capacity();
LibWeb: Add a super basic HTML syntax highlighter This can currently highlight tag names and attribute names/values.
2021-05-20 23:15:33 +04:30
m_source_positions.append(move(last_position));
}
LibWeb: Implement partial support for numeric character references
2020-05-27 16:16:23 +02:00
_StartOfFunction:
LibWeb: Allow HTML tokenizer to emit more than one token Tokens are now put on a queue when emitted, and we always pop from that queue when returning from next_token().
2020-05-26 15:50:05 +02:00
if (!m_queued_tokens.is_empty())
return m_queued_tokens.dequeue();
LibWeb: Implement aborting the HTML parser This is roughly on-spec, although I had to invent a simple "aborted" state for the tokenizer.
2022-09-20 21:08:14 +02:00
if (m_aborted)
return {};
LibWeb: Begin work on a spec-compliant HTML parser In order to actually view the web as it is, we're gonna need a proper HTML parser. So let's build one! This patch introduces the Web::HTMLTokenizer class, which currently operates on a StringView input stream where it fetches (ASCII only atm) codepoints and tokenizes acccording to the HTML spec tokenization algo. The tokenizer state machine looks a bit weird but is written in a way that tries to mimic the spec as closely as possible, in order to make development easier and bugs less likely. This initial version is far from finished, but it can parse a trivial document with a DOCTYPE and open/close tags. :^)
2020-05-22 21:46:13 +02:00
for (;;) {
LibWeb: Stop parsing after `document.write` at the insertion point If a call to `document.write` inserts an incomplete HTML tag, e.g.: document.write("<p"); we would previously continue parsing the document until we reached a closing angle bracket. However, the spec states we should stop once we reach the new insertion point.
2024-02-18 12:45:53 -05:00
if (stop_at_insertion_point == StopAtInsertionPoint::Yes && is_insertion_point_reached())
return {};
LibWeb: Teach more of HTMLTokenizer to stop at the insertion point In particular, input character lookahead now knows how to stop at the insertion point marker if needed. This makes it possible to do amazing things like having document.write() insert doctypes one character at a time.
2024-11-24 10:18:17 +01:00
auto current_input_character = next_code_point(stop_at_insertion_point);
LibWeb: Begin work on a spec-compliant HTML parser In order to actually view the web as it is, we're gonna need a proper HTML parser. So let's build one! This patch introduces the Web::HTMLTokenizer class, which currently operates on a StringView input stream where it fetches (ASCII only atm) codepoints and tokenizes acccording to the HTML spec tokenization algo. The tokenizer state machine looks a bit weird but is written in a way that tries to mimic the spec as closely as possible, in order to make development easier and bugs less likely. This initial version is far from finished, but it can parse a trivial document with a DOCTYPE and open/close tags. :^)
2020-05-22 21:46:13 +02:00
switch (m_state) {
LibWeb: Add spec links to each HTML tokenizer state section I didn't add full spec comments this time, but this is better than nothing :^)
2022-02-15 18:12:15 +00:00
// 13.2.5.1 Data state, https://html.spec.whatwg.org/multipage/parsing.html#data-state
LibWeb: Begin work on a spec-compliant HTML parser In order to actually view the web as it is, we're gonna need a proper HTML parser. So let's build one! This patch introduces the Web::HTMLTokenizer class, which currently operates on a StringView input stream where it fetches (ASCII only atm) codepoints and tokenizes acccording to the HTML spec tokenization algo. The tokenizer state machine looks a bit weird but is written in a way that tries to mimic the spec as closely as possible, in order to make development easier and bugs less likely. This initial version is far from finished, but it can parse a trivial document with a DOCTYPE and open/close tags. :^)
2020-05-22 21:46:13 +02:00
BEGIN_STATE(Data)
{
ON('&')
{
m_return_state = State::Data;
SWITCH_TO(CharacterReference);
}
ON('<')
{
SWITCH_TO(TagOpen);
}
LibWeb: Implement more of the tokenizer states Slowly adding more unimplemented options for tokenizer states.
2020-05-28 00:28:32 +02:00
ON(0)
{
LibWeb: Utilize SourceLocation for HTMLTokenizer logging
2021-04-24 20:22:30 -07:00
log_parse_error();
LibWeb: Implement more of the tokenizer states Slowly adding more unimplemented options for tokenizer states.
2020-05-28 00:28:32 +02:00
EMIT_CURRENT_CHARACTER;
}
LibWeb: Begin work on a spec-compliant HTML parser In order to actually view the web as it is, we're gonna need a proper HTML parser. So let's build one! This patch introduces the Web::HTMLTokenizer class, which currently operates on a StringView input stream where it fetches (ASCII only atm) codepoints and tokenizes acccording to the HTML spec tokenization algo. The tokenizer state machine looks a bit weird but is written in a way that tries to mimic the spec as closely as possible, in order to make development easier and bugs less likely. This initial version is far from finished, but it can parse a trivial document with a DOCTYPE and open/close tags. :^)
2020-05-22 21:46:13 +02:00
ON_EOF
{
LibWeb: Start building the tree building part of the new HTML parser This patch adds a new HTMLDocumentParser class. It keeps a tokenizer object internally and feeds itself with one token at a time from it. The names and idioms in this class are expressed as closely to the actual HTML parsing spec as possible, to make development as easy and bug free as possible. :^) This is going to become pretty large, but it's pretty cool!
2020-05-24 00:14:23 +02:00
EMIT_EOF;
LibWeb: Begin work on a spec-compliant HTML parser In order to actually view the web as it is, we're gonna need a proper HTML parser. So let's build one! This patch introduces the Web::HTMLTokenizer class, which currently operates on a StringView input stream where it fetches (ASCII only atm) codepoints and tokenizes acccording to the HTML spec tokenization algo. The tokenizer state machine looks a bit weird but is written in a way that tries to mimic the spec as closely as possible, in order to make development easier and bugs less likely. This initial version is far from finished, but it can parse a trivial document with a DOCTYPE and open/close tags. :^)
2020-05-22 21:46:13 +02:00
}
ANYTHING_ELSE
{
LibWeb: More work on the HTML parser and tokenizer The parser can now switch the state of the tokenizer! Very webby. :^)
2020-05-24 20:24:43 +02:00
EMIT_CURRENT_CHARACTER;
LibWeb: Begin work on a spec-compliant HTML parser In order to actually view the web as it is, we're gonna need a proper HTML parser. So let's build one! This patch introduces the Web::HTMLTokenizer class, which currently operates on a StringView input stream where it fetches (ASCII only atm) codepoints and tokenizes acccording to the HTML spec tokenization algo. The tokenizer state machine looks a bit weird but is written in a way that tries to mimic the spec as closely as possible, in order to make development easier and bugs less likely. This initial version is far from finished, but it can parse a trivial document with a DOCTYPE and open/close tags. :^)
2020-05-22 21:46:13 +02:00
}
}
END_STATE
LibWeb: Add spec links to each HTML tokenizer state section I didn't add full spec comments this time, but this is better than nothing :^)
2022-02-15 18:12:15 +00:00
// 13.2.5.6 Tag open state, https://html.spec.whatwg.org/multipage/parsing.html#tag-open-state
LibWeb: Begin work on a spec-compliant HTML parser In order to actually view the web as it is, we're gonna need a proper HTML parser. So let's build one! This patch introduces the Web::HTMLTokenizer class, which currently operates on a StringView input stream where it fetches (ASCII only atm) codepoints and tokenizes acccording to the HTML spec tokenization algo. The tokenizer state machine looks a bit weird but is written in a way that tries to mimic the spec as closely as possible, in order to make development easier and bugs less likely. This initial version is far from finished, but it can parse a trivial document with a DOCTYPE and open/close tags. :^)
2020-05-22 21:46:13 +02:00
BEGIN_STATE(TagOpen)
{
ON('!')
{
SWITCH_TO(MarkupDeclarationOpen);
}
ON('/')
{
SWITCH_TO(EndTagOpen);
}
ON_ASCII_ALPHA
{
create_new_token(HTMLToken::Type::StartTag);
RECONSUME_IN(TagName);
}
LibWeb: Emit character/comment tokens lazily to accumulate more data Instead of emitting data-bearing tokens immediately, do it lazily at the next state change. This allows us to accumulate full bursts of text in between tags instead of having one token per character. :^)
2020-05-23 18:43:09 +02:00
ON('?')
LibWeb: Implement more of the tokenizer states Slowly adding more unimplemented options for tokenizer states.
2020-05-28 00:28:32 +02:00
{
LibWeb: Utilize SourceLocation for HTMLTokenizer logging
2021-04-24 20:22:30 -07:00
log_parse_error();
LibWeb: Implement more of the tokenizer states Slowly adding more unimplemented options for tokenizer states.
2020-05-28 00:28:32 +02:00
create_new_token(HTMLToken::Type::Comment);
LibWeb: Use setter for HTMLToken::m_{start,end}_position
2021-07-15 15:53:54 +02:00
m_current_token.set_start_position({}, nth_last_position(2));
LibWeb: Implement more of the tokenizer states Slowly adding more unimplemented options for tokenizer states.
2020-05-28 00:28:32 +02:00
RECONSUME_IN(BogusComment);
}
ON_EOF
LibWeb: Emit character/comment tokens lazily to accumulate more data Instead of emitting data-bearing tokens immediately, do it lazily at the next state change. This allows us to accumulate full bursts of text in between tags instead of having one token per character. :^)
2020-05-23 18:43:09 +02:00
{
LibWeb: Utilize SourceLocation for HTMLTokenizer logging
2021-04-24 20:22:30 -07:00
log_parse_error();
LibWeb: Implement some missing tokenizer cases for EOF handling
2020-06-26 22:47:07 +02:00
m_queued_tokens.enqueue(HTMLToken::make_character('<'));
EMIT_EOF;
LibWeb: Emit character/comment tokens lazily to accumulate more data Instead of emitting data-bearing tokens immediately, do it lazily at the next state change. This allows us to accumulate full bursts of text in between tags instead of having one token per character. :^)
2020-05-23 18:43:09 +02:00
}
ANYTHING_ELSE
{
LibWeb: Utilize SourceLocation for HTMLTokenizer logging
2021-04-24 20:22:30 -07:00
log_parse_error();
LibWeb: Fully implement all DOCTYPE tokenizer states Also fixes TagOpen having a seperate emit and reconsume in ANYTHING_ELSE.
2020-06-11 05:00:45 +01:00
EMIT_CHARACTER_AND_RECONSUME_IN('<', Data);
LibWeb: Emit character/comment tokens lazily to accumulate more data Instead of emitting data-bearing tokens immediately, do it lazily at the next state change. This allows us to accumulate full bursts of text in between tags instead of having one token per character. :^)
2020-05-23 18:43:09 +02:00
}
LibWeb: Begin work on a spec-compliant HTML parser In order to actually view the web as it is, we're gonna need a proper HTML parser. So let's build one! This patch introduces the Web::HTMLTokenizer class, which currently operates on a StringView input stream where it fetches (ASCII only atm) codepoints and tokenizes acccording to the HTML spec tokenization algo. The tokenizer state machine looks a bit weird but is written in a way that tries to mimic the spec as closely as possible, in order to make development easier and bugs less likely. This initial version is far from finished, but it can parse a trivial document with a DOCTYPE and open/close tags. :^)
2020-05-22 21:46:13 +02:00
}
END_STATE
LibWeb: Add spec links to each HTML tokenizer state section I didn't add full spec comments this time, but this is better than nothing :^)
2022-02-15 18:12:15 +00:00
// 13.2.5.8 Tag name state, https://html.spec.whatwg.org/multipage/parsing.html#tag-name-state
LibWeb: Begin work on a spec-compliant HTML parser In order to actually view the web as it is, we're gonna need a proper HTML parser. So let's build one! This patch introduces the Web::HTMLTokenizer class, which currently operates on a StringView input stream where it fetches (ASCII only atm) codepoints and tokenizes acccording to the HTML spec tokenization algo. The tokenizer state machine looks a bit weird but is written in a way that tries to mimic the spec as closely as possible, in order to make development easier and bugs less likely. This initial version is far from finished, but it can parse a trivial document with a DOCTYPE and open/close tags. :^)
2020-05-22 21:46:13 +02:00
BEGIN_STATE(TagName)
{
LibWeb: Teach HTMLTokenizer how to tokenize attributes Properly tokenize single-quoted, double-quoted and unquoted attributes!
2020-05-23 01:19:42 +02:00
ON_WHITESPACE
{
LibWeb: User getter+setter for HTMLToken tag name and self-closing flag
2021-07-14 23:37:48 +02:00
m_current_token.set_tag_name(consume_current_builder());
LibWeb: Use setter for HTMLToken::m_{start,end}_position
2021-07-15 15:53:54 +02:00
m_current_token.set_end_position({}, nth_last_position(1));
LibWeb: Teach HTMLTokenizer how to tokenize attributes Properly tokenize single-quoted, double-quoted and unquoted attributes!
2020-05-23 01:19:42 +02:00
SWITCH_TO(BeforeAttributeName);
}
ON('/')
{
LibWeb: User getter+setter for HTMLToken tag name and self-closing flag
2021-07-14 23:37:48 +02:00
m_current_token.set_tag_name(consume_current_builder());
LibWeb: Use setter for HTMLToken::m_{start,end}_position
2021-07-15 15:53:54 +02:00
m_current_token.set_end_position({}, nth_last_position(0));
LibWeb: Teach HTMLTokenizer how to tokenize attributes Properly tokenize single-quoted, double-quoted and unquoted attributes!
2020-05-23 01:19:42 +02:00
SWITCH_TO(SelfClosingStartTag);
}
LibWeb: Begin work on a spec-compliant HTML parser In order to actually view the web as it is, we're gonna need a proper HTML parser. So let's build one! This patch introduces the Web::HTMLTokenizer class, which currently operates on a StringView input stream where it fetches (ASCII only atm) codepoints and tokenizes acccording to the HTML spec tokenization algo. The tokenizer state machine looks a bit weird but is written in a way that tries to mimic the spec as closely as possible, in order to make development easier and bugs less likely. This initial version is far from finished, but it can parse a trivial document with a DOCTYPE and open/close tags. :^)
2020-05-22 21:46:13 +02:00
ON('>')
{
LibWeb: User getter+setter for HTMLToken tag name and self-closing flag
2021-07-14 23:37:48 +02:00
m_current_token.set_tag_name(consume_current_builder());
LibWeb: Start building the tree building part of the new HTML parser This patch adds a new HTMLDocumentParser class. It keeps a tokenizer object internally and feeds itself with one token at a time from it. The names and idioms in this class are expressed as closely to the actual HTML parsing spec as possible, to make development as easy and bug free as possible. :^) This is going to become pretty large, but it's pretty cool!
2020-05-24 00:14:23 +02:00
SWITCH_TO_AND_EMIT_CURRENT_TOKEN(Data);
LibWeb: Begin work on a spec-compliant HTML parser In order to actually view the web as it is, we're gonna need a proper HTML parser. So let's build one! This patch introduces the Web::HTMLTokenizer class, which currently operates on a StringView input stream where it fetches (ASCII only atm) codepoints and tokenizes acccording to the HTML spec tokenization algo. The tokenizer state machine looks a bit weird but is written in a way that tries to mimic the spec as closely as possible, in order to make development easier and bugs less likely. This initial version is far from finished, but it can parse a trivial document with a DOCTYPE and open/close tags. :^)
2020-05-22 21:46:13 +02:00
}
LibWeb: Implement more of the tokenizer states Slowly adding more unimplemented options for tokenizer states.
2020-05-28 00:28:32 +02:00
ON_ASCII_UPPER_ALPHA
{
LibWeb: Remove StringBuilder from HTMLToken::m_tag
2021-05-23 09:16:07 +02:00
m_current_builder.append_code_point(to_ascii_lowercase(current_input_character.value()));
LibWeb: Use setter for HTMLToken::m_{start,end}_position
2021-07-15 15:53:54 +02:00
m_current_token.set_end_position({}, nth_last_position(0));
LibWeb: Implement more of the tokenizer states Slowly adding more unimplemented options for tokenizer states.
2020-05-28 00:28:32 +02:00
continue;
}
ON(0)
{
LibWeb: Utilize SourceLocation for HTMLTokenizer logging
2021-04-24 20:22:30 -07:00
log_parse_error();
LibWeb: Remove StringBuilder from HTMLToken::m_tag
2021-05-23 09:16:07 +02:00
m_current_builder.append_code_point(0xFFFD);
LibWeb: Use setter for HTMLToken::m_{start,end}_position
2021-07-15 15:53:54 +02:00
m_current_token.set_end_position({}, nth_last_position(0));
LibWeb: Implement more of the tokenizer states Slowly adding more unimplemented options for tokenizer states.
2020-05-28 00:28:32 +02:00
continue;
}
ON_EOF
{
LibWeb: Utilize SourceLocation for HTMLTokenizer logging
2021-04-24 20:22:30 -07:00
log_parse_error();
LibWeb: Implement more of the tokenizer states Slowly adding more unimplemented options for tokenizer states.
2020-05-28 00:28:32 +02:00
EMIT_EOF;
}
LibWeb: Begin work on a spec-compliant HTML parser In order to actually view the web as it is, we're gonna need a proper HTML parser. So let's build one! This patch introduces the Web::HTMLTokenizer class, which currently operates on a StringView input stream where it fetches (ASCII only atm) codepoints and tokenizes acccording to the HTML spec tokenization algo. The tokenizer state machine looks a bit weird but is written in a way that tries to mimic the spec as closely as possible, in order to make development easier and bugs less likely. This initial version is far from finished, but it can parse a trivial document with a DOCTYPE and open/close tags. :^)
2020-05-22 21:46:13 +02:00
ANYTHING_ELSE
{
LibWeb: Remove StringBuilder from HTMLToken::m_tag
2021-05-23 09:16:07 +02:00
m_current_builder.append_code_point(current_input_character.value());
LibWeb: Use setter for HTMLToken::m_{start,end}_position
2021-07-15 15:53:54 +02:00
m_current_token.set_end_position({}, nth_last_position(0));
LibWeb: Begin work on a spec-compliant HTML parser In order to actually view the web as it is, we're gonna need a proper HTML parser. So let's build one! This patch introduces the Web::HTMLTokenizer class, which currently operates on a StringView input stream where it fetches (ASCII only atm) codepoints and tokenizes acccording to the HTML spec tokenization algo. The tokenizer state machine looks a bit weird but is written in a way that tries to mimic the spec as closely as possible, in order to make development easier and bugs less likely. This initial version is far from finished, but it can parse a trivial document with a DOCTYPE and open/close tags. :^)
2020-05-22 21:46:13 +02:00
continue;
}
}
LibWeb: Add missing END_STATE for TagName Fixes #2339.
2020-05-23 10:33:23 +02:00
END_STATE
LibWeb: Begin work on a spec-compliant HTML parser In order to actually view the web as it is, we're gonna need a proper HTML parser. So let's build one! This patch introduces the Web::HTMLTokenizer class, which currently operates on a StringView input stream where it fetches (ASCII only atm) codepoints and tokenizes acccording to the HTML spec tokenization algo. The tokenizer state machine looks a bit weird but is written in a way that tries to mimic the spec as closely as possible, in order to make development easier and bugs less likely. This initial version is far from finished, but it can parse a trivial document with a DOCTYPE and open/close tags. :^)
2020-05-22 21:46:13 +02:00
LibWeb: Add spec links to each HTML tokenizer state section I didn't add full spec comments this time, but this is better than nothing :^)
2022-02-15 18:12:15 +00:00
// 13.2.5.7 End tag open state, https://html.spec.whatwg.org/multipage/parsing.html#end-tag-open-state
LibWeb: Begin work on a spec-compliant HTML parser In order to actually view the web as it is, we're gonna need a proper HTML parser. So let's build one! This patch introduces the Web::HTMLTokenizer class, which currently operates on a StringView input stream where it fetches (ASCII only atm) codepoints and tokenizes acccording to the HTML spec tokenization algo. The tokenizer state machine looks a bit weird but is written in a way that tries to mimic the spec as closely as possible, in order to make development easier and bugs less likely. This initial version is far from finished, but it can parse a trivial document with a DOCTYPE and open/close tags. :^)
2020-05-22 21:46:13 +02:00
BEGIN_STATE(EndTagOpen)
{
ON_ASCII_ALPHA
{
create_new_token(HTMLToken::Type::EndTag);
RECONSUME_IN(TagName);
}
LibWeb: Implement more of the tokenizer states Slowly adding more unimplemented options for tokenizer states.
2020-05-28 00:28:32 +02:00
ON('>')
{
LibWeb: Utilize SourceLocation for HTMLTokenizer logging
2021-04-24 20:22:30 -07:00
log_parse_error();
LibWeb: Implement more of the tokenizer states Slowly adding more unimplemented options for tokenizer states.
2020-05-28 00:28:32 +02:00
SWITCH_TO(Data);
}
ON_EOF
{
LibWeb: Utilize SourceLocation for HTMLTokenizer logging
2021-04-24 20:22:30 -07:00
log_parse_error();
LibWeb: Implement some missing tokenizer cases for EOF handling
2020-06-26 22:47:07 +02:00
m_queued_tokens.enqueue(HTMLToken::make_character('<'));
m_queued_tokens.enqueue(HTMLToken::make_character('/'));
EMIT_EOF;
LibWeb: Implement more of the tokenizer states Slowly adding more unimplemented options for tokenizer states.
2020-05-28 00:28:32 +02:00
}
ANYTHING_ELSE
{
LibWeb: Utilize SourceLocation for HTMLTokenizer logging
2021-04-24 20:22:30 -07:00
log_parse_error();
LibWeb: Implement more of the tokenizer states Slowly adding more unimplemented options for tokenizer states.
2020-05-28 00:28:32 +02:00
create_new_token(HTMLToken::Type::Comment);
RECONSUME_IN(BogusComment);
}
LibWeb: Begin work on a spec-compliant HTML parser In order to actually view the web as it is, we're gonna need a proper HTML parser. So let's build one! This patch introduces the Web::HTMLTokenizer class, which currently operates on a StringView input stream where it fetches (ASCII only atm) codepoints and tokenizes acccording to the HTML spec tokenization algo. The tokenizer state machine looks a bit weird but is written in a way that tries to mimic the spec as closely as possible, in order to make development easier and bugs less likely. This initial version is far from finished, but it can parse a trivial document with a DOCTYPE and open/close tags. :^)
2020-05-22 21:46:13 +02:00
}
END_STATE
LibWeb: Add spec links to each HTML tokenizer state section I didn't add full spec comments this time, but this is better than nothing :^)
2022-02-15 18:12:15 +00:00
// 13.2.5.42 Markup declaration open state, https://html.spec.whatwg.org/multipage/parsing.html#markup-declaration-open-state
LibWeb: Begin work on a spec-compliant HTML parser In order to actually view the web as it is, we're gonna need a proper HTML parser. So let's build one! This patch introduces the Web::HTMLTokenizer class, which currently operates on a StringView input stream where it fetches (ASCII only atm) codepoints and tokenizes acccording to the HTML spec tokenization algo. The tokenizer state machine looks a bit weird but is written in a way that tries to mimic the spec as closely as possible, in order to make development easier and bugs less likely. This initial version is far from finished, but it can parse a trivial document with a DOCTYPE and open/close tags. :^)
2020-05-22 21:46:13 +02:00
BEGIN_STATE(MarkupDeclarationOpen)
{
DONT_CONSUME_NEXT_INPUT_CHARACTER;
LibWeb: Teach more of HTMLTokenizer to stop at the insertion point In particular, input character lookahead now knows how to stop at the insertion point marker if needed. This makes it possible to do amazing things like having document.write() insert doctypes one character at a time.
2024-11-24 10:18:17 +01:00
switch (consume_next_if_match("--"sv, stop_at_insertion_point)) {
case ConsumeNextResult::Consumed:
LibWeb: Teach HTMLTokenizer how to tokenize comments We can now correctly tokenize the welcome.html test page. :^)
2020-05-23 01:54:26 +02:00
create_new_token(HTMLToken::Type::Comment);
LibWeb: Fix highlighting HTML comments Commit b193351a99 caused the HTML comments to flash when changing the text cursor. Also, when double-clicking on a comment, the selection started from the beginning of the file instead. The following message was displaying when `TOKENIZER_TRACE_DEBUG` was enabled: (Tokenizer::nth_last_position) Invalid position requested: 4th-last of 4. Returning (0-0). Changing the `nth_last_position` to 3 fixes this. I'm guessing that's because the parser is at that moment on the second hyphen of the `<!--` string, so it has to go back only by three characters.
2021-07-29 15:36:10 +02:00
m_current_token.set_start_position({}, nth_last_position(3));
LibWeb: Teach HTMLTokenizer how to tokenize comments We can now correctly tokenize the welcome.html test page. :^)
2020-05-23 01:54:26 +02:00
SWITCH_TO(CommentStart);
LibWeb: Teach more of HTMLTokenizer to stop at the insertion point In particular, input character lookahead now knows how to stop at the insertion point marker if needed. This makes it possible to do amazing things like having document.write() insert doctypes one character at a time.
2024-11-24 10:18:17 +01:00
break;
case ConsumeNextResult::NotConsumed:
break;
case ConsumeNextResult::RanOutOfCharacters:
return {};
LibWeb: Teach HTMLTokenizer how to tokenize comments We can now correctly tokenize the welcome.html test page. :^)
2020-05-23 01:54:26 +02:00
}
LibWeb: Teach more of HTMLTokenizer to stop at the insertion point In particular, input character lookahead now knows how to stop at the insertion point marker if needed. This makes it possible to do amazing things like having document.write() insert doctypes one character at a time.
2024-11-24 10:18:17 +01:00
switch (consume_next_if_match("DOCTYPE"sv, stop_at_insertion_point, CaseSensitivity::CaseInsensitive)) {
case ConsumeNextResult::Consumed:
LibWeb: Begin work on a spec-compliant HTML parser In order to actually view the web as it is, we're gonna need a proper HTML parser. So let's build one! This patch introduces the Web::HTMLTokenizer class, which currently operates on a StringView input stream where it fetches (ASCII only atm) codepoints and tokenizes acccording to the HTML spec tokenization algo. The tokenizer state machine looks a bit weird but is written in a way that tries to mimic the spec as closely as possible, in order to make development easier and bugs less likely. This initial version is far from finished, but it can parse a trivial document with a DOCTYPE and open/close tags. :^)
2020-05-22 21:46:13 +02:00
SWITCH_TO(DOCTYPE);
LibWeb: Teach more of HTMLTokenizer to stop at the insertion point In particular, input character lookahead now knows how to stop at the insertion point marker if needed. This makes it possible to do amazing things like having document.write() insert doctypes one character at a time.
2024-11-24 10:18:17 +01:00
break;
case ConsumeNextResult::NotConsumed:
break;
case ConsumeNextResult::RanOutOfCharacters:
return {};
LibWeb: Begin work on a spec-compliant HTML parser In order to actually view the web as it is, we're gonna need a proper HTML parser. So let's build one! This patch introduces the Web::HTMLTokenizer class, which currently operates on a StringView input stream where it fetches (ASCII only atm) codepoints and tokenizes acccording to the HTML spec tokenization algo. The tokenizer state machine looks a bit weird but is written in a way that tries to mimic the spec as closely as possible, in order to make development easier and bugs less likely. This initial version is far from finished, but it can parse a trivial document with a DOCTYPE and open/close tags. :^)
2020-05-22 21:46:13 +02:00
}
LibWeb: Teach more of HTMLTokenizer to stop at the insertion point In particular, input character lookahead now knows how to stop at the insertion point marker if needed. This makes it possible to do amazing things like having document.write() insert doctypes one character at a time.
2024-11-24 10:18:17 +01:00
switch (consume_next_if_match("[CDATA["sv, stop_at_insertion_point)) {
case ConsumeNextResult::Consumed:
LibWeb: Implement state switch for "[CDATA[" in HTML parser
2022-02-15 18:52:45 +00:00
// We keep the parser optional so that syntax highlighting can be lexer-only.
// The parser registers itself with the lexer it creates.
LibWeb: Make HTML parser "current node" APIs return nullable pointer It's actually possible for there to be no adjusted current node, when the stack of open elements is empty. This was covered by one of the WPT parsing tests.
2024-11-03 19:50:13 +01:00
if (m_parser != nullptr
&& m_parser->adjusted_current_node()
&& m_parser->adjusted_current_node()->namespace_uri() != Namespace::HTML) {
LibWeb: Implement state switch for "[CDATA[" in HTML parser
2022-02-15 18:52:45 +00:00
SWITCH_TO(CDATASection);
} else {
create_new_token(HTMLToken::Type::Comment);
Everywhere: Add sv suffix to strings relying on StringView(char const*) Each of these strings would previously rely on StringView's char const* constructor overload, which would call __builtin_strlen on the string. Since we now have operator ""sv, we can replace these with much simpler versions. This opens the door to being able to remove StringView(char const*). No functional changes.
2022-07-11 17:32:29 +00:00
m_current_builder.append("[CDATA["sv);
LibWeb: Implement state switch for "[CDATA[" in HTML parser
2022-02-15 18:52:45 +00:00
SWITCH_TO_WITH_UNCLEAN_BUILDER(BogusComment);
}
LibWeb: Teach more of HTMLTokenizer to stop at the insertion point In particular, input character lookahead now knows how to stop at the insertion point marker if needed. This makes it possible to do amazing things like having document.write() insert doctypes one character at a time.
2024-11-24 10:18:17 +01:00
break;
case ConsumeNextResult::NotConsumed:
break;
case ConsumeNextResult::RanOutOfCharacters:
return {};
LibWeb: Make the new HTML parser parse input as UTF-8 We already convert the input to UTF-8 before starting the tokenizer, so all this patch had to do was switch the tokenizer to use an Utf8View for its input (and to emit 32-bit codepoints.)
2020-06-04 21:06:54 +02:00
}
ANYTHING_ELSE
{
LibWeb: Utilize SourceLocation for HTMLTokenizer logging
2021-04-24 20:22:30 -07:00
log_parse_error();
LibWeb: Make the new HTML parser parse input as UTF-8 We already convert the input to UTF-8 before starting the tokenizer, so all this patch had to do was switch the tokenizer to use an Utf8View for its input (and to emit 32-bit codepoints.)
2020-06-04 21:06:54 +02:00
create_new_token(HTMLToken::Type::Comment);
SWITCH_TO(BogusComment);
}
LibWeb: Begin work on a spec-compliant HTML parser In order to actually view the web as it is, we're gonna need a proper HTML parser. So let's build one! This patch introduces the Web::HTMLTokenizer class, which currently operates on a StringView input stream where it fetches (ASCII only atm) codepoints and tokenizes acccording to the HTML spec tokenization algo. The tokenizer state machine looks a bit weird but is written in a way that tries to mimic the spec as closely as possible, in order to make development easier and bugs less likely. This initial version is far from finished, but it can parse a trivial document with a DOCTYPE and open/close tags. :^)
2020-05-22 21:46:13 +02:00
}
END_STATE
LibWeb: Add spec links to each HTML tokenizer state section I didn't add full spec comments this time, but this is better than nothing :^)
2022-02-15 18:12:15 +00:00
// 13.2.5.41 Bogus comment state, https://html.spec.whatwg.org/multipage/parsing.html#bogus-comment-state
LibWeb: Implement more of the tokenizer states Slowly adding more unimplemented options for tokenizer states.
2020-05-28 00:28:32 +02:00
BEGIN_STATE(BogusComment)
{
ON('>')
{
LibWeb: User getter and setter for Comment type HTMLTokens
2021-07-14 23:32:18 +02:00
m_current_token.set_comment(consume_current_builder());
LibWeb: Make the new HTML parser parse input as UTF-8 We already convert the input to UTF-8 before starting the tokenizer, so all this patch had to do was switch the tokenizer to use an Utf8View for its input (and to emit 32-bit codepoints.)
2020-06-04 21:06:54 +02:00
SWITCH_TO_AND_EMIT_CURRENT_TOKEN(Data);
LibWeb: Implement more of the tokenizer states Slowly adding more unimplemented options for tokenizer states.
2020-05-28 00:28:32 +02:00
}
ON_EOF
{
LibWeb: Emit comment token for unterminated bogus comments on EOF
2025-01-06 15:05:36 +00:00
m_current_token.set_comment(consume_current_builder());
EMIT_CURRENT_TOKEN_FOLLOWED_BY_EOF;
LibWeb: Implement more of the tokenizer states Slowly adding more unimplemented options for tokenizer states.
2020-05-28 00:28:32 +02:00
}
ON(0)
{
LibWeb: Utilize SourceLocation for HTMLTokenizer logging
2021-04-24 20:22:30 -07:00
log_parse_error();
LibWeb: Remove StringBuilder from HTMLToken::m_comment_or_character
2021-05-23 09:31:40 +02:00
m_current_builder.append_code_point(0xFFFD);
LibWeb: Make the new HTML parser parse input as UTF-8 We already convert the input to UTF-8 before starting the tokenizer, so all this patch had to do was switch the tokenizer to use an Utf8View for its input (and to emit 32-bit codepoints.)
2020-06-04 21:06:54 +02:00
continue;
LibWeb: Implement more of the tokenizer states Slowly adding more unimplemented options for tokenizer states.
2020-05-28 00:28:32 +02:00
}
ANYTHING_ELSE
{
LibWeb: Remove StringBuilder from HTMLToken::m_comment_or_character
2021-05-23 09:31:40 +02:00
m_current_builder.append_code_point(current_input_character.value());
LibWeb: Make the new HTML parser parse input as UTF-8 We already convert the input to UTF-8 before starting the tokenizer, so all this patch had to do was switch the tokenizer to use an Utf8View for its input (and to emit 32-bit codepoints.)
2020-06-04 21:06:54 +02:00
continue;
LibWeb: Implement more of the tokenizer states Slowly adding more unimplemented options for tokenizer states.
2020-05-28 00:28:32 +02:00
}
}
END_STATE
LibWeb: Add spec links to each HTML tokenizer state section I didn't add full spec comments this time, but this is better than nothing :^)
2022-02-15 18:12:15 +00:00
// 13.2.5.53 DOCTYPE state, https://html.spec.whatwg.org/multipage/parsing.html#doctype-state
LibWeb: Begin work on a spec-compliant HTML parser In order to actually view the web as it is, we're gonna need a proper HTML parser. So let's build one! This patch introduces the Web::HTMLTokenizer class, which currently operates on a StringView input stream where it fetches (ASCII only atm) codepoints and tokenizes acccording to the HTML spec tokenization algo. The tokenizer state machine looks a bit weird but is written in a way that tries to mimic the spec as closely as possible, in order to make development easier and bugs less likely. This initial version is far from finished, but it can parse a trivial document with a DOCTYPE and open/close tags. :^)
2020-05-22 21:46:13 +02:00
BEGIN_STATE(DOCTYPE)
{
ON_WHITESPACE
{
SWITCH_TO(BeforeDOCTYPEName);
}
LibWeb: Implement a bit more of DOCTYPE tokenization
2020-05-23 19:56:07 +02:00
ON('>')
{
RECONSUME_IN(BeforeDOCTYPEName);
}
ON_EOF
{
LibWeb: Utilize SourceLocation for HTMLTokenizer logging
2021-04-24 20:22:30 -07:00
log_parse_error();
LibWeb: Fully implement all DOCTYPE tokenizer states Also fixes TagOpen having a seperate emit and reconsume in ANYTHING_ELSE.
2020-06-11 05:00:45 +01:00
create_new_token(HTMLToken::Type::DOCTYPE);
LibWeb: Rename HTMLToken::doctype_data() => ensure_doctype_data() This renames the accessor to better reflect what it does, as this will allocate a DoctypeData struct if there is none.
2021-07-15 18:58:11 +02:00
m_current_token.ensure_doctype_data().force_quirks = true;
LibWeb: Use move() when enqueuing tokens in HTMLTokenizer We're not using the current token anymore once it's enqueued so let's use move() when enqueuing the tokens.
2021-05-23 12:39:00 +02:00
m_queued_tokens.enqueue(move(m_current_token));
LibWeb: Fully implement all DOCTYPE tokenizer states Also fixes TagOpen having a seperate emit and reconsume in ANYTHING_ELSE.
2020-06-11 05:00:45 +01:00
EMIT_EOF;
LibWeb: Implement a bit more of DOCTYPE tokenization
2020-05-23 19:56:07 +02:00
}
ANYTHING_ELSE
{
LibWeb: Utilize SourceLocation for HTMLTokenizer logging
2021-04-24 20:22:30 -07:00
log_parse_error();
LibWeb: Fully implement all DOCTYPE tokenizer states Also fixes TagOpen having a seperate emit and reconsume in ANYTHING_ELSE.
2020-06-11 05:00:45 +01:00
RECONSUME_IN(BeforeDOCTYPEName);
LibWeb: Implement a bit more of DOCTYPE tokenization
2020-05-23 19:56:07 +02:00
}
LibWeb: Begin work on a spec-compliant HTML parser In order to actually view the web as it is, we're gonna need a proper HTML parser. So let's build one! This patch introduces the Web::HTMLTokenizer class, which currently operates on a StringView input stream where it fetches (ASCII only atm) codepoints and tokenizes acccording to the HTML spec tokenization algo. The tokenizer state machine looks a bit weird but is written in a way that tries to mimic the spec as closely as possible, in order to make development easier and bugs less likely. This initial version is far from finished, but it can parse a trivial document with a DOCTYPE and open/close tags. :^)
2020-05-22 21:46:13 +02:00
}
END_STATE
LibWeb: Add spec links to each HTML tokenizer state section I didn't add full spec comments this time, but this is better than nothing :^)
2022-02-15 18:12:15 +00:00
// 13.2.5.54 Before DOCTYPE name state, https://html.spec.whatwg.org/multipage/parsing.html#before-doctype-name-state
LibWeb: Begin work on a spec-compliant HTML parser In order to actually view the web as it is, we're gonna need a proper HTML parser. So let's build one! This patch introduces the Web::HTMLTokenizer class, which currently operates on a StringView input stream where it fetches (ASCII only atm) codepoints and tokenizes acccording to the HTML spec tokenization algo. The tokenizer state machine looks a bit weird but is written in a way that tries to mimic the spec as closely as possible, in order to make development easier and bugs less likely. This initial version is far from finished, but it can parse a trivial document with a DOCTYPE and open/close tags. :^)
2020-05-22 21:46:13 +02:00
BEGIN_STATE(BeforeDOCTYPEName)
{
ON_WHITESPACE
{
continue;
}
LibWeb: Implement a bit more of DOCTYPE tokenization
2020-05-23 19:56:07 +02:00
ON_ASCII_UPPER_ALPHA
{
create_new_token(HTMLToken::Type::DOCTYPE);
LibWeb: Remove StringBuilders from HTMLToken::m_doctype
2021-05-23 08:20:03 +02:00
m_current_builder.append_code_point(to_ascii_lowercase(current_input_character.value()));
LibWeb: Rename HTMLToken::doctype_data() => ensure_doctype_data() This renames the accessor to better reflect what it does, as this will allocate a DoctypeData struct if there is none.
2021-07-15 18:58:11 +02:00
m_current_token.ensure_doctype_data().missing_name = false;
LibWeb: Remove StringBuilders from HTMLToken::m_doctype
2021-05-23 08:20:03 +02:00
SWITCH_TO_WITH_UNCLEAN_BUILDER(DOCTYPEName);
LibWeb: Implement a bit more of DOCTYPE tokenization
2020-05-23 19:56:07 +02:00
}
ON(0)
{
LibWeb: Utilize SourceLocation for HTMLTokenizer logging
2021-04-24 20:22:30 -07:00
log_parse_error();
LibWeb: Fully implement all DOCTYPE tokenizer states Also fixes TagOpen having a seperate emit and reconsume in ANYTHING_ELSE.
2020-06-11 05:00:45 +01:00
create_new_token(HTMLToken::Type::DOCTYPE);
LibWeb: Remove StringBuilders from HTMLToken::m_doctype
2021-05-23 08:20:03 +02:00
m_current_builder.append_code_point(0xFFFD);
LibWeb: Rename HTMLToken::doctype_data() => ensure_doctype_data() This renames the accessor to better reflect what it does, as this will allocate a DoctypeData struct if there is none.
2021-07-15 18:58:11 +02:00
m_current_token.ensure_doctype_data().missing_name = false;
LibWeb: Remove StringBuilders from HTMLToken::m_doctype
2021-05-23 08:20:03 +02:00
SWITCH_TO_WITH_UNCLEAN_BUILDER(DOCTYPEName);
LibWeb: Implement a bit more of DOCTYPE tokenization
2020-05-23 19:56:07 +02:00
}
ON('>')
{
LibWeb: Utilize SourceLocation for HTMLTokenizer logging
2021-04-24 20:22:30 -07:00
log_parse_error();
LibWeb: Fully implement all DOCTYPE tokenizer states Also fixes TagOpen having a seperate emit and reconsume in ANYTHING_ELSE.
2020-06-11 05:00:45 +01:00
create_new_token(HTMLToken::Type::DOCTYPE);
LibWeb: Rename HTMLToken::doctype_data() => ensure_doctype_data() This renames the accessor to better reflect what it does, as this will allocate a DoctypeData struct if there is none.
2021-07-15 18:58:11 +02:00
m_current_token.ensure_doctype_data().force_quirks = true;
LibWeb: Fully implement all DOCTYPE tokenizer states Also fixes TagOpen having a seperate emit and reconsume in ANYTHING_ELSE.
2020-06-11 05:00:45 +01:00
SWITCH_TO_AND_EMIT_CURRENT_TOKEN(Data);
LibWeb: Implement a bit more of DOCTYPE tokenization
2020-05-23 19:56:07 +02:00
}
ON_EOF
{
LibWeb: Utilize SourceLocation for HTMLTokenizer logging
2021-04-24 20:22:30 -07:00
log_parse_error();
LibWeb: Fully implement all DOCTYPE tokenizer states Also fixes TagOpen having a seperate emit and reconsume in ANYTHING_ELSE.
2020-06-11 05:00:45 +01:00
create_new_token(HTMLToken::Type::DOCTYPE);
LibWeb: Rename HTMLToken::doctype_data() => ensure_doctype_data() This renames the accessor to better reflect what it does, as this will allocate a DoctypeData struct if there is none.
2021-07-15 18:58:11 +02:00
m_current_token.ensure_doctype_data().force_quirks = true;
LibWeb: Use move() when enqueuing tokens in HTMLTokenizer We're not using the current token anymore once it's enqueued so let's use move() when enqueuing the tokens.
2021-05-23 12:39:00 +02:00
m_queued_tokens.enqueue(move(m_current_token));
LibWeb: Fully implement all DOCTYPE tokenizer states Also fixes TagOpen having a seperate emit and reconsume in ANYTHING_ELSE.
2020-06-11 05:00:45 +01:00
EMIT_EOF;
LibWeb: Implement a bit more of DOCTYPE tokenization
2020-05-23 19:56:07 +02:00
}
LibWeb: Begin work on a spec-compliant HTML parser In order to actually view the web as it is, we're gonna need a proper HTML parser. So let's build one! This patch introduces the Web::HTMLTokenizer class, which currently operates on a StringView input stream where it fetches (ASCII only atm) codepoints and tokenizes acccording to the HTML spec tokenization algo. The tokenizer state machine looks a bit weird but is written in a way that tries to mimic the spec as closely as possible, in order to make development easier and bugs less likely. This initial version is far from finished, but it can parse a trivial document with a DOCTYPE and open/close tags. :^)
2020-05-22 21:46:13 +02:00
ANYTHING_ELSE
{
create_new_token(HTMLToken::Type::DOCTYPE);
LibWeb: Remove StringBuilders from HTMLToken::m_doctype
2021-05-23 08:20:03 +02:00
m_current_builder.append_code_point(current_input_character.value());
LibWeb: Rename HTMLToken::doctype_data() => ensure_doctype_data() This renames the accessor to better reflect what it does, as this will allocate a DoctypeData struct if there is none.
2021-07-15 18:58:11 +02:00
m_current_token.ensure_doctype_data().missing_name = false;
LibWeb: Remove StringBuilders from HTMLToken::m_doctype
2021-05-23 08:20:03 +02:00
SWITCH_TO_WITH_UNCLEAN_BUILDER(DOCTYPEName);
LibWeb: Begin work on a spec-compliant HTML parser In order to actually view the web as it is, we're gonna need a proper HTML parser. So let's build one! This patch introduces the Web::HTMLTokenizer class, which currently operates on a StringView input stream where it fetches (ASCII only atm) codepoints and tokenizes acccording to the HTML spec tokenization algo. The tokenizer state machine looks a bit weird but is written in a way that tries to mimic the spec as closely as possible, in order to make development easier and bugs less likely. This initial version is far from finished, but it can parse a trivial document with a DOCTYPE and open/close tags. :^)
2020-05-22 21:46:13 +02:00
}
}
END_STATE
LibWeb: Add spec links to each HTML tokenizer state section I didn't add full spec comments this time, but this is better than nothing :^)
2022-02-15 18:12:15 +00:00
// 13.2.5.55 DOCTYPE name state, https://html.spec.whatwg.org/multipage/parsing.html#doctype-name-state
LibWeb: Begin work on a spec-compliant HTML parser In order to actually view the web as it is, we're gonna need a proper HTML parser. So let's build one! This patch introduces the Web::HTMLTokenizer class, which currently operates on a StringView input stream where it fetches (ASCII only atm) codepoints and tokenizes acccording to the HTML spec tokenization algo. The tokenizer state machine looks a bit weird but is written in a way that tries to mimic the spec as closely as possible, in order to make development easier and bugs less likely. This initial version is far from finished, but it can parse a trivial document with a DOCTYPE and open/close tags. :^)
2020-05-22 21:46:13 +02:00
BEGIN_STATE(DOCTYPEName)
{
LibWeb: Implement a bit more of DOCTYPE tokenization
2020-05-23 19:56:07 +02:00
ON_WHITESPACE
{
LibWeb: Rename HTMLToken::doctype_data() => ensure_doctype_data() This renames the accessor to better reflect what it does, as this will allocate a DoctypeData struct if there is none.
2021-07-15 18:58:11 +02:00
m_current_token.ensure_doctype_data().name = consume_current_builder();
LibWeb: Implement a bit more of DOCTYPE tokenization
2020-05-23 19:56:07 +02:00
SWITCH_TO(AfterDOCTYPEName);
}
LibWeb: Begin work on a spec-compliant HTML parser In order to actually view the web as it is, we're gonna need a proper HTML parser. So let's build one! This patch introduces the Web::HTMLTokenizer class, which currently operates on a StringView input stream where it fetches (ASCII only atm) codepoints and tokenizes acccording to the HTML spec tokenization algo. The tokenizer state machine looks a bit weird but is written in a way that tries to mimic the spec as closely as possible, in order to make development easier and bugs less likely. This initial version is far from finished, but it can parse a trivial document with a DOCTYPE and open/close tags. :^)
2020-05-22 21:46:13 +02:00
ON('>')
{
LibWeb: Rename HTMLToken::doctype_data() => ensure_doctype_data() This renames the accessor to better reflect what it does, as this will allocate a DoctypeData struct if there is none.
2021-07-15 18:58:11 +02:00
m_current_token.ensure_doctype_data().name = consume_current_builder();
LibWeb: Start building the tree building part of the new HTML parser This patch adds a new HTMLDocumentParser class. It keeps a tokenizer object internally and feeds itself with one token at a time from it. The names and idioms in this class are expressed as closely to the actual HTML parsing spec as possible, to make development as easy and bug free as possible. :^) This is going to become pretty large, but it's pretty cool!
2020-05-24 00:14:23 +02:00
SWITCH_TO_AND_EMIT_CURRENT_TOKEN(Data);
LibWeb: Begin work on a spec-compliant HTML parser In order to actually view the web as it is, we're gonna need a proper HTML parser. So let's build one! This patch introduces the Web::HTMLTokenizer class, which currently operates on a StringView input stream where it fetches (ASCII only atm) codepoints and tokenizes acccording to the HTML spec tokenization algo. The tokenizer state machine looks a bit weird but is written in a way that tries to mimic the spec as closely as possible, in order to make development easier and bugs less likely. This initial version is far from finished, but it can parse a trivial document with a DOCTYPE and open/close tags. :^)
2020-05-22 21:46:13 +02:00
}
LibWeb: Implement a bit more of DOCTYPE tokenization
2020-05-23 19:56:07 +02:00
ON_ASCII_UPPER_ALPHA
{
LibWeb: Remove StringBuilders from HTMLToken::m_doctype
2021-05-23 08:20:03 +02:00
m_current_builder.append_code_point(to_ascii_lowercase(current_input_character.value()));
LibWeb: Fully implement all DOCTYPE tokenizer states Also fixes TagOpen having a seperate emit and reconsume in ANYTHING_ELSE.
2020-06-11 05:00:45 +01:00
continue;
LibWeb: Implement a bit more of DOCTYPE tokenization
2020-05-23 19:56:07 +02:00
}
ON(0)
{
LibWeb: Utilize SourceLocation for HTMLTokenizer logging
2021-04-24 20:22:30 -07:00
log_parse_error();
LibWeb: Remove StringBuilders from HTMLToken::m_doctype
2021-05-23 08:20:03 +02:00
m_current_builder.append_code_point(0xFFFD);
LibWeb: Fully implement all DOCTYPE tokenizer states Also fixes TagOpen having a seperate emit and reconsume in ANYTHING_ELSE.
2020-06-11 05:00:45 +01:00
continue;
LibWeb: Implement a bit more of DOCTYPE tokenization
2020-05-23 19:56:07 +02:00
}
ON_EOF
{
LibWeb: Utilize SourceLocation for HTMLTokenizer logging
2021-04-24 20:22:30 -07:00
log_parse_error();
LibWeb: Rename HTMLToken::doctype_data() => ensure_doctype_data() This renames the accessor to better reflect what it does, as this will allocate a DoctypeData struct if there is none.
2021-07-15 18:58:11 +02:00
m_current_token.ensure_doctype_data().force_quirks = true;
LibWeb: Use move() when enqueuing tokens in HTMLTokenizer We're not using the current token anymore once it's enqueued so let's use move() when enqueuing the tokens.
2021-05-23 12:39:00 +02:00
m_queued_tokens.enqueue(move(m_current_token));
LibWeb: Fully implement all DOCTYPE tokenizer states Also fixes TagOpen having a seperate emit and reconsume in ANYTHING_ELSE.
2020-06-11 05:00:45 +01:00
EMIT_EOF;
LibWeb: Implement a bit more of DOCTYPE tokenization
2020-05-23 19:56:07 +02:00
}
LibWeb: Begin work on a spec-compliant HTML parser In order to actually view the web as it is, we're gonna need a proper HTML parser. So let's build one! This patch introduces the Web::HTMLTokenizer class, which currently operates on a StringView input stream where it fetches (ASCII only atm) codepoints and tokenizes acccording to the HTML spec tokenization algo. The tokenizer state machine looks a bit weird but is written in a way that tries to mimic the spec as closely as possible, in order to make development easier and bugs less likely. This initial version is far from finished, but it can parse a trivial document with a DOCTYPE and open/close tags. :^)
2020-05-22 21:46:13 +02:00
ANYTHING_ELSE
{
LibWeb: Remove StringBuilders from HTMLToken::m_doctype
2021-05-23 08:20:03 +02:00
m_current_builder.append_code_point(current_input_character.value());
LibWeb: Begin work on a spec-compliant HTML parser In order to actually view the web as it is, we're gonna need a proper HTML parser. So let's build one! This patch introduces the Web::HTMLTokenizer class, which currently operates on a StringView input stream where it fetches (ASCII only atm) codepoints and tokenizes acccording to the HTML spec tokenization algo. The tokenizer state machine looks a bit weird but is written in a way that tries to mimic the spec as closely as possible, in order to make development easier and bugs less likely. This initial version is far from finished, but it can parse a trivial document with a DOCTYPE and open/close tags. :^)
2020-05-22 21:46:13 +02:00
continue;
}
LibWeb: Implement a bit more of DOCTYPE tokenization
2020-05-23 19:56:07 +02:00
}
END_STATE
LibWeb: Add spec links to each HTML tokenizer state section I didn't add full spec comments this time, but this is better than nothing :^)
2022-02-15 18:12:15 +00:00
// 13.2.5.56 After DOCTYPE name state, https://html.spec.whatwg.org/multipage/parsing.html#after-doctype-name-state
LibWeb: Implement a bit more of DOCTYPE tokenization
2020-05-23 19:56:07 +02:00
BEGIN_STATE(AfterDOCTYPEName)
LibWeb: Flesh out the remaining DOCTYPE related tokenizer states We can now parse public and system identifiers! Not super useful, but at least we can do it :^)
2020-05-25 19:50:44 +02:00
{
ON_WHITESPACE
{
continue;
}
ON('>')
{
SWITCH_TO_AND_EMIT_CURRENT_TOKEN(Data);
}
ON_EOF
{
LibWeb: Utilize SourceLocation for HTMLTokenizer logging
2021-04-24 20:22:30 -07:00
log_parse_error();
LibWeb: Rename HTMLToken::doctype_data() => ensure_doctype_data() This renames the accessor to better reflect what it does, as this will allocate a DoctypeData struct if there is none.
2021-07-15 18:58:11 +02:00
m_current_token.ensure_doctype_data().force_quirks = true;
LibWeb: Use move() when enqueuing tokens in HTMLTokenizer We're not using the current token anymore once it's enqueued so let's use move() when enqueuing the tokens.
2021-05-23 12:39:00 +02:00
m_queued_tokens.enqueue(move(m_current_token));
LibWeb: Fully implement all DOCTYPE tokenizer states Also fixes TagOpen having a seperate emit and reconsume in ANYTHING_ELSE.
2020-06-11 05:00:45 +01:00
EMIT_EOF;
LibWeb: Flesh out the remaining DOCTYPE related tokenizer states We can now parse public and system identifiers! Not super useful, but at least we can do it :^)
2020-05-25 19:50:44 +02:00
}
ANYTHING_ELSE
{
LibWeb: Teach more of HTMLTokenizer to stop at the insertion point In particular, input character lookahead now knows how to stop at the insertion point marker if needed. This makes it possible to do amazing things like having document.write() insert doctypes one character at a time.
2024-11-24 10:18:17 +01:00
if (to_ascii_uppercase(current_input_character.value()) == 'P') {
switch (consume_next_if_match("UBLIC"sv, stop_at_insertion_point, CaseSensitivity::CaseInsensitive)) {
case ConsumeNextResult::Consumed:
SWITCH_TO(AfterDOCTYPEPublicKeyword);
break;
case ConsumeNextResult::NotConsumed:
break;
case ConsumeNextResult::RanOutOfCharacters:
DONT_CONSUME_NEXT_INPUT_CHARACTER;
return {};
}
LibWeb: Flesh out the remaining DOCTYPE related tokenizer states We can now parse public and system identifiers! Not super useful, but at least we can do it :^)
2020-05-25 19:50:44 +02:00
}
LibWeb: Teach more of HTMLTokenizer to stop at the insertion point In particular, input character lookahead now knows how to stop at the insertion point marker if needed. This makes it possible to do amazing things like having document.write() insert doctypes one character at a time.
2024-11-24 10:18:17 +01:00
if (to_ascii_uppercase(current_input_character.value()) == 'S') {
switch (consume_next_if_match("YSTEM"sv, stop_at_insertion_point, CaseSensitivity::CaseInsensitive)) {
case ConsumeNextResult::Consumed:
SWITCH_TO(AfterDOCTYPESystemKeyword);
break;
case ConsumeNextResult::NotConsumed:
break;
case ConsumeNextResult::RanOutOfCharacters:
DONT_CONSUME_NEXT_INPUT_CHARACTER;
return {};
}
LibWeb: Flesh out the remaining DOCTYPE related tokenizer states We can now parse public and system identifiers! Not super useful, but at least we can do it :^)
2020-05-25 19:50:44 +02:00
}
LibWeb: Utilize SourceLocation for HTMLTokenizer logging
2021-04-24 20:22:30 -07:00
log_parse_error();
LibWeb: Rename HTMLToken::doctype_data() => ensure_doctype_data() This renames the accessor to better reflect what it does, as this will allocate a DoctypeData struct if there is none.
2021-07-15 18:58:11 +02:00
m_current_token.ensure_doctype_data().force_quirks = true;
LibWeb: Fully implement all DOCTYPE tokenizer states Also fixes TagOpen having a seperate emit and reconsume in ANYTHING_ELSE.
2020-06-11 05:00:45 +01:00
RECONSUME_IN(BogusDOCTYPE);
LibWeb: Flesh out the remaining DOCTYPE related tokenizer states We can now parse public and system identifiers! Not super useful, but at least we can do it :^)
2020-05-25 19:50:44 +02:00
}
}
END_STATE
LibWeb: Add spec links to each HTML tokenizer state section I didn't add full spec comments this time, but this is better than nothing :^)
2022-02-15 18:12:15 +00:00
// 13.2.5.57 After DOCTYPE public keyword state, https://html.spec.whatwg.org/multipage/parsing.html#after-doctype-public-keyword-state
LibWeb: Flesh out the remaining DOCTYPE related tokenizer states We can now parse public and system identifiers! Not super useful, but at least we can do it :^)
2020-05-25 19:50:44 +02:00
BEGIN_STATE(AfterDOCTYPEPublicKeyword)
{
ON_WHITESPACE
{
SWITCH_TO(BeforeDOCTYPEPublicIdentifier);
}
ON('"')
{
LibWeb: Utilize SourceLocation for HTMLTokenizer logging
2021-04-24 20:22:30 -07:00
log_parse_error();
LibWeb: Rename HTMLToken::doctype_data() => ensure_doctype_data() This renames the accessor to better reflect what it does, as this will allocate a DoctypeData struct if there is none.
2021-07-15 18:58:11 +02:00
m_current_token.ensure_doctype_data().missing_public_identifier = false;
LibWeb: Fully implement all DOCTYPE tokenizer states Also fixes TagOpen having a seperate emit and reconsume in ANYTHING_ELSE.
2020-06-11 05:00:45 +01:00
SWITCH_TO(DOCTYPEPublicIdentifierDoubleQuoted);
LibWeb: Flesh out the remaining DOCTYPE related tokenizer states We can now parse public and system identifiers! Not super useful, but at least we can do it :^)
2020-05-25 19:50:44 +02:00
}
ON('\'')
{
LibWeb: Utilize SourceLocation for HTMLTokenizer logging
2021-04-24 20:22:30 -07:00
log_parse_error();
LibWeb: Rename HTMLToken::doctype_data() => ensure_doctype_data() This renames the accessor to better reflect what it does, as this will allocate a DoctypeData struct if there is none.
2021-07-15 18:58:11 +02:00
m_current_token.ensure_doctype_data().missing_public_identifier = false;
LibWeb: Fully implement all DOCTYPE tokenizer states Also fixes TagOpen having a seperate emit and reconsume in ANYTHING_ELSE.
2020-06-11 05:00:45 +01:00
SWITCH_TO(DOCTYPEPublicIdentifierSingleQuoted);
LibWeb: Flesh out the remaining DOCTYPE related tokenizer states We can now parse public and system identifiers! Not super useful, but at least we can do it :^)
2020-05-25 19:50:44 +02:00
}
ON('>')
{
LibWeb: Utilize SourceLocation for HTMLTokenizer logging
2021-04-24 20:22:30 -07:00
log_parse_error();
LibWeb: Rename HTMLToken::doctype_data() => ensure_doctype_data() This renames the accessor to better reflect what it does, as this will allocate a DoctypeData struct if there is none.
2021-07-15 18:58:11 +02:00
m_current_token.ensure_doctype_data().force_quirks = true;
LibWeb: Fully implement all DOCTYPE tokenizer states Also fixes TagOpen having a seperate emit and reconsume in ANYTHING_ELSE.
2020-06-11 05:00:45 +01:00
SWITCH_TO_AND_EMIT_CURRENT_TOKEN(Data);
LibWeb: Flesh out the remaining DOCTYPE related tokenizer states We can now parse public and system identifiers! Not super useful, but at least we can do it :^)
2020-05-25 19:50:44 +02:00
}
ON_EOF
{
LibWeb: Utilize SourceLocation for HTMLTokenizer logging
2021-04-24 20:22:30 -07:00
log_parse_error();
LibWeb: Rename HTMLToken::doctype_data() => ensure_doctype_data() This renames the accessor to better reflect what it does, as this will allocate a DoctypeData struct if there is none.
2021-07-15 18:58:11 +02:00
m_current_token.ensure_doctype_data().force_quirks = true;
LibWeb: Use move() when enqueuing tokens in HTMLTokenizer We're not using the current token anymore once it's enqueued so let's use move() when enqueuing the tokens.
2021-05-23 12:39:00 +02:00
m_queued_tokens.enqueue(move(m_current_token));
LibWeb: Fully implement all DOCTYPE tokenizer states Also fixes TagOpen having a seperate emit and reconsume in ANYTHING_ELSE.
2020-06-11 05:00:45 +01:00
EMIT_EOF;
LibWeb: Flesh out the remaining DOCTYPE related tokenizer states We can now parse public and system identifiers! Not super useful, but at least we can do it :^)
2020-05-25 19:50:44 +02:00
}
ANYTHING_ELSE
{
LibWeb: Utilize SourceLocation for HTMLTokenizer logging
2021-04-24 20:22:30 -07:00
log_parse_error();
LibWeb: Rename HTMLToken::doctype_data() => ensure_doctype_data() This renames the accessor to better reflect what it does, as this will allocate a DoctypeData struct if there is none.
2021-07-15 18:58:11 +02:00
m_current_token.ensure_doctype_data().force_quirks = true;
LibWeb: Fully implement all DOCTYPE tokenizer states Also fixes TagOpen having a seperate emit and reconsume in ANYTHING_ELSE.
2020-06-11 05:00:45 +01:00
RECONSUME_IN(BogusDOCTYPE);
LibWeb: Flesh out the remaining DOCTYPE related tokenizer states We can now parse public and system identifiers! Not super useful, but at least we can do it :^)
2020-05-25 19:50:44 +02:00
}
}
END_STATE
LibWeb: Add spec links to each HTML tokenizer state section I didn't add full spec comments this time, but this is better than nothing :^)
2022-02-15 18:12:15 +00:00
// 13.2.5.63 After DOCTYPE system keyword state, https://html.spec.whatwg.org/multipage/parsing.html#after-doctype-system-keyword-state
LibWeb: Flesh out the remaining DOCTYPE related tokenizer states We can now parse public and system identifiers! Not super useful, but at least we can do it :^)
2020-05-25 19:50:44 +02:00
BEGIN_STATE(AfterDOCTYPESystemKeyword)
{
ON_WHITESPACE
{
SWITCH_TO(BeforeDOCTYPESystemIdentifier);
}
ON('"')
{
LibWeb: Utilize SourceLocation for HTMLTokenizer logging
2021-04-24 20:22:30 -07:00
log_parse_error();
LibWeb: Rename HTMLToken::doctype_data() => ensure_doctype_data() This renames the accessor to better reflect what it does, as this will allocate a DoctypeData struct if there is none.
2021-07-15 18:58:11 +02:00
m_current_token.ensure_doctype_data().system_identifier = {};
m_current_token.ensure_doctype_data().missing_system_identifier = false;
LibWeb: Fully implement all DOCTYPE tokenizer states Also fixes TagOpen having a seperate emit and reconsume in ANYTHING_ELSE.
2020-06-11 05:00:45 +01:00
SWITCH_TO(DOCTYPESystemIdentifierDoubleQuoted);
LibWeb: Flesh out the remaining DOCTYPE related tokenizer states We can now parse public and system identifiers! Not super useful, but at least we can do it :^)
2020-05-25 19:50:44 +02:00
}
ON('\'')
{
LibWeb: Utilize SourceLocation for HTMLTokenizer logging
2021-04-24 20:22:30 -07:00
log_parse_error();
LibWeb: Rename HTMLToken::doctype_data() => ensure_doctype_data() This renames the accessor to better reflect what it does, as this will allocate a DoctypeData struct if there is none.
2021-07-15 18:58:11 +02:00
m_current_token.ensure_doctype_data().system_identifier = {};
m_current_token.ensure_doctype_data().missing_system_identifier = false;
LibWeb: Fully implement all DOCTYPE tokenizer states Also fixes TagOpen having a seperate emit and reconsume in ANYTHING_ELSE.
2020-06-11 05:00:45 +01:00
SWITCH_TO(DOCTYPESystemIdentifierSingleQuoted);
LibWeb: Flesh out the remaining DOCTYPE related tokenizer states We can now parse public and system identifiers! Not super useful, but at least we can do it :^)
2020-05-25 19:50:44 +02:00
}
ON('>')
{
LibWeb: Utilize SourceLocation for HTMLTokenizer logging
2021-04-24 20:22:30 -07:00
log_parse_error();
LibWeb: Rename HTMLToken::doctype_data() => ensure_doctype_data() This renames the accessor to better reflect what it does, as this will allocate a DoctypeData struct if there is none.
2021-07-15 18:58:11 +02:00
m_current_token.ensure_doctype_data().force_quirks = true;
LibWeb: Fully implement all DOCTYPE tokenizer states Also fixes TagOpen having a seperate emit and reconsume in ANYTHING_ELSE.
2020-06-11 05:00:45 +01:00
SWITCH_TO_AND_EMIT_CURRENT_TOKEN(Data);
LibWeb: Flesh out the remaining DOCTYPE related tokenizer states We can now parse public and system identifiers! Not super useful, but at least we can do it :^)
2020-05-25 19:50:44 +02:00
}
ON_EOF
{
LibWeb: Utilize SourceLocation for HTMLTokenizer logging
2021-04-24 20:22:30 -07:00
log_parse_error();
LibWeb: Rename HTMLToken::doctype_data() => ensure_doctype_data() This renames the accessor to better reflect what it does, as this will allocate a DoctypeData struct if there is none.
2021-07-15 18:58:11 +02:00
m_current_token.ensure_doctype_data().force_quirks = true;
LibWeb: Use move() when enqueuing tokens in HTMLTokenizer We're not using the current token anymore once it's enqueued so let's use move() when enqueuing the tokens.
2021-05-23 12:39:00 +02:00
m_queued_tokens.enqueue(move(m_current_token));
LibWeb: Fully implement all DOCTYPE tokenizer states Also fixes TagOpen having a seperate emit and reconsume in ANYTHING_ELSE.
2020-06-11 05:00:45 +01:00
EMIT_EOF;
LibWeb: Flesh out the remaining DOCTYPE related tokenizer states We can now parse public and system identifiers! Not super useful, but at least we can do it :^)
2020-05-25 19:50:44 +02:00
}
ANYTHING_ELSE
{
LibWeb: Utilize SourceLocation for HTMLTokenizer logging
2021-04-24 20:22:30 -07:00
log_parse_error();
LibWeb: Rename HTMLToken::doctype_data() => ensure_doctype_data() This renames the accessor to better reflect what it does, as this will allocate a DoctypeData struct if there is none.
2021-07-15 18:58:11 +02:00
m_current_token.ensure_doctype_data().force_quirks = true;
LibWeb: Fully implement all DOCTYPE tokenizer states Also fixes TagOpen having a seperate emit and reconsume in ANYTHING_ELSE.
2020-06-11 05:00:45 +01:00
RECONSUME_IN(BogusDOCTYPE);
LibWeb: Flesh out the remaining DOCTYPE related tokenizer states We can now parse public and system identifiers! Not super useful, but at least we can do it :^)
2020-05-25 19:50:44 +02:00
}
}
END_STATE
LibWeb: Add spec links to each HTML tokenizer state section I didn't add full spec comments this time, but this is better than nothing :^)
2022-02-15 18:12:15 +00:00
// 13.2.5.58 Before DOCTYPE public identifier state, https://html.spec.whatwg.org/multipage/parsing.html#before-doctype-public-identifier-state
LibWeb: Flesh out the remaining DOCTYPE related tokenizer states We can now parse public and system identifiers! Not super useful, but at least we can do it :^)
2020-05-25 19:50:44 +02:00
BEGIN_STATE(BeforeDOCTYPEPublicIdentifier)
{
ON_WHITESPACE
{
continue;
}
ON('"')
{
LibWeb: Rename HTMLToken::doctype_data() => ensure_doctype_data() This renames the accessor to better reflect what it does, as this will allocate a DoctypeData struct if there is none.
2021-07-15 18:58:11 +02:00
m_current_token.ensure_doctype_data().missing_public_identifier = false;
LibWeb: Flesh out the remaining DOCTYPE related tokenizer states We can now parse public and system identifiers! Not super useful, but at least we can do it :^)
2020-05-25 19:50:44 +02:00
SWITCH_TO(DOCTYPEPublicIdentifierDoubleQuoted);
}
ON('\'')
{
LibWeb: Rename HTMLToken::doctype_data() => ensure_doctype_data() This renames the accessor to better reflect what it does, as this will allocate a DoctypeData struct if there is none.
2021-07-15 18:58:11 +02:00
m_current_token.ensure_doctype_data().missing_public_identifier = false;
LibWeb: Flesh out the remaining DOCTYPE related tokenizer states We can now parse public and system identifiers! Not super useful, but at least we can do it :^)
2020-05-25 19:50:44 +02:00
SWITCH_TO(DOCTYPEPublicIdentifierSingleQuoted);
}
ON('>')
{
LibWeb: Utilize SourceLocation for HTMLTokenizer logging
2021-04-24 20:22:30 -07:00
log_parse_error();
LibWeb: Rename HTMLToken::doctype_data() => ensure_doctype_data() This renames the accessor to better reflect what it does, as this will allocate a DoctypeData struct if there is none.
2021-07-15 18:58:11 +02:00
m_current_token.ensure_doctype_data().force_quirks = true;
LibWeb: Fully implement all DOCTYPE tokenizer states Also fixes TagOpen having a seperate emit and reconsume in ANYTHING_ELSE.
2020-06-11 05:00:45 +01:00
SWITCH_TO_AND_EMIT_CURRENT_TOKEN(Data);
LibWeb: Flesh out the remaining DOCTYPE related tokenizer states We can now parse public and system identifiers! Not super useful, but at least we can do it :^)
2020-05-25 19:50:44 +02:00
}
ON_EOF
{
LibWeb: Utilize SourceLocation for HTMLTokenizer logging
2021-04-24 20:22:30 -07:00
log_parse_error();
LibWeb: Rename HTMLToken::doctype_data() => ensure_doctype_data() This renames the accessor to better reflect what it does, as this will allocate a DoctypeData struct if there is none.
2021-07-15 18:58:11 +02:00
m_current_token.ensure_doctype_data().force_quirks = true;
LibWeb: Use move() when enqueuing tokens in HTMLTokenizer We're not using the current token anymore once it's enqueued so let's use move() when enqueuing the tokens.
2021-05-23 12:39:00 +02:00
m_queued_tokens.enqueue(move(m_current_token));
LibWeb: Fully implement all DOCTYPE tokenizer states Also fixes TagOpen having a seperate emit and reconsume in ANYTHING_ELSE.
2020-06-11 05:00:45 +01:00
EMIT_EOF;
LibWeb: Flesh out the remaining DOCTYPE related tokenizer states We can now parse public and system identifiers! Not super useful, but at least we can do it :^)
2020-05-25 19:50:44 +02:00
}
ANYTHING_ELSE
{
LibWeb: Utilize SourceLocation for HTMLTokenizer logging
2021-04-24 20:22:30 -07:00
log_parse_error();
LibWeb: Rename HTMLToken::doctype_data() => ensure_doctype_data() This renames the accessor to better reflect what it does, as this will allocate a DoctypeData struct if there is none.
2021-07-15 18:58:11 +02:00
m_current_token.ensure_doctype_data().force_quirks = true;
LibWeb: Fully implement all DOCTYPE tokenizer states Also fixes TagOpen having a seperate emit and reconsume in ANYTHING_ELSE.
2020-06-11 05:00:45 +01:00
RECONSUME_IN(BogusDOCTYPE);
LibWeb: Flesh out the remaining DOCTYPE related tokenizer states We can now parse public and system identifiers! Not super useful, but at least we can do it :^)
2020-05-25 19:50:44 +02:00
}
}
END_STATE
LibWeb: Add spec links to each HTML tokenizer state section I didn't add full spec comments this time, but this is better than nothing :^)
2022-02-15 18:12:15 +00:00
// 13.2.5.64 Before DOCTYPE system identifier state, https://html.spec.whatwg.org/multipage/parsing.html#before-doctype-system-identifier-state
LibWeb: Flesh out the remaining DOCTYPE related tokenizer states We can now parse public and system identifiers! Not super useful, but at least we can do it :^)
2020-05-25 19:50:44 +02:00
BEGIN_STATE(BeforeDOCTYPESystemIdentifier)
{
ON_WHITESPACE
{
continue;
}
ON('"')
{
LibWeb: Rename HTMLToken::doctype_data() => ensure_doctype_data() This renames the accessor to better reflect what it does, as this will allocate a DoctypeData struct if there is none.
2021-07-15 18:58:11 +02:00
m_current_token.ensure_doctype_data().missing_system_identifier = false;
LibWeb: Flesh out the remaining DOCTYPE related tokenizer states We can now parse public and system identifiers! Not super useful, but at least we can do it :^)
2020-05-25 19:50:44 +02:00
SWITCH_TO(DOCTYPESystemIdentifierDoubleQuoted);
}
ON('\'')
{
LibWeb: Rename HTMLToken::doctype_data() => ensure_doctype_data() This renames the accessor to better reflect what it does, as this will allocate a DoctypeData struct if there is none.
2021-07-15 18:58:11 +02:00
m_current_token.ensure_doctype_data().missing_system_identifier = false;
LibWeb: Flesh out the remaining DOCTYPE related tokenizer states We can now parse public and system identifiers! Not super useful, but at least we can do it :^)
2020-05-25 19:50:44 +02:00
SWITCH_TO(DOCTYPESystemIdentifierSingleQuoted);
}
ON('>')
{
LibWeb: Utilize SourceLocation for HTMLTokenizer logging
2021-04-24 20:22:30 -07:00
log_parse_error();
LibWeb: Rename HTMLToken::doctype_data() => ensure_doctype_data() This renames the accessor to better reflect what it does, as this will allocate a DoctypeData struct if there is none.
2021-07-15 18:58:11 +02:00
m_current_token.ensure_doctype_data().force_quirks = true;
LibWeb: Fully implement all DOCTYPE tokenizer states Also fixes TagOpen having a seperate emit and reconsume in ANYTHING_ELSE.
2020-06-11 05:00:45 +01:00
SWITCH_TO_AND_EMIT_CURRENT_TOKEN(Data);
LibWeb: Flesh out the remaining DOCTYPE related tokenizer states We can now parse public and system identifiers! Not super useful, but at least we can do it :^)
2020-05-25 19:50:44 +02:00
}
ON_EOF
{
LibWeb: Utilize SourceLocation for HTMLTokenizer logging
2021-04-24 20:22:30 -07:00
log_parse_error();
LibWeb: Rename HTMLToken::doctype_data() => ensure_doctype_data() This renames the accessor to better reflect what it does, as this will allocate a DoctypeData struct if there is none.
2021-07-15 18:58:11 +02:00
m_current_token.ensure_doctype_data().force_quirks = true;
LibWeb: Use move() when enqueuing tokens in HTMLTokenizer We're not using the current token anymore once it's enqueued so let's use move() when enqueuing the tokens.
2021-05-23 12:39:00 +02:00
m_queued_tokens.enqueue(move(m_current_token));
LibWeb: Fully implement all DOCTYPE tokenizer states Also fixes TagOpen having a seperate emit and reconsume in ANYTHING_ELSE.
2020-06-11 05:00:45 +01:00
EMIT_EOF;
LibWeb: Flesh out the remaining DOCTYPE related tokenizer states We can now parse public and system identifiers! Not super useful, but at least we can do it :^)
2020-05-25 19:50:44 +02:00
}
ANYTHING_ELSE
{
LibWeb: Utilize SourceLocation for HTMLTokenizer logging
2021-04-24 20:22:30 -07:00
log_parse_error();
LibWeb: Rename HTMLToken::doctype_data() => ensure_doctype_data() This renames the accessor to better reflect what it does, as this will allocate a DoctypeData struct if there is none.
2021-07-15 18:58:11 +02:00
m_current_token.ensure_doctype_data().force_quirks = true;
LibWeb: Fully implement all DOCTYPE tokenizer states Also fixes TagOpen having a seperate emit and reconsume in ANYTHING_ELSE.
2020-06-11 05:00:45 +01:00
RECONSUME_IN(BogusDOCTYPE);
LibWeb: Flesh out the remaining DOCTYPE related tokenizer states We can now parse public and system identifiers! Not super useful, but at least we can do it :^)
2020-05-25 19:50:44 +02:00
}
}
END_STATE
LibWeb: Add spec links to each HTML tokenizer state section I didn't add full spec comments this time, but this is better than nothing :^)
2022-02-15 18:12:15 +00:00
// 13.2.5.59 DOCTYPE public identifier (double-quoted) state, https://html.spec.whatwg.org/multipage/parsing.html#doctype-public-identifier-(double-quoted)-state
LibWeb: Flesh out the remaining DOCTYPE related tokenizer states We can now parse public and system identifiers! Not super useful, but at least we can do it :^)
2020-05-25 19:50:44 +02:00
BEGIN_STATE(DOCTYPEPublicIdentifierDoubleQuoted)
{
ON('"')
{
LibWeb: Rename HTMLToken::doctype_data() => ensure_doctype_data() This renames the accessor to better reflect what it does, as this will allocate a DoctypeData struct if there is none.
2021-07-15 18:58:11 +02:00
m_current_token.ensure_doctype_data().public_identifier = consume_current_builder();
LibWeb: Flesh out the remaining DOCTYPE related tokenizer states We can now parse public and system identifiers! Not super useful, but at least we can do it :^)
2020-05-25 19:50:44 +02:00
SWITCH_TO(AfterDOCTYPEPublicIdentifier);
}
ON(0)
{
LibWeb: Utilize SourceLocation for HTMLTokenizer logging
2021-04-24 20:22:30 -07:00
log_parse_error();
LibWeb: Remove StringBuilders from HTMLToken::m_doctype
2021-05-23 08:20:03 +02:00
m_current_builder.append_code_point(0xFFFD);
LibWeb: Fully implement all DOCTYPE tokenizer states Also fixes TagOpen having a seperate emit and reconsume in ANYTHING_ELSE.
2020-06-11 05:00:45 +01:00
continue;
LibWeb: Flesh out the remaining DOCTYPE related tokenizer states We can now parse public and system identifiers! Not super useful, but at least we can do it :^)
2020-05-25 19:50:44 +02:00
}
ON('>')
{
LibWeb: Utilize SourceLocation for HTMLTokenizer logging
2021-04-24 20:22:30 -07:00
log_parse_error();
LibWeb: Rename HTMLToken::doctype_data() => ensure_doctype_data() This renames the accessor to better reflect what it does, as this will allocate a DoctypeData struct if there is none.
2021-07-15 18:58:11 +02:00
m_current_token.ensure_doctype_data().public_identifier = consume_current_builder();
m_current_token.ensure_doctype_data().force_quirks = true;
LibWeb: Fully implement all DOCTYPE tokenizer states Also fixes TagOpen having a seperate emit and reconsume in ANYTHING_ELSE.
2020-06-11 05:00:45 +01:00
SWITCH_TO_AND_EMIT_CURRENT_TOKEN(Data);
LibWeb: Flesh out the remaining DOCTYPE related tokenizer states We can now parse public and system identifiers! Not super useful, but at least we can do it :^)
2020-05-25 19:50:44 +02:00
}
ON_EOF
{
LibWeb: Utilize SourceLocation for HTMLTokenizer logging
2021-04-24 20:22:30 -07:00
log_parse_error();
LibWeb: Rename HTMLToken::doctype_data() => ensure_doctype_data() This renames the accessor to better reflect what it does, as this will allocate a DoctypeData struct if there is none.
2021-07-15 18:58:11 +02:00
m_current_token.ensure_doctype_data().force_quirks = true;
LibWeb: Use move() when enqueuing tokens in HTMLTokenizer We're not using the current token anymore once it's enqueued so let's use move() when enqueuing the tokens.
2021-05-23 12:39:00 +02:00
m_queued_tokens.enqueue(move(m_current_token));
LibWeb: Fully implement all DOCTYPE tokenizer states Also fixes TagOpen having a seperate emit and reconsume in ANYTHING_ELSE.
2020-06-11 05:00:45 +01:00
EMIT_EOF;
LibWeb: Flesh out the remaining DOCTYPE related tokenizer states We can now parse public and system identifiers! Not super useful, but at least we can do it :^)
2020-05-25 19:50:44 +02:00
}
ANYTHING_ELSE
{
LibWeb: Remove StringBuilders from HTMLToken::m_doctype
2021-05-23 08:20:03 +02:00
m_current_builder.append_code_point(current_input_character.value());
LibWeb: Flesh out the remaining DOCTYPE related tokenizer states We can now parse public and system identifiers! Not super useful, but at least we can do it :^)
2020-05-25 19:50:44 +02:00
continue;
}
}
END_STATE
LibWeb: Add spec links to each HTML tokenizer state section I didn't add full spec comments this time, but this is better than nothing :^)
2022-02-15 18:12:15 +00:00
// 13.2.5.60 DOCTYPE public identifier (single-quoted) state, https://html.spec.whatwg.org/multipage/parsing.html#doctype-public-identifier-(single-quoted)-state
LibWeb: Flesh out the remaining DOCTYPE related tokenizer states We can now parse public and system identifiers! Not super useful, but at least we can do it :^)
2020-05-25 19:50:44 +02:00
BEGIN_STATE(DOCTYPEPublicIdentifierSingleQuoted)
{
ON('\'')
{
LibWeb: Rename HTMLToken::doctype_data() => ensure_doctype_data() This renames the accessor to better reflect what it does, as this will allocate a DoctypeData struct if there is none.
2021-07-15 18:58:11 +02:00
m_current_token.ensure_doctype_data().public_identifier = consume_current_builder();
LibWeb: Flesh out the remaining DOCTYPE related tokenizer states We can now parse public and system identifiers! Not super useful, but at least we can do it :^)
2020-05-25 19:50:44 +02:00
SWITCH_TO(AfterDOCTYPEPublicIdentifier);
}
ON(0)
{
LibWeb: Utilize SourceLocation for HTMLTokenizer logging
2021-04-24 20:22:30 -07:00
log_parse_error();
LibWeb: Remove StringBuilders from HTMLToken::m_doctype
2021-05-23 08:20:03 +02:00
m_current_builder.append_code_point(0xFFFD);
LibWeb: Fully implement all DOCTYPE tokenizer states Also fixes TagOpen having a seperate emit and reconsume in ANYTHING_ELSE.
2020-06-11 05:00:45 +01:00
continue;
LibWeb: Flesh out the remaining DOCTYPE related tokenizer states We can now parse public and system identifiers! Not super useful, but at least we can do it :^)
2020-05-25 19:50:44 +02:00
}
ON('>')
{
LibWeb: Utilize SourceLocation for HTMLTokenizer logging
2021-04-24 20:22:30 -07:00
log_parse_error();
LibWeb: Rename HTMLToken::doctype_data() => ensure_doctype_data() This renames the accessor to better reflect what it does, as this will allocate a DoctypeData struct if there is none.
2021-07-15 18:58:11 +02:00
m_current_token.ensure_doctype_data().public_identifier = consume_current_builder();
m_current_token.ensure_doctype_data().force_quirks = true;
LibWeb: Fully implement all DOCTYPE tokenizer states Also fixes TagOpen having a seperate emit and reconsume in ANYTHING_ELSE.
2020-06-11 05:00:45 +01:00
SWITCH_TO_AND_EMIT_CURRENT_TOKEN(Data);
LibWeb: Flesh out the remaining DOCTYPE related tokenizer states We can now parse public and system identifiers! Not super useful, but at least we can do it :^)
2020-05-25 19:50:44 +02:00
}
ON_EOF
{
LibWeb: Utilize SourceLocation for HTMLTokenizer logging
2021-04-24 20:22:30 -07:00
log_parse_error();
LibWeb: Rename HTMLToken::doctype_data() => ensure_doctype_data() This renames the accessor to better reflect what it does, as this will allocate a DoctypeData struct if there is none.
2021-07-15 18:58:11 +02:00
m_current_token.ensure_doctype_data().force_quirks = true;
LibWeb: Use move() when enqueuing tokens in HTMLTokenizer We're not using the current token anymore once it's enqueued so let's use move() when enqueuing the tokens.
2021-05-23 12:39:00 +02:00
m_queued_tokens.enqueue(move(m_current_token));
LibWeb: Fully implement all DOCTYPE tokenizer states Also fixes TagOpen having a seperate emit and reconsume in ANYTHING_ELSE.
2020-06-11 05:00:45 +01:00
EMIT_EOF;
LibWeb: Flesh out the remaining DOCTYPE related tokenizer states We can now parse public and system identifiers! Not super useful, but at least we can do it :^)
2020-05-25 19:50:44 +02:00
}
ANYTHING_ELSE
{
LibWeb: Remove StringBuilders from HTMLToken::m_doctype
2021-05-23 08:20:03 +02:00
m_current_builder.append_code_point(current_input_character.value());
LibWeb: Flesh out the remaining DOCTYPE related tokenizer states We can now parse public and system identifiers! Not super useful, but at least we can do it :^)
2020-05-25 19:50:44 +02:00
continue;
}
}
END_STATE
LibWeb: Add spec links to each HTML tokenizer state section I didn't add full spec comments this time, but this is better than nothing :^)
2022-02-15 18:12:15 +00:00
// 13.2.5.65 DOCTYPE system identifier (double-quoted) state, https://html.spec.whatwg.org/multipage/parsing.html#doctype-system-identifier-(double-quoted)-state
LibWeb: Flesh out the remaining DOCTYPE related tokenizer states We can now parse public and system identifiers! Not super useful, but at least we can do it :^)
2020-05-25 19:50:44 +02:00
BEGIN_STATE(DOCTYPESystemIdentifierDoubleQuoted)
{
ON('"')
{
LibWeb: Fix issue where double-quoted doctype system ID was not captured We were storing double-quoted system ID's in the public ID field. 1% progression on ACID3. :^)
2022-03-02 12:26:31 +01:00
m_current_token.ensure_doctype_data().system_identifier = consume_current_builder();
LibWeb: Flesh out the remaining DOCTYPE related tokenizer states We can now parse public and system identifiers! Not super useful, but at least we can do it :^)
2020-05-25 19:50:44 +02:00
SWITCH_TO(AfterDOCTYPESystemIdentifier);
}
ON(0)
{
LibWeb: Utilize SourceLocation for HTMLTokenizer logging
2021-04-24 20:22:30 -07:00
log_parse_error();
LibWeb: Remove StringBuilders from HTMLToken::m_doctype
2021-05-23 08:20:03 +02:00
m_current_builder.append_code_point(0xFFFD);
LibWeb: Fully implement all DOCTYPE tokenizer states Also fixes TagOpen having a seperate emit and reconsume in ANYTHING_ELSE.
2020-06-11 05:00:45 +01:00
continue;
LibWeb: Flesh out the remaining DOCTYPE related tokenizer states We can now parse public and system identifiers! Not super useful, but at least we can do it :^)
2020-05-25 19:50:44 +02:00
}
ON('>')
{
LibWeb: Utilize SourceLocation for HTMLTokenizer logging
2021-04-24 20:22:30 -07:00
log_parse_error();
LibWeb: Fix issue where double-quoted doctype system ID was not captured We were storing double-quoted system ID's in the public ID field. 1% progression on ACID3. :^)
2022-03-02 12:26:31 +01:00
m_current_token.ensure_doctype_data().system_identifier = consume_current_builder();
LibWeb: Rename HTMLToken::doctype_data() => ensure_doctype_data() This renames the accessor to better reflect what it does, as this will allocate a DoctypeData struct if there is none.
2021-07-15 18:58:11 +02:00
m_current_token.ensure_doctype_data().force_quirks = true;
LibWeb: Fully implement all DOCTYPE tokenizer states Also fixes TagOpen having a seperate emit and reconsume in ANYTHING_ELSE.
2020-06-11 05:00:45 +01:00
SWITCH_TO_AND_EMIT_CURRENT_TOKEN(Data);
LibWeb: Flesh out the remaining DOCTYPE related tokenizer states We can now parse public and system identifiers! Not super useful, but at least we can do it :^)
2020-05-25 19:50:44 +02:00
}
ON_EOF
{
LibWeb: Utilize SourceLocation for HTMLTokenizer logging
2021-04-24 20:22:30 -07:00
log_parse_error();
LibWeb: Rename HTMLToken::doctype_data() => ensure_doctype_data() This renames the accessor to better reflect what it does, as this will allocate a DoctypeData struct if there is none.
2021-07-15 18:58:11 +02:00
m_current_token.ensure_doctype_data().force_quirks = true;
LibWeb: Use move() when enqueuing tokens in HTMLTokenizer We're not using the current token anymore once it's enqueued so let's use move() when enqueuing the tokens.
2021-05-23 12:39:00 +02:00
m_queued_tokens.enqueue(move(m_current_token));
LibWeb: Fully implement all DOCTYPE tokenizer states Also fixes TagOpen having a seperate emit and reconsume in ANYTHING_ELSE.
2020-06-11 05:00:45 +01:00
EMIT_EOF;
LibWeb: Flesh out the remaining DOCTYPE related tokenizer states We can now parse public and system identifiers! Not super useful, but at least we can do it :^)
2020-05-25 19:50:44 +02:00
}
ANYTHING_ELSE
{
LibWeb: Remove StringBuilders from HTMLToken::m_doctype
2021-05-23 08:20:03 +02:00
m_current_builder.append_code_point(current_input_character.value());
LibWeb: Flesh out the remaining DOCTYPE related tokenizer states We can now parse public and system identifiers! Not super useful, but at least we can do it :^)
2020-05-25 19:50:44 +02:00
continue;
}
}
END_STATE
LibWeb: Add spec links to each HTML tokenizer state section I didn't add full spec comments this time, but this is better than nothing :^)
2022-02-15 18:12:15 +00:00
// 13.2.5.66 DOCTYPE system identifier (single-quoted) state, https://html.spec.whatwg.org/multipage/parsing.html#doctype-system-identifier-(single-quoted)-state
LibWeb: Flesh out the remaining DOCTYPE related tokenizer states We can now parse public and system identifiers! Not super useful, but at least we can do it :^)
2020-05-25 19:50:44 +02:00
BEGIN_STATE(DOCTYPESystemIdentifierSingleQuoted)
{
ON('\'')
{
LibWeb: Rename HTMLToken::doctype_data() => ensure_doctype_data() This renames the accessor to better reflect what it does, as this will allocate a DoctypeData struct if there is none.
2021-07-15 18:58:11 +02:00
m_current_token.ensure_doctype_data().system_identifier = consume_current_builder();
LibWeb: Flesh out the remaining DOCTYPE related tokenizer states We can now parse public and system identifiers! Not super useful, but at least we can do it :^)
2020-05-25 19:50:44 +02:00
SWITCH_TO(AfterDOCTYPESystemIdentifier);
}
ON(0)
{
LibWeb: Utilize SourceLocation for HTMLTokenizer logging
2021-04-24 20:22:30 -07:00
log_parse_error();
LibWeb: Remove StringBuilders from HTMLToken::m_doctype
2021-05-23 08:20:03 +02:00
m_current_builder.append_code_point(0xFFFD);
LibWeb: Fully implement all DOCTYPE tokenizer states Also fixes TagOpen having a seperate emit and reconsume in ANYTHING_ELSE.
2020-06-11 05:00:45 +01:00
continue;
LibWeb: Flesh out the remaining DOCTYPE related tokenizer states We can now parse public and system identifiers! Not super useful, but at least we can do it :^)
2020-05-25 19:50:44 +02:00
}
ON('>')
{
LibWeb: Utilize SourceLocation for HTMLTokenizer logging
2021-04-24 20:22:30 -07:00
log_parse_error();
LibWeb: Rename HTMLToken::doctype_data() => ensure_doctype_data() This renames the accessor to better reflect what it does, as this will allocate a DoctypeData struct if there is none.
2021-07-15 18:58:11 +02:00
m_current_token.ensure_doctype_data().system_identifier = consume_current_builder();
m_current_token.ensure_doctype_data().force_quirks = true;
LibWeb: Fully implement all DOCTYPE tokenizer states Also fixes TagOpen having a seperate emit and reconsume in ANYTHING_ELSE.
2020-06-11 05:00:45 +01:00
SWITCH_TO_AND_EMIT_CURRENT_TOKEN(Data);
LibWeb: Flesh out the remaining DOCTYPE related tokenizer states We can now parse public and system identifiers! Not super useful, but at least we can do it :^)
2020-05-25 19:50:44 +02:00
}
ON_EOF
{
LibWeb: Utilize SourceLocation for HTMLTokenizer logging
2021-04-24 20:22:30 -07:00
log_parse_error();
LibWeb: Rename HTMLToken::doctype_data() => ensure_doctype_data() This renames the accessor to better reflect what it does, as this will allocate a DoctypeData struct if there is none.
2021-07-15 18:58:11 +02:00
m_current_token.ensure_doctype_data().force_quirks = true;
LibWeb: Use move() when enqueuing tokens in HTMLTokenizer We're not using the current token anymore once it's enqueued so let's use move() when enqueuing the tokens.
2021-05-23 12:39:00 +02:00
m_queued_tokens.enqueue(move(m_current_token));
LibWeb: Fully implement all DOCTYPE tokenizer states Also fixes TagOpen having a seperate emit and reconsume in ANYTHING_ELSE.
2020-06-11 05:00:45 +01:00
EMIT_EOF;
LibWeb: Flesh out the remaining DOCTYPE related tokenizer states We can now parse public and system identifiers! Not super useful, but at least we can do it :^)
2020-05-25 19:50:44 +02:00
}
ANYTHING_ELSE
{
LibWeb: Remove StringBuilders from HTMLToken::m_doctype
2021-05-23 08:20:03 +02:00
m_current_builder.append_code_point(current_input_character.value());
LibWeb: Flesh out the remaining DOCTYPE related tokenizer states We can now parse public and system identifiers! Not super useful, but at least we can do it :^)
2020-05-25 19:50:44 +02:00
continue;
}
}
END_STATE
LibWeb: Add spec links to each HTML tokenizer state section I didn't add full spec comments this time, but this is better than nothing :^)
2022-02-15 18:12:15 +00:00
// 13.2.5.61 After DOCTYPE public identifier state, https://html.spec.whatwg.org/multipage/parsing.html#after-doctype-public-identifier-state
LibWeb: Flesh out the remaining DOCTYPE related tokenizer states We can now parse public and system identifiers! Not super useful, but at least we can do it :^)
2020-05-25 19:50:44 +02:00
BEGIN_STATE(AfterDOCTYPEPublicIdentifier)
{
ON_WHITESPACE
{
SWITCH_TO(BetweenDOCTYPEPublicAndSystemIdentifiers);
}
ON('>')
{
SWITCH_TO_AND_EMIT_CURRENT_TOKEN(Data);
}
ON('"')
{
LibWeb: Utilize SourceLocation for HTMLTokenizer logging
2021-04-24 20:22:30 -07:00
log_parse_error();
LibWeb: Rename HTMLToken::doctype_data() => ensure_doctype_data() This renames the accessor to better reflect what it does, as this will allocate a DoctypeData struct if there is none.
2021-07-15 18:58:11 +02:00
m_current_token.ensure_doctype_data().missing_system_identifier = false;
LibWeb: Fully implement all DOCTYPE tokenizer states Also fixes TagOpen having a seperate emit and reconsume in ANYTHING_ELSE.
2020-06-11 05:00:45 +01:00
SWITCH_TO(DOCTYPESystemIdentifierDoubleQuoted);
LibWeb: Flesh out the remaining DOCTYPE related tokenizer states We can now parse public and system identifiers! Not super useful, but at least we can do it :^)
2020-05-25 19:50:44 +02:00
}
ON('\'')
{
LibWeb: Utilize SourceLocation for HTMLTokenizer logging
2021-04-24 20:22:30 -07:00
log_parse_error();
LibWeb: Rename HTMLToken::doctype_data() => ensure_doctype_data() This renames the accessor to better reflect what it does, as this will allocate a DoctypeData struct if there is none.
2021-07-15 18:58:11 +02:00
m_current_token.ensure_doctype_data().missing_system_identifier = false;
LibWeb: Fully implement all DOCTYPE tokenizer states Also fixes TagOpen having a seperate emit and reconsume in ANYTHING_ELSE.
2020-06-11 05:00:45 +01:00
SWITCH_TO(DOCTYPESystemIdentifierSingleQuoted);
LibWeb: Flesh out the remaining DOCTYPE related tokenizer states We can now parse public and system identifiers! Not super useful, but at least we can do it :^)
2020-05-25 19:50:44 +02:00
}
ON_EOF
{
LibWeb: Utilize SourceLocation for HTMLTokenizer logging
2021-04-24 20:22:30 -07:00
log_parse_error();
LibWeb: Rename HTMLToken::doctype_data() => ensure_doctype_data() This renames the accessor to better reflect what it does, as this will allocate a DoctypeData struct if there is none.
2021-07-15 18:58:11 +02:00
m_current_token.ensure_doctype_data().force_quirks = true;
LibWeb: Use move() when enqueuing tokens in HTMLTokenizer We're not using the current token anymore once it's enqueued so let's use move() when enqueuing the tokens.
2021-05-23 12:39:00 +02:00
m_queued_tokens.enqueue(move(m_current_token));
LibWeb: Fully implement all DOCTYPE tokenizer states Also fixes TagOpen having a seperate emit and reconsume in ANYTHING_ELSE.
2020-06-11 05:00:45 +01:00
EMIT_EOF;
LibWeb: Flesh out the remaining DOCTYPE related tokenizer states We can now parse public and system identifiers! Not super useful, but at least we can do it :^)
2020-05-25 19:50:44 +02:00
}
ANYTHING_ELSE
{
LibWeb: Utilize SourceLocation for HTMLTokenizer logging
2021-04-24 20:22:30 -07:00
log_parse_error();
LibWeb: Rename HTMLToken::doctype_data() => ensure_doctype_data() This renames the accessor to better reflect what it does, as this will allocate a DoctypeData struct if there is none.
2021-07-15 18:58:11 +02:00
m_current_token.ensure_doctype_data().force_quirks = true;
LibWeb: Fully implement all DOCTYPE tokenizer states Also fixes TagOpen having a seperate emit and reconsume in ANYTHING_ELSE.
2020-06-11 05:00:45 +01:00
RECONSUME_IN(BogusDOCTYPE);
LibWeb: Flesh out the remaining DOCTYPE related tokenizer states We can now parse public and system identifiers! Not super useful, but at least we can do it :^)
2020-05-25 19:50:44 +02:00
}
}
END_STATE
LibWeb: Add spec links to each HTML tokenizer state section I didn't add full spec comments this time, but this is better than nothing :^)
2022-02-15 18:12:15 +00:00
// 13.2.5.62 Between DOCTYPE public and system identifiers state, https://html.spec.whatwg.org/multipage/parsing.html#between-doctype-public-and-system-identifiers-state
LibWeb: Flesh out the remaining DOCTYPE related tokenizer states We can now parse public and system identifiers! Not super useful, but at least we can do it :^)
2020-05-25 19:50:44 +02:00
BEGIN_STATE(BetweenDOCTYPEPublicAndSystemIdentifiers)
{
ON_WHITESPACE
{
continue;
}
ON('>')
{
SWITCH_TO_AND_EMIT_CURRENT_TOKEN(Data);
}
ON('"')
{
LibWeb: Rename HTMLToken::doctype_data() => ensure_doctype_data() This renames the accessor to better reflect what it does, as this will allocate a DoctypeData struct if there is none.
2021-07-15 18:58:11 +02:00
m_current_token.ensure_doctype_data().missing_system_identifier = false;
LibWeb: Flesh out the remaining DOCTYPE related tokenizer states We can now parse public and system identifiers! Not super useful, but at least we can do it :^)
2020-05-25 19:50:44 +02:00
SWITCH_TO(DOCTYPESystemIdentifierDoubleQuoted);
}
ON('\'')
{
LibWeb: Rename HTMLToken::doctype_data() => ensure_doctype_data() This renames the accessor to better reflect what it does, as this will allocate a DoctypeData struct if there is none.
2021-07-15 18:58:11 +02:00
m_current_token.ensure_doctype_data().missing_system_identifier = false;
LibWeb: Flesh out the remaining DOCTYPE related tokenizer states We can now parse public and system identifiers! Not super useful, but at least we can do it :^)
2020-05-25 19:50:44 +02:00
SWITCH_TO(DOCTYPESystemIdentifierSingleQuoted);
}
ON_EOF
{
LibWeb: Utilize SourceLocation for HTMLTokenizer logging
2021-04-24 20:22:30 -07:00
log_parse_error();
LibWeb: Rename HTMLToken::doctype_data() => ensure_doctype_data() This renames the accessor to better reflect what it does, as this will allocate a DoctypeData struct if there is none.
2021-07-15 18:58:11 +02:00
m_current_token.ensure_doctype_data().force_quirks = true;
LibWeb: Use move() when enqueuing tokens in HTMLTokenizer We're not using the current token anymore once it's enqueued so let's use move() when enqueuing the tokens.
2021-05-23 12:39:00 +02:00
m_queued_tokens.enqueue(move(m_current_token));
LibWeb: Fully implement all DOCTYPE tokenizer states Also fixes TagOpen having a seperate emit and reconsume in ANYTHING_ELSE.
2020-06-11 05:00:45 +01:00
EMIT_EOF;
LibWeb: Flesh out the remaining DOCTYPE related tokenizer states We can now parse public and system identifiers! Not super useful, but at least we can do it :^)
2020-05-25 19:50:44 +02:00
}
ANYTHING_ELSE
{
LibWeb: Utilize SourceLocation for HTMLTokenizer logging
2021-04-24 20:22:30 -07:00
log_parse_error();
LibWeb: Rename HTMLToken::doctype_data() => ensure_doctype_data() This renames the accessor to better reflect what it does, as this will allocate a DoctypeData struct if there is none.
2021-07-15 18:58:11 +02:00
m_current_token.ensure_doctype_data().force_quirks = true;
LibWeb: Fully implement all DOCTYPE tokenizer states Also fixes TagOpen having a seperate emit and reconsume in ANYTHING_ELSE.
2020-06-11 05:00:45 +01:00
RECONSUME_IN(BogusDOCTYPE);
LibWeb: Flesh out the remaining DOCTYPE related tokenizer states We can now parse public and system identifiers! Not super useful, but at least we can do it :^)
2020-05-25 19:50:44 +02:00
}
}
END_STATE
LibWeb: Add spec links to each HTML tokenizer state section I didn't add full spec comments this time, but this is better than nothing :^)
2022-02-15 18:12:15 +00:00
// 13.2.5.67 After DOCTYPE system identifier state, https://html.spec.whatwg.org/multipage/parsing.html#after-doctype-system-identifier-state
LibWeb: Flesh out the remaining DOCTYPE related tokenizer states We can now parse public and system identifiers! Not super useful, but at least we can do it :^)
2020-05-25 19:50:44 +02:00
BEGIN_STATE(AfterDOCTYPESystemIdentifier)
LibWeb: Implement a bit more of DOCTYPE tokenization
2020-05-23 19:56:07 +02:00
{
ON_WHITESPACE
{
continue;
}
ON('>')
{
LibWeb: Start building the tree building part of the new HTML parser This patch adds a new HTMLDocumentParser class. It keeps a tokenizer object internally and feeds itself with one token at a time from it. The names and idioms in this class are expressed as closely to the actual HTML parsing spec as possible, to make development as easy and bug free as possible. :^) This is going to become pretty large, but it's pretty cool!
2020-05-24 00:14:23 +02:00
SWITCH_TO_AND_EMIT_CURRENT_TOKEN(Data);
LibWeb: Implement a bit more of DOCTYPE tokenization
2020-05-23 19:56:07 +02:00
}
ON_EOF
{
LibWeb: Utilize SourceLocation for HTMLTokenizer logging
2021-04-24 20:22:30 -07:00
log_parse_error();
LibWeb: Rename HTMLToken::doctype_data() => ensure_doctype_data() This renames the accessor to better reflect what it does, as this will allocate a DoctypeData struct if there is none.
2021-07-15 18:58:11 +02:00
m_current_token.ensure_doctype_data().force_quirks = true;
LibWeb: Use move() when enqueuing tokens in HTMLTokenizer We're not using the current token anymore once it's enqueued so let's use move() when enqueuing the tokens.
2021-05-23 12:39:00 +02:00
m_queued_tokens.enqueue(move(m_current_token));
LibWeb: Fully implement all DOCTYPE tokenizer states Also fixes TagOpen having a seperate emit and reconsume in ANYTHING_ELSE.
2020-06-11 05:00:45 +01:00
EMIT_EOF;
LibWeb: Implement a bit more of DOCTYPE tokenization
2020-05-23 19:56:07 +02:00
}
ANYTHING_ELSE
{
LibWeb: Utilize SourceLocation for HTMLTokenizer logging
2021-04-24 20:22:30 -07:00
log_parse_error();
LibWeb: Fully implement all DOCTYPE tokenizer states Also fixes TagOpen having a seperate emit and reconsume in ANYTHING_ELSE.
2020-06-11 05:00:45 +01:00
RECONSUME_IN(BogusDOCTYPE);
}
}
END_STATE
LibWeb: Add spec links to each HTML tokenizer state section I didn't add full spec comments this time, but this is better than nothing :^)
2022-02-15 18:12:15 +00:00
// 13.2.5.68 Bogus DOCTYPE state, https://html.spec.whatwg.org/multipage/parsing.html#bogus-doctype-state
LibWeb: Fully implement all DOCTYPE tokenizer states Also fixes TagOpen having a seperate emit and reconsume in ANYTHING_ELSE.
2020-06-11 05:00:45 +01:00
BEGIN_STATE(BogusDOCTYPE)
{
ON('>')
{
SWITCH_TO_AND_EMIT_CURRENT_TOKEN(Data);
}
ON(0)
{
LibWeb: Utilize SourceLocation for HTMLTokenizer logging
2021-04-24 20:22:30 -07:00
log_parse_error();
LibWeb: Fully implement all DOCTYPE tokenizer states Also fixes TagOpen having a seperate emit and reconsume in ANYTHING_ELSE.
2020-06-11 05:00:45 +01:00
continue;
}
ON_EOF
{
LibWeb: Use move() when enqueuing tokens in HTMLTokenizer We're not using the current token anymore once it's enqueued so let's use move() when enqueuing the tokens.
2021-05-23 12:39:00 +02:00
m_queued_tokens.enqueue(move(m_current_token));
LibWeb: Fully implement all DOCTYPE tokenizer states Also fixes TagOpen having a seperate emit and reconsume in ANYTHING_ELSE.
2020-06-11 05:00:45 +01:00
EMIT_EOF;
}
ANYTHING_ELSE
{
continue;
LibWeb: Implement a bit more of DOCTYPE tokenization
2020-05-23 19:56:07 +02:00
}
LibWeb: Begin work on a spec-compliant HTML parser In order to actually view the web as it is, we're gonna need a proper HTML parser. So let's build one! This patch introduces the Web::HTMLTokenizer class, which currently operates on a StringView input stream where it fetches (ASCII only atm) codepoints and tokenizes acccording to the HTML spec tokenization algo. The tokenizer state machine looks a bit weird but is written in a way that tries to mimic the spec as closely as possible, in order to make development easier and bugs less likely. This initial version is far from finished, but it can parse a trivial document with a DOCTYPE and open/close tags. :^)
2020-05-22 21:46:13 +02:00
}
END_STATE
LibWeb: Add spec links to each HTML tokenizer state section I didn't add full spec comments this time, but this is better than nothing :^)
2022-02-15 18:12:15 +00:00
// 13.2.5.32 Before attribute name state, https://html.spec.whatwg.org/multipage/parsing.html#before-attribute-name-state
LibWeb: Teach HTMLTokenizer how to tokenize attributes Properly tokenize single-quoted, double-quoted and unquoted attributes!
2020-05-23 01:19:42 +02:00
BEGIN_STATE(BeforeAttributeName)
{
ON_WHITESPACE
{
continue;
}
ON('/')
{
RECONSUME_IN(AfterAttributeName);
}
ON('>')
{
RECONSUME_IN(AfterAttributeName);
}
ON_EOF
{
RECONSUME_IN(AfterAttributeName);
}
ON('=')
{
LibWeb: Utilize SourceLocation for HTMLTokenizer logging
2021-04-24 20:22:30 -07:00
log_parse_error();
LibWeb: Rename HTMLToken::AttributeBuilder struct to Attribute This does not contain StringBuilders anymore, so it can do with a simpler name: Attribute.
2021-07-14 23:17:35 +02:00
HTMLToken::Attribute new_attribute;
LibWeb: Add position tracking information to HTML tokens
2021-05-20 23:11:41 +04:30
new_attribute.name_start_position = nth_last_position(1);
LibWeb: Remove StringBuilders from HTMLToken::AttributeBuilder
2021-05-23 08:50:48 +02:00
m_current_builder.append_code_point(current_input_character.value());
LibWeb: Hide implementation details of HTMLToken attribute list Previously, HTMLToken would expose the Vector<Attribute> directly to its users. In preparation for a future change, all users now use implementation-agnostic APIs which do not expose the Vector directly.
2021-07-14 23:53:11 +02:00
m_current_token.add_attribute(move(new_attribute));
LibWeb: Remove StringBuilders from HTMLToken::AttributeBuilder
2021-05-23 08:50:48 +02:00
SWITCH_TO_WITH_UNCLEAN_BUILDER(AttributeName);
LibWeb: Teach HTMLTokenizer how to tokenize attributes Properly tokenize single-quoted, double-quoted and unquoted attributes!
2020-05-23 01:19:42 +02:00
}
ANYTHING_ELSE
{
LibWeb: Rename HTMLToken::AttributeBuilder struct to Attribute This does not contain StringBuilders anymore, so it can do with a simpler name: Attribute.
2021-07-14 23:17:35 +02:00
HTMLToken::Attribute new_attribute;
LibWeb: Add position tracking information to HTML tokens
2021-05-20 23:11:41 +04:30
new_attribute.name_start_position = nth_last_position(1);
LibWeb: Hide implementation details of HTMLToken attribute list Previously, HTMLToken would expose the Vector<Attribute> directly to its users. In preparation for a future change, all users now use implementation-agnostic APIs which do not expose the Vector directly.
2021-07-14 23:53:11 +02:00
m_current_token.add_attribute(move(new_attribute));
LibWeb: Teach HTMLTokenizer how to tokenize attributes Properly tokenize single-quoted, double-quoted and unquoted attributes!
2020-05-23 01:19:42 +02:00
RECONSUME_IN(AttributeName);
}
}
END_STATE
LibWeb: Add spec links to each HTML tokenizer state section I didn't add full spec comments this time, but this is better than nothing :^)
2022-02-15 18:12:15 +00:00
// 13.2.5.40 Self-closing start tag state, https://html.spec.whatwg.org/multipage/parsing.html#self-closing-start-tag-state
LibWeb: Teach HTMLTokenizer how to tokenize attributes Properly tokenize single-quoted, double-quoted and unquoted attributes!
2020-05-23 01:19:42 +02:00
BEGIN_STATE(SelfClosingStartTag)
{
LibWeb: Implement the "self closing start tag" tokenizer state
2020-05-27 18:07:51 +02:00
ON('>')
{
LibWeb: User getter+setter for HTMLToken tag name and self-closing flag
2021-07-14 23:37:48 +02:00
m_current_token.set_self_closing(true);
LibWeb: Fix accidental swallow of self-closing tag tokens Instead of dropping self-closing tags on the floor, we now emit them into the token stream. :^)
2020-05-30 11:30:45 +02:00
SWITCH_TO_AND_EMIT_CURRENT_TOKEN(Data);
LibWeb: Implement the "self closing start tag" tokenizer state
2020-05-27 18:07:51 +02:00
}
ON_EOF
{
LibWeb: Utilize SourceLocation for HTMLTokenizer logging
2021-04-24 20:22:30 -07:00
log_parse_error();
LibWeb: Handle more cases in the SelfClosingStartTag tokenizer state
2020-06-01 12:37:28 +02:00
EMIT_EOF;
LibWeb: Implement the "self closing start tag" tokenizer state
2020-05-27 18:07:51 +02:00
}
ANYTHING_ELSE
{
LibWeb: Utilize SourceLocation for HTMLTokenizer logging
2021-04-24 20:22:30 -07:00
log_parse_error();
LibWeb: Handle more cases in the SelfClosingStartTag tokenizer state
2020-06-01 12:37:28 +02:00
RECONSUME_IN(BeforeAttributeName);
LibWeb: Implement the "self closing start tag" tokenizer state
2020-05-27 18:07:51 +02:00
}
LibWeb: Teach HTMLTokenizer how to tokenize attributes Properly tokenize single-quoted, double-quoted and unquoted attributes!
2020-05-23 01:19:42 +02:00
}
END_STATE
LibWeb: Add spec links to each HTML tokenizer state section I didn't add full spec comments this time, but this is better than nothing :^)
2022-02-15 18:12:15 +00:00
// 13.2.5.33 Attribute name state, https://html.spec.whatwg.org/multipage/parsing.html#attribute-name-state
LibWeb: Teach HTMLTokenizer how to tokenize attributes Properly tokenize single-quoted, double-quoted and unquoted attributes!
2020-05-23 01:19:42 +02:00
BEGIN_STATE(AttributeName)
{
ON_WHITESPACE
{
LibWeb: Set the correct end position of HTML attribute names We were previously setting the end position of attribute names in self- closing HTML tags to the end of the attribute value. To illustrate the previous behavior, consider this tag and its attribute's start and end positions (shown inclusively below): <meta charset="UTF-8" /> ^ name start ^ value start ^ value end ^ name end Rather than setting the end position of the attribute name when we parse the closing slash, ensure the end position is already set while we are in the AttributeName state. We now have: <meta charset="UTF-8" /> ^ name start ^ name end ^ value start ^ value end The tokenizer unit test has been extended to test these positions.
2023-08-24 17:43:05 -04:00
m_current_token.last_attribute().name_end_position = nth_last_position(1);
LibWeb: Switch HTMLToken and HTMLTokenizer to String & FlyString
2023-11-04 10:08:07 +01:00
m_current_token.last_attribute().local_name = consume_current_builder();
LibWeb: Teach HTMLTokenizer how to tokenize attributes Properly tokenize single-quoted, double-quoted and unquoted attributes!
2020-05-23 01:19:42 +02:00
RECONSUME_IN(AfterAttributeName);
}
ON('/')
{
LibWeb: Set the correct end position of HTML attribute names We were previously setting the end position of attribute names in self- closing HTML tags to the end of the attribute value. To illustrate the previous behavior, consider this tag and its attribute's start and end positions (shown inclusively below): <meta charset="UTF-8" /> ^ name start ^ value start ^ value end ^ name end Rather than setting the end position of the attribute name when we parse the closing slash, ensure the end position is already set while we are in the AttributeName state. We now have: <meta charset="UTF-8" /> ^ name start ^ name end ^ value start ^ value end The tokenizer unit test has been extended to test these positions.
2023-08-24 17:43:05 -04:00
m_current_token.last_attribute().name_end_position = nth_last_position(1);
LibWeb: Switch HTMLToken and HTMLTokenizer to String & FlyString
2023-11-04 10:08:07 +01:00
m_current_token.last_attribute().local_name = consume_current_builder();
LibWeb: Teach HTMLTokenizer how to tokenize attributes Properly tokenize single-quoted, double-quoted and unquoted attributes!
2020-05-23 01:19:42 +02:00
RECONSUME_IN(AfterAttributeName);
}
ON('>')
{
LibWeb: Set the correct end position of HTML attribute names We were previously setting the end position of attribute names in self- closing HTML tags to the end of the attribute value. To illustrate the previous behavior, consider this tag and its attribute's start and end positions (shown inclusively below): <meta charset="UTF-8" /> ^ name start ^ value start ^ value end ^ name end Rather than setting the end position of the attribute name when we parse the closing slash, ensure the end position is already set while we are in the AttributeName state. We now have: <meta charset="UTF-8" /> ^ name start ^ name end ^ value start ^ value end The tokenizer unit test has been extended to test these positions.
2023-08-24 17:43:05 -04:00
m_current_token.last_attribute().name_end_position = nth_last_position(1);
LibWeb: Switch HTMLToken and HTMLTokenizer to String & FlyString
2023-11-04 10:08:07 +01:00
m_current_token.last_attribute().local_name = consume_current_builder();
LibWeb: Teach HTMLTokenizer how to tokenize attributes Properly tokenize single-quoted, double-quoted and unquoted attributes!
2020-05-23 01:19:42 +02:00
RECONSUME_IN(AfterAttributeName);
}
ON_EOF
{
LibWeb: Set the correct end position of HTML attribute names We were previously setting the end position of attribute names in self- closing HTML tags to the end of the attribute value. To illustrate the previous behavior, consider this tag and its attribute's start and end positions (shown inclusively below): <meta charset="UTF-8" /> ^ name start ^ value start ^ value end ^ name end Rather than setting the end position of the attribute name when we parse the closing slash, ensure the end position is already set while we are in the AttributeName state. We now have: <meta charset="UTF-8" /> ^ name start ^ name end ^ value start ^ value end The tokenizer unit test has been extended to test these positions.
2023-08-24 17:43:05 -04:00
m_current_token.last_attribute().name_end_position = nth_last_position(1);
LibWeb: Switch HTMLToken and HTMLTokenizer to String & FlyString
2023-11-04 10:08:07 +01:00
m_current_token.last_attribute().local_name = consume_current_builder();
LibWeb: Teach HTMLTokenizer how to tokenize attributes Properly tokenize single-quoted, double-quoted and unquoted attributes!
2020-05-23 01:19:42 +02:00
RECONSUME_IN(AfterAttributeName);
}
ON('=')
{
LibWeb: Hide implementation details of HTMLToken attribute list Previously, HTMLToken would expose the Vector<Attribute> directly to its users. In preparation for a future change, all users now use implementation-agnostic APIs which do not expose the Vector directly.
2021-07-14 23:53:11 +02:00
m_current_token.last_attribute().name_end_position = nth_last_position(1);
LibWeb: Switch HTMLToken and HTMLTokenizer to String & FlyString
2023-11-04 10:08:07 +01:00
m_current_token.last_attribute().local_name = consume_current_builder();
LibWeb: Teach HTMLTokenizer how to tokenize attributes Properly tokenize single-quoted, double-quoted and unquoted attributes!
2020-05-23 01:19:42 +02:00
SWITCH_TO(BeforeAttributeValue);
}
LibWeb: Implement almost all missing tokenizer cases
2020-06-26 22:41:35 +01:00
ON_ASCII_UPPER_ALPHA
{
LibWeb: Remove StringBuilders from HTMLToken::AttributeBuilder
2021-05-23 08:50:48 +02:00
m_current_builder.append_code_point(to_ascii_lowercase(current_input_character.value()));
LibWeb: Implement almost all missing tokenizer cases
2020-06-26 22:41:35 +01:00
continue;
}
ON(0)
{
LibWeb: Utilize SourceLocation for HTMLTokenizer logging
2021-04-24 20:22:30 -07:00
log_parse_error();
LibWeb: Remove StringBuilders from HTMLToken::AttributeBuilder
2021-05-23 08:50:48 +02:00
m_current_builder.append_code_point(0xFFFD);
LibWeb: Implement almost all missing tokenizer cases
2020-06-26 22:41:35 +01:00
continue;
}
ON('"')
{
LibWeb: Utilize SourceLocation for HTMLTokenizer logging
2021-04-24 20:22:30 -07:00
log_parse_error();
LibWeb: Implement almost all missing tokenizer cases
2020-06-26 22:41:35 +01:00
goto AnythingElseAttributeName;
}
ON('\'')
{
LibWeb: Utilize SourceLocation for HTMLTokenizer logging
2021-04-24 20:22:30 -07:00
log_parse_error();
LibWeb: Implement almost all missing tokenizer cases
2020-06-26 22:41:35 +01:00
goto AnythingElseAttributeName;
}
ON('<')
{
LibWeb: Utilize SourceLocation for HTMLTokenizer logging
2021-04-24 20:22:30 -07:00
log_parse_error();
LibWeb: Implement almost all missing tokenizer cases
2020-06-26 22:41:35 +01:00
goto AnythingElseAttributeName;
}
LibWeb: Teach HTMLTokenizer how to tokenize attributes Properly tokenize single-quoted, double-quoted and unquoted attributes!
2020-05-23 01:19:42 +02:00
ANYTHING_ELSE
{
LibWeb: Implement almost all missing tokenizer cases
2020-06-26 22:41:35 +01:00
AnythingElseAttributeName:
LibWeb: Remove StringBuilders from HTMLToken::AttributeBuilder
2021-05-23 08:50:48 +02:00
m_current_builder.append_code_point(current_input_character.value());
LibWeb: Teach HTMLTokenizer how to tokenize attributes Properly tokenize single-quoted, double-quoted and unquoted attributes!
2020-05-23 01:19:42 +02:00
continue;
}
}
END_STATE
LibWeb: Add spec links to each HTML tokenizer state section I didn't add full spec comments this time, but this is better than nothing :^)
2022-02-15 18:12:15 +00:00
// 13.2.5.34 After attribute name state, https://html.spec.whatwg.org/multipage/parsing.html#after-attribute-name-state
LibWeb: Teach HTMLTokenizer how to tokenize attributes Properly tokenize single-quoted, double-quoted and unquoted attributes!
2020-05-23 01:19:42 +02:00
BEGIN_STATE(AfterAttributeName)
{
LibWeb: Implement the "after attribute name" tokenizer state One little step at a time towards parsing the monster blob of HTML we get from twitter.com :^)
2020-05-27 18:27:32 +02:00
ON_WHITESPACE
{
continue;
}
ON('/')
{
SWITCH_TO(SelfClosingStartTag);
}
ON('=')
{
LibWeb: Hide implementation details of HTMLToken attribute list Previously, HTMLToken would expose the Vector<Attribute> directly to its users. In preparation for a future change, all users now use implementation-agnostic APIs which do not expose the Vector directly.
2021-07-14 23:53:11 +02:00
m_current_token.last_attribute().name_end_position = nth_last_position(1);
LibWeb: Implement the "after attribute name" tokenizer state One little step at a time towards parsing the monster blob of HTML we get from twitter.com :^)
2020-05-27 18:27:32 +02:00
SWITCH_TO(BeforeAttributeValue);
}
ON('>')
{
LibWeb: Fix tokenization of attributes with empty attributes We were neglecting to emit start tags for tags where the last attribute had no value. Also fix a parse error TODO that I hit while looking at this.
2020-06-04 11:59:14 +02:00
SWITCH_TO_AND_EMIT_CURRENT_TOKEN(Data);
LibWeb: Implement the "after attribute name" tokenizer state One little step at a time towards parsing the monster blob of HTML we get from twitter.com :^)
2020-05-27 18:27:32 +02:00
}
ON_EOF
{
LibWeb: Utilize SourceLocation for HTMLTokenizer logging
2021-04-24 20:22:30 -07:00
log_parse_error();
LibWeb: Implement some missing tokenizer cases for EOF handling
2020-06-26 22:47:07 +02:00
EMIT_EOF;
LibWeb: Implement the "after attribute name" tokenizer state One little step at a time towards parsing the monster blob of HTML we get from twitter.com :^)
2020-05-27 18:27:32 +02:00
}
ANYTHING_ELSE
{
LibWeb: Hide implementation details of HTMLToken attribute list Previously, HTMLToken would expose the Vector<Attribute> directly to its users. In preparation for a future change, all users now use implementation-agnostic APIs which do not expose the Vector directly.
2021-07-14 23:53:11 +02:00
m_current_token.add_attribute({});
LibWeb: Fix Array OOBs in the HTMLTokenizer Accessing last() if there are no elements makes WebContent crash :^)
2022-06-02 01:03:44 +02:00
if (!m_source_positions.is_empty())
LibWeb: Set the correct end position of HTML attribute names We were previously setting the end position of attribute names in self- closing HTML tags to the end of the attribute value. To illustrate the previous behavior, consider this tag and its attribute's start and end positions (shown inclusively below): <meta charset="UTF-8" /> ^ name start ^ value start ^ value end ^ name end Rather than setting the end position of the attribute name when we parse the closing slash, ensure the end position is already set while we are in the AttributeName state. We now have: <meta charset="UTF-8" /> ^ name start ^ name end ^ value start ^ value end The tokenizer unit test has been extended to test these positions.
2023-08-24 17:43:05 -04:00
m_current_token.last_attribute().name_start_position = nth_last_position(1);
LibWeb: Implement the "after attribute name" tokenizer state One little step at a time towards parsing the monster blob of HTML we get from twitter.com :^)
2020-05-27 18:27:32 +02:00
RECONSUME_IN(AttributeName);
}
LibWeb: Teach HTMLTokenizer how to tokenize attributes Properly tokenize single-quoted, double-quoted and unquoted attributes!
2020-05-23 01:19:42 +02:00
}
END_STATE
LibWeb: Add spec links to each HTML tokenizer state section I didn't add full spec comments this time, but this is better than nothing :^)
2022-02-15 18:12:15 +00:00
// 13.2.5.35 Before attribute value state, https://html.spec.whatwg.org/multipage/parsing.html#before-attribute-value-state
LibWeb: Teach HTMLTokenizer how to tokenize attributes Properly tokenize single-quoted, double-quoted and unquoted attributes!
2020-05-23 01:19:42 +02:00
BEGIN_STATE(BeforeAttributeValue)
{
LibWeb: Hide implementation details of HTMLToken attribute list Previously, HTMLToken would expose the Vector<Attribute> directly to its users. In preparation for a future change, all users now use implementation-agnostic APIs which do not expose the Vector directly.
2021-07-14 23:53:11 +02:00
m_current_token.last_attribute().value_start_position = nth_last_position(1);
LibWeb: Teach HTMLTokenizer how to tokenize attributes Properly tokenize single-quoted, double-quoted and unquoted attributes!
2020-05-23 01:19:42 +02:00
ON_WHITESPACE
{
continue;
}
ON('"')
{
SWITCH_TO(AttributeValueDoubleQuoted);
}
ON('\'')
{
SWITCH_TO(AttributeValueSingleQuoted);
}
ON('>')
{
LibWeb: Utilize SourceLocation for HTMLTokenizer logging
2021-04-24 20:22:30 -07:00
log_parse_error();
LibWeb: Fix tokenization of attributes with empty attributes We were neglecting to emit start tags for tags where the last attribute had no value. Also fix a parse error TODO that I hit while looking at this.
2020-06-04 11:59:14 +02:00
SWITCH_TO_AND_EMIT_CURRENT_TOKEN(Data);
LibWeb: Teach HTMLTokenizer how to tokenize attributes Properly tokenize single-quoted, double-quoted and unquoted attributes!
2020-05-23 01:19:42 +02:00
}
ANYTHING_ELSE
{
RECONSUME_IN(AttributeValueUnquoted);
}
}
END_STATE
LibWeb: Add spec links to each HTML tokenizer state section I didn't add full spec comments this time, but this is better than nothing :^)
2022-02-15 18:12:15 +00:00
// 13.2.5.36 Attribute value (double-quoted) state, https://html.spec.whatwg.org/multipage/parsing.html#attribute-value-(double-quoted)-state
LibWeb: Teach HTMLTokenizer how to tokenize attributes Properly tokenize single-quoted, double-quoted and unquoted attributes!
2020-05-23 01:19:42 +02:00
BEGIN_STATE(AttributeValueDoubleQuoted)
{
ON('"')
{
LibWeb: Switch HTMLToken and HTMLTokenizer to String & FlyString
2023-11-04 10:08:07 +01:00
m_current_token.last_attribute().value = consume_current_builder();
LibWeb: Teach HTMLTokenizer how to tokenize attributes Properly tokenize single-quoted, double-quoted and unquoted attributes!
2020-05-23 01:19:42 +02:00
SWITCH_TO(AfterAttributeValueQuoted);
}
ON('&')
{
m_return_state = State::AttributeValueDoubleQuoted;
LibWeb: Fix parsing of character references in attribute values
2021-07-24 23:15:47 +01:00
SWITCH_TO_WITH_UNCLEAN_BUILDER(CharacterReference);
LibWeb: Teach HTMLTokenizer how to tokenize attributes Properly tokenize single-quoted, double-quoted and unquoted attributes!
2020-05-23 01:19:42 +02:00
}
ON(0)
{
LibWeb: Utilize SourceLocation for HTMLTokenizer logging
2021-04-24 20:22:30 -07:00
log_parse_error();
LibWeb: Remove StringBuilders from HTMLToken::AttributeBuilder
2021-05-23 08:50:48 +02:00
m_current_builder.append_code_point(0xFFFD);
LibWeb: Implement almost all missing tokenizer cases
2020-06-26 22:41:35 +01:00
continue;
LibWeb: Teach HTMLTokenizer how to tokenize attributes Properly tokenize single-quoted, double-quoted and unquoted attributes!
2020-05-23 01:19:42 +02:00
}
ON_EOF
{
LibWeb: Utilize SourceLocation for HTMLTokenizer logging
2021-04-24 20:22:30 -07:00
log_parse_error();
LibWeb: Implement remaining missing tokenizer EOF cases
2020-06-27 12:05:21 +02:00
EMIT_EOF;
LibWeb: Teach HTMLTokenizer how to tokenize attributes Properly tokenize single-quoted, double-quoted and unquoted attributes!
2020-05-23 01:19:42 +02:00
}
ANYTHING_ELSE
{
LibWeb: Remove StringBuilders from HTMLToken::AttributeBuilder
2021-05-23 08:50:48 +02:00
m_current_builder.append_code_point(current_input_character.value());
LibWeb: Teach HTMLTokenizer how to tokenize attributes Properly tokenize single-quoted, double-quoted and unquoted attributes!
2020-05-23 01:19:42 +02:00
continue;
}
}
END_STATE
LibWeb: Add spec links to each HTML tokenizer state section I didn't add full spec comments this time, but this is better than nothing :^)
2022-02-15 18:12:15 +00:00
// 13.2.5.37 Attribute value (single-quoted) state, https://html.spec.whatwg.org/multipage/parsing.html#attribute-value-(single-quoted)-state
LibWeb: Teach HTMLTokenizer how to tokenize attributes Properly tokenize single-quoted, double-quoted and unquoted attributes!
2020-05-23 01:19:42 +02:00
BEGIN_STATE(AttributeValueSingleQuoted)
{
ON('\'')
{
LibWeb: Switch HTMLToken and HTMLTokenizer to String & FlyString
2023-11-04 10:08:07 +01:00
m_current_token.last_attribute().value = consume_current_builder();
LibWeb: Teach HTMLTokenizer how to tokenize attributes Properly tokenize single-quoted, double-quoted and unquoted attributes!
2020-05-23 01:19:42 +02:00
SWITCH_TO(AfterAttributeValueQuoted);
}
ON('&')
{
m_return_state = State::AttributeValueSingleQuoted;
LibWeb: Fix parsing of character references in attribute values
2021-07-24 23:15:47 +01:00
SWITCH_TO_WITH_UNCLEAN_BUILDER(CharacterReference);
LibWeb: Teach HTMLTokenizer how to tokenize attributes Properly tokenize single-quoted, double-quoted and unquoted attributes!
2020-05-23 01:19:42 +02:00
}
ON(0)
{
LibWeb: Utilize SourceLocation for HTMLTokenizer logging
2021-04-24 20:22:30 -07:00
log_parse_error();
LibWeb: Remove StringBuilders from HTMLToken::AttributeBuilder
2021-05-23 08:50:48 +02:00
m_current_builder.append_code_point(0xFFFD);
LibWeb: Implement almost all missing tokenizer cases
2020-06-26 22:41:35 +01:00
continue;
LibWeb: Teach HTMLTokenizer how to tokenize attributes Properly tokenize single-quoted, double-quoted and unquoted attributes!
2020-05-23 01:19:42 +02:00
}
ON_EOF
{
LibWeb: Utilize SourceLocation for HTMLTokenizer logging
2021-04-24 20:22:30 -07:00
log_parse_error();
LibWeb: Implement remaining missing tokenizer EOF cases
2020-06-27 12:05:21 +02:00
EMIT_EOF;
LibWeb: Teach HTMLTokenizer how to tokenize attributes Properly tokenize single-quoted, double-quoted and unquoted attributes!
2020-05-23 01:19:42 +02:00
}
ANYTHING_ELSE
{
LibWeb: Remove StringBuilders from HTMLToken::AttributeBuilder
2021-05-23 08:50:48 +02:00
m_current_builder.append_code_point(current_input_character.value());
LibWeb: Teach HTMLTokenizer how to tokenize attributes Properly tokenize single-quoted, double-quoted and unquoted attributes!
2020-05-23 01:19:42 +02:00
continue;
}
}
END_STATE
LibWeb: Add spec links to each HTML tokenizer state section I didn't add full spec comments this time, but this is better than nothing :^)
2022-02-15 18:12:15 +00:00
// 13.2.5.38 Attribute value (unquoted) state, https://html.spec.whatwg.org/multipage/parsing.html#attribute-value-(single-quoted)-state
LibWeb: Teach HTMLTokenizer how to tokenize attributes Properly tokenize single-quoted, double-quoted and unquoted attributes!
2020-05-23 01:19:42 +02:00
BEGIN_STATE(AttributeValueUnquoted)
{
ON_WHITESPACE
{
LibWeb: Switch HTMLToken and HTMLTokenizer to String & FlyString
2023-11-04 10:08:07 +01:00
m_current_token.last_attribute().value = consume_current_builder();
LibWeb: Fix off-by-one error when highlighting unquoted HTML attributes This fixes #11166
2021-12-10 17:17:12 +00:00
m_current_token.last_attribute().value_end_position = nth_last_position(1);
LibWeb: Teach HTMLTokenizer how to tokenize attributes Properly tokenize single-quoted, double-quoted and unquoted attributes!
2020-05-23 01:19:42 +02:00
SWITCH_TO(BeforeAttributeName);
}
ON('&')
{
m_return_state = State::AttributeValueUnquoted;
LibWeb: Fix parsing of character references in attribute values
2021-07-24 23:15:47 +01:00
SWITCH_TO_WITH_UNCLEAN_BUILDER(CharacterReference);
LibWeb: Teach HTMLTokenizer how to tokenize attributes Properly tokenize single-quoted, double-quoted and unquoted attributes!
2020-05-23 01:19:42 +02:00
}
ON('>')
{
LibWeb: Switch HTMLToken and HTMLTokenizer to String & FlyString
2023-11-04 10:08:07 +01:00
m_current_token.last_attribute().value = consume_current_builder();
LibWeb: Hide implementation details of HTMLToken attribute list Previously, HTMLToken would expose the Vector<Attribute> directly to its users. In preparation for a future change, all users now use implementation-agnostic APIs which do not expose the Vector directly.
2021-07-14 23:53:11 +02:00
m_current_token.last_attribute().value_end_position = nth_last_position(1);
LibWeb: Start building the tree building part of the new HTML parser This patch adds a new HTMLDocumentParser class. It keeps a tokenizer object internally and feeds itself with one token at a time from it. The names and idioms in this class are expressed as closely to the actual HTML parsing spec as possible, to make development as easy and bug free as possible. :^) This is going to become pretty large, but it's pretty cool!
2020-05-24 00:14:23 +02:00
SWITCH_TO_AND_EMIT_CURRENT_TOKEN(Data);
LibWeb: Teach HTMLTokenizer how to tokenize attributes Properly tokenize single-quoted, double-quoted and unquoted attributes!
2020-05-23 01:19:42 +02:00
}
ON(0)
{
LibWeb: Utilize SourceLocation for HTMLTokenizer logging
2021-04-24 20:22:30 -07:00
log_parse_error();
LibWeb: Remove StringBuilders from HTMLToken::AttributeBuilder
2021-05-23 08:50:48 +02:00
m_current_builder.append_code_point(0xFFFD);
LibWeb: Implement almost all missing tokenizer cases
2020-06-26 22:41:35 +01:00
continue;
}
ON('"')
{
LibWeb: Utilize SourceLocation for HTMLTokenizer logging
2021-04-24 20:22:30 -07:00
log_parse_error();
LibWeb: Implement almost all missing tokenizer cases
2020-06-26 22:41:35 +01:00
goto AnythingElseAttributeValueUnquoted;
}
ON('\'')
{
LibWeb: Utilize SourceLocation for HTMLTokenizer logging
2021-04-24 20:22:30 -07:00
log_parse_error();
LibWeb: Implement almost all missing tokenizer cases
2020-06-26 22:41:35 +01:00
goto AnythingElseAttributeValueUnquoted;
}
ON('<')
{
LibWeb: Utilize SourceLocation for HTMLTokenizer logging
2021-04-24 20:22:30 -07:00
log_parse_error();
LibWeb: Implement almost all missing tokenizer cases
2020-06-26 22:41:35 +01:00
goto AnythingElseAttributeValueUnquoted;
}
ON('=')
{
LibWeb: Utilize SourceLocation for HTMLTokenizer logging
2021-04-24 20:22:30 -07:00
log_parse_error();
LibWeb: Implement almost all missing tokenizer cases
2020-06-26 22:41:35 +01:00
goto AnythingElseAttributeValueUnquoted;
}
ON('`')
{
LibWeb: Utilize SourceLocation for HTMLTokenizer logging
2021-04-24 20:22:30 -07:00
log_parse_error();
LibWeb: Implement almost all missing tokenizer cases
2020-06-26 22:41:35 +01:00
goto AnythingElseAttributeValueUnquoted;
LibWeb: Teach HTMLTokenizer how to tokenize attributes Properly tokenize single-quoted, double-quoted and unquoted attributes!
2020-05-23 01:19:42 +02:00
}
ON_EOF
{
LibWeb: Utilize SourceLocation for HTMLTokenizer logging
2021-04-24 20:22:30 -07:00
log_parse_error();
LibWeb: Implement remaining missing tokenizer EOF cases
2020-06-27 12:05:21 +02:00
EMIT_EOF;
LibWeb: Teach HTMLTokenizer how to tokenize attributes Properly tokenize single-quoted, double-quoted and unquoted attributes!
2020-05-23 01:19:42 +02:00
}
ANYTHING_ELSE
{
LibWeb: Implement almost all missing tokenizer cases
2020-06-26 22:41:35 +01:00
AnythingElseAttributeValueUnquoted:
LibWeb: Remove StringBuilders from HTMLToken::AttributeBuilder
2021-05-23 08:50:48 +02:00
m_current_builder.append_code_point(current_input_character.value());
LibWeb: Teach HTMLTokenizer how to tokenize attributes Properly tokenize single-quoted, double-quoted and unquoted attributes!
2020-05-23 01:19:42 +02:00
continue;
}
}
END_STATE
LibWeb: Add spec links to each HTML tokenizer state section I didn't add full spec comments this time, but this is better than nothing :^)
2022-02-15 18:12:15 +00:00
// 13.2.5.39 After attribute value (quoted) state, https://html.spec.whatwg.org/multipage/parsing.html#after-attribute-value-(quoted)-state
LibWeb: Teach HTMLTokenizer how to tokenize attributes Properly tokenize single-quoted, double-quoted and unquoted attributes!
2020-05-23 01:19:42 +02:00
BEGIN_STATE(AfterAttributeValueQuoted)
{
LibWeb: Hide implementation details of HTMLToken attribute list Previously, HTMLToken would expose the Vector<Attribute> directly to its users. In preparation for a future change, all users now use implementation-agnostic APIs which do not expose the Vector directly.
2021-07-14 23:53:11 +02:00
m_current_token.last_attribute().value_end_position = nth_last_position(1);
LibWeb: Teach HTMLTokenizer how to tokenize attributes Properly tokenize single-quoted, double-quoted and unquoted attributes!
2020-05-23 01:19:42 +02:00
ON_WHITESPACE
{
SWITCH_TO(BeforeAttributeName);
}
ON('/')
{
SWITCH_TO(SelfClosingStartTag);
}
ON('>')
{
LibWeb: Start building the tree building part of the new HTML parser This patch adds a new HTMLDocumentParser class. It keeps a tokenizer object internally and feeds itself with one token at a time from it. The names and idioms in this class are expressed as closely to the actual HTML parsing spec as possible, to make development as easy and bug free as possible. :^) This is going to become pretty large, but it's pretty cool!
2020-05-24 00:14:23 +02:00
SWITCH_TO_AND_EMIT_CURRENT_TOKEN(Data);
LibWeb: Teach HTMLTokenizer how to tokenize attributes Properly tokenize single-quoted, double-quoted and unquoted attributes!
2020-05-23 01:19:42 +02:00
}
ON_EOF
{
LibWeb: Utilize SourceLocation for HTMLTokenizer logging
2021-04-24 20:22:30 -07:00
log_parse_error();
LibWeb: Parse "textarea" tags during the "in body" insertion mode Had to handle some more cases in the tokenizer to support this.
2020-05-30 18:40:23 +02:00
EMIT_EOF;
LibWeb: Teach HTMLTokenizer how to tokenize attributes Properly tokenize single-quoted, double-quoted and unquoted attributes!
2020-05-23 01:19:42 +02:00
}
ANYTHING_ELSE
{
LibWeb: Utilize SourceLocation for HTMLTokenizer logging
2021-04-24 20:22:30 -07:00
log_parse_error();
LibWeb: Parse "textarea" tags during the "in body" insertion mode Had to handle some more cases in the tokenizer to support this.
2020-05-30 18:40:23 +02:00
RECONSUME_IN(BeforeAttributeName);
LibWeb: Teach HTMLTokenizer how to tokenize attributes Properly tokenize single-quoted, double-quoted and unquoted attributes!
2020-05-23 01:19:42 +02:00
}
}
END_STATE
LibWeb: Add spec links to each HTML tokenizer state section I didn't add full spec comments this time, but this is better than nothing :^)
2022-02-15 18:12:15 +00:00
// 13.2.5.43 Comment start state, https://html.spec.whatwg.org/multipage/parsing.html#comment-start-state
LibWeb: Teach HTMLTokenizer how to tokenize comments We can now correctly tokenize the welcome.html test page. :^)
2020-05-23 01:54:26 +02:00
BEGIN_STATE(CommentStart)
{
ON('-')
{
SWITCH_TO(CommentStartDash);
}
ON('>')
{
LibWeb: Utilize SourceLocation for HTMLTokenizer logging
2021-04-24 20:22:30 -07:00
log_parse_error();
LibWeb: Fully implement all comment tokenizer states
2020-06-11 02:09:07 +01:00
SWITCH_TO_AND_EMIT_CURRENT_TOKEN(Data);
LibWeb: Teach HTMLTokenizer how to tokenize comments We can now correctly tokenize the welcome.html test page. :^)
2020-05-23 01:54:26 +02:00
}
ANYTHING_ELSE
{
RECONSUME_IN(Comment);
}
}
END_STATE
LibWeb: Add spec links to each HTML tokenizer state section I didn't add full spec comments this time, but this is better than nothing :^)
2022-02-15 18:12:15 +00:00
// 13.2.5.44 Comment start dash state, https://html.spec.whatwg.org/multipage/parsing.html#comment-start-dash-state
LibWeb: Teach HTMLTokenizer how to tokenize comments We can now correctly tokenize the welcome.html test page. :^)
2020-05-23 01:54:26 +02:00
BEGIN_STATE(CommentStartDash)
{
ON('-')
{
LibWeb: Correct behavior of Comment* states in HTMLTokenizer Previously, this would lead to assertion failures when parsing HTML comments. This fixes #8757.
2021-07-15 00:35:53 +02:00
SWITCH_TO(CommentEnd);
LibWeb: Teach HTMLTokenizer how to tokenize comments We can now correctly tokenize the welcome.html test page. :^)
2020-05-23 01:54:26 +02:00
}
ON('>')
{
LibWeb: Utilize SourceLocation for HTMLTokenizer logging
2021-04-24 20:22:30 -07:00
log_parse_error();
LibWeb: Fully implement all comment tokenizer states
2020-06-11 02:09:07 +01:00
SWITCH_TO_AND_EMIT_CURRENT_TOKEN(Data);
LibWeb: Teach HTMLTokenizer how to tokenize comments We can now correctly tokenize the welcome.html test page. :^)
2020-05-23 01:54:26 +02:00
}
ON_EOF
{
LibWeb: Utilize SourceLocation for HTMLTokenizer logging
2021-04-24 20:22:30 -07:00
log_parse_error();
LibWeb: Emit the current token before EOF on invalid comments The spec for each of these state: -> EOF: This is an eof-in-comment parse error. Emit the current comment token. Emit an end-of-file token. We were neglecting to emit the current comment token before emitting an EOF token. Note the existing EMIT_CURRENT_TOKEN macro was unused.
2024-03-23 10:11:29 -04:00
EMIT_CURRENT_TOKEN_FOLLOWED_BY_EOF;
LibWeb: Teach HTMLTokenizer how to tokenize comments We can now correctly tokenize the welcome.html test page. :^)
2020-05-23 01:54:26 +02:00
}
ANYTHING_ELSE
{
LibWeb: Remove StringBuilder from HTMLToken::m_comment_or_character
2021-05-23 09:31:40 +02:00
m_current_builder.append('-');
LibWeb: Teach HTMLTokenizer how to tokenize comments We can now correctly tokenize the welcome.html test page. :^)
2020-05-23 01:54:26 +02:00
RECONSUME_IN(Comment);
}
}
END_STATE
LibWeb: Add spec links to each HTML tokenizer state section I didn't add full spec comments this time, but this is better than nothing :^)
2022-02-15 18:12:15 +00:00
// 13.2.5.45 Comment state, https://html.spec.whatwg.org/multipage/parsing.html#comment-state
LibWeb: Teach HTMLTokenizer how to tokenize comments We can now correctly tokenize the welcome.html test page. :^)
2020-05-23 01:54:26 +02:00
BEGIN_STATE(Comment)
{
ON('<')
{
LibWeb: Remove StringBuilder from HTMLToken::m_comment_or_character
2021-05-23 09:31:40 +02:00
m_current_builder.append_code_point(current_input_character.value());
SWITCH_TO_WITH_UNCLEAN_BUILDER(CommentLessThanSign);
LibWeb: Teach HTMLTokenizer how to tokenize comments We can now correctly tokenize the welcome.html test page. :^)
2020-05-23 01:54:26 +02:00
}
ON('-')
{
LibWeb: Remove StringBuilder from HTMLToken::m_comment_or_character
2021-05-23 09:31:40 +02:00
SWITCH_TO_WITH_UNCLEAN_BUILDER(CommentEndDash);
LibWeb: Teach HTMLTokenizer how to tokenize comments We can now correctly tokenize the welcome.html test page. :^)
2020-05-23 01:54:26 +02:00
}
ON(0)
{
LibWeb: Utilize SourceLocation for HTMLTokenizer logging
2021-04-24 20:22:30 -07:00
log_parse_error();
LibWeb: Remove StringBuilder from HTMLToken::m_comment_or_character
2021-05-23 09:31:40 +02:00
m_current_builder.append_code_point(0xFFFD);
LibWeb: Fully implement all comment tokenizer states
2020-06-11 02:09:07 +01:00
continue;
LibWeb: Teach HTMLTokenizer how to tokenize comments We can now correctly tokenize the welcome.html test page. :^)
2020-05-23 01:54:26 +02:00
}
ON_EOF
{
LibWeb: Utilize SourceLocation for HTMLTokenizer logging
2021-04-24 20:22:30 -07:00
log_parse_error();
LibWeb: User getter and setter for Comment type HTMLTokens
2021-07-14 23:32:18 +02:00
m_current_token.set_comment(consume_current_builder());
LibWeb: Emit the current token before EOF on invalid comments The spec for each of these state: -> EOF: This is an eof-in-comment parse error. Emit the current comment token. Emit an end-of-file token. We were neglecting to emit the current comment token before emitting an EOF token. Note the existing EMIT_CURRENT_TOKEN macro was unused.
2024-03-23 10:11:29 -04:00
EMIT_CURRENT_TOKEN_FOLLOWED_BY_EOF;
LibWeb: Teach HTMLTokenizer how to tokenize comments We can now correctly tokenize the welcome.html test page. :^)
2020-05-23 01:54:26 +02:00
}
ANYTHING_ELSE
{
LibWeb: Remove StringBuilder from HTMLToken::m_comment_or_character
2021-05-23 09:31:40 +02:00
m_current_builder.append_code_point(current_input_character.value());
LibWeb: Teach HTMLTokenizer how to tokenize comments We can now correctly tokenize the welcome.html test page. :^)
2020-05-23 01:54:26 +02:00
continue;
}
}
LibWeb: Make BEGIN_STATE and END_STATE include some {{{ and }}} This makes it a compile error to omit the END_STATE. Also add some more missing END_STATE's exposed by this (nice!) Thanks to @predmond for suggesting the multi-pair trick! :^)
2020-05-23 14:04:53 +02:00
END_STATE
LibWeb: Teach HTMLTokenizer how to tokenize comments We can now correctly tokenize the welcome.html test page. :^)
2020-05-23 01:54:26 +02:00
LibWeb: Add spec links to each HTML tokenizer state section I didn't add full spec comments this time, but this is better than nothing :^)
2022-02-15 18:12:15 +00:00
// 13.2.5.51 Comment end state, https://html.spec.whatwg.org/multipage/parsing.html#comment-end-state
LibWeb: Teach HTMLTokenizer how to tokenize comments We can now correctly tokenize the welcome.html test page. :^)
2020-05-23 01:54:26 +02:00
BEGIN_STATE(CommentEnd)
{
ON('>')
{
LibWeb: User getter and setter for Comment type HTMLTokens
2021-07-14 23:32:18 +02:00
m_current_token.set_comment(consume_current_builder());
LibWeb: Start building the tree building part of the new HTML parser This patch adds a new HTMLDocumentParser class. It keeps a tokenizer object internally and feeds itself with one token at a time from it. The names and idioms in this class are expressed as closely to the actual HTML parsing spec as possible, to make development as easy and bug free as possible. :^) This is going to become pretty large, but it's pretty cool!
2020-05-24 00:14:23 +02:00
SWITCH_TO_AND_EMIT_CURRENT_TOKEN(Data);
LibWeb: Teach HTMLTokenizer how to tokenize comments We can now correctly tokenize the welcome.html test page. :^)
2020-05-23 01:54:26 +02:00
}
ON('!')
{
LibWeb: Correct behavior of Comment* states in HTMLTokenizer Previously, this would lead to assertion failures when parsing HTML comments. This fixes #8757.
2021-07-15 00:35:53 +02:00
SWITCH_TO_WITH_UNCLEAN_BUILDER(CommentEndBang);
LibWeb: Teach HTMLTokenizer how to tokenize comments We can now correctly tokenize the welcome.html test page. :^)
2020-05-23 01:54:26 +02:00
}
ON('-')
{
LibWeb: Remove StringBuilder from HTMLToken::m_comment_or_character
2021-05-23 09:31:40 +02:00
m_current_builder.append('-');
LibWeb: Teach HTMLTokenizer how to tokenize comments We can now correctly tokenize the welcome.html test page. :^)
2020-05-23 01:54:26 +02:00
continue;
}
ON_EOF
{
LibWeb: Utilize SourceLocation for HTMLTokenizer logging
2021-04-24 20:22:30 -07:00
log_parse_error();
LibWeb: User getter and setter for Comment type HTMLTokens
2021-07-14 23:32:18 +02:00
m_current_token.set_comment(consume_current_builder());
LibWeb: Emit the current token before EOF on invalid comments The spec for each of these state: -> EOF: This is an eof-in-comment parse error. Emit the current comment token. Emit an end-of-file token. We were neglecting to emit the current comment token before emitting an EOF token. Note the existing EMIT_CURRENT_TOKEN macro was unused.
2024-03-23 10:11:29 -04:00
EMIT_CURRENT_TOKEN_FOLLOWED_BY_EOF;
LibWeb: Teach HTMLTokenizer how to tokenize comments We can now correctly tokenize the welcome.html test page. :^)
2020-05-23 01:54:26 +02:00
}
ANYTHING_ELSE
{
Everywhere: Add sv suffix to strings relying on StringView(char const*) Each of these strings would previously rely on StringView's char const* constructor overload, which would call __builtin_strlen on the string. Since we now have operator ""sv, we can replace these with much simpler versions. This opens the door to being able to remove StringView(char const*). No functional changes.
2022-07-11 17:32:29 +00:00
m_current_builder.append("--"sv);
LibWeb: Teach HTMLTokenizer how to tokenize comments We can now correctly tokenize the welcome.html test page. :^)
2020-05-23 01:54:26 +02:00
RECONSUME_IN(Comment);
}
}
END_STATE
LibWeb: Add spec links to each HTML tokenizer state section I didn't add full spec comments this time, but this is better than nothing :^)
2022-02-15 18:12:15 +00:00
// 13.2.5.52 Comment end bang state, https://html.spec.whatwg.org/multipage/parsing.html#comment-end-bang-state
LibWeb: Teach HTMLTokenizer how to tokenize comments We can now correctly tokenize the welcome.html test page. :^)
2020-05-23 01:54:26 +02:00
BEGIN_STATE(CommentEndBang)
{
ON('-')
{
Everywhere: Add sv suffix to strings relying on StringView(char const*) Each of these strings would previously rely on StringView's char const* constructor overload, which would call __builtin_strlen on the string. Since we now have operator ""sv, we can replace these with much simpler versions. This opens the door to being able to remove StringView(char const*). No functional changes.
2022-07-11 17:32:29 +00:00
m_current_builder.append("--!"sv);
LibWeb: Correct behavior of Comment* states in HTMLTokenizer Previously, this would lead to assertion failures when parsing HTML comments. This fixes #8757.
2021-07-15 00:35:53 +02:00
SWITCH_TO_WITH_UNCLEAN_BUILDER(CommentEndDash);
LibWeb: Teach HTMLTokenizer how to tokenize comments We can now correctly tokenize the welcome.html test page. :^)
2020-05-23 01:54:26 +02:00
}
ON('>')
{
LibWeb: Utilize SourceLocation for HTMLTokenizer logging
2021-04-24 20:22:30 -07:00
log_parse_error();
LibWeb: User getter and setter for Comment type HTMLTokens
2021-07-14 23:32:18 +02:00
m_current_token.set_comment(consume_current_builder());
LibWeb: Fully implement all comment tokenizer states
2020-06-11 02:09:07 +01:00
SWITCH_TO_AND_EMIT_CURRENT_TOKEN(Data);
LibWeb: Teach HTMLTokenizer how to tokenize comments We can now correctly tokenize the welcome.html test page. :^)
2020-05-23 01:54:26 +02:00
}
ON_EOF
{
LibWeb: Utilize SourceLocation for HTMLTokenizer logging
2021-04-24 20:22:30 -07:00
log_parse_error();
LibWeb: User getter and setter for Comment type HTMLTokens
2021-07-14 23:32:18 +02:00
m_current_token.set_comment(consume_current_builder());
LibWeb: Emit the current token before EOF on invalid comments The spec for each of these state: -> EOF: This is an eof-in-comment parse error. Emit the current comment token. Emit an end-of-file token. We were neglecting to emit the current comment token before emitting an EOF token. Note the existing EMIT_CURRENT_TOKEN macro was unused.
2024-03-23 10:11:29 -04:00
EMIT_CURRENT_TOKEN_FOLLOWED_BY_EOF;
LibWeb: Teach HTMLTokenizer how to tokenize comments We can now correctly tokenize the welcome.html test page. :^)
2020-05-23 01:54:26 +02:00
}
ANYTHING_ELSE
{
Everywhere: Add sv suffix to strings relying on StringView(char const*) Each of these strings would previously rely on StringView's char const* constructor overload, which would call __builtin_strlen on the string. Since we now have operator ""sv, we can replace these with much simpler versions. This opens the door to being able to remove StringView(char const*). No functional changes.
2022-07-11 17:32:29 +00:00
m_current_builder.append("--!"sv);
LibWeb: Teach HTMLTokenizer how to tokenize comments We can now correctly tokenize the welcome.html test page. :^)
2020-05-23 01:54:26 +02:00
RECONSUME_IN(Comment);
}
}
END_STATE
LibWeb: Add spec links to each HTML tokenizer state section I didn't add full spec comments this time, but this is better than nothing :^)
2022-02-15 18:12:15 +00:00
// 13.2.5.50 Comment end dash state, https://html.spec.whatwg.org/multipage/parsing.html#comment-end-dash-state
LibWeb: Teach HTMLTokenizer how to tokenize comments We can now correctly tokenize the welcome.html test page. :^)
2020-05-23 01:54:26 +02:00
BEGIN_STATE(CommentEndDash)
{
ON('-')
{
LibWeb: Remove StringBuilder from HTMLToken::m_comment_or_character
2021-05-23 09:31:40 +02:00
SWITCH_TO_WITH_UNCLEAN_BUILDER(CommentEnd);
LibWeb: Teach HTMLTokenizer how to tokenize comments We can now correctly tokenize the welcome.html test page. :^)
2020-05-23 01:54:26 +02:00
}
ON_EOF
{
LibWeb: Utilize SourceLocation for HTMLTokenizer logging
2021-04-24 20:22:30 -07:00
log_parse_error();
LibWeb: User getter and setter for Comment type HTMLTokens
2021-07-14 23:32:18 +02:00
m_current_token.set_comment(consume_current_builder());
LibWeb: Emit the current token before EOF on invalid comments The spec for each of these state: -> EOF: This is an eof-in-comment parse error. Emit the current comment token. Emit an end-of-file token. We were neglecting to emit the current comment token before emitting an EOF token. Note the existing EMIT_CURRENT_TOKEN macro was unused.
2024-03-23 10:11:29 -04:00
EMIT_CURRENT_TOKEN_FOLLOWED_BY_EOF;
LibWeb: Teach HTMLTokenizer how to tokenize comments We can now correctly tokenize the welcome.html test page. :^)
2020-05-23 01:54:26 +02:00
}
ANYTHING_ELSE
{
LibWeb: Remove StringBuilder from HTMLToken::m_comment_or_character
2021-05-23 09:31:40 +02:00
m_current_builder.append('-');
LibWeb: Teach HTMLTokenizer how to tokenize comments We can now correctly tokenize the welcome.html test page. :^)
2020-05-23 01:54:26 +02:00
RECONSUME_IN(Comment);
}
}
LibWeb: Make BEGIN_STATE and END_STATE include some {{{ and }}} This makes it a compile error to omit the END_STATE. Also add some more missing END_STATE's exposed by this (nice!) Thanks to @predmond for suggesting the multi-pair trick! :^)
2020-05-23 14:04:53 +02:00
END_STATE
LibWeb: Teach HTMLTokenizer how to tokenize comments We can now correctly tokenize the welcome.html test page. :^)
2020-05-23 01:54:26 +02:00
LibWeb: Add spec links to each HTML tokenizer state section I didn't add full spec comments this time, but this is better than nothing :^)
2022-02-15 18:12:15 +00:00
// 13.2.5.46 Comment less-than sign state, https://html.spec.whatwg.org/multipage/parsing.html#comment-less-than-sign-state
LibWeb: Teach HTMLTokenizer how to tokenize comments We can now correctly tokenize the welcome.html test page. :^)
2020-05-23 01:54:26 +02:00
BEGIN_STATE(CommentLessThanSign)
{
ON('!')
{
LibWeb: Remove StringBuilder from HTMLToken::m_comment_or_character
2021-05-23 09:31:40 +02:00
m_current_builder.append_code_point(current_input_character.value());
SWITCH_TO_WITH_UNCLEAN_BUILDER(CommentLessThanSignBang);
LibWeb: Teach HTMLTokenizer how to tokenize comments We can now correctly tokenize the welcome.html test page. :^)
2020-05-23 01:54:26 +02:00
}
ON('<')
{
LibWeb: Remove StringBuilder from HTMLToken::m_comment_or_character
2021-05-23 09:31:40 +02:00
m_current_builder.append_code_point(current_input_character.value());
LibWeb: Teach HTMLTokenizer how to tokenize comments We can now correctly tokenize the welcome.html test page. :^)
2020-05-23 01:54:26 +02:00
continue;
}
ANYTHING_ELSE
{
RECONSUME_IN(Comment);
}
}
END_STATE
LibWeb: Add spec links to each HTML tokenizer state section I didn't add full spec comments this time, but this is better than nothing :^)
2022-02-15 18:12:15 +00:00
// 13.2.5.47 Comment less-than sign bang state, https://html.spec.whatwg.org/multipage/parsing.html#comment-less-than-sign-bang-state
LibWeb: Teach HTMLTokenizer how to tokenize comments We can now correctly tokenize the welcome.html test page. :^)
2020-05-23 01:54:26 +02:00
BEGIN_STATE(CommentLessThanSignBang)
{
ON('-')
{
LibWeb: Correct behavior of Comment* states in HTMLTokenizer Previously, this would lead to assertion failures when parsing HTML comments. This fixes #8757.
2021-07-15 00:35:53 +02:00
SWITCH_TO_WITH_UNCLEAN_BUILDER(CommentLessThanSignBangDash);
LibWeb: Teach HTMLTokenizer how to tokenize comments We can now correctly tokenize the welcome.html test page. :^)
2020-05-23 01:54:26 +02:00
}
ANYTHING_ELSE
{
RECONSUME_IN(Comment);
}
}
END_STATE
LibWeb: Add spec links to each HTML tokenizer state section I didn't add full spec comments this time, but this is better than nothing :^)
2022-02-15 18:12:15 +00:00
// 13.2.5.48 Comment less-than sign bang dash state, https://html.spec.whatwg.org/multipage/parsing.html#comment-less-than-sign-bang-dash-state
LibWeb: Teach HTMLTokenizer how to tokenize comments We can now correctly tokenize the welcome.html test page. :^)
2020-05-23 01:54:26 +02:00
BEGIN_STATE(CommentLessThanSignBangDash)
{
ON('-')
{
LibWeb: Correct behavior of Comment* states in HTMLTokenizer Previously, this would lead to assertion failures when parsing HTML comments. This fixes #8757.
2021-07-15 00:35:53 +02:00
SWITCH_TO_WITH_UNCLEAN_BUILDER(CommentLessThanSignBangDashDash);
LibWeb: Teach HTMLTokenizer how to tokenize comments We can now correctly tokenize the welcome.html test page. :^)
2020-05-23 01:54:26 +02:00
}
ANYTHING_ELSE
{
LibWeb: Fully implement all comment tokenizer states
2020-06-11 02:09:07 +01:00
RECONSUME_IN(CommentEndDash);
LibWeb: Teach HTMLTokenizer how to tokenize comments We can now correctly tokenize the welcome.html test page. :^)
2020-05-23 01:54:26 +02:00
}
}
END_STATE
LibWeb: Add spec links to each HTML tokenizer state section I didn't add full spec comments this time, but this is better than nothing :^)
2022-02-15 18:12:15 +00:00
// 13.2.5.49 Comment less-than sign bang dash dash state, https://html.spec.whatwg.org/multipage/parsing.html#comment-less-than-sign-bang-dash-dash-state
LibWeb: Teach HTMLTokenizer how to tokenize comments We can now correctly tokenize the welcome.html test page. :^)
2020-05-23 01:54:26 +02:00
BEGIN_STATE(CommentLessThanSignBangDashDash)
{
ON('>')
{
LibWeb: Fully implement all comment tokenizer states
2020-06-11 02:09:07 +01:00
RECONSUME_IN(CommentEnd);
}
ON_EOF
{
RECONSUME_IN(CommentEnd);
LibWeb: Teach HTMLTokenizer how to tokenize comments We can now correctly tokenize the welcome.html test page. :^)
2020-05-23 01:54:26 +02:00
}
ANYTHING_ELSE
{
LibWeb: Utilize SourceLocation for HTMLTokenizer logging
2021-04-24 20:22:30 -07:00
log_parse_error();
LibWeb: Tokenize "anything else" in CommentLessThanSignBangDashDash
2020-06-01 20:14:23 +02:00
RECONSUME_IN(CommentEnd);
LibWeb: Teach HTMLTokenizer how to tokenize comments We can now correctly tokenize the welcome.html test page. :^)
2020-05-23 01:54:26 +02:00
}
}
END_STATE
LibWeb: Add spec links to each HTML tokenizer state section I didn't add full spec comments this time, but this is better than nothing :^)
2022-02-15 18:12:15 +00:00
// 13.2.5.72 Character reference state, https://html.spec.whatwg.org/multipage/parsing.html#character-reference-state
LibWeb: Begin work on a spec-compliant HTML parser In order to actually view the web as it is, we're gonna need a proper HTML parser. So let's build one! This patch introduces the Web::HTMLTokenizer class, which currently operates on a StringView input stream where it fetches (ASCII only atm) codepoints and tokenizes acccording to the HTML spec tokenization algo. The tokenizer state machine looks a bit weird but is written in a way that tries to mimic the spec as closely as possible, in order to make development easier and bugs less likely. This initial version is far from finished, but it can parse a trivial document with a DOCTYPE and open/close tags. :^)
2020-05-22 21:46:13 +02:00
BEGIN_STATE(CharacterReference)
{
LibWeb: Implement partial support for numeric character references
2020-05-27 16:16:23 +02:00
m_temporary_buffer.clear();
m_temporary_buffer.append('&');
ON_ASCII_ALPHANUMERIC
{
LibWeb: Make named character references more spec-compliant & efficient There are two changes happening here: a correctness fix, and an optimization. In theory they are unrelated, but the optimization actually paves the way for the correctness fix. Before this commit, the HTML tokenizer would attempt to look for named character references by checking from after the `&` until the end of m_decoded_input, which meant that it was unable to recognize things like named character references that are inserted via `document.write` one byte at a time. For example, if `&notin;` was written one-byte-at-a-time with `document.write`, then the tokenizer would only check against `n` since that's all that would exist at the time of the check and therefore erroneously conclude that it was an invalid named character reference. This commit modifies the approach taken for named character reference matching by using a trie-like structure (specifically, a deterministic acyclic finite state automaton or DAFSA), which allows for efficiently matching one-character-at-a-time and therefore it is able to pick up matching where it left off after each code point is consumed. Note: Because it's possible for a partial match to not actually develop into a full match (e.g. `&notindo` which could lead to `&notindot;`), some backtracking is performed after-the-fact in order to only consume the code points within the longest match found (e.g. `&notindo` would backtrack back to `&not`). With this new approach, `document.write` being called one-byte-at-a-time is handled correctly, which allows for passing more WPT tests, with the most directly relevant tests being `/html/syntax/parsing/html5lib_entities01.html` and `/html/syntax/parsing/html5lib_entities02.html` when run with `?run_type=write_single`. Additionally, the implementation now better conforms to the language of the spec (and resolves a FIXME) because exactly the matched characters are consumed and nothing more, so SWITCH_TO is able to be used as the spec says instead of RECONSUME_IN. The new approach is also an optimization: - Instead of a linear search using `starts_with`, the usage of a DAFSA means that it is always aware of which characters can lead to a match at any given point, and will bail out whenever a match is no longer possible. - The DAFSA is able to take advantage of the note in the section `13.5 Named character references` that says "This list is static and will not be expanded or changed in the future." and tailor its Node struct accordingly to tightly pack each node's data into 32-bits. Together with the inherent DAFSA property of redundant node deduplication, the amount of data stored for named character reference matching is minimized. In my testing: - A benchmark tokenizing an arbitrary set of HTML test files was about 1.23x faster (2070ms to 1682ms). - A benchmark tokenizing a file with tens of thousands of named character references mixed in with truncated named character references and arbitrary ASCII characters/ampersands runs about 8x faster (758ms to 93ms). - The size of `liblagom-web.so` was reduced by 94.96KiB. Some technical details: A DAFSA (deterministic acyclic finite state automaton) is essentially a trie flattened into an array, but it also uses techniques to minimize redundant nodes. This provides fast lookups while minimizing the required data size, but normally does not allow for associating data related to each word. However, by adding a count of the number of possible words from each node, it becomes possible to also use it to achieve minimal perfect hashing for the set of words (which allows going from word -> unique index as well as unique index -> word). This allows us to store a second array of data so that the DAFSA can be used as a lookup for e.g. the associated code points. For the Swift implementation, the new NamedCharacterReferenceMatcher was used to satisfy the previous API and the tokenizer was left alone otherwise. In the future, the Swift implementation should be updated to use the same implementation for its NamedCharacterReference state as the updated C++ implementation.
2024-12-22 07:09:31 -08:00
m_named_character_reference_matcher = {};
LibWeb: Implement partial support for numeric character references
2020-05-27 16:16:23 +02:00
RECONSUME_IN(NamedCharacterReference);
}
ON('#')
{
m_temporary_buffer.append(current_input_character.value());
LibWeb: Avoid assertion failure on parsing numeric character references
2021-07-28 00:37:26 +01:00
SWITCH_TO_WITH_UNCLEAN_BUILDER(NumericCharacterReference);
LibWeb: Implement partial support for numeric character references
2020-05-27 16:16:23 +02:00
}
ANYTHING_ELSE
{
LibWeb: Fix character references losing characters in certain situations This fixes 4 issues: - RECONSUME_IN_RETURN_STATE was functionally equivalent to SWITCH_TO_RETURN_STATE, which caused us to lose characters. For example, &test= would lose the = - & characters by themselves would be lost. For example, 1 & 2 would become 1 2. This is because we forgot to flush characters in the the ANYTHING_ELSE path in CharacterReference - Named character references didn't work at all in attributes. This is because there was a path that was checking the entity code points instead of the entity itself. Plus, the path that was checking the entity itself wasn't quite spec compliant. - If we fail to match a named character reference, the first character is lost. For example &test would become &est. However, this relies on a little hack since I can't wrap my head around on how to change the code to do as the spec says. The hack is to reconsume in AmbigiousAmpersand instead of just switching to it. Fixes #3957
2020-12-29 14:42:47 +00:00
FLUSH_CODEPOINTS_CONSUMED_AS_A_CHARACTER_REFERENCE;
LibWeb: Implement partial support for numeric character references
2020-05-27 16:16:23 +02:00
RECONSUME_IN_RETURN_STATE;
}
}
END_STATE
LibWeb: Add spec links to each HTML tokenizer state section I didn't add full spec comments this time, but this is better than nothing :^)
2022-02-15 18:12:15 +00:00
// 13.2.5.73 Named character reference state, https://html.spec.whatwg.org/multipage/parsing.html#named-character-reference-state
LibWeb: Implement partial support for numeric character references
2020-05-27 16:16:23 +02:00
BEGIN_STATE(NamedCharacterReference)
{
LibWeb/HTML: Use lookahead when possible for named character references When there is an active insertion point, it's necessary to tokenize code-point-by-code-point to handle the case of document.write being used to insert a named character reference one code point at a time. However, when there is no insertion point defined, looking ahead at the input and doing the matching all-at-once is more efficient since it allows: - Avoiding the work done in next_code_point between each code point being matched (leading to better CPU cache usage in theory) - Skipping ahead to the end of the match all at once, which does less work overall than the equivalent number of next_code_point calls (that is, skip(N) does less work than next_code_point called N times) In my benchmarking, this provides a small performance boost (fewer instructions, fewer cpu cycles, fewer branch misses) essentially for free.
2025-07-04 01:04:01 -07:00
if (stop_at_insertion_point == StopAtInsertionPoint::Yes && is_insertion_point_defined()) {
// If there is an insertion point, match code-point-by-code-point to handle the possibility of
// document.write being used to insert a named character reference one-code-point-at-a-time.
if (current_input_character.has_value()) {
if (m_named_character_reference_matcher.try_consume_code_point(current_input_character.value())) {
m_temporary_buffer.append(current_input_character.value());
continue;
} else {
DONT_CONSUME_NEXT_INPUT_CHARACTER;
}
}
} else {
// If there's no insertion point (this is the common case), it is safe to look ahead at the rest
// of the input and try to match a named character reference all-at-once. This is worthwhile
// because matching all-at-once ends up being more efficient.
auto starting_consumed_count = m_temporary_buffer.size();
auto remaining_source = m_decoded_input.span().slice(m_prev_offset);
for (auto const code_point : remaining_source) {
if (m_named_character_reference_matcher.try_consume_code_point(code_point)) {
m_temporary_buffer.append(code_point);
} else {
break;
}
}
auto num_consumed = m_temporary_buffer.size() - starting_consumed_count;
if (num_consumed == 0) {
LibWeb: Make named character references more spec-compliant & efficient There are two changes happening here: a correctness fix, and an optimization. In theory they are unrelated, but the optimization actually paves the way for the correctness fix. Before this commit, the HTML tokenizer would attempt to look for named character references by checking from after the `&` until the end of m_decoded_input, which meant that it was unable to recognize things like named character references that are inserted via `document.write` one byte at a time. For example, if `&notin;` was written one-byte-at-a-time with `document.write`, then the tokenizer would only check against `n` since that's all that would exist at the time of the check and therefore erroneously conclude that it was an invalid named character reference. This commit modifies the approach taken for named character reference matching by using a trie-like structure (specifically, a deterministic acyclic finite state automaton or DAFSA), which allows for efficiently matching one-character-at-a-time and therefore it is able to pick up matching where it left off after each code point is consumed. Note: Because it's possible for a partial match to not actually develop into a full match (e.g. `&notindo` which could lead to `&notindot;`), some backtracking is performed after-the-fact in order to only consume the code points within the longest match found (e.g. `&notindo` would backtrack back to `&not`). With this new approach, `document.write` being called one-byte-at-a-time is handled correctly, which allows for passing more WPT tests, with the most directly relevant tests being `/html/syntax/parsing/html5lib_entities01.html` and `/html/syntax/parsing/html5lib_entities02.html` when run with `?run_type=write_single`. Additionally, the implementation now better conforms to the language of the spec (and resolves a FIXME) because exactly the matched characters are consumed and nothing more, so SWITCH_TO is able to be used as the spec says instead of RECONSUME_IN. The new approach is also an optimization: - Instead of a linear search using `starts_with`, the usage of a DAFSA means that it is always aware of which characters can lead to a match at any given point, and will bail out whenever a match is no longer possible. - The DAFSA is able to take advantage of the note in the section `13.5 Named character references` that says "This list is static and will not be expanded or changed in the future." and tailor its Node struct accordingly to tightly pack each node's data into 32-bits. Together with the inherent DAFSA property of redundant node deduplication, the amount of data stored for named character reference matching is minimized. In my testing: - A benchmark tokenizing an arbitrary set of HTML test files was about 1.23x faster (2070ms to 1682ms). - A benchmark tokenizing a file with tens of thousands of named character references mixed in with truncated named character references and arbitrary ASCII characters/ampersands runs about 8x faster (758ms to 93ms). - The size of `liblagom-web.so` was reduced by 94.96KiB. Some technical details: A DAFSA (deterministic acyclic finite state automaton) is essentially a trie flattened into an array, but it also uses techniques to minimize redundant nodes. This provides fast lookups while minimizing the required data size, but normally does not allow for associating data related to each word. However, by adding a count of the number of possible words from each node, it becomes possible to also use it to achieve minimal perfect hashing for the set of words (which allows going from word -> unique index as well as unique index -> word). This allows us to store a second array of data so that the DAFSA can be used as a lookup for e.g. the associated code points. For the Swift implementation, the new NamedCharacterReferenceMatcher was used to satisfy the previous API and the tokenizer was left alone otherwise. In the future, the Swift implementation should be updated to use the same implementation for its NamedCharacterReference state as the updated C++ implementation.
2024-12-22 07:09:31 -08:00
DONT_CONSUME_NEXT_INPUT_CHARACTER;
LibWeb/HTML: Use lookahead when possible for named character references When there is an active insertion point, it's necessary to tokenize code-point-by-code-point to handle the case of document.write being used to insert a named character reference one code point at a time. However, when there is no insertion point defined, looking ahead at the input and doing the matching all-at-once is more efficient since it allows: - Avoiding the work done in next_code_point between each code point being matched (leading to better CPU cache usage in theory) - Skipping ahead to the end of the match all at once, which does less work overall than the equivalent number of next_code_point calls (that is, skip(N) does less work than next_code_point called N times) In my benchmarking, this provides a small performance boost (fewer instructions, fewer cpu cycles, fewer branch misses) essentially for free.
2025-07-04 01:04:01 -07:00
} else {
skip(num_consumed - 1);
LibWeb: Make named character references more spec-compliant & efficient There are two changes happening here: a correctness fix, and an optimization. In theory they are unrelated, but the optimization actually paves the way for the correctness fix. Before this commit, the HTML tokenizer would attempt to look for named character references by checking from after the `&` until the end of m_decoded_input, which meant that it was unable to recognize things like named character references that are inserted via `document.write` one byte at a time. For example, if `&notin;` was written one-byte-at-a-time with `document.write`, then the tokenizer would only check against `n` since that's all that would exist at the time of the check and therefore erroneously conclude that it was an invalid named character reference. This commit modifies the approach taken for named character reference matching by using a trie-like structure (specifically, a deterministic acyclic finite state automaton or DAFSA), which allows for efficiently matching one-character-at-a-time and therefore it is able to pick up matching where it left off after each code point is consumed. Note: Because it's possible for a partial match to not actually develop into a full match (e.g. `&notindo` which could lead to `&notindot;`), some backtracking is performed after-the-fact in order to only consume the code points within the longest match found (e.g. `&notindo` would backtrack back to `&not`). With this new approach, `document.write` being called one-byte-at-a-time is handled correctly, which allows for passing more WPT tests, with the most directly relevant tests being `/html/syntax/parsing/html5lib_entities01.html` and `/html/syntax/parsing/html5lib_entities02.html` when run with `?run_type=write_single`. Additionally, the implementation now better conforms to the language of the spec (and resolves a FIXME) because exactly the matched characters are consumed and nothing more, so SWITCH_TO is able to be used as the spec says instead of RECONSUME_IN. The new approach is also an optimization: - Instead of a linear search using `starts_with`, the usage of a DAFSA means that it is always aware of which characters can lead to a match at any given point, and will bail out whenever a match is no longer possible. - The DAFSA is able to take advantage of the note in the section `13.5 Named character references` that says "This list is static and will not be expanded or changed in the future." and tailor its Node struct accordingly to tightly pack each node's data into 32-bits. Together with the inherent DAFSA property of redundant node deduplication, the amount of data stored for named character reference matching is minimized. In my testing: - A benchmark tokenizing an arbitrary set of HTML test files was about 1.23x faster (2070ms to 1682ms). - A benchmark tokenizing a file with tens of thousands of named character references mixed in with truncated named character references and arbitrary ASCII characters/ampersands runs about 8x faster (758ms to 93ms). - The size of `liblagom-web.so` was reduced by 94.96KiB. Some technical details: A DAFSA (deterministic acyclic finite state automaton) is essentially a trie flattened into an array, but it also uses techniques to minimize redundant nodes. This provides fast lookups while minimizing the required data size, but normally does not allow for associating data related to each word. However, by adding a count of the number of possible words from each node, it becomes possible to also use it to achieve minimal perfect hashing for the set of words (which allows going from word -> unique index as well as unique index -> word). This allows us to store a second array of data so that the DAFSA can be used as a lookup for e.g. the associated code points. For the Swift implementation, the new NamedCharacterReferenceMatcher was used to satisfy the previous API and the tokenizer was left alone otherwise. In the future, the Swift implementation should be updated to use the same implementation for its NamedCharacterReference state as the updated C++ implementation.
2024-12-22 07:09:31 -08:00
}
}
LibWeb: Support named character references (e.g "&amp;")
2020-05-28 11:44:19 +02:00
LibWeb: Make named character references more spec-compliant & efficient There are two changes happening here: a correctness fix, and an optimization. In theory they are unrelated, but the optimization actually paves the way for the correctness fix. Before this commit, the HTML tokenizer would attempt to look for named character references by checking from after the `&` until the end of m_decoded_input, which meant that it was unable to recognize things like named character references that are inserted via `document.write` one byte at a time. For example, if `&notin;` was written one-byte-at-a-time with `document.write`, then the tokenizer would only check against `n` since that's all that would exist at the time of the check and therefore erroneously conclude that it was an invalid named character reference. This commit modifies the approach taken for named character reference matching by using a trie-like structure (specifically, a deterministic acyclic finite state automaton or DAFSA), which allows for efficiently matching one-character-at-a-time and therefore it is able to pick up matching where it left off after each code point is consumed. Note: Because it's possible for a partial match to not actually develop into a full match (e.g. `&notindo` which could lead to `&notindot;`), some backtracking is performed after-the-fact in order to only consume the code points within the longest match found (e.g. `&notindo` would backtrack back to `&not`). With this new approach, `document.write` being called one-byte-at-a-time is handled correctly, which allows for passing more WPT tests, with the most directly relevant tests being `/html/syntax/parsing/html5lib_entities01.html` and `/html/syntax/parsing/html5lib_entities02.html` when run with `?run_type=write_single`. Additionally, the implementation now better conforms to the language of the spec (and resolves a FIXME) because exactly the matched characters are consumed and nothing more, so SWITCH_TO is able to be used as the spec says instead of RECONSUME_IN. The new approach is also an optimization: - Instead of a linear search using `starts_with`, the usage of a DAFSA means that it is always aware of which characters can lead to a match at any given point, and will bail out whenever a match is no longer possible. - The DAFSA is able to take advantage of the note in the section `13.5 Named character references` that says "This list is static and will not be expanded or changed in the future." and tailor its Node struct accordingly to tightly pack each node's data into 32-bits. Together with the inherent DAFSA property of redundant node deduplication, the amount of data stored for named character reference matching is minimized. In my testing: - A benchmark tokenizing an arbitrary set of HTML test files was about 1.23x faster (2070ms to 1682ms). - A benchmark tokenizing a file with tens of thousands of named character references mixed in with truncated named character references and arbitrary ASCII characters/ampersands runs about 8x faster (758ms to 93ms). - The size of `liblagom-web.so` was reduced by 94.96KiB. Some technical details: A DAFSA (deterministic acyclic finite state automaton) is essentially a trie flattened into an array, but it also uses techniques to minimize redundant nodes. This provides fast lookups while minimizing the required data size, but normally does not allow for associating data related to each word. However, by adding a count of the number of possible words from each node, it becomes possible to also use it to achieve minimal perfect hashing for the set of words (which allows going from word -> unique index as well as unique index -> word). This allows us to store a second array of data so that the DAFSA can be used as a lookup for e.g. the associated code points. For the Swift implementation, the new NamedCharacterReferenceMatcher was used to satisfy the previous API and the tokenizer was left alone otherwise. In the future, the Swift implementation should be updated to use the same implementation for its NamedCharacterReference state as the updated C++ implementation.
2024-12-22 07:09:31 -08:00
// Only consume the characters within the longest match. It's possible that we've overconsumed code points,
// though, so we want to backtrack to the longest match found. For example, `&notindo` (which could still
LibWeb/HTML: Use lookahead when possible for named character references When there is an active insertion point, it's necessary to tokenize code-point-by-code-point to handle the case of document.write being used to insert a named character reference one code point at a time. However, when there is no insertion point defined, looking ahead at the input and doing the matching all-at-once is more efficient since it allows: - Avoiding the work done in next_code_point between each code point being matched (leading to better CPU cache usage in theory) - Skipping ahead to the end of the match all at once, which does less work overall than the equivalent number of next_code_point calls (that is, skip(N) does less work than next_code_point called N times) In my benchmarking, this provides a small performance boost (fewer instructions, fewer cpu cycles, fewer branch misses) essentially for free.
2025-07-04 01:04:01 -07:00
// have lead to `&notindot;`) would need to backtrack back to `&not`.
LibWeb: Make named character references more spec-compliant & efficient There are two changes happening here: a correctness fix, and an optimization. In theory they are unrelated, but the optimization actually paves the way for the correctness fix. Before this commit, the HTML tokenizer would attempt to look for named character references by checking from after the `&` until the end of m_decoded_input, which meant that it was unable to recognize things like named character references that are inserted via `document.write` one byte at a time. For example, if `&notin;` was written one-byte-at-a-time with `document.write`, then the tokenizer would only check against `n` since that's all that would exist at the time of the check and therefore erroneously conclude that it was an invalid named character reference. This commit modifies the approach taken for named character reference matching by using a trie-like structure (specifically, a deterministic acyclic finite state automaton or DAFSA), which allows for efficiently matching one-character-at-a-time and therefore it is able to pick up matching where it left off after each code point is consumed. Note: Because it's possible for a partial match to not actually develop into a full match (e.g. `&notindo` which could lead to `&notindot;`), some backtracking is performed after-the-fact in order to only consume the code points within the longest match found (e.g. `&notindo` would backtrack back to `&not`). With this new approach, `document.write` being called one-byte-at-a-time is handled correctly, which allows for passing more WPT tests, with the most directly relevant tests being `/html/syntax/parsing/html5lib_entities01.html` and `/html/syntax/parsing/html5lib_entities02.html` when run with `?run_type=write_single`. Additionally, the implementation now better conforms to the language of the spec (and resolves a FIXME) because exactly the matched characters are consumed and nothing more, so SWITCH_TO is able to be used as the spec says instead of RECONSUME_IN. The new approach is also an optimization: - Instead of a linear search using `starts_with`, the usage of a DAFSA means that it is always aware of which characters can lead to a match at any given point, and will bail out whenever a match is no longer possible. - The DAFSA is able to take advantage of the note in the section `13.5 Named character references` that says "This list is static and will not be expanded or changed in the future." and tailor its Node struct accordingly to tightly pack each node's data into 32-bits. Together with the inherent DAFSA property of redundant node deduplication, the amount of data stored for named character reference matching is minimized. In my testing: - A benchmark tokenizing an arbitrary set of HTML test files was about 1.23x faster (2070ms to 1682ms). - A benchmark tokenizing a file with tens of thousands of named character references mixed in with truncated named character references and arbitrary ASCII characters/ampersands runs about 8x faster (758ms to 93ms). - The size of `liblagom-web.so` was reduced by 94.96KiB. Some technical details: A DAFSA (deterministic acyclic finite state automaton) is essentially a trie flattened into an array, but it also uses techniques to minimize redundant nodes. This provides fast lookups while minimizing the required data size, but normally does not allow for associating data related to each word. However, by adding a count of the number of possible words from each node, it becomes possible to also use it to achieve minimal perfect hashing for the set of words (which allows going from word -> unique index as well as unique index -> word). This allows us to store a second array of data so that the DAFSA can be used as a lookup for e.g. the associated code points. For the Swift implementation, the new NamedCharacterReferenceMatcher was used to satisfy the previous API and the tokenizer was left alone otherwise. In the future, the Swift implementation should be updated to use the same implementation for its NamedCharacterReference state as the updated C++ implementation.
2024-12-22 07:09:31 -08:00
auto overconsumed_code_points = m_named_character_reference_matcher.overconsumed_code_points();
if (overconsumed_code_points > 0) {
LibWeb/HTML: Use lookahead when possible for named character references When there is an active insertion point, it's necessary to tokenize code-point-by-code-point to handle the case of document.write being used to insert a named character reference one code point at a time. However, when there is no insertion point defined, looking ahead at the input and doing the matching all-at-once is more efficient since it allows: - Avoiding the work done in next_code_point between each code point being matched (leading to better CPU cache usage in theory) - Skipping ahead to the end of the match all at once, which does less work overall than the equivalent number of next_code_point calls (that is, skip(N) does less work than next_code_point called N times) In my benchmarking, this provides a small performance boost (fewer instructions, fewer cpu cycles, fewer branch misses) essentially for free.
2025-07-04 01:04:01 -07:00
restore_to(m_current_offset - overconsumed_code_points);
LibWeb: Make named character references more spec-compliant & efficient There are two changes happening here: a correctness fix, and an optimization. In theory they are unrelated, but the optimization actually paves the way for the correctness fix. Before this commit, the HTML tokenizer would attempt to look for named character references by checking from after the `&` until the end of m_decoded_input, which meant that it was unable to recognize things like named character references that are inserted via `document.write` one byte at a time. For example, if `&notin;` was written one-byte-at-a-time with `document.write`, then the tokenizer would only check against `n` since that's all that would exist at the time of the check and therefore erroneously conclude that it was an invalid named character reference. This commit modifies the approach taken for named character reference matching by using a trie-like structure (specifically, a deterministic acyclic finite state automaton or DAFSA), which allows for efficiently matching one-character-at-a-time and therefore it is able to pick up matching where it left off after each code point is consumed. Note: Because it's possible for a partial match to not actually develop into a full match (e.g. `&notindo` which could lead to `&notindot;`), some backtracking is performed after-the-fact in order to only consume the code points within the longest match found (e.g. `&notindo` would backtrack back to `&not`). With this new approach, `document.write` being called one-byte-at-a-time is handled correctly, which allows for passing more WPT tests, with the most directly relevant tests being `/html/syntax/parsing/html5lib_entities01.html` and `/html/syntax/parsing/html5lib_entities02.html` when run with `?run_type=write_single`. Additionally, the implementation now better conforms to the language of the spec (and resolves a FIXME) because exactly the matched characters are consumed and nothing more, so SWITCH_TO is able to be used as the spec says instead of RECONSUME_IN. The new approach is also an optimization: - Instead of a linear search using `starts_with`, the usage of a DAFSA means that it is always aware of which characters can lead to a match at any given point, and will bail out whenever a match is no longer possible. - The DAFSA is able to take advantage of the note in the section `13.5 Named character references` that says "This list is static and will not be expanded or changed in the future." and tailor its Node struct accordingly to tightly pack each node's data into 32-bits. Together with the inherent DAFSA property of redundant node deduplication, the amount of data stored for named character reference matching is minimized. In my testing: - A benchmark tokenizing an arbitrary set of HTML test files was about 1.23x faster (2070ms to 1682ms). - A benchmark tokenizing a file with tens of thousands of named character references mixed in with truncated named character references and arbitrary ASCII characters/ampersands runs about 8x faster (758ms to 93ms). - The size of `liblagom-web.so` was reduced by 94.96KiB. Some technical details: A DAFSA (deterministic acyclic finite state automaton) is essentially a trie flattened into an array, but it also uses techniques to minimize redundant nodes. This provides fast lookups while minimizing the required data size, but normally does not allow for associating data related to each word. However, by adding a count of the number of possible words from each node, it becomes possible to also use it to achieve minimal perfect hashing for the set of words (which allows going from word -> unique index as well as unique index -> word). This allows us to store a second array of data so that the DAFSA can be used as a lookup for e.g. the associated code points. For the Swift implementation, the new NamedCharacterReferenceMatcher was used to satisfy the previous API and the tokenizer was left alone otherwise. In the future, the Swift implementation should be updated to use the same implementation for its NamedCharacterReference state as the updated C++ implementation.
2024-12-22 07:09:31 -08:00
m_temporary_buffer.resize_and_keep_capacity(m_temporary_buffer.size() - overconsumed_code_points);
}
LibWeb: Support named character references (e.g "&amp;")
2020-05-28 11:44:19 +02:00
LibWeb: Make named character references more spec-compliant & efficient There are two changes happening here: a correctness fix, and an optimization. In theory they are unrelated, but the optimization actually paves the way for the correctness fix. Before this commit, the HTML tokenizer would attempt to look for named character references by checking from after the `&` until the end of m_decoded_input, which meant that it was unable to recognize things like named character references that are inserted via `document.write` one byte at a time. For example, if `&notin;` was written one-byte-at-a-time with `document.write`, then the tokenizer would only check against `n` since that's all that would exist at the time of the check and therefore erroneously conclude that it was an invalid named character reference. This commit modifies the approach taken for named character reference matching by using a trie-like structure (specifically, a deterministic acyclic finite state automaton or DAFSA), which allows for efficiently matching one-character-at-a-time and therefore it is able to pick up matching where it left off after each code point is consumed. Note: Because it's possible for a partial match to not actually develop into a full match (e.g. `&notindo` which could lead to `&notindot;`), some backtracking is performed after-the-fact in order to only consume the code points within the longest match found (e.g. `&notindo` would backtrack back to `&not`). With this new approach, `document.write` being called one-byte-at-a-time is handled correctly, which allows for passing more WPT tests, with the most directly relevant tests being `/html/syntax/parsing/html5lib_entities01.html` and `/html/syntax/parsing/html5lib_entities02.html` when run with `?run_type=write_single`. Additionally, the implementation now better conforms to the language of the spec (and resolves a FIXME) because exactly the matched characters are consumed and nothing more, so SWITCH_TO is able to be used as the spec says instead of RECONSUME_IN. The new approach is also an optimization: - Instead of a linear search using `starts_with`, the usage of a DAFSA means that it is always aware of which characters can lead to a match at any given point, and will bail out whenever a match is no longer possible. - The DAFSA is able to take advantage of the note in the section `13.5 Named character references` that says "This list is static and will not be expanded or changed in the future." and tailor its Node struct accordingly to tightly pack each node's data into 32-bits. Together with the inherent DAFSA property of redundant node deduplication, the amount of data stored for named character reference matching is minimized. In my testing: - A benchmark tokenizing an arbitrary set of HTML test files was about 1.23x faster (2070ms to 1682ms). - A benchmark tokenizing a file with tens of thousands of named character references mixed in with truncated named character references and arbitrary ASCII characters/ampersands runs about 8x faster (758ms to 93ms). - The size of `liblagom-web.so` was reduced by 94.96KiB. Some technical details: A DAFSA (deterministic acyclic finite state automaton) is essentially a trie flattened into an array, but it also uses techniques to minimize redundant nodes. This provides fast lookups while minimizing the required data size, but normally does not allow for associating data related to each word. However, by adding a count of the number of possible words from each node, it becomes possible to also use it to achieve minimal perfect hashing for the set of words (which allows going from word -> unique index as well as unique index -> word). This allows us to store a second array of data so that the DAFSA can be used as a lookup for e.g. the associated code points. For the Swift implementation, the new NamedCharacterReferenceMatcher was used to satisfy the previous API and the tokenizer was left alone otherwise. In the future, the Swift implementation should be updated to use the same implementation for its NamedCharacterReference state as the updated C++ implementation.
2024-12-22 07:09:31 -08:00
auto mapped_codepoints = m_named_character_reference_matcher.code_points();
// If there is a match
if (mapped_codepoints.has_value()) {
if (consumed_as_part_of_an_attribute() && !m_named_character_reference_matcher.last_match_ends_with_semicolon()) {
LibWeb: Teach more of HTMLTokenizer to stop at the insertion point In particular, input character lookahead now knows how to stop at the insertion point marker if needed. This makes it possible to do amazing things like having document.write() insert doctypes one character at a time.
2024-11-24 10:18:17 +01:00
auto next_code_point = peek_code_point(0, stop_at_insertion_point);
Everywhere: Replace ctype.h to avoid narrowing conversions This replaces ctype.h with CharacterType.h everywhere I could find issues with narrowing conversions. While using it will probably make sense almost everywhere in the future, the most critical places should have been addressed.
2021-06-01 21:18:08 +02:00
if (next_code_point.has_value() && (next_code_point.value() == '=' || is_ascii_alphanumeric(next_code_point.value()))) {
Revert "Unicode: s/codepoint/code_point/g" This reverts commit ea9ac3155d1774f13ac4e9a96605c0e85a8f299e. It replaced "codepoint" with "code_points", not "code_point".
2020-08-05 16:28:19 -04:00
FLUSH_CODEPOINTS_CONSUMED_AS_A_CHARACTER_REFERENCE;
LibWeb: Support named character references (e.g "&amp;")
2020-05-28 11:44:19 +02:00
SWITCH_TO_RETURN_STATE;
}
}
LibWeb: Make named character references more spec-compliant & efficient There are two changes happening here: a correctness fix, and an optimization. In theory they are unrelated, but the optimization actually paves the way for the correctness fix. Before this commit, the HTML tokenizer would attempt to look for named character references by checking from after the `&` until the end of m_decoded_input, which meant that it was unable to recognize things like named character references that are inserted via `document.write` one byte at a time. For example, if `&notin;` was written one-byte-at-a-time with `document.write`, then the tokenizer would only check against `n` since that's all that would exist at the time of the check and therefore erroneously conclude that it was an invalid named character reference. This commit modifies the approach taken for named character reference matching by using a trie-like structure (specifically, a deterministic acyclic finite state automaton or DAFSA), which allows for efficiently matching one-character-at-a-time and therefore it is able to pick up matching where it left off after each code point is consumed. Note: Because it's possible for a partial match to not actually develop into a full match (e.g. `&notindo` which could lead to `&notindot;`), some backtracking is performed after-the-fact in order to only consume the code points within the longest match found (e.g. `&notindo` would backtrack back to `&not`). With this new approach, `document.write` being called one-byte-at-a-time is handled correctly, which allows for passing more WPT tests, with the most directly relevant tests being `/html/syntax/parsing/html5lib_entities01.html` and `/html/syntax/parsing/html5lib_entities02.html` when run with `?run_type=write_single`. Additionally, the implementation now better conforms to the language of the spec (and resolves a FIXME) because exactly the matched characters are consumed and nothing more, so SWITCH_TO is able to be used as the spec says instead of RECONSUME_IN. The new approach is also an optimization: - Instead of a linear search using `starts_with`, the usage of a DAFSA means that it is always aware of which characters can lead to a match at any given point, and will bail out whenever a match is no longer possible. - The DAFSA is able to take advantage of the note in the section `13.5 Named character references` that says "This list is static and will not be expanded or changed in the future." and tailor its Node struct accordingly to tightly pack each node's data into 32-bits. Together with the inherent DAFSA property of redundant node deduplication, the amount of data stored for named character reference matching is minimized. In my testing: - A benchmark tokenizing an arbitrary set of HTML test files was about 1.23x faster (2070ms to 1682ms). - A benchmark tokenizing a file with tens of thousands of named character references mixed in with truncated named character references and arbitrary ASCII characters/ampersands runs about 8x faster (758ms to 93ms). - The size of `liblagom-web.so` was reduced by 94.96KiB. Some technical details: A DAFSA (deterministic acyclic finite state automaton) is essentially a trie flattened into an array, but it also uses techniques to minimize redundant nodes. This provides fast lookups while minimizing the required data size, but normally does not allow for associating data related to each word. However, by adding a count of the number of possible words from each node, it becomes possible to also use it to achieve minimal perfect hashing for the set of words (which allows going from word -> unique index as well as unique index -> word). This allows us to store a second array of data so that the DAFSA can be used as a lookup for e.g. the associated code points. For the Swift implementation, the new NamedCharacterReferenceMatcher was used to satisfy the previous API and the tokenizer was left alone otherwise. In the future, the Swift implementation should be updated to use the same implementation for its NamedCharacterReference state as the updated C++ implementation.
2024-12-22 07:09:31 -08:00
if (!m_named_character_reference_matcher.last_match_ends_with_semicolon()) {
LibWeb: Utilize SourceLocation for HTMLTokenizer logging
2021-04-24 20:22:30 -07:00
log_parse_error();
LibWeb: Support named character references (e.g "&amp;")
2020-05-28 11:44:19 +02:00
}
LibWeb: Make named character references more spec-compliant & efficient There are two changes happening here: a correctness fix, and an optimization. In theory they are unrelated, but the optimization actually paves the way for the correctness fix. Before this commit, the HTML tokenizer would attempt to look for named character references by checking from after the `&` until the end of m_decoded_input, which meant that it was unable to recognize things like named character references that are inserted via `document.write` one byte at a time. For example, if `&notin;` was written one-byte-at-a-time with `document.write`, then the tokenizer would only check against `n` since that's all that would exist at the time of the check and therefore erroneously conclude that it was an invalid named character reference. This commit modifies the approach taken for named character reference matching by using a trie-like structure (specifically, a deterministic acyclic finite state automaton or DAFSA), which allows for efficiently matching one-character-at-a-time and therefore it is able to pick up matching where it left off after each code point is consumed. Note: Because it's possible for a partial match to not actually develop into a full match (e.g. `&notindo` which could lead to `&notindot;`), some backtracking is performed after-the-fact in order to only consume the code points within the longest match found (e.g. `&notindo` would backtrack back to `&not`). With this new approach, `document.write` being called one-byte-at-a-time is handled correctly, which allows for passing more WPT tests, with the most directly relevant tests being `/html/syntax/parsing/html5lib_entities01.html` and `/html/syntax/parsing/html5lib_entities02.html` when run with `?run_type=write_single`. Additionally, the implementation now better conforms to the language of the spec (and resolves a FIXME) because exactly the matched characters are consumed and nothing more, so SWITCH_TO is able to be used as the spec says instead of RECONSUME_IN. The new approach is also an optimization: - Instead of a linear search using `starts_with`, the usage of a DAFSA means that it is always aware of which characters can lead to a match at any given point, and will bail out whenever a match is no longer possible. - The DAFSA is able to take advantage of the note in the section `13.5 Named character references` that says "This list is static and will not be expanded or changed in the future." and tailor its Node struct accordingly to tightly pack each node's data into 32-bits. Together with the inherent DAFSA property of redundant node deduplication, the amount of data stored for named character reference matching is minimized. In my testing: - A benchmark tokenizing an arbitrary set of HTML test files was about 1.23x faster (2070ms to 1682ms). - A benchmark tokenizing a file with tens of thousands of named character references mixed in with truncated named character references and arbitrary ASCII characters/ampersands runs about 8x faster (758ms to 93ms). - The size of `liblagom-web.so` was reduced by 94.96KiB. Some technical details: A DAFSA (deterministic acyclic finite state automaton) is essentially a trie flattened into an array, but it also uses techniques to minimize redundant nodes. This provides fast lookups while minimizing the required data size, but normally does not allow for associating data related to each word. However, by adding a count of the number of possible words from each node, it becomes possible to also use it to achieve minimal perfect hashing for the set of words (which allows going from word -> unique index as well as unique index -> word). This allows us to store a second array of data so that the DAFSA can be used as a lookup for e.g. the associated code points. For the Swift implementation, the new NamedCharacterReferenceMatcher was used to satisfy the previous API and the tokenizer was left alone otherwise. In the future, the Swift implementation should be updated to use the same implementation for its NamedCharacterReference state as the updated C++ implementation.
2024-12-22 07:09:31 -08:00
m_temporary_buffer.clear_with_capacity();
m_temporary_buffer.append(mapped_codepoints.value().first);
auto second_codepoint = named_character_reference_second_codepoint_value(mapped_codepoints.value().second);
if (second_codepoint.has_value()) {
m_temporary_buffer.append(second_codepoint.value());
}
LibWeb: Support named character references (e.g "&amp;")
2020-05-28 11:44:19 +02:00
Revert "Unicode: s/codepoint/code_point/g" This reverts commit ea9ac3155d1774f13ac4e9a96605c0e85a8f299e. It replaced "codepoint" with "code_points", not "code_point".
2020-08-05 16:28:19 -04:00
FLUSH_CODEPOINTS_CONSUMED_AS_A_CHARACTER_REFERENCE;
LibWeb: Support named character references (e.g "&amp;")
2020-05-28 11:44:19 +02:00
SWITCH_TO_RETURN_STATE;
} else {
Revert "Unicode: s/codepoint/code_point/g" This reverts commit ea9ac3155d1774f13ac4e9a96605c0e85a8f299e. It replaced "codepoint" with "code_points", not "code_point".
2020-08-05 16:28:19 -04:00
FLUSH_CODEPOINTS_CONSUMED_AS_A_CHARACTER_REFERENCE;
LibWeb: Make named character references more spec-compliant & efficient There are two changes happening here: a correctness fix, and an optimization. In theory they are unrelated, but the optimization actually paves the way for the correctness fix. Before this commit, the HTML tokenizer would attempt to look for named character references by checking from after the `&` until the end of m_decoded_input, which meant that it was unable to recognize things like named character references that are inserted via `document.write` one byte at a time. For example, if `&notin;` was written one-byte-at-a-time with `document.write`, then the tokenizer would only check against `n` since that's all that would exist at the time of the check and therefore erroneously conclude that it was an invalid named character reference. This commit modifies the approach taken for named character reference matching by using a trie-like structure (specifically, a deterministic acyclic finite state automaton or DAFSA), which allows for efficiently matching one-character-at-a-time and therefore it is able to pick up matching where it left off after each code point is consumed. Note: Because it's possible for a partial match to not actually develop into a full match (e.g. `&notindo` which could lead to `&notindot;`), some backtracking is performed after-the-fact in order to only consume the code points within the longest match found (e.g. `&notindo` would backtrack back to `&not`). With this new approach, `document.write` being called one-byte-at-a-time is handled correctly, which allows for passing more WPT tests, with the most directly relevant tests being `/html/syntax/parsing/html5lib_entities01.html` and `/html/syntax/parsing/html5lib_entities02.html` when run with `?run_type=write_single`. Additionally, the implementation now better conforms to the language of the spec (and resolves a FIXME) because exactly the matched characters are consumed and nothing more, so SWITCH_TO is able to be used as the spec says instead of RECONSUME_IN. The new approach is also an optimization: - Instead of a linear search using `starts_with`, the usage of a DAFSA means that it is always aware of which characters can lead to a match at any given point, and will bail out whenever a match is no longer possible. - The DAFSA is able to take advantage of the note in the section `13.5 Named character references` that says "This list is static and will not be expanded or changed in the future." and tailor its Node struct accordingly to tightly pack each node's data into 32-bits. Together with the inherent DAFSA property of redundant node deduplication, the amount of data stored for named character reference matching is minimized. In my testing: - A benchmark tokenizing an arbitrary set of HTML test files was about 1.23x faster (2070ms to 1682ms). - A benchmark tokenizing a file with tens of thousands of named character references mixed in with truncated named character references and arbitrary ASCII characters/ampersands runs about 8x faster (758ms to 93ms). - The size of `liblagom-web.so` was reduced by 94.96KiB. Some technical details: A DAFSA (deterministic acyclic finite state automaton) is essentially a trie flattened into an array, but it also uses techniques to minimize redundant nodes. This provides fast lookups while minimizing the required data size, but normally does not allow for associating data related to each word. However, by adding a count of the number of possible words from each node, it becomes possible to also use it to achieve minimal perfect hashing for the set of words (which allows going from word -> unique index as well as unique index -> word). This allows us to store a second array of data so that the DAFSA can be used as a lookup for e.g. the associated code points. For the Swift implementation, the new NamedCharacterReferenceMatcher was used to satisfy the previous API and the tokenizer was left alone otherwise. In the future, the Swift implementation should be updated to use the same implementation for its NamedCharacterReference state as the updated C++ implementation.
2024-12-22 07:09:31 -08:00
SWITCH_TO_WITH_UNCLEAN_BUILDER(AmbiguousAmpersand);
LibWeb: Support named character references (e.g "&amp;")
2020-05-28 11:44:19 +02:00
}
}
END_STATE
LibWeb: Add spec links to each HTML tokenizer state section I didn't add full spec comments this time, but this is better than nothing :^)
2022-02-15 18:12:15 +00:00
// 13.2.5.74 Ambiguous ampersand state, https://html.spec.whatwg.org/multipage/parsing.html#ambiguous-ampersand-state
LibWeb: Support named character references (e.g "&amp;")
2020-05-28 11:44:19 +02:00
BEGIN_STATE(AmbiguousAmpersand)
{
ON_ASCII_ALPHANUMERIC
{
if (consumed_as_part_of_an_attribute()) {
LibWeb: Remove StringBuilders from HTMLToken::AttributeBuilder
2021-05-23 08:50:48 +02:00
m_current_builder.append_code_point(current_input_character.value());
LibWeb: Support named character references (e.g "&amp;")
2020-05-28 11:44:19 +02:00
continue;
} else {
EMIT_CURRENT_CHARACTER;
}
}
LibWeb: Implement partial support for numeric character references
2020-05-27 16:16:23 +02:00
ON(';')
{
LibWeb: Utilize SourceLocation for HTMLTokenizer logging
2021-04-24 20:22:30 -07:00
log_parse_error();
LibWeb: Implement almost all missing tokenizer cases
2020-06-26 22:41:35 +01:00
RECONSUME_IN_RETURN_STATE;
LibWeb: Implement partial support for numeric character references
2020-05-27 16:16:23 +02:00
}
ANYTHING_ELSE
{
LibWeb: Support named character references (e.g "&amp;")
2020-05-28 11:44:19 +02:00
RECONSUME_IN_RETURN_STATE;
LibWeb: Implement partial support for numeric character references
2020-05-27 16:16:23 +02:00
}
}
END_STATE
LibWeb: Add spec links to each HTML tokenizer state section I didn't add full spec comments this time, but this is better than nothing :^)
2022-02-15 18:12:15 +00:00
// 13.2.5.75 Numeric character reference state, https://html.spec.whatwg.org/multipage/parsing.html#numeric-character-reference-state
LibWeb: Implement partial support for numeric character references
2020-05-27 16:16:23 +02:00
BEGIN_STATE(NumericCharacterReference)
{
m_character_reference_code = 0;
ON('X')
{
m_temporary_buffer.append(current_input_character.value());
LibWeb: Avoid assertion failure on parsing numeric character references
2021-07-28 00:37:26 +01:00
SWITCH_TO_WITH_UNCLEAN_BUILDER(HexadecimalCharacterReferenceStart);
LibWeb: Implement partial support for numeric character references
2020-05-27 16:16:23 +02:00
}
ON('x')
{
m_temporary_buffer.append(current_input_character.value());
LibWeb: Avoid assertion failure on parsing numeric character references
2021-07-28 00:37:26 +01:00
SWITCH_TO_WITH_UNCLEAN_BUILDER(HexadecimalCharacterReferenceStart);
LibWeb: Implement partial support for numeric character references
2020-05-27 16:16:23 +02:00
}
ANYTHING_ELSE
{
RECONSUME_IN(DecimalCharacterReferenceStart);
}
}
END_STATE
LibWeb: Add spec links to each HTML tokenizer state section I didn't add full spec comments this time, but this is better than nothing :^)
2022-02-15 18:12:15 +00:00
// 13.2.5.76 Hexadecimal character reference start state, https://html.spec.whatwg.org/multipage/parsing.html#hexadecimal-character-reference-start-state
LibWeb: Implement partial support for numeric character references
2020-05-27 16:16:23 +02:00
BEGIN_STATE(HexadecimalCharacterReferenceStart)
{
ON_ASCII_HEX_DIGIT
{
RECONSUME_IN(HexadecimalCharacterReference);
}
ANYTHING_ELSE
{
LibWeb: Utilize SourceLocation for HTMLTokenizer logging
2021-04-24 20:22:30 -07:00
log_parse_error();
Revert "Unicode: s/codepoint/code_point/g" This reverts commit ea9ac3155d1774f13ac4e9a96605c0e85a8f299e. It replaced "codepoint" with "code_points", not "code_point".
2020-08-05 16:28:19 -04:00
FLUSH_CODEPOINTS_CONSUMED_AS_A_CHARACTER_REFERENCE;
LibWeb: Implement almost all missing tokenizer cases
2020-06-26 22:41:35 +01:00
RECONSUME_IN_RETURN_STATE;
LibWeb: Implement partial support for numeric character references
2020-05-27 16:16:23 +02:00
}
}
END_STATE
LibWeb: Add spec links to each HTML tokenizer state section I didn't add full spec comments this time, but this is better than nothing :^)
2022-02-15 18:12:15 +00:00
// 13.2.5.77 Decimal character reference start state, https://html.spec.whatwg.org/multipage/parsing.html#decimal-character-reference-start-state
LibWeb: Implement partial support for numeric character references
2020-05-27 16:16:23 +02:00
BEGIN_STATE(DecimalCharacterReferenceStart)
{
ON_ASCII_DIGIT
{
RECONSUME_IN(DecimalCharacterReference);
}
ANYTHING_ELSE
{
LibWeb: Utilize SourceLocation for HTMLTokenizer logging
2021-04-24 20:22:30 -07:00
log_parse_error();
Revert "Unicode: s/codepoint/code_point/g" This reverts commit ea9ac3155d1774f13ac4e9a96605c0e85a8f299e. It replaced "codepoint" with "code_points", not "code_point".
2020-08-05 16:28:19 -04:00
FLUSH_CODEPOINTS_CONSUMED_AS_A_CHARACTER_REFERENCE;
LibWeb: Implement almost all missing tokenizer cases
2020-06-26 22:41:35 +01:00
RECONSUME_IN_RETURN_STATE;
LibWeb: Implement partial support for numeric character references
2020-05-27 16:16:23 +02:00
}
}
END_STATE
LibWeb: Add spec links to each HTML tokenizer state section I didn't add full spec comments this time, but this is better than nothing :^)
2022-02-15 18:12:15 +00:00
// 13.2.5.78 Hexadecimal character reference state, https://html.spec.whatwg.org/multipage/parsing.html#decimal-character-reference-start-state
LibWeb: Implement partial support for numeric character references
2020-05-27 16:16:23 +02:00
BEGIN_STATE(HexadecimalCharacterReference)
{
ON_ASCII_DIGIT
{
m_character_reference_code *= 16;
m_character_reference_code += current_input_character.value() - 0x30;
continue;
}
LibWeb: Fix hex character references accepting all alphabetic ASCII Instead of just A-F/a-f, any char A-Z/a-z was being accepted as a valid hexadecimal digit.
2024-12-20 06:17:13 -08:00
ON_ASCII_HEX_DIGIT
LibWeb: Implement partial support for numeric character references
2020-05-27 16:16:23 +02:00
{
m_character_reference_code *= 16;
LibWeb: Fix hex character references accepting all alphabetic ASCII Instead of just A-F/a-f, any char A-Z/a-z was being accepted as a valid hexadecimal digit.
2024-12-20 06:17:13 -08:00
auto hex_digit_min_ascii_value = is_ascii_upper_alpha(current_input_character.value()) ? 0x37 : 0x57;
m_character_reference_code += current_input_character.value() - hex_digit_min_ascii_value;
LibWeb: Implement partial support for numeric character references
2020-05-27 16:16:23 +02:00
continue;
}
ON(';')
{
LibWeb: Avoid assertion failure on parsing numeric character references
2021-07-28 00:37:26 +01:00
SWITCH_TO_WITH_UNCLEAN_BUILDER(NumericCharacterReferenceEnd);
LibWeb: Implement partial support for numeric character references
2020-05-27 16:16:23 +02:00
}
ANYTHING_ELSE
{
LibWeb: Utilize SourceLocation for HTMLTokenizer logging
2021-04-24 20:22:30 -07:00
log_parse_error();
LibWeb: Implement almost all missing tokenizer cases
2020-06-26 22:41:35 +01:00
RECONSUME_IN(NumericCharacterReferenceEnd);
LibWeb: Implement partial support for numeric character references
2020-05-27 16:16:23 +02:00
}
}
END_STATE
LibWeb: Add spec links to each HTML tokenizer state section I didn't add full spec comments this time, but this is better than nothing :^)
2022-02-15 18:12:15 +00:00
// 13.2.5.79 Decimal character reference state, https://html.spec.whatwg.org/multipage/parsing.html#decimal-character-reference-state
LibWeb: Implement partial support for numeric character references
2020-05-27 16:16:23 +02:00
BEGIN_STATE(DecimalCharacterReference)
{
ON_ASCII_DIGIT
{
m_character_reference_code *= 10;
m_character_reference_code += current_input_character.value() - 0x30;
continue;
}
ON(';')
{
LibWeb: Avoid assertion failure on parsing numeric character references
2021-07-28 00:37:26 +01:00
SWITCH_TO_WITH_UNCLEAN_BUILDER(NumericCharacterReferenceEnd);
LibWeb: Implement partial support for numeric character references
2020-05-27 16:16:23 +02:00
}
ANYTHING_ELSE
{
LibWeb: Utilize SourceLocation for HTMLTokenizer logging
2021-04-24 20:22:30 -07:00
log_parse_error();
LibWeb: Implement almost all missing tokenizer cases
2020-06-26 22:41:35 +01:00
RECONSUME_IN(NumericCharacterReferenceEnd);
LibWeb: Implement partial support for numeric character references
2020-05-27 16:16:23 +02:00
}
}
END_STATE
LibWeb: Add spec links to each HTML tokenizer state section I didn't add full spec comments this time, but this is better than nothing :^)
2022-02-15 18:12:15 +00:00
// 13.2.5.80 Numeric character reference end state, https://html.spec.whatwg.org/multipage/parsing.html#numeric-character-reference-end-state
LibWeb: Implement partial support for numeric character references
2020-05-27 16:16:23 +02:00
BEGIN_STATE(NumericCharacterReferenceEnd)
{
LibWeb: Fix incorrectly consumed characters after reference tokens The NumericCharacterReferenceEnd tokenizer state should not advance the input stream.
2020-06-04 16:48:15 +02:00
DONT_CONSUME_NEXT_INPUT_CHARACTER;
LibWeb: Implement partial support for numeric character references
2020-05-27 16:16:23 +02:00
if (m_character_reference_code == 0) {
LibWeb: Utilize SourceLocation for HTMLTokenizer logging
2021-04-24 20:22:30 -07:00
log_parse_error();
LibWeb: Implement almost all missing tokenizer cases
2020-06-26 22:41:35 +01:00
m_character_reference_code = 0xFFFD;
LibWeb: Implement partial support for numeric character references
2020-05-27 16:16:23 +02:00
}
if (m_character_reference_code > 0x10ffff) {
LibWeb: Utilize SourceLocation for HTMLTokenizer logging
2021-04-24 20:22:30 -07:00
log_parse_error();
LibWeb: Implement almost all missing tokenizer cases
2020-06-26 22:41:35 +01:00
m_character_reference_code = 0xFFFD;
LibWeb: Implement partial support for numeric character references
2020-05-27 16:16:23 +02:00
}
Everywhere: Replace ctype.h to avoid narrowing conversions This replaces ctype.h with CharacterType.h everywhere I could find issues with narrowing conversions. While using it will probably make sense almost everywhere in the future, the most critical places should have been addressed.
2021-06-01 21:18:08 +02:00
if (is_unicode_surrogate(m_character_reference_code)) {
LibWeb: Utilize SourceLocation for HTMLTokenizer logging
2021-04-24 20:22:30 -07:00
log_parse_error();
LibWeb: Implement almost all missing tokenizer cases
2020-06-26 22:41:35 +01:00
m_character_reference_code = 0xFFFD;
LibWeb: Implement partial support for numeric character references
2020-05-27 16:16:23 +02:00
}
Everywhere: Replace ctype.h to avoid narrowing conversions This replaces ctype.h with CharacterType.h everywhere I could find issues with narrowing conversions. While using it will probably make sense almost everywhere in the future, the most critical places should have been addressed.
2021-06-01 21:18:08 +02:00
if (is_unicode_noncharacter(m_character_reference_code)) {
LibWeb: Utilize SourceLocation for HTMLTokenizer logging
2021-04-24 20:22:30 -07:00
log_parse_error();
LibWeb: Implement partial support for numeric character references
2020-05-27 16:16:23 +02:00
}
Everywhere: Replace ctype.h to avoid narrowing conversions This replaces ctype.h with CharacterType.h everywhere I could find issues with narrowing conversions. While using it will probably make sense almost everywhere in the future, the most critical places should have been addressed.
2021-06-01 21:18:08 +02:00
if (m_character_reference_code == 0xd || (is_unicode_control(m_character_reference_code) && !is_ascii_space(m_character_reference_code))) {
LibWeb: Utilize SourceLocation for HTMLTokenizer logging
2021-04-24 20:22:30 -07:00
log_parse_error();
LibWeb: Implement partial support for numeric character references
2020-05-27 16:16:23 +02:00
constexpr struct {
u32 number;
Unicode: Try s/codepoint/code_point/g again This time, without trailing 's'. Ran: git grep -l 'codepoint' | xargs sed -ie 's/codepoint/code_point/g
2020-08-05 16:31:20 -04:00
u32 code_point;
LibWeb: Implement partial support for numeric character references
2020-05-27 16:16:23 +02:00
} conversion_table[] = {
{ 0x80, 0x20AC },
{ 0x82, 0x201A },
{ 0x83, 0x0192 },
{ 0x84, 0x201E },
{ 0x85, 0x2026 },
{ 0x86, 0x2020 },
{ 0x87, 0x2021 },
{ 0x88, 0x02C6 },
{ 0x89, 0x2030 },
{ 0x8A, 0x0160 },
{ 0x8B, 0x2039 },
{ 0x8C, 0x0152 },
{ 0x8E, 0x017D },
{ 0x91, 0x2018 },
{ 0x92, 0x2019 },
{ 0x93, 0x201C },
{ 0x94, 0x201D },
{ 0x95, 0x2022 },
{ 0x96, 0x2013 },
{ 0x97, 0x2014 },
{ 0x98, 0x02DC },
{ 0x99, 0x2122 },
{ 0x9A, 0x0161 },
{ 0x9B, 0x203A },
{ 0x9C, 0x0153 },
{ 0x9E, 0x017E },
{ 0x9F, 0x0178 },
};
for (auto& entry : conversion_table) {
if (m_character_reference_code == entry.number) {
Unicode: Try s/codepoint/code_point/g again This time, without trailing 's'. Ran: git grep -l 'codepoint' | xargs sed -ie 's/codepoint/code_point/g
2020-08-05 16:31:20 -04:00
m_character_reference_code = entry.code_point;
LibWeb: Implement partial support for numeric character references
2020-05-27 16:16:23 +02:00
break;
}
}
}
m_temporary_buffer.clear();
m_temporary_buffer.append(m_character_reference_code);
Revert "Unicode: s/codepoint/code_point/g" This reverts commit ea9ac3155d1774f13ac4e9a96605c0e85a8f299e. It replaced "codepoint" with "code_points", not "code_point".
2020-08-05 16:28:19 -04:00
FLUSH_CODEPOINTS_CONSUMED_AS_A_CHARACTER_REFERENCE;
LibWeb: Implement partial support for numeric character references
2020-05-27 16:16:23 +02:00
SWITCH_TO_RETURN_STATE;
LibWeb: Begin work on a spec-compliant HTML parser In order to actually view the web as it is, we're gonna need a proper HTML parser. So let's build one! This patch introduces the Web::HTMLTokenizer class, which currently operates on a StringView input stream where it fetches (ASCII only atm) codepoints and tokenizes acccording to the HTML spec tokenization algo. The tokenizer state machine looks a bit weird but is written in a way that tries to mimic the spec as closely as possible, in order to make development easier and bugs less likely. This initial version is far from finished, but it can parse a trivial document with a DOCTYPE and open/close tags. :^)
2020-05-22 21:46:13 +02:00
}
END_STATE
LibWeb: Add spec links to each HTML tokenizer state section I didn't add full spec comments this time, but this is better than nothing :^)
2022-02-15 18:12:15 +00:00
// 13.2.5.2 RCDATA state, https://html.spec.whatwg.org/multipage/parsing.html#rcdata-state
LibWeb: More work on the HTML parser and tokenizer The parser can now switch the state of the tokenizer! Very webby. :^)
2020-05-24 20:24:43 +02:00
BEGIN_STATE(RCDATA)
{
ON('&')
{
m_return_state = State::RCDATA;
SWITCH_TO(CharacterReference);
}
ON('<')
{
SWITCH_TO(RCDATALessThanSign);
}
ON(0)
{
LibWeb: Utilize SourceLocation for HTMLTokenizer logging
2021-04-24 20:22:30 -07:00
log_parse_error();
LibWeb: Make the new HTML parser parse input as UTF-8 We already convert the input to UTF-8 before starting the tokenizer, so all this patch had to do was switch the tokenizer to use an Utf8View for its input (and to emit 32-bit codepoints.)
2020-06-04 21:06:54 +02:00
EMIT_CHARACTER(0xFFFD);
LibWeb: More work on the HTML parser and tokenizer The parser can now switch the state of the tokenizer! Very webby. :^)
2020-05-24 20:24:43 +02:00
}
ON_EOF
{
EMIT_EOF;
}
ANYTHING_ELSE
{
EMIT_CURRENT_CHARACTER;
}
}
END_STATE
LibWeb: Add spec links to each HTML tokenizer state section I didn't add full spec comments this time, but this is better than nothing :^)
2022-02-15 18:12:15 +00:00
// 13.2.5.9 RCDATA less-than sign state, https://html.spec.whatwg.org/multipage/parsing.html#rcdata-less-than-sign-state
LibWeb: More work on the HTML parser and tokenizer The parser can now switch the state of the tokenizer! Very webby. :^)
2020-05-24 20:24:43 +02:00
BEGIN_STATE(RCDATALessThanSign)
{
ON('/')
{
m_temporary_buffer.clear();
SWITCH_TO(RCDATAEndTagOpen);
}
ANYTHING_ELSE
{
LibWeb: Fix missing tokenizer state change in RCDATALessThanSign We can't RECONSUME_IN after we've used EMIT_CHARACTER since we'll have returned from the function.
2020-06-05 12:02:30 +02:00
EMIT_CHARACTER_AND_RECONSUME_IN('<', RCDATA);
LibWeb: More work on the HTML parser and tokenizer The parser can now switch the state of the tokenizer! Very webby. :^)
2020-05-24 20:24:43 +02:00
}
}
END_STATE
LibWeb: Add spec links to each HTML tokenizer state section I didn't add full spec comments this time, but this is better than nothing :^)
2022-02-15 18:12:15 +00:00
// 13.2.5.10 RCDATA end tag open state, https://html.spec.whatwg.org/multipage/parsing.html#rcdata-end-tag-open-state
LibWeb: More work on the HTML parser and tokenizer The parser can now switch the state of the tokenizer! Very webby. :^)
2020-05-24 20:24:43 +02:00
BEGIN_STATE(RCDATAEndTagOpen)
{
ON_ASCII_ALPHA
{
create_new_token(HTMLToken::Type::EndTag);
RECONSUME_IN(RCDATAEndTagName);
}
ANYTHING_ELSE
{
LibWeb: Parse "textarea" tags during the "in body" insertion mode Had to handle some more cases in the tokenizer to support this.
2020-05-30 18:40:23 +02:00
m_queued_tokens.enqueue(HTMLToken::make_character('<'));
m_queued_tokens.enqueue(HTMLToken::make_character('/'));
RECONSUME_IN(RCDATA);
LibWeb: More work on the HTML parser and tokenizer The parser can now switch the state of the tokenizer! Very webby. :^)
2020-05-24 20:24:43 +02:00
}
}
END_STATE
LibWeb: Add spec links to each HTML tokenizer state section I didn't add full spec comments this time, but this is better than nothing :^)
2022-02-15 18:12:15 +00:00
// 13.2.5.11 RCDATA end tag name state, https://html.spec.whatwg.org/multipage/parsing.html#rcdata-end-tag-name-state
LibWeb: More work on the HTML parser and tokenizer The parser can now switch the state of the tokenizer! Very webby. :^)
2020-05-24 20:24:43 +02:00
BEGIN_STATE(RCDATAEndTagName)
{
ON_WHITESPACE
{
LibWeb: User getter+setter for HTMLToken tag name and self-closing flag
2021-07-14 23:37:48 +02:00
m_current_token.set_tag_name(consume_current_builder());
LibWeb: Implement more of the tokenizer states Slowly adding more unimplemented options for tokenizer states.
2020-05-28 00:28:32 +02:00
if (!current_end_tag_token_is_appropriate()) {
LibWeb: Parse "textarea" tags during the "in body" insertion mode Had to handle some more cases in the tokenizer to support this.
2020-05-30 18:40:23 +02:00
m_queued_tokens.enqueue(HTMLToken::make_character('<'));
m_queued_tokens.enqueue(HTMLToken::make_character('/'));
Unicode: Try s/codepoint/code_point/g again This time, without trailing 's'. Ran: git grep -l 'codepoint' | xargs sed -ie 's/codepoint/code_point/g
2020-08-05 16:31:20 -04:00
for (auto code_point : m_temporary_buffer)
m_queued_tokens.enqueue(HTMLToken::make_character(code_point));
LibWeb: Parse "textarea" tags during the "in body" insertion mode Had to handle some more cases in the tokenizer to support this.
2020-05-30 18:40:23 +02:00
RECONSUME_IN(RCDATA);
LibWeb: Implement more of the tokenizer states Slowly adding more unimplemented options for tokenizer states.
2020-05-28 00:28:32 +02:00
}
SWITCH_TO(BeforeAttributeName);
LibWeb: More work on the HTML parser and tokenizer The parser can now switch the state of the tokenizer! Very webby. :^)
2020-05-24 20:24:43 +02:00
}
ON('/')
{
LibWeb: User getter+setter for HTMLToken tag name and self-closing flag
2021-07-14 23:37:48 +02:00
m_current_token.set_tag_name(consume_current_builder());
LibWeb: Implement more of the tokenizer states Slowly adding more unimplemented options for tokenizer states.
2020-05-28 00:28:32 +02:00
if (!current_end_tag_token_is_appropriate()) {
LibWeb: Parse "textarea" tags during the "in body" insertion mode Had to handle some more cases in the tokenizer to support this.
2020-05-30 18:40:23 +02:00
m_queued_tokens.enqueue(HTMLToken::make_character('<'));
m_queued_tokens.enqueue(HTMLToken::make_character('/'));
Unicode: Try s/codepoint/code_point/g again This time, without trailing 's'. Ran: git grep -l 'codepoint' | xargs sed -ie 's/codepoint/code_point/g
2020-08-05 16:31:20 -04:00
for (auto code_point : m_temporary_buffer)
m_queued_tokens.enqueue(HTMLToken::make_character(code_point));
LibWeb: Parse "textarea" tags during the "in body" insertion mode Had to handle some more cases in the tokenizer to support this.
2020-05-30 18:40:23 +02:00
RECONSUME_IN(RCDATA);
LibWeb: Implement more of the tokenizer states Slowly adding more unimplemented options for tokenizer states.
2020-05-28 00:28:32 +02:00
}
SWITCH_TO(SelfClosingStartTag);
LibWeb: More work on the HTML parser and tokenizer The parser can now switch the state of the tokenizer! Very webby. :^)
2020-05-24 20:24:43 +02:00
}
ON('>')
{
LibWeb: User getter+setter for HTMLToken tag name and self-closing flag
2021-07-14 23:37:48 +02:00
m_current_token.set_tag_name(consume_current_builder());
LibWeb: More work on the HTML parser and tokenizer The parser can now switch the state of the tokenizer! Very webby. :^)
2020-05-24 20:24:43 +02:00
if (!current_end_tag_token_is_appropriate()) {
LibWeb: Parse "textarea" tags during the "in body" insertion mode Had to handle some more cases in the tokenizer to support this.
2020-05-30 18:40:23 +02:00
m_queued_tokens.enqueue(HTMLToken::make_character('<'));
m_queued_tokens.enqueue(HTMLToken::make_character('/'));
Unicode: Try s/codepoint/code_point/g again This time, without trailing 's'. Ran: git grep -l 'codepoint' | xargs sed -ie 's/codepoint/code_point/g
2020-08-05 16:31:20 -04:00
for (auto code_point : m_temporary_buffer)
m_queued_tokens.enqueue(HTMLToken::make_character(code_point));
LibWeb: Parse "textarea" tags during the "in body" insertion mode Had to handle some more cases in the tokenizer to support this.
2020-05-30 18:40:23 +02:00
RECONSUME_IN(RCDATA);
LibWeb: More work on the HTML parser and tokenizer The parser can now switch the state of the tokenizer! Very webby. :^)
2020-05-24 20:24:43 +02:00
}
SWITCH_TO_AND_EMIT_CURRENT_TOKEN(Data);
}
ON_ASCII_UPPER_ALPHA
{
LibWeb: Remove StringBuilder from HTMLToken::m_tag
2021-05-23 09:16:07 +02:00
m_current_builder.append_code_point(to_ascii_lowercase(current_input_character.value()));
LibWeb: More work on the HTML parser and tokenizer The parser can now switch the state of the tokenizer! Very webby. :^)
2020-05-24 20:24:43 +02:00
m_temporary_buffer.append(current_input_character.value());
continue;
}
ON_ASCII_LOWER_ALPHA
{
LibWeb: Remove StringBuilder from HTMLToken::m_tag
2021-05-23 09:16:07 +02:00
m_current_builder.append_code_point(current_input_character.value());
LibWeb: More work on the HTML parser and tokenizer The parser can now switch the state of the tokenizer! Very webby. :^)
2020-05-24 20:24:43 +02:00
m_temporary_buffer.append(current_input_character.value());
LibWeb: Parse enough to handle a <style> inside a <head> :^)
2020-05-24 20:36:43 +02:00
continue;
}
ANYTHING_ELSE
{
LibWeb: Parse "textarea" tags during the "in body" insertion mode Had to handle some more cases in the tokenizer to support this.
2020-05-30 18:40:23 +02:00
m_queued_tokens.enqueue(HTMLToken::make_character('<'));
m_queued_tokens.enqueue(HTMLToken::make_character('/'));
LibWeb: Fix assertion failures in HTMLTokenizer The *TagName states are all very similar, so it seems to be correct to apply the fix from #8761 to all of those states. This fixes #8788.
2021-07-16 00:36:10 +02:00
// NOTE: The spec doesn't mention this, but it seems that m_current_token (an end tag) is just dropped in this case.
m_current_builder.clear();
Unicode: Try s/codepoint/code_point/g again This time, without trailing 's'. Ran: git grep -l 'codepoint' | xargs sed -ie 's/codepoint/code_point/g
2020-08-05 16:31:20 -04:00
for (auto code_point : m_temporary_buffer)
m_queued_tokens.enqueue(HTMLToken::make_character(code_point));
LibWeb: Parse "textarea" tags during the "in body" insertion mode Had to handle some more cases in the tokenizer to support this.
2020-05-30 18:40:23 +02:00
RECONSUME_IN(RCDATA);
LibWeb: Parse enough to handle a <style> inside a <head> :^)
2020-05-24 20:36:43 +02:00
}
}
END_STATE
LibWeb: Add spec links to each HTML tokenizer state section I didn't add full spec comments this time, but this is better than nothing :^)
2022-02-15 18:12:15 +00:00
// 13.2.5.3 RAWTEXT state, https://html.spec.whatwg.org/multipage/parsing.html#rawtext-state
LibWeb: Parse enough to handle a <style> inside a <head> :^)
2020-05-24 20:36:43 +02:00
BEGIN_STATE(RAWTEXT)
{
ON('<')
{
SWITCH_TO(RAWTEXTLessThanSign);
}
ON(0)
{
LibWeb: Utilize SourceLocation for HTMLTokenizer logging
2021-04-24 20:22:30 -07:00
log_parse_error();
LibWeb: Make the new HTML parser parse input as UTF-8 We already convert the input to UTF-8 before starting the tokenizer, so all this patch had to do was switch the tokenizer to use an Utf8View for its input (and to emit 32-bit codepoints.)
2020-06-04 21:06:54 +02:00
EMIT_CHARACTER(0xFFFD);
LibWeb: Parse enough to handle a <style> inside a <head> :^)
2020-05-24 20:36:43 +02:00
}
ON_EOF
{
EMIT_EOF;
}
ANYTHING_ELSE
{
EMIT_CURRENT_CHARACTER;
}
}
END_STATE
LibWeb: Add spec links to each HTML tokenizer state section I didn't add full spec comments this time, but this is better than nothing :^)
2022-02-15 18:12:15 +00:00
// 13.2.5.12 RAWTEXT less-than sign state, https://html.spec.whatwg.org/multipage/parsing.html#rawtext-less-than-sign-state
LibWeb: Parse enough to handle a <style> inside a <head> :^)
2020-05-24 20:36:43 +02:00
BEGIN_STATE(RAWTEXTLessThanSign)
{
ON('/')
{
m_temporary_buffer.clear();
SWITCH_TO(RAWTEXTEndTagOpen);
}
ANYTHING_ELSE
{
LibWeb: Fully implement all script tokenizer states Also fixes RAWTEXTLessThanSign having a separate emit and reconsume.
2020-06-06 07:06:46 +01:00
EMIT_CHARACTER_AND_RECONSUME_IN('<', RAWTEXT);
LibWeb: Parse enough to handle a <style> inside a <head> :^)
2020-05-24 20:36:43 +02:00
}
}
END_STATE
LibWeb: Add spec links to each HTML tokenizer state section I didn't add full spec comments this time, but this is better than nothing :^)
2022-02-15 18:12:15 +00:00
// 13.2.5.13 RAWTEXT end tag open state, https://html.spec.whatwg.org/multipage/parsing.html#rawtext-end-tag-open-state
LibWeb: Parse enough to handle a <style> inside a <head> :^)
2020-05-24 20:36:43 +02:00
BEGIN_STATE(RAWTEXTEndTagOpen)
{
ON_ASCII_ALPHA
{
create_new_token(HTMLToken::Type::EndTag);
RECONSUME_IN(RAWTEXTEndTagName);
}
ANYTHING_ELSE
{
LibWeb: Implement some more RAWTEXT stuff in the tokenizer
2020-05-30 16:15:16 +02:00
m_queued_tokens.enqueue(HTMLToken::make_character('<'));
m_queued_tokens.enqueue(HTMLToken::make_character('/'));
RECONSUME_IN(RAWTEXT);
LibWeb: Parse enough to handle a <style> inside a <head> :^)
2020-05-24 20:36:43 +02:00
}
}
END_STATE
LibWeb: Add spec links to each HTML tokenizer state section I didn't add full spec comments this time, but this is better than nothing :^)
2022-02-15 18:12:15 +00:00
// 13.2.5.14 RAWTEXT end tag name state, https://html.spec.whatwg.org/multipage/parsing.html#rawtext-end-tag-name-state
LibWeb: Parse enough to handle a <style> inside a <head> :^)
2020-05-24 20:36:43 +02:00
BEGIN_STATE(RAWTEXTEndTagName)
{
ON_WHITESPACE
{
LibWeb: User getter+setter for HTMLToken tag name and self-closing flag
2021-07-14 23:37:48 +02:00
m_current_token.set_tag_name(consume_current_builder());
LibWeb: Implement more of the tokenizer states Slowly adding more unimplemented options for tokenizer states.
2020-05-28 00:28:32 +02:00
if (!current_end_tag_token_is_appropriate()) {
LibWeb: Implement some more RAWTEXT stuff in the tokenizer
2020-05-30 16:15:16 +02:00
m_queued_tokens.enqueue(HTMLToken::make_character('<'));
m_queued_tokens.enqueue(HTMLToken::make_character('/'));
Unicode: Try s/codepoint/code_point/g again This time, without trailing 's'. Ran: git grep -l 'codepoint' | xargs sed -ie 's/codepoint/code_point/g
2020-08-05 16:31:20 -04:00
for (auto code_point : m_temporary_buffer)
m_queued_tokens.enqueue(HTMLToken::make_character(code_point));
LibWeb: Implement some more RAWTEXT stuff in the tokenizer
2020-05-30 16:15:16 +02:00
RECONSUME_IN(RAWTEXT);
LibWeb: Implement more of the tokenizer states Slowly adding more unimplemented options for tokenizer states.
2020-05-28 00:28:32 +02:00
}
SWITCH_TO(BeforeAttributeName);
LibWeb: Parse enough to handle a <style> inside a <head> :^)
2020-05-24 20:36:43 +02:00
}
ON('/')
{
LibWeb: User getter+setter for HTMLToken tag name and self-closing flag
2021-07-14 23:37:48 +02:00
m_current_token.set_tag_name(consume_current_builder());
LibWeb: Implement more of the tokenizer states Slowly adding more unimplemented options for tokenizer states.
2020-05-28 00:28:32 +02:00
if (!current_end_tag_token_is_appropriate()) {
LibWeb: Implement some more RAWTEXT stuff in the tokenizer
2020-05-30 16:15:16 +02:00
m_queued_tokens.enqueue(HTMLToken::make_character('<'));
m_queued_tokens.enqueue(HTMLToken::make_character('/'));
Unicode: Try s/codepoint/code_point/g again This time, without trailing 's'. Ran: git grep -l 'codepoint' | xargs sed -ie 's/codepoint/code_point/g
2020-08-05 16:31:20 -04:00
for (auto code_point : m_temporary_buffer)
m_queued_tokens.enqueue(HTMLToken::make_character(code_point));
LibWeb: Implement some more RAWTEXT stuff in the tokenizer
2020-05-30 16:15:16 +02:00
RECONSUME_IN(RAWTEXT);
LibWeb: Implement more of the tokenizer states Slowly adding more unimplemented options for tokenizer states.
2020-05-28 00:28:32 +02:00
}
SWITCH_TO(SelfClosingStartTag);
LibWeb: Parse enough to handle a <style> inside a <head> :^)
2020-05-24 20:36:43 +02:00
}
ON('>')
{
LibWeb: User getter+setter for HTMLToken tag name and self-closing flag
2021-07-14 23:37:48 +02:00
m_current_token.set_tag_name(consume_current_builder());
LibWeb: Parse enough to handle a <style> inside a <head> :^)
2020-05-24 20:36:43 +02:00
if (!current_end_tag_token_is_appropriate()) {
LibWeb: Implement some more RAWTEXT stuff in the tokenizer
2020-05-30 16:15:16 +02:00
m_queued_tokens.enqueue(HTMLToken::make_character('<'));
m_queued_tokens.enqueue(HTMLToken::make_character('/'));
Unicode: Try s/codepoint/code_point/g again This time, without trailing 's'. Ran: git grep -l 'codepoint' | xargs sed -ie 's/codepoint/code_point/g
2020-08-05 16:31:20 -04:00
for (auto code_point : m_temporary_buffer)
m_queued_tokens.enqueue(HTMLToken::make_character(code_point));
LibWeb: Implement some more RAWTEXT stuff in the tokenizer
2020-05-30 16:15:16 +02:00
RECONSUME_IN(RAWTEXT);
LibWeb: Parse enough to handle a <style> inside a <head> :^)
2020-05-24 20:36:43 +02:00
}
SWITCH_TO_AND_EMIT_CURRENT_TOKEN(Data);
}
ON_ASCII_UPPER_ALPHA
{
LibWeb: Remove StringBuilder from HTMLToken::m_tag
2021-05-23 09:16:07 +02:00
m_current_builder.append_code_point(to_ascii_lowercase(current_input_character.value()));
LibWeb: Parse enough to handle a <style> inside a <head> :^)
2020-05-24 20:36:43 +02:00
m_temporary_buffer.append(current_input_character.value());
continue;
}
ON_ASCII_LOWER_ALPHA
{
LibWeb: Remove StringBuilder from HTMLToken::m_tag
2021-05-23 09:16:07 +02:00
m_current_builder.append(current_input_character.value());
LibWeb: Parse enough to handle a <style> inside a <head> :^)
2020-05-24 20:36:43 +02:00
m_temporary_buffer.append(current_input_character.value());
LibWeb: More work on the HTML parser and tokenizer The parser can now switch the state of the tokenizer! Very webby. :^)
2020-05-24 20:24:43 +02:00
continue;
}
ANYTHING_ELSE
{
LibWeb: Implement some more RAWTEXT stuff in the tokenizer
2020-05-30 16:15:16 +02:00
m_queued_tokens.enqueue(HTMLToken::make_character('<'));
m_queued_tokens.enqueue(HTMLToken::make_character('/'));
LibWeb: Fix assertion failures in HTMLTokenizer The *TagName states are all very similar, so it seems to be correct to apply the fix from #8761 to all of those states. This fixes #8788.
2021-07-16 00:36:10 +02:00
// NOTE: The spec doesn't mention this, but it seems that m_current_token (an end tag) is just dropped in this case.
m_current_builder.clear();
Unicode: Try s/codepoint/code_point/g again This time, without trailing 's'. Ran: git grep -l 'codepoint' | xargs sed -ie 's/codepoint/code_point/g
2020-08-05 16:31:20 -04:00
for (auto code_point : m_temporary_buffer)
m_queued_tokens.enqueue(HTMLToken::make_character(code_point));
LibWeb: Implement some more RAWTEXT stuff in the tokenizer
2020-05-30 16:15:16 +02:00
RECONSUME_IN(RAWTEXT);
LibWeb: More work on the HTML parser and tokenizer The parser can now switch the state of the tokenizer! Very webby. :^)
2020-05-24 20:24:43 +02:00
}
}
END_STATE
LibWeb: Add spec links to each HTML tokenizer state section I didn't add full spec comments this time, but this is better than nothing :^)
2022-02-15 18:12:15 +00:00
// 13.2.5.4 Script data state, https://html.spec.whatwg.org/multipage/parsing.html#script-data-state
LibWeb: A whole bunch of work towards spec-compliant <script> elements This is still very unfinished, but there's at least a skeleton of code.
2020-05-24 22:00:46 +02:00
BEGIN_STATE(ScriptData)
{
ON('<')
{
SWITCH_TO(ScriptDataLessThanSign);
}
ON(0)
{
LibWeb: Utilize SourceLocation for HTMLTokenizer logging
2021-04-24 20:22:30 -07:00
log_parse_error();
LibWeb: Make the new HTML parser parse input as UTF-8 We already convert the input to UTF-8 before starting the tokenizer, so all this patch had to do was switch the tokenizer to use an Utf8View for its input (and to emit 32-bit codepoints.)
2020-06-04 21:06:54 +02:00
EMIT_CHARACTER(0xFFFD);
LibWeb: Implement more of the tokenizer states Slowly adding more unimplemented options for tokenizer states.
2020-05-28 00:28:32 +02:00
}
ON_EOF
{
EMIT_EOF;
}
ANYTHING_ELSE
{
EMIT_CURRENT_CHARACTER;
}
}
END_STATE
LibWeb: Add spec links to each HTML tokenizer state section I didn't add full spec comments this time, but this is better than nothing :^)
2022-02-15 18:12:15 +00:00
// 13.2.5.5 PLAINTEXT state, https://html.spec.whatwg.org/multipage/parsing.html#plaintext-state
LibWeb: Implement more of the tokenizer states Slowly adding more unimplemented options for tokenizer states.
2020-05-28 00:28:32 +02:00
BEGIN_STATE(PLAINTEXT)
{
ON(0)
{
LibWeb: Utilize SourceLocation for HTMLTokenizer logging
2021-04-24 20:22:30 -07:00
log_parse_error();
LibWeb: Make the new HTML parser parse input as UTF-8 We already convert the input to UTF-8 before starting the tokenizer, so all this patch had to do was switch the tokenizer to use an Utf8View for its input (and to emit 32-bit codepoints.)
2020-06-04 21:06:54 +02:00
EMIT_CHARACTER(0xFFFD);
LibWeb: A whole bunch of work towards spec-compliant <script> elements This is still very unfinished, but there's at least a skeleton of code.
2020-05-24 22:00:46 +02:00
}
ON_EOF
{
EMIT_EOF;
}
ANYTHING_ELSE
{
EMIT_CURRENT_CHARACTER;
}
}
END_STATE
LibWeb: Add spec links to each HTML tokenizer state section I didn't add full spec comments this time, but this is better than nothing :^)
2022-02-15 18:12:15 +00:00
// 13.2.5.15 Script data less-than sign state, https://html.spec.whatwg.org/multipage/parsing.html#script-data-less-than-sign-state
LibWeb: A whole bunch of work towards spec-compliant <script> elements This is still very unfinished, but there's at least a skeleton of code.
2020-05-24 22:00:46 +02:00
BEGIN_STATE(ScriptDataLessThanSign)
{
ON('/')
{
m_temporary_buffer.clear();
SWITCH_TO(ScriptDataEndTagOpen);
}
ON('!')
{
LibWeb: Implement a bunch more script-related tokenization states
2020-05-28 18:44:17 +02:00
m_queued_tokens.enqueue(HTMLToken::make_character('<'));
m_queued_tokens.enqueue(HTMLToken::make_character('!'));
SWITCH_TO(ScriptDataEscapeStart);
LibWeb: A whole bunch of work towards spec-compliant <script> elements This is still very unfinished, but there's at least a skeleton of code.
2020-05-24 22:00:46 +02:00
}
ANYTHING_ELSE
{
LibWeb: Allow HTML tokenizer to emit more than one token Tokens are now put on a queue when emitted, and we always pop from that queue when returning from next_token().
2020-05-26 15:50:05 +02:00
EMIT_CHARACTER_AND_RECONSUME_IN('<', ScriptData);
LibWeb: A whole bunch of work towards spec-compliant <script> elements This is still very unfinished, but there's at least a skeleton of code.
2020-05-24 22:00:46 +02:00
}
}
END_STATE
LibWeb: Add spec links to each HTML tokenizer state section I didn't add full spec comments this time, but this is better than nothing :^)
2022-02-15 18:12:15 +00:00
// 13.2.5.18 Script data escape start state, https://html.spec.whatwg.org/multipage/parsing.html#script-data-escape-start-state
LibWeb: Implement a bunch more script-related tokenization states
2020-05-28 18:44:17 +02:00
BEGIN_STATE(ScriptDataEscapeStart)
{
ON('-')
{
LibWeb: Fully implement all script tokenizer states Also fixes RAWTEXTLessThanSign having a separate emit and reconsume.
2020-06-06 07:06:46 +01:00
SWITCH_TO_AND_EMIT_CHARACTER('-', ScriptDataEscapeStartDash);
LibWeb: Implement a bunch more script-related tokenization states
2020-05-28 18:44:17 +02:00
}
ANYTHING_ELSE
{
RECONSUME_IN(ScriptData);
}
}
END_STATE
LibWeb: Add spec links to each HTML tokenizer state section I didn't add full spec comments this time, but this is better than nothing :^)
2022-02-15 18:12:15 +00:00
// 13.2.5.19 Script data escape start dash state, https://html.spec.whatwg.org/multipage/parsing.html#script-data-escape-start-dash-state
LibWeb: Implement a bunch more script-related tokenization states
2020-05-28 18:44:17 +02:00
BEGIN_STATE(ScriptDataEscapeStartDash)
{
ON('-')
{
LibWeb: Fully implement all script tokenizer states Also fixes RAWTEXTLessThanSign having a separate emit and reconsume.
2020-06-06 07:06:46 +01:00
SWITCH_TO_AND_EMIT_CHARACTER('-', ScriptDataEscapedDashDash);
LibWeb: Implement a bunch more script-related tokenization states
2020-05-28 18:44:17 +02:00
}
ANYTHING_ELSE
{
RECONSUME_IN(ScriptData);
}
}
END_STATE
LibWeb: Add spec links to each HTML tokenizer state section I didn't add full spec comments this time, but this is better than nothing :^)
2022-02-15 18:12:15 +00:00
// 13.2.5.22 Script data escaped dash dash state, https://html.spec.whatwg.org/multipage/parsing.html#script-data-escaped-dash-dash-state
LibWeb: Implement a bunch more script-related tokenization states
2020-05-28 18:44:17 +02:00
BEGIN_STATE(ScriptDataEscapedDashDash)
{
ON('-')
{
EMIT_CHARACTER('-');
}
ON('<')
{
SWITCH_TO(ScriptDataEscapedLessThanSign);
}
ON('>')
{
LibWeb: Fully implement all script tokenizer states Also fixes RAWTEXTLessThanSign having a separate emit and reconsume.
2020-06-06 07:06:46 +01:00
SWITCH_TO_AND_EMIT_CHARACTER('>', ScriptData);
LibWeb: Implement a bunch more script-related tokenization states
2020-05-28 18:44:17 +02:00
}
ON(0)
{
LibWeb: Utilize SourceLocation for HTMLTokenizer logging
2021-04-24 20:22:30 -07:00
log_parse_error();
LibWeb: Fully implement all script tokenizer states Also fixes RAWTEXTLessThanSign having a separate emit and reconsume.
2020-06-06 07:06:46 +01:00
SWITCH_TO_AND_EMIT_CHARACTER(0xFFFD, ScriptDataEscaped);
LibWeb: Implement a bunch more script-related tokenization states
2020-05-28 18:44:17 +02:00
}
ON_EOF
{
LibWeb: Utilize SourceLocation for HTMLTokenizer logging
2021-04-24 20:22:30 -07:00
log_parse_error();
LibWeb: Fully implement all script tokenizer states Also fixes RAWTEXTLessThanSign having a separate emit and reconsume.
2020-06-06 07:06:46 +01:00
EMIT_EOF;
LibWeb: Implement a bunch more script-related tokenization states
2020-05-28 18:44:17 +02:00
}
ANYTHING_ELSE
{
LibWeb: Fully implement all script tokenizer states Also fixes RAWTEXTLessThanSign having a separate emit and reconsume.
2020-06-06 07:06:46 +01:00
SWITCH_TO_AND_EMIT_CURRENT_CHARACTER(ScriptDataEscaped);
LibWeb: Implement a bunch more script-related tokenization states
2020-05-28 18:44:17 +02:00
}
}
END_STATE
LibWeb: Add spec links to each HTML tokenizer state section I didn't add full spec comments this time, but this is better than nothing :^)
2022-02-15 18:12:15 +00:00
// 13.2.5.23 Script data escaped less-than sign state, https://html.spec.whatwg.org/multipage/parsing.html#script-data-escaped-less-than-sign-state
LibWeb: Implement a bunch more script-related tokenization states
2020-05-28 18:44:17 +02:00
BEGIN_STATE(ScriptDataEscapedLessThanSign)
{
ON('/')
{
m_temporary_buffer.clear();
SWITCH_TO(ScriptDataEscapedEndTagOpen);
}
ON_ASCII_ALPHA
{
m_temporary_buffer.clear();
EMIT_CHARACTER_AND_RECONSUME_IN('<', ScriptDataDoubleEscapeStart);
}
ANYTHING_ELSE
{
EMIT_CHARACTER_AND_RECONSUME_IN('<', ScriptDataEscaped);
}
}
END_STATE
LibWeb: Add spec links to each HTML tokenizer state section I didn't add full spec comments this time, but this is better than nothing :^)
2022-02-15 18:12:15 +00:00
// 13.2.5.24 Script data escaped end tag open state, https://html.spec.whatwg.org/multipage/parsing.html#script-data-escaped-end-tag-open-state
LibWeb: Implement a bunch more script-related tokenization states
2020-05-28 18:44:17 +02:00
BEGIN_STATE(ScriptDataEscapedEndTagOpen)
{
ON_ASCII_ALPHA
{
create_new_token(HTMLToken::Type::EndTag);
RECONSUME_IN(ScriptDataEscapedEndTagName);
}
ANYTHING_ELSE
{
m_queued_tokens.enqueue(HTMLToken::make_character('<'));
m_queued_tokens.enqueue(HTMLToken::make_character('/'));
RECONSUME_IN(ScriptDataEscaped);
}
}
END_STATE
LibWeb: Add spec links to each HTML tokenizer state section I didn't add full spec comments this time, but this is better than nothing :^)
2022-02-15 18:12:15 +00:00
// 13.2.5.25 Script data escaped end tag name state, https://html.spec.whatwg.org/multipage/parsing.html#script-data-escaped-end-tag-name-state
LibWeb: Implement a bunch more script-related tokenization states
2020-05-28 18:44:17 +02:00
BEGIN_STATE(ScriptDataEscapedEndTagName)
{
ON_WHITESPACE
{
LibWeb: User getter+setter for HTMLToken tag name and self-closing flag
2021-07-14 23:37:48 +02:00
m_current_token.set_tag_name(consume_current_builder());
LibWeb: Fully implement all script tokenizer states Also fixes RAWTEXTLessThanSign having a separate emit and reconsume.
2020-06-06 07:06:46 +01:00
if (current_end_tag_token_is_appropriate())
LibWeb: Implement a bunch more script-related tokenization states
2020-05-28 18:44:17 +02:00
SWITCH_TO(BeforeAttributeName);
LibWeb: Fully implement all script tokenizer states Also fixes RAWTEXTLessThanSign having a separate emit and reconsume.
2020-06-06 07:06:46 +01:00
m_queued_tokens.enqueue(HTMLToken::make_character('<'));
m_queued_tokens.enqueue(HTMLToken::make_character('/'));
LibWeb: Fix assertion failures in HTMLTokenizer The *TagName states are all very similar, so it seems to be correct to apply the fix from #8761 to all of those states. This fixes #8788.
2021-07-16 00:36:10 +02:00
// NOTE: The spec doesn't mention this, but it seems that m_current_token (an end tag) is just dropped in this case.
m_current_builder.clear();
Unicode: Try s/codepoint/code_point/g again This time, without trailing 's'. Ran: git grep -l 'codepoint' | xargs sed -ie 's/codepoint/code_point/g
2020-08-05 16:31:20 -04:00
for (auto code_point : m_temporary_buffer) {
m_queued_tokens.enqueue(HTMLToken::make_character(code_point));
LibWeb: Implement a bunch more script-related tokenization states
2020-05-28 18:44:17 +02:00
}
LibWeb: Fully implement all script tokenizer states Also fixes RAWTEXTLessThanSign having a separate emit and reconsume.
2020-06-06 07:06:46 +01:00
RECONSUME_IN(ScriptDataEscaped);
LibWeb: Implement a bunch more script-related tokenization states
2020-05-28 18:44:17 +02:00
}
ON('/')
{
LibWeb: User getter+setter for HTMLToken tag name and self-closing flag
2021-07-14 23:37:48 +02:00
m_current_token.set_tag_name(consume_current_builder());
LibWeb: Fully implement all script tokenizer states Also fixes RAWTEXTLessThanSign having a separate emit and reconsume.
2020-06-06 07:06:46 +01:00
if (current_end_tag_token_is_appropriate())
LibWeb: Implement a bunch more script-related tokenization states
2020-05-28 18:44:17 +02:00
SWITCH_TO(SelfClosingStartTag);
LibWeb: Fully implement all script tokenizer states Also fixes RAWTEXTLessThanSign having a separate emit and reconsume.
2020-06-06 07:06:46 +01:00
m_queued_tokens.enqueue(HTMLToken::make_character('<'));
m_queued_tokens.enqueue(HTMLToken::make_character('/'));
LibWeb: Fix assertion failures in HTMLTokenizer The *TagName states are all very similar, so it seems to be correct to apply the fix from #8761 to all of those states. This fixes #8788.
2021-07-16 00:36:10 +02:00
// NOTE: The spec doesn't mention this, but it seems that m_current_token (an end tag) is just dropped in this case.
m_current_builder.clear();
Unicode: Try s/codepoint/code_point/g again This time, without trailing 's'. Ran: git grep -l 'codepoint' | xargs sed -ie 's/codepoint/code_point/g
2020-08-05 16:31:20 -04:00
for (auto code_point : m_temporary_buffer) {
m_queued_tokens.enqueue(HTMLToken::make_character(code_point));
LibWeb: Implement a bunch more script-related tokenization states
2020-05-28 18:44:17 +02:00
}
LibWeb: Fully implement all script tokenizer states Also fixes RAWTEXTLessThanSign having a separate emit and reconsume.
2020-06-06 07:06:46 +01:00
RECONSUME_IN(ScriptDataEscaped);
LibWeb: Implement a bunch more script-related tokenization states
2020-05-28 18:44:17 +02:00
}
ON('>')
{
LibWeb: User getter+setter for HTMLToken tag name and self-closing flag
2021-07-14 23:37:48 +02:00
m_current_token.set_tag_name(consume_current_builder());
LibWeb: Fully implement all script tokenizer states Also fixes RAWTEXTLessThanSign having a separate emit and reconsume.
2020-06-06 07:06:46 +01:00
if (current_end_tag_token_is_appropriate())
LibWeb: Implement a bunch more script-related tokenization states
2020-05-28 18:44:17 +02:00
SWITCH_TO_AND_EMIT_CURRENT_TOKEN(Data);
LibWeb: Fully implement all script tokenizer states Also fixes RAWTEXTLessThanSign having a separate emit and reconsume.
2020-06-06 07:06:46 +01:00
m_queued_tokens.enqueue(HTMLToken::make_character('<'));
m_queued_tokens.enqueue(HTMLToken::make_character('/'));
LibWeb: Fix assertion failures in HTMLTokenizer The *TagName states are all very similar, so it seems to be correct to apply the fix from #8761 to all of those states. This fixes #8788.
2021-07-16 00:36:10 +02:00
// NOTE: The spec doesn't mention this, but it seems that m_current_token (an end tag) is just dropped in this case.
m_current_builder.clear();
Unicode: Try s/codepoint/code_point/g again This time, without trailing 's'. Ran: git grep -l 'codepoint' | xargs sed -ie 's/codepoint/code_point/g
2020-08-05 16:31:20 -04:00
for (auto code_point : m_temporary_buffer) {
m_queued_tokens.enqueue(HTMLToken::make_character(code_point));
LibWeb: Implement a bunch more script-related tokenization states
2020-05-28 18:44:17 +02:00
}
LibWeb: Fully implement all script tokenizer states Also fixes RAWTEXTLessThanSign having a separate emit and reconsume.
2020-06-06 07:06:46 +01:00
RECONSUME_IN(ScriptDataEscaped);
LibWeb: Implement a bunch more script-related tokenization states
2020-05-28 18:44:17 +02:00
}
ON_ASCII_UPPER_ALPHA
{
LibWeb: Remove StringBuilder from HTMLToken::m_tag
2021-05-23 09:16:07 +02:00
m_current_builder.append_code_point(to_ascii_lowercase(current_input_character.value()));
LibWeb: Implement a bunch more script-related tokenization states
2020-05-28 18:44:17 +02:00
m_temporary_buffer.append(current_input_character.value());
continue;
}
ON_ASCII_LOWER_ALPHA
{
LibWeb: Remove StringBuilder from HTMLToken::m_tag
2021-05-23 09:16:07 +02:00
m_current_builder.append(current_input_character.value());
LibWeb: Implement a bunch more script-related tokenization states
2020-05-28 18:44:17 +02:00
m_temporary_buffer.append(current_input_character.value());
continue;
}
ANYTHING_ELSE
{
m_queued_tokens.enqueue(HTMLToken::make_character('<'));
m_queued_tokens.enqueue(HTMLToken::make_character('/'));
LibWeb: Fix assertion failures in HTMLTokenizer The *TagName states are all very similar, so it seems to be correct to apply the fix from #8761 to all of those states. This fixes #8788.
2021-07-16 00:36:10 +02:00
// NOTE: The spec doesn't mention this, but it seems that m_current_token (an end tag) is just dropped in this case.
m_current_builder.clear();
Unicode: Try s/codepoint/code_point/g again This time, without trailing 's'. Ran: git grep -l 'codepoint' | xargs sed -ie 's/codepoint/code_point/g
2020-08-05 16:31:20 -04:00
for (auto code_point : m_temporary_buffer) {
m_queued_tokens.enqueue(HTMLToken::make_character(code_point));
LibWeb: Implement a bunch more script-related tokenization states
2020-05-28 18:44:17 +02:00
}
RECONSUME_IN(ScriptDataEscaped);
}
}
END_STATE
LibWeb: Add spec links to each HTML tokenizer state section I didn't add full spec comments this time, but this is better than nothing :^)
2022-02-15 18:12:15 +00:00
// 13.2.5.26 Script data double escape start state, https://html.spec.whatwg.org/multipage/parsing.html#script-data-double-escape-start-state
LibWeb: Implement a bunch more script-related tokenization states
2020-05-28 18:44:17 +02:00
BEGIN_STATE(ScriptDataDoubleEscapeStart)
{
LibWeb: Fully implement all script tokenizer states Also fixes RAWTEXTLessThanSign having a separate emit and reconsume.
2020-06-06 07:06:46 +01:00
auto temporary_buffer_equal_to_script = [this]() -> bool {
if (m_temporary_buffer.size() != 6)
return false;
// FIXME: Is there a better way of doing this?
LibWeb: Move the HTML parser into HTML/Parser/
2020-07-28 19:18:23 +02:00
return m_temporary_buffer[0] == 's' && m_temporary_buffer[1] == 'c' && m_temporary_buffer[2] == 'r' && m_temporary_buffer[3] == 'i' && m_temporary_buffer[4] == 'p' && m_temporary_buffer[5] == 't';
LibWeb: Fully implement all script tokenizer states Also fixes RAWTEXTLessThanSign having a separate emit and reconsume.
2020-06-06 07:06:46 +01:00
};
ON_WHITESPACE
{
if (temporary_buffer_equal_to_script())
SWITCH_TO_AND_EMIT_CURRENT_CHARACTER(ScriptDataDoubleEscaped);
else
SWITCH_TO_AND_EMIT_CURRENT_CHARACTER(ScriptDataEscaped);
}
ON('/')
{
if (temporary_buffer_equal_to_script())
SWITCH_TO_AND_EMIT_CURRENT_CHARACTER(ScriptDataDoubleEscaped);
else
SWITCH_TO_AND_EMIT_CURRENT_CHARACTER(ScriptDataEscaped);
}
ON('>')
{
if (temporary_buffer_equal_to_script())
SWITCH_TO_AND_EMIT_CURRENT_CHARACTER(ScriptDataDoubleEscaped);
else
SWITCH_TO_AND_EMIT_CURRENT_CHARACTER(ScriptDataEscaped);
}
ON_ASCII_UPPER_ALPHA
{
Everywhere: Replace ctype.h to avoid narrowing conversions This replaces ctype.h with CharacterType.h everywhere I could find issues with narrowing conversions. While using it will probably make sense almost everywhere in the future, the most critical places should have been addressed.
2021-06-01 21:18:08 +02:00
m_temporary_buffer.append(to_ascii_lowercase(current_input_character.value()));
LibWeb: Fully implement all script tokenizer states Also fixes RAWTEXTLessThanSign having a separate emit and reconsume.
2020-06-06 07:06:46 +01:00
EMIT_CURRENT_CHARACTER;
}
ON_ASCII_LOWER_ALPHA
{
m_temporary_buffer.append(current_input_character.value());
EMIT_CURRENT_CHARACTER;
}
ANYTHING_ELSE
{
RECONSUME_IN(ScriptDataEscaped);
}
}
END_STATE
LibWeb: Add spec links to each HTML tokenizer state section I didn't add full spec comments this time, but this is better than nothing :^)
2022-02-15 18:12:15 +00:00
// 13.2.5.27 Script data double escaped state, https://html.spec.whatwg.org/multipage/parsing.html#script-data-double-escaped-state
LibWeb: Fully implement all script tokenizer states Also fixes RAWTEXTLessThanSign having a separate emit and reconsume.
2020-06-06 07:06:46 +01:00
BEGIN_STATE(ScriptDataDoubleEscaped)
{
ON('-')
{
SWITCH_TO_AND_EMIT_CHARACTER('-', ScriptDataDoubleEscapedDash);
}
ON('<')
{
SWITCH_TO_AND_EMIT_CHARACTER('<', ScriptDataDoubleEscapedLessThanSign);
}
ON(0)
{
LibWeb: Utilize SourceLocation for HTMLTokenizer logging
2021-04-24 20:22:30 -07:00
log_parse_error();
LibWeb: Fully implement all script tokenizer states Also fixes RAWTEXTLessThanSign having a separate emit and reconsume.
2020-06-06 07:06:46 +01:00
EMIT_CHARACTER(0xFFFD);
}
ON_EOF
{
LibWeb: Utilize SourceLocation for HTMLTokenizer logging
2021-04-24 20:22:30 -07:00
log_parse_error();
LibWeb: Fully implement all script tokenizer states Also fixes RAWTEXTLessThanSign having a separate emit and reconsume.
2020-06-06 07:06:46 +01:00
EMIT_EOF;
}
ANYTHING_ELSE
{
EMIT_CURRENT_CHARACTER;
}
}
END_STATE
LibWeb: Add spec links to each HTML tokenizer state section I didn't add full spec comments this time, but this is better than nothing :^)
2022-02-15 18:12:15 +00:00
// 13.2.5.28 Script data double escaped dash state, https://html.spec.whatwg.org/multipage/parsing.html#script-data-double-escaped-dash-state
LibWeb: Fully implement all script tokenizer states Also fixes RAWTEXTLessThanSign having a separate emit and reconsume.
2020-06-06 07:06:46 +01:00
BEGIN_STATE(ScriptDataDoubleEscapedDash)
{
ON('-')
{
SWITCH_TO_AND_EMIT_CHARACTER('-', ScriptDataDoubleEscapedDashDash);
}
ON('<')
{
SWITCH_TO_AND_EMIT_CHARACTER('<', ScriptDataDoubleEscapedLessThanSign);
}
ON(0)
{
LibWeb: Utilize SourceLocation for HTMLTokenizer logging
2021-04-24 20:22:30 -07:00
log_parse_error();
LibWeb: Fully implement all script tokenizer states Also fixes RAWTEXTLessThanSign having a separate emit and reconsume.
2020-06-06 07:06:46 +01:00
SWITCH_TO_AND_EMIT_CHARACTER(0xFFFD, ScriptDataDoubleEscaped);
}
ON_EOF
{
LibWeb: Utilize SourceLocation for HTMLTokenizer logging
2021-04-24 20:22:30 -07:00
log_parse_error();
LibWeb: Fully implement all script tokenizer states Also fixes RAWTEXTLessThanSign having a separate emit and reconsume.
2020-06-06 07:06:46 +01:00
EMIT_EOF;
}
ANYTHING_ELSE
{
SWITCH_TO_AND_EMIT_CURRENT_CHARACTER(ScriptDataDoubleEscaped);
}
}
END_STATE
LibWeb: Add spec links to each HTML tokenizer state section I didn't add full spec comments this time, but this is better than nothing :^)
2022-02-15 18:12:15 +00:00
// 13.2.5.29 Script data double escaped dash dash state, https://html.spec.whatwg.org/multipage/parsing.html#script-data-double-escaped-dash-dash-state
LibWeb: Fully implement all script tokenizer states Also fixes RAWTEXTLessThanSign having a separate emit and reconsume.
2020-06-06 07:06:46 +01:00
BEGIN_STATE(ScriptDataDoubleEscapedDashDash)
{
ON('-')
{
EMIT_CHARACTER('-');
}
ON('<')
{
SWITCH_TO_AND_EMIT_CHARACTER('<', ScriptDataDoubleEscapedLessThanSign);
}
ON('>')
{
SWITCH_TO_AND_EMIT_CHARACTER('>', ScriptData);
}
ON(0)
{
LibWeb: Utilize SourceLocation for HTMLTokenizer logging
2021-04-24 20:22:30 -07:00
log_parse_error();
LibWeb: Fully implement all script tokenizer states Also fixes RAWTEXTLessThanSign having a separate emit and reconsume.
2020-06-06 07:06:46 +01:00
SWITCH_TO_AND_EMIT_CHARACTER(0xFFFD, ScriptDataDoubleEscaped);
}
ON_EOF
{
LibWeb: Utilize SourceLocation for HTMLTokenizer logging
2021-04-24 20:22:30 -07:00
log_parse_error();
LibWeb: Fully implement all script tokenizer states Also fixes RAWTEXTLessThanSign having a separate emit and reconsume.
2020-06-06 07:06:46 +01:00
EMIT_EOF;
}
ANYTHING_ELSE
{
SWITCH_TO_AND_EMIT_CURRENT_CHARACTER(ScriptDataDoubleEscaped);
}
}
END_STATE
LibWeb: Add spec links to each HTML tokenizer state section I didn't add full spec comments this time, but this is better than nothing :^)
2022-02-15 18:12:15 +00:00
// 13.2.5.30 Script data double escaped less-than sign state, https://html.spec.whatwg.org/multipage/parsing.html#script-data-double-escaped-less-than-sign-state
LibWeb: Fully implement all script tokenizer states Also fixes RAWTEXTLessThanSign having a separate emit and reconsume.
2020-06-06 07:06:46 +01:00
BEGIN_STATE(ScriptDataDoubleEscapedLessThanSign)
{
ON('/')
{
m_temporary_buffer.clear();
SWITCH_TO_AND_EMIT_CHARACTER('/', ScriptDataDoubleEscapeEnd);
}
ANYTHING_ELSE
{
RECONSUME_IN(ScriptDataDoubleEscaped);
}
}
END_STATE
LibWeb: Add spec links to each HTML tokenizer state section I didn't add full spec comments this time, but this is better than nothing :^)
2022-02-15 18:12:15 +00:00
// 13.2.5.31 Script data double escape end state, https://html.spec.whatwg.org/multipage/parsing.html#script-data-double-escape-end-state
LibWeb: Fully implement all script tokenizer states Also fixes RAWTEXTLessThanSign having a separate emit and reconsume.
2020-06-06 07:06:46 +01:00
BEGIN_STATE(ScriptDataDoubleEscapeEnd)
{
auto temporary_buffer_equal_to_script = [this]() -> bool {
if (m_temporary_buffer.size() != 6)
return false;
// FIXME: Is there a better way of doing this?
LibWeb: Move the HTML parser into HTML/Parser/
2020-07-28 19:18:23 +02:00
return m_temporary_buffer[0] == 's' && m_temporary_buffer[1] == 'c' && m_temporary_buffer[2] == 'r' && m_temporary_buffer[3] == 'i' && m_temporary_buffer[4] == 'p' && m_temporary_buffer[5] == 't';
LibWeb: Fully implement all script tokenizer states Also fixes RAWTEXTLessThanSign having a separate emit and reconsume.
2020-06-06 07:06:46 +01:00
};
ON_WHITESPACE
{
if (temporary_buffer_equal_to_script())
SWITCH_TO_AND_EMIT_CURRENT_CHARACTER(ScriptDataEscaped);
else
SWITCH_TO_AND_EMIT_CURRENT_CHARACTER(ScriptDataDoubleEscaped);
}
ON('/')
{
if (temporary_buffer_equal_to_script())
SWITCH_TO_AND_EMIT_CURRENT_CHARACTER(ScriptDataEscaped);
else
SWITCH_TO_AND_EMIT_CURRENT_CHARACTER(ScriptDataDoubleEscaped);
}
ON('>')
{
if (temporary_buffer_equal_to_script())
SWITCH_TO_AND_EMIT_CURRENT_CHARACTER(ScriptDataEscaped);
else
SWITCH_TO_AND_EMIT_CURRENT_CHARACTER(ScriptDataDoubleEscaped);
}
ON_ASCII_UPPER_ALPHA
{
Everywhere: Replace ctype.h to avoid narrowing conversions This replaces ctype.h with CharacterType.h everywhere I could find issues with narrowing conversions. While using it will probably make sense almost everywhere in the future, the most critical places should have been addressed.
2021-06-01 21:18:08 +02:00
m_temporary_buffer.append(to_ascii_lowercase(current_input_character.value()));
LibWeb: Fully implement all script tokenizer states Also fixes RAWTEXTLessThanSign having a separate emit and reconsume.
2020-06-06 07:06:46 +01:00
EMIT_CURRENT_CHARACTER;
}
ON_ASCII_LOWER_ALPHA
{
m_temporary_buffer.append(current_input_character.value());
EMIT_CURRENT_CHARACTER;
}
ANYTHING_ELSE
{
RECONSUME_IN(ScriptDataDoubleEscaped);
}
LibWeb: Implement a bunch more script-related tokenization states
2020-05-28 18:44:17 +02:00
}
END_STATE
LibWeb: Add spec links to each HTML tokenizer state section I didn't add full spec comments this time, but this is better than nothing :^)
2022-02-15 18:12:15 +00:00
// 13.2.5.21 Script data escaped dash state, https://html.spec.whatwg.org/multipage/parsing.html#script-data-escaped-dash-state
LibWeb: Implement a bunch more script-related tokenization states
2020-05-28 18:44:17 +02:00
BEGIN_STATE(ScriptDataEscapedDash)
{
ON('-')
{
LibWeb: Fully implement all script tokenizer states Also fixes RAWTEXTLessThanSign having a separate emit and reconsume.
2020-06-06 07:06:46 +01:00
SWITCH_TO_AND_EMIT_CHARACTER('-', ScriptDataEscapedDashDash);
LibWeb: Implement a bunch more script-related tokenization states
2020-05-28 18:44:17 +02:00
}
ON('<')
{
SWITCH_TO(ScriptDataEscapedLessThanSign);
}
ON(0)
{
LibWeb: Utilize SourceLocation for HTMLTokenizer logging
2021-04-24 20:22:30 -07:00
log_parse_error();
LibWeb: Fully implement all script tokenizer states Also fixes RAWTEXTLessThanSign having a separate emit and reconsume.
2020-06-06 07:06:46 +01:00
SWITCH_TO_AND_EMIT_CHARACTER(0xFFFD, ScriptDataEscaped);
LibWeb: Implement a bunch more script-related tokenization states
2020-05-28 18:44:17 +02:00
}
ON_EOF
{
LibWeb: Utilize SourceLocation for HTMLTokenizer logging
2021-04-24 20:22:30 -07:00
log_parse_error();
LibWeb: Fully implement all script tokenizer states Also fixes RAWTEXTLessThanSign having a separate emit and reconsume.
2020-06-06 07:06:46 +01:00
EMIT_EOF;
LibWeb: Implement a bunch more script-related tokenization states
2020-05-28 18:44:17 +02:00
}
ANYTHING_ELSE
{
LibWeb: Fully implement all script tokenizer states Also fixes RAWTEXTLessThanSign having a separate emit and reconsume.
2020-06-06 07:06:46 +01:00
SWITCH_TO_AND_EMIT_CURRENT_CHARACTER(ScriptDataEscaped);
LibWeb: Implement a bunch more script-related tokenization states
2020-05-28 18:44:17 +02:00
}
}
END_STATE
LibWeb: Add spec links to each HTML tokenizer state section I didn't add full spec comments this time, but this is better than nothing :^)
2022-02-15 18:12:15 +00:00
// 13.2.5.20 Script data escaped state, https://html.spec.whatwg.org/multipage/parsing.html#script-data-escaped-state
LibWeb: Implement a bunch more script-related tokenization states
2020-05-28 18:44:17 +02:00
BEGIN_STATE(ScriptDataEscaped)
{
ON('-')
{
LibWeb: Fully implement all script tokenizer states Also fixes RAWTEXTLessThanSign having a separate emit and reconsume.
2020-06-06 07:06:46 +01:00
SWITCH_TO_AND_EMIT_CHARACTER('-', ScriptDataEscapedDash);
LibWeb: Implement a bunch more script-related tokenization states
2020-05-28 18:44:17 +02:00
}
ON('<')
{
SWITCH_TO(ScriptDataEscapedLessThanSign);
}
ON(0)
{
LibWeb: Utilize SourceLocation for HTMLTokenizer logging
2021-04-24 20:22:30 -07:00
log_parse_error();
LibWeb: Fully implement all script tokenizer states Also fixes RAWTEXTLessThanSign having a separate emit and reconsume.
2020-06-06 07:06:46 +01:00
EMIT_CHARACTER(0xFFFD);
LibWeb: Implement a bunch more script-related tokenization states
2020-05-28 18:44:17 +02:00
}
ON_EOF
{
LibWeb: Utilize SourceLocation for HTMLTokenizer logging
2021-04-24 20:22:30 -07:00
log_parse_error();
LibWeb: Fully implement all script tokenizer states Also fixes RAWTEXTLessThanSign having a separate emit and reconsume.
2020-06-06 07:06:46 +01:00
EMIT_EOF;
LibWeb: Implement a bunch more script-related tokenization states
2020-05-28 18:44:17 +02:00
}
ANYTHING_ELSE
{
EMIT_CURRENT_CHARACTER;
}
}
END_STATE
LibWeb: Add spec links to each HTML tokenizer state section I didn't add full spec comments this time, but this is better than nothing :^)
2022-02-15 18:12:15 +00:00
// 13.2.5.16 Script data end tag open state, https://html.spec.whatwg.org/multipage/parsing.html#script-data-end-tag-open-state
LibWeb: A whole bunch of work towards spec-compliant <script> elements This is still very unfinished, but there's at least a skeleton of code.
2020-05-24 22:00:46 +02:00
BEGIN_STATE(ScriptDataEndTagOpen)
{
ON_ASCII_ALPHA
{
create_new_token(HTMLToken::Type::EndTag);
RECONSUME_IN(ScriptDataEndTagName);
}
ANYTHING_ELSE
{
LibWeb: Fully implement all script tokenizer states Also fixes RAWTEXTLessThanSign having a separate emit and reconsume.
2020-06-06 07:06:46 +01:00
m_queued_tokens.enqueue(HTMLToken::make_character('<'));
m_queued_tokens.enqueue(HTMLToken::make_character('/'));
RECONSUME_IN(ScriptData);
LibWeb: A whole bunch of work towards spec-compliant <script> elements This is still very unfinished, but there's at least a skeleton of code.
2020-05-24 22:00:46 +02:00
}
}
END_STATE
LibWeb: Add spec links to each HTML tokenizer state section I didn't add full spec comments this time, but this is better than nothing :^)
2022-02-15 18:12:15 +00:00
// 13.2.5.17 Script data end tag name state, https://html.spec.whatwg.org/multipage/parsing.html#script-data-end-tag-name-state
LibWeb: A whole bunch of work towards spec-compliant <script> elements This is still very unfinished, but there's at least a skeleton of code.
2020-05-24 22:00:46 +02:00
BEGIN_STATE(ScriptDataEndTagName)
{
ON_WHITESPACE
{
LibWeb: User getter+setter for HTMLToken tag name and self-closing flag
2021-07-14 23:37:48 +02:00
m_current_token.set_tag_name(consume_current_builder());
LibWeb: A whole bunch of work towards spec-compliant <script> elements This is still very unfinished, but there's at least a skeleton of code.
2020-05-24 22:00:46 +02:00
if (current_end_tag_token_is_appropriate())
SWITCH_TO(BeforeAttributeName);
LibWeb: Implement more of the ScriptDataEndTagName tokenizer state Some of this is extremely repetitive. We'll need to rethink how we do queue/emit to improve this.
2020-05-30 22:59:41 +02:00
m_queued_tokens.enqueue(HTMLToken::make_character('<'));
m_queued_tokens.enqueue(HTMLToken::make_character('/'));
LibWeb: Fix assertion failures in HTMLTokenizer The *TagName states are all very similar, so it seems to be correct to apply the fix from #8761 to all of those states. This fixes #8788.
2021-07-16 00:36:10 +02:00
// NOTE: The spec doesn't mention this, but it seems that m_current_token (an end tag) is just dropped in this case.
m_current_builder.clear();
Unicode: Try s/codepoint/code_point/g again This time, without trailing 's'. Ran: git grep -l 'codepoint' | xargs sed -ie 's/codepoint/code_point/g
2020-08-05 16:31:20 -04:00
for (auto code_point : m_temporary_buffer)
m_queued_tokens.enqueue(HTMLToken::make_character(code_point));
LibWeb: Implement more of the ScriptDataEndTagName tokenizer state Some of this is extremely repetitive. We'll need to rethink how we do queue/emit to improve this.
2020-05-30 22:59:41 +02:00
RECONSUME_IN(ScriptData);
LibWeb: A whole bunch of work towards spec-compliant <script> elements This is still very unfinished, but there's at least a skeleton of code.
2020-05-24 22:00:46 +02:00
}
ON('/')
{
LibWeb: User getter+setter for HTMLToken tag name and self-closing flag
2021-07-14 23:37:48 +02:00
m_current_token.set_tag_name(consume_current_builder());
LibWeb: A whole bunch of work towards spec-compliant <script> elements This is still very unfinished, but there's at least a skeleton of code.
2020-05-24 22:00:46 +02:00
if (current_end_tag_token_is_appropriate())
SWITCH_TO(SelfClosingStartTag);
LibWeb: Implement more of the ScriptDataEndTagName tokenizer state Some of this is extremely repetitive. We'll need to rethink how we do queue/emit to improve this.
2020-05-30 22:59:41 +02:00
m_queued_tokens.enqueue(HTMLToken::make_character('<'));
m_queued_tokens.enqueue(HTMLToken::make_character('/'));
LibWeb: Fix assertion failures in HTMLTokenizer The *TagName states are all very similar, so it seems to be correct to apply the fix from #8761 to all of those states. This fixes #8788.
2021-07-16 00:36:10 +02:00
// NOTE: The spec doesn't mention this, but it seems that m_current_token (an end tag) is just dropped in this case.
m_current_builder.clear();
Unicode: Try s/codepoint/code_point/g again This time, without trailing 's'. Ran: git grep -l 'codepoint' | xargs sed -ie 's/codepoint/code_point/g
2020-08-05 16:31:20 -04:00
for (auto code_point : m_temporary_buffer)
m_queued_tokens.enqueue(HTMLToken::make_character(code_point));
LibWeb: Implement more of the ScriptDataEndTagName tokenizer state Some of this is extremely repetitive. We'll need to rethink how we do queue/emit to improve this.
2020-05-30 22:59:41 +02:00
RECONSUME_IN(ScriptData);
LibWeb: A whole bunch of work towards spec-compliant <script> elements This is still very unfinished, but there's at least a skeleton of code.
2020-05-24 22:00:46 +02:00
}
ON('>')
{
LibWeb: User getter+setter for HTMLToken tag name and self-closing flag
2021-07-14 23:37:48 +02:00
m_current_token.set_tag_name(consume_current_builder());
LibWeb: A whole bunch of work towards spec-compliant <script> elements This is still very unfinished, but there's at least a skeleton of code.
2020-05-24 22:00:46 +02:00
if (current_end_tag_token_is_appropriate())
SWITCH_TO_AND_EMIT_CURRENT_TOKEN(Data);
LibWeb: Implement more of the ScriptDataEndTagName tokenizer state Some of this is extremely repetitive. We'll need to rethink how we do queue/emit to improve this.
2020-05-30 22:59:41 +02:00
m_queued_tokens.enqueue(HTMLToken::make_character('<'));
m_queued_tokens.enqueue(HTMLToken::make_character('/'));
LibWeb: Fix assertion failures in HTMLTokenizer The *TagName states are all very similar, so it seems to be correct to apply the fix from #8761 to all of those states. This fixes #8788.
2021-07-16 00:36:10 +02:00
// NOTE: The spec doesn't mention this, but it seems that m_current_token (an end tag) is just dropped in this case.
m_current_builder.clear();
Unicode: Try s/codepoint/code_point/g again This time, without trailing 's'. Ran: git grep -l 'codepoint' | xargs sed -ie 's/codepoint/code_point/g
2020-08-05 16:31:20 -04:00
for (auto code_point : m_temporary_buffer)
m_queued_tokens.enqueue(HTMLToken::make_character(code_point));
LibWeb: Implement more of the ScriptDataEndTagName tokenizer state Some of this is extremely repetitive. We'll need to rethink how we do queue/emit to improve this.
2020-05-30 22:59:41 +02:00
RECONSUME_IN(ScriptData);
LibWeb: A whole bunch of work towards spec-compliant <script> elements This is still very unfinished, but there's at least a skeleton of code.
2020-05-24 22:00:46 +02:00
}
ON_ASCII_UPPER_ALPHA
{
LibWeb: Remove StringBuilder from HTMLToken::m_tag
2021-05-23 09:16:07 +02:00
m_current_builder.append_code_point(to_ascii_lowercase(current_input_character.value()));
LibWeb: A whole bunch of work towards spec-compliant <script> elements This is still very unfinished, but there's at least a skeleton of code.
2020-05-24 22:00:46 +02:00
m_temporary_buffer.append(current_input_character.value());
continue;
}
ON_ASCII_LOWER_ALPHA
{
LibWeb: Remove StringBuilder from HTMLToken::m_tag
2021-05-23 09:16:07 +02:00
m_current_builder.append(current_input_character.value());
LibWeb: A whole bunch of work towards spec-compliant <script> elements This is still very unfinished, but there's at least a skeleton of code.
2020-05-24 22:00:46 +02:00
m_temporary_buffer.append(current_input_character.value());
continue;
}
ANYTHING_ELSE
{
LibWeb: Implement more of the ScriptDataEndTagName tokenizer state Some of this is extremely repetitive. We'll need to rethink how we do queue/emit to improve this.
2020-05-30 22:59:41 +02:00
m_queued_tokens.enqueue(HTMLToken::make_character('<'));
m_queued_tokens.enqueue(HTMLToken::make_character('/'));
LibWeb: Fix assertion failure when tokenizing JS regex literals This fixes parsing the following regular expression: /</g; It also adds a simple script element to the HTMLTokenizer regression test, which also contains that specific regex.
2021-07-15 01:25:34 +02:00
// NOTE: The spec doesn't mention this, but it seems that m_current_token (an end tag) is just dropped in this case.
m_current_builder.clear();
Unicode: Try s/codepoint/code_point/g again This time, without trailing 's'. Ran: git grep -l 'codepoint' | xargs sed -ie 's/codepoint/code_point/g
2020-08-05 16:31:20 -04:00
for (auto code_point : m_temporary_buffer)
m_queued_tokens.enqueue(HTMLToken::make_character(code_point));
LibWeb: Implement more of the ScriptDataEndTagName tokenizer state Some of this is extremely repetitive. We'll need to rethink how we do queue/emit to improve this.
2020-05-30 22:59:41 +02:00
RECONSUME_IN(ScriptData);
LibWeb: A whole bunch of work towards spec-compliant <script> elements This is still very unfinished, but there's at least a skeleton of code.
2020-05-24 22:00:46 +02:00
}
}
END_STATE
LibWeb: Add spec links to each HTML tokenizer state section I didn't add full spec comments this time, but this is better than nothing :^)
2022-02-15 18:12:15 +00:00
// 13.2.5.69 CDATA section state, https://html.spec.whatwg.org/multipage/parsing.html#cdata-section-state
LibWeb: Implement all CDATA tokenizer states Even though we haven't implemented any switches to these states yet, we may as well have them ready for when we do implement the switches.
2020-06-12 23:43:06 +01:00
BEGIN_STATE(CDATASection)
{
ON(']')
{
SWITCH_TO(CDATASectionBracket);
}
ON_EOF
{
LibWeb: Utilize SourceLocation for HTMLTokenizer logging
2021-04-24 20:22:30 -07:00
log_parse_error();
LibWeb: Implement all CDATA tokenizer states Even though we haven't implemented any switches to these states yet, we may as well have them ready for when we do implement the switches.
2020-06-12 23:43:06 +01:00
EMIT_EOF;
}
ANYTHING_ELSE
{
EMIT_CURRENT_CHARACTER;
}
}
END_STATE
LibWeb: Add spec links to each HTML tokenizer state section I didn't add full spec comments this time, but this is better than nothing :^)
2022-02-15 18:12:15 +00:00
// 13.2.5.70 CDATA section bracket state, https://html.spec.whatwg.org/multipage/parsing.html#cdata-section-bracket-state
LibWeb: Implement all CDATA tokenizer states Even though we haven't implemented any switches to these states yet, we may as well have them ready for when we do implement the switches.
2020-06-12 23:43:06 +01:00
BEGIN_STATE(CDATASectionBracket)
{
ON(']')
{
SWITCH_TO(CDATASectionEnd);
}
ANYTHING_ELSE
{
EMIT_CHARACTER_AND_RECONSUME_IN(']', CDATASection);
}
}
END_STATE
LibWeb: Add spec links to each HTML tokenizer state section I didn't add full spec comments this time, but this is better than nothing :^)
2022-02-15 18:12:15 +00:00
// 13.2.5.71 CDATA section end state, https://html.spec.whatwg.org/multipage/parsing.html#cdata-section-end-state
LibWeb: Implement all CDATA tokenizer states Even though we haven't implemented any switches to these states yet, we may as well have them ready for when we do implement the switches.
2020-06-12 23:43:06 +01:00
BEGIN_STATE(CDATASectionEnd)
{
ON(']')
{
EMIT_CHARACTER(']');
}
ON('>')
{
SWITCH_TO(Data);
}
ANYTHING_ELSE
{
m_queued_tokens.enqueue(HTMLToken::make_character(']'));
m_queued_tokens.enqueue(HTMLToken::make_character(']'));
RECONSUME_IN(CDATASection);
}
}
END_STATE
LibWeb: Begin work on a spec-compliant HTML parser In order to actually view the web as it is, we're gonna need a proper HTML parser. So let's build one! This patch introduces the Web::HTMLTokenizer class, which currently operates on a StringView input stream where it fetches (ASCII only atm) codepoints and tokenizes acccording to the HTML spec tokenization algo. The tokenizer state machine looks a bit weird but is written in a way that tries to mimic the spec as closely as possible, in order to make development easier and bugs less likely. This initial version is far from finished, but it can parse a trivial document with a DOCTYPE and open/close tags. :^)
2020-05-22 21:46:13 +02:00
default:
LibWeb: A whole bunch of work towards spec-compliant <script> elements This is still very unfinished, but there's at least a skeleton of code.
2020-05-24 22:00:46 +02:00
TODO();
LibWeb: Begin work on a spec-compliant HTML parser In order to actually view the web as it is, we're gonna need a proper HTML parser. So let's build one! This patch introduces the Web::HTMLTokenizer class, which currently operates on a StringView input stream where it fetches (ASCII only atm) codepoints and tokenizes acccording to the HTML spec tokenization algo. The tokenizer state machine looks a bit weird but is written in a way that tries to mimic the spec as closely as possible, in order to make development easier and bugs less likely. This initial version is far from finished, but it can parse a trivial document with a DOCTYPE and open/close tags. :^)
2020-05-22 21:46:13 +02:00
}
}
}
LibWeb: Teach more of HTMLTokenizer to stop at the insertion point In particular, input character lookahead now knows how to stop at the insertion point marker if needed. This makes it possible to do amazing things like having document.write() insert doctypes one character at a time.
2024-11-24 10:18:17 +01:00
HTMLTokenizer::ConsumeNextResult HTMLTokenizer::consume_next_if_match(StringView string, StopAtInsertionPoint stop_at_insertion_point, CaseSensitivity case_sensitivity)
LibWeb: Begin work on a spec-compliant HTML parser In order to actually view the web as it is, we're gonna need a proper HTML parser. So let's build one! This patch introduces the Web::HTMLTokenizer class, which currently operates on a StringView input stream where it fetches (ASCII only atm) codepoints and tokenizes acccording to the HTML spec tokenization algo. The tokenizer state machine looks a bit weird but is written in a way that tries to mimic the spec as closely as possible, in order to make development easier and bugs less likely. This initial version is far from finished, but it can parse a trivial document with a DOCTYPE and open/close tags. :^)
2020-05-22 21:46:13 +02:00
{
for (size_t i = 0; i < string.length(); ++i) {
LibWeb: Teach more of HTMLTokenizer to stop at the insertion point In particular, input character lookahead now knows how to stop at the insertion point marker if needed. This makes it possible to do amazing things like having document.write() insert doctypes one character at a time.
2024-11-24 10:18:17 +01:00
auto code_point = peek_code_point(i, stop_at_insertion_point);
if (!code_point.has_value()) {
if (StopAtInsertionPoint::Yes == stop_at_insertion_point) {
return ConsumeNextResult::RanOutOfCharacters;
}
return ConsumeNextResult::NotConsumed;
}
LibWeb: Begin work on a spec-compliant HTML parser In order to actually view the web as it is, we're gonna need a proper HTML parser. So let's build one! This patch introduces the Web::HTMLTokenizer class, which currently operates on a StringView input stream where it fetches (ASCII only atm) codepoints and tokenizes acccording to the HTML spec tokenization algo. The tokenizer state machine looks a bit weird but is written in a way that tries to mimic the spec as closely as possible, in order to make development easier and bugs less likely. This initial version is far from finished, but it can parse a trivial document with a DOCTYPE and open/close tags. :^)
2020-05-22 21:46:13 +02:00
// FIXME: This should be more Unicode-aware.
LibWeb: Checking for "DOCTYPE" should be case insensitive in tokenizer
2020-05-25 19:22:23 +02:00
if (case_sensitivity == CaseSensitivity::CaseInsensitive) {
Unicode: Try s/codepoint/code_point/g again This time, without trailing 's'. Ran: git grep -l 'codepoint' | xargs sed -ie 's/codepoint/code_point/g
2020-08-05 16:31:20 -04:00
if (code_point.value() < 0x80) {
Everywhere: Replace ctype.h to avoid narrowing conversions This replaces ctype.h with CharacterType.h everywhere I could find issues with narrowing conversions. While using it will probably make sense almost everywhere in the future, the most critical places should have been addressed.
2021-06-01 21:18:08 +02:00
if (to_ascii_lowercase(code_point.value()) != to_ascii_lowercase(string[i]))
LibWeb: Teach more of HTMLTokenizer to stop at the insertion point In particular, input character lookahead now knows how to stop at the insertion point marker if needed. This makes it possible to do amazing things like having document.write() insert doctypes one character at a time.
2024-11-24 10:18:17 +01:00
return ConsumeNextResult::NotConsumed;
LibWeb: Checking for "DOCTYPE" should be case insensitive in tokenizer
2020-05-25 19:22:23 +02:00
continue;
}
}
Unicode: Try s/codepoint/code_point/g again This time, without trailing 's'. Ran: git grep -l 'codepoint' | xargs sed -ie 's/codepoint/code_point/g
2020-08-05 16:31:20 -04:00
if (code_point.value() != (u32)string[i])
LibWeb: Teach more of HTMLTokenizer to stop at the insertion point In particular, input character lookahead now knows how to stop at the insertion point marker if needed. This makes it possible to do amazing things like having document.write() insert doctypes one character at a time.
2024-11-24 10:18:17 +01:00
return ConsumeNextResult::NotConsumed;
LibWeb: Begin work on a spec-compliant HTML parser In order to actually view the web as it is, we're gonna need a proper HTML parser. So let's build one! This patch introduces the Web::HTMLTokenizer class, which currently operates on a StringView input stream where it fetches (ASCII only atm) codepoints and tokenizes acccording to the HTML spec tokenization algo. The tokenizer state machine looks a bit weird but is written in a way that tries to mimic the spec as closely as possible, in order to make development easier and bugs less likely. This initial version is far from finished, but it can parse a trivial document with a DOCTYPE and open/close tags. :^)
2020-05-22 21:46:13 +02:00
}
LibWeb: Add position tracking information to HTML tokens
2021-05-20 23:11:41 +04:30
skip(string.length());
LibWeb: Teach more of HTMLTokenizer to stop at the insertion point In particular, input character lookahead now knows how to stop at the insertion point marker if needed. This makes it possible to do amazing things like having document.write() insert doctypes one character at a time.
2024-11-24 10:18:17 +01:00
return ConsumeNextResult::Consumed;
LibWeb: Begin work on a spec-compliant HTML parser In order to actually view the web as it is, we're gonna need a proper HTML parser. So let's build one! This patch introduces the Web::HTMLTokenizer class, which currently operates on a StringView input stream where it fetches (ASCII only atm) codepoints and tokenizes acccording to the HTML spec tokenization algo. The tokenizer state machine looks a bit weird but is written in a way that tries to mimic the spec as closely as possible, in order to make development easier and bugs less likely. This initial version is far from finished, but it can parse a trivial document with a DOCTYPE and open/close tags. :^)
2020-05-22 21:46:13 +02:00
}
void HTMLTokenizer::create_new_token(HTMLToken::Type type)
{
LibWeb: Add HTMLToken(Type) constructor and use it
2021-07-15 15:55:46 +02:00
m_current_token = { type };
LibWeb: Add position tracking information to HTML tokens
2021-05-20 23:11:41 +04:30
LibWeb: Set consistent positions for the start and end of HTML tags To illustrate the previous behavior, consider these tags and their start and end positions (shown inclusively below): Start tag: End tag: <span> </span> ^ start ^ start ^end ^end The start position of a tag is the first ASCII-alpha code point after the opening brace. The start position of a close tag is the slash just before the first ASCII-alpha code point. And the end position of both is the closing brace. So the opening brace is not included in the emitted tag, but the closing brace is. And the end tag including the slash is an oddity that had to be worked around in its only use case (syntax highlighting). We now consistently exclude the braces from the emitted tag, and also exclude the slash from the end tag, so that it does not need to be accounted for in syntax highlighting. That is, we now have: Start tag: End tag: <span> </span> ^ start ^ start ^end ^end The tokenizer unit test has been extended to test these positions.
2023-08-24 17:01:19 -04:00
auto is_start_or_end_tag = type == HTMLToken::Type::StartTag || type == HTMLToken::Type::EndTag;
m_current_token.set_start_position({}, nth_last_position(is_start_or_end_tag ? 1 : 0));
LibWeb: Begin work on a spec-compliant HTML parser In order to actually view the web as it is, we're gonna need a proper HTML parser. So let's build one! This patch introduces the Web::HTMLTokenizer class, which currently operates on a StringView input stream where it fetches (ASCII only atm) codepoints and tokenizes acccording to the HTML spec tokenization algo. The tokenizer state machine looks a bit weird but is written in a way that tries to mimic the spec as closely as possible, in order to make development easier and bugs less likely. This initial version is far from finished, but it can parse a trivial document with a DOCTYPE and open/close tags. :^)
2020-05-22 21:46:13 +02:00
}
LibWeb: Add basic support for dynamic markup insertion This implements basic support for dynamic markup insertion, adding * Document::open() * Document::write(Vector<String> const&) * Document::writeln(Vector<String> const&) * Document::close() The HTMLParser is modified to make it possible to create a script-created parser which initially only contains a HTMLTokenizer without any data. Aditionally the HTMLParser::run method gains an overload which does not modify the Document and does not run HTMLParser::the_end() so that we can reenter the parser at a later time. Furthermore all FIXMEs that consern the insertion point are implemented wich is defined in the HTMLTokenizer. Additionally the following member-variables of the HTMLParser are now exposed by getter funcions: * m_tokenizer * m_aborted * m_script_nesting_level The HTMLTokenizer is modified so that it contains an insertion point which keeps track of where the next input from the Document::write functions will be inserted. The insertion point is implemented as the charakter offset into m_decoded_input and a boolean describing if the insertion point is defined. Functions to update, check and {re}store the insertion point are also added. The function HTMLTokenizer::insert_eof is added to tell a script-created parser that document::close was called and HTMLParser::the_end() should be called. Lastly an explicit default constructor is added to HTMLTokenizer to create a empty HTMLTokenizer into which data can be inserted.
2022-02-19 15:58:21 +01:00
HTMLTokenizer::HTMLTokenizer()
{
LibWeb: Let HTMLTokenizer walk over code points instead of UTF-8 Instead of using UTF-8 iterators to traverse the HTMLTokenizer input stream one code point at a time, we now do a one-shot conversion up front from the input encoding to a Vector<u32> of Unicode code points. This simplifies the tokenizer logic somewhat, and ends up being faster as well, so win-win. 1.02x speedup on Speedometer 2.1
2025-05-10 11:28:35 +02:00
m_decoded_input = {};
m_current_offset = 0;
m_prev_offset = 0;
LibWeb: Add basic support for dynamic markup insertion This implements basic support for dynamic markup insertion, adding * Document::open() * Document::write(Vector<String> const&) * Document::writeln(Vector<String> const&) * Document::close() The HTMLParser is modified to make it possible to create a script-created parser which initially only contains a HTMLTokenizer without any data. Aditionally the HTMLParser::run method gains an overload which does not modify the Document and does not run HTMLParser::the_end() so that we can reenter the parser at a later time. Furthermore all FIXMEs that consern the insertion point are implemented wich is defined in the HTMLTokenizer. Additionally the following member-variables of the HTMLParser are now exposed by getter funcions: * m_tokenizer * m_aborted * m_script_nesting_level The HTMLTokenizer is modified so that it contains an insertion point which keeps track of where the next input from the Document::write functions will be inserted. The insertion point is implemented as the charakter offset into m_decoded_input and a boolean describing if the insertion point is defined. Functions to update, check and {re}store the insertion point are also added. The function HTMLTokenizer::insert_eof is added to tell a script-created parser that document::close was called and HTMLParser::the_end() should be called. Lastly an explicit default constructor is added to HTMLTokenizer to create a empty HTMLTokenizer into which data can be inserted.
2022-02-19 15:58:21 +01:00
m_source_positions.empend(0u, 0u);
}
Everywhere: Rename {Deprecated => Byte}String This commit un-deprecates DeprecatedString, and repurposes it as a byte string. As the null state has already been removed, there are no other particularly hairy blockers in repurposing this type as a byte string (what it _really_ is). This commit is auto-generated: $ xs=$(ack -l \bDeprecatedString\b\|deprecated_string AK Userland \ Meta Ports Ladybird Tests Kernel) $ perl -pie 's/\bDeprecatedString\b/ByteString/g; s/deprecated_string/byte_string/g' $xs $ clang-format --style=file -i \ $(git diff --name-only | grep \.cpp\|\.h) $ gn format $(git ls-files '*.gn' '*.gni')
2023-12-16 17:49:34 +03:30
HTMLTokenizer::HTMLTokenizer(StringView input, ByteString const& encoding)
LibWeb: Begin work on a spec-compliant HTML parser In order to actually view the web as it is, we're gonna need a proper HTML parser. So let's build one! This patch introduces the Web::HTMLTokenizer class, which currently operates on a StringView input stream where it fetches (ASCII only atm) codepoints and tokenizes acccording to the HTML spec tokenization algo. The tokenizer state machine looks a bit weird but is written in a way that tries to mimic the spec as closely as possible, in order to make development easier and bugs less likely. This initial version is far from finished, but it can parse a trivial document with a DOCTYPE and open/close tags. :^)
2020-05-22 21:46:13 +02:00
{
LibTextCodec+Everywhere: Return Optional<Decoder&> from `decoder_for()`
2023-02-17 17:45:08 +00:00
auto decoder = TextCodec::decoder_for(encoding);
VERIFY(decoder.has_value());
LibWeb: Let HTMLTokenizer walk over code points instead of UTF-8 Instead of using UTF-8 iterators to traverse the HTMLTokenizer input stream one code point at a time, we now do a one-shot conversion up front from the input encoding to a Vector<u32> of Unicode code points. This simplifies the tokenizer logic somewhat, and ends up being faster as well, so win-win. 1.02x speedup on Speedometer 2.1
2025-05-10 11:28:35 +02:00
m_source = MUST(decoder->to_utf8(input));
m_decoded_input.ensure_capacity(m_source.bytes().size());
for (auto code_point : m_source.code_points())
m_decoded_input.append(code_point);
m_current_offset = 0;
m_prev_offset = 0;
LibWeb: Add position tracking information to HTML tokens
2021-05-20 23:11:41 +04:30
m_source_positions.empend(0u, 0u);
LibWeb: Begin work on a spec-compliant HTML parser In order to actually view the web as it is, we're gonna need a proper HTML parser. So let's build one! This patch introduces the Web::HTMLTokenizer class, which currently operates on a StringView input stream where it fetches (ASCII only atm) codepoints and tokenizes acccording to the HTML spec tokenization algo. The tokenizer state machine looks a bit weird but is written in a way that tries to mimic the spec as closely as possible, in order to make development easier and bugs less likely. This initial version is far from finished, but it can parse a trivial document with a DOCTYPE and open/close tags. :^)
2020-05-22 21:46:13 +02:00
}
LibWeb: Make Document::run_the_document_write_steps take a StringView Which flows on down into HTMLTokenizer::insert_input_at_insertion_point.
2023-09-12 23:16:10 +12:00
void HTMLTokenizer::insert_input_at_insertion_point(StringView input)
LibWeb: Add basic support for dynamic markup insertion This implements basic support for dynamic markup insertion, adding * Document::open() * Document::write(Vector<String> const&) * Document::writeln(Vector<String> const&) * Document::close() The HTMLParser is modified to make it possible to create a script-created parser which initially only contains a HTMLTokenizer without any data. Aditionally the HTMLParser::run method gains an overload which does not modify the Document and does not run HTMLParser::the_end() so that we can reenter the parser at a later time. Furthermore all FIXMEs that consern the insertion point are implemented wich is defined in the HTMLTokenizer. Additionally the following member-variables of the HTMLParser are now exposed by getter funcions: * m_tokenizer * m_aborted * m_script_nesting_level The HTMLTokenizer is modified so that it contains an insertion point which keeps track of where the next input from the Document::write functions will be inserted. The insertion point is implemented as the charakter offset into m_decoded_input and a boolean describing if the insertion point is defined. Functions to update, check and {re}store the insertion point are also added. The function HTMLTokenizer::insert_eof is added to tell a script-created parser that document::close was called and HTMLParser::the_end() should be called. Lastly an explicit default constructor is added to HTMLTokenizer to create a empty HTMLTokenizer into which data can be inserted.
2022-02-19 15:58:21 +01:00
{
LibWeb: Let HTMLTokenizer walk over code points instead of UTF-8 Instead of using UTF-8 iterators to traverse the HTMLTokenizer input stream one code point at a time, we now do a one-shot conversion up front from the input encoding to a Vector<u32> of Unicode code points. This simplifies the tokenizer logic somewhat, and ends up being faster as well, so win-win. 1.02x speedup on Speedometer 2.1
2025-05-10 11:28:35 +02:00
Vector<u32> new_decoded_input;
new_decoded_input.ensure_capacity(m_decoded_input.size() + input.length());
LibWeb: Add basic support for dynamic markup insertion This implements basic support for dynamic markup insertion, adding * Document::open() * Document::write(Vector<String> const&) * Document::writeln(Vector<String> const&) * Document::close() The HTMLParser is modified to make it possible to create a script-created parser which initially only contains a HTMLTokenizer without any data. Aditionally the HTMLParser::run method gains an overload which does not modify the Document and does not run HTMLParser::the_end() so that we can reenter the parser at a later time. Furthermore all FIXMEs that consern the insertion point are implemented wich is defined in the HTMLTokenizer. Additionally the following member-variables of the HTMLParser are now exposed by getter funcions: * m_tokenizer * m_aborted * m_script_nesting_level The HTMLTokenizer is modified so that it contains an insertion point which keeps track of where the next input from the Document::write functions will be inserted. The insertion point is implemented as the charakter offset into m_decoded_input and a boolean describing if the insertion point is defined. Functions to update, check and {re}store the insertion point are also added. The function HTMLTokenizer::insert_eof is added to tell a script-created parser that document::close was called and HTMLParser::the_end() should be called. Lastly an explicit default constructor is added to HTMLTokenizer to create a empty HTMLTokenizer into which data can be inserted.
2022-02-19 15:58:21 +01:00
LibWeb: Let HTMLTokenizer walk over code points instead of UTF-8 Instead of using UTF-8 iterators to traverse the HTMLTokenizer input stream one code point at a time, we now do a one-shot conversion up front from the input encoding to a Vector<u32> of Unicode code points. This simplifies the tokenizer logic somewhat, and ends up being faster as well, so win-win. 1.02x speedup on Speedometer 2.1
2025-05-10 11:28:35 +02:00
auto before = m_decoded_input.span().slice(0, m_insertion_point.position);
new_decoded_input.append(before.data(), before.size());
auto utf8_to_insert = MUST(String::from_utf8(input));
ssize_t code_points_inserted = 0;
for (auto code_point : utf8_to_insert.code_points()) {
new_decoded_input.append(code_point);
++code_points_inserted;
}
LibWeb: Add basic support for dynamic markup insertion This implements basic support for dynamic markup insertion, adding * Document::open() * Document::write(Vector<String> const&) * Document::writeln(Vector<String> const&) * Document::close() The HTMLParser is modified to make it possible to create a script-created parser which initially only contains a HTMLTokenizer without any data. Aditionally the HTMLParser::run method gains an overload which does not modify the Document and does not run HTMLParser::the_end() so that we can reenter the parser at a later time. Furthermore all FIXMEs that consern the insertion point are implemented wich is defined in the HTMLTokenizer. Additionally the following member-variables of the HTMLParser are now exposed by getter funcions: * m_tokenizer * m_aborted * m_script_nesting_level The HTMLTokenizer is modified so that it contains an insertion point which keeps track of where the next input from the Document::write functions will be inserted. The insertion point is implemented as the charakter offset into m_decoded_input and a boolean describing if the insertion point is defined. Functions to update, check and {re}store the insertion point are also added. The function HTMLTokenizer::insert_eof is added to tell a script-created parser that document::close was called and HTMLParser::the_end() should be called. Lastly an explicit default constructor is added to HTMLTokenizer to create a empty HTMLTokenizer into which data can be inserted.
2022-02-19 15:58:21 +01:00
LibWeb: Let HTMLTokenizer walk over code points instead of UTF-8 Instead of using UTF-8 iterators to traverse the HTMLTokenizer input stream one code point at a time, we now do a one-shot conversion up front from the input encoding to a Vector<u32> of Unicode code points. This simplifies the tokenizer logic somewhat, and ends up being faster as well, so win-win. 1.02x speedup on Speedometer 2.1
2025-05-10 11:28:35 +02:00
auto after = m_decoded_input.span().slice(m_insertion_point.position);
new_decoded_input.append(after.data(), after.size());
m_decoded_input = move(new_decoded_input);
LibWeb: Add basic support for dynamic markup insertion This implements basic support for dynamic markup insertion, adding * Document::open() * Document::write(Vector<String> const&) * Document::writeln(Vector<String> const&) * Document::close() The HTMLParser is modified to make it possible to create a script-created parser which initially only contains a HTMLTokenizer without any data. Aditionally the HTMLParser::run method gains an overload which does not modify the Document and does not run HTMLParser::the_end() so that we can reenter the parser at a later time. Furthermore all FIXMEs that consern the insertion point are implemented wich is defined in the HTMLTokenizer. Additionally the following member-variables of the HTMLParser are now exposed by getter funcions: * m_tokenizer * m_aborted * m_script_nesting_level The HTMLTokenizer is modified so that it contains an insertion point which keeps track of where the next input from the Document::write functions will be inserted. The insertion point is implemented as the charakter offset into m_decoded_input and a boolean describing if the insertion point is defined. Functions to update, check and {re}store the insertion point are also added. The function HTMLTokenizer::insert_eof is added to tell a script-created parser that document::close was called and HTMLParser::the_end() should be called. Lastly an explicit default constructor is added to HTMLTokenizer to create a empty HTMLTokenizer into which data can be inserted.
2022-02-19 15:58:21 +01:00
LibWeb: Let HTMLTokenizer walk over code points instead of UTF-8 Instead of using UTF-8 iterators to traverse the HTMLTokenizer input stream one code point at a time, we now do a one-shot conversion up front from the input encoding to a Vector<u32> of Unicode code points. This simplifies the tokenizer logic somewhat, and ends up being faster as well, so win-win. 1.02x speedup on Speedometer 2.1
2025-05-10 11:28:35 +02:00
m_insertion_point.position += code_points_inserted;
LibWeb: Add basic support for dynamic markup insertion This implements basic support for dynamic markup insertion, adding * Document::open() * Document::write(Vector<String> const&) * Document::writeln(Vector<String> const&) * Document::close() The HTMLParser is modified to make it possible to create a script-created parser which initially only contains a HTMLTokenizer without any data. Aditionally the HTMLParser::run method gains an overload which does not modify the Document and does not run HTMLParser::the_end() so that we can reenter the parser at a later time. Furthermore all FIXMEs that consern the insertion point are implemented wich is defined in the HTMLTokenizer. Additionally the following member-variables of the HTMLParser are now exposed by getter funcions: * m_tokenizer * m_aborted * m_script_nesting_level The HTMLTokenizer is modified so that it contains an insertion point which keeps track of where the next input from the Document::write functions will be inserted. The insertion point is implemented as the charakter offset into m_decoded_input and a boolean describing if the insertion point is defined. Functions to update, check and {re}store the insertion point are also added. The function HTMLTokenizer::insert_eof is added to tell a script-created parser that document::close was called and HTMLParser::the_end() should be called. Lastly an explicit default constructor is added to HTMLTokenizer to create a empty HTMLTokenizer into which data can be inserted.
2022-02-19 15:58:21 +01:00
}
void HTMLTokenizer::insert_eof()
{
m_explicit_eof_inserted = true;
}
bool HTMLTokenizer::is_eof_inserted()
{
return m_explicit_eof_inserted;
}
LibWeb: Begin work on a spec-compliant HTML parser In order to actually view the web as it is, we're gonna need a proper HTML parser. So let's build one! This patch introduces the Web::HTMLTokenizer class, which currently operates on a StringView input stream where it fetches (ASCII only atm) codepoints and tokenizes acccording to the HTML spec tokenization algo. The tokenizer state machine looks a bit weird but is written in a way that tries to mimic the spec as closely as possible, in order to make development easier and bugs less likely. This initial version is far from finished, but it can parse a trivial document with a DOCTYPE and open/close tags. :^)
2020-05-22 21:46:13 +02:00
void HTMLTokenizer::will_switch_to([[maybe_unused]] State new_state)
{
Everywhere: Replace dbgln<flag>(...) with dbgln_if(flag, ...) Replacement made by `find Kernel Userland -name '*.h' -o -name '*.cpp' | sed -i -Ee 's/dbgln\b<(\w+)>\(/dbgln_if(\1, /g'`
2021-02-07 15:33:24 +03:30
dbgln_if(TOKENIZER_TRACE_DEBUG, "[{}] Switch to {}", state_name(m_state), state_name(new_state));
LibWeb: Begin work on a spec-compliant HTML parser In order to actually view the web as it is, we're gonna need a proper HTML parser. So let's build one! This patch introduces the Web::HTMLTokenizer class, which currently operates on a StringView input stream where it fetches (ASCII only atm) codepoints and tokenizes acccording to the HTML spec tokenization algo. The tokenizer state machine looks a bit weird but is written in a way that tries to mimic the spec as closely as possible, in order to make development easier and bugs less likely. This initial version is far from finished, but it can parse a trivial document with a DOCTYPE and open/close tags. :^)
2020-05-22 21:46:13 +02:00
}
void HTMLTokenizer::will_reconsume_in([[maybe_unused]] State new_state)
{
Everywhere: Replace dbgln<flag>(...) with dbgln_if(flag, ...) Replacement made by `find Kernel Userland -name '*.h' -o -name '*.cpp' | sed -i -Ee 's/dbgln\b<(\w+)>\(/dbgln_if(\1, /g'`
2021-02-07 15:33:24 +03:30
dbgln_if(TOKENIZER_TRACE_DEBUG, "[{}] Reconsume in {}", state_name(m_state), state_name(new_state));
LibWeb: Begin work on a spec-compliant HTML parser In order to actually view the web as it is, we're gonna need a proper HTML parser. So let's build one! This patch introduces the Web::HTMLTokenizer class, which currently operates on a StringView input stream where it fetches (ASCII only atm) codepoints and tokenizes acccording to the HTML spec tokenization algo. The tokenizer state machine looks a bit weird but is written in a way that tries to mimic the spec as closely as possible, in order to make development easier and bugs less likely. This initial version is far from finished, but it can parse a trivial document with a DOCTYPE and open/close tags. :^)
2020-05-22 21:46:13 +02:00
}
LibWeb: Rename HTMLDocumentParser => HTMLParser
2021-09-25 23:15:48 +02:00
void HTMLTokenizer::switch_to(Badge<HTMLParser>, State new_state)
LibWeb: More work on the HTML parser and tokenizer The parser can now switch the state of the tokenizer! Very webby. :^)
2020-05-24 20:24:43 +02:00
{
Everywhere: Replace dbgln<flag>(...) with dbgln_if(flag, ...) Replacement made by `find Kernel Userland -name '*.h' -o -name '*.cpp' | sed -i -Ee 's/dbgln\b<(\w+)>\(/dbgln_if(\1, /g'`
2021-02-07 15:33:24 +03:30
dbgln_if(TOKENIZER_TRACE_DEBUG, "[{}] Parser switches tokenizer state to {}", state_name(m_state), state_name(new_state));
LibWeb: More work on the HTML parser and tokenizer The parser can now switch the state of the tokenizer! Very webby. :^)
2020-05-24 20:24:43 +02:00
m_state = new_state;
}
void HTMLTokenizer::will_emit(HTMLToken& token)
{
if (token.is_start_tag())
LibJS+LibWeb: Remove more uses of DeprecatedFlyString
2025-03-24 11:27:27 +00:00
m_last_emitted_start_tag_name = token.tag_name();
LibWeb: Set consistent positions for the start and end of HTML tags To illustrate the previous behavior, consider these tags and their start and end positions (shown inclusively below): Start tag: End tag: <span> </span> ^ start ^ start ^end ^end The start position of a tag is the first ASCII-alpha code point after the opening brace. The start position of a close tag is the slash just before the first ASCII-alpha code point. And the end position of both is the closing brace. So the opening brace is not included in the emitted tag, but the closing brace is. And the end tag including the slash is an oddity that had to be worked around in its only use case (syntax highlighting). We now consistently exclude the braces from the emitted tag, and also exclude the slash from the end tag, so that it does not need to be accounted for in syntax highlighting. That is, we now have: Start tag: End tag: <span> </span> ^ start ^ start ^end ^end The tokenizer unit test has been extended to test these positions.
2023-08-24 17:01:19 -04:00
auto is_start_or_end_tag = token.type() == HTMLToken::Type::StartTag || token.type() == HTMLToken::Type::EndTag;
token.set_end_position({}, nth_last_position(is_start_or_end_tag ? 1 : 0));
LibWeb: Deduplicate attributes when emitting start and end tags The HTML tokenizer specification says that we're supposed to do this when leaving the Attribute name or when emitting the token, as appropriate. Hopefully 'as appropriate' can mean only when emitting the token, as that's the easiest place to insert this logic without complicating the tokenizer any more.
2024-09-30 17:52:30 -06:00
if (is_start_or_end_tag)
token.normalize_attributes();
LibWeb: More work on the HTML parser and tokenizer The parser can now switch the state of the tokenizer! Very webby. :^)
2020-05-24 20:24:43 +02:00
}
bool HTMLTokenizer::current_end_tag_token_is_appropriate() const
{
Everywhere: Rename ASSERT => VERIFY (...and ASSERT_NOT_REACHED => VERIFY_NOT_REACHED) Since all of these checks are done in release builds as well, let's rename them to VERIFY to prevent confusion, as everyone is used to assertions being compiled out in release. We can introduce a new ASSERT macro that is specifically for debug checks, but I'm doing this wholesale conversion first since we've accumulated thousands of these already, and it's not immediately obvious which ones are suitable for ASSERT.
2021-02-23 20:42:32 +01:00
VERIFY(m_current_token.is_end_tag());
LibWeb: Use an Optional<String> to track the last HTML start tag Using an HTMLToken object here is unnecessary because the only attribute we're interested in is the tag_name.
2021-05-22 20:17:09 +02:00
if (!m_last_emitted_start_tag_name.has_value())
LibWeb: More work on the HTML parser and tokenizer The parser can now switch the state of the tokenizer! Very webby. :^)
2020-05-24 20:24:43 +02:00
return false;
LibJS+LibWeb: Remove more uses of DeprecatedFlyString
2025-03-24 11:27:27 +00:00
return m_current_token.tag_name() == m_last_emitted_start_tag_name.value();
LibWeb: More work on the HTML parser and tokenizer The parser can now switch the state of the tokenizer! Very webby. :^)
2020-05-24 20:24:43 +02:00
}
LibWeb: Implement partial support for numeric character references
2020-05-27 16:16:23 +02:00
bool HTMLTokenizer::consumed_as_part_of_an_attribute() const
{
return m_return_state == State::AttributeValueUnquoted || m_return_state == State::AttributeValueSingleQuoted || m_return_state == State::AttributeValueDoubleQuoted;
}
LibWeb: Let HTMLTokenizer walk over code points instead of UTF-8 Instead of using UTF-8 iterators to traverse the HTMLTokenizer input stream one code point at a time, we now do a one-shot conversion up front from the input encoding to a Vector<u32> of Unicode code points. This simplifies the tokenizer logic somewhat, and ends up being faster as well, so win-win. 1.02x speedup on Speedometer 2.1
2025-05-10 11:28:35 +02:00
void HTMLTokenizer::restore_to(ssize_t new_iterator)
LibWeb: Add position tracking information to HTML tokens
2021-05-20 23:11:41 +04:30
{
LibWeb: Let HTMLTokenizer walk over code points instead of UTF-8 Instead of using UTF-8 iterators to traverse the HTMLTokenizer input stream one code point at a time, we now do a one-shot conversion up front from the input encoding to a Vector<u32> of Unicode code points. This simplifies the tokenizer logic somewhat, and ends up being faster as well, so win-win. 1.02x speedup on Speedometer 2.1
2025-05-10 11:28:35 +02:00
auto diff = m_current_offset - new_iterator;
LibWeb: Fix off-by-one in HTMLTokenizer::restore_to() The difference should be between m_utf8_iterator and the the new position, if m_prev_utf8_iterator is used one fewer source position is popped than required. This issue was not apparent on most pages since restore_to used for tokens such <!doctype> that are normally followed by a newline that resets the column to zero, but it can be seen on pages with minified HTML.
2022-02-13 14:08:53 +00:00
if (diff > 0) {
LibWeb: Fix Array OOBs in the HTMLTokenizer Accessing last() if there are no elements makes WebContent crash :^)
2022-06-02 01:03:44 +02:00
for (ssize_t i = 0; i < diff; ++i) {
if (!m_source_positions.is_empty())
m_source_positions.take_last();
}
LibWeb: Fix off-by-one in HTMLTokenizer::restore_to() The difference should be between m_utf8_iterator and the the new position, if m_prev_utf8_iterator is used one fewer source position is popped than required. This issue was not apparent on most pages since restore_to used for tokens such <!doctype> that are normally followed by a newline that resets the column to zero, but it can be seen on pages with minified HTML.
2022-02-13 14:08:53 +00:00
} else {
// Going forwards...?
TODO();
LibWeb: Add position tracking information to HTML tokens
2021-05-20 23:11:41 +04:30
}
LibWeb: Let HTMLTokenizer walk over code points instead of UTF-8 Instead of using UTF-8 iterators to traverse the HTMLTokenizer input stream one code point at a time, we now do a one-shot conversion up front from the input encoding to a Vector<u32> of Unicode code points. This simplifies the tokenizer logic somewhat, and ends up being faster as well, so win-win. 1.02x speedup on Speedometer 2.1
2025-05-10 11:28:35 +02:00
m_current_offset = new_iterator;
LibWeb: Add position tracking information to HTML tokens
2021-05-20 23:11:41 +04:30
}
LibWeb: Switch HTMLToken and HTMLTokenizer to String & FlyString
2023-11-04 10:08:07 +01:00
String HTMLTokenizer::consume_current_builder()
LibWeb: Remove StringBuilders from HTMLToken::m_doctype
2021-05-23 08:20:03 +02:00
{
LibWeb: Avoid redundant UTF-8 validation in HTML tokenizer
2024-04-21 14:45:49 +02:00
auto string = m_current_builder.to_string_without_validation();
LibWeb: Remove StringBuilders from HTMLToken::m_doctype
2021-05-23 08:20:03 +02:00
m_current_builder.clear();
return string;
}
LibWeb: Begin work on a spec-compliant HTML parser In order to actually view the web as it is, we're gonna need a proper HTML parser. So let's build one! This patch introduces the Web::HTMLTokenizer class, which currently operates on a StringView input stream where it fetches (ASCII only atm) codepoints and tokenizes acccording to the HTML spec tokenization algo. The tokenizer state machine looks a bit weird but is written in a way that tries to mimic the spec as closely as possible, in order to make development easier and bugs less likely. This initial version is far from finished, but it can parse a trivial document with a DOCTYPE and open/close tags. :^)
2020-05-22 21:46:13 +02:00
}
Reference in a new issue Copy permalink