Stowage/ladybird
2
0
Fork 0
mirror of https://github.com/LadybirdBrowser/ladybird.git synced 2025-10-20 08:03:21 +00:00
Code Issues Projects Releases Packages Wiki Activity
ladybird/Userland/Libraries/LibWeb/HTML/Parser/HTMLTokenizer.cpp

2893 lines
110 KiB
C++
Raw Normal View History

LibWeb: Begin work on a spec-compliant HTML parser In order to actually view the web as it is, we're gonna need a proper HTML parser. So let's build one! This patch introduces the Web::HTMLTokenizer class, which currently operates on a StringView input stream where it fetches (ASCII only atm) codepoints and tokenizes acccording to the HTML spec tokenization algo. The tokenizer state machine looks a bit weird but is written in a way that tries to mimic the spec as closely as possible, in order to make development easier and bugs less likely. This initial version is far from finished, but it can parse a trivial document with a DOCTYPE and open/close tags. :^)
2020-05-22 21:46:13 +02:00
/*
* Copyright (c) 2020, Andreas Kling <kling@serenityos.org>
LibWeb: Implement state switch for "[CDATA[" in HTML parser
2022-02-15 18:52:45 +00:00
* Copyright (c) 2022, Linus Groh <linusg@serenityos.org>
LibWeb: Begin work on a spec-compliant HTML parser In order to actually view the web as it is, we're gonna need a proper HTML parser. So let's build one! This patch introduces the Web::HTMLTokenizer class, which currently operates on a StringView input stream where it fetches (ASCII only atm) codepoints and tokenizes acccording to the HTML spec tokenization algo. The tokenizer state machine looks a bit weird but is written in a way that tries to mimic the spec as closely as possible, in order to make development easier and bugs less likely. This initial version is far from finished, but it can parse a trivial document with a DOCTYPE and open/close tags. :^)
2020-05-22 21:46:13 +02:00
*
Everything: Move to SPDX license identifiers in all files. SPDX License Identifiers are a more compact / standardized way of representing file license information. See: https://spdx.dev/resources/use/#identifiers This was done with the `ambr` search and replace tool. ambr --no-parent-ignore --key-from-file --rep-from-file key.txt rep.txt *
2021-04-22 01:24:48 -07:00
* SPDX-License-Identifier: BSD-2-Clause
LibWeb: Begin work on a spec-compliant HTML parser In order to actually view the web as it is, we're gonna need a proper HTML parser. So let's build one! This patch introduces the Web::HTMLTokenizer class, which currently operates on a StringView input stream where it fetches (ASCII only atm) codepoints and tokenizes acccording to the HTML spec tokenization algo. The tokenizer state machine looks a bit weird but is written in a way that tries to mimic the spec as closely as possible, in order to make development easier and bugs less likely. This initial version is far from finished, but it can parse a trivial document with a DOCTYPE and open/close tags. :^)
2020-05-22 21:46:13 +02:00
*/
Everywhere: Replace ctype.h to avoid narrowing conversions This replaces ctype.h with CharacterType.h everywhere I could find issues with narrowing conversions. While using it will probably make sense almost everywhere in the future, the most critical places should have been addressed.
2021-06-01 21:18:08 +02:00
#include <AK/CharacterTypes.h>
Everywhere: Replace a bundle of dbg with dbgln. These changes are arbitrarily divided into multiple commits to make it easier to find potentially introduced bugs with git bisect.
2021-01-17 16:57:17 +01:00
#include <AK/Debug.h>
LibWeb: Utilize SourceLocation for HTMLTokenizer logging
2021-04-24 20:22:30 -07:00
#include <AK/SourceLocation.h>
LibWeb: Plumb content encoding into the new HTML parser We still don't handle non-ASCII input correctly, but at least now we'll convert e.g ISO-8859-1 to UTF-8 before starting to tokenize. This patch also makes "view source" work with the new parser. :^)
2020-05-28 12:35:19 +02:00
#include <LibTextCodec/Decoder.h>
LibWeb: Move the HTML parser into HTML/Parser/
2020-07-28 19:18:23 +02:00
#include <LibWeb/HTML/Parser/Entities.h>
LibWeb: Implement state switch for "[CDATA[" in HTML parser
2022-02-15 18:52:45 +00:00
#include <LibWeb/HTML/Parser/HTMLParser.h>
LibWeb: Move the HTML parser into HTML/Parser/
2020-07-28 19:18:23 +02:00
#include <LibWeb/HTML/Parser/HTMLToken.h>
#include <LibWeb/HTML/Parser/HTMLTokenizer.h>
LibWeb: Implement state switch for "[CDATA[" in HTML parser
2022-02-15 18:52:45 +00:00
#include <LibWeb/Namespace.h>
LibWeb: In the HTML tokenizer, pretty up ON_WHITESPACE a tiny bit No behavior change.
2020-10-09 11:25:10 -04:00
#include <string.h>
LibWeb: Begin work on a spec-compliant HTML parser In order to actually view the web as it is, we're gonna need a proper HTML parser. So let's build one! This patch introduces the Web::HTMLTokenizer class, which currently operates on a StringView input stream where it fetches (ASCII only atm) codepoints and tokenizes acccording to the HTML spec tokenization algo. The tokenizer state machine looks a bit weird but is written in a way that tries to mimic the spec as closely as possible, in order to make development easier and bugs less likely. This initial version is far from finished, but it can parse a trivial document with a DOCTYPE and open/close tags. :^)
2020-05-22 21:46:13 +02:00
LibWeb: Move HTML classes into the Web::HTML namespace
2020-07-28 18:20:36 +02:00
namespace Web::HTML {
LibWeb: Start building the tree building part of the new HTML parser This patch adds a new HTMLDocumentParser class. It keeps a tokenizer object internally and feeds itself with one token at a time from it. The names and idioms in this class are expressed as closely to the actual HTML parsing spec as possible, to make development as easy and bug free as possible. :^) This is going to become pretty large, but it's pretty cool!
2020-05-24 00:14:23 +02:00
#pragma GCC diagnostic ignored "-Wunused-label"
LibWeb: Implement partial support for numeric character references
2020-05-27 16:16:23 +02:00
#define CONSUME_NEXT_INPUT_CHARACTER \
Unicode: Try s/codepoint/code_point/g again This time, without trailing 's'. Ran: git grep -l 'codepoint' | xargs sed -ie 's/codepoint/code_point/g
2020-08-05 16:31:20 -04:00
current_input_character = next_code_point();
LibWeb: Implement partial support for numeric character references
2020-05-27 16:16:23 +02:00
LibWeb: Remove StringBuilders from HTMLToken::m_doctype
2021-05-23 08:20:03 +02:00
#define SWITCH_TO(new_state) \
do { \
VERIFY(m_current_builder.is_empty()); \
SWITCH_TO_WITH_UNCLEAN_BUILDER(new_state); \
} while (0)
#define SWITCH_TO_WITH_UNCLEAN_BUILDER(new_state) \
do { \
will_switch_to(State::new_state); \
m_state = State::new_state; \
CONSUME_NEXT_INPUT_CHARACTER; \
goto new_state; \
LibWeb: A whole bunch of work towards spec-compliant <script> elements This is still very unfinished, but there's at least a skeleton of code.
2020-05-24 22:00:46 +02:00
} while (0)
LibWeb: Begin work on a spec-compliant HTML parser In order to actually view the web as it is, we're gonna need a proper HTML parser. So let's build one! This patch introduces the Web::HTMLTokenizer class, which currently operates on a StringView input stream where it fetches (ASCII only atm) codepoints and tokenizes acccording to the HTML spec tokenization algo. The tokenizer state machine looks a bit weird but is written in a way that tries to mimic the spec as closely as possible, in order to make development easier and bugs less likely. This initial version is far from finished, but it can parse a trivial document with a DOCTYPE and open/close tags. :^)
2020-05-22 21:46:13 +02:00
LibWeb: A whole bunch of work towards spec-compliant <script> elements This is still very unfinished, but there's at least a skeleton of code.
2020-05-24 22:00:46 +02:00
#define RECONSUME_IN(new_state) \
do { \
will_reconsume_in(State::new_state); \
m_state = State::new_state; \
goto new_state; \
} while (0)
LibWeb: Begin work on a spec-compliant HTML parser In order to actually view the web as it is, we're gonna need a proper HTML parser. So let's build one! This patch introduces the Web::HTMLTokenizer class, which currently operates on a StringView input stream where it fetches (ASCII only atm) codepoints and tokenizes acccording to the HTML spec tokenization algo. The tokenizer state machine looks a bit weird but is written in a way that tries to mimic the spec as closely as possible, in order to make development easier and bugs less likely. This initial version is far from finished, but it can parse a trivial document with a DOCTYPE and open/close tags. :^)
2020-05-22 21:46:13 +02:00
LibWeb: Implement partial support for numeric character references
2020-05-27 16:16:23 +02:00
#define SWITCH_TO_RETURN_STATE \
do { \
will_switch_to(m_return_state); \
m_state = m_return_state; \
goto _StartOfFunction; \
} while (0)
LibWeb: Add position tracking information to HTML tokens
2021-05-20 23:11:41 +04:30
#define RECONSUME_IN_RETURN_STATE \
do { \
will_reconsume_in(m_return_state); \
m_state = m_return_state; \
if (current_input_character.has_value()) \
restore_to(m_prev_utf8_iterator); \
goto _StartOfFunction; \
LibWeb: Implement partial support for numeric character references
2020-05-27 16:16:23 +02:00
} while (0)
LibWeb: Use move() when enqueuing tokens in HTMLTokenizer We're not using the current token anymore once it's enqueued so let's use move() when enqueuing the tokens.
2021-05-23 12:39:00 +02:00
#define SWITCH_TO_AND_EMIT_CURRENT_TOKEN(new_state) \
do { \
LibWeb: VERIFY an empty builder when emitting tokens in HTMLTokenizer
2021-07-15 00:34:54 +02:00
VERIFY(m_current_builder.is_empty()); \
LibWeb: Use move() when enqueuing tokens in HTMLTokenizer We're not using the current token anymore once it's enqueued so let's use move() when enqueuing the tokens.
2021-05-23 12:39:00 +02:00
will_switch_to(State::new_state); \
m_state = State::new_state; \
will_emit(m_current_token); \
m_queued_tokens.enqueue(move(m_current_token)); \
return m_queued_tokens.dequeue(); \
LibWeb: Allow HTML tokenizer to emit more than one token Tokens are now put on a queue when emitted, and we always pop from that queue when returning from next_token().
2020-05-26 15:50:05 +02:00
} while (0)
Unicode: Try s/codepoint/code_point/g again This time, without trailing 's'. Ran: git grep -l 'codepoint' | xargs sed -ie 's/codepoint/code_point/g
2020-08-05 16:31:20 -04:00
#define EMIT_CHARACTER_AND_RECONSUME_IN(code_point, new_state) \
Meta+LibHTTP through LibWeb: Make clang-format-10 clean
2020-09-18 09:49:51 +02:00
do { \
Unicode: Try s/codepoint/code_point/g again This time, without trailing 's'. Ran: git grep -l 'codepoint' | xargs sed -ie 's/codepoint/code_point/g
2020-08-05 16:31:20 -04:00
m_queued_tokens.enqueue(HTMLToken::make_character(code_point)); \
Meta+LibHTTP through LibWeb: Make clang-format-10 clean
2020-09-18 09:49:51 +02:00
will_reconsume_in(State::new_state); \
m_state = State::new_state; \
goto new_state; \
LibWeb: A whole bunch of work towards spec-compliant <script> elements This is still very unfinished, but there's at least a skeleton of code.
2020-05-24 22:00:46 +02:00
} while (0)
LibWeb: Start building the tree building part of the new HTML parser This patch adds a new HTMLDocumentParser class. It keeps a tokenizer object internally and feeds itself with one token at a time from it. The names and idioms in this class are expressed as closely to the actual HTML parsing spec as possible, to make development as easy and bug free as possible. :^) This is going to become pretty large, but it's pretty cool!
2020-05-24 00:14:23 +02:00
LibWeb: Use getter and setter for Character type HTMLTokens While storing the code point in a UTF-8 encoded String in horrendously inefficient, this problem will be addressed at a later stage.
2021-07-14 23:33:12 +02:00
#define FLUSH_CODEPOINTS_CONSUMED_AS_A_CHARACTER_REFERENCE \
do { \
for (auto code_point : m_temporary_buffer) { \
if (consumed_as_part_of_an_attribute()) { \
m_current_builder.append_code_point(code_point); \
} else { \
create_new_token(HTMLToken::Type::Character); \
m_current_token.set_code_point(code_point); \
m_queued_tokens.enqueue(move(m_current_token)); \
} \
} \
LibWeb: Implement partial support for numeric character references
2020-05-27 16:16:23 +02:00
} while (0)
LibWeb: Add position tracking information to HTML tokens
2021-05-20 23:11:41 +04:30
#define DONT_CONSUME_NEXT_INPUT_CHARACTER \
do { \
restore_to(m_prev_utf8_iterator); \
LibWeb: Make the new HTML parser parse input as UTF-8 We already convert the input to UTF-8 before starting the tokenizer, so all this patch had to do was switch the tokenizer to use an Utf8View for its input (and to emit 32-bit codepoints.)
2020-06-04 21:06:54 +02:00
} while (0)
LibWeb: Begin work on a spec-compliant HTML parser In order to actually view the web as it is, we're gonna need a proper HTML parser. So let's build one! This patch introduces the Web::HTMLTokenizer class, which currently operates on a StringView input stream where it fetches (ASCII only atm) codepoints and tokenizes acccording to the HTML spec tokenization algo. The tokenizer state machine looks a bit weird but is written in a way that tries to mimic the spec as closely as possible, in order to make development easier and bugs less likely. This initial version is far from finished, but it can parse a trivial document with a DOCTYPE and open/close tags. :^)
2020-05-22 21:46:13 +02:00
Unicode: Try s/codepoint/code_point/g again This time, without trailing 's'. Ran: git grep -l 'codepoint' | xargs sed -ie 's/codepoint/code_point/g
2020-08-05 16:31:20 -04:00
#define ON(code_point) \
if (current_input_character.has_value() && current_input_character.value() == code_point)
LibWeb: Begin work on a spec-compliant HTML parser In order to actually view the web as it is, we're gonna need a proper HTML parser. So let's build one! This patch introduces the Web::HTMLTokenizer class, which currently operates on a StringView input stream where it fetches (ASCII only atm) codepoints and tokenizes acccording to the HTML spec tokenization algo. The tokenizer state machine looks a bit weird but is written in a way that tries to mimic the spec as closely as possible, in order to make development easier and bugs less likely. This initial version is far from finished, but it can parse a trivial document with a DOCTYPE and open/close tags. :^)
2020-05-22 21:46:13 +02:00
#define ON_EOF \
if (!current_input_character.has_value())
#define ON_ASCII_ALPHA \
Everywhere: Replace ctype.h to avoid narrowing conversions This replaces ctype.h with CharacterType.h everywhere I could find issues with narrowing conversions. While using it will probably make sense almost everywhere in the future, the most critical places should have been addressed.
2021-06-01 21:18:08 +02:00
if (current_input_character.has_value() && is_ascii_alpha(current_input_character.value()))
LibWeb: Begin work on a spec-compliant HTML parser In order to actually view the web as it is, we're gonna need a proper HTML parser. So let's build one! This patch introduces the Web::HTMLTokenizer class, which currently operates on a StringView input stream where it fetches (ASCII only atm) codepoints and tokenizes acccording to the HTML spec tokenization algo. The tokenizer state machine looks a bit weird but is written in a way that tries to mimic the spec as closely as possible, in order to make development easier and bugs less likely. This initial version is far from finished, but it can parse a trivial document with a DOCTYPE and open/close tags. :^)
2020-05-22 21:46:13 +02:00
LibWeb: Implement partial support for numeric character references
2020-05-27 16:16:23 +02:00
#define ON_ASCII_ALPHANUMERIC \
Everywhere: Replace ctype.h to avoid narrowing conversions This replaces ctype.h with CharacterType.h everywhere I could find issues with narrowing conversions. While using it will probably make sense almost everywhere in the future, the most critical places should have been addressed.
2021-06-01 21:18:08 +02:00
if (current_input_character.has_value() && is_ascii_alphanumeric(current_input_character.value()))
LibWeb: Implement partial support for numeric character references
2020-05-27 16:16:23 +02:00
LibWeb: Implement a bit more of DOCTYPE tokenization
2020-05-23 19:56:07 +02:00
#define ON_ASCII_UPPER_ALPHA \
Everywhere: Replace ctype.h to avoid narrowing conversions This replaces ctype.h with CharacterType.h everywhere I could find issues with narrowing conversions. While using it will probably make sense almost everywhere in the future, the most critical places should have been addressed.
2021-06-01 21:18:08 +02:00
if (current_input_character.has_value() && is_ascii_upper_alpha(current_input_character.value()))
LibWeb: Implement a bit more of DOCTYPE tokenization
2020-05-23 19:56:07 +02:00
LibWeb: More work on the HTML parser and tokenizer The parser can now switch the state of the tokenizer! Very webby. :^)
2020-05-24 20:24:43 +02:00
#define ON_ASCII_LOWER_ALPHA \
Everywhere: Replace ctype.h to avoid narrowing conversions This replaces ctype.h with CharacterType.h everywhere I could find issues with narrowing conversions. While using it will probably make sense almost everywhere in the future, the most critical places should have been addressed.
2021-06-01 21:18:08 +02:00
if (current_input_character.has_value() && is_ascii_lower_alpha(current_input_character.value()))
LibWeb: More work on the HTML parser and tokenizer The parser can now switch the state of the tokenizer! Very webby. :^)
2020-05-24 20:24:43 +02:00
LibWeb: Implement partial support for numeric character references
2020-05-27 16:16:23 +02:00
#define ON_ASCII_DIGIT \
Everywhere: Replace ctype.h to avoid narrowing conversions This replaces ctype.h with CharacterType.h everywhere I could find issues with narrowing conversions. While using it will probably make sense almost everywhere in the future, the most critical places should have been addressed.
2021-06-01 21:18:08 +02:00
if (current_input_character.has_value() && is_ascii_digit(current_input_character.value()))
LibWeb: Implement partial support for numeric character references
2020-05-27 16:16:23 +02:00
#define ON_ASCII_HEX_DIGIT \
Everywhere: Replace ctype.h to avoid narrowing conversions This replaces ctype.h with CharacterType.h everywhere I could find issues with narrowing conversions. While using it will probably make sense almost everywhere in the future, the most critical places should have been addressed.
2021-06-01 21:18:08 +02:00
if (current_input_character.has_value() && is_ascii_hex_digit(current_input_character.value()))
LibWeb: Implement partial support for numeric character references
2020-05-27 16:16:23 +02:00
LibWeb: Begin work on a spec-compliant HTML parser In order to actually view the web as it is, we're gonna need a proper HTML parser. So let's build one! This patch introduces the Web::HTMLTokenizer class, which currently operates on a StringView input stream where it fetches (ASCII only atm) codepoints and tokenizes acccording to the HTML spec tokenization algo. The tokenizer state machine looks a bit weird but is written in a way that tries to mimic the spec as closely as possible, in order to make development easier and bugs less likely. This initial version is far from finished, but it can parse a trivial document with a DOCTYPE and open/close tags. :^)
2020-05-22 21:46:13 +02:00
#define ON_WHITESPACE \
Everywhere: Replace ctype.h to avoid narrowing conversions This replaces ctype.h with CharacterType.h everywhere I could find issues with narrowing conversions. While using it will probably make sense almost everywhere in the future, the most critical places should have been addressed.
2021-06-01 21:18:08 +02:00
if (current_input_character.has_value() && is_ascii(current_input_character.value()) && "\t\n\f "sv.contains(current_input_character.value()))
LibWeb: Begin work on a spec-compliant HTML parser In order to actually view the web as it is, we're gonna need a proper HTML parser. So let's build one! This patch introduces the Web::HTMLTokenizer class, which currently operates on a StringView input stream where it fetches (ASCII only atm) codepoints and tokenizes acccording to the HTML spec tokenization algo. The tokenizer state machine looks a bit weird but is written in a way that tries to mimic the spec as closely as possible, in order to make development easier and bugs less likely. This initial version is far from finished, but it can parse a trivial document with a DOCTYPE and open/close tags. :^)
2020-05-22 21:46:13 +02:00
#define ANYTHING_ELSE if (1)
LibWeb: Use move() when enqueuing tokens in HTMLTokenizer We're not using the current token anymore once it's enqueued so let's use move() when enqueuing the tokens.
2021-05-23 12:39:00 +02:00
#define EMIT_EOF \
do { \
if (m_has_emitted_eof) \
return {}; \
m_has_emitted_eof = true; \
create_new_token(HTMLToken::Type::EndOfFile); \
will_emit(m_current_token); \
m_queued_tokens.enqueue(move(m_current_token)); \
return m_queued_tokens.dequeue(); \
LibWeb: A whole bunch of work towards spec-compliant <script> elements This is still very unfinished, but there's at least a skeleton of code.
2020-05-24 22:00:46 +02:00
} while (0)
LibWeb: More work on the HTML parser and tokenizer The parser can now switch the state of the tokenizer! Very webby. :^)
2020-05-24 20:24:43 +02:00
LibWeb: Use move() when enqueuing tokens in HTMLTokenizer We're not using the current token anymore once it's enqueued so let's use move() when enqueuing the tokens.
2021-05-23 12:39:00 +02:00
#define EMIT_CURRENT_TOKEN \
do { \
LibWeb: VERIFY an empty builder when emitting tokens in HTMLTokenizer
2021-07-15 00:34:54 +02:00
VERIFY(m_current_builder.is_empty()); \
LibWeb: Use move() when enqueuing tokens in HTMLTokenizer We're not using the current token anymore once it's enqueued so let's use move() when enqueuing the tokens.
2021-05-23 12:39:00 +02:00
will_emit(m_current_token); \
m_queued_tokens.enqueue(move(m_current_token)); \
return m_queued_tokens.dequeue(); \
LibWeb: A whole bunch of work towards spec-compliant <script> elements This is still very unfinished, but there's at least a skeleton of code.
2020-05-24 22:00:46 +02:00
} while (0)
LibWeb: Start building the tree building part of the new HTML parser This patch adds a new HTMLDocumentParser class. It keeps a tokenizer object internally and feeds itself with one token at a time from it. The names and idioms in this class are expressed as closely to the actual HTML parsing spec as possible, to make development as easy and bug free as possible. :^) This is going to become pretty large, but it's pretty cool!
2020-05-24 00:14:23 +02:00
LibWeb: Use getter and setter for Character type HTMLTokens While storing the code point in a UTF-8 encoded String in horrendously inefficient, this problem will be addressed at a later stage.
2021-07-14 23:33:12 +02:00
#define EMIT_CHARACTER(code_point) \
do { \
create_new_token(HTMLToken::Type::Character); \
m_current_token.set_code_point(code_point); \
m_queued_tokens.enqueue(move(m_current_token)); \
return m_queued_tokens.dequeue(); \
LibWeb: A whole bunch of work towards spec-compliant <script> elements This is still very unfinished, but there's at least a skeleton of code.
2020-05-24 22:00:46 +02:00
} while (0)
LibWeb: Begin work on a spec-compliant HTML parser In order to actually view the web as it is, we're gonna need a proper HTML parser. So let's build one! This patch introduces the Web::HTMLTokenizer class, which currently operates on a StringView input stream where it fetches (ASCII only atm) codepoints and tokenizes acccording to the HTML spec tokenization algo. The tokenizer state machine looks a bit weird but is written in a way that tries to mimic the spec as closely as possible, in order to make development easier and bugs less likely. This initial version is far from finished, but it can parse a trivial document with a DOCTYPE and open/close tags. :^)
2020-05-22 21:46:13 +02:00
LibWeb: More work on the HTML parser and tokenizer The parser can now switch the state of the tokenizer! Very webby. :^)
2020-05-24 20:24:43 +02:00
#define EMIT_CURRENT_CHARACTER \
EMIT_CHARACTER(current_input_character.value());
Unicode: Try s/codepoint/code_point/g again This time, without trailing 's'. Ran: git grep -l 'codepoint' | xargs sed -ie 's/codepoint/code_point/g
2020-08-05 16:31:20 -04:00
#define SWITCH_TO_AND_EMIT_CHARACTER(code_point, new_state) \
Meta+LibHTTP through LibWeb: Make clang-format-10 clean
2020-09-18 09:49:51 +02:00
do { \
will_switch_to(State::new_state); \
m_state = State::new_state; \
Unicode: Try s/codepoint/code_point/g again This time, without trailing 's'. Ran: git grep -l 'codepoint' | xargs sed -ie 's/codepoint/code_point/g
2020-08-05 16:31:20 -04:00
EMIT_CHARACTER(code_point); \
LibWeb: Fully implement all script tokenizer states Also fixes RAWTEXTLessThanSign having a separate emit and reconsume.
2020-06-06 07:06:46 +01:00
} while (0)
#define SWITCH_TO_AND_EMIT_CURRENT_CHARACTER(new_state) \
SWITCH_TO_AND_EMIT_CHARACTER(current_input_character.value(), new_state)
LibWeb: Begin work on a spec-compliant HTML parser In order to actually view the web as it is, we're gonna need a proper HTML parser. So let's build one! This patch introduces the Web::HTMLTokenizer class, which currently operates on a StringView input stream where it fetches (ASCII only atm) codepoints and tokenizes acccording to the HTML spec tokenization algo. The tokenizer state machine looks a bit weird but is written in a way that tries to mimic the spec as closely as possible, in order to make development easier and bugs less likely. This initial version is far from finished, but it can parse a trivial document with a DOCTYPE and open/close tags. :^)
2020-05-22 21:46:13 +02:00
#define BEGIN_STATE(state) \
state: \
LibWeb: Make BEGIN_STATE and END_STATE include some {{{ and }}} This makes it a compile error to omit the END_STATE. Also add some more missing END_STATE's exposed by this (nice!) Thanks to @predmond for suggesting the multi-pair trick! :^)
2020-05-23 14:04:53 +02:00
case State::state: { \
{ \
{
LibWeb: Begin work on a spec-compliant HTML parser In order to actually view the web as it is, we're gonna need a proper HTML parser. So let's build one! This patch introduces the Web::HTMLTokenizer class, which currently operates on a StringView input stream where it fetches (ASCII only atm) codepoints and tokenizes acccording to the HTML spec tokenization algo. The tokenizer state machine looks a bit weird but is written in a way that tries to mimic the spec as closely as possible, in order to make development easier and bugs less likely. This initial version is far from finished, but it can parse a trivial document with a DOCTYPE and open/close tags. :^)
2020-05-22 21:46:13 +02:00
#define END_STATE \
Everywhere: Rename ASSERT => VERIFY (...and ASSERT_NOT_REACHED => VERIFY_NOT_REACHED) Since all of these checks are done in release builds as well, let's rename them to VERIFY to prevent confusion, as everyone is used to assertions being compiled out in release. We can introduce a new ASSERT macro that is specifically for debug checks, but I'm doing this wholesale conversion first since we've accumulated thousands of these already, and it's not immediately obvious which ones are suitable for ASSERT.
2021-02-23 20:42:32 +01:00
VERIFY_NOT_REACHED(); \
LibWeb: Make BEGIN_STATE and END_STATE include some {{{ and }}} This makes it a compile error to omit the END_STATE. Also add some more missing END_STATE's exposed by this (nice!) Thanks to @predmond for suggesting the multi-pair trick! :^)
2020-05-23 14:04:53 +02:00
break; \
} \
} \
}
LibWeb: Begin work on a spec-compliant HTML parser In order to actually view the web as it is, we're gonna need a proper HTML parser. So let's build one! This patch introduces the Web::HTMLTokenizer class, which currently operates on a StringView input stream where it fetches (ASCII only atm) codepoints and tokenizes acccording to the HTML spec tokenization algo. The tokenizer state machine looks a bit weird but is written in a way that tries to mimic the spec as closely as possible, in order to make development easier and bugs less likely. This initial version is far from finished, but it can parse a trivial document with a DOCTYPE and open/close tags. :^)
2020-05-22 21:46:13 +02:00
LibWeb: Change HTMLTokenizer.{cpp,h} to east const style
2021-07-12 12:44:21 +02:00
static inline void log_parse_error(SourceLocation const& location = SourceLocation::current())
LibWeb: Utilize SourceLocation for HTMLTokenizer logging
2021-04-24 20:22:30 -07:00
{
dbgln_if(TOKENIZER_TRACE_DEBUG, "Parse error (tokenization) {}", location);
}
Unicode: Try s/codepoint/code_point/g again This time, without trailing 's'. Ran: git grep -l 'codepoint' | xargs sed -ie 's/codepoint/code_point/g
2020-08-05 16:31:20 -04:00
Optional<u32> HTMLTokenizer::next_code_point()
LibWeb: Begin work on a spec-compliant HTML parser In order to actually view the web as it is, we're gonna need a proper HTML parser. So let's build one! This patch introduces the Web::HTMLTokenizer class, which currently operates on a StringView input stream where it fetches (ASCII only atm) codepoints and tokenizes acccording to the HTML spec tokenization algo. The tokenizer state machine looks a bit weird but is written in a way that tries to mimic the spec as closely as possible, in order to make development easier and bugs less likely. This initial version is far from finished, but it can parse a trivial document with a DOCTYPE and open/close tags. :^)
2020-05-22 21:46:13 +02:00
{
LibWeb: Make the new HTML parser parse input as UTF-8 We already convert the input to UTF-8 before starting the tokenizer, so all this patch had to do was switch the tokenizer to use an Utf8View for its input (and to emit 32-bit codepoints.)
2020-06-04 21:06:54 +02:00
if (m_utf8_iterator == m_utf8_view.end())
LibWeb: Begin work on a spec-compliant HTML parser In order to actually view the web as it is, we're gonna need a proper HTML parser. So let's build one! This patch introduces the Web::HTMLTokenizer class, which currently operates on a StringView input stream where it fetches (ASCII only atm) codepoints and tokenizes acccording to the HTML spec tokenization algo. The tokenizer state machine looks a bit weird but is written in a way that tries to mimic the spec as closely as possible, in order to make development easier and bugs less likely. This initial version is far from finished, but it can parse a trivial document with a DOCTYPE and open/close tags. :^)
2020-05-22 21:46:13 +02:00
return {};
LibWeb: Implement tokenization newline preprocessing Newline normalization will replace \r and \r\n with \n. The spec specifically states > Before the tokenization stage, the input stream must be preprocessed > by normalizing newlines. wheras this is implemented the processing during the tokenization itself. This should still exhibit the same behaviour, while keeping the tokenization logic in the same place.
2022-02-18 22:46:28 +00:00
u32 code_point;
// https://html.spec.whatwg.org/multipage/parsing.html#preprocessing-the-input-stream:tokenization
// https://infra.spec.whatwg.org/#normalize-newlines
if (peek_code_point(0).value_or(0) == '\r' && peek_code_point(1).value_or(0) == '\n') {
// replace every U+000D CR U+000A LF code point pair with a single U+000A LF code point,
skip(2);
code_point = '\n';
} else if (peek_code_point(0).value_or(0) == '\r') {
// replace every remaining U+000D CR code point with a U+000A LF code point.
skip(1);
code_point = '\n';
} else {
skip(1);
code_point = *m_prev_utf8_iterator;
}
dbgln_if(TOKENIZER_TRACE_DEBUG, "(Tokenizer) Next code_point: {}", code_point);
return code_point;
LibWeb: Begin work on a spec-compliant HTML parser In order to actually view the web as it is, we're gonna need a proper HTML parser. So let's build one! This patch introduces the Web::HTMLTokenizer class, which currently operates on a StringView input stream where it fetches (ASCII only atm) codepoints and tokenizes acccording to the HTML spec tokenization algo. The tokenizer state machine looks a bit weird but is written in a way that tries to mimic the spec as closely as possible, in order to make development easier and bugs less likely. This initial version is far from finished, but it can parse a trivial document with a DOCTYPE and open/close tags. :^)
2020-05-22 21:46:13 +02:00
}
LibWeb: Add position tracking information to HTML tokens
2021-05-20 23:11:41 +04:30
void HTMLTokenizer::skip(size_t count)
{
LibWeb: Fix Array OOBs in the HTMLTokenizer Accessing last() if there are no elements makes WebContent crash :^)
2022-06-02 01:03:44 +02:00
if (!m_source_positions.is_empty())
m_source_positions.append(m_source_positions.last());
LibWeb: Add position tracking information to HTML tokens
2021-05-20 23:11:41 +04:30
for (size_t i = 0; i < count; ++i) {
LibWeb: Fix invalid behaviour of HTMLTokenizer::skip() and restore_to() skip() is supposed to end up keeping the previous iterator only one index behind the current one, and restore_to() should actually do the restore instead of just removing the now-useless source positions. Fixes #7331.
2021-05-21 11:03:31 +04:30
m_prev_utf8_iterator = m_utf8_iterator;
LibWeb: Add position tracking information to HTML tokens
2021-05-20 23:11:41 +04:30
auto code_point = *m_utf8_iterator;
LibWeb: Fix Array OOBs in the HTMLTokenizer Accessing last() if there are no elements makes WebContent crash :^)
2022-06-02 01:03:44 +02:00
if (!m_source_positions.is_empty()) {
if (code_point == '\n') {
m_source_positions.last().column = 0;
m_source_positions.last().line++;
} else {
m_source_positions.last().column++;
}
LibWeb: Add position tracking information to HTML tokens
2021-05-20 23:11:41 +04:30
}
++m_utf8_iterator;
}
}
Unicode: Try s/codepoint/code_point/g again This time, without trailing 's'. Ran: git grep -l 'codepoint' | xargs sed -ie 's/codepoint/code_point/g
2020-08-05 16:31:20 -04:00
Optional<u32> HTMLTokenizer::peek_code_point(size_t offset) const
LibWeb: Begin work on a spec-compliant HTML parser In order to actually view the web as it is, we're gonna need a proper HTML parser. So let's build one! This patch introduces the Web::HTMLTokenizer class, which currently operates on a StringView input stream where it fetches (ASCII only atm) codepoints and tokenizes acccording to the HTML spec tokenization algo. The tokenizer state machine looks a bit weird but is written in a way that tries to mimic the spec as closely as possible, in order to make development easier and bugs less likely. This initial version is far from finished, but it can parse a trivial document with a DOCTYPE and open/close tags. :^)
2020-05-22 21:46:13 +02:00
{
LibWeb: Unbreak character reference and DOCTYPE parsing post-UTF-8 Oops, these were still using the byte-offset cursor. My goodness is it unergonomic to index into UTF-8 strings, but Dr. Bugaev says it's good. There is lots of room for improvement here. Just like the rest of the tokenizer and parser. We'll have to do a few optimization passes over them once they mature.
2020-06-04 22:02:54 +02:00
auto it = m_utf8_iterator;
for (size_t i = 0; i < offset && it != m_utf8_view.end(); ++i)
++it;
if (it == m_utf8_view.end())
LibWeb: Begin work on a spec-compliant HTML parser In order to actually view the web as it is, we're gonna need a proper HTML parser. So let's build one! This patch introduces the Web::HTMLTokenizer class, which currently operates on a StringView input stream where it fetches (ASCII only atm) codepoints and tokenizes acccording to the HTML spec tokenization algo. The tokenizer state machine looks a bit weird but is written in a way that tries to mimic the spec as closely as possible, in order to make development easier and bugs less likely. This initial version is far from finished, but it can parse a trivial document with a DOCTYPE and open/close tags. :^)
2020-05-22 21:46:13 +02:00
return {};
LibWeb: Unbreak character reference and DOCTYPE parsing post-UTF-8 Oops, these were still using the byte-offset cursor. My goodness is it unergonomic to index into UTF-8 strings, but Dr. Bugaev says it's good. There is lots of room for improvement here. Just like the rest of the tokenizer and parser. We'll have to do a few optimization passes over them once they mature.
2020-06-04 22:02:54 +02:00
return *it;
LibWeb: Begin work on a spec-compliant HTML parser In order to actually view the web as it is, we're gonna need a proper HTML parser. So let's build one! This patch introduces the Web::HTMLTokenizer class, which currently operates on a StringView input stream where it fetches (ASCII only atm) codepoints and tokenizes acccording to the HTML spec tokenization algo. The tokenizer state machine looks a bit weird but is written in a way that tries to mimic the spec as closely as possible, in order to make development easier and bugs less likely. This initial version is far from finished, but it can parse a trivial document with a DOCTYPE and open/close tags. :^)
2020-05-22 21:46:13 +02:00
}
LibWeb: Be more forgiving when adding source positions in HTMLTokenizer This patch changes HTMLTokenizer::nth_last_position to not fail if the requested position is not available. Rather, it will just return (0-0). While this is not the correct solution, it prevents the tokenizer from crashing just because it cannot find a source position. This should only affect SyntaxHighlighter.
2021-06-04 11:31:43 +02:00
HTMLToken::Position HTMLTokenizer::nth_last_position(size_t n)
{
if (n + 1 > m_source_positions.size()) {
dbgln_if(TOKENIZER_TRACE_DEBUG, "(Tokenizer::nth_last_position) Invalid position requested: {}th-last of {}. Returning (0-0).", n, m_source_positions.size());
return HTMLToken::Position { 0, 0 };
};
return m_source_positions.at(m_source_positions.size() - 1 - n);
}
LibWeb: Start building the tree building part of the new HTML parser This patch adds a new HTMLDocumentParser class. It keeps a tokenizer object internally and feeds itself with one token at a time from it. The names and idioms in this class are expressed as closely to the actual HTML parsing spec as possible, to make development as easy and bug free as possible. :^) This is going to become pretty large, but it's pretty cool!
2020-05-24 00:14:23 +02:00
Optional<HTMLToken> HTMLTokenizer::next_token()
LibWeb: Begin work on a spec-compliant HTML parser In order to actually view the web as it is, we're gonna need a proper HTML parser. So let's build one! This patch introduces the Web::HTMLTokenizer class, which currently operates on a StringView input stream where it fetches (ASCII only atm) codepoints and tokenizes acccording to the HTML spec tokenization algo. The tokenizer state machine looks a bit weird but is written in a way that tries to mimic the spec as closely as possible, in order to make development easier and bugs less likely. This initial version is far from finished, but it can parse a trivial document with a DOCTYPE and open/close tags. :^)
2020-05-22 21:46:13 +02:00
{
LibWeb: Fix Array OOBs in the HTMLTokenizer Accessing last() if there are no elements makes WebContent crash :^)
2022-06-02 01:03:44 +02:00
if (!m_source_positions.is_empty()) {
LibWeb: Add a super basic HTML syntax highlighter This can currently highlight tag names and attribute names/values.
2021-05-20 23:15:33 +04:30
auto last_position = m_source_positions.last();
LibWeb: Use Vector::clear_with_capacity() in HTMLTokenizer This avoids constantly reallocating the Vector<HTMLToken>.
2022-02-18 23:49:51 +01:00
m_source_positions.clear_with_capacity();
LibWeb: Add a super basic HTML syntax highlighter This can currently highlight tag names and attribute names/values.
2021-05-20 23:15:33 +04:30
m_source_positions.append(move(last_position));
}
LibWeb: Implement partial support for numeric character references
2020-05-27 16:16:23 +02:00
_StartOfFunction:
LibWeb: Allow HTML tokenizer to emit more than one token Tokens are now put on a queue when emitted, and we always pop from that queue when returning from next_token().
2020-05-26 15:50:05 +02:00
if (!m_queued_tokens.is_empty())
return m_queued_tokens.dequeue();
LibWeb: Begin work on a spec-compliant HTML parser In order to actually view the web as it is, we're gonna need a proper HTML parser. So let's build one! This patch introduces the Web::HTMLTokenizer class, which currently operates on a StringView input stream where it fetches (ASCII only atm) codepoints and tokenizes acccording to the HTML spec tokenization algo. The tokenizer state machine looks a bit weird but is written in a way that tries to mimic the spec as closely as possible, in order to make development easier and bugs less likely. This initial version is far from finished, but it can parse a trivial document with a DOCTYPE and open/close tags. :^)
2020-05-22 21:46:13 +02:00
for (;;) {
Unicode: Try s/codepoint/code_point/g again This time, without trailing 's'. Ran: git grep -l 'codepoint' | xargs sed -ie 's/codepoint/code_point/g
2020-08-05 16:31:20 -04:00
auto current_input_character = next_code_point();
LibWeb: Begin work on a spec-compliant HTML parser In order to actually view the web as it is, we're gonna need a proper HTML parser. So let's build one! This patch introduces the Web::HTMLTokenizer class, which currently operates on a StringView input stream where it fetches (ASCII only atm) codepoints and tokenizes acccording to the HTML spec tokenization algo. The tokenizer state machine looks a bit weird but is written in a way that tries to mimic the spec as closely as possible, in order to make development easier and bugs less likely. This initial version is far from finished, but it can parse a trivial document with a DOCTYPE and open/close tags. :^)
2020-05-22 21:46:13 +02:00
switch (m_state) {
LibWeb: Add spec links to each HTML tokenizer state section I didn't add full spec comments this time, but this is better than nothing :^)
2022-02-15 18:12:15 +00:00
// 13.2.5.1 Data state, https://html.spec.whatwg.org/multipage/parsing.html#data-state
LibWeb: Begin work on a spec-compliant HTML parser In order to actually view the web as it is, we're gonna need a proper HTML parser. So let's build one! This patch introduces the Web::HTMLTokenizer class, which currently operates on a StringView input stream where it fetches (ASCII only atm) codepoints and tokenizes acccording to the HTML spec tokenization algo. The tokenizer state machine looks a bit weird but is written in a way that tries to mimic the spec as closely as possible, in order to make development easier and bugs less likely. This initial version is far from finished, but it can parse a trivial document with a DOCTYPE and open/close tags. :^)
2020-05-22 21:46:13 +02:00
BEGIN_STATE(Data)
{
ON('&')
{
m_return_state = State::Data;
SWITCH_TO(CharacterReference);
}
ON('<')
{
SWITCH_TO(TagOpen);
}
LibWeb: Implement more of the tokenizer states Slowly adding more unimplemented options for tokenizer states.
2020-05-28 00:28:32 +02:00
ON(0)
{
LibWeb: Utilize SourceLocation for HTMLTokenizer logging
2021-04-24 20:22:30 -07:00
log_parse_error();
LibWeb: Implement more of the tokenizer states Slowly adding more unimplemented options for tokenizer states.
2020-05-28 00:28:32 +02:00
EMIT_CURRENT_CHARACTER;
}
LibWeb: Begin work on a spec-compliant HTML parser In order to actually view the web as it is, we're gonna need a proper HTML parser. So let's build one! This patch introduces the Web::HTMLTokenizer class, which currently operates on a StringView input stream where it fetches (ASCII only atm) codepoints and tokenizes acccording to the HTML spec tokenization algo. The tokenizer state machine looks a bit weird but is written in a way that tries to mimic the spec as closely as possible, in order to make development easier and bugs less likely. This initial version is far from finished, but it can parse a trivial document with a DOCTYPE and open/close tags. :^)
2020-05-22 21:46:13 +02:00
ON_EOF
{
LibWeb: Start building the tree building part of the new HTML parser This patch adds a new HTMLDocumentParser class. It keeps a tokenizer object internally and feeds itself with one token at a time from it. The names and idioms in this class are expressed as closely to the actual HTML parsing spec as possible, to make development as easy and bug free as possible. :^) This is going to become pretty large, but it's pretty cool!
2020-05-24 00:14:23 +02:00
EMIT_EOF;
LibWeb: Begin work on a spec-compliant HTML parser In order to actually view the web as it is, we're gonna need a proper HTML parser. So let's build one! This patch introduces the Web::HTMLTokenizer class, which currently operates on a StringView input stream where it fetches (ASCII only atm) codepoints and tokenizes acccording to the HTML spec tokenization algo. The tokenizer state machine looks a bit weird but is written in a way that tries to mimic the spec as closely as possible, in order to make development easier and bugs less likely. This initial version is far from finished, but it can parse a trivial document with a DOCTYPE and open/close tags. :^)
2020-05-22 21:46:13 +02:00
}
ANYTHING_ELSE
{
LibWeb: More work on the HTML parser and tokenizer The parser can now switch the state of the tokenizer! Very webby. :^)
2020-05-24 20:24:43 +02:00
EMIT_CURRENT_CHARACTER;
LibWeb: Begin work on a spec-compliant HTML parser In order to actually view the web as it is, we're gonna need a proper HTML parser. So let's build one! This patch introduces the Web::HTMLTokenizer class, which currently operates on a StringView input stream where it fetches (ASCII only atm) codepoints and tokenizes acccording to the HTML spec tokenization algo. The tokenizer state machine looks a bit weird but is written in a way that tries to mimic the spec as closely as possible, in order to make development easier and bugs less likely. This initial version is far from finished, but it can parse a trivial document with a DOCTYPE and open/close tags. :^)
2020-05-22 21:46:13 +02:00
}
}
END_STATE
LibWeb: Add spec links to each HTML tokenizer state section I didn't add full spec comments this time, but this is better than nothing :^)
2022-02-15 18:12:15 +00:00
// 13.2.5.6 Tag open state, https://html.spec.whatwg.org/multipage/parsing.html#tag-open-state
LibWeb: Begin work on a spec-compliant HTML parser In order to actually view the web as it is, we're gonna need a proper HTML parser. So let's build one! This patch introduces the Web::HTMLTokenizer class, which currently operates on a StringView input stream where it fetches (ASCII only atm) codepoints and tokenizes acccording to the HTML spec tokenization algo. The tokenizer state machine looks a bit weird but is written in a way that tries to mimic the spec as closely as possible, in order to make development easier and bugs less likely. This initial version is far from finished, but it can parse a trivial document with a DOCTYPE and open/close tags. :^)
2020-05-22 21:46:13 +02:00
BEGIN_STATE(TagOpen)
{
ON('!')
{
SWITCH_TO(MarkupDeclarationOpen);
}
ON('/')
{
SWITCH_TO(EndTagOpen);
}
ON_ASCII_ALPHA
{
create_new_token(HTMLToken::Type::StartTag);
RECONSUME_IN(TagName);
}
LibWeb: Emit character/comment tokens lazily to accumulate more data Instead of emitting data-bearing tokens immediately, do it lazily at the next state change. This allows us to accumulate full bursts of text in between tags instead of having one token per character. :^)
2020-05-23 18:43:09 +02:00
ON('?')
LibWeb: Implement more of the tokenizer states Slowly adding more unimplemented options for tokenizer states.
2020-05-28 00:28:32 +02:00
{
LibWeb: Utilize SourceLocation for HTMLTokenizer logging
2021-04-24 20:22:30 -07:00
log_parse_error();
LibWeb: Implement more of the tokenizer states Slowly adding more unimplemented options for tokenizer states.
2020-05-28 00:28:32 +02:00
create_new_token(HTMLToken::Type::Comment);
LibWeb: Use setter for HTMLToken::m_{start,end}_position
2021-07-15 15:53:54 +02:00
m_current_token.set_start_position({}, nth_last_position(2));
LibWeb: Implement more of the tokenizer states Slowly adding more unimplemented options for tokenizer states.
2020-05-28 00:28:32 +02:00
RECONSUME_IN(BogusComment);
}
ON_EOF
LibWeb: Emit character/comment tokens lazily to accumulate more data Instead of emitting data-bearing tokens immediately, do it lazily at the next state change. This allows us to accumulate full bursts of text in between tags instead of having one token per character. :^)
2020-05-23 18:43:09 +02:00
{
LibWeb: Utilize SourceLocation for HTMLTokenizer logging
2021-04-24 20:22:30 -07:00
log_parse_error();
LibWeb: Implement some missing tokenizer cases for EOF handling
2020-06-26 22:47:07 +02:00
m_queued_tokens.enqueue(HTMLToken::make_character('<'));
EMIT_EOF;
LibWeb: Emit character/comment tokens lazily to accumulate more data Instead of emitting data-bearing tokens immediately, do it lazily at the next state change. This allows us to accumulate full bursts of text in between tags instead of having one token per character. :^)
2020-05-23 18:43:09 +02:00
}
ANYTHING_ELSE
{
LibWeb: Utilize SourceLocation for HTMLTokenizer logging
2021-04-24 20:22:30 -07:00
log_parse_error();
LibWeb: Fully implement all DOCTYPE tokenizer states Also fixes TagOpen having a seperate emit and reconsume in ANYTHING_ELSE.
2020-06-11 05:00:45 +01:00
EMIT_CHARACTER_AND_RECONSUME_IN('<', Data);
LibWeb: Emit character/comment tokens lazily to accumulate more data Instead of emitting data-bearing tokens immediately, do it lazily at the next state change. This allows us to accumulate full bursts of text in between tags instead of having one token per character. :^)
2020-05-23 18:43:09 +02:00
}
LibWeb: Begin work on a spec-compliant HTML parser In order to actually view the web as it is, we're gonna need a proper HTML parser. So let's build one! This patch introduces the Web::HTMLTokenizer class, which currently operates on a StringView input stream where it fetches (ASCII only atm) codepoints and tokenizes acccording to the HTML spec tokenization algo. The tokenizer state machine looks a bit weird but is written in a way that tries to mimic the spec as closely as possible, in order to make development easier and bugs less likely. This initial version is far from finished, but it can parse a trivial document with a DOCTYPE and open/close tags. :^)
2020-05-22 21:46:13 +02:00
}
END_STATE
LibWeb: Add spec links to each HTML tokenizer state section I didn't add full spec comments this time, but this is better than nothing :^)
2022-02-15 18:12:15 +00:00
// 13.2.5.8 Tag name state, https://html.spec.whatwg.org/multipage/parsing.html#tag-name-state
LibWeb: Begin work on a spec-compliant HTML parser In order to actually view the web as it is, we're gonna need a proper HTML parser. So let's build one! This patch introduces the Web::HTMLTokenizer class, which currently operates on a StringView input stream where it fetches (ASCII only atm) codepoints and tokenizes acccording to the HTML spec tokenization algo. The tokenizer state machine looks a bit weird but is written in a way that tries to mimic the spec as closely as possible, in order to make development easier and bugs less likely. This initial version is far from finished, but it can parse a trivial document with a DOCTYPE and open/close tags. :^)
2020-05-22 21:46:13 +02:00
BEGIN_STATE(TagName)
{
LibWeb: Teach HTMLTokenizer how to tokenize attributes Properly tokenize single-quoted, double-quoted and unquoted attributes!
2020-05-23 01:19:42 +02:00
ON_WHITESPACE
{
LibWeb: User getter+setter for HTMLToken tag name and self-closing flag
2021-07-14 23:37:48 +02:00
m_current_token.set_tag_name(consume_current_builder());
LibWeb: Use setter for HTMLToken::m_{start,end}_position
2021-07-15 15:53:54 +02:00
m_current_token.set_end_position({}, nth_last_position(1));
LibWeb: Teach HTMLTokenizer how to tokenize attributes Properly tokenize single-quoted, double-quoted and unquoted attributes!
2020-05-23 01:19:42 +02:00
SWITCH_TO(BeforeAttributeName);
}
ON('/')
{
LibWeb: User getter+setter for HTMLToken tag name and self-closing flag
2021-07-14 23:37:48 +02:00
m_current_token.set_tag_name(consume_current_builder());
LibWeb: Use setter for HTMLToken::m_{start,end}_position
2021-07-15 15:53:54 +02:00
m_current_token.set_end_position({}, nth_last_position(0));
LibWeb: Teach HTMLTokenizer how to tokenize attributes Properly tokenize single-quoted, double-quoted and unquoted attributes!
2020-05-23 01:19:42 +02:00
SWITCH_TO(SelfClosingStartTag);
}
LibWeb: Begin work on a spec-compliant HTML parser In order to actually view the web as it is, we're gonna need a proper HTML parser. So let's build one! This patch introduces the Web::HTMLTokenizer class, which currently operates on a StringView input stream where it fetches (ASCII only atm) codepoints and tokenizes acccording to the HTML spec tokenization algo. The tokenizer state machine looks a bit weird but is written in a way that tries to mimic the spec as closely as possible, in order to make development easier and bugs less likely. This initial version is far from finished, but it can parse a trivial document with a DOCTYPE and open/close tags. :^)
2020-05-22 21:46:13 +02:00
ON('>')
{
LibWeb: User getter+setter for HTMLToken tag name and self-closing flag
2021-07-14 23:37:48 +02:00
m_current_token.set_tag_name(consume_current_builder());
LibWeb: Use setter for HTMLToken::m_{start,end}_position
2021-07-15 15:53:54 +02:00
m_current_token.set_end_position({}, nth_last_position(1));
LibWeb: Start building the tree building part of the new HTML parser This patch adds a new HTMLDocumentParser class. It keeps a tokenizer object internally and feeds itself with one token at a time from it. The names and idioms in this class are expressed as closely to the actual HTML parsing spec as possible, to make development as easy and bug free as possible. :^) This is going to become pretty large, but it's pretty cool!
2020-05-24 00:14:23 +02:00
SWITCH_TO_AND_EMIT_CURRENT_TOKEN(Data);
LibWeb: Begin work on a spec-compliant HTML parser In order to actually view the web as it is, we're gonna need a proper HTML parser. So let's build one! This patch introduces the Web::HTMLTokenizer class, which currently operates on a StringView input stream where it fetches (ASCII only atm) codepoints and tokenizes acccording to the HTML spec tokenization algo. The tokenizer state machine looks a bit weird but is written in a way that tries to mimic the spec as closely as possible, in order to make development easier and bugs less likely. This initial version is far from finished, but it can parse a trivial document with a DOCTYPE and open/close tags. :^)
2020-05-22 21:46:13 +02:00
}
LibWeb: Implement more of the tokenizer states Slowly adding more unimplemented options for tokenizer states.
2020-05-28 00:28:32 +02:00
ON_ASCII_UPPER_ALPHA
{
LibWeb: Remove StringBuilder from HTMLToken::m_tag
2021-05-23 09:16:07 +02:00
m_current_builder.append_code_point(to_ascii_lowercase(current_input_character.value()));
LibWeb: Use setter for HTMLToken::m_{start,end}_position
2021-07-15 15:53:54 +02:00
m_current_token.set_end_position({}, nth_last_position(0));
LibWeb: Implement more of the tokenizer states Slowly adding more unimplemented options for tokenizer states.
2020-05-28 00:28:32 +02:00
continue;
}
ON(0)
{
LibWeb: Utilize SourceLocation for HTMLTokenizer logging
2021-04-24 20:22:30 -07:00
log_parse_error();
LibWeb: Remove StringBuilder from HTMLToken::m_tag
2021-05-23 09:16:07 +02:00
m_current_builder.append_code_point(0xFFFD);
LibWeb: Use setter for HTMLToken::m_{start,end}_position
2021-07-15 15:53:54 +02:00
m_current_token.set_end_position({}, nth_last_position(0));
LibWeb: Implement more of the tokenizer states Slowly adding more unimplemented options for tokenizer states.
2020-05-28 00:28:32 +02:00
continue;
}
ON_EOF
{
LibWeb: Utilize SourceLocation for HTMLTokenizer logging
2021-04-24 20:22:30 -07:00
log_parse_error();
LibWeb: Use setter for HTMLToken::m_{start,end}_position
2021-07-15 15:53:54 +02:00
m_current_token.set_end_position({}, nth_last_position(0));
LibWeb: Implement more of the tokenizer states Slowly adding more unimplemented options for tokenizer states.
2020-05-28 00:28:32 +02:00
EMIT_EOF;
}
LibWeb: Begin work on a spec-compliant HTML parser In order to actually view the web as it is, we're gonna need a proper HTML parser. So let's build one! This patch introduces the Web::HTMLTokenizer class, which currently operates on a StringView input stream where it fetches (ASCII only atm) codepoints and tokenizes acccording to the HTML spec tokenization algo. The tokenizer state machine looks a bit weird but is written in a way that tries to mimic the spec as closely as possible, in order to make development easier and bugs less likely. This initial version is far from finished, but it can parse a trivial document with a DOCTYPE and open/close tags. :^)
2020-05-22 21:46:13 +02:00
ANYTHING_ELSE
{
LibWeb: Remove StringBuilder from HTMLToken::m_tag
2021-05-23 09:16:07 +02:00
m_current_builder.append_code_point(current_input_character.value());
LibWeb: Use setter for HTMLToken::m_{start,end}_position
2021-07-15 15:53:54 +02:00
m_current_token.set_end_position({}, nth_last_position(0));
LibWeb: Begin work on a spec-compliant HTML parser In order to actually view the web as it is, we're gonna need a proper HTML parser. So let's build one! This patch introduces the Web::HTMLTokenizer class, which currently operates on a StringView input stream where it fetches (ASCII only atm) codepoints and tokenizes acccording to the HTML spec tokenization algo. The tokenizer state machine looks a bit weird but is written in a way that tries to mimic the spec as closely as possible, in order to make development easier and bugs less likely. This initial version is far from finished, but it can parse a trivial document with a DOCTYPE and open/close tags. :^)
2020-05-22 21:46:13 +02:00
continue;
}
}
LibWeb: Add missing END_STATE for TagName Fixes #2339.
2020-05-23 10:33:23 +02:00
END_STATE
LibWeb: Begin work on a spec-compliant HTML parser In order to actually view the web as it is, we're gonna need a proper HTML parser. So let's build one! This patch introduces the Web::HTMLTokenizer class, which currently operates on a StringView input stream where it fetches (ASCII only atm) codepoints and tokenizes acccording to the HTML spec tokenization algo. The tokenizer state machine looks a bit weird but is written in a way that tries to mimic the spec as closely as possible, in order to make development easier and bugs less likely. This initial version is far from finished, but it can parse a trivial document with a DOCTYPE and open/close tags. :^)
2020-05-22 21:46:13 +02:00
LibWeb: Add spec links to each HTML tokenizer state section I didn't add full spec comments this time, but this is better than nothing :^)
2022-02-15 18:12:15 +00:00
// 13.2.5.7 End tag open state, https://html.spec.whatwg.org/multipage/parsing.html#end-tag-open-state
LibWeb: Begin work on a spec-compliant HTML parser In order to actually view the web as it is, we're gonna need a proper HTML parser. So let's build one! This patch introduces the Web::HTMLTokenizer class, which currently operates on a StringView input stream where it fetches (ASCII only atm) codepoints and tokenizes acccording to the HTML spec tokenization algo. The tokenizer state machine looks a bit weird but is written in a way that tries to mimic the spec as closely as possible, in order to make development easier and bugs less likely. This initial version is far from finished, but it can parse a trivial document with a DOCTYPE and open/close tags. :^)
2020-05-22 21:46:13 +02:00
BEGIN_STATE(EndTagOpen)
{
ON_ASCII_ALPHA
{
create_new_token(HTMLToken::Type::EndTag);
RECONSUME_IN(TagName);
}
LibWeb: Implement more of the tokenizer states Slowly adding more unimplemented options for tokenizer states.
2020-05-28 00:28:32 +02:00
ON('>')
{
LibWeb: Utilize SourceLocation for HTMLTokenizer logging
2021-04-24 20:22:30 -07:00
log_parse_error();
LibWeb: Implement more of the tokenizer states Slowly adding more unimplemented options for tokenizer states.
2020-05-28 00:28:32 +02:00
SWITCH_TO(Data);
}
ON_EOF
{
LibWeb: Utilize SourceLocation for HTMLTokenizer logging
2021-04-24 20:22:30 -07:00
log_parse_error();
LibWeb: Implement some missing tokenizer cases for EOF handling
2020-06-26 22:47:07 +02:00
m_queued_tokens.enqueue(HTMLToken::make_character('<'));
m_queued_tokens.enqueue(HTMLToken::make_character('/'));
EMIT_EOF;
LibWeb: Implement more of the tokenizer states Slowly adding more unimplemented options for tokenizer states.
2020-05-28 00:28:32 +02:00
}
ANYTHING_ELSE
{
LibWeb: Utilize SourceLocation for HTMLTokenizer logging
2021-04-24 20:22:30 -07:00
log_parse_error();
LibWeb: Implement more of the tokenizer states Slowly adding more unimplemented options for tokenizer states.
2020-05-28 00:28:32 +02:00
create_new_token(HTMLToken::Type::Comment);
RECONSUME_IN(BogusComment);
}
LibWeb: Begin work on a spec-compliant HTML parser In order to actually view the web as it is, we're gonna need a proper HTML parser. So let's build one! This patch introduces the Web::HTMLTokenizer class, which currently operates on a StringView input stream where it fetches (ASCII only atm) codepoints and tokenizes acccording to the HTML spec tokenization algo. The tokenizer state machine looks a bit weird but is written in a way that tries to mimic the spec as closely as possible, in order to make development easier and bugs less likely. This initial version is far from finished, but it can parse a trivial document with a DOCTYPE and open/close tags. :^)
2020-05-22 21:46:13 +02:00
}
END_STATE
LibWeb: Add spec links to each HTML tokenizer state section I didn't add full spec comments this time, but this is better than nothing :^)
2022-02-15 18:12:15 +00:00
// 13.2.5.42 Markup declaration open state, https://html.spec.whatwg.org/multipage/parsing.html#markup-declaration-open-state
LibWeb: Begin work on a spec-compliant HTML parser In order to actually view the web as it is, we're gonna need a proper HTML parser. So let's build one! This patch introduces the Web::HTMLTokenizer class, which currently operates on a StringView input stream where it fetches (ASCII only atm) codepoints and tokenizes acccording to the HTML spec tokenization algo. The tokenizer state machine looks a bit weird but is written in a way that tries to mimic the spec as closely as possible, in order to make development easier and bugs less likely. This initial version is far from finished, but it can parse a trivial document with a DOCTYPE and open/close tags. :^)
2020-05-22 21:46:13 +02:00
BEGIN_STATE(MarkupDeclarationOpen)
{
DONT_CONSUME_NEXT_INPUT_CHARACTER;
LibWeb: Checking for "DOCTYPE" should be case insensitive in tokenizer
2020-05-25 19:22:23 +02:00
if (consume_next_if_match("--")) {
LibWeb: Teach HTMLTokenizer how to tokenize comments We can now correctly tokenize the welcome.html test page. :^)
2020-05-23 01:54:26 +02:00
create_new_token(HTMLToken::Type::Comment);
LibWeb: Fix highlighting HTML comments Commit b193351a99 caused the HTML comments to flash when changing the text cursor. Also, when double-clicking on a comment, the selection started from the beginning of the file instead. The following message was displaying when `TOKENIZER_TRACE_DEBUG` was enabled: (Tokenizer::nth_last_position) Invalid position requested: 4th-last of 4. Returning (0-0). Changing the `nth_last_position` to 3 fixes this. I'm guessing that's because the parser is at that moment on the second hyphen of the `<!--` string, so it has to go back only by three characters.
2021-07-29 15:36:10 +02:00
m_current_token.set_start_position({}, nth_last_position(3));
LibWeb: Teach HTMLTokenizer how to tokenize comments We can now correctly tokenize the welcome.html test page. :^)
2020-05-23 01:54:26 +02:00
SWITCH_TO(CommentStart);
}
LibWeb: Checking for "DOCTYPE" should be case insensitive in tokenizer
2020-05-25 19:22:23 +02:00
if (consume_next_if_match("DOCTYPE", CaseSensitivity::CaseInsensitive)) {
LibWeb: Begin work on a spec-compliant HTML parser In order to actually view the web as it is, we're gonna need a proper HTML parser. So let's build one! This patch introduces the Web::HTMLTokenizer class, which currently operates on a StringView input stream where it fetches (ASCII only atm) codepoints and tokenizes acccording to the HTML spec tokenization algo. The tokenizer state machine looks a bit weird but is written in a way that tries to mimic the spec as closely as possible, in order to make development easier and bugs less likely. This initial version is far from finished, but it can parse a trivial document with a DOCTYPE and open/close tags. :^)
2020-05-22 21:46:13 +02:00
SWITCH_TO(DOCTYPE);
}
LibWeb: Make the new HTML parser parse input as UTF-8 We already convert the input to UTF-8 before starting the tokenizer, so all this patch had to do was switch the tokenizer to use an Utf8View for its input (and to emit 32-bit codepoints.)
2020-06-04 21:06:54 +02:00
if (consume_next_if_match("[CDATA[")) {
LibWeb: Implement state switch for "[CDATA[" in HTML parser
2022-02-15 18:52:45 +00:00
// We keep the parser optional so that syntax highlighting can be lexer-only.
// The parser registers itself with the lexer it creates.
if (m_parser != nullptr && m_parser->adjusted_current_node().namespace_() != Namespace::HTML) {
SWITCH_TO(CDATASection);
} else {
create_new_token(HTMLToken::Type::Comment);
m_current_builder.append("[CDATA[");
SWITCH_TO_WITH_UNCLEAN_BUILDER(BogusComment);
}
LibWeb: Make the new HTML parser parse input as UTF-8 We already convert the input to UTF-8 before starting the tokenizer, so all this patch had to do was switch the tokenizer to use an Utf8View for its input (and to emit 32-bit codepoints.)
2020-06-04 21:06:54 +02:00
}
ANYTHING_ELSE
{
LibWeb: Utilize SourceLocation for HTMLTokenizer logging
2021-04-24 20:22:30 -07:00
log_parse_error();
LibWeb: Make the new HTML parser parse input as UTF-8 We already convert the input to UTF-8 before starting the tokenizer, so all this patch had to do was switch the tokenizer to use an Utf8View for its input (and to emit 32-bit codepoints.)
2020-06-04 21:06:54 +02:00
create_new_token(HTMLToken::Type::Comment);
SWITCH_TO(BogusComment);
}
LibWeb: Begin work on a spec-compliant HTML parser In order to actually view the web as it is, we're gonna need a proper HTML parser. So let's build one! This patch introduces the Web::HTMLTokenizer class, which currently operates on a StringView input stream where it fetches (ASCII only atm) codepoints and tokenizes acccording to the HTML spec tokenization algo. The tokenizer state machine looks a bit weird but is written in a way that tries to mimic the spec as closely as possible, in order to make development easier and bugs less likely. This initial version is far from finished, but it can parse a trivial document with a DOCTYPE and open/close tags. :^)
2020-05-22 21:46:13 +02:00
}
END_STATE
LibWeb: Add spec links to each HTML tokenizer state section I didn't add full spec comments this time, but this is better than nothing :^)
2022-02-15 18:12:15 +00:00
// 13.2.5.41 Bogus comment state, https://html.spec.whatwg.org/multipage/parsing.html#bogus-comment-state
LibWeb: Implement more of the tokenizer states Slowly adding more unimplemented options for tokenizer states.
2020-05-28 00:28:32 +02:00
BEGIN_STATE(BogusComment)
{
ON('>')
{
LibWeb: User getter and setter for Comment type HTMLTokens
2021-07-14 23:32:18 +02:00
m_current_token.set_comment(consume_current_builder());
LibWeb: Make the new HTML parser parse input as UTF-8 We already convert the input to UTF-8 before starting the tokenizer, so all this patch had to do was switch the tokenizer to use an Utf8View for its input (and to emit 32-bit codepoints.)
2020-06-04 21:06:54 +02:00
SWITCH_TO_AND_EMIT_CURRENT_TOKEN(Data);
LibWeb: Implement more of the tokenizer states Slowly adding more unimplemented options for tokenizer states.
2020-05-28 00:28:32 +02:00
}
ON_EOF
{
LibWeb: Use move() when enqueuing tokens in HTMLTokenizer We're not using the current token anymore once it's enqueued so let's use move() when enqueuing the tokens.
2021-05-23 12:39:00 +02:00
m_queued_tokens.enqueue(move(m_current_token));
LibWeb: Make the new HTML parser parse input as UTF-8 We already convert the input to UTF-8 before starting the tokenizer, so all this patch had to do was switch the tokenizer to use an Utf8View for its input (and to emit 32-bit codepoints.)
2020-06-04 21:06:54 +02:00
EMIT_EOF;
LibWeb: Implement more of the tokenizer states Slowly adding more unimplemented options for tokenizer states.
2020-05-28 00:28:32 +02:00
}
ON(0)
{
LibWeb: Utilize SourceLocation for HTMLTokenizer logging
2021-04-24 20:22:30 -07:00
log_parse_error();
LibWeb: Remove StringBuilder from HTMLToken::m_comment_or_character
2021-05-23 09:31:40 +02:00
m_current_builder.append_code_point(0xFFFD);
LibWeb: Make the new HTML parser parse input as UTF-8 We already convert the input to UTF-8 before starting the tokenizer, so all this patch had to do was switch the tokenizer to use an Utf8View for its input (and to emit 32-bit codepoints.)
2020-06-04 21:06:54 +02:00
continue;
LibWeb: Implement more of the tokenizer states Slowly adding more unimplemented options for tokenizer states.
2020-05-28 00:28:32 +02:00
}
ANYTHING_ELSE
{
LibWeb: Remove StringBuilder from HTMLToken::m_comment_or_character
2021-05-23 09:31:40 +02:00
m_current_builder.append_code_point(current_input_character.value());
LibWeb: Make the new HTML parser parse input as UTF-8 We already convert the input to UTF-8 before starting the tokenizer, so all this patch had to do was switch the tokenizer to use an Utf8View for its input (and to emit 32-bit codepoints.)
2020-06-04 21:06:54 +02:00
continue;
LibWeb: Implement more of the tokenizer states Slowly adding more unimplemented options for tokenizer states.
2020-05-28 00:28:32 +02:00
}
}
END_STATE
LibWeb: Add spec links to each HTML tokenizer state section I didn't add full spec comments this time, but this is better than nothing :^)
2022-02-15 18:12:15 +00:00
// 13.2.5.53 DOCTYPE state, https://html.spec.whatwg.org/multipage/parsing.html#doctype-state
LibWeb: Begin work on a spec-compliant HTML parser In order to actually view the web as it is, we're gonna need a proper HTML parser. So let's build one! This patch introduces the Web::HTMLTokenizer class, which currently operates on a StringView input stream where it fetches (ASCII only atm) codepoints and tokenizes acccording to the HTML spec tokenization algo. The tokenizer state machine looks a bit weird but is written in a way that tries to mimic the spec as closely as possible, in order to make development easier and bugs less likely. This initial version is far from finished, but it can parse a trivial document with a DOCTYPE and open/close tags. :^)
2020-05-22 21:46:13 +02:00
BEGIN_STATE(DOCTYPE)
{
ON_WHITESPACE
{
SWITCH_TO(BeforeDOCTYPEName);
}
LibWeb: Implement a bit more of DOCTYPE tokenization
2020-05-23 19:56:07 +02:00
ON('>')
{
RECONSUME_IN(BeforeDOCTYPEName);
}
ON_EOF
{
LibWeb: Utilize SourceLocation for HTMLTokenizer logging
2021-04-24 20:22:30 -07:00
log_parse_error();
LibWeb: Fully implement all DOCTYPE tokenizer states Also fixes TagOpen having a seperate emit and reconsume in ANYTHING_ELSE.
2020-06-11 05:00:45 +01:00
create_new_token(HTMLToken::Type::DOCTYPE);
LibWeb: Rename HTMLToken::doctype_data() => ensure_doctype_data() This renames the accessor to better reflect what it does, as this will allocate a DoctypeData struct if there is none.
2021-07-15 18:58:11 +02:00
m_current_token.ensure_doctype_data().force_quirks = true;
LibWeb: Use move() when enqueuing tokens in HTMLTokenizer We're not using the current token anymore once it's enqueued so let's use move() when enqueuing the tokens.
2021-05-23 12:39:00 +02:00
m_queued_tokens.enqueue(move(m_current_token));
LibWeb: Fully implement all DOCTYPE tokenizer states Also fixes TagOpen having a seperate emit and reconsume in ANYTHING_ELSE.
2020-06-11 05:00:45 +01:00
EMIT_EOF;
LibWeb: Implement a bit more of DOCTYPE tokenization
2020-05-23 19:56:07 +02:00
}
ANYTHING_ELSE
{
LibWeb: Utilize SourceLocation for HTMLTokenizer logging
2021-04-24 20:22:30 -07:00
log_parse_error();
LibWeb: Fully implement all DOCTYPE tokenizer states Also fixes TagOpen having a seperate emit and reconsume in ANYTHING_ELSE.
2020-06-11 05:00:45 +01:00
RECONSUME_IN(BeforeDOCTYPEName);
LibWeb: Implement a bit more of DOCTYPE tokenization
2020-05-23 19:56:07 +02:00
}
LibWeb: Begin work on a spec-compliant HTML parser In order to actually view the web as it is, we're gonna need a proper HTML parser. So let's build one! This patch introduces the Web::HTMLTokenizer class, which currently operates on a StringView input stream where it fetches (ASCII only atm) codepoints and tokenizes acccording to the HTML spec tokenization algo. The tokenizer state machine looks a bit weird but is written in a way that tries to mimic the spec as closely as possible, in order to make development easier and bugs less likely. This initial version is far from finished, but it can parse a trivial document with a DOCTYPE and open/close tags. :^)
2020-05-22 21:46:13 +02:00
}
END_STATE
LibWeb: Add spec links to each HTML tokenizer state section I didn't add full spec comments this time, but this is better than nothing :^)
2022-02-15 18:12:15 +00:00
// 13.2.5.54 Before DOCTYPE name state, https://html.spec.whatwg.org/multipage/parsing.html#before-doctype-name-state
LibWeb: Begin work on a spec-compliant HTML parser In order to actually view the web as it is, we're gonna need a proper HTML parser. So let's build one! This patch introduces the Web::HTMLTokenizer class, which currently operates on a StringView input stream where it fetches (ASCII only atm) codepoints and tokenizes acccording to the HTML spec tokenization algo. The tokenizer state machine looks a bit weird but is written in a way that tries to mimic the spec as closely as possible, in order to make development easier and bugs less likely. This initial version is far from finished, but it can parse a trivial document with a DOCTYPE and open/close tags. :^)
2020-05-22 21:46:13 +02:00
BEGIN_STATE(BeforeDOCTYPEName)
{
ON_WHITESPACE
{
continue;
}
LibWeb: Implement a bit more of DOCTYPE tokenization
2020-05-23 19:56:07 +02:00
ON_ASCII_UPPER_ALPHA
{
create_new_token(HTMLToken::Type::DOCTYPE);
LibWeb: Remove StringBuilders from HTMLToken::m_doctype
2021-05-23 08:20:03 +02:00
m_current_builder.append_code_point(to_ascii_lowercase(current_input_character.value()));
LibWeb: Rename HTMLToken::doctype_data() => ensure_doctype_data() This renames the accessor to better reflect what it does, as this will allocate a DoctypeData struct if there is none.
2021-07-15 18:58:11 +02:00
m_current_token.ensure_doctype_data().missing_name = false;
LibWeb: Remove StringBuilders from HTMLToken::m_doctype
2021-05-23 08:20:03 +02:00
SWITCH_TO_WITH_UNCLEAN_BUILDER(DOCTYPEName);
LibWeb: Implement a bit more of DOCTYPE tokenization
2020-05-23 19:56:07 +02:00
}
ON(0)
{
LibWeb: Utilize SourceLocation for HTMLTokenizer logging
2021-04-24 20:22:30 -07:00
log_parse_error();
LibWeb: Fully implement all DOCTYPE tokenizer states Also fixes TagOpen having a seperate emit and reconsume in ANYTHING_ELSE.
2020-06-11 05:00:45 +01:00
create_new_token(HTMLToken::Type::DOCTYPE);
LibWeb: Remove StringBuilders from HTMLToken::m_doctype
2021-05-23 08:20:03 +02:00
m_current_builder.append_code_point(0xFFFD);
LibWeb: Rename HTMLToken::doctype_data() => ensure_doctype_data() This renames the accessor to better reflect what it does, as this will allocate a DoctypeData struct if there is none.
2021-07-15 18:58:11 +02:00
m_current_token.ensure_doctype_data().missing_name = false;
LibWeb: Remove StringBuilders from HTMLToken::m_doctype
2021-05-23 08:20:03 +02:00
SWITCH_TO_WITH_UNCLEAN_BUILDER(DOCTYPEName);
LibWeb: Implement a bit more of DOCTYPE tokenization
2020-05-23 19:56:07 +02:00
}
ON('>')
{
LibWeb: Utilize SourceLocation for HTMLTokenizer logging
2021-04-24 20:22:30 -07:00
log_parse_error();
LibWeb: Fully implement all DOCTYPE tokenizer states Also fixes TagOpen having a seperate emit and reconsume in ANYTHING_ELSE.
2020-06-11 05:00:45 +01:00
create_new_token(HTMLToken::Type::DOCTYPE);
LibWeb: Rename HTMLToken::doctype_data() => ensure_doctype_data() This renames the accessor to better reflect what it does, as this will allocate a DoctypeData struct if there is none.
2021-07-15 18:58:11 +02:00
m_current_token.ensure_doctype_data().force_quirks = true;
LibWeb: Fully implement all DOCTYPE tokenizer states Also fixes TagOpen having a seperate emit and reconsume in ANYTHING_ELSE.
2020-06-11 05:00:45 +01:00
SWITCH_TO_AND_EMIT_CURRENT_TOKEN(Data);
LibWeb: Implement a bit more of DOCTYPE tokenization
2020-05-23 19:56:07 +02:00
}
ON_EOF
{
LibWeb: Utilize SourceLocation for HTMLTokenizer logging
2021-04-24 20:22:30 -07:00
log_parse_error();
LibWeb: Fully implement all DOCTYPE tokenizer states Also fixes TagOpen having a seperate emit and reconsume in ANYTHING_ELSE.
2020-06-11 05:00:45 +01:00
create_new_token(HTMLToken::Type::DOCTYPE);
LibWeb: Rename HTMLToken::doctype_data() => ensure_doctype_data() This renames the accessor to better reflect what it does, as this will allocate a DoctypeData struct if there is none.
2021-07-15 18:58:11 +02:00
m_current_token.ensure_doctype_data().force_quirks = true;
LibWeb: Use move() when enqueuing tokens in HTMLTokenizer We're not using the current token anymore once it's enqueued so let's use move() when enqueuing the tokens.
2021-05-23 12:39:00 +02:00
m_queued_tokens.enqueue(move(m_current_token));
LibWeb: Fully implement all DOCTYPE tokenizer states Also fixes TagOpen having a seperate emit and reconsume in ANYTHING_ELSE.
2020-06-11 05:00:45 +01:00
EMIT_EOF;
LibWeb: Implement a bit more of DOCTYPE tokenization
2020-05-23 19:56:07 +02:00
}
LibWeb: Begin work on a spec-compliant HTML parser In order to actually view the web as it is, we're gonna need a proper HTML parser. So let's build one! This patch introduces the Web::HTMLTokenizer class, which currently operates on a StringView input stream where it fetches (ASCII only atm) codepoints and tokenizes acccording to the HTML spec tokenization algo. The tokenizer state machine looks a bit weird but is written in a way that tries to mimic the spec as closely as possible, in order to make development easier and bugs less likely. This initial version is far from finished, but it can parse a trivial document with a DOCTYPE and open/close tags. :^)
2020-05-22 21:46:13 +02:00
ANYTHING_ELSE
{
create_new_token(HTMLToken::Type::DOCTYPE);
LibWeb: Remove StringBuilders from HTMLToken::m_doctype
2021-05-23 08:20:03 +02:00
m_current_builder.append_code_point(current_input_character.value());
LibWeb: Rename HTMLToken::doctype_data() => ensure_doctype_data() This renames the accessor to better reflect what it does, as this will allocate a DoctypeData struct if there is none.
2021-07-15 18:58:11 +02:00
m_current_token.ensure_doctype_data().missing_name = false;
LibWeb: Remove StringBuilders from HTMLToken::m_doctype
2021-05-23 08:20:03 +02:00
SWITCH_TO_WITH_UNCLEAN_BUILDER(DOCTYPEName);
LibWeb: Begin work on a spec-compliant HTML parser In order to actually view the web as it is, we're gonna need a proper HTML parser. So let's build one! This patch introduces the Web::HTMLTokenizer class, which currently operates on a StringView input stream where it fetches (ASCII only atm) codepoints and tokenizes acccording to the HTML spec tokenization algo. The tokenizer state machine looks a bit weird but is written in a way that tries to mimic the spec as closely as possible, in order to make development easier and bugs less likely. This initial version is far from finished, but it can parse a trivial document with a DOCTYPE and open/close tags. :^)
2020-05-22 21:46:13 +02:00
}
}
END_STATE
LibWeb: Add spec links to each HTML tokenizer state section I didn't add full spec comments this time, but this is better than nothing :^)
2022-02-15 18:12:15 +00:00
// 13.2.5.55 DOCTYPE name state, https://html.spec.whatwg.org/multipage/parsing.html#doctype-name-state
LibWeb: Begin work on a spec-compliant HTML parser In order to actually view the web as it is, we're gonna need a proper HTML parser. So let's build one! This patch introduces the Web::HTMLTokenizer class, which currently operates on a StringView input stream where it fetches (ASCII only atm) codepoints and tokenizes acccording to the HTML spec tokenization algo. The tokenizer state machine looks a bit weird but is written in a way that tries to mimic the spec as closely as possible, in order to make development easier and bugs less likely. This initial version is far from finished, but it can parse a trivial document with a DOCTYPE and open/close tags. :^)
2020-05-22 21:46:13 +02:00
BEGIN_STATE(DOCTYPEName)
{
LibWeb: Implement a bit more of DOCTYPE tokenization
2020-05-23 19:56:07 +02:00
ON_WHITESPACE
{
LibWeb: Rename HTMLToken::doctype_data() => ensure_doctype_data() This renames the accessor to better reflect what it does, as this will allocate a DoctypeData struct if there is none.
2021-07-15 18:58:11 +02:00
m_current_token.ensure_doctype_data().name = consume_current_builder();
LibWeb: Implement a bit more of DOCTYPE tokenization
2020-05-23 19:56:07 +02:00
SWITCH_TO(AfterDOCTYPEName);
}
LibWeb: Begin work on a spec-compliant HTML parser In order to actually view the web as it is, we're gonna need a proper HTML parser. So let's build one! This patch introduces the Web::HTMLTokenizer class, which currently operates on a StringView input stream where it fetches (ASCII only atm) codepoints and tokenizes acccording to the HTML spec tokenization algo. The tokenizer state machine looks a bit weird but is written in a way that tries to mimic the spec as closely as possible, in order to make development easier and bugs less likely. This initial version is far from finished, but it can parse a trivial document with a DOCTYPE and open/close tags. :^)
2020-05-22 21:46:13 +02:00
ON('>')
{
LibWeb: Rename HTMLToken::doctype_data() => ensure_doctype_data() This renames the accessor to better reflect what it does, as this will allocate a DoctypeData struct if there is none.
2021-07-15 18:58:11 +02:00
m_current_token.ensure_doctype_data().name = consume_current_builder();
LibWeb: Start building the tree building part of the new HTML parser This patch adds a new HTMLDocumentParser class. It keeps a tokenizer object internally and feeds itself with one token at a time from it. The names and idioms in this class are expressed as closely to the actual HTML parsing spec as possible, to make development as easy and bug free as possible. :^) This is going to become pretty large, but it's pretty cool!
2020-05-24 00:14:23 +02:00
SWITCH_TO_AND_EMIT_CURRENT_TOKEN(Data);
LibWeb: Begin work on a spec-compliant HTML parser In order to actually view the web as it is, we're gonna need a proper HTML parser. So let's build one! This patch introduces the Web::HTMLTokenizer class, which currently operates on a StringView input stream where it fetches (ASCII only atm) codepoints and tokenizes acccording to the HTML spec tokenization algo. The tokenizer state machine looks a bit weird but is written in a way that tries to mimic the spec as closely as possible, in order to make development easier and bugs less likely. This initial version is far from finished, but it can parse a trivial document with a DOCTYPE and open/close tags. :^)
2020-05-22 21:46:13 +02:00
}
LibWeb: Implement a bit more of DOCTYPE tokenization
2020-05-23 19:56:07 +02:00
ON_ASCII_UPPER_ALPHA
{
LibWeb: Remove StringBuilders from HTMLToken::m_doctype
2021-05-23 08:20:03 +02:00
m_current_builder.append_code_point(to_ascii_lowercase(current_input_character.value()));
LibWeb: Fully implement all DOCTYPE tokenizer states Also fixes TagOpen having a seperate emit and reconsume in ANYTHING_ELSE.
2020-06-11 05:00:45 +01:00
continue;
LibWeb: Implement a bit more of DOCTYPE tokenization
2020-05-23 19:56:07 +02:00
}
ON(0)
{
LibWeb: Utilize SourceLocation for HTMLTokenizer logging
2021-04-24 20:22:30 -07:00
log_parse_error();
LibWeb: Remove StringBuilders from HTMLToken::m_doctype
2021-05-23 08:20:03 +02:00
m_current_builder.append_code_point(0xFFFD);
LibWeb: Fully implement all DOCTYPE tokenizer states Also fixes TagOpen having a seperate emit and reconsume in ANYTHING_ELSE.
2020-06-11 05:00:45 +01:00
continue;
LibWeb: Implement a bit more of DOCTYPE tokenization
2020-05-23 19:56:07 +02:00
}
ON_EOF
{
LibWeb: Utilize SourceLocation for HTMLTokenizer logging
2021-04-24 20:22:30 -07:00
log_parse_error();
LibWeb: Rename HTMLToken::doctype_data() => ensure_doctype_data() This renames the accessor to better reflect what it does, as this will allocate a DoctypeData struct if there is none.
2021-07-15 18:58:11 +02:00
m_current_token.ensure_doctype_data().force_quirks = true;
LibWeb: Use move() when enqueuing tokens in HTMLTokenizer We're not using the current token anymore once it's enqueued so let's use move() when enqueuing the tokens.
2021-05-23 12:39:00 +02:00
m_queued_tokens.enqueue(move(m_current_token));
LibWeb: Fully implement all DOCTYPE tokenizer states Also fixes TagOpen having a seperate emit and reconsume in ANYTHING_ELSE.
2020-06-11 05:00:45 +01:00
EMIT_EOF;
LibWeb: Implement a bit more of DOCTYPE tokenization
2020-05-23 19:56:07 +02:00
}
LibWeb: Begin work on a spec-compliant HTML parser In order to actually view the web as it is, we're gonna need a proper HTML parser. So let's build one! This patch introduces the Web::HTMLTokenizer class, which currently operates on a StringView input stream where it fetches (ASCII only atm) codepoints and tokenizes acccording to the HTML spec tokenization algo. The tokenizer state machine looks a bit weird but is written in a way that tries to mimic the spec as closely as possible, in order to make development easier and bugs less likely. This initial version is far from finished, but it can parse a trivial document with a DOCTYPE and open/close tags. :^)
2020-05-22 21:46:13 +02:00
ANYTHING_ELSE
{
LibWeb: Remove StringBuilders from HTMLToken::m_doctype
2021-05-23 08:20:03 +02:00
m_current_builder.append_code_point(current_input_character.value());
LibWeb: Begin work on a spec-compliant HTML parser In order to actually view the web as it is, we're gonna need a proper HTML parser. So let's build one! This patch introduces the Web::HTMLTokenizer class, which currently operates on a StringView input stream where it fetches (ASCII only atm) codepoints and tokenizes acccording to the HTML spec tokenization algo. The tokenizer state machine looks a bit weird but is written in a way that tries to mimic the spec as closely as possible, in order to make development easier and bugs less likely. This initial version is far from finished, but it can parse a trivial document with a DOCTYPE and open/close tags. :^)
2020-05-22 21:46:13 +02:00
continue;
}
LibWeb: Implement a bit more of DOCTYPE tokenization
2020-05-23 19:56:07 +02:00
}
END_STATE
LibWeb: Add spec links to each HTML tokenizer state section I didn't add full spec comments this time, but this is better than nothing :^)
2022-02-15 18:12:15 +00:00
// 13.2.5.56 After DOCTYPE name state, https://html.spec.whatwg.org/multipage/parsing.html#after-doctype-name-state
LibWeb: Implement a bit more of DOCTYPE tokenization
2020-05-23 19:56:07 +02:00
BEGIN_STATE(AfterDOCTYPEName)
LibWeb: Flesh out the remaining DOCTYPE related tokenizer states We can now parse public and system identifiers! Not super useful, but at least we can do it :^)
2020-05-25 19:50:44 +02:00
{
ON_WHITESPACE
{
continue;
}
ON('>')
{
SWITCH_TO_AND_EMIT_CURRENT_TOKEN(Data);
}
ON_EOF
{
LibWeb: Utilize SourceLocation for HTMLTokenizer logging
2021-04-24 20:22:30 -07:00
log_parse_error();
LibWeb: Rename HTMLToken::doctype_data() => ensure_doctype_data() This renames the accessor to better reflect what it does, as this will allocate a DoctypeData struct if there is none.
2021-07-15 18:58:11 +02:00
m_current_token.ensure_doctype_data().force_quirks = true;
LibWeb: Use move() when enqueuing tokens in HTMLTokenizer We're not using the current token anymore once it's enqueued so let's use move() when enqueuing the tokens.
2021-05-23 12:39:00 +02:00
m_queued_tokens.enqueue(move(m_current_token));
LibWeb: Fully implement all DOCTYPE tokenizer states Also fixes TagOpen having a seperate emit and reconsume in ANYTHING_ELSE.
2020-06-11 05:00:45 +01:00
EMIT_EOF;
LibWeb: Flesh out the remaining DOCTYPE related tokenizer states We can now parse public and system identifiers! Not super useful, but at least we can do it :^)
2020-05-25 19:50:44 +02:00
}
ANYTHING_ELSE
{
Everywhere: Replace ctype.h to avoid narrowing conversions This replaces ctype.h with CharacterType.h everywhere I could find issues with narrowing conversions. While using it will probably make sense almost everywhere in the future, the most critical places should have been addressed.
2021-06-01 21:18:08 +02:00
if (to_ascii_uppercase(current_input_character.value()) == 'P' && consume_next_if_match("UBLIC", CaseSensitivity::CaseInsensitive)) {
LibWeb: Flesh out the remaining DOCTYPE related tokenizer states We can now parse public and system identifiers! Not super useful, but at least we can do it :^)
2020-05-25 19:50:44 +02:00
SWITCH_TO(AfterDOCTYPEPublicKeyword);
}
Everywhere: Replace ctype.h to avoid narrowing conversions This replaces ctype.h with CharacterType.h everywhere I could find issues with narrowing conversions. While using it will probably make sense almost everywhere in the future, the most critical places should have been addressed.
2021-06-01 21:18:08 +02:00
if (to_ascii_uppercase(current_input_character.value()) == 'S' && consume_next_if_match("YSTEM", CaseSensitivity::CaseInsensitive)) {
LibWeb: Flesh out the remaining DOCTYPE related tokenizer states We can now parse public and system identifiers! Not super useful, but at least we can do it :^)
2020-05-25 19:50:44 +02:00
SWITCH_TO(AfterDOCTYPESystemKeyword);
}
LibWeb: Utilize SourceLocation for HTMLTokenizer logging
2021-04-24 20:22:30 -07:00
log_parse_error();
LibWeb: Rename HTMLToken::doctype_data() => ensure_doctype_data() This renames the accessor to better reflect what it does, as this will allocate a DoctypeData struct if there is none.
2021-07-15 18:58:11 +02:00
m_current_token.ensure_doctype_data().force_quirks = true;
LibWeb: Fully implement all DOCTYPE tokenizer states Also fixes TagOpen having a seperate emit and reconsume in ANYTHING_ELSE.
2020-06-11 05:00:45 +01:00
RECONSUME_IN(BogusDOCTYPE);
LibWeb: Flesh out the remaining DOCTYPE related tokenizer states We can now parse public and system identifiers! Not super useful, but at least we can do it :^)
2020-05-25 19:50:44 +02:00
}
}
END_STATE
LibWeb: Add spec links to each HTML tokenizer state section I didn't add full spec comments this time, but this is better than nothing :^)
2022-02-15 18:12:15 +00:00
// 13.2.5.57 After DOCTYPE public keyword state, https://html.spec.whatwg.org/multipage/parsing.html#after-doctype-public-keyword-state
LibWeb: Flesh out the remaining DOCTYPE related tokenizer states We can now parse public and system identifiers! Not super useful, but at least we can do it :^)
2020-05-25 19:50:44 +02:00
BEGIN_STATE(AfterDOCTYPEPublicKeyword)
{
ON_WHITESPACE
{
SWITCH_TO(BeforeDOCTYPEPublicIdentifier);
}
ON('"')
{
LibWeb: Utilize SourceLocation for HTMLTokenizer logging
2021-04-24 20:22:30 -07:00
log_parse_error();
LibWeb: Rename HTMLToken::doctype_data() => ensure_doctype_data() This renames the accessor to better reflect what it does, as this will allocate a DoctypeData struct if there is none.
2021-07-15 18:58:11 +02:00
m_current_token.ensure_doctype_data().missing_public_identifier = false;
LibWeb: Fully implement all DOCTYPE tokenizer states Also fixes TagOpen having a seperate emit and reconsume in ANYTHING_ELSE.
2020-06-11 05:00:45 +01:00
SWITCH_TO(DOCTYPEPublicIdentifierDoubleQuoted);
LibWeb: Flesh out the remaining DOCTYPE related tokenizer states We can now parse public and system identifiers! Not super useful, but at least we can do it :^)
2020-05-25 19:50:44 +02:00
}
ON('\'')
{
LibWeb: Utilize SourceLocation for HTMLTokenizer logging
2021-04-24 20:22:30 -07:00
log_parse_error();
LibWeb: Rename HTMLToken::doctype_data() => ensure_doctype_data() This renames the accessor to better reflect what it does, as this will allocate a DoctypeData struct if there is none.
2021-07-15 18:58:11 +02:00
m_current_token.ensure_doctype_data().missing_public_identifier = false;
LibWeb: Fully implement all DOCTYPE tokenizer states Also fixes TagOpen having a seperate emit and reconsume in ANYTHING_ELSE.
2020-06-11 05:00:45 +01:00
SWITCH_TO(DOCTYPEPublicIdentifierSingleQuoted);
LibWeb: Flesh out the remaining DOCTYPE related tokenizer states We can now parse public and system identifiers! Not super useful, but at least we can do it :^)
2020-05-25 19:50:44 +02:00
}
ON('>')
{
LibWeb: Utilize SourceLocation for HTMLTokenizer logging
2021-04-24 20:22:30 -07:00
log_parse_error();
LibWeb: Rename HTMLToken::doctype_data() => ensure_doctype_data() This renames the accessor to better reflect what it does, as this will allocate a DoctypeData struct if there is none.
2021-07-15 18:58:11 +02:00
m_current_token.ensure_doctype_data().force_quirks = true;
LibWeb: Fully implement all DOCTYPE tokenizer states Also fixes TagOpen having a seperate emit and reconsume in ANYTHING_ELSE.
2020-06-11 05:00:45 +01:00
SWITCH_TO_AND_EMIT_CURRENT_TOKEN(Data);
LibWeb: Flesh out the remaining DOCTYPE related tokenizer states We can now parse public and system identifiers! Not super useful, but at least we can do it :^)
2020-05-25 19:50:44 +02:00
}
ON_EOF
{
LibWeb: Utilize SourceLocation for HTMLTokenizer logging
2021-04-24 20:22:30 -07:00
log_parse_error();
LibWeb: Rename HTMLToken::doctype_data() => ensure_doctype_data() This renames the accessor to better reflect what it does, as this will allocate a DoctypeData struct if there is none.
2021-07-15 18:58:11 +02:00
m_current_token.ensure_doctype_data().force_quirks = true;
LibWeb: Use move() when enqueuing tokens in HTMLTokenizer We're not using the current token anymore once it's enqueued so let's use move() when enqueuing the tokens.
2021-05-23 12:39:00 +02:00
m_queued_tokens.enqueue(move(m_current_token));
LibWeb: Fully implement all DOCTYPE tokenizer states Also fixes TagOpen having a seperate emit and reconsume in ANYTHING_ELSE.
2020-06-11 05:00:45 +01:00
EMIT_EOF;
LibWeb: Flesh out the remaining DOCTYPE related tokenizer states We can now parse public and system identifiers! Not super useful, but at least we can do it :^)
2020-05-25 19:50:44 +02:00
}
ANYTHING_ELSE
{
LibWeb: Utilize SourceLocation for HTMLTokenizer logging
2021-04-24 20:22:30 -07:00
log_parse_error();
LibWeb: Rename HTMLToken::doctype_data() => ensure_doctype_data() This renames the accessor to better reflect what it does, as this will allocate a DoctypeData struct if there is none.
2021-07-15 18:58:11 +02:00
m_current_token.ensure_doctype_data().force_quirks = true;
LibWeb: Fully implement all DOCTYPE tokenizer states Also fixes TagOpen having a seperate emit and reconsume in ANYTHING_ELSE.
2020-06-11 05:00:45 +01:00
RECONSUME_IN(BogusDOCTYPE);
LibWeb: Flesh out the remaining DOCTYPE related tokenizer states We can now parse public and system identifiers! Not super useful, but at least we can do it :^)
2020-05-25 19:50:44 +02:00
}
}
END_STATE
LibWeb: Add spec links to each HTML tokenizer state section I didn't add full spec comments this time, but this is better than nothing :^)
2022-02-15 18:12:15 +00:00
// 13.2.5.63 After DOCTYPE system keyword state, https://html.spec.whatwg.org/multipage/parsing.html#after-doctype-system-keyword-state
LibWeb: Flesh out the remaining DOCTYPE related tokenizer states We can now parse public and system identifiers! Not super useful, but at least we can do it :^)
2020-05-25 19:50:44 +02:00
BEGIN_STATE(AfterDOCTYPESystemKeyword)
{
ON_WHITESPACE
{
SWITCH_TO(BeforeDOCTYPESystemIdentifier);
}
ON('"')
{
LibWeb: Utilize SourceLocation for HTMLTokenizer logging
2021-04-24 20:22:30 -07:00
log_parse_error();
LibWeb: Rename HTMLToken::doctype_data() => ensure_doctype_data() This renames the accessor to better reflect what it does, as this will allocate a DoctypeData struct if there is none.
2021-07-15 18:58:11 +02:00
m_current_token.ensure_doctype_data().system_identifier = {};
m_current_token.ensure_doctype_data().missing_system_identifier = false;
LibWeb: Fully implement all DOCTYPE tokenizer states Also fixes TagOpen having a seperate emit and reconsume in ANYTHING_ELSE.
2020-06-11 05:00:45 +01:00
SWITCH_TO(DOCTYPESystemIdentifierDoubleQuoted);
LibWeb: Flesh out the remaining DOCTYPE related tokenizer states We can now parse public and system identifiers! Not super useful, but at least we can do it :^)
2020-05-25 19:50:44 +02:00
}
ON('\'')
{
LibWeb: Utilize SourceLocation for HTMLTokenizer logging
2021-04-24 20:22:30 -07:00
log_parse_error();
LibWeb: Rename HTMLToken::doctype_data() => ensure_doctype_data() This renames the accessor to better reflect what it does, as this will allocate a DoctypeData struct if there is none.
2021-07-15 18:58:11 +02:00
m_current_token.ensure_doctype_data().system_identifier = {};
m_current_token.ensure_doctype_data().missing_system_identifier = false;
LibWeb: Fully implement all DOCTYPE tokenizer states Also fixes TagOpen having a seperate emit and reconsume in ANYTHING_ELSE.
2020-06-11 05:00:45 +01:00
SWITCH_TO(DOCTYPESystemIdentifierSingleQuoted);
LibWeb: Flesh out the remaining DOCTYPE related tokenizer states We can now parse public and system identifiers! Not super useful, but at least we can do it :^)
2020-05-25 19:50:44 +02:00
}
ON('>')
{
LibWeb: Utilize SourceLocation for HTMLTokenizer logging
2021-04-24 20:22:30 -07:00
log_parse_error();
LibWeb: Rename HTMLToken::doctype_data() => ensure_doctype_data() This renames the accessor to better reflect what it does, as this will allocate a DoctypeData struct if there is none.
2021-07-15 18:58:11 +02:00
m_current_token.ensure_doctype_data().force_quirks = true;
LibWeb: Fully implement all DOCTYPE tokenizer states Also fixes TagOpen having a seperate emit and reconsume in ANYTHING_ELSE.
2020-06-11 05:00:45 +01:00
SWITCH_TO_AND_EMIT_CURRENT_TOKEN(Data);
LibWeb: Flesh out the remaining DOCTYPE related tokenizer states We can now parse public and system identifiers! Not super useful, but at least we can do it :^)
2020-05-25 19:50:44 +02:00
}
ON_EOF
{
LibWeb: Utilize SourceLocation for HTMLTokenizer logging
2021-04-24 20:22:30 -07:00
log_parse_error();
LibWeb: Rename HTMLToken::doctype_data() => ensure_doctype_data() This renames the accessor to better reflect what it does, as this will allocate a DoctypeData struct if there is none.
2021-07-15 18:58:11 +02:00
m_current_token.ensure_doctype_data().force_quirks = true;
LibWeb: Use move() when enqueuing tokens in HTMLTokenizer We're not using the current token anymore once it's enqueued so let's use move() when enqueuing the tokens.
2021-05-23 12:39:00 +02:00
m_queued_tokens.enqueue(move(m_current_token));
LibWeb: Fully implement all DOCTYPE tokenizer states Also fixes TagOpen having a seperate emit and reconsume in ANYTHING_ELSE.
2020-06-11 05:00:45 +01:00
EMIT_EOF;
LibWeb: Flesh out the remaining DOCTYPE related tokenizer states We can now parse public and system identifiers! Not super useful, but at least we can do it :^)
2020-05-25 19:50:44 +02:00
}
ANYTHING_ELSE
{
LibWeb: Utilize SourceLocation for HTMLTokenizer logging
2021-04-24 20:22:30 -07:00
log_parse_error();
LibWeb: Rename HTMLToken::doctype_data() => ensure_doctype_data() This renames the accessor to better reflect what it does, as this will allocate a DoctypeData struct if there is none.
2021-07-15 18:58:11 +02:00
m_current_token.ensure_doctype_data().force_quirks = true;
LibWeb: Fully implement all DOCTYPE tokenizer states Also fixes TagOpen having a seperate emit and reconsume in ANYTHING_ELSE.
2020-06-11 05:00:45 +01:00
RECONSUME_IN(BogusDOCTYPE);
LibWeb: Flesh out the remaining DOCTYPE related tokenizer states We can now parse public and system identifiers! Not super useful, but at least we can do it :^)
2020-05-25 19:50:44 +02:00
}
}
END_STATE
LibWeb: Add spec links to each HTML tokenizer state section I didn't add full spec comments this time, but this is better than nothing :^)
2022-02-15 18:12:15 +00:00
// 13.2.5.58 Before DOCTYPE public identifier state, https://html.spec.whatwg.org/multipage/parsing.html#before-doctype-public-identifier-state
LibWeb: Flesh out the remaining DOCTYPE related tokenizer states We can now parse public and system identifiers! Not super useful, but at least we can do it :^)
2020-05-25 19:50:44 +02:00
BEGIN_STATE(BeforeDOCTYPEPublicIdentifier)
{
ON_WHITESPACE
{
continue;
}
ON('"')
{
LibWeb: Rename HTMLToken::doctype_data() => ensure_doctype_data() This renames the accessor to better reflect what it does, as this will allocate a DoctypeData struct if there is none.
2021-07-15 18:58:11 +02:00
m_current_token.ensure_doctype_data().missing_public_identifier = false;
LibWeb: Flesh out the remaining DOCTYPE related tokenizer states We can now parse public and system identifiers! Not super useful, but at least we can do it :^)
2020-05-25 19:50:44 +02:00
SWITCH_TO(DOCTYPEPublicIdentifierDoubleQuoted);
}
ON('\'')
{
LibWeb: Rename HTMLToken::doctype_data() => ensure_doctype_data() This renames the accessor to better reflect what it does, as this will allocate a DoctypeData struct if there is none.
2021-07-15 18:58:11 +02:00
m_current_token.ensure_doctype_data().missing_public_identifier = false;
LibWeb: Flesh out the remaining DOCTYPE related tokenizer states We can now parse public and system identifiers! Not super useful, but at least we can do it :^)
2020-05-25 19:50:44 +02:00
SWITCH_TO(DOCTYPEPublicIdentifierSingleQuoted);
}
ON('>')
{
LibWeb: Utilize SourceLocation for HTMLTokenizer logging
2021-04-24 20:22:30 -07:00
log_parse_error();
LibWeb: Rename HTMLToken::doctype_data() => ensure_doctype_data() This renames the accessor to better reflect what it does, as this will allocate a DoctypeData struct if there is none.
2021-07-15 18:58:11 +02:00
m_current_token.ensure_doctype_data().force_quirks = true;
LibWeb: Fully implement all DOCTYPE tokenizer states Also fixes TagOpen having a seperate emit and reconsume in ANYTHING_ELSE.
2020-06-11 05:00:45 +01:00
SWITCH_TO_AND_EMIT_CURRENT_TOKEN(Data);
LibWeb: Flesh out the remaining DOCTYPE related tokenizer states We can now parse public and system identifiers! Not super useful, but at least we can do it :^)
2020-05-25 19:50:44 +02:00
}
ON_EOF
{
LibWeb: Utilize SourceLocation for HTMLTokenizer logging
2021-04-24 20:22:30 -07:00
log_parse_error();
LibWeb: Rename HTMLToken::doctype_data() => ensure_doctype_data() This renames the accessor to better reflect what it does, as this will allocate a DoctypeData struct if there is none.
2021-07-15 18:58:11 +02:00
m_current_token.ensure_doctype_data().force_quirks = true;
LibWeb: Use move() when enqueuing tokens in HTMLTokenizer We're not using the current token anymore once it's enqueued so let's use move() when enqueuing the tokens.
2021-05-23 12:39:00 +02:00
m_queued_tokens.enqueue(move(m_current_token));
LibWeb: Fully implement all DOCTYPE tokenizer states Also fixes TagOpen having a seperate emit and reconsume in ANYTHING_ELSE.
2020-06-11 05:00:45 +01:00
EMIT_EOF;
LibWeb: Flesh out the remaining DOCTYPE related tokenizer states We can now parse public and system identifiers! Not super useful, but at least we can do it :^)
2020-05-25 19:50:44 +02:00
}
ANYTHING_ELSE
{
LibWeb: Utilize SourceLocation for HTMLTokenizer logging
2021-04-24 20:22:30 -07:00
log_parse_error();
LibWeb: Rename HTMLToken::doctype_data() => ensure_doctype_data() This renames the accessor to better reflect what it does, as this will allocate a DoctypeData struct if there is none.
2021-07-15 18:58:11 +02:00
m_current_token.ensure_doctype_data().force_quirks = true;
LibWeb: Fully implement all DOCTYPE tokenizer states Also fixes TagOpen having a seperate emit and reconsume in ANYTHING_ELSE.
2020-06-11 05:00:45 +01:00
RECONSUME_IN(BogusDOCTYPE);
LibWeb: Flesh out the remaining DOCTYPE related tokenizer states We can now parse public and system identifiers! Not super useful, but at least we can do it :^)
2020-05-25 19:50:44 +02:00
}
}
END_STATE
LibWeb: Add spec links to each HTML tokenizer state section I didn't add full spec comments this time, but this is better than nothing :^)
2022-02-15 18:12:15 +00:00
// 13.2.5.64 Before DOCTYPE system identifier state, https://html.spec.whatwg.org/multipage/parsing.html#before-doctype-system-identifier-state
LibWeb: Flesh out the remaining DOCTYPE related tokenizer states We can now parse public and system identifiers! Not super useful, but at least we can do it :^)
2020-05-25 19:50:44 +02:00
BEGIN_STATE(BeforeDOCTYPESystemIdentifier)
{
ON_WHITESPACE
{
continue;
}
ON('"')
{
LibWeb: Rename HTMLToken::doctype_data() => ensure_doctype_data() This renames the accessor to better reflect what it does, as this will allocate a DoctypeData struct if there is none.
2021-07-15 18:58:11 +02:00
m_current_token.ensure_doctype_data().missing_system_identifier = false;
LibWeb: Flesh out the remaining DOCTYPE related tokenizer states We can now parse public and system identifiers! Not super useful, but at least we can do it :^)
2020-05-25 19:50:44 +02:00
SWITCH_TO(DOCTYPESystemIdentifierDoubleQuoted);
}
ON('\'')
{
LibWeb: Rename HTMLToken::doctype_data() => ensure_doctype_data() This renames the accessor to better reflect what it does, as this will allocate a DoctypeData struct if there is none.
2021-07-15 18:58:11 +02:00
m_current_token.ensure_doctype_data().missing_system_identifier = false;
LibWeb: Flesh out the remaining DOCTYPE related tokenizer states We can now parse public and system identifiers! Not super useful, but at least we can do it :^)
2020-05-25 19:50:44 +02:00
SWITCH_TO(DOCTYPESystemIdentifierSingleQuoted);
}
ON('>')
{
LibWeb: Utilize SourceLocation for HTMLTokenizer logging
2021-04-24 20:22:30 -07:00
log_parse_error();
LibWeb: Rename HTMLToken::doctype_data() => ensure_doctype_data() This renames the accessor to better reflect what it does, as this will allocate a DoctypeData struct if there is none.
2021-07-15 18:58:11 +02:00
m_current_token.ensure_doctype_data().force_quirks = true;
LibWeb: Fully implement all DOCTYPE tokenizer states Also fixes TagOpen having a seperate emit and reconsume in ANYTHING_ELSE.
2020-06-11 05:00:45 +01:00
SWITCH_TO_AND_EMIT_CURRENT_TOKEN(Data);
LibWeb: Flesh out the remaining DOCTYPE related tokenizer states We can now parse public and system identifiers! Not super useful, but at least we can do it :^)
2020-05-25 19:50:44 +02:00
}
ON_EOF
{
LibWeb: Utilize SourceLocation for HTMLTokenizer logging
2021-04-24 20:22:30 -07:00
log_parse_error();
LibWeb: Rename HTMLToken::doctype_data() => ensure_doctype_data() This renames the accessor to better reflect what it does, as this will allocate a DoctypeData struct if there is none.
2021-07-15 18:58:11 +02:00
m_current_token.ensure_doctype_data().force_quirks = true;
LibWeb: Use move() when enqueuing tokens in HTMLTokenizer We're not using the current token anymore once it's enqueued so let's use move() when enqueuing the tokens.
2021-05-23 12:39:00 +02:00
m_queued_tokens.enqueue(move(m_current_token));
LibWeb: Fully implement all DOCTYPE tokenizer states Also fixes TagOpen having a seperate emit and reconsume in ANYTHING_ELSE.
2020-06-11 05:00:45 +01:00
EMIT_EOF;
LibWeb: Flesh out the remaining DOCTYPE related tokenizer states We can now parse public and system identifiers! Not super useful, but at least we can do it :^)
2020-05-25 19:50:44 +02:00
}
ANYTHING_ELSE
{
LibWeb: Utilize SourceLocation for HTMLTokenizer logging
2021-04-24 20:22:30 -07:00
log_parse_error();
LibWeb: Rename HTMLToken::doctype_data() => ensure_doctype_data() This renames the accessor to better reflect what it does, as this will allocate a DoctypeData struct if there is none.
2021-07-15 18:58:11 +02:00
m_current_token.ensure_doctype_data().force_quirks = true;
LibWeb: Fully implement all DOCTYPE tokenizer states Also fixes TagOpen having a seperate emit and reconsume in ANYTHING_ELSE.
2020-06-11 05:00:45 +01:00
RECONSUME_IN(BogusDOCTYPE);
LibWeb: Flesh out the remaining DOCTYPE related tokenizer states We can now parse public and system identifiers! Not super useful, but at least we can do it :^)
2020-05-25 19:50:44 +02:00
}
}
END_STATE
LibWeb: Add spec links to each HTML tokenizer state section I didn't add full spec comments this time, but this is better than nothing :^)
2022-02-15 18:12:15 +00:00
// 13.2.5.59 DOCTYPE public identifier (double-quoted) state, https://html.spec.whatwg.org/multipage/parsing.html#doctype-public-identifier-(double-quoted)-state
LibWeb: Flesh out the remaining DOCTYPE related tokenizer states We can now parse public and system identifiers! Not super useful, but at least we can do it :^)
2020-05-25 19:50:44 +02:00
BEGIN_STATE(DOCTYPEPublicIdentifierDoubleQuoted)
{
ON('"')
{
LibWeb: Rename HTMLToken::doctype_data() => ensure_doctype_data() This renames the accessor to better reflect what it does, as this will allocate a DoctypeData struct if there is none.
2021-07-15 18:58:11 +02:00
m_current_token.ensure_doctype_data().public_identifier = consume_current_builder();
LibWeb: Flesh out the remaining DOCTYPE related tokenizer states We can now parse public and system identifiers! Not super useful, but at least we can do it :^)
2020-05-25 19:50:44 +02:00
SWITCH_TO(AfterDOCTYPEPublicIdentifier);
}
ON(0)
{
LibWeb: Utilize SourceLocation for HTMLTokenizer logging
2021-04-24 20:22:30 -07:00
log_parse_error();
LibWeb: Remove StringBuilders from HTMLToken::m_doctype
2021-05-23 08:20:03 +02:00
m_current_builder.append_code_point(0xFFFD);
LibWeb: Fully implement all DOCTYPE tokenizer states Also fixes TagOpen having a seperate emit and reconsume in ANYTHING_ELSE.
2020-06-11 05:00:45 +01:00
continue;
LibWeb: Flesh out the remaining DOCTYPE related tokenizer states We can now parse public and system identifiers! Not super useful, but at least we can do it :^)
2020-05-25 19:50:44 +02:00
}
ON('>')
{
LibWeb: Utilize SourceLocation for HTMLTokenizer logging
2021-04-24 20:22:30 -07:00
log_parse_error();
LibWeb: Rename HTMLToken::doctype_data() => ensure_doctype_data() This renames the accessor to better reflect what it does, as this will allocate a DoctypeData struct if there is none.
2021-07-15 18:58:11 +02:00
m_current_token.ensure_doctype_data().public_identifier = consume_current_builder();
m_current_token.ensure_doctype_data().force_quirks = true;
LibWeb: Fully implement all DOCTYPE tokenizer states Also fixes TagOpen having a seperate emit and reconsume in ANYTHING_ELSE.
2020-06-11 05:00:45 +01:00
SWITCH_TO_AND_EMIT_CURRENT_TOKEN(Data);
LibWeb: Flesh out the remaining DOCTYPE related tokenizer states We can now parse public and system identifiers! Not super useful, but at least we can do it :^)
2020-05-25 19:50:44 +02:00
}
ON_EOF
{
LibWeb: Utilize SourceLocation for HTMLTokenizer logging
2021-04-24 20:22:30 -07:00
log_parse_error();
LibWeb: Rename HTMLToken::doctype_data() => ensure_doctype_data() This renames the accessor to better reflect what it does, as this will allocate a DoctypeData struct if there is none.
2021-07-15 18:58:11 +02:00
m_current_token.ensure_doctype_data().force_quirks = true;
LibWeb: Use move() when enqueuing tokens in HTMLTokenizer We're not using the current token anymore once it's enqueued so let's use move() when enqueuing the tokens.
2021-05-23 12:39:00 +02:00
m_queued_tokens.enqueue(move(m_current_token));
LibWeb: Fully implement all DOCTYPE tokenizer states Also fixes TagOpen having a seperate emit and reconsume in ANYTHING_ELSE.
2020-06-11 05:00:45 +01:00
EMIT_EOF;
LibWeb: Flesh out the remaining DOCTYPE related tokenizer states We can now parse public and system identifiers! Not super useful, but at least we can do it :^)
2020-05-25 19:50:44 +02:00
}
ANYTHING_ELSE
{
LibWeb: Remove StringBuilders from HTMLToken::m_doctype
2021-05-23 08:20:03 +02:00
m_current_builder.append_code_point(current_input_character.value());
LibWeb: Flesh out the remaining DOCTYPE related tokenizer states We can now parse public and system identifiers! Not super useful, but at least we can do it :^)
2020-05-25 19:50:44 +02:00
continue;
}
}
END_STATE
LibWeb: Add spec links to each HTML tokenizer state section I didn't add full spec comments this time, but this is better than nothing :^)
2022-02-15 18:12:15 +00:00
// 13.2.5.60 DOCTYPE public identifier (single-quoted) state, https://html.spec.whatwg.org/multipage/parsing.html#doctype-public-identifier-(single-quoted)-state
LibWeb: Flesh out the remaining DOCTYPE related tokenizer states We can now parse public and system identifiers! Not super useful, but at least we can do it :^)
2020-05-25 19:50:44 +02:00
BEGIN_STATE(DOCTYPEPublicIdentifierSingleQuoted)
{
ON('\'')
{
LibWeb: Rename HTMLToken::doctype_data() => ensure_doctype_data() This renames the accessor to better reflect what it does, as this will allocate a DoctypeData struct if there is none.
2021-07-15 18:58:11 +02:00
m_current_token.ensure_doctype_data().public_identifier = consume_current_builder();
LibWeb: Flesh out the remaining DOCTYPE related tokenizer states We can now parse public and system identifiers! Not super useful, but at least we can do it :^)
2020-05-25 19:50:44 +02:00
SWITCH_TO(AfterDOCTYPEPublicIdentifier);
}
ON(0)
{
LibWeb: Utilize SourceLocation for HTMLTokenizer logging
2021-04-24 20:22:30 -07:00
log_parse_error();
LibWeb: Remove StringBuilders from HTMLToken::m_doctype
2021-05-23 08:20:03 +02:00
m_current_builder.append_code_point(0xFFFD);
LibWeb: Fully implement all DOCTYPE tokenizer states Also fixes TagOpen having a seperate emit and reconsume in ANYTHING_ELSE.
2020-06-11 05:00:45 +01:00
continue;
LibWeb: Flesh out the remaining DOCTYPE related tokenizer states We can now parse public and system identifiers! Not super useful, but at least we can do it :^)
2020-05-25 19:50:44 +02:00
}
ON('>')
{
LibWeb: Utilize SourceLocation for HTMLTokenizer logging
2021-04-24 20:22:30 -07:00
log_parse_error();
LibWeb: Rename HTMLToken::doctype_data() => ensure_doctype_data() This renames the accessor to better reflect what it does, as this will allocate a DoctypeData struct if there is none.
2021-07-15 18:58:11 +02:00
m_current_token.ensure_doctype_data().public_identifier = consume_current_builder();
m_current_token.ensure_doctype_data().force_quirks = true;
LibWeb: Fully implement all DOCTYPE tokenizer states Also fixes TagOpen having a seperate emit and reconsume in ANYTHING_ELSE.
2020-06-11 05:00:45 +01:00
SWITCH_TO_AND_EMIT_CURRENT_TOKEN(Data);
LibWeb: Flesh out the remaining DOCTYPE related tokenizer states We can now parse public and system identifiers! Not super useful, but at least we can do it :^)
2020-05-25 19:50:44 +02:00
}
ON_EOF
{
LibWeb: Utilize SourceLocation for HTMLTokenizer logging
2021-04-24 20:22:30 -07:00
log_parse_error();
LibWeb: Rename HTMLToken::doctype_data() => ensure_doctype_data() This renames the accessor to better reflect what it does, as this will allocate a DoctypeData struct if there is none.
2021-07-15 18:58:11 +02:00
m_current_token.ensure_doctype_data().force_quirks = true;
LibWeb: Use move() when enqueuing tokens in HTMLTokenizer We're not using the current token anymore once it's enqueued so let's use move() when enqueuing the tokens.
2021-05-23 12:39:00 +02:00
m_queued_tokens.enqueue(move(m_current_token));
LibWeb: Fully implement all DOCTYPE tokenizer states Also fixes TagOpen having a seperate emit and reconsume in ANYTHING_ELSE.
2020-06-11 05:00:45 +01:00
EMIT_EOF;
LibWeb: Flesh out the remaining DOCTYPE related tokenizer states We can now parse public and system identifiers! Not super useful, but at least we can do it :^)
2020-05-25 19:50:44 +02:00
}
ANYTHING_ELSE
{
LibWeb: Remove StringBuilders from HTMLToken::m_doctype
2021-05-23 08:20:03 +02:00
m_current_builder.append_code_point(current_input_character.value());
LibWeb: Flesh out the remaining DOCTYPE related tokenizer states We can now parse public and system identifiers! Not super useful, but at least we can do it :^)
2020-05-25 19:50:44 +02:00
continue;
}
}
END_STATE
LibWeb: Add spec links to each HTML tokenizer state section I didn't add full spec comments this time, but this is better than nothing :^)
2022-02-15 18:12:15 +00:00
// 13.2.5.65 DOCTYPE system identifier (double-quoted) state, https://html.spec.whatwg.org/multipage/parsing.html#doctype-system-identifier-(double-quoted)-state
LibWeb: Flesh out the remaining DOCTYPE related tokenizer states We can now parse public and system identifiers! Not super useful, but at least we can do it :^)
2020-05-25 19:50:44 +02:00
BEGIN_STATE(DOCTYPESystemIdentifierDoubleQuoted)
{
ON('"')
{
LibWeb: Fix issue where double-quoted doctype system ID was not captured We were storing double-quoted system ID's in the public ID field. 1% progression on ACID3. :^)
2022-03-02 12:26:31 +01:00
m_current_token.ensure_doctype_data().system_identifier = consume_current_builder();
LibWeb: Flesh out the remaining DOCTYPE related tokenizer states We can now parse public and system identifiers! Not super useful, but at least we can do it :^)
2020-05-25 19:50:44 +02:00
SWITCH_TO(AfterDOCTYPESystemIdentifier);
}
ON(0)
{
LibWeb: Utilize SourceLocation for HTMLTokenizer logging
2021-04-24 20:22:30 -07:00
log_parse_error();
LibWeb: Remove StringBuilders from HTMLToken::m_doctype
2021-05-23 08:20:03 +02:00
m_current_builder.append_code_point(0xFFFD);
LibWeb: Fully implement all DOCTYPE tokenizer states Also fixes TagOpen having a seperate emit and reconsume in ANYTHING_ELSE.
2020-06-11 05:00:45 +01:00
continue;
LibWeb: Flesh out the remaining DOCTYPE related tokenizer states We can now parse public and system identifiers! Not super useful, but at least we can do it :^)
2020-05-25 19:50:44 +02:00
}
ON('>')
{
LibWeb: Utilize SourceLocation for HTMLTokenizer logging
2021-04-24 20:22:30 -07:00
log_parse_error();
LibWeb: Fix issue where double-quoted doctype system ID was not captured We were storing double-quoted system ID's in the public ID field. 1% progression on ACID3. :^)
2022-03-02 12:26:31 +01:00
m_current_token.ensure_doctype_data().system_identifier = consume_current_builder();
LibWeb: Rename HTMLToken::doctype_data() => ensure_doctype_data() This renames the accessor to better reflect what it does, as this will allocate a DoctypeData struct if there is none.
2021-07-15 18:58:11 +02:00
m_current_token.ensure_doctype_data().force_quirks = true;
LibWeb: Fully implement all DOCTYPE tokenizer states Also fixes TagOpen having a seperate emit and reconsume in ANYTHING_ELSE.
2020-06-11 05:00:45 +01:00
SWITCH_TO_AND_EMIT_CURRENT_TOKEN(Data);
LibWeb: Flesh out the remaining DOCTYPE related tokenizer states We can now parse public and system identifiers! Not super useful, but at least we can do it :^)
2020-05-25 19:50:44 +02:00
}
ON_EOF
{
LibWeb: Utilize SourceLocation for HTMLTokenizer logging
2021-04-24 20:22:30 -07:00
log_parse_error();
LibWeb: Rename HTMLToken::doctype_data() => ensure_doctype_data() This renames the accessor to better reflect what it does, as this will allocate a DoctypeData struct if there is none.
2021-07-15 18:58:11 +02:00
m_current_token.ensure_doctype_data().force_quirks = true;
LibWeb: Use move() when enqueuing tokens in HTMLTokenizer We're not using the current token anymore once it's enqueued so let's use move() when enqueuing the tokens.
2021-05-23 12:39:00 +02:00
m_queued_tokens.enqueue(move(m_current_token));
LibWeb: Fully implement all DOCTYPE tokenizer states Also fixes TagOpen having a seperate emit and reconsume in ANYTHING_ELSE.
2020-06-11 05:00:45 +01:00
EMIT_EOF;
LibWeb: Flesh out the remaining DOCTYPE related tokenizer states We can now parse public and system identifiers! Not super useful, but at least we can do it :^)
2020-05-25 19:50:44 +02:00
}
ANYTHING_ELSE
{
LibWeb: Remove StringBuilders from HTMLToken::m_doctype
2021-05-23 08:20:03 +02:00
m_current_builder.append_code_point(current_input_character.value());
LibWeb: Flesh out the remaining DOCTYPE related tokenizer states We can now parse public and system identifiers! Not super useful, but at least we can do it :^)
2020-05-25 19:50:44 +02:00
continue;
}
}
END_STATE
LibWeb: Add spec links to each HTML tokenizer state section I didn't add full spec comments this time, but this is better than nothing :^)
2022-02-15 18:12:15 +00:00
// 13.2.5.66 DOCTYPE system identifier (single-quoted) state, https://html.spec.whatwg.org/multipage/parsing.html#doctype-system-identifier-(single-quoted)-state
LibWeb: Flesh out the remaining DOCTYPE related tokenizer states We can now parse public and system identifiers! Not super useful, but at least we can do it :^)
2020-05-25 19:50:44 +02:00
BEGIN_STATE(DOCTYPESystemIdentifierSingleQuoted)
{
ON('\'')
{
LibWeb: Rename HTMLToken::doctype_data() => ensure_doctype_data() This renames the accessor to better reflect what it does, as this will allocate a DoctypeData struct if there is none.
2021-07-15 18:58:11 +02:00
m_current_token.ensure_doctype_data().system_identifier = consume_current_builder();
LibWeb: Flesh out the remaining DOCTYPE related tokenizer states We can now parse public and system identifiers! Not super useful, but at least we can do it :^)
2020-05-25 19:50:44 +02:00
SWITCH_TO(AfterDOCTYPESystemIdentifier);
}
ON(0)
{
LibWeb: Utilize SourceLocation for HTMLTokenizer logging
2021-04-24 20:22:30 -07:00
log_parse_error();
LibWeb: Remove StringBuilders from HTMLToken::m_doctype
2021-05-23 08:20:03 +02:00
m_current_builder.append_code_point(0xFFFD);
LibWeb: Fully implement all DOCTYPE tokenizer states Also fixes TagOpen having a seperate emit and reconsume in ANYTHING_ELSE.
2020-06-11 05:00:45 +01:00
continue;
LibWeb: Flesh out the remaining DOCTYPE related tokenizer states We can now parse public and system identifiers! Not super useful, but at least we can do it :^)
2020-05-25 19:50:44 +02:00
}
ON('>')
{
LibWeb: Utilize SourceLocation for HTMLTokenizer logging
2021-04-24 20:22:30 -07:00
log_parse_error();
LibWeb: Rename HTMLToken::doctype_data() => ensure_doctype_data() This renames the accessor to better reflect what it does, as this will allocate a DoctypeData struct if there is none.
2021-07-15 18:58:11 +02:00
m_current_token.ensure_doctype_data().system_identifier = consume_current_builder();
m_current_token.ensure_doctype_data().force_quirks = true;
LibWeb: Fully implement all DOCTYPE tokenizer states Also fixes TagOpen having a seperate emit and reconsume in ANYTHING_ELSE.
2020-06-11 05:00:45 +01:00
SWITCH_TO_AND_EMIT_CURRENT_TOKEN(Data);
LibWeb: Flesh out the remaining DOCTYPE related tokenizer states We can now parse public and system identifiers! Not super useful, but at least we can do it :^)
2020-05-25 19:50:44 +02:00
}
ON_EOF
{
LibWeb: Utilize SourceLocation for HTMLTokenizer logging
2021-04-24 20:22:30 -07:00
log_parse_error();
LibWeb: Rename HTMLToken::doctype_data() => ensure_doctype_data() This renames the accessor to better reflect what it does, as this will allocate a DoctypeData struct if there is none.
2021-07-15 18:58:11 +02:00
m_current_token.ensure_doctype_data().force_quirks = true;
LibWeb: Use move() when enqueuing tokens in HTMLTokenizer We're not using the current token anymore once it's enqueued so let's use move() when enqueuing the tokens.
2021-05-23 12:39:00 +02:00
m_queued_tokens.enqueue(move(m_current_token));
LibWeb: Fully implement all DOCTYPE tokenizer states Also fixes TagOpen having a seperate emit and reconsume in ANYTHING_ELSE.
2020-06-11 05:00:45 +01:00
EMIT_EOF;
LibWeb: Flesh out the remaining DOCTYPE related tokenizer states We can now parse public and system identifiers! Not super useful, but at least we can do it :^)
2020-05-25 19:50:44 +02:00
}
ANYTHING_ELSE
{
LibWeb: Remove StringBuilders from HTMLToken::m_doctype
2021-05-23 08:20:03 +02:00
m_current_builder.append_code_point(current_input_character.value());
LibWeb: Flesh out the remaining DOCTYPE related tokenizer states We can now parse public and system identifiers! Not super useful, but at least we can do it :^)
2020-05-25 19:50:44 +02:00
continue;
}
}
END_STATE
LibWeb: Add spec links to each HTML tokenizer state section I didn't add full spec comments this time, but this is better than nothing :^)
2022-02-15 18:12:15 +00:00
// 13.2.5.61 After DOCTYPE public identifier state, https://html.spec.whatwg.org/multipage/parsing.html#after-doctype-public-identifier-state
LibWeb: Flesh out the remaining DOCTYPE related tokenizer states We can now parse public and system identifiers! Not super useful, but at least we can do it :^)
2020-05-25 19:50:44 +02:00
BEGIN_STATE(AfterDOCTYPEPublicIdentifier)
{
ON_WHITESPACE
{
SWITCH_TO(BetweenDOCTYPEPublicAndSystemIdentifiers);
}
ON('>')
{
SWITCH_TO_AND_EMIT_CURRENT_TOKEN(Data);
}
ON('"')
{
LibWeb: Utilize SourceLocation for HTMLTokenizer logging
2021-04-24 20:22:30 -07:00
log_parse_error();
LibWeb: Rename HTMLToken::doctype_data() => ensure_doctype_data() This renames the accessor to better reflect what it does, as this will allocate a DoctypeData struct if there is none.
2021-07-15 18:58:11 +02:00
m_current_token.ensure_doctype_data().missing_system_identifier = false;
LibWeb: Fully implement all DOCTYPE tokenizer states Also fixes TagOpen having a seperate emit and reconsume in ANYTHING_ELSE.
2020-06-11 05:00:45 +01:00
SWITCH_TO(DOCTYPESystemIdentifierDoubleQuoted);
LibWeb: Flesh out the remaining DOCTYPE related tokenizer states We can now parse public and system identifiers! Not super useful, but at least we can do it :^)
2020-05-25 19:50:44 +02:00
}
ON('\'')
{
LibWeb: Utilize SourceLocation for HTMLTokenizer logging
2021-04-24 20:22:30 -07:00
log_parse_error();
LibWeb: Rename HTMLToken::doctype_data() => ensure_doctype_data() This renames the accessor to better reflect what it does, as this will allocate a DoctypeData struct if there is none.
2021-07-15 18:58:11 +02:00
m_current_token.ensure_doctype_data().missing_system_identifier = false;
LibWeb: Fully implement all DOCTYPE tokenizer states Also fixes TagOpen having a seperate emit and reconsume in ANYTHING_ELSE.
2020-06-11 05:00:45 +01:00
SWITCH_TO(DOCTYPESystemIdentifierSingleQuoted);
LibWeb: Flesh out the remaining DOCTYPE related tokenizer states We can now parse public and system identifiers! Not super useful, but at least we can do it :^)
2020-05-25 19:50:44 +02:00
}
ON_EOF
{
LibWeb: Utilize SourceLocation for HTMLTokenizer logging
2021-04-24 20:22:30 -07:00
log_parse_error();
LibWeb: Rename HTMLToken::doctype_data() => ensure_doctype_data() This renames the accessor to better reflect what it does, as this will allocate a DoctypeData struct if there is none.
2021-07-15 18:58:11 +02:00
m_current_token.ensure_doctype_data().force_quirks = true;
LibWeb: Use move() when enqueuing tokens in HTMLTokenizer We're not using the current token anymore once it's enqueued so let's use move() when enqueuing the tokens.
2021-05-23 12:39:00 +02:00
m_queued_tokens.enqueue(move(m_current_token));
LibWeb: Fully implement all DOCTYPE tokenizer states Also fixes TagOpen having a seperate emit and reconsume in ANYTHING_ELSE.
2020-06-11 05:00:45 +01:00
EMIT_EOF;
LibWeb: Flesh out the remaining DOCTYPE related tokenizer states We can now parse public and system identifiers! Not super useful, but at least we can do it :^)
2020-05-25 19:50:44 +02:00
}
ANYTHING_ELSE
{
LibWeb: Utilize SourceLocation for HTMLTokenizer logging
2021-04-24 20:22:30 -07:00
log_parse_error();
LibWeb: Rename HTMLToken::doctype_data() => ensure_doctype_data() This renames the accessor to better reflect what it does, as this will allocate a DoctypeData struct if there is none.
2021-07-15 18:58:11 +02:00
m_current_token.ensure_doctype_data().force_quirks = true;
LibWeb: Fully implement all DOCTYPE tokenizer states Also fixes TagOpen having a seperate emit and reconsume in ANYTHING_ELSE.
2020-06-11 05:00:45 +01:00
RECONSUME_IN(BogusDOCTYPE);
LibWeb: Flesh out the remaining DOCTYPE related tokenizer states We can now parse public and system identifiers! Not super useful, but at least we can do it :^)
2020-05-25 19:50:44 +02:00
}
}
END_STATE
LibWeb: Add spec links to each HTML tokenizer state section I didn't add full spec comments this time, but this is better than nothing :^)
2022-02-15 18:12:15 +00:00
// 13.2.5.62 Between DOCTYPE public and system identifiers state, https://html.spec.whatwg.org/multipage/parsing.html#between-doctype-public-and-system-identifiers-state
LibWeb: Flesh out the remaining DOCTYPE related tokenizer states We can now parse public and system identifiers! Not super useful, but at least we can do it :^)
2020-05-25 19:50:44 +02:00
BEGIN_STATE(BetweenDOCTYPEPublicAndSystemIdentifiers)
{
ON_WHITESPACE
{
continue;
}
ON('>')
{
SWITCH_TO_AND_EMIT_CURRENT_TOKEN(Data);
}
ON('"')
{
LibWeb: Rename HTMLToken::doctype_data() => ensure_doctype_data() This renames the accessor to better reflect what it does, as this will allocate a DoctypeData struct if there is none.
2021-07-15 18:58:11 +02:00
m_current_token.ensure_doctype_data().missing_system_identifier = false;
LibWeb: Flesh out the remaining DOCTYPE related tokenizer states We can now parse public and system identifiers! Not super useful, but at least we can do it :^)
2020-05-25 19:50:44 +02:00
SWITCH_TO(DOCTYPESystemIdentifierDoubleQuoted);
}
ON('\'')
{
LibWeb: Rename HTMLToken::doctype_data() => ensure_doctype_data() This renames the accessor to better reflect what it does, as this will allocate a DoctypeData struct if there is none.
2021-07-15 18:58:11 +02:00
m_current_token.ensure_doctype_data().missing_system_identifier = false;
LibWeb: Flesh out the remaining DOCTYPE related tokenizer states We can now parse public and system identifiers! Not super useful, but at least we can do it :^)
2020-05-25 19:50:44 +02:00
SWITCH_TO(DOCTYPESystemIdentifierSingleQuoted);
}
ON_EOF
{
LibWeb: Utilize SourceLocation for HTMLTokenizer logging
2021-04-24 20:22:30 -07:00
log_parse_error();
LibWeb: Rename HTMLToken::doctype_data() => ensure_doctype_data() This renames the accessor to better reflect what it does, as this will allocate a DoctypeData struct if there is none.
2021-07-15 18:58:11 +02:00
m_current_token.ensure_doctype_data().force_quirks = true;
LibWeb: Use move() when enqueuing tokens in HTMLTokenizer We're not using the current token anymore once it's enqueued so let's use move() when enqueuing the tokens.
2021-05-23 12:39:00 +02:00
m_queued_tokens.enqueue(move(m_current_token));
LibWeb: Fully implement all DOCTYPE tokenizer states Also fixes TagOpen having a seperate emit and reconsume in ANYTHING_ELSE.
2020-06-11 05:00:45 +01:00
EMIT_EOF;
LibWeb: Flesh out the remaining DOCTYPE related tokenizer states We can now parse public and system identifiers! Not super useful, but at least we can do it :^)
2020-05-25 19:50:44 +02:00
}
ANYTHING_ELSE
{
LibWeb: Utilize SourceLocation for HTMLTokenizer logging
2021-04-24 20:22:30 -07:00
log_parse_error();
LibWeb: Rename HTMLToken::doctype_data() => ensure_doctype_data() This renames the accessor to better reflect what it does, as this will allocate a DoctypeData struct if there is none.
2021-07-15 18:58:11 +02:00
m_current_token.ensure_doctype_data().force_quirks = true;
LibWeb: Fully implement all DOCTYPE tokenizer states Also fixes TagOpen having a seperate emit and reconsume in ANYTHING_ELSE.
2020-06-11 05:00:45 +01:00
RECONSUME_IN(BogusDOCTYPE);
LibWeb: Flesh out the remaining DOCTYPE related tokenizer states We can now parse public and system identifiers! Not super useful, but at least we can do it :^)
2020-05-25 19:50:44 +02:00
}
}
END_STATE
LibWeb: Add spec links to each HTML tokenizer state section I didn't add full spec comments this time, but this is better than nothing :^)
2022-02-15 18:12:15 +00:00
// 13.2.5.67 After DOCTYPE system identifier state, https://html.spec.whatwg.org/multipage/parsing.html#after-doctype-system-identifier-state
LibWeb: Flesh out the remaining DOCTYPE related tokenizer states We can now parse public and system identifiers! Not super useful, but at least we can do it :^)
2020-05-25 19:50:44 +02:00
BEGIN_STATE(AfterDOCTYPESystemIdentifier)
LibWeb: Implement a bit more of DOCTYPE tokenization
2020-05-23 19:56:07 +02:00
{
ON_WHITESPACE
{
continue;
}
ON('>')
{
LibWeb: Start building the tree building part of the new HTML parser This patch adds a new HTMLDocumentParser class. It keeps a tokenizer object internally and feeds itself with one token at a time from it. The names and idioms in this class are expressed as closely to the actual HTML parsing spec as possible, to make development as easy and bug free as possible. :^) This is going to become pretty large, but it's pretty cool!
2020-05-24 00:14:23 +02:00
SWITCH_TO_AND_EMIT_CURRENT_TOKEN(Data);
LibWeb: Implement a bit more of DOCTYPE tokenization
2020-05-23 19:56:07 +02:00
}
ON_EOF
{
LibWeb: Utilize SourceLocation for HTMLTokenizer logging
2021-04-24 20:22:30 -07:00
log_parse_error();
LibWeb: Rename HTMLToken::doctype_data() => ensure_doctype_data() This renames the accessor to better reflect what it does, as this will allocate a DoctypeData struct if there is none.
2021-07-15 18:58:11 +02:00
m_current_token.ensure_doctype_data().force_quirks = true;
LibWeb: Use move() when enqueuing tokens in HTMLTokenizer We're not using the current token anymore once it's enqueued so let's use move() when enqueuing the tokens.
2021-05-23 12:39:00 +02:00
m_queued_tokens.enqueue(move(m_current_token));
LibWeb: Fully implement all DOCTYPE tokenizer states Also fixes TagOpen having a seperate emit and reconsume in ANYTHING_ELSE.
2020-06-11 05:00:45 +01:00
EMIT_EOF;
LibWeb: Implement a bit more of DOCTYPE tokenization
2020-05-23 19:56:07 +02:00
}
ANYTHING_ELSE
{
LibWeb: Utilize SourceLocation for HTMLTokenizer logging
2021-04-24 20:22:30 -07:00
log_parse_error();
LibWeb: Fully implement all DOCTYPE tokenizer states Also fixes TagOpen having a seperate emit and reconsume in ANYTHING_ELSE.
2020-06-11 05:00:45 +01:00
RECONSUME_IN(BogusDOCTYPE);
}
}
END_STATE
LibWeb: Add spec links to each HTML tokenizer state section I didn't add full spec comments this time, but this is better than nothing :^)
2022-02-15 18:12:15 +00:00
// 13.2.5.68 Bogus DOCTYPE state, https://html.spec.whatwg.org/multipage/parsing.html#bogus-doctype-state
LibWeb: Fully implement all DOCTYPE tokenizer states Also fixes TagOpen having a seperate emit and reconsume in ANYTHING_ELSE.
2020-06-11 05:00:45 +01:00
BEGIN_STATE(BogusDOCTYPE)
{
ON('>')
{
SWITCH_TO_AND_EMIT_CURRENT_TOKEN(Data);
}
ON(0)
{
LibWeb: Utilize SourceLocation for HTMLTokenizer logging
2021-04-24 20:22:30 -07:00
log_parse_error();
LibWeb: Fully implement all DOCTYPE tokenizer states Also fixes TagOpen having a seperate emit and reconsume in ANYTHING_ELSE.
2020-06-11 05:00:45 +01:00
continue;
}
ON_EOF
{
LibWeb: Use move() when enqueuing tokens in HTMLTokenizer We're not using the current token anymore once it's enqueued so let's use move() when enqueuing the tokens.
2021-05-23 12:39:00 +02:00
m_queued_tokens.enqueue(move(m_current_token));
LibWeb: Fully implement all DOCTYPE tokenizer states Also fixes TagOpen having a seperate emit and reconsume in ANYTHING_ELSE.
2020-06-11 05:00:45 +01:00
EMIT_EOF;
}
ANYTHING_ELSE
{
continue;
LibWeb: Implement a bit more of DOCTYPE tokenization
2020-05-23 19:56:07 +02:00
}
LibWeb: Begin work on a spec-compliant HTML parser In order to actually view the web as it is, we're gonna need a proper HTML parser. So let's build one! This patch introduces the Web::HTMLTokenizer class, which currently operates on a StringView input stream where it fetches (ASCII only atm) codepoints and tokenizes acccording to the HTML spec tokenization algo. The tokenizer state machine looks a bit weird but is written in a way that tries to mimic the spec as closely as possible, in order to make development easier and bugs less likely. This initial version is far from finished, but it can parse a trivial document with a DOCTYPE and open/close tags. :^)
2020-05-22 21:46:13 +02:00
}
END_STATE
LibWeb: Add spec links to each HTML tokenizer state section I didn't add full spec comments this time, but this is better than nothing :^)
2022-02-15 18:12:15 +00:00
// 13.2.5.32 Before attribute name state, https://html.spec.whatwg.org/multipage/parsing.html#before-attribute-name-state
LibWeb: Teach HTMLTokenizer how to tokenize attributes Properly tokenize single-quoted, double-quoted and unquoted attributes!
2020-05-23 01:19:42 +02:00
BEGIN_STATE(BeforeAttributeName)
{
ON_WHITESPACE
{
continue;
}
ON('/')
{
LibWeb: Hide implementation details of HTMLToken attribute list Previously, HTMLToken would expose the Vector<Attribute> directly to its users. In preparation for a future change, all users now use implementation-agnostic APIs which do not expose the Vector directly.
2021-07-14 23:53:11 +02:00
if (m_current_token.has_attributes())
m_current_token.last_attribute().name_end_position = nth_last_position(1);
LibWeb: Teach HTMLTokenizer how to tokenize attributes Properly tokenize single-quoted, double-quoted and unquoted attributes!
2020-05-23 01:19:42 +02:00
RECONSUME_IN(AfterAttributeName);
}
ON('>')
{
RECONSUME_IN(AfterAttributeName);
}
ON_EOF
{
RECONSUME_IN(AfterAttributeName);
}
ON('=')
{
LibWeb: Utilize SourceLocation for HTMLTokenizer logging
2021-04-24 20:22:30 -07:00
log_parse_error();
LibWeb: Rename HTMLToken::AttributeBuilder struct to Attribute This does not contain StringBuilders anymore, so it can do with a simpler name: Attribute.
2021-07-14 23:17:35 +02:00
HTMLToken::Attribute new_attribute;
LibWeb: Add position tracking information to HTML tokens
2021-05-20 23:11:41 +04:30
new_attribute.name_start_position = nth_last_position(1);
LibWeb: Remove StringBuilders from HTMLToken::AttributeBuilder
2021-05-23 08:50:48 +02:00
m_current_builder.append_code_point(current_input_character.value());
LibWeb: Hide implementation details of HTMLToken attribute list Previously, HTMLToken would expose the Vector<Attribute> directly to its users. In preparation for a future change, all users now use implementation-agnostic APIs which do not expose the Vector directly.
2021-07-14 23:53:11 +02:00
m_current_token.add_attribute(move(new_attribute));
LibWeb: Remove StringBuilders from HTMLToken::AttributeBuilder
2021-05-23 08:50:48 +02:00
SWITCH_TO_WITH_UNCLEAN_BUILDER(AttributeName);
LibWeb: Teach HTMLTokenizer how to tokenize attributes Properly tokenize single-quoted, double-quoted and unquoted attributes!
2020-05-23 01:19:42 +02:00
}
ANYTHING_ELSE
{
LibWeb: Rename HTMLToken::AttributeBuilder struct to Attribute This does not contain StringBuilders anymore, so it can do with a simpler name: Attribute.
2021-07-14 23:17:35 +02:00
HTMLToken::Attribute new_attribute;
LibWeb: Add position tracking information to HTML tokens
2021-05-20 23:11:41 +04:30
new_attribute.name_start_position = nth_last_position(1);
LibWeb: Hide implementation details of HTMLToken attribute list Previously, HTMLToken would expose the Vector<Attribute> directly to its users. In preparation for a future change, all users now use implementation-agnostic APIs which do not expose the Vector directly.
2021-07-14 23:53:11 +02:00
m_current_token.add_attribute(move(new_attribute));
LibWeb: Teach HTMLTokenizer how to tokenize attributes Properly tokenize single-quoted, double-quoted and unquoted attributes!
2020-05-23 01:19:42 +02:00
RECONSUME_IN(AttributeName);
}
}
END_STATE
LibWeb: Add spec links to each HTML tokenizer state section I didn't add full spec comments this time, but this is better than nothing :^)
2022-02-15 18:12:15 +00:00
// 13.2.5.40 Self-closing start tag state, https://html.spec.whatwg.org/multipage/parsing.html#self-closing-start-tag-state
LibWeb: Teach HTMLTokenizer how to tokenize attributes Properly tokenize single-quoted, double-quoted and unquoted attributes!
2020-05-23 01:19:42 +02:00
BEGIN_STATE(SelfClosingStartTag)
{
LibWeb: Implement the "self closing start tag" tokenizer state
2020-05-27 18:07:51 +02:00
ON('>')
{
LibWeb: User getter+setter for HTMLToken tag name and self-closing flag
2021-07-14 23:37:48 +02:00
m_current_token.set_self_closing(true);
LibWeb: Fix accidental swallow of self-closing tag tokens Instead of dropping self-closing tags on the floor, we now emit them into the token stream. :^)
2020-05-30 11:30:45 +02:00
SWITCH_TO_AND_EMIT_CURRENT_TOKEN(Data);
LibWeb: Implement the "self closing start tag" tokenizer state
2020-05-27 18:07:51 +02:00
}
ON_EOF
{
LibWeb: Utilize SourceLocation for HTMLTokenizer logging
2021-04-24 20:22:30 -07:00
log_parse_error();
LibWeb: Handle more cases in the SelfClosingStartTag tokenizer state
2020-06-01 12:37:28 +02:00
EMIT_EOF;
LibWeb: Implement the "self closing start tag" tokenizer state
2020-05-27 18:07:51 +02:00
}
ANYTHING_ELSE
{
LibWeb: Utilize SourceLocation for HTMLTokenizer logging
2021-04-24 20:22:30 -07:00
log_parse_error();
LibWeb: Handle more cases in the SelfClosingStartTag tokenizer state
2020-06-01 12:37:28 +02:00
RECONSUME_IN(BeforeAttributeName);
LibWeb: Implement the "self closing start tag" tokenizer state
2020-05-27 18:07:51 +02:00
}
LibWeb: Teach HTMLTokenizer how to tokenize attributes Properly tokenize single-quoted, double-quoted and unquoted attributes!
2020-05-23 01:19:42 +02:00
}
END_STATE
LibWeb: Add spec links to each HTML tokenizer state section I didn't add full spec comments this time, but this is better than nothing :^)
2022-02-15 18:12:15 +00:00
// 13.2.5.33 Attribute name state, https://html.spec.whatwg.org/multipage/parsing.html#attribute-name-state
LibWeb: Teach HTMLTokenizer how to tokenize attributes Properly tokenize single-quoted, double-quoted and unquoted attributes!
2020-05-23 01:19:42 +02:00
BEGIN_STATE(AttributeName)
{
ON_WHITESPACE
{
LibWeb: Hide implementation details of HTMLToken attribute list Previously, HTMLToken would expose the Vector<Attribute> directly to its users. In preparation for a future change, all users now use implementation-agnostic APIs which do not expose the Vector directly.
2021-07-14 23:53:11 +02:00
m_current_token.last_attribute().local_name = consume_current_builder();
LibWeb: Teach HTMLTokenizer how to tokenize attributes Properly tokenize single-quoted, double-quoted and unquoted attributes!
2020-05-23 01:19:42 +02:00
RECONSUME_IN(AfterAttributeName);
}
ON('/')
{
LibWeb: Hide implementation details of HTMLToken attribute list Previously, HTMLToken would expose the Vector<Attribute> directly to its users. In preparation for a future change, all users now use implementation-agnostic APIs which do not expose the Vector directly.
2021-07-14 23:53:11 +02:00
m_current_token.last_attribute().local_name = consume_current_builder();
LibWeb: Teach HTMLTokenizer how to tokenize attributes Properly tokenize single-quoted, double-quoted and unquoted attributes!
2020-05-23 01:19:42 +02:00
RECONSUME_IN(AfterAttributeName);
}
ON('>')
{
LibWeb: Hide implementation details of HTMLToken attribute list Previously, HTMLToken would expose the Vector<Attribute> directly to its users. In preparation for a future change, all users now use implementation-agnostic APIs which do not expose the Vector directly.
2021-07-14 23:53:11 +02:00
m_current_token.last_attribute().local_name = consume_current_builder();
LibWeb: Teach HTMLTokenizer how to tokenize attributes Properly tokenize single-quoted, double-quoted and unquoted attributes!
2020-05-23 01:19:42 +02:00
RECONSUME_IN(AfterAttributeName);
}
ON_EOF
{
LibWeb: Hide implementation details of HTMLToken attribute list Previously, HTMLToken would expose the Vector<Attribute> directly to its users. In preparation for a future change, all users now use implementation-agnostic APIs which do not expose the Vector directly.
2021-07-14 23:53:11 +02:00
m_current_token.last_attribute().local_name = consume_current_builder();
LibWeb: Teach HTMLTokenizer how to tokenize attributes Properly tokenize single-quoted, double-quoted and unquoted attributes!
2020-05-23 01:19:42 +02:00
RECONSUME_IN(AfterAttributeName);
}
ON('=')
{
LibWeb: Hide implementation details of HTMLToken attribute list Previously, HTMLToken would expose the Vector<Attribute> directly to its users. In preparation for a future change, all users now use implementation-agnostic APIs which do not expose the Vector directly.
2021-07-14 23:53:11 +02:00
m_current_token.last_attribute().name_end_position = nth_last_position(1);
m_current_token.last_attribute().local_name = consume_current_builder();
LibWeb: Teach HTMLTokenizer how to tokenize attributes Properly tokenize single-quoted, double-quoted and unquoted attributes!
2020-05-23 01:19:42 +02:00
SWITCH_TO(BeforeAttributeValue);
}
LibWeb: Implement almost all missing tokenizer cases
2020-06-26 22:41:35 +01:00
ON_ASCII_UPPER_ALPHA
{
LibWeb: Remove StringBuilders from HTMLToken::AttributeBuilder
2021-05-23 08:50:48 +02:00
m_current_builder.append_code_point(to_ascii_lowercase(current_input_character.value()));
LibWeb: Implement almost all missing tokenizer cases
2020-06-26 22:41:35 +01:00
continue;
}
ON(0)
{
LibWeb: Utilize SourceLocation for HTMLTokenizer logging
2021-04-24 20:22:30 -07:00
log_parse_error();
LibWeb: Remove StringBuilders from HTMLToken::AttributeBuilder
2021-05-23 08:50:48 +02:00
m_current_builder.append_code_point(0xFFFD);
LibWeb: Implement almost all missing tokenizer cases
2020-06-26 22:41:35 +01:00
continue;
}
ON('"')
{
LibWeb: Utilize SourceLocation for HTMLTokenizer logging
2021-04-24 20:22:30 -07:00
log_parse_error();
LibWeb: Implement almost all missing tokenizer cases
2020-06-26 22:41:35 +01:00
goto AnythingElseAttributeName;
}
ON('\'')
{
LibWeb: Utilize SourceLocation for HTMLTokenizer logging
2021-04-24 20:22:30 -07:00
log_parse_error();
LibWeb: Implement almost all missing tokenizer cases
2020-06-26 22:41:35 +01:00
goto AnythingElseAttributeName;
}
ON('<')
{
LibWeb: Utilize SourceLocation for HTMLTokenizer logging
2021-04-24 20:22:30 -07:00
log_parse_error();
LibWeb: Implement almost all missing tokenizer cases
2020-06-26 22:41:35 +01:00
goto AnythingElseAttributeName;
}
LibWeb: Teach HTMLTokenizer how to tokenize attributes Properly tokenize single-quoted, double-quoted and unquoted attributes!
2020-05-23 01:19:42 +02:00
ANYTHING_ELSE
{
LibWeb: Implement almost all missing tokenizer cases
2020-06-26 22:41:35 +01:00
AnythingElseAttributeName:
LibWeb: Remove StringBuilders from HTMLToken::AttributeBuilder
2021-05-23 08:50:48 +02:00
m_current_builder.append_code_point(current_input_character.value());
LibWeb: Teach HTMLTokenizer how to tokenize attributes Properly tokenize single-quoted, double-quoted and unquoted attributes!
2020-05-23 01:19:42 +02:00
continue;
}
}
END_STATE
LibWeb: Add spec links to each HTML tokenizer state section I didn't add full spec comments this time, but this is better than nothing :^)
2022-02-15 18:12:15 +00:00
// 13.2.5.34 After attribute name state, https://html.spec.whatwg.org/multipage/parsing.html#after-attribute-name-state
LibWeb: Teach HTMLTokenizer how to tokenize attributes Properly tokenize single-quoted, double-quoted and unquoted attributes!
2020-05-23 01:19:42 +02:00
BEGIN_STATE(AfterAttributeName)
{
LibWeb: Implement the "after attribute name" tokenizer state One little step at a time towards parsing the monster blob of HTML we get from twitter.com :^)
2020-05-27 18:27:32 +02:00
ON_WHITESPACE
{
continue;
}
ON('/')
{
SWITCH_TO(SelfClosingStartTag);
}
ON('=')
{
LibWeb: Hide implementation details of HTMLToken attribute list Previously, HTMLToken would expose the Vector<Attribute> directly to its users. In preparation for a future change, all users now use implementation-agnostic APIs which do not expose the Vector directly.
2021-07-14 23:53:11 +02:00
m_current_token.last_attribute().name_end_position = nth_last_position(1);
LibWeb: Implement the "after attribute name" tokenizer state One little step at a time towards parsing the monster blob of HTML we get from twitter.com :^)
2020-05-27 18:27:32 +02:00
SWITCH_TO(BeforeAttributeValue);
}
ON('>')
{
LibWeb: Fix tokenization of attributes with empty attributes We were neglecting to emit start tags for tags where the last attribute had no value. Also fix a parse error TODO that I hit while looking at this.
2020-06-04 11:59:14 +02:00
SWITCH_TO_AND_EMIT_CURRENT_TOKEN(Data);
LibWeb: Implement the "after attribute name" tokenizer state One little step at a time towards parsing the monster blob of HTML we get from twitter.com :^)
2020-05-27 18:27:32 +02:00
}
ON_EOF
{
LibWeb: Utilize SourceLocation for HTMLTokenizer logging
2021-04-24 20:22:30 -07:00
log_parse_error();
LibWeb: Implement some missing tokenizer cases for EOF handling
2020-06-26 22:47:07 +02:00
EMIT_EOF;
LibWeb: Implement the "after attribute name" tokenizer state One little step at a time towards parsing the monster blob of HTML we get from twitter.com :^)
2020-05-27 18:27:32 +02:00
}
ANYTHING_ELSE
{
LibWeb: Hide implementation details of HTMLToken attribute list Previously, HTMLToken would expose the Vector<Attribute> directly to its users. In preparation for a future change, all users now use implementation-agnostic APIs which do not expose the Vector directly.
2021-07-14 23:53:11 +02:00
m_current_token.add_attribute({});
LibWeb: Fix Array OOBs in the HTMLTokenizer Accessing last() if there are no elements makes WebContent crash :^)
2022-06-02 01:03:44 +02:00
if (!m_source_positions.is_empty())
m_current_token.last_attribute().name_start_position = m_source_positions.last();
LibWeb: Implement the "after attribute name" tokenizer state One little step at a time towards parsing the monster blob of HTML we get from twitter.com :^)
2020-05-27 18:27:32 +02:00
RECONSUME_IN(AttributeName);
}
LibWeb: Teach HTMLTokenizer how to tokenize attributes Properly tokenize single-quoted, double-quoted and unquoted attributes!
2020-05-23 01:19:42 +02:00
}
END_STATE
LibWeb: Add spec links to each HTML tokenizer state section I didn't add full spec comments this time, but this is better than nothing :^)
2022-02-15 18:12:15 +00:00
// 13.2.5.35 Before attribute value state, https://html.spec.whatwg.org/multipage/parsing.html#before-attribute-value-state
LibWeb: Teach HTMLTokenizer how to tokenize attributes Properly tokenize single-quoted, double-quoted and unquoted attributes!
2020-05-23 01:19:42 +02:00
BEGIN_STATE(BeforeAttributeValue)
{
LibWeb: Hide implementation details of HTMLToken attribute list Previously, HTMLToken would expose the Vector<Attribute> directly to its users. In preparation for a future change, all users now use implementation-agnostic APIs which do not expose the Vector directly.
2021-07-14 23:53:11 +02:00
m_current_token.last_attribute().value_start_position = nth_last_position(1);
LibWeb: Teach HTMLTokenizer how to tokenize attributes Properly tokenize single-quoted, double-quoted and unquoted attributes!
2020-05-23 01:19:42 +02:00
ON_WHITESPACE
{
continue;
}
ON('"')
{
SWITCH_TO(AttributeValueDoubleQuoted);
}
ON('\'')
{
SWITCH_TO(AttributeValueSingleQuoted);
}
ON('>')
{
LibWeb: Utilize SourceLocation for HTMLTokenizer logging
2021-04-24 20:22:30 -07:00
log_parse_error();
LibWeb: Fix tokenization of attributes with empty attributes We were neglecting to emit start tags for tags where the last attribute had no value. Also fix a parse error TODO that I hit while looking at this.
2020-06-04 11:59:14 +02:00
SWITCH_TO_AND_EMIT_CURRENT_TOKEN(Data);
LibWeb: Teach HTMLTokenizer how to tokenize attributes Properly tokenize single-quoted, double-quoted and unquoted attributes!
2020-05-23 01:19:42 +02:00
}
ANYTHING_ELSE
{
RECONSUME_IN(AttributeValueUnquoted);
}
}
END_STATE
LibWeb: Add spec links to each HTML tokenizer state section I didn't add full spec comments this time, but this is better than nothing :^)
2022-02-15 18:12:15 +00:00
// 13.2.5.36 Attribute value (double-quoted) state, https://html.spec.whatwg.org/multipage/parsing.html#attribute-value-(double-quoted)-state
LibWeb: Teach HTMLTokenizer how to tokenize attributes Properly tokenize single-quoted, double-quoted and unquoted attributes!
2020-05-23 01:19:42 +02:00
BEGIN_STATE(AttributeValueDoubleQuoted)
{
ON('"')
{
LibWeb: Hide implementation details of HTMLToken attribute list Previously, HTMLToken would expose the Vector<Attribute> directly to its users. In preparation for a future change, all users now use implementation-agnostic APIs which do not expose the Vector directly.
2021-07-14 23:53:11 +02:00
m_current_token.last_attribute().value = consume_current_builder();
LibWeb: Teach HTMLTokenizer how to tokenize attributes Properly tokenize single-quoted, double-quoted and unquoted attributes!
2020-05-23 01:19:42 +02:00
SWITCH_TO(AfterAttributeValueQuoted);
}
ON('&')
{
m_return_state = State::AttributeValueDoubleQuoted;
LibWeb: Fix parsing of character references in attribute values
2021-07-24 23:15:47 +01:00
SWITCH_TO_WITH_UNCLEAN_BUILDER(CharacterReference);
LibWeb: Teach HTMLTokenizer how to tokenize attributes Properly tokenize single-quoted, double-quoted and unquoted attributes!
2020-05-23 01:19:42 +02:00
}
ON(0)
{
LibWeb: Utilize SourceLocation for HTMLTokenizer logging
2021-04-24 20:22:30 -07:00
log_parse_error();
LibWeb: Remove StringBuilders from HTMLToken::AttributeBuilder
2021-05-23 08:50:48 +02:00
m_current_builder.append_code_point(0xFFFD);
LibWeb: Implement almost all missing tokenizer cases
2020-06-26 22:41:35 +01:00
continue;
LibWeb: Teach HTMLTokenizer how to tokenize attributes Properly tokenize single-quoted, double-quoted and unquoted attributes!
2020-05-23 01:19:42 +02:00
}
ON_EOF
{
LibWeb: Utilize SourceLocation for HTMLTokenizer logging
2021-04-24 20:22:30 -07:00
log_parse_error();
LibWeb: Implement remaining missing tokenizer EOF cases
2020-06-27 12:05:21 +02:00
EMIT_EOF;
LibWeb: Teach HTMLTokenizer how to tokenize attributes Properly tokenize single-quoted, double-quoted and unquoted attributes!
2020-05-23 01:19:42 +02:00
}
ANYTHING_ELSE
{
LibWeb: Remove StringBuilders from HTMLToken::AttributeBuilder
2021-05-23 08:50:48 +02:00
m_current_builder.append_code_point(current_input_character.value());
LibWeb: Teach HTMLTokenizer how to tokenize attributes Properly tokenize single-quoted, double-quoted and unquoted attributes!
2020-05-23 01:19:42 +02:00
continue;
}
}
END_STATE
LibWeb: Add spec links to each HTML tokenizer state section I didn't add full spec comments this time, but this is better than nothing :^)
2022-02-15 18:12:15 +00:00
// 13.2.5.37 Attribute value (single-quoted) state, https://html.spec.whatwg.org/multipage/parsing.html#attribute-value-(single-quoted)-state
LibWeb: Teach HTMLTokenizer how to tokenize attributes Properly tokenize single-quoted, double-quoted and unquoted attributes!
2020-05-23 01:19:42 +02:00
BEGIN_STATE(AttributeValueSingleQuoted)
{
ON('\'')
{
LibWeb: Hide implementation details of HTMLToken attribute list Previously, HTMLToken would expose the Vector<Attribute> directly to its users. In preparation for a future change, all users now use implementation-agnostic APIs which do not expose the Vector directly.
2021-07-14 23:53:11 +02:00
m_current_token.last_attribute().value = consume_current_builder();
LibWeb: Teach HTMLTokenizer how to tokenize attributes Properly tokenize single-quoted, double-quoted and unquoted attributes!
2020-05-23 01:19:42 +02:00
SWITCH_TO(AfterAttributeValueQuoted);
}
ON('&')
{
m_return_state = State::AttributeValueSingleQuoted;
LibWeb: Fix parsing of character references in attribute values
2021-07-24 23:15:47 +01:00
SWITCH_TO_WITH_UNCLEAN_BUILDER(CharacterReference);
LibWeb: Teach HTMLTokenizer how to tokenize attributes Properly tokenize single-quoted, double-quoted and unquoted attributes!
2020-05-23 01:19:42 +02:00
}
ON(0)
{
LibWeb: Utilize SourceLocation for HTMLTokenizer logging
2021-04-24 20:22:30 -07:00
log_parse_error();
LibWeb: Remove StringBuilders from HTMLToken::AttributeBuilder
2021-05-23 08:50:48 +02:00
m_current_builder.append_code_point(0xFFFD);
LibWeb: Implement almost all missing tokenizer cases
2020-06-26 22:41:35 +01:00
continue;
LibWeb: Teach HTMLTokenizer how to tokenize attributes Properly tokenize single-quoted, double-quoted and unquoted attributes!
2020-05-23 01:19:42 +02:00
}
ON_EOF
{
LibWeb: Utilize SourceLocation for HTMLTokenizer logging
2021-04-24 20:22:30 -07:00
log_parse_error();
LibWeb: Implement remaining missing tokenizer EOF cases
2020-06-27 12:05:21 +02:00
EMIT_EOF;
LibWeb: Teach HTMLTokenizer how to tokenize attributes Properly tokenize single-quoted, double-quoted and unquoted attributes!
2020-05-23 01:19:42 +02:00
}
ANYTHING_ELSE
{
LibWeb: Remove StringBuilders from HTMLToken::AttributeBuilder
2021-05-23 08:50:48 +02:00
m_current_builder.append_code_point(current_input_character.value());
LibWeb: Teach HTMLTokenizer how to tokenize attributes Properly tokenize single-quoted, double-quoted and unquoted attributes!
2020-05-23 01:19:42 +02:00
continue;
}
}
END_STATE
LibWeb: Add spec links to each HTML tokenizer state section I didn't add full spec comments this time, but this is better than nothing :^)
2022-02-15 18:12:15 +00:00
// 13.2.5.38 Attribute value (unquoted) state, https://html.spec.whatwg.org/multipage/parsing.html#attribute-value-(single-quoted)-state
LibWeb: Teach HTMLTokenizer how to tokenize attributes Properly tokenize single-quoted, double-quoted and unquoted attributes!
2020-05-23 01:19:42 +02:00
BEGIN_STATE(AttributeValueUnquoted)
{
ON_WHITESPACE
{
LibWeb: Hide implementation details of HTMLToken attribute list Previously, HTMLToken would expose the Vector<Attribute> directly to its users. In preparation for a future change, all users now use implementation-agnostic APIs which do not expose the Vector directly.
2021-07-14 23:53:11 +02:00
m_current_token.last_attribute().value = consume_current_builder();
LibWeb: Fix off-by-one error when highlighting unquoted HTML attributes This fixes #11166
2021-12-10 17:17:12 +00:00
m_current_token.last_attribute().value_end_position = nth_last_position(1);
LibWeb: Teach HTMLTokenizer how to tokenize attributes Properly tokenize single-quoted, double-quoted and unquoted attributes!
2020-05-23 01:19:42 +02:00
SWITCH_TO(BeforeAttributeName);
}
ON('&')
{
m_return_state = State::AttributeValueUnquoted;
LibWeb: Fix parsing of character references in attribute values
2021-07-24 23:15:47 +01:00
SWITCH_TO_WITH_UNCLEAN_BUILDER(CharacterReference);
LibWeb: Teach HTMLTokenizer how to tokenize attributes Properly tokenize single-quoted, double-quoted and unquoted attributes!
2020-05-23 01:19:42 +02:00
}
ON('>')
{
LibWeb: Hide implementation details of HTMLToken attribute list Previously, HTMLToken would expose the Vector<Attribute> directly to its users. In preparation for a future change, all users now use implementation-agnostic APIs which do not expose the Vector directly.
2021-07-14 23:53:11 +02:00
m_current_token.last_attribute().value = consume_current_builder();
m_current_token.last_attribute().value_end_position = nth_last_position(1);
LibWeb: Start building the tree building part of the new HTML parser This patch adds a new HTMLDocumentParser class. It keeps a tokenizer object internally and feeds itself with one token at a time from it. The names and idioms in this class are expressed as closely to the actual HTML parsing spec as possible, to make development as easy and bug free as possible. :^) This is going to become pretty large, but it's pretty cool!
2020-05-24 00:14:23 +02:00
SWITCH_TO_AND_EMIT_CURRENT_TOKEN(Data);
LibWeb: Teach HTMLTokenizer how to tokenize attributes Properly tokenize single-quoted, double-quoted and unquoted attributes!
2020-05-23 01:19:42 +02:00
}
ON(0)
{
LibWeb: Utilize SourceLocation for HTMLTokenizer logging
2021-04-24 20:22:30 -07:00
log_parse_error();
LibWeb: Remove StringBuilders from HTMLToken::AttributeBuilder
2021-05-23 08:50:48 +02:00
m_current_builder.append_code_point(0xFFFD);
LibWeb: Implement almost all missing tokenizer cases
2020-06-26 22:41:35 +01:00
continue;
}
ON('"')
{
LibWeb: Utilize SourceLocation for HTMLTokenizer logging
2021-04-24 20:22:30 -07:00
log_parse_error();
LibWeb: Implement almost all missing tokenizer cases
2020-06-26 22:41:35 +01:00
goto AnythingElseAttributeValueUnquoted;
}
ON('\'')
{
LibWeb: Utilize SourceLocation for HTMLTokenizer logging
2021-04-24 20:22:30 -07:00
log_parse_error();
LibWeb: Implement almost all missing tokenizer cases
2020-06-26 22:41:35 +01:00
goto AnythingElseAttributeValueUnquoted;
}
ON('<')
{
LibWeb: Utilize SourceLocation for HTMLTokenizer logging
2021-04-24 20:22:30 -07:00
log_parse_error();
LibWeb: Implement almost all missing tokenizer cases
2020-06-26 22:41:35 +01:00
goto AnythingElseAttributeValueUnquoted;
}
ON('=')
{
LibWeb: Utilize SourceLocation for HTMLTokenizer logging
2021-04-24 20:22:30 -07:00
log_parse_error();
LibWeb: Implement almost all missing tokenizer cases
2020-06-26 22:41:35 +01:00
goto AnythingElseAttributeValueUnquoted;
}
ON('`')
{
LibWeb: Utilize SourceLocation for HTMLTokenizer logging
2021-04-24 20:22:30 -07:00
log_parse_error();
LibWeb: Implement almost all missing tokenizer cases
2020-06-26 22:41:35 +01:00
goto AnythingElseAttributeValueUnquoted;
LibWeb: Teach HTMLTokenizer how to tokenize attributes Properly tokenize single-quoted, double-quoted and unquoted attributes!
2020-05-23 01:19:42 +02:00
}
ON_EOF
{
LibWeb: Utilize SourceLocation for HTMLTokenizer logging
2021-04-24 20:22:30 -07:00
log_parse_error();
LibWeb: Implement remaining missing tokenizer EOF cases
2020-06-27 12:05:21 +02:00
EMIT_EOF;
LibWeb: Teach HTMLTokenizer how to tokenize attributes Properly tokenize single-quoted, double-quoted and unquoted attributes!
2020-05-23 01:19:42 +02:00
}
ANYTHING_ELSE
{
LibWeb: Implement almost all missing tokenizer cases
2020-06-26 22:41:35 +01:00
AnythingElseAttributeValueUnquoted:
LibWeb: Remove StringBuilders from HTMLToken::AttributeBuilder
2021-05-23 08:50:48 +02:00
m_current_builder.append_code_point(current_input_character.value());
LibWeb: Teach HTMLTokenizer how to tokenize attributes Properly tokenize single-quoted, double-quoted and unquoted attributes!
2020-05-23 01:19:42 +02:00
continue;
}
}
END_STATE
LibWeb: Add spec links to each HTML tokenizer state section I didn't add full spec comments this time, but this is better than nothing :^)
2022-02-15 18:12:15 +00:00
// 13.2.5.39 After attribute value (quoted) state, https://html.spec.whatwg.org/multipage/parsing.html#after-attribute-value-(quoted)-state
LibWeb: Teach HTMLTokenizer how to tokenize attributes Properly tokenize single-quoted, double-quoted and unquoted attributes!
2020-05-23 01:19:42 +02:00
BEGIN_STATE(AfterAttributeValueQuoted)
{
LibWeb: Hide implementation details of HTMLToken attribute list Previously, HTMLToken would expose the Vector<Attribute> directly to its users. In preparation for a future change, all users now use implementation-agnostic APIs which do not expose the Vector directly.
2021-07-14 23:53:11 +02:00
m_current_token.last_attribute().value_end_position = nth_last_position(1);
LibWeb: Teach HTMLTokenizer how to tokenize attributes Properly tokenize single-quoted, double-quoted and unquoted attributes!
2020-05-23 01:19:42 +02:00
ON_WHITESPACE
{
SWITCH_TO(BeforeAttributeName);
}
ON('/')
{
SWITCH_TO(SelfClosingStartTag);
}
ON('>')
{
LibWeb: Start building the tree building part of the new HTML parser This patch adds a new HTMLDocumentParser class. It keeps a tokenizer object internally and feeds itself with one token at a time from it. The names and idioms in this class are expressed as closely to the actual HTML parsing spec as possible, to make development as easy and bug free as possible. :^) This is going to become pretty large, but it's pretty cool!
2020-05-24 00:14:23 +02:00
SWITCH_TO_AND_EMIT_CURRENT_TOKEN(Data);
LibWeb: Teach HTMLTokenizer how to tokenize attributes Properly tokenize single-quoted, double-quoted and unquoted attributes!
2020-05-23 01:19:42 +02:00
}
ON_EOF
{
LibWeb: Utilize SourceLocation for HTMLTokenizer logging
2021-04-24 20:22:30 -07:00
log_parse_error();
LibWeb: Parse "textarea" tags during the "in body" insertion mode Had to handle some more cases in the tokenizer to support this.
2020-05-30 18:40:23 +02:00
EMIT_EOF;
LibWeb: Teach HTMLTokenizer how to tokenize attributes Properly tokenize single-quoted, double-quoted and unquoted attributes!
2020-05-23 01:19:42 +02:00
}
ANYTHING_ELSE
{
LibWeb: Utilize SourceLocation for HTMLTokenizer logging
2021-04-24 20:22:30 -07:00
log_parse_error();
LibWeb: Parse "textarea" tags during the "in body" insertion mode Had to handle some more cases in the tokenizer to support this.
2020-05-30 18:40:23 +02:00
RECONSUME_IN(BeforeAttributeName);
LibWeb: Teach HTMLTokenizer how to tokenize attributes Properly tokenize single-quoted, double-quoted and unquoted attributes!
2020-05-23 01:19:42 +02:00
}
}
END_STATE
LibWeb: Add spec links to each HTML tokenizer state section I didn't add full spec comments this time, but this is better than nothing :^)
2022-02-15 18:12:15 +00:00
// 13.2.5.43 Comment start state, https://html.spec.whatwg.org/multipage/parsing.html#comment-start-state
LibWeb: Teach HTMLTokenizer how to tokenize comments We can now correctly tokenize the welcome.html test page. :^)
2020-05-23 01:54:26 +02:00
BEGIN_STATE(CommentStart)
{
ON('-')
{
SWITCH_TO(CommentStartDash);
}
ON('>')
{
LibWeb: Utilize SourceLocation for HTMLTokenizer logging
2021-04-24 20:22:30 -07:00
log_parse_error();
LibWeb: Fully implement all comment tokenizer states
2020-06-11 02:09:07 +01:00
SWITCH_TO_AND_EMIT_CURRENT_TOKEN(Data);
LibWeb: Teach HTMLTokenizer how to tokenize comments We can now correctly tokenize the welcome.html test page. :^)
2020-05-23 01:54:26 +02:00
}
ANYTHING_ELSE
{
RECONSUME_IN(Comment);
}
}
END_STATE
LibWeb: Add spec links to each HTML tokenizer state section I didn't add full spec comments this time, but this is better than nothing :^)
2022-02-15 18:12:15 +00:00
// 13.2.5.44 Comment start dash state, https://html.spec.whatwg.org/multipage/parsing.html#comment-start-dash-state
LibWeb: Teach HTMLTokenizer how to tokenize comments We can now correctly tokenize the welcome.html test page. :^)
2020-05-23 01:54:26 +02:00
BEGIN_STATE(CommentStartDash)
{
ON('-')
{
LibWeb: Correct behavior of Comment* states in HTMLTokenizer Previously, this would lead to assertion failures when parsing HTML comments. This fixes #8757.
2021-07-15 00:35:53 +02:00
SWITCH_TO(CommentEnd);
LibWeb: Teach HTMLTokenizer how to tokenize comments We can now correctly tokenize the welcome.html test page. :^)
2020-05-23 01:54:26 +02:00
}
ON('>')
{
LibWeb: Utilize SourceLocation for HTMLTokenizer logging
2021-04-24 20:22:30 -07:00
log_parse_error();
LibWeb: Fully implement all comment tokenizer states
2020-06-11 02:09:07 +01:00
SWITCH_TO_AND_EMIT_CURRENT_TOKEN(Data);
LibWeb: Teach HTMLTokenizer how to tokenize comments We can now correctly tokenize the welcome.html test page. :^)
2020-05-23 01:54:26 +02:00
}
ON_EOF
{
LibWeb: Utilize SourceLocation for HTMLTokenizer logging
2021-04-24 20:22:30 -07:00
log_parse_error();
LibWeb: Fully implement all comment tokenizer states
2020-06-11 02:09:07 +01:00
EMIT_EOF;
LibWeb: Teach HTMLTokenizer how to tokenize comments We can now correctly tokenize the welcome.html test page. :^)
2020-05-23 01:54:26 +02:00
}
ANYTHING_ELSE
{
LibWeb: Remove StringBuilder from HTMLToken::m_comment_or_character
2021-05-23 09:31:40 +02:00
m_current_builder.append('-');
LibWeb: Teach HTMLTokenizer how to tokenize comments We can now correctly tokenize the welcome.html test page. :^)
2020-05-23 01:54:26 +02:00
RECONSUME_IN(Comment);
}
}
END_STATE
LibWeb: Add spec links to each HTML tokenizer state section I didn't add full spec comments this time, but this is better than nothing :^)
2022-02-15 18:12:15 +00:00
// 13.2.5.45 Comment state, https://html.spec.whatwg.org/multipage/parsing.html#comment-state
LibWeb: Teach HTMLTokenizer how to tokenize comments We can now correctly tokenize the welcome.html test page. :^)
2020-05-23 01:54:26 +02:00
BEGIN_STATE(Comment)
{
ON('<')
{
LibWeb: Remove StringBuilder from HTMLToken::m_comment_or_character
2021-05-23 09:31:40 +02:00
m_current_builder.append_code_point(current_input_character.value());
SWITCH_TO_WITH_UNCLEAN_BUILDER(CommentLessThanSign);
LibWeb: Teach HTMLTokenizer how to tokenize comments We can now correctly tokenize the welcome.html test page. :^)
2020-05-23 01:54:26 +02:00
}
ON('-')
{
LibWeb: Remove StringBuilder from HTMLToken::m_comment_or_character
2021-05-23 09:31:40 +02:00
SWITCH_TO_WITH_UNCLEAN_BUILDER(CommentEndDash);
LibWeb: Teach HTMLTokenizer how to tokenize comments We can now correctly tokenize the welcome.html test page. :^)
2020-05-23 01:54:26 +02:00
}
ON(0)
{
LibWeb: Utilize SourceLocation for HTMLTokenizer logging
2021-04-24 20:22:30 -07:00
log_parse_error();
LibWeb: Remove StringBuilder from HTMLToken::m_comment_or_character
2021-05-23 09:31:40 +02:00
m_current_builder.append_code_point(0xFFFD);
LibWeb: Fully implement all comment tokenizer states
2020-06-11 02:09:07 +01:00
continue;
LibWeb: Teach HTMLTokenizer how to tokenize comments We can now correctly tokenize the welcome.html test page. :^)
2020-05-23 01:54:26 +02:00
}
ON_EOF
{
LibWeb: Utilize SourceLocation for HTMLTokenizer logging
2021-04-24 20:22:30 -07:00
log_parse_error();
LibWeb: User getter and setter for Comment type HTMLTokens
2021-07-14 23:32:18 +02:00
m_current_token.set_comment(consume_current_builder());
LibWeb: Fully implement all comment tokenizer states
2020-06-11 02:09:07 +01:00
EMIT_EOF;
LibWeb: Teach HTMLTokenizer how to tokenize comments We can now correctly tokenize the welcome.html test page. :^)
2020-05-23 01:54:26 +02:00
}
ANYTHING_ELSE
{
LibWeb: Remove StringBuilder from HTMLToken::m_comment_or_character
2021-05-23 09:31:40 +02:00
m_current_builder.append_code_point(current_input_character.value());
LibWeb: Teach HTMLTokenizer how to tokenize comments We can now correctly tokenize the welcome.html test page. :^)
2020-05-23 01:54:26 +02:00
continue;
}
}
LibWeb: Make BEGIN_STATE and END_STATE include some {{{ and }}} This makes it a compile error to omit the END_STATE. Also add some more missing END_STATE's exposed by this (nice!) Thanks to @predmond for suggesting the multi-pair trick! :^)
2020-05-23 14:04:53 +02:00
END_STATE
LibWeb: Teach HTMLTokenizer how to tokenize comments We can now correctly tokenize the welcome.html test page. :^)
2020-05-23 01:54:26 +02:00
LibWeb: Add spec links to each HTML tokenizer state section I didn't add full spec comments this time, but this is better than nothing :^)
2022-02-15 18:12:15 +00:00
// 13.2.5.51 Comment end state, https://html.spec.whatwg.org/multipage/parsing.html#comment-end-state
LibWeb: Teach HTMLTokenizer how to tokenize comments We can now correctly tokenize the welcome.html test page. :^)
2020-05-23 01:54:26 +02:00
BEGIN_STATE(CommentEnd)
{
ON('>')
{
LibWeb: User getter and setter for Comment type HTMLTokens
2021-07-14 23:32:18 +02:00
m_current_token.set_comment(consume_current_builder());
LibWeb: Start building the tree building part of the new HTML parser This patch adds a new HTMLDocumentParser class. It keeps a tokenizer object internally and feeds itself with one token at a time from it. The names and idioms in this class are expressed as closely to the actual HTML parsing spec as possible, to make development as easy and bug free as possible. :^) This is going to become pretty large, but it's pretty cool!
2020-05-24 00:14:23 +02:00
SWITCH_TO_AND_EMIT_CURRENT_TOKEN(Data);
LibWeb: Teach HTMLTokenizer how to tokenize comments We can now correctly tokenize the welcome.html test page. :^)
2020-05-23 01:54:26 +02:00
}
ON('!')
{
LibWeb: Correct behavior of Comment* states in HTMLTokenizer Previously, this would lead to assertion failures when parsing HTML comments. This fixes #8757.
2021-07-15 00:35:53 +02:00
SWITCH_TO_WITH_UNCLEAN_BUILDER(CommentEndBang);
LibWeb: Teach HTMLTokenizer how to tokenize comments We can now correctly tokenize the welcome.html test page. :^)
2020-05-23 01:54:26 +02:00
}
ON('-')
{
LibWeb: Remove StringBuilder from HTMLToken::m_comment_or_character
2021-05-23 09:31:40 +02:00
m_current_builder.append('-');
LibWeb: Teach HTMLTokenizer how to tokenize comments We can now correctly tokenize the welcome.html test page. :^)
2020-05-23 01:54:26 +02:00
continue;
}
ON_EOF
{
LibWeb: Utilize SourceLocation for HTMLTokenizer logging
2021-04-24 20:22:30 -07:00
log_parse_error();
LibWeb: User getter and setter for Comment type HTMLTokens
2021-07-14 23:32:18 +02:00
m_current_token.set_comment(consume_current_builder());
LibWeb: Fully implement all comment tokenizer states
2020-06-11 02:09:07 +01:00
EMIT_EOF;
LibWeb: Teach HTMLTokenizer how to tokenize comments We can now correctly tokenize the welcome.html test page. :^)
2020-05-23 01:54:26 +02:00
}
ANYTHING_ELSE
{
LibWeb: Fix 'Comment end state' in HTML Tokenizer Also, update the expected hash in the LibWeb TestHTMLTokenizer regression test. This is due to the "This comment has a few too many dashes." comment token being updated.
2022-02-18 23:02:52 +00:00
m_current_builder.append("--");
LibWeb: Teach HTMLTokenizer how to tokenize comments We can now correctly tokenize the welcome.html test page. :^)
2020-05-23 01:54:26 +02:00
RECONSUME_IN(Comment);
}
}
END_STATE
LibWeb: Add spec links to each HTML tokenizer state section I didn't add full spec comments this time, but this is better than nothing :^)
2022-02-15 18:12:15 +00:00
// 13.2.5.52 Comment end bang state, https://html.spec.whatwg.org/multipage/parsing.html#comment-end-bang-state
LibWeb: Teach HTMLTokenizer how to tokenize comments We can now correctly tokenize the welcome.html test page. :^)
2020-05-23 01:54:26 +02:00
BEGIN_STATE(CommentEndBang)
{
ON('-')
{
LibWeb: Remove StringBuilder from HTMLToken::m_comment_or_character
2021-05-23 09:31:40 +02:00
m_current_builder.append("--!");
LibWeb: Correct behavior of Comment* states in HTMLTokenizer Previously, this would lead to assertion failures when parsing HTML comments. This fixes #8757.
2021-07-15 00:35:53 +02:00
SWITCH_TO_WITH_UNCLEAN_BUILDER(CommentEndDash);
LibWeb: Teach HTMLTokenizer how to tokenize comments We can now correctly tokenize the welcome.html test page. :^)
2020-05-23 01:54:26 +02:00
}
ON('>')
{
LibWeb: Utilize SourceLocation for HTMLTokenizer logging
2021-04-24 20:22:30 -07:00
log_parse_error();
LibWeb: User getter and setter for Comment type HTMLTokens
2021-07-14 23:32:18 +02:00
m_current_token.set_comment(consume_current_builder());
LibWeb: Fully implement all comment tokenizer states
2020-06-11 02:09:07 +01:00
SWITCH_TO_AND_EMIT_CURRENT_TOKEN(Data);
LibWeb: Teach HTMLTokenizer how to tokenize comments We can now correctly tokenize the welcome.html test page. :^)
2020-05-23 01:54:26 +02:00
}
ON_EOF
{
LibWeb: Utilize SourceLocation for HTMLTokenizer logging
2021-04-24 20:22:30 -07:00
log_parse_error();
LibWeb: User getter and setter for Comment type HTMLTokens
2021-07-14 23:32:18 +02:00
m_current_token.set_comment(consume_current_builder());
LibWeb: Fully implement all comment tokenizer states
2020-06-11 02:09:07 +01:00
EMIT_EOF;
LibWeb: Teach HTMLTokenizer how to tokenize comments We can now correctly tokenize the welcome.html test page. :^)
2020-05-23 01:54:26 +02:00
}
ANYTHING_ELSE
{
LibWeb: Remove StringBuilder from HTMLToken::m_comment_or_character
2021-05-23 09:31:40 +02:00
m_current_builder.append("--!");
LibWeb: Teach HTMLTokenizer how to tokenize comments We can now correctly tokenize the welcome.html test page. :^)
2020-05-23 01:54:26 +02:00
RECONSUME_IN(Comment);
}
}
END_STATE
LibWeb: Add spec links to each HTML tokenizer state section I didn't add full spec comments this time, but this is better than nothing :^)
2022-02-15 18:12:15 +00:00
// 13.2.5.50 Comment end dash state, https://html.spec.whatwg.org/multipage/parsing.html#comment-end-dash-state
LibWeb: Teach HTMLTokenizer how to tokenize comments We can now correctly tokenize the welcome.html test page. :^)
2020-05-23 01:54:26 +02:00
BEGIN_STATE(CommentEndDash)
{
ON('-')
{
LibWeb: Remove StringBuilder from HTMLToken::m_comment_or_character
2021-05-23 09:31:40 +02:00
SWITCH_TO_WITH_UNCLEAN_BUILDER(CommentEnd);
LibWeb: Teach HTMLTokenizer how to tokenize comments We can now correctly tokenize the welcome.html test page. :^)
2020-05-23 01:54:26 +02:00
}
ON_EOF
{
LibWeb: Utilize SourceLocation for HTMLTokenizer logging
2021-04-24 20:22:30 -07:00
log_parse_error();
LibWeb: User getter and setter for Comment type HTMLTokens
2021-07-14 23:32:18 +02:00
m_current_token.set_comment(consume_current_builder());
LibWeb: Fully implement all comment tokenizer states
2020-06-11 02:09:07 +01:00
EMIT_EOF;
LibWeb: Teach HTMLTokenizer how to tokenize comments We can now correctly tokenize the welcome.html test page. :^)
2020-05-23 01:54:26 +02:00
}
ANYTHING_ELSE
{
LibWeb: Remove StringBuilder from HTMLToken::m_comment_or_character
2021-05-23 09:31:40 +02:00
m_current_builder.append('-');
LibWeb: Teach HTMLTokenizer how to tokenize comments We can now correctly tokenize the welcome.html test page. :^)
2020-05-23 01:54:26 +02:00
RECONSUME_IN(Comment);
}
}
LibWeb: Make BEGIN_STATE and END_STATE include some {{{ and }}} This makes it a compile error to omit the END_STATE. Also add some more missing END_STATE's exposed by this (nice!) Thanks to @predmond for suggesting the multi-pair trick! :^)
2020-05-23 14:04:53 +02:00
END_STATE
LibWeb: Teach HTMLTokenizer how to tokenize comments We can now correctly tokenize the welcome.html test page. :^)
2020-05-23 01:54:26 +02:00
LibWeb: Add spec links to each HTML tokenizer state section I didn't add full spec comments this time, but this is better than nothing :^)
2022-02-15 18:12:15 +00:00
// 13.2.5.46 Comment less-than sign state, https://html.spec.whatwg.org/multipage/parsing.html#comment-less-than-sign-state
LibWeb: Teach HTMLTokenizer how to tokenize comments We can now correctly tokenize the welcome.html test page. :^)
2020-05-23 01:54:26 +02:00
BEGIN_STATE(CommentLessThanSign)
{
ON('!')
{
LibWeb: Remove StringBuilder from HTMLToken::m_comment_or_character
2021-05-23 09:31:40 +02:00
m_current_builder.append_code_point(current_input_character.value());
SWITCH_TO_WITH_UNCLEAN_BUILDER(CommentLessThanSignBang);
LibWeb: Teach HTMLTokenizer how to tokenize comments We can now correctly tokenize the welcome.html test page. :^)
2020-05-23 01:54:26 +02:00
}
ON('<')
{
LibWeb: Remove StringBuilder from HTMLToken::m_comment_or_character
2021-05-23 09:31:40 +02:00
m_current_builder.append_code_point(current_input_character.value());
LibWeb: Teach HTMLTokenizer how to tokenize comments We can now correctly tokenize the welcome.html test page. :^)
2020-05-23 01:54:26 +02:00
continue;
}
ANYTHING_ELSE
{
RECONSUME_IN(Comment);
}
}
END_STATE
LibWeb: Add spec links to each HTML tokenizer state section I didn't add full spec comments this time, but this is better than nothing :^)
2022-02-15 18:12:15 +00:00
// 13.2.5.47 Comment less-than sign bang state, https://html.spec.whatwg.org/multipage/parsing.html#comment-less-than-sign-bang-state
LibWeb: Teach HTMLTokenizer how to tokenize comments We can now correctly tokenize the welcome.html test page. :^)
2020-05-23 01:54:26 +02:00
BEGIN_STATE(CommentLessThanSignBang)
{
ON('-')
{
LibWeb: Correct behavior of Comment* states in HTMLTokenizer Previously, this would lead to assertion failures when parsing HTML comments. This fixes #8757.
2021-07-15 00:35:53 +02:00
SWITCH_TO_WITH_UNCLEAN_BUILDER(CommentLessThanSignBangDash);
LibWeb: Teach HTMLTokenizer how to tokenize comments We can now correctly tokenize the welcome.html test page. :^)
2020-05-23 01:54:26 +02:00
}
ANYTHING_ELSE
{
RECONSUME_IN(Comment);
}
}
END_STATE
LibWeb: Add spec links to each HTML tokenizer state section I didn't add full spec comments this time, but this is better than nothing :^)
2022-02-15 18:12:15 +00:00
// 13.2.5.48 Comment less-than sign bang dash state, https://html.spec.whatwg.org/multipage/parsing.html#comment-less-than-sign-bang-dash-state
LibWeb: Teach HTMLTokenizer how to tokenize comments We can now correctly tokenize the welcome.html test page. :^)
2020-05-23 01:54:26 +02:00
BEGIN_STATE(CommentLessThanSignBangDash)
{
ON('-')
{
LibWeb: Correct behavior of Comment* states in HTMLTokenizer Previously, this would lead to assertion failures when parsing HTML comments. This fixes #8757.
2021-07-15 00:35:53 +02:00
SWITCH_TO_WITH_UNCLEAN_BUILDER(CommentLessThanSignBangDashDash);
LibWeb: Teach HTMLTokenizer how to tokenize comments We can now correctly tokenize the welcome.html test page. :^)
2020-05-23 01:54:26 +02:00
}
ANYTHING_ELSE
{
LibWeb: Fully implement all comment tokenizer states
2020-06-11 02:09:07 +01:00
RECONSUME_IN(CommentEndDash);
LibWeb: Teach HTMLTokenizer how to tokenize comments We can now correctly tokenize the welcome.html test page. :^)
2020-05-23 01:54:26 +02:00
}
}
END_STATE
LibWeb: Add spec links to each HTML tokenizer state section I didn't add full spec comments this time, but this is better than nothing :^)
2022-02-15 18:12:15 +00:00
// 13.2.5.49 Comment less-than sign bang dash dash state, https://html.spec.whatwg.org/multipage/parsing.html#comment-less-than-sign-bang-dash-dash-state
LibWeb: Teach HTMLTokenizer how to tokenize comments We can now correctly tokenize the welcome.html test page. :^)
2020-05-23 01:54:26 +02:00
BEGIN_STATE(CommentLessThanSignBangDashDash)
{
ON('>')
{
LibWeb: Fully implement all comment tokenizer states
2020-06-11 02:09:07 +01:00
RECONSUME_IN(CommentEnd);
}
ON_EOF
{
RECONSUME_IN(CommentEnd);
LibWeb: Teach HTMLTokenizer how to tokenize comments We can now correctly tokenize the welcome.html test page. :^)
2020-05-23 01:54:26 +02:00
}
ANYTHING_ELSE
{
LibWeb: Utilize SourceLocation for HTMLTokenizer logging
2021-04-24 20:22:30 -07:00
log_parse_error();
LibWeb: Tokenize "anything else" in CommentLessThanSignBangDashDash
2020-06-01 20:14:23 +02:00
RECONSUME_IN(CommentEnd);
LibWeb: Teach HTMLTokenizer how to tokenize comments We can now correctly tokenize the welcome.html test page. :^)
2020-05-23 01:54:26 +02:00
}
}
END_STATE
LibWeb: Add spec links to each HTML tokenizer state section I didn't add full spec comments this time, but this is better than nothing :^)
2022-02-15 18:12:15 +00:00
// 13.2.5.72 Character reference state, https://html.spec.whatwg.org/multipage/parsing.html#character-reference-state
LibWeb: Begin work on a spec-compliant HTML parser In order to actually view the web as it is, we're gonna need a proper HTML parser. So let's build one! This patch introduces the Web::HTMLTokenizer class, which currently operates on a StringView input stream where it fetches (ASCII only atm) codepoints and tokenizes acccording to the HTML spec tokenization algo. The tokenizer state machine looks a bit weird but is written in a way that tries to mimic the spec as closely as possible, in order to make development easier and bugs less likely. This initial version is far from finished, but it can parse a trivial document with a DOCTYPE and open/close tags. :^)
2020-05-22 21:46:13 +02:00
BEGIN_STATE(CharacterReference)
{
LibWeb: Implement partial support for numeric character references
2020-05-27 16:16:23 +02:00
m_temporary_buffer.clear();
m_temporary_buffer.append('&');
ON_ASCII_ALPHANUMERIC
{
RECONSUME_IN(NamedCharacterReference);
}
ON('#')
{
m_temporary_buffer.append(current_input_character.value());
LibWeb: Avoid assertion failure on parsing numeric character references
2021-07-28 00:37:26 +01:00
SWITCH_TO_WITH_UNCLEAN_BUILDER(NumericCharacterReference);
LibWeb: Implement partial support for numeric character references
2020-05-27 16:16:23 +02:00
}
ANYTHING_ELSE
{
LibWeb: Fix character references losing characters in certain situations This fixes 4 issues: - RECONSUME_IN_RETURN_STATE was functionally equivalent to SWITCH_TO_RETURN_STATE, which caused us to lose characters. For example, &test= would lose the = - & characters by themselves would be lost. For example, 1 & 2 would become 1 2. This is because we forgot to flush characters in the the ANYTHING_ELSE path in CharacterReference - Named character references didn't work at all in attributes. This is because there was a path that was checking the entity code points instead of the entity itself. Plus, the path that was checking the entity itself wasn't quite spec compliant. - If we fail to match a named character reference, the first character is lost. For example &test would become &est. However, this relies on a little hack since I can't wrap my head around on how to change the code to do as the spec says. The hack is to reconsume in AmbigiousAmpersand instead of just switching to it. Fixes #3957
2020-12-29 14:42:47 +00:00
FLUSH_CODEPOINTS_CONSUMED_AS_A_CHARACTER_REFERENCE;
LibWeb: Implement partial support for numeric character references
2020-05-27 16:16:23 +02:00
RECONSUME_IN_RETURN_STATE;
}
}
END_STATE
LibWeb: Add spec links to each HTML tokenizer state section I didn't add full spec comments this time, but this is better than nothing :^)
2022-02-15 18:12:15 +00:00
// 13.2.5.73 Named character reference state, https://html.spec.whatwg.org/multipage/parsing.html#named-character-reference-state
LibWeb: Implement partial support for numeric character references
2020-05-27 16:16:23 +02:00
BEGIN_STATE(NamedCharacterReference)
{
LibWeb: Unbreak character reference and DOCTYPE parsing post-UTF-8 Oops, these were still using the byte-offset cursor. My goodness is it unergonomic to index into UTF-8 strings, but Dr. Bugaev says it's good. There is lots of room for improvement here. Just like the rest of the tokenizer and parser. We'll have to do a few optimization passes over them once they mature.
2020-06-04 22:02:54 +02:00
size_t byte_offset = m_utf8_view.byte_offset_of(m_prev_utf8_iterator);
LibWeb: Fix off by one error in HTML Tokenizer In 'NamedCharacterReference' we attempt to lookup the code point by a identifier, eg apos; becomes ' This is done by passing the entire rest of the document to the `HTML::code_points_from_entity` function. However, before this change we didn't sent the final character which meant if the document ended in a named character reference the lookup would fail.
2022-02-18 22:12:47 +00:00
auto match = HTML::code_points_from_entity(m_decoded_input.substring_view(byte_offset, m_decoded_input.length() - byte_offset));
LibWeb: Support named character references (e.g "&amp;")
2020-05-28 11:44:19 +02:00
if (match.has_value()) {
LibWeb: Add position tracking information to HTML tokens
2021-05-20 23:11:41 +04:30
skip(match->entity.length() - 1);
LibWeb: Support named character references (e.g "&amp;")
2020-05-28 11:44:19 +02:00
for (auto ch : match.value().entity)
m_temporary_buffer.append(ch);
LibWeb: Fix character references losing characters in certain situations This fixes 4 issues: - RECONSUME_IN_RETURN_STATE was functionally equivalent to SWITCH_TO_RETURN_STATE, which caused us to lose characters. For example, &test= would lose the = - & characters by themselves would be lost. For example, 1 & 2 would become 1 2. This is because we forgot to flush characters in the the ANYTHING_ELSE path in CharacterReference - Named character references didn't work at all in attributes. This is because there was a path that was checking the entity code points instead of the entity itself. Plus, the path that was checking the entity itself wasn't quite spec compliant. - If we fail to match a named character reference, the first character is lost. For example &test would become &est. However, this relies on a little hack since I can't wrap my head around on how to change the code to do as the spec says. The hack is to reconsume in AmbigiousAmpersand instead of just switching to it. Fixes #3957
2020-12-29 14:42:47 +00:00
if (consumed_as_part_of_an_attribute() && !match.value().entity.ends_with(';')) {
Unicode: Try s/codepoint/code_point/g again This time, without trailing 's'. Ran: git grep -l 'codepoint' | xargs sed -ie 's/codepoint/code_point/g
2020-08-05 16:31:20 -04:00
auto next_code_point = peek_code_point(0);
Everywhere: Replace ctype.h to avoid narrowing conversions This replaces ctype.h with CharacterType.h everywhere I could find issues with narrowing conversions. While using it will probably make sense almost everywhere in the future, the most critical places should have been addressed.
2021-06-01 21:18:08 +02:00
if (next_code_point.has_value() && (next_code_point.value() == '=' || is_ascii_alphanumeric(next_code_point.value()))) {
Revert "Unicode: s/codepoint/code_point/g" This reverts commit ea9ac3155d1774f13ac4e9a96605c0e85a8f299e. It replaced "codepoint" with "code_points", not "code_point".
2020-08-05 16:28:19 -04:00
FLUSH_CODEPOINTS_CONSUMED_AS_A_CHARACTER_REFERENCE;
LibWeb: Support named character references (e.g "&amp;")
2020-05-28 11:44:19 +02:00
SWITCH_TO_RETURN_STATE;
}
}
if (!match.value().entity.ends_with(';')) {
LibWeb: Utilize SourceLocation for HTMLTokenizer logging
2021-04-24 20:22:30 -07:00
log_parse_error();
LibWeb: Support named character references (e.g "&amp;")
2020-05-28 11:44:19 +02:00
}
AK: Rename Vector::append(Vector) => Vector::extend(Vector) Let's make it a bit more clear when we're appending the elements from one vector to the end of another vector.
2021-06-12 13:24:45 +02:00
m_temporary_buffer = match.value().code_points;
LibWeb: Support named character references (e.g "&amp;")
2020-05-28 11:44:19 +02:00
Revert "Unicode: s/codepoint/code_point/g" This reverts commit ea9ac3155d1774f13ac4e9a96605c0e85a8f299e. It replaced "codepoint" with "code_points", not "code_point".
2020-08-05 16:28:19 -04:00
FLUSH_CODEPOINTS_CONSUMED_AS_A_CHARACTER_REFERENCE;
LibWeb: Support named character references (e.g "&amp;")
2020-05-28 11:44:19 +02:00
SWITCH_TO_RETURN_STATE;
} else {
Revert "Unicode: s/codepoint/code_point/g" This reverts commit ea9ac3155d1774f13ac4e9a96605c0e85a8f299e. It replaced "codepoint" with "code_points", not "code_point".
2020-08-05 16:28:19 -04:00
FLUSH_CODEPOINTS_CONSUMED_AS_A_CHARACTER_REFERENCE;
LibWeb: Fix character references losing characters in certain situations This fixes 4 issues: - RECONSUME_IN_RETURN_STATE was functionally equivalent to SWITCH_TO_RETURN_STATE, which caused us to lose characters. For example, &test= would lose the = - & characters by themselves would be lost. For example, 1 & 2 would become 1 2. This is because we forgot to flush characters in the the ANYTHING_ELSE path in CharacterReference - Named character references didn't work at all in attributes. This is because there was a path that was checking the entity code points instead of the entity itself. Plus, the path that was checking the entity itself wasn't quite spec compliant. - If we fail to match a named character reference, the first character is lost. For example &test would become &est. However, this relies on a little hack since I can't wrap my head around on how to change the code to do as the spec says. The hack is to reconsume in AmbigiousAmpersand instead of just switching to it. Fixes #3957
2020-12-29 14:42:47 +00:00
// FIXME: This should be SWITCH_TO, but we always lose the first character on this path, so just reconsume it.
// I can't wrap my head around how to do it as the spec says.
RECONSUME_IN(AmbiguousAmpersand);
LibWeb: Support named character references (e.g "&amp;")
2020-05-28 11:44:19 +02:00
}
}
END_STATE
LibWeb: Add spec links to each HTML tokenizer state section I didn't add full spec comments this time, but this is better than nothing :^)
2022-02-15 18:12:15 +00:00
// 13.2.5.74 Ambiguous ampersand state, https://html.spec.whatwg.org/multipage/parsing.html#ambiguous-ampersand-state
LibWeb: Support named character references (e.g "&amp;")
2020-05-28 11:44:19 +02:00
BEGIN_STATE(AmbiguousAmpersand)
{
ON_ASCII_ALPHANUMERIC
{
if (consumed_as_part_of_an_attribute()) {
LibWeb: Remove StringBuilders from HTMLToken::AttributeBuilder
2021-05-23 08:50:48 +02:00
m_current_builder.append_code_point(current_input_character.value());
LibWeb: Support named character references (e.g "&amp;")
2020-05-28 11:44:19 +02:00
continue;
} else {
EMIT_CURRENT_CHARACTER;
}
}
LibWeb: Implement partial support for numeric character references
2020-05-27 16:16:23 +02:00
ON(';')
{
LibWeb: Utilize SourceLocation for HTMLTokenizer logging
2021-04-24 20:22:30 -07:00
log_parse_error();
LibWeb: Implement almost all missing tokenizer cases
2020-06-26 22:41:35 +01:00
RECONSUME_IN_RETURN_STATE;
LibWeb: Implement partial support for numeric character references
2020-05-27 16:16:23 +02:00
}
ANYTHING_ELSE
{
LibWeb: Support named character references (e.g "&amp;")
2020-05-28 11:44:19 +02:00
RECONSUME_IN_RETURN_STATE;
LibWeb: Implement partial support for numeric character references
2020-05-27 16:16:23 +02:00
}
}
END_STATE
LibWeb: Add spec links to each HTML tokenizer state section I didn't add full spec comments this time, but this is better than nothing :^)
2022-02-15 18:12:15 +00:00
// 13.2.5.75 Numeric character reference state, https://html.spec.whatwg.org/multipage/parsing.html#numeric-character-reference-state
LibWeb: Implement partial support for numeric character references
2020-05-27 16:16:23 +02:00
BEGIN_STATE(NumericCharacterReference)
{
m_character_reference_code = 0;
ON('X')
{
m_temporary_buffer.append(current_input_character.value());
LibWeb: Avoid assertion failure on parsing numeric character references
2021-07-28 00:37:26 +01:00
SWITCH_TO_WITH_UNCLEAN_BUILDER(HexadecimalCharacterReferenceStart);
LibWeb: Implement partial support for numeric character references
2020-05-27 16:16:23 +02:00
}
ON('x')
{
m_temporary_buffer.append(current_input_character.value());
LibWeb: Avoid assertion failure on parsing numeric character references
2021-07-28 00:37:26 +01:00
SWITCH_TO_WITH_UNCLEAN_BUILDER(HexadecimalCharacterReferenceStart);
LibWeb: Implement partial support for numeric character references
2020-05-27 16:16:23 +02:00
}
ANYTHING_ELSE
{
RECONSUME_IN(DecimalCharacterReferenceStart);
}
}
END_STATE
LibWeb: Add spec links to each HTML tokenizer state section I didn't add full spec comments this time, but this is better than nothing :^)
2022-02-15 18:12:15 +00:00
// 13.2.5.76 Hexadecimal character reference start state, https://html.spec.whatwg.org/multipage/parsing.html#hexadecimal-character-reference-start-state
LibWeb: Implement partial support for numeric character references
2020-05-27 16:16:23 +02:00
BEGIN_STATE(HexadecimalCharacterReferenceStart)
{
ON_ASCII_HEX_DIGIT
{
RECONSUME_IN(HexadecimalCharacterReference);
}
ANYTHING_ELSE
{
LibWeb: Utilize SourceLocation for HTMLTokenizer logging
2021-04-24 20:22:30 -07:00
log_parse_error();
Revert "Unicode: s/codepoint/code_point/g" This reverts commit ea9ac3155d1774f13ac4e9a96605c0e85a8f299e. It replaced "codepoint" with "code_points", not "code_point".
2020-08-05 16:28:19 -04:00
FLUSH_CODEPOINTS_CONSUMED_AS_A_CHARACTER_REFERENCE;
LibWeb: Implement almost all missing tokenizer cases
2020-06-26 22:41:35 +01:00
RECONSUME_IN_RETURN_STATE;
LibWeb: Implement partial support for numeric character references
2020-05-27 16:16:23 +02:00
}
}
END_STATE
LibWeb: Add spec links to each HTML tokenizer state section I didn't add full spec comments this time, but this is better than nothing :^)
2022-02-15 18:12:15 +00:00
// 13.2.5.77 Decimal character reference start state, https://html.spec.whatwg.org/multipage/parsing.html#decimal-character-reference-start-state
LibWeb: Implement partial support for numeric character references
2020-05-27 16:16:23 +02:00
BEGIN_STATE(DecimalCharacterReferenceStart)
{
ON_ASCII_DIGIT
{
RECONSUME_IN(DecimalCharacterReference);
}
ANYTHING_ELSE
{
LibWeb: Utilize SourceLocation for HTMLTokenizer logging
2021-04-24 20:22:30 -07:00
log_parse_error();
Revert "Unicode: s/codepoint/code_point/g" This reverts commit ea9ac3155d1774f13ac4e9a96605c0e85a8f299e. It replaced "codepoint" with "code_points", not "code_point".
2020-08-05 16:28:19 -04:00
FLUSH_CODEPOINTS_CONSUMED_AS_A_CHARACTER_REFERENCE;
LibWeb: Implement almost all missing tokenizer cases
2020-06-26 22:41:35 +01:00
RECONSUME_IN_RETURN_STATE;
LibWeb: Implement partial support for numeric character references
2020-05-27 16:16:23 +02:00
}
}
END_STATE
LibWeb: Add spec links to each HTML tokenizer state section I didn't add full spec comments this time, but this is better than nothing :^)
2022-02-15 18:12:15 +00:00
// 13.2.5.78 Hexadecimal character reference state, https://html.spec.whatwg.org/multipage/parsing.html#decimal-character-reference-start-state
LibWeb: Implement partial support for numeric character references
2020-05-27 16:16:23 +02:00
BEGIN_STATE(HexadecimalCharacterReference)
{
ON_ASCII_DIGIT
{
m_character_reference_code *= 16;
m_character_reference_code += current_input_character.value() - 0x30;
continue;
}
ON_ASCII_UPPER_ALPHA
{
m_character_reference_code *= 16;
m_character_reference_code += current_input_character.value() - 0x37;
continue;
}
ON_ASCII_LOWER_ALPHA
{
m_character_reference_code *= 16;
m_character_reference_code += current_input_character.value() - 0x57;
continue;
}
ON(';')
{
LibWeb: Avoid assertion failure on parsing numeric character references
2021-07-28 00:37:26 +01:00
SWITCH_TO_WITH_UNCLEAN_BUILDER(NumericCharacterReferenceEnd);
LibWeb: Implement partial support for numeric character references
2020-05-27 16:16:23 +02:00
}
ANYTHING_ELSE
{
LibWeb: Utilize SourceLocation for HTMLTokenizer logging
2021-04-24 20:22:30 -07:00
log_parse_error();
LibWeb: Implement almost all missing tokenizer cases
2020-06-26 22:41:35 +01:00
RECONSUME_IN(NumericCharacterReferenceEnd);
LibWeb: Implement partial support for numeric character references
2020-05-27 16:16:23 +02:00
}
}
END_STATE
LibWeb: Add spec links to each HTML tokenizer state section I didn't add full spec comments this time, but this is better than nothing :^)
2022-02-15 18:12:15 +00:00
// 13.2.5.79 Decimal character reference state, https://html.spec.whatwg.org/multipage/parsing.html#decimal-character-reference-state
LibWeb: Implement partial support for numeric character references
2020-05-27 16:16:23 +02:00
BEGIN_STATE(DecimalCharacterReference)
{
ON_ASCII_DIGIT
{
m_character_reference_code *= 10;
m_character_reference_code += current_input_character.value() - 0x30;
continue;
}
ON(';')
{
LibWeb: Avoid assertion failure on parsing numeric character references
2021-07-28 00:37:26 +01:00
SWITCH_TO_WITH_UNCLEAN_BUILDER(NumericCharacterReferenceEnd);
LibWeb: Implement partial support for numeric character references
2020-05-27 16:16:23 +02:00
}
ANYTHING_ELSE
{
LibWeb: Utilize SourceLocation for HTMLTokenizer logging
2021-04-24 20:22:30 -07:00
log_parse_error();
LibWeb: Implement almost all missing tokenizer cases
2020-06-26 22:41:35 +01:00
RECONSUME_IN(NumericCharacterReferenceEnd);
LibWeb: Implement partial support for numeric character references
2020-05-27 16:16:23 +02:00
}
}
END_STATE
LibWeb: Add spec links to each HTML tokenizer state section I didn't add full spec comments this time, but this is better than nothing :^)
2022-02-15 18:12:15 +00:00
// 13.2.5.80 Numeric character reference end state, https://html.spec.whatwg.org/multipage/parsing.html#numeric-character-reference-end-state
LibWeb: Implement partial support for numeric character references
2020-05-27 16:16:23 +02:00
BEGIN_STATE(NumericCharacterReferenceEnd)
{
LibWeb: Fix incorrectly consumed characters after reference tokens The NumericCharacterReferenceEnd tokenizer state should not advance the input stream.
2020-06-04 16:48:15 +02:00
DONT_CONSUME_NEXT_INPUT_CHARACTER;
LibWeb: Implement partial support for numeric character references
2020-05-27 16:16:23 +02:00
if (m_character_reference_code == 0) {
LibWeb: Utilize SourceLocation for HTMLTokenizer logging
2021-04-24 20:22:30 -07:00
log_parse_error();
LibWeb: Implement almost all missing tokenizer cases
2020-06-26 22:41:35 +01:00
m_character_reference_code = 0xFFFD;
LibWeb: Implement partial support for numeric character references
2020-05-27 16:16:23 +02:00
}
if (m_character_reference_code > 0x10ffff) {
LibWeb: Utilize SourceLocation for HTMLTokenizer logging
2021-04-24 20:22:30 -07:00
log_parse_error();
LibWeb: Implement almost all missing tokenizer cases
2020-06-26 22:41:35 +01:00
m_character_reference_code = 0xFFFD;
LibWeb: Implement partial support for numeric character references
2020-05-27 16:16:23 +02:00
}
Everywhere: Replace ctype.h to avoid narrowing conversions This replaces ctype.h with CharacterType.h everywhere I could find issues with narrowing conversions. While using it will probably make sense almost everywhere in the future, the most critical places should have been addressed.
2021-06-01 21:18:08 +02:00
if (is_unicode_surrogate(m_character_reference_code)) {
LibWeb: Utilize SourceLocation for HTMLTokenizer logging
2021-04-24 20:22:30 -07:00
log_parse_error();
LibWeb: Implement almost all missing tokenizer cases
2020-06-26 22:41:35 +01:00
m_character_reference_code = 0xFFFD;
LibWeb: Implement partial support for numeric character references
2020-05-27 16:16:23 +02:00
}
Everywhere: Replace ctype.h to avoid narrowing conversions This replaces ctype.h with CharacterType.h everywhere I could find issues with narrowing conversions. While using it will probably make sense almost everywhere in the future, the most critical places should have been addressed.
2021-06-01 21:18:08 +02:00
if (is_unicode_noncharacter(m_character_reference_code)) {
LibWeb: Utilize SourceLocation for HTMLTokenizer logging
2021-04-24 20:22:30 -07:00
log_parse_error();
LibWeb: Implement partial support for numeric character references
2020-05-27 16:16:23 +02:00
}
Everywhere: Replace ctype.h to avoid narrowing conversions This replaces ctype.h with CharacterType.h everywhere I could find issues with narrowing conversions. While using it will probably make sense almost everywhere in the future, the most critical places should have been addressed.
2021-06-01 21:18:08 +02:00
if (m_character_reference_code == 0xd || (is_unicode_control(m_character_reference_code) && !is_ascii_space(m_character_reference_code))) {
LibWeb: Utilize SourceLocation for HTMLTokenizer logging
2021-04-24 20:22:30 -07:00
log_parse_error();
LibWeb: Implement partial support for numeric character references
2020-05-27 16:16:23 +02:00
constexpr struct {
u32 number;
Unicode: Try s/codepoint/code_point/g again This time, without trailing 's'. Ran: git grep -l 'codepoint' | xargs sed -ie 's/codepoint/code_point/g
2020-08-05 16:31:20 -04:00
u32 code_point;
LibWeb: Implement partial support for numeric character references
2020-05-27 16:16:23 +02:00
} conversion_table[] = {
{ 0x80, 0x20AC },
{ 0x82, 0x201A },
{ 0x83, 0x0192 },
{ 0x84, 0x201E },
{ 0x85, 0x2026 },
{ 0x86, 0x2020 },
{ 0x87, 0x2021 },
{ 0x88, 0x02C6 },
{ 0x89, 0x2030 },
{ 0x8A, 0x0160 },
{ 0x8B, 0x2039 },
{ 0x8C, 0x0152 },
{ 0x8E, 0x017D },
{ 0x91, 0x2018 },
{ 0x92, 0x2019 },
{ 0x93, 0x201C },
{ 0x94, 0x201D },
{ 0x95, 0x2022 },
{ 0x96, 0x2013 },
{ 0x97, 0x2014 },
{ 0x98, 0x02DC },
{ 0x99, 0x2122 },
{ 0x9A, 0x0161 },
{ 0x9B, 0x203A },
{ 0x9C, 0x0153 },
{ 0x9E, 0x017E },
{ 0x9F, 0x0178 },
};
for (auto& entry : conversion_table) {
if (m_character_reference_code == entry.number) {
Unicode: Try s/codepoint/code_point/g again This time, without trailing 's'. Ran: git grep -l 'codepoint' | xargs sed -ie 's/codepoint/code_point/g
2020-08-05 16:31:20 -04:00
m_character_reference_code = entry.code_point;
LibWeb: Implement partial support for numeric character references
2020-05-27 16:16:23 +02:00
break;
}
}
}
m_temporary_buffer.clear();
m_temporary_buffer.append(m_character_reference_code);
Revert "Unicode: s/codepoint/code_point/g" This reverts commit ea9ac3155d1774f13ac4e9a96605c0e85a8f299e. It replaced "codepoint" with "code_points", not "code_point".
2020-08-05 16:28:19 -04:00
FLUSH_CODEPOINTS_CONSUMED_AS_A_CHARACTER_REFERENCE;
LibWeb: Implement partial support for numeric character references
2020-05-27 16:16:23 +02:00
SWITCH_TO_RETURN_STATE;
LibWeb: Begin work on a spec-compliant HTML parser In order to actually view the web as it is, we're gonna need a proper HTML parser. So let's build one! This patch introduces the Web::HTMLTokenizer class, which currently operates on a StringView input stream where it fetches (ASCII only atm) codepoints and tokenizes acccording to the HTML spec tokenization algo. The tokenizer state machine looks a bit weird but is written in a way that tries to mimic the spec as closely as possible, in order to make development easier and bugs less likely. This initial version is far from finished, but it can parse a trivial document with a DOCTYPE and open/close tags. :^)
2020-05-22 21:46:13 +02:00
}
END_STATE
LibWeb: Add spec links to each HTML tokenizer state section I didn't add full spec comments this time, but this is better than nothing :^)
2022-02-15 18:12:15 +00:00
// 13.2.5.2 RCDATA state, https://html.spec.whatwg.org/multipage/parsing.html#rcdata-state
LibWeb: More work on the HTML parser and tokenizer The parser can now switch the state of the tokenizer! Very webby. :^)
2020-05-24 20:24:43 +02:00
BEGIN_STATE(RCDATA)
{
ON('&')
{
m_return_state = State::RCDATA;
SWITCH_TO(CharacterReference);
}
ON('<')
{
SWITCH_TO(RCDATALessThanSign);
}
ON(0)
{
LibWeb: Utilize SourceLocation for HTMLTokenizer logging
2021-04-24 20:22:30 -07:00
log_parse_error();
LibWeb: Make the new HTML parser parse input as UTF-8 We already convert the input to UTF-8 before starting the tokenizer, so all this patch had to do was switch the tokenizer to use an Utf8View for its input (and to emit 32-bit codepoints.)
2020-06-04 21:06:54 +02:00
EMIT_CHARACTER(0xFFFD);
LibWeb: More work on the HTML parser and tokenizer The parser can now switch the state of the tokenizer! Very webby. :^)
2020-05-24 20:24:43 +02:00
}
ON_EOF
{
EMIT_EOF;
}
ANYTHING_ELSE
{
EMIT_CURRENT_CHARACTER;
}
}
END_STATE
LibWeb: Add spec links to each HTML tokenizer state section I didn't add full spec comments this time, but this is better than nothing :^)
2022-02-15 18:12:15 +00:00
// 13.2.5.9 RCDATA less-than sign state, https://html.spec.whatwg.org/multipage/parsing.html#rcdata-less-than-sign-state
LibWeb: More work on the HTML parser and tokenizer The parser can now switch the state of the tokenizer! Very webby. :^)
2020-05-24 20:24:43 +02:00
BEGIN_STATE(RCDATALessThanSign)
{
ON('/')
{
m_temporary_buffer.clear();
SWITCH_TO(RCDATAEndTagOpen);
}
ANYTHING_ELSE
{
LibWeb: Fix missing tokenizer state change in RCDATALessThanSign We can't RECONSUME_IN after we've used EMIT_CHARACTER since we'll have returned from the function.
2020-06-05 12:02:30 +02:00
EMIT_CHARACTER_AND_RECONSUME_IN('<', RCDATA);
LibWeb: More work on the HTML parser and tokenizer The parser can now switch the state of the tokenizer! Very webby. :^)
2020-05-24 20:24:43 +02:00
}
}
END_STATE
LibWeb: Add spec links to each HTML tokenizer state section I didn't add full spec comments this time, but this is better than nothing :^)
2022-02-15 18:12:15 +00:00
// 13.2.5.10 RCDATA end tag open state, https://html.spec.whatwg.org/multipage/parsing.html#rcdata-end-tag-open-state
LibWeb: More work on the HTML parser and tokenizer The parser can now switch the state of the tokenizer! Very webby. :^)
2020-05-24 20:24:43 +02:00
BEGIN_STATE(RCDATAEndTagOpen)
{
ON_ASCII_ALPHA
{
create_new_token(HTMLToken::Type::EndTag);
RECONSUME_IN(RCDATAEndTagName);
}
ANYTHING_ELSE
{
LibWeb: Parse "textarea" tags during the "in body" insertion mode Had to handle some more cases in the tokenizer to support this.
2020-05-30 18:40:23 +02:00
m_queued_tokens.enqueue(HTMLToken::make_character('<'));
m_queued_tokens.enqueue(HTMLToken::make_character('/'));
RECONSUME_IN(RCDATA);
LibWeb: More work on the HTML parser and tokenizer The parser can now switch the state of the tokenizer! Very webby. :^)
2020-05-24 20:24:43 +02:00
}
}
END_STATE
LibWeb: Add spec links to each HTML tokenizer state section I didn't add full spec comments this time, but this is better than nothing :^)
2022-02-15 18:12:15 +00:00
// 13.2.5.11 RCDATA end tag name state, https://html.spec.whatwg.org/multipage/parsing.html#rcdata-end-tag-name-state
LibWeb: More work on the HTML parser and tokenizer The parser can now switch the state of the tokenizer! Very webby. :^)
2020-05-24 20:24:43 +02:00
BEGIN_STATE(RCDATAEndTagName)
{
ON_WHITESPACE
{
LibWeb: User getter+setter for HTMLToken tag name and self-closing flag
2021-07-14 23:37:48 +02:00
m_current_token.set_tag_name(consume_current_builder());
LibWeb: Implement more of the tokenizer states Slowly adding more unimplemented options for tokenizer states.
2020-05-28 00:28:32 +02:00
if (!current_end_tag_token_is_appropriate()) {
LibWeb: Parse "textarea" tags during the "in body" insertion mode Had to handle some more cases in the tokenizer to support this.
2020-05-30 18:40:23 +02:00
m_queued_tokens.enqueue(HTMLToken::make_character('<'));
m_queued_tokens.enqueue(HTMLToken::make_character('/'));
Unicode: Try s/codepoint/code_point/g again This time, without trailing 's'. Ran: git grep -l 'codepoint' | xargs sed -ie 's/codepoint/code_point/g
2020-08-05 16:31:20 -04:00
for (auto code_point : m_temporary_buffer)
m_queued_tokens.enqueue(HTMLToken::make_character(code_point));
LibWeb: Parse "textarea" tags during the "in body" insertion mode Had to handle some more cases in the tokenizer to support this.
2020-05-30 18:40:23 +02:00
RECONSUME_IN(RCDATA);
LibWeb: Implement more of the tokenizer states Slowly adding more unimplemented options for tokenizer states.
2020-05-28 00:28:32 +02:00
}
SWITCH_TO(BeforeAttributeName);
LibWeb: More work on the HTML parser and tokenizer The parser can now switch the state of the tokenizer! Very webby. :^)
2020-05-24 20:24:43 +02:00
}
ON('/')
{
LibWeb: User getter+setter for HTMLToken tag name and self-closing flag
2021-07-14 23:37:48 +02:00
m_current_token.set_tag_name(consume_current_builder());
LibWeb: Implement more of the tokenizer states Slowly adding more unimplemented options for tokenizer states.
2020-05-28 00:28:32 +02:00
if (!current_end_tag_token_is_appropriate()) {
LibWeb: Parse "textarea" tags during the "in body" insertion mode Had to handle some more cases in the tokenizer to support this.
2020-05-30 18:40:23 +02:00
m_queued_tokens.enqueue(HTMLToken::make_character('<'));
m_queued_tokens.enqueue(HTMLToken::make_character('/'));
Unicode: Try s/codepoint/code_point/g again This time, without trailing 's'. Ran: git grep -l 'codepoint' | xargs sed -ie 's/codepoint/code_point/g
2020-08-05 16:31:20 -04:00
for (auto code_point : m_temporary_buffer)
m_queued_tokens.enqueue(HTMLToken::make_character(code_point));
LibWeb: Parse "textarea" tags during the "in body" insertion mode Had to handle some more cases in the tokenizer to support this.
2020-05-30 18:40:23 +02:00
RECONSUME_IN(RCDATA);
LibWeb: Implement more of the tokenizer states Slowly adding more unimplemented options for tokenizer states.
2020-05-28 00:28:32 +02:00
}
SWITCH_TO(SelfClosingStartTag);
LibWeb: More work on the HTML parser and tokenizer The parser can now switch the state of the tokenizer! Very webby. :^)
2020-05-24 20:24:43 +02:00
}
ON('>')
{
LibWeb: User getter+setter for HTMLToken tag name and self-closing flag
2021-07-14 23:37:48 +02:00
m_current_token.set_tag_name(consume_current_builder());
LibWeb: More work on the HTML parser and tokenizer The parser can now switch the state of the tokenizer! Very webby. :^)
2020-05-24 20:24:43 +02:00
if (!current_end_tag_token_is_appropriate()) {
LibWeb: Parse "textarea" tags during the "in body" insertion mode Had to handle some more cases in the tokenizer to support this.
2020-05-30 18:40:23 +02:00
m_queued_tokens.enqueue(HTMLToken::make_character('<'));
m_queued_tokens.enqueue(HTMLToken::make_character('/'));
Unicode: Try s/codepoint/code_point/g again This time, without trailing 's'. Ran: git grep -l 'codepoint' | xargs sed -ie 's/codepoint/code_point/g
2020-08-05 16:31:20 -04:00
for (auto code_point : m_temporary_buffer)
m_queued_tokens.enqueue(HTMLToken::make_character(code_point));
LibWeb: Parse "textarea" tags during the "in body" insertion mode Had to handle some more cases in the tokenizer to support this.
2020-05-30 18:40:23 +02:00
RECONSUME_IN(RCDATA);
LibWeb: More work on the HTML parser and tokenizer The parser can now switch the state of the tokenizer! Very webby. :^)
2020-05-24 20:24:43 +02:00
}
SWITCH_TO_AND_EMIT_CURRENT_TOKEN(Data);
}
ON_ASCII_UPPER_ALPHA
{
LibWeb: Remove StringBuilder from HTMLToken::m_tag
2021-05-23 09:16:07 +02:00
m_current_builder.append_code_point(to_ascii_lowercase(current_input_character.value()));
LibWeb: More work on the HTML parser and tokenizer The parser can now switch the state of the tokenizer! Very webby. :^)
2020-05-24 20:24:43 +02:00
m_temporary_buffer.append(current_input_character.value());
continue;
}
ON_ASCII_LOWER_ALPHA
{
LibWeb: Remove StringBuilder from HTMLToken::m_tag
2021-05-23 09:16:07 +02:00
m_current_builder.append_code_point(current_input_character.value());
LibWeb: More work on the HTML parser and tokenizer The parser can now switch the state of the tokenizer! Very webby. :^)
2020-05-24 20:24:43 +02:00
m_temporary_buffer.append(current_input_character.value());
LibWeb: Parse enough to handle a <style> inside a <head> :^)
2020-05-24 20:36:43 +02:00
continue;
}
ANYTHING_ELSE
{
LibWeb: Parse "textarea" tags during the "in body" insertion mode Had to handle some more cases in the tokenizer to support this.
2020-05-30 18:40:23 +02:00
m_queued_tokens.enqueue(HTMLToken::make_character('<'));
m_queued_tokens.enqueue(HTMLToken::make_character('/'));
LibWeb: Fix assertion failures in HTMLTokenizer The *TagName states are all very similar, so it seems to be correct to apply the fix from #8761 to all of those states. This fixes #8788.
2021-07-16 00:36:10 +02:00
// NOTE: The spec doesn't mention this, but it seems that m_current_token (an end tag) is just dropped in this case.
m_current_builder.clear();
Unicode: Try s/codepoint/code_point/g again This time, without trailing 's'. Ran: git grep -l 'codepoint' | xargs sed -ie 's/codepoint/code_point/g
2020-08-05 16:31:20 -04:00
for (auto code_point : m_temporary_buffer)
m_queued_tokens.enqueue(HTMLToken::make_character(code_point));
LibWeb: Parse "textarea" tags during the "in body" insertion mode Had to handle some more cases in the tokenizer to support this.
2020-05-30 18:40:23 +02:00
RECONSUME_IN(RCDATA);
LibWeb: Parse enough to handle a <style> inside a <head> :^)
2020-05-24 20:36:43 +02:00
}
}
END_STATE
LibWeb: Add spec links to each HTML tokenizer state section I didn't add full spec comments this time, but this is better than nothing :^)
2022-02-15 18:12:15 +00:00
// 13.2.5.3 RAWTEXT state, https://html.spec.whatwg.org/multipage/parsing.html#rawtext-state
LibWeb: Parse enough to handle a <style> inside a <head> :^)
2020-05-24 20:36:43 +02:00
BEGIN_STATE(RAWTEXT)
{
ON('<')
{
SWITCH_TO(RAWTEXTLessThanSign);
}
ON(0)
{
LibWeb: Utilize SourceLocation for HTMLTokenizer logging
2021-04-24 20:22:30 -07:00
log_parse_error();
LibWeb: Make the new HTML parser parse input as UTF-8 We already convert the input to UTF-8 before starting the tokenizer, so all this patch had to do was switch the tokenizer to use an Utf8View for its input (and to emit 32-bit codepoints.)
2020-06-04 21:06:54 +02:00
EMIT_CHARACTER(0xFFFD);
LibWeb: Parse enough to handle a <style> inside a <head> :^)
2020-05-24 20:36:43 +02:00
}
ON_EOF
{
EMIT_EOF;
}
ANYTHING_ELSE
{
EMIT_CURRENT_CHARACTER;
}
}
END_STATE
LibWeb: Add spec links to each HTML tokenizer state section I didn't add full spec comments this time, but this is better than nothing :^)
2022-02-15 18:12:15 +00:00
// 13.2.5.12 RAWTEXT less-than sign state, https://html.spec.whatwg.org/multipage/parsing.html#rawtext-less-than-sign-state
LibWeb: Parse enough to handle a <style> inside a <head> :^)
2020-05-24 20:36:43 +02:00
BEGIN_STATE(RAWTEXTLessThanSign)
{
ON('/')
{
m_temporary_buffer.clear();
SWITCH_TO(RAWTEXTEndTagOpen);
}
ANYTHING_ELSE
{
LibWeb: Fully implement all script tokenizer states Also fixes RAWTEXTLessThanSign having a separate emit and reconsume.
2020-06-06 07:06:46 +01:00
EMIT_CHARACTER_AND_RECONSUME_IN('<', RAWTEXT);
LibWeb: Parse enough to handle a <style> inside a <head> :^)
2020-05-24 20:36:43 +02:00
}
}
END_STATE
LibWeb: Add spec links to each HTML tokenizer state section I didn't add full spec comments this time, but this is better than nothing :^)
2022-02-15 18:12:15 +00:00
// 13.2.5.13 RAWTEXT end tag open state, https://html.spec.whatwg.org/multipage/parsing.html#rawtext-end-tag-open-state
LibWeb: Parse enough to handle a <style> inside a <head> :^)
2020-05-24 20:36:43 +02:00
BEGIN_STATE(RAWTEXTEndTagOpen)
{
ON_ASCII_ALPHA
{
create_new_token(HTMLToken::Type::EndTag);
RECONSUME_IN(RAWTEXTEndTagName);
}
ANYTHING_ELSE
{
LibWeb: Implement some more RAWTEXT stuff in the tokenizer
2020-05-30 16:15:16 +02:00
m_queued_tokens.enqueue(HTMLToken::make_character('<'));
m_queued_tokens.enqueue(HTMLToken::make_character('/'));
RECONSUME_IN(RAWTEXT);
LibWeb: Parse enough to handle a <style> inside a <head> :^)
2020-05-24 20:36:43 +02:00
}
}
END_STATE
LibWeb: Add spec links to each HTML tokenizer state section I didn't add full spec comments this time, but this is better than nothing :^)
2022-02-15 18:12:15 +00:00
// 13.2.5.14 RAWTEXT end tag name state, https://html.spec.whatwg.org/multipage/parsing.html#rawtext-end-tag-name-state
LibWeb: Parse enough to handle a <style> inside a <head> :^)
2020-05-24 20:36:43 +02:00
BEGIN_STATE(RAWTEXTEndTagName)
{
ON_WHITESPACE
{
LibWeb: User getter+setter for HTMLToken tag name and self-closing flag
2021-07-14 23:37:48 +02:00
m_current_token.set_tag_name(consume_current_builder());
LibWeb: Implement more of the tokenizer states Slowly adding more unimplemented options for tokenizer states.
2020-05-28 00:28:32 +02:00
if (!current_end_tag_token_is_appropriate()) {
LibWeb: Implement some more RAWTEXT stuff in the tokenizer
2020-05-30 16:15:16 +02:00
m_queued_tokens.enqueue(HTMLToken::make_character('<'));
m_queued_tokens.enqueue(HTMLToken::make_character('/'));
Unicode: Try s/codepoint/code_point/g again This time, without trailing 's'. Ran: git grep -l 'codepoint' | xargs sed -ie 's/codepoint/code_point/g
2020-08-05 16:31:20 -04:00
for (auto code_point : m_temporary_buffer)
m_queued_tokens.enqueue(HTMLToken::make_character(code_point));
LibWeb: Implement some more RAWTEXT stuff in the tokenizer
2020-05-30 16:15:16 +02:00
RECONSUME_IN(RAWTEXT);
LibWeb: Implement more of the tokenizer states Slowly adding more unimplemented options for tokenizer states.
2020-05-28 00:28:32 +02:00
}
SWITCH_TO(BeforeAttributeName);
LibWeb: Parse enough to handle a <style> inside a <head> :^)
2020-05-24 20:36:43 +02:00
}
ON('/')
{
LibWeb: User getter+setter for HTMLToken tag name and self-closing flag
2021-07-14 23:37:48 +02:00
m_current_token.set_tag_name(consume_current_builder());
LibWeb: Implement more of the tokenizer states Slowly adding more unimplemented options for tokenizer states.
2020-05-28 00:28:32 +02:00
if (!current_end_tag_token_is_appropriate()) {
LibWeb: Implement some more RAWTEXT stuff in the tokenizer
2020-05-30 16:15:16 +02:00
m_queued_tokens.enqueue(HTMLToken::make_character('<'));
m_queued_tokens.enqueue(HTMLToken::make_character('/'));
Unicode: Try s/codepoint/code_point/g again This time, without trailing 's'. Ran: git grep -l 'codepoint' | xargs sed -ie 's/codepoint/code_point/g
2020-08-05 16:31:20 -04:00
for (auto code_point : m_temporary_buffer)
m_queued_tokens.enqueue(HTMLToken::make_character(code_point));
LibWeb: Implement some more RAWTEXT stuff in the tokenizer
2020-05-30 16:15:16 +02:00
RECONSUME_IN(RAWTEXT);
LibWeb: Implement more of the tokenizer states Slowly adding more unimplemented options for tokenizer states.
2020-05-28 00:28:32 +02:00
}
SWITCH_TO(SelfClosingStartTag);
LibWeb: Parse enough to handle a <style> inside a <head> :^)
2020-05-24 20:36:43 +02:00
}
ON('>')
{
LibWeb: User getter+setter for HTMLToken tag name and self-closing flag
2021-07-14 23:37:48 +02:00
m_current_token.set_tag_name(consume_current_builder());
LibWeb: Parse enough to handle a <style> inside a <head> :^)
2020-05-24 20:36:43 +02:00
if (!current_end_tag_token_is_appropriate()) {
LibWeb: Implement some more RAWTEXT stuff in the tokenizer
2020-05-30 16:15:16 +02:00
m_queued_tokens.enqueue(HTMLToken::make_character('<'));
m_queued_tokens.enqueue(HTMLToken::make_character('/'));
Unicode: Try s/codepoint/code_point/g again This time, without trailing 's'. Ran: git grep -l 'codepoint' | xargs sed -ie 's/codepoint/code_point/g
2020-08-05 16:31:20 -04:00
for (auto code_point : m_temporary_buffer)
m_queued_tokens.enqueue(HTMLToken::make_character(code_point));
LibWeb: Implement some more RAWTEXT stuff in the tokenizer
2020-05-30 16:15:16 +02:00
RECONSUME_IN(RAWTEXT);
LibWeb: Parse enough to handle a <style> inside a <head> :^)
2020-05-24 20:36:43 +02:00
}
SWITCH_TO_AND_EMIT_CURRENT_TOKEN(Data);
}
ON_ASCII_UPPER_ALPHA
{
LibWeb: Remove StringBuilder from HTMLToken::m_tag
2021-05-23 09:16:07 +02:00
m_current_builder.append_code_point(to_ascii_lowercase(current_input_character.value()));
LibWeb: Parse enough to handle a <style> inside a <head> :^)
2020-05-24 20:36:43 +02:00
m_temporary_buffer.append(current_input_character.value());
continue;
}
ON_ASCII_LOWER_ALPHA
{
LibWeb: Remove StringBuilder from HTMLToken::m_tag
2021-05-23 09:16:07 +02:00
m_current_builder.append(current_input_character.value());
LibWeb: Parse enough to handle a <style> inside a <head> :^)
2020-05-24 20:36:43 +02:00
m_temporary_buffer.append(current_input_character.value());
LibWeb: More work on the HTML parser and tokenizer The parser can now switch the state of the tokenizer! Very webby. :^)
2020-05-24 20:24:43 +02:00
continue;
}
ANYTHING_ELSE
{
LibWeb: Implement some more RAWTEXT stuff in the tokenizer
2020-05-30 16:15:16 +02:00
m_queued_tokens.enqueue(HTMLToken::make_character('<'));
m_queued_tokens.enqueue(HTMLToken::make_character('/'));
LibWeb: Fix assertion failures in HTMLTokenizer The *TagName states are all very similar, so it seems to be correct to apply the fix from #8761 to all of those states. This fixes #8788.
2021-07-16 00:36:10 +02:00
// NOTE: The spec doesn't mention this, but it seems that m_current_token (an end tag) is just dropped in this case.
m_current_builder.clear();
Unicode: Try s/codepoint/code_point/g again This time, without trailing 's'. Ran: git grep -l 'codepoint' | xargs sed -ie 's/codepoint/code_point/g
2020-08-05 16:31:20 -04:00
for (auto code_point : m_temporary_buffer)
m_queued_tokens.enqueue(HTMLToken::make_character(code_point));
LibWeb: Implement some more RAWTEXT stuff in the tokenizer
2020-05-30 16:15:16 +02:00
RECONSUME_IN(RAWTEXT);
LibWeb: More work on the HTML parser and tokenizer The parser can now switch the state of the tokenizer! Very webby. :^)
2020-05-24 20:24:43 +02:00
}
}
END_STATE
LibWeb: Add spec links to each HTML tokenizer state section I didn't add full spec comments this time, but this is better than nothing :^)
2022-02-15 18:12:15 +00:00
// 13.2.5.4 Script data state, https://html.spec.whatwg.org/multipage/parsing.html#script-data-state
LibWeb: A whole bunch of work towards spec-compliant <script> elements This is still very unfinished, but there's at least a skeleton of code.
2020-05-24 22:00:46 +02:00
BEGIN_STATE(ScriptData)
{
ON('<')
{
SWITCH_TO(ScriptDataLessThanSign);
}
ON(0)
{
LibWeb: Utilize SourceLocation for HTMLTokenizer logging
2021-04-24 20:22:30 -07:00
log_parse_error();
LibWeb: Make the new HTML parser parse input as UTF-8 We already convert the input to UTF-8 before starting the tokenizer, so all this patch had to do was switch the tokenizer to use an Utf8View for its input (and to emit 32-bit codepoints.)
2020-06-04 21:06:54 +02:00
EMIT_CHARACTER(0xFFFD);
LibWeb: Implement more of the tokenizer states Slowly adding more unimplemented options for tokenizer states.
2020-05-28 00:28:32 +02:00
}
ON_EOF
{
EMIT_EOF;
}
ANYTHING_ELSE
{
EMIT_CURRENT_CHARACTER;
}
}
END_STATE
LibWeb: Add spec links to each HTML tokenizer state section I didn't add full spec comments this time, but this is better than nothing :^)
2022-02-15 18:12:15 +00:00
// 13.2.5.5 PLAINTEXT state, https://html.spec.whatwg.org/multipage/parsing.html#plaintext-state
LibWeb: Implement more of the tokenizer states Slowly adding more unimplemented options for tokenizer states.
2020-05-28 00:28:32 +02:00
BEGIN_STATE(PLAINTEXT)
{
ON(0)
{
LibWeb: Utilize SourceLocation for HTMLTokenizer logging
2021-04-24 20:22:30 -07:00
log_parse_error();
LibWeb: Make the new HTML parser parse input as UTF-8 We already convert the input to UTF-8 before starting the tokenizer, so all this patch had to do was switch the tokenizer to use an Utf8View for its input (and to emit 32-bit codepoints.)
2020-06-04 21:06:54 +02:00
EMIT_CHARACTER(0xFFFD);
LibWeb: A whole bunch of work towards spec-compliant <script> elements This is still very unfinished, but there's at least a skeleton of code.
2020-05-24 22:00:46 +02:00
}
ON_EOF
{
EMIT_EOF;
}
ANYTHING_ELSE
{
EMIT_CURRENT_CHARACTER;
}
}
END_STATE
LibWeb: Add spec links to each HTML tokenizer state section I didn't add full spec comments this time, but this is better than nothing :^)
2022-02-15 18:12:15 +00:00
// 13.2.5.15 Script data less-than sign state, https://html.spec.whatwg.org/multipage/parsing.html#script-data-less-than-sign-state
LibWeb: A whole bunch of work towards spec-compliant <script> elements This is still very unfinished, but there's at least a skeleton of code.
2020-05-24 22:00:46 +02:00
BEGIN_STATE(ScriptDataLessThanSign)
{
ON('/')
{
m_temporary_buffer.clear();
SWITCH_TO(ScriptDataEndTagOpen);
}
ON('!')
{
LibWeb: Implement a bunch more script-related tokenization states
2020-05-28 18:44:17 +02:00
m_queued_tokens.enqueue(HTMLToken::make_character('<'));
m_queued_tokens.enqueue(HTMLToken::make_character('!'));
SWITCH_TO(ScriptDataEscapeStart);
LibWeb: A whole bunch of work towards spec-compliant <script> elements This is still very unfinished, but there's at least a skeleton of code.
2020-05-24 22:00:46 +02:00
}
ANYTHING_ELSE
{
LibWeb: Allow HTML tokenizer to emit more than one token Tokens are now put on a queue when emitted, and we always pop from that queue when returning from next_token().
2020-05-26 15:50:05 +02:00
EMIT_CHARACTER_AND_RECONSUME_IN('<', ScriptData);
LibWeb: A whole bunch of work towards spec-compliant <script> elements This is still very unfinished, but there's at least a skeleton of code.
2020-05-24 22:00:46 +02:00
}
}
END_STATE
LibWeb: Add spec links to each HTML tokenizer state section I didn't add full spec comments this time, but this is better than nothing :^)
2022-02-15 18:12:15 +00:00
// 13.2.5.18 Script data escape start state, https://html.spec.whatwg.org/multipage/parsing.html#script-data-escape-start-state
LibWeb: Implement a bunch more script-related tokenization states
2020-05-28 18:44:17 +02:00
BEGIN_STATE(ScriptDataEscapeStart)
{
ON('-')
{
LibWeb: Fully implement all script tokenizer states Also fixes RAWTEXTLessThanSign having a separate emit and reconsume.
2020-06-06 07:06:46 +01:00
SWITCH_TO_AND_EMIT_CHARACTER('-', ScriptDataEscapeStartDash);
LibWeb: Implement a bunch more script-related tokenization states
2020-05-28 18:44:17 +02:00
}
ANYTHING_ELSE
{
RECONSUME_IN(ScriptData);
}
}
END_STATE
LibWeb: Add spec links to each HTML tokenizer state section I didn't add full spec comments this time, but this is better than nothing :^)
2022-02-15 18:12:15 +00:00
// 13.2.5.19 Script data escape start dash state, https://html.spec.whatwg.org/multipage/parsing.html#script-data-escape-start-dash-state
LibWeb: Implement a bunch more script-related tokenization states
2020-05-28 18:44:17 +02:00
BEGIN_STATE(ScriptDataEscapeStartDash)
{
ON('-')
{
LibWeb: Fully implement all script tokenizer states Also fixes RAWTEXTLessThanSign having a separate emit and reconsume.
2020-06-06 07:06:46 +01:00
SWITCH_TO_AND_EMIT_CHARACTER('-', ScriptDataEscapedDashDash);
LibWeb: Implement a bunch more script-related tokenization states
2020-05-28 18:44:17 +02:00
}
ANYTHING_ELSE
{
RECONSUME_IN(ScriptData);
}
}
END_STATE
LibWeb: Add spec links to each HTML tokenizer state section I didn't add full spec comments this time, but this is better than nothing :^)
2022-02-15 18:12:15 +00:00
// 13.2.5.22 Script data escaped dash dash state, https://html.spec.whatwg.org/multipage/parsing.html#script-data-escaped-dash-dash-state
LibWeb: Implement a bunch more script-related tokenization states
2020-05-28 18:44:17 +02:00
BEGIN_STATE(ScriptDataEscapedDashDash)
{
ON('-')
{
EMIT_CHARACTER('-');
}
ON('<')
{
SWITCH_TO(ScriptDataEscapedLessThanSign);
}
ON('>')
{
LibWeb: Fully implement all script tokenizer states Also fixes RAWTEXTLessThanSign having a separate emit and reconsume.
2020-06-06 07:06:46 +01:00
SWITCH_TO_AND_EMIT_CHARACTER('>', ScriptData);
LibWeb: Implement a bunch more script-related tokenization states
2020-05-28 18:44:17 +02:00
}
ON(0)
{
LibWeb: Utilize SourceLocation for HTMLTokenizer logging
2021-04-24 20:22:30 -07:00
log_parse_error();
LibWeb: Fully implement all script tokenizer states Also fixes RAWTEXTLessThanSign having a separate emit and reconsume.
2020-06-06 07:06:46 +01:00
SWITCH_TO_AND_EMIT_CHARACTER(0xFFFD, ScriptDataEscaped);
LibWeb: Implement a bunch more script-related tokenization states
2020-05-28 18:44:17 +02:00
}
ON_EOF
{
LibWeb: Utilize SourceLocation for HTMLTokenizer logging
2021-04-24 20:22:30 -07:00
log_parse_error();
LibWeb: Fully implement all script tokenizer states Also fixes RAWTEXTLessThanSign having a separate emit and reconsume.
2020-06-06 07:06:46 +01:00
EMIT_EOF;
LibWeb: Implement a bunch more script-related tokenization states
2020-05-28 18:44:17 +02:00
}
ANYTHING_ELSE
{
LibWeb: Fully implement all script tokenizer states Also fixes RAWTEXTLessThanSign having a separate emit and reconsume.
2020-06-06 07:06:46 +01:00
SWITCH_TO_AND_EMIT_CURRENT_CHARACTER(ScriptDataEscaped);
LibWeb: Implement a bunch more script-related tokenization states
2020-05-28 18:44:17 +02:00
}
}
END_STATE
LibWeb: Add spec links to each HTML tokenizer state section I didn't add full spec comments this time, but this is better than nothing :^)
2022-02-15 18:12:15 +00:00
// 13.2.5.23 Script data escaped less-than sign state, https://html.spec.whatwg.org/multipage/parsing.html#script-data-escaped-less-than-sign-state
LibWeb: Implement a bunch more script-related tokenization states
2020-05-28 18:44:17 +02:00
BEGIN_STATE(ScriptDataEscapedLessThanSign)
{
ON('/')
{
m_temporary_buffer.clear();
SWITCH_TO(ScriptDataEscapedEndTagOpen);
}
ON_ASCII_ALPHA
{
m_temporary_buffer.clear();
EMIT_CHARACTER_AND_RECONSUME_IN('<', ScriptDataDoubleEscapeStart);
}
ANYTHING_ELSE
{
EMIT_CHARACTER_AND_RECONSUME_IN('<', ScriptDataEscaped);
}
}
END_STATE
LibWeb: Add spec links to each HTML tokenizer state section I didn't add full spec comments this time, but this is better than nothing :^)
2022-02-15 18:12:15 +00:00
// 13.2.5.24 Script data escaped end tag open state, https://html.spec.whatwg.org/multipage/parsing.html#script-data-escaped-end-tag-open-state
LibWeb: Implement a bunch more script-related tokenization states
2020-05-28 18:44:17 +02:00
BEGIN_STATE(ScriptDataEscapedEndTagOpen)
{
ON_ASCII_ALPHA
{
create_new_token(HTMLToken::Type::EndTag);
RECONSUME_IN(ScriptDataEscapedEndTagName);
}
ANYTHING_ELSE
{
m_queued_tokens.enqueue(HTMLToken::make_character('<'));
m_queued_tokens.enqueue(HTMLToken::make_character('/'));
RECONSUME_IN(ScriptDataEscaped);
}
}
END_STATE
LibWeb: Add spec links to each HTML tokenizer state section I didn't add full spec comments this time, but this is better than nothing :^)
2022-02-15 18:12:15 +00:00
// 13.2.5.25 Script data escaped end tag name state, https://html.spec.whatwg.org/multipage/parsing.html#script-data-escaped-end-tag-name-state
LibWeb: Implement a bunch more script-related tokenization states
2020-05-28 18:44:17 +02:00
BEGIN_STATE(ScriptDataEscapedEndTagName)
{
ON_WHITESPACE
{
LibWeb: User getter+setter for HTMLToken tag name and self-closing flag
2021-07-14 23:37:48 +02:00
m_current_token.set_tag_name(consume_current_builder());
LibWeb: Fully implement all script tokenizer states Also fixes RAWTEXTLessThanSign having a separate emit and reconsume.
2020-06-06 07:06:46 +01:00
if (current_end_tag_token_is_appropriate())
LibWeb: Implement a bunch more script-related tokenization states
2020-05-28 18:44:17 +02:00
SWITCH_TO(BeforeAttributeName);
LibWeb: Fully implement all script tokenizer states Also fixes RAWTEXTLessThanSign having a separate emit and reconsume.
2020-06-06 07:06:46 +01:00
m_queued_tokens.enqueue(HTMLToken::make_character('<'));
m_queued_tokens.enqueue(HTMLToken::make_character('/'));
LibWeb: Fix assertion failures in HTMLTokenizer The *TagName states are all very similar, so it seems to be correct to apply the fix from #8761 to all of those states. This fixes #8788.
2021-07-16 00:36:10 +02:00
// NOTE: The spec doesn't mention this, but it seems that m_current_token (an end tag) is just dropped in this case.
m_current_builder.clear();
Unicode: Try s/codepoint/code_point/g again This time, without trailing 's'. Ran: git grep -l 'codepoint' | xargs sed -ie 's/codepoint/code_point/g
2020-08-05 16:31:20 -04:00
for (auto code_point : m_temporary_buffer) {
m_queued_tokens.enqueue(HTMLToken::make_character(code_point));
LibWeb: Implement a bunch more script-related tokenization states
2020-05-28 18:44:17 +02:00
}
LibWeb: Fully implement all script tokenizer states Also fixes RAWTEXTLessThanSign having a separate emit and reconsume.
2020-06-06 07:06:46 +01:00
RECONSUME_IN(ScriptDataEscaped);
LibWeb: Implement a bunch more script-related tokenization states
2020-05-28 18:44:17 +02:00
}
ON('/')
{
LibWeb: User getter+setter for HTMLToken tag name and self-closing flag
2021-07-14 23:37:48 +02:00
m_current_token.set_tag_name(consume_current_builder());
LibWeb: Fully implement all script tokenizer states Also fixes RAWTEXTLessThanSign having a separate emit and reconsume.
2020-06-06 07:06:46 +01:00
if (current_end_tag_token_is_appropriate())
LibWeb: Implement a bunch more script-related tokenization states
2020-05-28 18:44:17 +02:00
SWITCH_TO(SelfClosingStartTag);
LibWeb: Fully implement all script tokenizer states Also fixes RAWTEXTLessThanSign having a separate emit and reconsume.
2020-06-06 07:06:46 +01:00
m_queued_tokens.enqueue(HTMLToken::make_character('<'));
m_queued_tokens.enqueue(HTMLToken::make_character('/'));
LibWeb: Fix assertion failures in HTMLTokenizer The *TagName states are all very similar, so it seems to be correct to apply the fix from #8761 to all of those states. This fixes #8788.
2021-07-16 00:36:10 +02:00
// NOTE: The spec doesn't mention this, but it seems that m_current_token (an end tag) is just dropped in this case.
m_current_builder.clear();
Unicode: Try s/codepoint/code_point/g again This time, without trailing 's'. Ran: git grep -l 'codepoint' | xargs sed -ie 's/codepoint/code_point/g
2020-08-05 16:31:20 -04:00
for (auto code_point : m_temporary_buffer) {
m_queued_tokens.enqueue(HTMLToken::make_character(code_point));
LibWeb: Implement a bunch more script-related tokenization states
2020-05-28 18:44:17 +02:00
}
LibWeb: Fully implement all script tokenizer states Also fixes RAWTEXTLessThanSign having a separate emit and reconsume.
2020-06-06 07:06:46 +01:00
RECONSUME_IN(ScriptDataEscaped);
LibWeb: Implement a bunch more script-related tokenization states
2020-05-28 18:44:17 +02:00
}
ON('>')
{
LibWeb: User getter+setter for HTMLToken tag name and self-closing flag
2021-07-14 23:37:48 +02:00
m_current_token.set_tag_name(consume_current_builder());
LibWeb: Fully implement all script tokenizer states Also fixes RAWTEXTLessThanSign having a separate emit and reconsume.
2020-06-06 07:06:46 +01:00
if (current_end_tag_token_is_appropriate())
LibWeb: Implement a bunch more script-related tokenization states
2020-05-28 18:44:17 +02:00
SWITCH_TO_AND_EMIT_CURRENT_TOKEN(Data);
LibWeb: Fully implement all script tokenizer states Also fixes RAWTEXTLessThanSign having a separate emit and reconsume.
2020-06-06 07:06:46 +01:00
m_queued_tokens.enqueue(HTMLToken::make_character('<'));
m_queued_tokens.enqueue(HTMLToken::make_character('/'));
LibWeb: Fix assertion failures in HTMLTokenizer The *TagName states are all very similar, so it seems to be correct to apply the fix from #8761 to all of those states. This fixes #8788.
2021-07-16 00:36:10 +02:00
// NOTE: The spec doesn't mention this, but it seems that m_current_token (an end tag) is just dropped in this case.
m_current_builder.clear();
Unicode: Try s/codepoint/code_point/g again This time, without trailing 's'. Ran: git grep -l 'codepoint' | xargs sed -ie 's/codepoint/code_point/g
2020-08-05 16:31:20 -04:00
for (auto code_point : m_temporary_buffer) {
m_queued_tokens.enqueue(HTMLToken::make_character(code_point));
LibWeb: Implement a bunch more script-related tokenization states
2020-05-28 18:44:17 +02:00
}
LibWeb: Fully implement all script tokenizer states Also fixes RAWTEXTLessThanSign having a separate emit and reconsume.
2020-06-06 07:06:46 +01:00
RECONSUME_IN(ScriptDataEscaped);
LibWeb: Implement a bunch more script-related tokenization states
2020-05-28 18:44:17 +02:00
}
ON_ASCII_UPPER_ALPHA
{
LibWeb: Remove StringBuilder from HTMLToken::m_tag
2021-05-23 09:16:07 +02:00
m_current_builder.append_code_point(to_ascii_lowercase(current_input_character.value()));
LibWeb: Implement a bunch more script-related tokenization states
2020-05-28 18:44:17 +02:00
m_temporary_buffer.append(current_input_character.value());
continue;
}
ON_ASCII_LOWER_ALPHA
{
LibWeb: Remove StringBuilder from HTMLToken::m_tag
2021-05-23 09:16:07 +02:00
m_current_builder.append(current_input_character.value());
LibWeb: Implement a bunch more script-related tokenization states
2020-05-28 18:44:17 +02:00
m_temporary_buffer.append(current_input_character.value());
continue;
}
ANYTHING_ELSE
{
m_queued_tokens.enqueue(HTMLToken::make_character('<'));
m_queued_tokens.enqueue(HTMLToken::make_character('/'));
LibWeb: Fix assertion failures in HTMLTokenizer The *TagName states are all very similar, so it seems to be correct to apply the fix from #8761 to all of those states. This fixes #8788.
2021-07-16 00:36:10 +02:00
// NOTE: The spec doesn't mention this, but it seems that m_current_token (an end tag) is just dropped in this case.
m_current_builder.clear();
Unicode: Try s/codepoint/code_point/g again This time, without trailing 's'. Ran: git grep -l 'codepoint' | xargs sed -ie 's/codepoint/code_point/g
2020-08-05 16:31:20 -04:00
for (auto code_point : m_temporary_buffer) {
m_queued_tokens.enqueue(HTMLToken::make_character(code_point));
LibWeb: Implement a bunch more script-related tokenization states
2020-05-28 18:44:17 +02:00
}
RECONSUME_IN(ScriptDataEscaped);
}
}
END_STATE
LibWeb: Add spec links to each HTML tokenizer state section I didn't add full spec comments this time, but this is better than nothing :^)
2022-02-15 18:12:15 +00:00
// 13.2.5.26 Script data double escape start state, https://html.spec.whatwg.org/multipage/parsing.html#script-data-double-escape-start-state
LibWeb: Implement a bunch more script-related tokenization states
2020-05-28 18:44:17 +02:00
BEGIN_STATE(ScriptDataDoubleEscapeStart)
{
LibWeb: Fully implement all script tokenizer states Also fixes RAWTEXTLessThanSign having a separate emit and reconsume.
2020-06-06 07:06:46 +01:00
auto temporary_buffer_equal_to_script = [this]() -> bool {
if (m_temporary_buffer.size() != 6)
return false;
// FIXME: Is there a better way of doing this?
LibWeb: Move the HTML parser into HTML/Parser/
2020-07-28 19:18:23 +02:00
return m_temporary_buffer[0] == 's' && m_temporary_buffer[1] == 'c' && m_temporary_buffer[2] == 'r' && m_temporary_buffer[3] == 'i' && m_temporary_buffer[4] == 'p' && m_temporary_buffer[5] == 't';
LibWeb: Fully implement all script tokenizer states Also fixes RAWTEXTLessThanSign having a separate emit and reconsume.
2020-06-06 07:06:46 +01:00
};
ON_WHITESPACE
{
if (temporary_buffer_equal_to_script())
SWITCH_TO_AND_EMIT_CURRENT_CHARACTER(ScriptDataDoubleEscaped);
else
SWITCH_TO_AND_EMIT_CURRENT_CHARACTER(ScriptDataEscaped);
}
ON('/')
{
if (temporary_buffer_equal_to_script())
SWITCH_TO_AND_EMIT_CURRENT_CHARACTER(ScriptDataDoubleEscaped);
else
SWITCH_TO_AND_EMIT_CURRENT_CHARACTER(ScriptDataEscaped);
}
ON('>')
{
if (temporary_buffer_equal_to_script())
SWITCH_TO_AND_EMIT_CURRENT_CHARACTER(ScriptDataDoubleEscaped);
else
SWITCH_TO_AND_EMIT_CURRENT_CHARACTER(ScriptDataEscaped);
}
ON_ASCII_UPPER_ALPHA
{
Everywhere: Replace ctype.h to avoid narrowing conversions This replaces ctype.h with CharacterType.h everywhere I could find issues with narrowing conversions. While using it will probably make sense almost everywhere in the future, the most critical places should have been addressed.
2021-06-01 21:18:08 +02:00
m_temporary_buffer.append(to_ascii_lowercase(current_input_character.value()));
LibWeb: Fully implement all script tokenizer states Also fixes RAWTEXTLessThanSign having a separate emit and reconsume.
2020-06-06 07:06:46 +01:00
EMIT_CURRENT_CHARACTER;
}
ON_ASCII_LOWER_ALPHA
{
m_temporary_buffer.append(current_input_character.value());
EMIT_CURRENT_CHARACTER;
}
ANYTHING_ELSE
{
RECONSUME_IN(ScriptDataEscaped);
}
}
END_STATE
LibWeb: Add spec links to each HTML tokenizer state section I didn't add full spec comments this time, but this is better than nothing :^)
2022-02-15 18:12:15 +00:00
// 13.2.5.27 Script data double escaped state, https://html.spec.whatwg.org/multipage/parsing.html#script-data-double-escaped-state
LibWeb: Fully implement all script tokenizer states Also fixes RAWTEXTLessThanSign having a separate emit and reconsume.
2020-06-06 07:06:46 +01:00
BEGIN_STATE(ScriptDataDoubleEscaped)
{
ON('-')
{
SWITCH_TO_AND_EMIT_CHARACTER('-', ScriptDataDoubleEscapedDash);
}
ON('<')
{
SWITCH_TO_AND_EMIT_CHARACTER('<', ScriptDataDoubleEscapedLessThanSign);
}
ON(0)
{
LibWeb: Utilize SourceLocation for HTMLTokenizer logging
2021-04-24 20:22:30 -07:00
log_parse_error();
LibWeb: Fully implement all script tokenizer states Also fixes RAWTEXTLessThanSign having a separate emit and reconsume.
2020-06-06 07:06:46 +01:00
EMIT_CHARACTER(0xFFFD);
}
ON_EOF
{
LibWeb: Utilize SourceLocation for HTMLTokenizer logging
2021-04-24 20:22:30 -07:00
log_parse_error();
LibWeb: Fully implement all script tokenizer states Also fixes RAWTEXTLessThanSign having a separate emit and reconsume.
2020-06-06 07:06:46 +01:00
EMIT_EOF;
}
ANYTHING_ELSE
{
EMIT_CURRENT_CHARACTER;
}
}
END_STATE
LibWeb: Add spec links to each HTML tokenizer state section I didn't add full spec comments this time, but this is better than nothing :^)
2022-02-15 18:12:15 +00:00
// 13.2.5.28 Script data double escaped dash state, https://html.spec.whatwg.org/multipage/parsing.html#script-data-double-escaped-dash-state
LibWeb: Fully implement all script tokenizer states Also fixes RAWTEXTLessThanSign having a separate emit and reconsume.
2020-06-06 07:06:46 +01:00
BEGIN_STATE(ScriptDataDoubleEscapedDash)
{
ON('-')
{
SWITCH_TO_AND_EMIT_CHARACTER('-', ScriptDataDoubleEscapedDashDash);
}
ON('<')
{
SWITCH_TO_AND_EMIT_CHARACTER('<', ScriptDataDoubleEscapedLessThanSign);
}
ON(0)
{
LibWeb: Utilize SourceLocation for HTMLTokenizer logging
2021-04-24 20:22:30 -07:00
log_parse_error();
LibWeb: Fully implement all script tokenizer states Also fixes RAWTEXTLessThanSign having a separate emit and reconsume.
2020-06-06 07:06:46 +01:00
SWITCH_TO_AND_EMIT_CHARACTER(0xFFFD, ScriptDataDoubleEscaped);
}
ON_EOF
{
LibWeb: Utilize SourceLocation for HTMLTokenizer logging
2021-04-24 20:22:30 -07:00
log_parse_error();
LibWeb: Fully implement all script tokenizer states Also fixes RAWTEXTLessThanSign having a separate emit and reconsume.
2020-06-06 07:06:46 +01:00
EMIT_EOF;
}
ANYTHING_ELSE
{
SWITCH_TO_AND_EMIT_CURRENT_CHARACTER(ScriptDataDoubleEscaped);
}
}
END_STATE
LibWeb: Add spec links to each HTML tokenizer state section I didn't add full spec comments this time, but this is better than nothing :^)
2022-02-15 18:12:15 +00:00
// 13.2.5.29 Script data double escaped dash dash state, https://html.spec.whatwg.org/multipage/parsing.html#script-data-double-escaped-dash-dash-state
LibWeb: Fully implement all script tokenizer states Also fixes RAWTEXTLessThanSign having a separate emit and reconsume.
2020-06-06 07:06:46 +01:00
BEGIN_STATE(ScriptDataDoubleEscapedDashDash)
{
ON('-')
{
EMIT_CHARACTER('-');
}
ON('<')
{
SWITCH_TO_AND_EMIT_CHARACTER('<', ScriptDataDoubleEscapedLessThanSign);
}
ON('>')
{
SWITCH_TO_AND_EMIT_CHARACTER('>', ScriptData);
}
ON(0)
{
LibWeb: Utilize SourceLocation for HTMLTokenizer logging
2021-04-24 20:22:30 -07:00
log_parse_error();
LibWeb: Fully implement all script tokenizer states Also fixes RAWTEXTLessThanSign having a separate emit and reconsume.
2020-06-06 07:06:46 +01:00
SWITCH_TO_AND_EMIT_CHARACTER(0xFFFD, ScriptDataDoubleEscaped);
}
ON_EOF
{
LibWeb: Utilize SourceLocation for HTMLTokenizer logging
2021-04-24 20:22:30 -07:00
log_parse_error();
LibWeb: Fully implement all script tokenizer states Also fixes RAWTEXTLessThanSign having a separate emit and reconsume.
2020-06-06 07:06:46 +01:00
EMIT_EOF;
}
ANYTHING_ELSE
{
SWITCH_TO_AND_EMIT_CURRENT_CHARACTER(ScriptDataDoubleEscaped);
}
}
END_STATE
LibWeb: Add spec links to each HTML tokenizer state section I didn't add full spec comments this time, but this is better than nothing :^)
2022-02-15 18:12:15 +00:00
// 13.2.5.30 Script data double escaped less-than sign state, https://html.spec.whatwg.org/multipage/parsing.html#script-data-double-escaped-less-than-sign-state
LibWeb: Fully implement all script tokenizer states Also fixes RAWTEXTLessThanSign having a separate emit and reconsume.
2020-06-06 07:06:46 +01:00
BEGIN_STATE(ScriptDataDoubleEscapedLessThanSign)
{
ON('/')
{
m_temporary_buffer.clear();
SWITCH_TO_AND_EMIT_CHARACTER('/', ScriptDataDoubleEscapeEnd);
}
ANYTHING_ELSE
{
RECONSUME_IN(ScriptDataDoubleEscaped);
}
}
END_STATE
LibWeb: Add spec links to each HTML tokenizer state section I didn't add full spec comments this time, but this is better than nothing :^)
2022-02-15 18:12:15 +00:00
// 13.2.5.31 Script data double escape end state, https://html.spec.whatwg.org/multipage/parsing.html#script-data-double-escape-end-state
LibWeb: Fully implement all script tokenizer states Also fixes RAWTEXTLessThanSign having a separate emit and reconsume.
2020-06-06 07:06:46 +01:00
BEGIN_STATE(ScriptDataDoubleEscapeEnd)
{
auto temporary_buffer_equal_to_script = [this]() -> bool {
if (m_temporary_buffer.size() != 6)
return false;
// FIXME: Is there a better way of doing this?
LibWeb: Move the HTML parser into HTML/Parser/
2020-07-28 19:18:23 +02:00
return m_temporary_buffer[0] == 's' && m_temporary_buffer[1] == 'c' && m_temporary_buffer[2] == 'r' && m_temporary_buffer[3] == 'i' && m_temporary_buffer[4] == 'p' && m_temporary_buffer[5] == 't';
LibWeb: Fully implement all script tokenizer states Also fixes RAWTEXTLessThanSign having a separate emit and reconsume.
2020-06-06 07:06:46 +01:00
};
ON_WHITESPACE
{
if (temporary_buffer_equal_to_script())
SWITCH_TO_AND_EMIT_CURRENT_CHARACTER(ScriptDataEscaped);
else
SWITCH_TO_AND_EMIT_CURRENT_CHARACTER(ScriptDataDoubleEscaped);
}
ON('/')
{
if (temporary_buffer_equal_to_script())
SWITCH_TO_AND_EMIT_CURRENT_CHARACTER(ScriptDataEscaped);
else
SWITCH_TO_AND_EMIT_CURRENT_CHARACTER(ScriptDataDoubleEscaped);
}
ON('>')
{
if (temporary_buffer_equal_to_script())
SWITCH_TO_AND_EMIT_CURRENT_CHARACTER(ScriptDataEscaped);
else
SWITCH_TO_AND_EMIT_CURRENT_CHARACTER(ScriptDataDoubleEscaped);
}
ON_ASCII_UPPER_ALPHA
{
Everywhere: Replace ctype.h to avoid narrowing conversions This replaces ctype.h with CharacterType.h everywhere I could find issues with narrowing conversions. While using it will probably make sense almost everywhere in the future, the most critical places should have been addressed.
2021-06-01 21:18:08 +02:00
m_temporary_buffer.append(to_ascii_lowercase(current_input_character.value()));
LibWeb: Fully implement all script tokenizer states Also fixes RAWTEXTLessThanSign having a separate emit and reconsume.
2020-06-06 07:06:46 +01:00
EMIT_CURRENT_CHARACTER;
}
ON_ASCII_LOWER_ALPHA
{
m_temporary_buffer.append(current_input_character.value());
EMIT_CURRENT_CHARACTER;
}
ANYTHING_ELSE
{
RECONSUME_IN(ScriptDataDoubleEscaped);
}
LibWeb: Implement a bunch more script-related tokenization states
2020-05-28 18:44:17 +02:00
}
END_STATE
LibWeb: Add spec links to each HTML tokenizer state section I didn't add full spec comments this time, but this is better than nothing :^)
2022-02-15 18:12:15 +00:00
// 13.2.5.21 Script data escaped dash state, https://html.spec.whatwg.org/multipage/parsing.html#script-data-escaped-dash-state
LibWeb: Implement a bunch more script-related tokenization states
2020-05-28 18:44:17 +02:00
BEGIN_STATE(ScriptDataEscapedDash)
{
ON('-')
{
LibWeb: Fully implement all script tokenizer states Also fixes RAWTEXTLessThanSign having a separate emit and reconsume.
2020-06-06 07:06:46 +01:00
SWITCH_TO_AND_EMIT_CHARACTER('-', ScriptDataEscapedDashDash);
LibWeb: Implement a bunch more script-related tokenization states
2020-05-28 18:44:17 +02:00
}
ON('<')
{
SWITCH_TO(ScriptDataEscapedLessThanSign);
}
ON(0)
{
LibWeb: Utilize SourceLocation for HTMLTokenizer logging
2021-04-24 20:22:30 -07:00
log_parse_error();
LibWeb: Fully implement all script tokenizer states Also fixes RAWTEXTLessThanSign having a separate emit and reconsume.
2020-06-06 07:06:46 +01:00
SWITCH_TO_AND_EMIT_CHARACTER(0xFFFD, ScriptDataEscaped);
LibWeb: Implement a bunch more script-related tokenization states
2020-05-28 18:44:17 +02:00
}
ON_EOF
{
LibWeb: Utilize SourceLocation for HTMLTokenizer logging
2021-04-24 20:22:30 -07:00
log_parse_error();
LibWeb: Fully implement all script tokenizer states Also fixes RAWTEXTLessThanSign having a separate emit and reconsume.
2020-06-06 07:06:46 +01:00
EMIT_EOF;
LibWeb: Implement a bunch more script-related tokenization states
2020-05-28 18:44:17 +02:00
}
ANYTHING_ELSE
{
LibWeb: Fully implement all script tokenizer states Also fixes RAWTEXTLessThanSign having a separate emit and reconsume.
2020-06-06 07:06:46 +01:00
SWITCH_TO_AND_EMIT_CURRENT_CHARACTER(ScriptDataEscaped);
LibWeb: Implement a bunch more script-related tokenization states
2020-05-28 18:44:17 +02:00
}
}
END_STATE
LibWeb: Add spec links to each HTML tokenizer state section I didn't add full spec comments this time, but this is better than nothing :^)
2022-02-15 18:12:15 +00:00
// 13.2.5.20 Script data escaped state, https://html.spec.whatwg.org/multipage/parsing.html#script-data-escaped-state
LibWeb: Implement a bunch more script-related tokenization states
2020-05-28 18:44:17 +02:00
BEGIN_STATE(ScriptDataEscaped)
{
ON('-')
{
LibWeb: Fully implement all script tokenizer states Also fixes RAWTEXTLessThanSign having a separate emit and reconsume.
2020-06-06 07:06:46 +01:00
SWITCH_TO_AND_EMIT_CHARACTER('-', ScriptDataEscapedDash);
LibWeb: Implement a bunch more script-related tokenization states
2020-05-28 18:44:17 +02:00
}
ON('<')
{
SWITCH_TO(ScriptDataEscapedLessThanSign);
}
ON(0)
{
LibWeb: Utilize SourceLocation for HTMLTokenizer logging
2021-04-24 20:22:30 -07:00
log_parse_error();
LibWeb: Fully implement all script tokenizer states Also fixes RAWTEXTLessThanSign having a separate emit and reconsume.
2020-06-06 07:06:46 +01:00
EMIT_CHARACTER(0xFFFD);
LibWeb: Implement a bunch more script-related tokenization states
2020-05-28 18:44:17 +02:00
}
ON_EOF
{
LibWeb: Utilize SourceLocation for HTMLTokenizer logging
2021-04-24 20:22:30 -07:00
log_parse_error();
LibWeb: Fully implement all script tokenizer states Also fixes RAWTEXTLessThanSign having a separate emit and reconsume.
2020-06-06 07:06:46 +01:00
EMIT_EOF;
LibWeb: Implement a bunch more script-related tokenization states
2020-05-28 18:44:17 +02:00
}
ANYTHING_ELSE
{
EMIT_CURRENT_CHARACTER;
}
}
END_STATE
LibWeb: Add spec links to each HTML tokenizer state section I didn't add full spec comments this time, but this is better than nothing :^)
2022-02-15 18:12:15 +00:00
// 13.2.5.16 Script data end tag open state, https://html.spec.whatwg.org/multipage/parsing.html#script-data-end-tag-open-state
LibWeb: A whole bunch of work towards spec-compliant <script> elements This is still very unfinished, but there's at least a skeleton of code.
2020-05-24 22:00:46 +02:00
BEGIN_STATE(ScriptDataEndTagOpen)
{
ON_ASCII_ALPHA
{
create_new_token(HTMLToken::Type::EndTag);
RECONSUME_IN(ScriptDataEndTagName);
}
ANYTHING_ELSE
{
LibWeb: Fully implement all script tokenizer states Also fixes RAWTEXTLessThanSign having a separate emit and reconsume.
2020-06-06 07:06:46 +01:00
m_queued_tokens.enqueue(HTMLToken::make_character('<'));
m_queued_tokens.enqueue(HTMLToken::make_character('/'));
RECONSUME_IN(ScriptData);
LibWeb: A whole bunch of work towards spec-compliant <script> elements This is still very unfinished, but there's at least a skeleton of code.
2020-05-24 22:00:46 +02:00
}
}
END_STATE
LibWeb: Add spec links to each HTML tokenizer state section I didn't add full spec comments this time, but this is better than nothing :^)
2022-02-15 18:12:15 +00:00
// 13.2.5.17 Script data end tag name state, https://html.spec.whatwg.org/multipage/parsing.html#script-data-end-tag-name-state
LibWeb: A whole bunch of work towards spec-compliant <script> elements This is still very unfinished, but there's at least a skeleton of code.
2020-05-24 22:00:46 +02:00
BEGIN_STATE(ScriptDataEndTagName)
{
ON_WHITESPACE
{
LibWeb: User getter+setter for HTMLToken tag name and self-closing flag
2021-07-14 23:37:48 +02:00
m_current_token.set_tag_name(consume_current_builder());
LibWeb: A whole bunch of work towards spec-compliant <script> elements This is still very unfinished, but there's at least a skeleton of code.
2020-05-24 22:00:46 +02:00
if (current_end_tag_token_is_appropriate())
SWITCH_TO(BeforeAttributeName);
LibWeb: Implement more of the ScriptDataEndTagName tokenizer state Some of this is extremely repetitive. We'll need to rethink how we do queue/emit to improve this.
2020-05-30 22:59:41 +02:00
m_queued_tokens.enqueue(HTMLToken::make_character('<'));
m_queued_tokens.enqueue(HTMLToken::make_character('/'));
LibWeb: Fix assertion failures in HTMLTokenizer The *TagName states are all very similar, so it seems to be correct to apply the fix from #8761 to all of those states. This fixes #8788.
2021-07-16 00:36:10 +02:00
// NOTE: The spec doesn't mention this, but it seems that m_current_token (an end tag) is just dropped in this case.
m_current_builder.clear();
Unicode: Try s/codepoint/code_point/g again This time, without trailing 's'. Ran: git grep -l 'codepoint' | xargs sed -ie 's/codepoint/code_point/g
2020-08-05 16:31:20 -04:00
for (auto code_point : m_temporary_buffer)
m_queued_tokens.enqueue(HTMLToken::make_character(code_point));
LibWeb: Implement more of the ScriptDataEndTagName tokenizer state Some of this is extremely repetitive. We'll need to rethink how we do queue/emit to improve this.
2020-05-30 22:59:41 +02:00
RECONSUME_IN(ScriptData);
LibWeb: A whole bunch of work towards spec-compliant <script> elements This is still very unfinished, but there's at least a skeleton of code.
2020-05-24 22:00:46 +02:00
}
ON('/')
{
LibWeb: User getter+setter for HTMLToken tag name and self-closing flag
2021-07-14 23:37:48 +02:00
m_current_token.set_tag_name(consume_current_builder());
LibWeb: A whole bunch of work towards spec-compliant <script> elements This is still very unfinished, but there's at least a skeleton of code.
2020-05-24 22:00:46 +02:00
if (current_end_tag_token_is_appropriate())
SWITCH_TO(SelfClosingStartTag);
LibWeb: Implement more of the ScriptDataEndTagName tokenizer state Some of this is extremely repetitive. We'll need to rethink how we do queue/emit to improve this.
2020-05-30 22:59:41 +02:00
m_queued_tokens.enqueue(HTMLToken::make_character('<'));
m_queued_tokens.enqueue(HTMLToken::make_character('/'));
LibWeb: Fix assertion failures in HTMLTokenizer The *TagName states are all very similar, so it seems to be correct to apply the fix from #8761 to all of those states. This fixes #8788.
2021-07-16 00:36:10 +02:00
// NOTE: The spec doesn't mention this, but it seems that m_current_token (an end tag) is just dropped in this case.
m_current_builder.clear();
Unicode: Try s/codepoint/code_point/g again This time, without trailing 's'. Ran: git grep -l 'codepoint' | xargs sed -ie 's/codepoint/code_point/g
2020-08-05 16:31:20 -04:00
for (auto code_point : m_temporary_buffer)
m_queued_tokens.enqueue(HTMLToken::make_character(code_point));
LibWeb: Implement more of the ScriptDataEndTagName tokenizer state Some of this is extremely repetitive. We'll need to rethink how we do queue/emit to improve this.
2020-05-30 22:59:41 +02:00
RECONSUME_IN(ScriptData);
LibWeb: A whole bunch of work towards spec-compliant <script> elements This is still very unfinished, but there's at least a skeleton of code.
2020-05-24 22:00:46 +02:00
}
ON('>')
{
LibWeb: User getter+setter for HTMLToken tag name and self-closing flag
2021-07-14 23:37:48 +02:00
m_current_token.set_tag_name(consume_current_builder());
LibWeb: A whole bunch of work towards spec-compliant <script> elements This is still very unfinished, but there's at least a skeleton of code.
2020-05-24 22:00:46 +02:00
if (current_end_tag_token_is_appropriate())
SWITCH_TO_AND_EMIT_CURRENT_TOKEN(Data);
LibWeb: Implement more of the ScriptDataEndTagName tokenizer state Some of this is extremely repetitive. We'll need to rethink how we do queue/emit to improve this.
2020-05-30 22:59:41 +02:00
m_queued_tokens.enqueue(HTMLToken::make_character('<'));
m_queued_tokens.enqueue(HTMLToken::make_character('/'));
LibWeb: Fix assertion failures in HTMLTokenizer The *TagName states are all very similar, so it seems to be correct to apply the fix from #8761 to all of those states. This fixes #8788.
2021-07-16 00:36:10 +02:00
// NOTE: The spec doesn't mention this, but it seems that m_current_token (an end tag) is just dropped in this case.
m_current_builder.clear();
Unicode: Try s/codepoint/code_point/g again This time, without trailing 's'. Ran: git grep -l 'codepoint' | xargs sed -ie 's/codepoint/code_point/g
2020-08-05 16:31:20 -04:00
for (auto code_point : m_temporary_buffer)
m_queued_tokens.enqueue(HTMLToken::make_character(code_point));
LibWeb: Implement more of the ScriptDataEndTagName tokenizer state Some of this is extremely repetitive. We'll need to rethink how we do queue/emit to improve this.
2020-05-30 22:59:41 +02:00
RECONSUME_IN(ScriptData);
LibWeb: A whole bunch of work towards spec-compliant <script> elements This is still very unfinished, but there's at least a skeleton of code.
2020-05-24 22:00:46 +02:00
}
ON_ASCII_UPPER_ALPHA
{
LibWeb: Remove StringBuilder from HTMLToken::m_tag
2021-05-23 09:16:07 +02:00
m_current_builder.append_code_point(to_ascii_lowercase(current_input_character.value()));
LibWeb: A whole bunch of work towards spec-compliant <script> elements This is still very unfinished, but there's at least a skeleton of code.
2020-05-24 22:00:46 +02:00
m_temporary_buffer.append(current_input_character.value());
continue;
}
ON_ASCII_LOWER_ALPHA
{
LibWeb: Remove StringBuilder from HTMLToken::m_tag
2021-05-23 09:16:07 +02:00
m_current_builder.append(current_input_character.value());
LibWeb: A whole bunch of work towards spec-compliant <script> elements This is still very unfinished, but there's at least a skeleton of code.
2020-05-24 22:00:46 +02:00
m_temporary_buffer.append(current_input_character.value());
continue;
}
ANYTHING_ELSE
{
LibWeb: Implement more of the ScriptDataEndTagName tokenizer state Some of this is extremely repetitive. We'll need to rethink how we do queue/emit to improve this.
2020-05-30 22:59:41 +02:00
m_queued_tokens.enqueue(HTMLToken::make_character('<'));
m_queued_tokens.enqueue(HTMLToken::make_character('/'));
LibWeb: Fix assertion failure when tokenizing JS regex literals This fixes parsing the following regular expression: /</g; It also adds a simple script element to the HTMLTokenizer regression test, which also contains that specific regex.
2021-07-15 01:25:34 +02:00
// NOTE: The spec doesn't mention this, but it seems that m_current_token (an end tag) is just dropped in this case.
m_current_builder.clear();
Unicode: Try s/codepoint/code_point/g again This time, without trailing 's'. Ran: git grep -l 'codepoint' | xargs sed -ie 's/codepoint/code_point/g
2020-08-05 16:31:20 -04:00
for (auto code_point : m_temporary_buffer)
m_queued_tokens.enqueue(HTMLToken::make_character(code_point));
LibWeb: Implement more of the ScriptDataEndTagName tokenizer state Some of this is extremely repetitive. We'll need to rethink how we do queue/emit to improve this.
2020-05-30 22:59:41 +02:00
RECONSUME_IN(ScriptData);
LibWeb: A whole bunch of work towards spec-compliant <script> elements This is still very unfinished, but there's at least a skeleton of code.
2020-05-24 22:00:46 +02:00
}
}
END_STATE
LibWeb: Add spec links to each HTML tokenizer state section I didn't add full spec comments this time, but this is better than nothing :^)
2022-02-15 18:12:15 +00:00
// 13.2.5.69 CDATA section state, https://html.spec.whatwg.org/multipage/parsing.html#cdata-section-state
LibWeb: Implement all CDATA tokenizer states Even though we haven't implemented any switches to these states yet, we may as well have them ready for when we do implement the switches.
2020-06-12 23:43:06 +01:00
BEGIN_STATE(CDATASection)
{
ON(']')
{
SWITCH_TO(CDATASectionBracket);
}
ON_EOF
{
LibWeb: Utilize SourceLocation for HTMLTokenizer logging
2021-04-24 20:22:30 -07:00
log_parse_error();
LibWeb: Implement all CDATA tokenizer states Even though we haven't implemented any switches to these states yet, we may as well have them ready for when we do implement the switches.
2020-06-12 23:43:06 +01:00
EMIT_EOF;
}
ANYTHING_ELSE
{
EMIT_CURRENT_CHARACTER;
}
}
END_STATE
LibWeb: Add spec links to each HTML tokenizer state section I didn't add full spec comments this time, but this is better than nothing :^)
2022-02-15 18:12:15 +00:00
// 13.2.5.70 CDATA section bracket state, https://html.spec.whatwg.org/multipage/parsing.html#cdata-section-bracket-state
LibWeb: Implement all CDATA tokenizer states Even though we haven't implemented any switches to these states yet, we may as well have them ready for when we do implement the switches.
2020-06-12 23:43:06 +01:00
BEGIN_STATE(CDATASectionBracket)
{
ON(']')
{
SWITCH_TO(CDATASectionEnd);
}
ANYTHING_ELSE
{
EMIT_CHARACTER_AND_RECONSUME_IN(']', CDATASection);
}
}
END_STATE
LibWeb: Add spec links to each HTML tokenizer state section I didn't add full spec comments this time, but this is better than nothing :^)
2022-02-15 18:12:15 +00:00
// 13.2.5.71 CDATA section end state, https://html.spec.whatwg.org/multipage/parsing.html#cdata-section-end-state
LibWeb: Implement all CDATA tokenizer states Even though we haven't implemented any switches to these states yet, we may as well have them ready for when we do implement the switches.
2020-06-12 23:43:06 +01:00
BEGIN_STATE(CDATASectionEnd)
{
ON(']')
{
EMIT_CHARACTER(']');
}
ON('>')
{
SWITCH_TO(Data);
}
ANYTHING_ELSE
{
m_queued_tokens.enqueue(HTMLToken::make_character(']'));
m_queued_tokens.enqueue(HTMLToken::make_character(']'));
RECONSUME_IN(CDATASection);
}
}
END_STATE
LibWeb: Begin work on a spec-compliant HTML parser In order to actually view the web as it is, we're gonna need a proper HTML parser. So let's build one! This patch introduces the Web::HTMLTokenizer class, which currently operates on a StringView input stream where it fetches (ASCII only atm) codepoints and tokenizes acccording to the HTML spec tokenization algo. The tokenizer state machine looks a bit weird but is written in a way that tries to mimic the spec as closely as possible, in order to make development easier and bugs less likely. This initial version is far from finished, but it can parse a trivial document with a DOCTYPE and open/close tags. :^)
2020-05-22 21:46:13 +02:00
default:
LibWeb: A whole bunch of work towards spec-compliant <script> elements This is still very unfinished, but there's at least a skeleton of code.
2020-05-24 22:00:46 +02:00
TODO();
LibWeb: Begin work on a spec-compliant HTML parser In order to actually view the web as it is, we're gonna need a proper HTML parser. So let's build one! This patch introduces the Web::HTMLTokenizer class, which currently operates on a StringView input stream where it fetches (ASCII only atm) codepoints and tokenizes acccording to the HTML spec tokenization algo. The tokenizer state machine looks a bit weird but is written in a way that tries to mimic the spec as closely as possible, in order to make development easier and bugs less likely. This initial version is far from finished, but it can parse a trivial document with a DOCTYPE and open/close tags. :^)
2020-05-22 21:46:13 +02:00
}
}
}
Everywhere: Pass AK::StringView by value
2021-11-11 00:55:02 +01:00
bool HTMLTokenizer::consume_next_if_match(StringView string, CaseSensitivity case_sensitivity)
LibWeb: Begin work on a spec-compliant HTML parser In order to actually view the web as it is, we're gonna need a proper HTML parser. So let's build one! This patch introduces the Web::HTMLTokenizer class, which currently operates on a StringView input stream where it fetches (ASCII only atm) codepoints and tokenizes acccording to the HTML spec tokenization algo. The tokenizer state machine looks a bit weird but is written in a way that tries to mimic the spec as closely as possible, in order to make development easier and bugs less likely. This initial version is far from finished, but it can parse a trivial document with a DOCTYPE and open/close tags. :^)
2020-05-22 21:46:13 +02:00
{
for (size_t i = 0; i < string.length(); ++i) {
Unicode: Try s/codepoint/code_point/g again This time, without trailing 's'. Ran: git grep -l 'codepoint' | xargs sed -ie 's/codepoint/code_point/g
2020-08-05 16:31:20 -04:00
auto code_point = peek_code_point(i);
if (!code_point.has_value())
LibWeb: Begin work on a spec-compliant HTML parser In order to actually view the web as it is, we're gonna need a proper HTML parser. So let's build one! This patch introduces the Web::HTMLTokenizer class, which currently operates on a StringView input stream where it fetches (ASCII only atm) codepoints and tokenizes acccording to the HTML spec tokenization algo. The tokenizer state machine looks a bit weird but is written in a way that tries to mimic the spec as closely as possible, in order to make development easier and bugs less likely. This initial version is far from finished, but it can parse a trivial document with a DOCTYPE and open/close tags. :^)
2020-05-22 21:46:13 +02:00
return false;
// FIXME: This should be more Unicode-aware.
LibWeb: Checking for "DOCTYPE" should be case insensitive in tokenizer
2020-05-25 19:22:23 +02:00
if (case_sensitivity == CaseSensitivity::CaseInsensitive) {
Unicode: Try s/codepoint/code_point/g again This time, without trailing 's'. Ran: git grep -l 'codepoint' | xargs sed -ie 's/codepoint/code_point/g
2020-08-05 16:31:20 -04:00
if (code_point.value() < 0x80) {
Everywhere: Replace ctype.h to avoid narrowing conversions This replaces ctype.h with CharacterType.h everywhere I could find issues with narrowing conversions. While using it will probably make sense almost everywhere in the future, the most critical places should have been addressed.
2021-06-01 21:18:08 +02:00
if (to_ascii_lowercase(code_point.value()) != to_ascii_lowercase(string[i]))
LibWeb: Checking for "DOCTYPE" should be case insensitive in tokenizer
2020-05-25 19:22:23 +02:00
return false;
continue;
}
}
Unicode: Try s/codepoint/code_point/g again This time, without trailing 's'. Ran: git grep -l 'codepoint' | xargs sed -ie 's/codepoint/code_point/g
2020-08-05 16:31:20 -04:00
if (code_point.value() != (u32)string[i])
LibWeb: Begin work on a spec-compliant HTML parser In order to actually view the web as it is, we're gonna need a proper HTML parser. So let's build one! This patch introduces the Web::HTMLTokenizer class, which currently operates on a StringView input stream where it fetches (ASCII only atm) codepoints and tokenizes acccording to the HTML spec tokenization algo. The tokenizer state machine looks a bit weird but is written in a way that tries to mimic the spec as closely as possible, in order to make development easier and bugs less likely. This initial version is far from finished, but it can parse a trivial document with a DOCTYPE and open/close tags. :^)
2020-05-22 21:46:13 +02:00
return false;
}
LibWeb: Add position tracking information to HTML tokens
2021-05-20 23:11:41 +04:30
skip(string.length());
LibWeb: Begin work on a spec-compliant HTML parser In order to actually view the web as it is, we're gonna need a proper HTML parser. So let's build one! This patch introduces the Web::HTMLTokenizer class, which currently operates on a StringView input stream where it fetches (ASCII only atm) codepoints and tokenizes acccording to the HTML spec tokenization algo. The tokenizer state machine looks a bit weird but is written in a way that tries to mimic the spec as closely as possible, in order to make development easier and bugs less likely. This initial version is far from finished, but it can parse a trivial document with a DOCTYPE and open/close tags. :^)
2020-05-22 21:46:13 +02:00
return true;
}
void HTMLTokenizer::create_new_token(HTMLToken::Type type)
{
LibWeb: Add HTMLToken(Type) constructor and use it
2021-07-15 15:55:46 +02:00
m_current_token = { type };
LibWeb: Add position tracking information to HTML tokens
2021-05-20 23:11:41 +04:30
size_t offset = 0;
switch (type) {
case HTMLToken::Type::StartTag:
offset = 1;
break;
case HTMLToken::Type::EndTag:
offset = 2;
break;
default:
break;
}
LibWeb: Use setter for HTMLToken::m_{start,end}_position
2021-07-15 15:53:54 +02:00
m_current_token.set_start_position({}, nth_last_position(offset));
LibWeb: Begin work on a spec-compliant HTML parser In order to actually view the web as it is, we're gonna need a proper HTML parser. So let's build one! This patch introduces the Web::HTMLTokenizer class, which currently operates on a StringView input stream where it fetches (ASCII only atm) codepoints and tokenizes acccording to the HTML spec tokenization algo. The tokenizer state machine looks a bit weird but is written in a way that tries to mimic the spec as closely as possible, in order to make development easier and bugs less likely. This initial version is far from finished, but it can parse a trivial document with a DOCTYPE and open/close tags. :^)
2020-05-22 21:46:13 +02:00
}
LibWeb: Add basic support for dynamic markup insertion This implements basic support for dynamic markup insertion, adding * Document::open() * Document::write(Vector<String> const&) * Document::writeln(Vector<String> const&) * Document::close() The HTMLParser is modified to make it possible to create a script-created parser which initially only contains a HTMLTokenizer without any data. Aditionally the HTMLParser::run method gains an overload which does not modify the Document and does not run HTMLParser::the_end() so that we can reenter the parser at a later time. Furthermore all FIXMEs that consern the insertion point are implemented wich is defined in the HTMLTokenizer. Additionally the following member-variables of the HTMLParser are now exposed by getter funcions: * m_tokenizer * m_aborted * m_script_nesting_level The HTMLTokenizer is modified so that it contains an insertion point which keeps track of where the next input from the Document::write functions will be inserted. The insertion point is implemented as the charakter offset into m_decoded_input and a boolean describing if the insertion point is defined. Functions to update, check and {re}store the insertion point are also added. The function HTMLTokenizer::insert_eof is added to tell a script-created parser that document::close was called and HTMLParser::the_end() should be called. Lastly an explicit default constructor is added to HTMLTokenizer to create a empty HTMLTokenizer into which data can be inserted.
2022-02-19 15:58:21 +01:00
HTMLTokenizer::HTMLTokenizer()
{
m_decoded_input = "";
m_utf8_view = Utf8View(m_decoded_input);
m_utf8_iterator = m_utf8_view.begin();
m_prev_utf8_iterator = m_utf8_view.begin();
m_source_positions.empend(0u, 0u);
}
Everywhere: Pass AK::StringView by value
2021-11-11 00:55:02 +01:00
HTMLTokenizer::HTMLTokenizer(StringView input, String const& encoding)
LibWeb: Begin work on a spec-compliant HTML parser In order to actually view the web as it is, we're gonna need a proper HTML parser. So let's build one! This patch introduces the Web::HTMLTokenizer class, which currently operates on a StringView input stream where it fetches (ASCII only atm) codepoints and tokenizes acccording to the HTML spec tokenization algo. The tokenizer state machine looks a bit weird but is written in a way that tries to mimic the spec as closely as possible, in order to make development easier and bugs less likely. This initial version is far from finished, but it can parse a trivial document with a DOCTYPE and open/close tags. :^)
2020-05-22 21:46:13 +02:00
{
LibWeb: Plumb content encoding into the new HTML parser We still don't handle non-ASCII input correctly, but at least now we'll convert e.g ISO-8859-1 to UTF-8 before starting to tokenize. This patch also makes "view source" work with the new parser. :^)
2020-05-28 12:35:19 +02:00
auto* decoder = TextCodec::decoder_for(encoding);
Everywhere: Rename ASSERT => VERIFY (...and ASSERT_NOT_REACHED => VERIFY_NOT_REACHED) Since all of these checks are done in release builds as well, let's rename them to VERIFY to prevent confusion, as everyone is used to assertions being compiled out in release. We can introduce a new ASSERT macro that is specifically for debug checks, but I'm doing this wholesale conversion first since we've accumulated thousands of these already, and it's not immediately obvious which ones are suitable for ASSERT.
2021-02-23 20:42:32 +01:00
VERIFY(decoder);
LibWeb: Plumb content encoding into the new HTML parser We still don't handle non-ASCII input correctly, but at least now we'll convert e.g ISO-8859-1 to UTF-8 before starting to tokenize. This patch also makes "view source" work with the new parser. :^)
2020-05-28 12:35:19 +02:00
m_decoded_input = decoder->to_utf8(input);
LibWeb: Make the new HTML parser parse input as UTF-8 We already convert the input to UTF-8 before starting the tokenizer, so all this patch had to do was switch the tokenizer to use an Utf8View for its input (and to emit 32-bit codepoints.)
2020-06-04 21:06:54 +02:00
m_utf8_view = Utf8View(m_decoded_input);
m_utf8_iterator = m_utf8_view.begin();
LibWeb: Add basic support for dynamic markup insertion This implements basic support for dynamic markup insertion, adding * Document::open() * Document::write(Vector<String> const&) * Document::writeln(Vector<String> const&) * Document::close() The HTMLParser is modified to make it possible to create a script-created parser which initially only contains a HTMLTokenizer without any data. Aditionally the HTMLParser::run method gains an overload which does not modify the Document and does not run HTMLParser::the_end() so that we can reenter the parser at a later time. Furthermore all FIXMEs that consern the insertion point are implemented wich is defined in the HTMLTokenizer. Additionally the following member-variables of the HTMLParser are now exposed by getter funcions: * m_tokenizer * m_aborted * m_script_nesting_level The HTMLTokenizer is modified so that it contains an insertion point which keeps track of where the next input from the Document::write functions will be inserted. The insertion point is implemented as the charakter offset into m_decoded_input and a boolean describing if the insertion point is defined. Functions to update, check and {re}store the insertion point are also added. The function HTMLTokenizer::insert_eof is added to tell a script-created parser that document::close was called and HTMLParser::the_end() should be called. Lastly an explicit default constructor is added to HTMLTokenizer to create a empty HTMLTokenizer into which data can be inserted.
2022-02-19 15:58:21 +01:00
m_prev_utf8_iterator = m_utf8_view.begin();
LibWeb: Add position tracking information to HTML tokens
2021-05-20 23:11:41 +04:30
m_source_positions.empend(0u, 0u);
LibWeb: Begin work on a spec-compliant HTML parser In order to actually view the web as it is, we're gonna need a proper HTML parser. So let's build one! This patch introduces the Web::HTMLTokenizer class, which currently operates on a StringView input stream where it fetches (ASCII only atm) codepoints and tokenizes acccording to the HTML spec tokenization algo. The tokenizer state machine looks a bit weird but is written in a way that tries to mimic the spec as closely as possible, in order to make development easier and bugs less likely. This initial version is far from finished, but it can parse a trivial document with a DOCTYPE and open/close tags. :^)
2020-05-22 21:46:13 +02:00
}
LibWeb: Add basic support for dynamic markup insertion This implements basic support for dynamic markup insertion, adding * Document::open() * Document::write(Vector<String> const&) * Document::writeln(Vector<String> const&) * Document::close() The HTMLParser is modified to make it possible to create a script-created parser which initially only contains a HTMLTokenizer without any data. Aditionally the HTMLParser::run method gains an overload which does not modify the Document and does not run HTMLParser::the_end() so that we can reenter the parser at a later time. Furthermore all FIXMEs that consern the insertion point are implemented wich is defined in the HTMLTokenizer. Additionally the following member-variables of the HTMLParser are now exposed by getter funcions: * m_tokenizer * m_aborted * m_script_nesting_level The HTMLTokenizer is modified so that it contains an insertion point which keeps track of where the next input from the Document::write functions will be inserted. The insertion point is implemented as the charakter offset into m_decoded_input and a boolean describing if the insertion point is defined. Functions to update, check and {re}store the insertion point are also added. The function HTMLTokenizer::insert_eof is added to tell a script-created parser that document::close was called and HTMLParser::the_end() should be called. Lastly an explicit default constructor is added to HTMLTokenizer to create a empty HTMLTokenizer into which data can be inserted.
2022-02-19 15:58:21 +01:00
void HTMLTokenizer::insert_input_at_insertion_point(String const& input)
{
auto utf8_iterator_byte_offset = m_utf8_view.byte_offset_of(m_utf8_iterator);
// FIXME: Implement a InputStream to handle insertion_point and iterators.
StringBuilder builder {};
builder.append(m_decoded_input.substring(0, m_insertion_point.position));
builder.append(input);
builder.append(m_decoded_input.substring(m_insertion_point.position));
m_decoded_input = builder.build();
m_utf8_view = Utf8View(m_decoded_input);
m_utf8_iterator = m_utf8_view.iterator_at_byte_offset(utf8_iterator_byte_offset);
m_insertion_point.position += input.length();
}
void HTMLTokenizer::insert_eof()
{
m_explicit_eof_inserted = true;
}
bool HTMLTokenizer::is_eof_inserted()
{
return m_explicit_eof_inserted;
}
LibWeb: Begin work on a spec-compliant HTML parser In order to actually view the web as it is, we're gonna need a proper HTML parser. So let's build one! This patch introduces the Web::HTMLTokenizer class, which currently operates on a StringView input stream where it fetches (ASCII only atm) codepoints and tokenizes acccording to the HTML spec tokenization algo. The tokenizer state machine looks a bit weird but is written in a way that tries to mimic the spec as closely as possible, in order to make development easier and bugs less likely. This initial version is far from finished, but it can parse a trivial document with a DOCTYPE and open/close tags. :^)
2020-05-22 21:46:13 +02:00
void HTMLTokenizer::will_switch_to([[maybe_unused]] State new_state)
{
Everywhere: Replace dbgln<flag>(...) with dbgln_if(flag, ...) Replacement made by `find Kernel Userland -name '*.h' -o -name '*.cpp' | sed -i -Ee 's/dbgln\b<(\w+)>\(/dbgln_if(\1, /g'`
2021-02-07 15:33:24 +03:30
dbgln_if(TOKENIZER_TRACE_DEBUG, "[{}] Switch to {}", state_name(m_state), state_name(new_state));
LibWeb: Begin work on a spec-compliant HTML parser In order to actually view the web as it is, we're gonna need a proper HTML parser. So let's build one! This patch introduces the Web::HTMLTokenizer class, which currently operates on a StringView input stream where it fetches (ASCII only atm) codepoints and tokenizes acccording to the HTML spec tokenization algo. The tokenizer state machine looks a bit weird but is written in a way that tries to mimic the spec as closely as possible, in order to make development easier and bugs less likely. This initial version is far from finished, but it can parse a trivial document with a DOCTYPE and open/close tags. :^)
2020-05-22 21:46:13 +02:00
}
void HTMLTokenizer::will_reconsume_in([[maybe_unused]] State new_state)
{
Everywhere: Replace dbgln<flag>(...) with dbgln_if(flag, ...) Replacement made by `find Kernel Userland -name '*.h' -o -name '*.cpp' | sed -i -Ee 's/dbgln\b<(\w+)>\(/dbgln_if(\1, /g'`
2021-02-07 15:33:24 +03:30
dbgln_if(TOKENIZER_TRACE_DEBUG, "[{}] Reconsume in {}", state_name(m_state), state_name(new_state));
LibWeb: Begin work on a spec-compliant HTML parser In order to actually view the web as it is, we're gonna need a proper HTML parser. So let's build one! This patch introduces the Web::HTMLTokenizer class, which currently operates on a StringView input stream where it fetches (ASCII only atm) codepoints and tokenizes acccording to the HTML spec tokenization algo. The tokenizer state machine looks a bit weird but is written in a way that tries to mimic the spec as closely as possible, in order to make development easier and bugs less likely. This initial version is far from finished, but it can parse a trivial document with a DOCTYPE and open/close tags. :^)
2020-05-22 21:46:13 +02:00
}
LibWeb: Rename HTMLDocumentParser => HTMLParser
2021-09-25 23:15:48 +02:00
void HTMLTokenizer::switch_to(Badge<HTMLParser>, State new_state)
LibWeb: More work on the HTML parser and tokenizer The parser can now switch the state of the tokenizer! Very webby. :^)
2020-05-24 20:24:43 +02:00
{
Everywhere: Replace dbgln<flag>(...) with dbgln_if(flag, ...) Replacement made by `find Kernel Userland -name '*.h' -o -name '*.cpp' | sed -i -Ee 's/dbgln\b<(\w+)>\(/dbgln_if(\1, /g'`
2021-02-07 15:33:24 +03:30
dbgln_if(TOKENIZER_TRACE_DEBUG, "[{}] Parser switches tokenizer state to {}", state_name(m_state), state_name(new_state));
LibWeb: More work on the HTML parser and tokenizer The parser can now switch the state of the tokenizer! Very webby. :^)
2020-05-24 20:24:43 +02:00
m_state = new_state;
}
void HTMLTokenizer::will_emit(HTMLToken& token)
{
if (token.is_start_tag())
LibWeb: Use an Optional<String> to track the last HTML start tag Using an HTMLToken object here is unnecessary because the only attribute we're interested in is the tag_name.
2021-05-22 20:17:09 +02:00
m_last_emitted_start_tag_name = token.tag_name();
LibWeb: Use setter for HTMLToken::m_{start,end}_position
2021-07-15 15:53:54 +02:00
token.set_end_position({}, nth_last_position(0));
LibWeb: More work on the HTML parser and tokenizer The parser can now switch the state of the tokenizer! Very webby. :^)
2020-05-24 20:24:43 +02:00
}
bool HTMLTokenizer::current_end_tag_token_is_appropriate() const
{
Everywhere: Rename ASSERT => VERIFY (...and ASSERT_NOT_REACHED => VERIFY_NOT_REACHED) Since all of these checks are done in release builds as well, let's rename them to VERIFY to prevent confusion, as everyone is used to assertions being compiled out in release. We can introduce a new ASSERT macro that is specifically for debug checks, but I'm doing this wholesale conversion first since we've accumulated thousands of these already, and it's not immediately obvious which ones are suitable for ASSERT.
2021-02-23 20:42:32 +01:00
VERIFY(m_current_token.is_end_tag());
LibWeb: Use an Optional<String> to track the last HTML start tag Using an HTMLToken object here is unnecessary because the only attribute we're interested in is the tag_name.
2021-05-22 20:17:09 +02:00
if (!m_last_emitted_start_tag_name.has_value())
LibWeb: More work on the HTML parser and tokenizer The parser can now switch the state of the tokenizer! Very webby. :^)
2020-05-24 20:24:43 +02:00
return false;
LibWeb: Use an Optional<String> to track the last HTML start tag Using an HTMLToken object here is unnecessary because the only attribute we're interested in is the tag_name.
2021-05-22 20:17:09 +02:00
return m_current_token.tag_name() == m_last_emitted_start_tag_name.value();
LibWeb: More work on the HTML parser and tokenizer The parser can now switch the state of the tokenizer! Very webby. :^)
2020-05-24 20:24:43 +02:00
}
LibWeb: Implement partial support for numeric character references
2020-05-27 16:16:23 +02:00
bool HTMLTokenizer::consumed_as_part_of_an_attribute() const
{
return m_return_state == State::AttributeValueUnquoted || m_return_state == State::AttributeValueSingleQuoted || m_return_state == State::AttributeValueDoubleQuoted;
}
LibWeb: Change HTMLTokenizer.{cpp,h} to east const style
2021-07-12 12:44:21 +02:00
void HTMLTokenizer::restore_to(Utf8CodePointIterator const& new_iterator)
LibWeb: Add position tracking information to HTML tokens
2021-05-20 23:11:41 +04:30
{
LibWeb: Fix off-by-one in HTMLTokenizer::restore_to() The difference should be between m_utf8_iterator and the the new position, if m_prev_utf8_iterator is used one fewer source position is popped than required. This issue was not apparent on most pages since restore_to used for tokens such <!doctype> that are normally followed by a newline that resets the column to zero, but it can be seen on pages with minified HTML.
2022-02-13 14:08:53 +00:00
auto diff = m_utf8_iterator - new_iterator;
if (diff > 0) {
LibWeb: Fix Array OOBs in the HTMLTokenizer Accessing last() if there are no elements makes WebContent crash :^)
2022-06-02 01:03:44 +02:00
for (ssize_t i = 0; i < diff; ++i) {
if (!m_source_positions.is_empty())
m_source_positions.take_last();
}
LibWeb: Fix off-by-one in HTMLTokenizer::restore_to() The difference should be between m_utf8_iterator and the the new position, if m_prev_utf8_iterator is used one fewer source position is popped than required. This issue was not apparent on most pages since restore_to used for tokens such <!doctype> that are normally followed by a newline that resets the column to zero, but it can be seen on pages with minified HTML.
2022-02-13 14:08:53 +00:00
} else {
// Going forwards...?
TODO();
LibWeb: Add position tracking information to HTML tokens
2021-05-20 23:11:41 +04:30
}
LibWeb: Fix invalid behaviour of HTMLTokenizer::skip() and restore_to() skip() is supposed to end up keeping the previous iterator only one index behind the current one, and restore_to() should actually do the restore instead of just removing the now-useless source positions. Fixes #7331.
2021-05-21 11:03:31 +04:30
m_utf8_iterator = new_iterator;
LibWeb: Add position tracking information to HTML tokens
2021-05-20 23:11:41 +04:30
}
LibWeb: Remove StringBuilders from HTMLToken::m_doctype
2021-05-23 08:20:03 +02:00
String HTMLTokenizer::consume_current_builder()
{
auto string = m_current_builder.to_string();
m_current_builder.clear();
return string;
}
LibWeb: Begin work on a spec-compliant HTML parser In order to actually view the web as it is, we're gonna need a proper HTML parser. So let's build one! This patch introduces the Web::HTMLTokenizer class, which currently operates on a StringView input stream where it fetches (ASCII only atm) codepoints and tokenizes acccording to the HTML spec tokenization algo. The tokenizer state machine looks a bit weird but is written in a way that tries to mimic the spec as closely as possible, in order to make development easier and bugs less likely. This initial version is far from finished, but it can parse a trivial document with a DOCTYPE and open/close tags. :^)
2020-05-22 21:46:13 +02:00
}
Reference in a new issue Copy permalink