Stowage/ladybird
2
0
Fork 0
mirror of https://github.com/LadybirdBrowser/ladybird.git synced 2025-10-20 16:13:21 +00:00
Code Issues Projects Releases Packages Wiki Activity
ladybird/Libraries/LibWeb/HTML/Parser/HTMLToken.cpp

107 lines
3.5 KiB
C++
Raw Normal View History

LibWeb: Start building the tree building part of the new HTML parser This patch adds a new HTMLDocumentParser class. It keeps a tokenizer object internally and feeds itself with one token at a time from it. The names and idioms in this class are expressed as closely to the actual HTML parsing spec as possible, to make development as easy and bug free as possible. :^) This is going to become pretty large, but it's pretty cool!
2020-05-24 00:14:23 +02:00
/*
Meta: Update my e-mail address everywhere
2024-10-04 13:19:50 +02:00
* Copyright (c) 2020, Andreas Kling <andreas@ladybird.org>
LibWeb: Start building the tree building part of the new HTML parser This patch adds a new HTMLDocumentParser class. It keeps a tokenizer object internally and feeds itself with one token at a time from it. The names and idioms in this class are expressed as closely to the actual HTML parsing spec as possible, to make development as easy and bug free as possible. :^) This is going to become pretty large, but it's pretty cool!
2020-05-24 00:14:23 +02:00
*
Everything: Move to SPDX license identifiers in all files. SPDX License Identifiers are a more compact / standardized way of representing file license information. See: https://spdx.dev/resources/use/#identifiers This was done with the `ambr` search and replace tool. ambr --no-parent-ignore --key-from-file --rep-from-file key.txt rep.txt *
2021-04-22 01:24:48 -07:00
* SPDX-License-Identifier: BSD-2-Clause
LibWeb: Start building the tree building part of the new HTML parser This patch adds a new HTMLDocumentParser class. It keeps a tokenizer object internally and feeds itself with one token at a time from it. The names and idioms in this class are expressed as closely to the actual HTML parsing spec as possible, to make development as easy and bug free as possible. :^) This is going to become pretty large, but it's pretty cool!
2020-05-24 00:14:23 +02:00
*/
LibWeb: Deduplicate attributes when emitting start and end tags The HTML tokenizer specification says that we're supposed to do this when leaving the Attribute name or when emitting the token, as appropriate. Hopefully 'as appropriate' can mean only when emitting the token, as that's the easiest place to insert this logic without complicating the tokenizer any more.
2024-09-30 17:52:30 -06:00
#include <AK/HashTable.h>
LibWeb: Move the HTML parser into HTML/Parser/
2020-07-28 19:18:23 +02:00
#include <LibWeb/HTML/Parser/HTMLToken.h>
LibWeb: Start building the tree building part of the new HTML parser This patch adds a new HTMLDocumentParser class. It keeps a tokenizer object internally and feeds itself with one token at a time from it. The names and idioms in this class are expressed as closely to the actual HTML parsing spec as possible, to make development as easy and bug free as possible. :^) This is going to become pretty large, but it's pretty cool!
2020-05-24 00:14:23 +02:00
LibWeb: Move HTML classes into the Web::HTML namespace
2020-07-28 18:20:36 +02:00
namespace Web::HTML {
LibWeb: Start building the tree building part of the new HTML parser This patch adds a new HTMLDocumentParser class. It keeps a tokenizer object internally and feeds itself with one token at a time from it. The names and idioms in this class are expressed as closely to the actual HTML parsing spec as possible, to make development as easy and bug free as possible. :^) This is going to become pretty large, but it's pretty cool!
2020-05-24 00:14:23 +02:00
LibWeb: Port HTMLToken::to_deprecated_string to new AK String
2023-11-05 11:45:55 +13:00
String HTMLToken::to_string() const
LibWeb: Start building the tree building part of the new HTML parser This patch adds a new HTMLDocumentParser class. It keeps a tokenizer object internally and feeds itself with one token at a time from it. The names and idioms in this class are expressed as closely to the actual HTML parsing spec as possible, to make development as easy and bug free as possible. :^) This is going to become pretty large, but it's pretty cool!
2020-05-24 00:14:23 +02:00
{
StringBuilder builder;
switch (type()) {
case HTMLToken::Type::DOCTYPE:
Everywhere: Add sv suffix to strings relying on StringView(char const*) Each of these strings would previously rely on StringView's char const* constructor overload, which would call __builtin_strlen on the string. Since we now have operator ""sv, we can replace these with much simpler versions. This opens the door to being able to remove StringView(char const*). No functional changes.
2022-07-11 17:32:29 +00:00
builder.append("DOCTYPE"sv);
builder.append(" { name: '"sv);
LibWeb: Change HTMLToken::m_doctype into named DoctypeData struct This is in preparation for an upcoming storage change of HTMLToken. In contrast to the other token types, the accessor can hand out a mutable reference to allow users to change parts of the DoctypeData easily.
2021-07-15 00:03:50 +02:00
builder.append(doctype_data().name);
Everywhere: Add sv suffix to strings relying on StringView(char const*) Each of these strings would previously rely on StringView's char const* constructor overload, which would call __builtin_strlen on the string. Since we now have operator ""sv, we can replace these with much simpler versions. This opens the door to being able to remove StringView(char const*). No functional changes.
2022-07-11 17:32:29 +00:00
builder.append("' }"sv);
LibWeb: Start building the tree building part of the new HTML parser This patch adds a new HTMLDocumentParser class. It keeps a tokenizer object internally and feeds itself with one token at a time from it. The names and idioms in this class are expressed as closely to the actual HTML parsing spec as possible, to make development as easy and bug free as possible. :^) This is going to become pretty large, but it's pretty cool!
2020-05-24 00:14:23 +02:00
break;
case HTMLToken::Type::StartTag:
Everywhere: Add sv suffix to strings relying on StringView(char const*) Each of these strings would previously rely on StringView's char const* constructor overload, which would call __builtin_strlen on the string. Since we now have operator ""sv, we can replace these with much simpler versions. This opens the door to being able to remove StringView(char const*). No functional changes.
2022-07-11 17:32:29 +00:00
builder.append("StartTag"sv);
LibWeb: Start building the tree building part of the new HTML parser This patch adds a new HTMLDocumentParser class. It keeps a tokenizer object internally and feeds itself with one token at a time from it. The names and idioms in this class are expressed as closely to the actual HTML parsing spec as possible, to make development as easy and bug free as possible. :^) This is going to become pretty large, but it's pretty cool!
2020-05-24 00:14:23 +02:00
break;
case HTMLToken::Type::EndTag:
Everywhere: Add sv suffix to strings relying on StringView(char const*) Each of these strings would previously rely on StringView's char const* constructor overload, which would call __builtin_strlen on the string. Since we now have operator ""sv, we can replace these with much simpler versions. This opens the door to being able to remove StringView(char const*). No functional changes.
2022-07-11 17:32:29 +00:00
builder.append("EndTag"sv);
LibWeb: Start building the tree building part of the new HTML parser This patch adds a new HTMLDocumentParser class. It keeps a tokenizer object internally and feeds itself with one token at a time from it. The names and idioms in this class are expressed as closely to the actual HTML parsing spec as possible, to make development as easy and bug free as possible. :^) This is going to become pretty large, but it's pretty cool!
2020-05-24 00:14:23 +02:00
break;
case HTMLToken::Type::Comment:
Everywhere: Add sv suffix to strings relying on StringView(char const*) Each of these strings would previously rely on StringView's char const* constructor overload, which would call __builtin_strlen on the string. Since we now have operator ""sv, we can replace these with much simpler versions. This opens the door to being able to remove StringView(char const*). No functional changes.
2022-07-11 17:32:29 +00:00
builder.append("Comment"sv);
LibWeb: Start building the tree building part of the new HTML parser This patch adds a new HTMLDocumentParser class. It keeps a tokenizer object internally and feeds itself with one token at a time from it. The names and idioms in this class are expressed as closely to the actual HTML parsing spec as possible, to make development as easy and bug free as possible. :^) This is going to become pretty large, but it's pretty cool!
2020-05-24 00:14:23 +02:00
break;
case HTMLToken::Type::Character:
Everywhere: Add sv suffix to strings relying on StringView(char const*) Each of these strings would previously rely on StringView's char const* constructor overload, which would call __builtin_strlen on the string. Since we now have operator ""sv, we can replace these with much simpler versions. This opens the door to being able to remove StringView(char const*). No functional changes.
2022-07-11 17:32:29 +00:00
builder.append("Character"sv);
LibWeb: Start building the tree building part of the new HTML parser This patch adds a new HTMLDocumentParser class. It keeps a tokenizer object internally and feeds itself with one token at a time from it. The names and idioms in this class are expressed as closely to the actual HTML parsing spec as possible, to make development as easy and bug free as possible. :^) This is going to become pretty large, but it's pretty cool!
2020-05-24 00:14:23 +02:00
break;
case HTMLToken::Type::EndOfFile:
Everywhere: Add sv suffix to strings relying on StringView(char const*) Each of these strings would previously rely on StringView's char const* constructor overload, which would call __builtin_strlen on the string. Since we now have operator ""sv, we can replace these with much simpler versions. This opens the door to being able to remove StringView(char const*). No functional changes.
2022-07-11 17:32:29 +00:00
builder.append("EndOfFile"sv);
LibWeb: Start building the tree building part of the new HTML parser This patch adds a new HTMLDocumentParser class. It keeps a tokenizer object internally and feeds itself with one token at a time from it. The names and idioms in this class are expressed as closely to the actual HTML parsing spec as possible, to make development as easy and bug free as possible. :^) This is going to become pretty large, but it's pretty cool!
2020-05-24 00:14:23 +02:00
break;
LibWeb: More work on the HTML parser and tokenizer The parser can now switch the state of the tokenizer! Very webby. :^)
2020-05-24 20:24:43 +02:00
case HTMLToken::Type::Invalid:
Everywhere: Rename ASSERT => VERIFY (...and ASSERT_NOT_REACHED => VERIFY_NOT_REACHED) Since all of these checks are done in release builds as well, let's rename them to VERIFY to prevent confusion, as everyone is used to assertions being compiled out in release. We can introduce a new ASSERT macro that is specifically for debug checks, but I'm doing this wholesale conversion first since we've accumulated thousands of these already, and it's not immediately obvious which ones are suitable for ASSERT.
2021-02-23 20:42:32 +01:00
VERIFY_NOT_REACHED();
LibWeb: Start building the tree building part of the new HTML parser This patch adds a new HTMLDocumentParser class. It keeps a tokenizer object internally and feeds itself with one token at a time from it. The names and idioms in this class are expressed as closely to the actual HTML parsing spec as possible, to make development as easy and bug free as possible. :^) This is going to become pretty large, but it's pretty cool!
2020-05-24 00:14:23 +02:00
}
if (type() == HTMLToken::Type::StartTag || type() == HTMLToken::Type::EndTag) {
Everywhere: Add sv suffix to strings relying on StringView(char const*) Each of these strings would previously rely on StringView's char const* constructor overload, which would call __builtin_strlen on the string. Since we now have operator ""sv, we can replace these with much simpler versions. This opens the door to being able to remove StringView(char const*). No functional changes.
2022-07-11 17:32:29 +00:00
builder.append(" { name: '"sv);
LibWeb: User getter+setter for HTMLToken tag name and self-closing flag
2021-07-14 23:37:48 +02:00
builder.append(tag_name());
Everywhere: Add sv suffix to strings relying on StringView(char const*) Each of these strings would previously rely on StringView's char const* constructor overload, which would call __builtin_strlen on the string. Since we now have operator ""sv, we can replace these with much simpler versions. This opens the door to being able to remove StringView(char const*). No functional changes.
2022-07-11 17:32:29 +00:00
builder.append("', { "sv);
LibWeb: Hide implementation details of HTMLToken attribute list Previously, HTMLToken would expose the Vector<Attribute> directly to its users. In preparation for a future change, all users now use implementation-agnostic APIs which do not expose the Vector directly.
2021-07-14 23:53:11 +02:00
for_each_attribute([&](auto& attribute) {
LibWeb: Remove StringBuilders from HTMLToken::AttributeBuilder
2021-05-23 08:50:48 +02:00
builder.append(attribute.local_name);
Everywhere: Add sv suffix to strings relying on StringView(char const*) Each of these strings would previously rely on StringView's char const* constructor overload, which would call __builtin_strlen on the string. Since we now have operator ""sv, we can replace these with much simpler versions. This opens the door to being able to remove StringView(char const*). No functional changes.
2022-07-11 17:32:29 +00:00
builder.append("=\""sv);
LibWeb: Remove StringBuilders from HTMLToken::AttributeBuilder
2021-05-23 08:50:48 +02:00
builder.append(attribute.value);
Everywhere: Add sv suffix to strings relying on StringView(char const*) Each of these strings would previously rely on StringView's char const* constructor overload, which would call __builtin_strlen on the string. Since we now have operator ""sv, we can replace these with much simpler versions. This opens the door to being able to remove StringView(char const*). No functional changes.
2022-07-11 17:32:29 +00:00
builder.append("\" "sv);
LibWeb: Hide implementation details of HTMLToken attribute list Previously, HTMLToken would expose the Vector<Attribute> directly to its users. In preparation for a future change, all users now use implementation-agnostic APIs which do not expose the Vector directly.
2021-07-14 23:53:11 +02:00
return IterationDecision::Continue;
});
Everywhere: Add sv suffix to strings relying on StringView(char const*) Each of these strings would previously rely on StringView's char const* constructor overload, which would call __builtin_strlen on the string. Since we now have operator ""sv, we can replace these with much simpler versions. This opens the door to being able to remove StringView(char const*). No functional changes.
2022-07-11 17:32:29 +00:00
builder.append("} }"sv);
LibWeb: Start building the tree building part of the new HTML parser This patch adds a new HTMLDocumentParser class. It keeps a tokenizer object internally and feeds itself with one token at a time from it. The names and idioms in this class are expressed as closely to the actual HTML parsing spec as possible, to make development as easy and bug free as possible. :^) This is going to become pretty large, but it's pretty cool!
2020-05-24 00:14:23 +02:00
}
LibWeb: Change HTMLToken storage architecture This completely changes how HTMLTokens store their data. Previously, space was allocated for all token types separately. Now, the HTMLToken's data is stored in just a String, two booleans and a Variant. This change reduces sizeof(HTMLToken) from 68 to 32. Also, this reduces raw tokenization time by around 20 to 50 percent, depending on the page. Full document parsing time (with HTMLDocumentParser, on a local HTML page without any dependency files) is reduced by between 4 and 20 percent, depending on the page. Since tokenizing HTML pages can easily generated 50'000 tokens and more, the storage has been designed in a way that avoids heap allocations where possible, while trying to reduce the size of the tokens. The only tokens which need to allocate on the heap are thus DOCTYPE tokens (max. 1 per document), and tag tokens (but only if they have attributes). This way, only around 5 percent of all tokens generated need to allocate on the heap (except for StringImpl allocations).
2021-07-15 18:49:39 +02:00
if (is_comment()) {
Everywhere: Add sv suffix to strings relying on StringView(char const*) Each of these strings would previously rely on StringView's char const* constructor overload, which would call __builtin_strlen on the string. Since we now have operator ""sv, we can replace these with much simpler versions. This opens the door to being able to remove StringView(char const*). No functional changes.
2022-07-11 17:32:29 +00:00
builder.append(" { data: '"sv);
LibWeb: Change HTMLToken storage architecture This completely changes how HTMLTokens store their data. Previously, space was allocated for all token types separately. Now, the HTMLToken's data is stored in just a String, two booleans and a Variant. This change reduces sizeof(HTMLToken) from 68 to 32. Also, this reduces raw tokenization time by around 20 to 50 percent, depending on the page. Full document parsing time (with HTMLDocumentParser, on a local HTML page without any dependency files) is reduced by between 4 and 20 percent, depending on the page. Since tokenizing HTML pages can easily generated 50'000 tokens and more, the storage has been designed in a way that avoids heap allocations where possible, while trying to reduce the size of the tokens. The only tokens which need to allocate on the heap are thus DOCTYPE tokens (max. 1 per document), and tag tokens (but only if they have attributes). This way, only around 5 percent of all tokens generated need to allocate on the heap (except for StringImpl allocations).
2021-07-15 18:49:39 +02:00
builder.append(comment());
Everywhere: Add sv suffix to strings relying on StringView(char const*) Each of these strings would previously rely on StringView's char const* constructor overload, which would call __builtin_strlen on the string. Since we now have operator ""sv, we can replace these with much simpler versions. This opens the door to being able to remove StringView(char const*). No functional changes.
2022-07-11 17:32:29 +00:00
builder.append("' }"sv);
LibWeb: Change HTMLToken storage architecture This completely changes how HTMLTokens store their data. Previously, space was allocated for all token types separately. Now, the HTMLToken's data is stored in just a String, two booleans and a Variant. This change reduces sizeof(HTMLToken) from 68 to 32. Also, this reduces raw tokenization time by around 20 to 50 percent, depending on the page. Full document parsing time (with HTMLDocumentParser, on a local HTML page without any dependency files) is reduced by between 4 and 20 percent, depending on the page. Since tokenizing HTML pages can easily generated 50'000 tokens and more, the storage has been designed in a way that avoids heap allocations where possible, while trying to reduce the size of the tokens. The only tokens which need to allocate on the heap are thus DOCTYPE tokens (max. 1 per document), and tag tokens (but only if they have attributes). This way, only around 5 percent of all tokens generated need to allocate on the heap (except for StringImpl allocations).
2021-07-15 18:49:39 +02:00
}
if (is_character()) {
Everywhere: Add sv suffix to strings relying on StringView(char const*) Each of these strings would previously rely on StringView's char const* constructor overload, which would call __builtin_strlen on the string. Since we now have operator ""sv, we can replace these with much simpler versions. This opens the door to being able to remove StringView(char const*). No functional changes.
2022-07-11 17:32:29 +00:00
builder.append(" { data: '"sv);
LibWeb: Change HTMLToken storage architecture This completely changes how HTMLTokens store their data. Previously, space was allocated for all token types separately. Now, the HTMLToken's data is stored in just a String, two booleans and a Variant. This change reduces sizeof(HTMLToken) from 68 to 32. Also, this reduces raw tokenization time by around 20 to 50 percent, depending on the page. Full document parsing time (with HTMLDocumentParser, on a local HTML page without any dependency files) is reduced by between 4 and 20 percent, depending on the page. Since tokenizing HTML pages can easily generated 50'000 tokens and more, the storage has been designed in a way that avoids heap allocations where possible, while trying to reduce the size of the tokens. The only tokens which need to allocate on the heap are thus DOCTYPE tokens (max. 1 per document), and tag tokens (but only if they have attributes). This way, only around 5 percent of all tokens generated need to allocate on the heap (except for StringImpl allocations).
2021-07-15 18:49:39 +02:00
builder.append_code_point(code_point());
Everywhere: Add sv suffix to strings relying on StringView(char const*) Each of these strings would previously rely on StringView's char const* constructor overload, which would call __builtin_strlen on the string. Since we now have operator ""sv, we can replace these with much simpler versions. This opens the door to being able to remove StringView(char const*). No functional changes.
2022-07-11 17:32:29 +00:00
builder.append("' }"sv);
LibWeb: Start building the tree building part of the new HTML parser This patch adds a new HTMLDocumentParser class. It keeps a tokenizer object internally and feeds itself with one token at a time from it. The names and idioms in this class are expressed as closely to the actual HTML parsing spec as possible, to make development as easy and bug free as possible. :^) This is going to become pretty large, but it's pretty cool!
2020-05-24 00:14:23 +02:00
}
LibWeb: Be more forgiving when adding source positions in HTMLTokenizer This patch changes HTMLTokenizer::nth_last_position to not fail if the requested position is not available. Rather, it will just return (0-0). While this is not the correct solution, it prevents the tokenizer from crashing just because it cannot find a source position. This should only affect SyntaxHighlighter.
2021-06-04 11:31:43 +02:00
if (type() == HTMLToken::Type::Character) {
builder.appendff("@{}:{}", m_start_position.line, m_start_position.column);
} else {
builder.appendff("@{}:{}-{}:{}", m_start_position.line, m_start_position.column, m_end_position.line, m_end_position.column);
}
LibWeb: Add position tracking information to HTML tokens
2021-05-20 23:11:41 +04:30
LibWeb: Port HTMLToken::to_deprecated_string to new AK String
2023-11-05 11:45:55 +13:00
return MUST(builder.to_string());
LibWeb: Start building the tree building part of the new HTML parser This patch adds a new HTMLDocumentParser class. It keeps a tokenizer object internally and feeds itself with one token at a time from it. The names and idioms in this class are expressed as closely to the actual HTML parsing spec as possible, to make development as easy and bug free as possible. :^) This is going to become pretty large, but it's pretty cool!
2020-05-24 00:14:23 +02:00
}
LibWeb: Deduplicate attributes when emitting start and end tags The HTML tokenizer specification says that we're supposed to do this when leaving the Attribute name or when emitting the token, as appropriate. Hopefully 'as appropriate' can mean only when emitting the token, as that's the easiest place to insert this logic without complicating the tokenizer any more.
2024-09-30 17:52:30 -06:00
void HTMLToken::normalize_attributes()
{
// From AttributeNameState: https://html.spec.whatwg.org/multipage/parsing.html#attribute-name-state
//
// When the user agent leaves the attribute name state (and before emitting the tag token, if appropriate),
// the complete attribute's name must be compared to the other attributes on the same token;
// if there is already an attribute on the token with the exact same name, then this is a duplicate-attribute
// parse error and the new attribute must be removed from the token.
// NOTE: If an attribute is so removed from a token, it, and the value that gets associated with it, if any,
// are never subsequently used by the parser, and are therefore effectively discarded. Removing the attribute
// in this way does not change its status as the "current attribute" for the purposes of the tokenizer, however.
HashTable<FlyString> seen_attributes;
auto* ptr = tag_attributes();
if (!ptr)
return;
auto& tag_attributes = *ptr;
for (size_t i = 0; i < tag_attributes.size(); ++i) {
auto& attribute = tag_attributes[i];
if (seen_attributes.set(attribute.local_name, AK::HashSetExistingEntryBehavior::Keep) == AK::HashSetResult::KeptExistingEntry) {
// This is a duplicate attribute, remove it.
tag_attributes.remove(i);
--i;
LibWeb: Track if element was created from token with dupe attributes This is required for CSP to ignore the nonce attribute to prevent duplicate attributes hijacking the attribute. See https://w3c.github.io/webappsec-csp/#security-nonce-hijacking
2024-12-02 12:33:52 +00:00
m_had_duplicate_attribute = true;
LibWeb: Deduplicate attributes when emitting start and end tags The HTML tokenizer specification says that we're supposed to do this when leaving the Attribute name or when emitting the token, as appropriate. Hopefully 'as appropriate' can mean only when emitting the token, as that's the easiest place to insert this logic without complicating the tokenizer any more.
2024-09-30 17:52:30 -06:00
}
}
}
LibWeb: Start building the tree building part of the new HTML parser This patch adds a new HTMLDocumentParser class. It keeps a tokenizer object internally and feeds itself with one token at a time from it. The names and idioms in this class are expressed as closely to the actual HTML parsing spec as possible, to make development as easy and bug free as possible. :^) This is going to become pretty large, but it's pretty cool!
2020-05-24 00:14:23 +02:00
}
Reference in a new issue Copy permalink