ladybird/Libraries/LibWeb/HTML/Parser/Entities.cpp

100 lines
4.1 KiB
C++
Raw Normal View History

/*
* Copyright (c) 2020, Andreas Kling <andreas@ladybird.org>
*
* SPDX-License-Identifier: BSD-2-Clause
*/
LibWeb/HTML: Improve data structure of named character reference data Introduces a few ad-hoc modifications to the DAFSA aimed to increase performance while keeping the data size small. - The 'first layer' of nodes is extracted out and replaced with a lookup table. This turns the search for the first character from O(n) to O (1), and doesn't increase the data size because all first characters in the set of named character references have the values 'a'-'z'/'A'-'Z', so a lookup array of exactly 52 elements can be used. The lookup table stores the cumulative "number" fields that would be calculated by a linear scan that matches a given node, thus allowing the unique index to be built-up as normal with a O(1) search instead of a linear scan. - The 'second layer' of nodes is also extracted out and searches of the second layer are done using a bit field of 52 bits (the set bits of the bit field depend on the first character's value), where each set bit corresponds to one of 'a'-'z'/'A'-'Z' (similar to the first layer, the second layer can only contain ASCII alphabetic characters). The bit field is then re-used (along with an offset) to get the index into the array of second layer nodes. This technique ultimately allows for storing the minimum number of nodes in the second layer, and therefore only increasing the size of the data by the size of the 'first to second layer link' info which is 52 * 8 = 416 bytes. - After the second layer, the rest of the data is stored using a mostly-normal DAFSA, but there are still a few differences: - The "number" field is cumulative, in the same way that the first/second layer store a cumulative "number" field. This cuts down slightly on the amount of work done during the search of a list of children, and we can get away with it because the cumulative "number" fields of the remaining nodes in the DAFSA (after the first and second layer nodes were extracted out) happens to require few enough bits that we can store the cumulative version while staying under our 32-bit budget. - Instead of storing a 'last sibling' flag to denote the end of a list of children, the length of each node's list of children is stored. Again, this is mostly done just because there are enough bits available to do so while keeping the DAFSA node within 32 bits. - Note: Together, these modifications open up the possibility of using a binary search instead of a linear search over the children, but due to the consistently small lengths of the lists of children in the remaining DAFSA, a linear search actually seems to be the better option. The new data size is 24,724 bytes, up from 24,412 bytes (+312, -104 from the 52 first layer nodes going from 4-bytes to 2-bytes, and +416 from the addition of the 'first to second layer link' data). In terms of raw matching speed (outside the context of the tokenizer), this provides about a 1.72x speedup. In very named-character-reference-heavy tokenizer benchmarks, this provides about a 1.05x speedup (the effect of named character reference matching speed is diluted when benchmarking the tokenizer). Additionally, fixes the size of the named character reference data when targeting Windows.
2025-07-09 00:05:28 -07:00
#include <AK/Assertions.h>
#include <AK/BuiltinWrappers.h>
#include <AK/CharacterTypes.h>
#include <LibWeb/HTML/Parser/Entities.h>
LibWeb: Make named character references more spec-compliant & efficient There are two changes happening here: a correctness fix, and an optimization. In theory they are unrelated, but the optimization actually paves the way for the correctness fix. Before this commit, the HTML tokenizer would attempt to look for named character references by checking from after the `&` until the end of m_decoded_input, which meant that it was unable to recognize things like named character references that are inserted via `document.write` one byte at a time. For example, if `&notin;` was written one-byte-at-a-time with `document.write`, then the tokenizer would only check against `n` since that's all that would exist at the time of the check and therefore erroneously conclude that it was an invalid named character reference. This commit modifies the approach taken for named character reference matching by using a trie-like structure (specifically, a deterministic acyclic finite state automaton or DAFSA), which allows for efficiently matching one-character-at-a-time and therefore it is able to pick up matching where it left off after each code point is consumed. Note: Because it's possible for a partial match to not actually develop into a full match (e.g. `&notindo` which could lead to `&notindot;`), some backtracking is performed after-the-fact in order to only consume the code points within the longest match found (e.g. `&notindo` would backtrack back to `&not`). With this new approach, `document.write` being called one-byte-at-a-time is handled correctly, which allows for passing more WPT tests, with the most directly relevant tests being `/html/syntax/parsing/html5lib_entities01.html` and `/html/syntax/parsing/html5lib_entities02.html` when run with `?run_type=write_single`. Additionally, the implementation now better conforms to the language of the spec (and resolves a FIXME) because exactly the matched characters are consumed and nothing more, so SWITCH_TO is able to be used as the spec says instead of RECONSUME_IN. The new approach is also an optimization: - Instead of a linear search using `starts_with`, the usage of a DAFSA means that it is always aware of which characters can lead to a match at any given point, and will bail out whenever a match is no longer possible. - The DAFSA is able to take advantage of the note in the section `13.5 Named character references` that says "This list is static and will not be expanded or changed in the future." and tailor its Node struct accordingly to tightly pack each node's data into 32-bits. Together with the inherent DAFSA property of redundant node deduplication, the amount of data stored for named character reference matching is minimized. In my testing: - A benchmark tokenizing an arbitrary set of HTML test files was about 1.23x faster (2070ms to 1682ms). - A benchmark tokenizing a file with tens of thousands of named character references mixed in with truncated named character references and arbitrary ASCII characters/ampersands runs about 8x faster (758ms to 93ms). - The size of `liblagom-web.so` was reduced by 94.96KiB. Some technical details: A DAFSA (deterministic acyclic finite state automaton) is essentially a trie flattened into an array, but it also uses techniques to minimize redundant nodes. This provides fast lookups while minimizing the required data size, but normally does not allow for associating data related to each word. However, by adding a count of the number of possible words from each node, it becomes possible to also use it to achieve minimal perfect hashing for the set of words (which allows going from word -> unique index as well as unique index -> word). This allows us to store a second array of data so that the DAFSA can be used as a lookup for e.g. the associated code points. For the Swift implementation, the new NamedCharacterReferenceMatcher was used to satisfy the previous API and the tokenizer was left alone otherwise. In the future, the Swift implementation should be updated to use the same implementation for its NamedCharacterReference state as the updated C++ implementation.
2024-12-22 07:09:31 -08:00
#include <LibWeb/HTML/Parser/NamedCharacterReferences.h>
namespace Web::HTML {
LibWeb/HTML: Improve data structure of named character reference data Introduces a few ad-hoc modifications to the DAFSA aimed to increase performance while keeping the data size small. - The 'first layer' of nodes is extracted out and replaced with a lookup table. This turns the search for the first character from O(n) to O (1), and doesn't increase the data size because all first characters in the set of named character references have the values 'a'-'z'/'A'-'Z', so a lookup array of exactly 52 elements can be used. The lookup table stores the cumulative "number" fields that would be calculated by a linear scan that matches a given node, thus allowing the unique index to be built-up as normal with a O(1) search instead of a linear scan. - The 'second layer' of nodes is also extracted out and searches of the second layer are done using a bit field of 52 bits (the set bits of the bit field depend on the first character's value), where each set bit corresponds to one of 'a'-'z'/'A'-'Z' (similar to the first layer, the second layer can only contain ASCII alphabetic characters). The bit field is then re-used (along with an offset) to get the index into the array of second layer nodes. This technique ultimately allows for storing the minimum number of nodes in the second layer, and therefore only increasing the size of the data by the size of the 'first to second layer link' info which is 52 * 8 = 416 bytes. - After the second layer, the rest of the data is stored using a mostly-normal DAFSA, but there are still a few differences: - The "number" field is cumulative, in the same way that the first/second layer store a cumulative "number" field. This cuts down slightly on the amount of work done during the search of a list of children, and we can get away with it because the cumulative "number" fields of the remaining nodes in the DAFSA (after the first and second layer nodes were extracted out) happens to require few enough bits that we can store the cumulative version while staying under our 32-bit budget. - Instead of storing a 'last sibling' flag to denote the end of a list of children, the length of each node's list of children is stored. Again, this is mostly done just because there are enough bits available to do so while keeping the DAFSA node within 32 bits. - Note: Together, these modifications open up the possibility of using a binary search instead of a linear search over the children, but due to the consistently small lengths of the lists of children in the remaining DAFSA, a linear search actually seems to be the better option. The new data size is 24,724 bytes, up from 24,412 bytes (+312, -104 from the 52 first layer nodes going from 4-bytes to 2-bytes, and +416 from the addition of the 'first to second layer link' data). In terms of raw matching speed (outside the context of the tokenizer), this provides about a 1.72x speedup. In very named-character-reference-heavy tokenizer benchmarks, this provides about a 1.05x speedup (the effect of named character reference matching speed is diluted when benchmarking the tokenizer). Additionally, fixes the size of the named character reference data when targeting Windows.
2025-07-09 00:05:28 -07:00
static u8 ascii_alphabetic_to_index(u8 c)
{
ASSERT(AK::is_ascii_alpha(c));
return c <= 'Z' ? (c - 'A') : (c - 'a' + 26);
}
LibWeb: Make named character references more spec-compliant & efficient There are two changes happening here: a correctness fix, and an optimization. In theory they are unrelated, but the optimization actually paves the way for the correctness fix. Before this commit, the HTML tokenizer would attempt to look for named character references by checking from after the `&` until the end of m_decoded_input, which meant that it was unable to recognize things like named character references that are inserted via `document.write` one byte at a time. For example, if `&notin;` was written one-byte-at-a-time with `document.write`, then the tokenizer would only check against `n` since that's all that would exist at the time of the check and therefore erroneously conclude that it was an invalid named character reference. This commit modifies the approach taken for named character reference matching by using a trie-like structure (specifically, a deterministic acyclic finite state automaton or DAFSA), which allows for efficiently matching one-character-at-a-time and therefore it is able to pick up matching where it left off after each code point is consumed. Note: Because it's possible for a partial match to not actually develop into a full match (e.g. `&notindo` which could lead to `&notindot;`), some backtracking is performed after-the-fact in order to only consume the code points within the longest match found (e.g. `&notindo` would backtrack back to `&not`). With this new approach, `document.write` being called one-byte-at-a-time is handled correctly, which allows for passing more WPT tests, with the most directly relevant tests being `/html/syntax/parsing/html5lib_entities01.html` and `/html/syntax/parsing/html5lib_entities02.html` when run with `?run_type=write_single`. Additionally, the implementation now better conforms to the language of the spec (and resolves a FIXME) because exactly the matched characters are consumed and nothing more, so SWITCH_TO is able to be used as the spec says instead of RECONSUME_IN. The new approach is also an optimization: - Instead of a linear search using `starts_with`, the usage of a DAFSA means that it is always aware of which characters can lead to a match at any given point, and will bail out whenever a match is no longer possible. - The DAFSA is able to take advantage of the note in the section `13.5 Named character references` that says "This list is static and will not be expanded or changed in the future." and tailor its Node struct accordingly to tightly pack each node's data into 32-bits. Together with the inherent DAFSA property of redundant node deduplication, the amount of data stored for named character reference matching is minimized. In my testing: - A benchmark tokenizing an arbitrary set of HTML test files was about 1.23x faster (2070ms to 1682ms). - A benchmark tokenizing a file with tens of thousands of named character references mixed in with truncated named character references and arbitrary ASCII characters/ampersands runs about 8x faster (758ms to 93ms). - The size of `liblagom-web.so` was reduced by 94.96KiB. Some technical details: A DAFSA (deterministic acyclic finite state automaton) is essentially a trie flattened into an array, but it also uses techniques to minimize redundant nodes. This provides fast lookups while minimizing the required data size, but normally does not allow for associating data related to each word. However, by adding a count of the number of possible words from each node, it becomes possible to also use it to achieve minimal perfect hashing for the set of words (which allows going from word -> unique index as well as unique index -> word). This allows us to store a second array of data so that the DAFSA can be used as a lookup for e.g. the associated code points. For the Swift implementation, the new NamedCharacterReferenceMatcher was used to satisfy the previous API and the tokenizer was left alone otherwise. In the future, the Swift implementation should be updated to use the same implementation for its NamedCharacterReference state as the updated C++ implementation.
2024-12-22 07:09:31 -08:00
bool NamedCharacterReferenceMatcher::try_consume_ascii_char(u8 c)
{
LibWeb/HTML: Improve data structure of named character reference data Introduces a few ad-hoc modifications to the DAFSA aimed to increase performance while keeping the data size small. - The 'first layer' of nodes is extracted out and replaced with a lookup table. This turns the search for the first character from O(n) to O (1), and doesn't increase the data size because all first characters in the set of named character references have the values 'a'-'z'/'A'-'Z', so a lookup array of exactly 52 elements can be used. The lookup table stores the cumulative "number" fields that would be calculated by a linear scan that matches a given node, thus allowing the unique index to be built-up as normal with a O(1) search instead of a linear scan. - The 'second layer' of nodes is also extracted out and searches of the second layer are done using a bit field of 52 bits (the set bits of the bit field depend on the first character's value), where each set bit corresponds to one of 'a'-'z'/'A'-'Z' (similar to the first layer, the second layer can only contain ASCII alphabetic characters). The bit field is then re-used (along with an offset) to get the index into the array of second layer nodes. This technique ultimately allows for storing the minimum number of nodes in the second layer, and therefore only increasing the size of the data by the size of the 'first to second layer link' info which is 52 * 8 = 416 bytes. - After the second layer, the rest of the data is stored using a mostly-normal DAFSA, but there are still a few differences: - The "number" field is cumulative, in the same way that the first/second layer store a cumulative "number" field. This cuts down slightly on the amount of work done during the search of a list of children, and we can get away with it because the cumulative "number" fields of the remaining nodes in the DAFSA (after the first and second layer nodes were extracted out) happens to require few enough bits that we can store the cumulative version while staying under our 32-bit budget. - Instead of storing a 'last sibling' flag to denote the end of a list of children, the length of each node's list of children is stored. Again, this is mostly done just because there are enough bits available to do so while keeping the DAFSA node within 32 bits. - Note: Together, these modifications open up the possibility of using a binary search instead of a linear search over the children, but due to the consistently small lengths of the lists of children in the remaining DAFSA, a linear search actually seems to be the better option. The new data size is 24,724 bytes, up from 24,412 bytes (+312, -104 from the 52 first layer nodes going from 4-bytes to 2-bytes, and +416 from the addition of the 'first to second layer link' data). In terms of raw matching speed (outside the context of the tokenizer), this provides about a 1.72x speedup. In very named-character-reference-heavy tokenizer benchmarks, this provides about a 1.05x speedup (the effect of named character reference matching speed is diluted when benchmarking the tokenizer). Additionally, fixes the size of the named character reference data when targeting Windows.
2025-07-09 00:05:28 -07:00
switch (m_search_state_tag) {
case NamedCharacterReferenceMatcher::SearchStateTag::Init: {
if (!AK::is_ascii_alpha(c))
return false;
auto index = ascii_alphabetic_to_index(c);
m_search_state_tag = NamedCharacterReferenceMatcher::SearchStateTag::FirstToSecondLayer;
m_search_state = { .first_to_second_layer = g_named_character_reference_first_to_second_layer[index] };
m_pending_unique_index = g_named_character_reference_first_layer[index].number;
m_overconsumed_code_points++;
return true;
}
case NamedCharacterReferenceMatcher::SearchStateTag::FirstToSecondLayer: {
if (!AK::is_ascii_alpha(c))
return false;
auto bit_index = ascii_alphabetic_to_index(c);
if (((1ull << bit_index) & m_search_state.first_to_second_layer.mask) == 0)
return false;
// Get the second layer node by re-using the first_to_second_layer.mask.
// For example, if the first character is 'n' and the second character is 'o':
//
// This is the first_to_second_layer.mask when the first character is 'n':
// 0001111110110110111111111100001000100000100001000000
// └ bit_index of 'o'
//
// Create a mask where all of the less significant bits than the
// bit index of the current character ('o') are set:
// 0000000000001111111111111111111111111111111111111111
// └ bit_index of 'o'
//
// Bitwise AND this new mask with the first_to_second_layer.mask
// to get only the set bits less significant than the bit index of the
// current character:
// 0000000000000110111111111100001000100000100001000000
//
// Take the popcount of this to get the index of the node within the
// second layer. In this case, there are 16 bits set, so the index
// of 'o' in the second layer is first_to_second_layer.second_layer_offset + 16.
u64 mask = (1ull << bit_index) - 1;
u8 char_index = AK::popcount(m_search_state.first_to_second_layer.mask & mask);
auto const& node = g_named_character_reference_second_layer[m_search_state.first_to_second_layer.second_layer_offset + char_index];
m_pending_unique_index += node.number;
m_overconsumed_code_points++;
if (node.end_of_word) {
m_pending_unique_index++;
m_last_matched_unique_index = m_pending_unique_index;
m_ends_with_semicolon = c == ';';
m_overconsumed_code_points = 0;
}
m_search_state_tag = NamedCharacterReferenceMatcher::SearchStateTag::DafsaChildren;
m_search_state = { .dafsa_children = { &g_named_character_reference_nodes[node.child_index], node.children_len } };
return true;
}
case NamedCharacterReferenceMatcher::SearchStateTag::DafsaChildren: {
for (auto const& node : m_search_state.dafsa_children) {
if (node.character == c) {
m_pending_unique_index += node.number;
m_overconsumed_code_points++;
if (node.end_of_word) {
m_pending_unique_index++;
m_last_matched_unique_index = m_pending_unique_index;
m_ends_with_semicolon = c == ';';
m_overconsumed_code_points = 0;
}
m_search_state = { .dafsa_children = { &g_named_character_reference_nodes[node.child_index], node.children_len } };
return true;
}
}
LibWeb: Make named character references more spec-compliant & efficient There are two changes happening here: a correctness fix, and an optimization. In theory they are unrelated, but the optimization actually paves the way for the correctness fix. Before this commit, the HTML tokenizer would attempt to look for named character references by checking from after the `&` until the end of m_decoded_input, which meant that it was unable to recognize things like named character references that are inserted via `document.write` one byte at a time. For example, if `&notin;` was written one-byte-at-a-time with `document.write`, then the tokenizer would only check against `n` since that's all that would exist at the time of the check and therefore erroneously conclude that it was an invalid named character reference. This commit modifies the approach taken for named character reference matching by using a trie-like structure (specifically, a deterministic acyclic finite state automaton or DAFSA), which allows for efficiently matching one-character-at-a-time and therefore it is able to pick up matching where it left off after each code point is consumed. Note: Because it's possible for a partial match to not actually develop into a full match (e.g. `&notindo` which could lead to `&notindot;`), some backtracking is performed after-the-fact in order to only consume the code points within the longest match found (e.g. `&notindo` would backtrack back to `&not`). With this new approach, `document.write` being called one-byte-at-a-time is handled correctly, which allows for passing more WPT tests, with the most directly relevant tests being `/html/syntax/parsing/html5lib_entities01.html` and `/html/syntax/parsing/html5lib_entities02.html` when run with `?run_type=write_single`. Additionally, the implementation now better conforms to the language of the spec (and resolves a FIXME) because exactly the matched characters are consumed and nothing more, so SWITCH_TO is able to be used as the spec says instead of RECONSUME_IN. The new approach is also an optimization: - Instead of a linear search using `starts_with`, the usage of a DAFSA means that it is always aware of which characters can lead to a match at any given point, and will bail out whenever a match is no longer possible. - The DAFSA is able to take advantage of the note in the section `13.5 Named character references` that says "This list is static and will not be expanded or changed in the future." and tailor its Node struct accordingly to tightly pack each node's data into 32-bits. Together with the inherent DAFSA property of redundant node deduplication, the amount of data stored for named character reference matching is minimized. In my testing: - A benchmark tokenizing an arbitrary set of HTML test files was about 1.23x faster (2070ms to 1682ms). - A benchmark tokenizing a file with tens of thousands of named character references mixed in with truncated named character references and arbitrary ASCII characters/ampersands runs about 8x faster (758ms to 93ms). - The size of `liblagom-web.so` was reduced by 94.96KiB. Some technical details: A DAFSA (deterministic acyclic finite state automaton) is essentially a trie flattened into an array, but it also uses techniques to minimize redundant nodes. This provides fast lookups while minimizing the required data size, but normally does not allow for associating data related to each word. However, by adding a count of the number of possible words from each node, it becomes possible to also use it to achieve minimal perfect hashing for the set of words (which allows going from word -> unique index as well as unique index -> word). This allows us to store a second array of data so that the DAFSA can be used as a lookup for e.g. the associated code points. For the Swift implementation, the new NamedCharacterReferenceMatcher was used to satisfy the previous API and the tokenizer was left alone otherwise. In the future, the Swift implementation should be updated to use the same implementation for its NamedCharacterReference state as the updated C++ implementation.
2024-12-22 07:09:31 -08:00
return false;
}
LibWeb/HTML: Improve data structure of named character reference data Introduces a few ad-hoc modifications to the DAFSA aimed to increase performance while keeping the data size small. - The 'first layer' of nodes is extracted out and replaced with a lookup table. This turns the search for the first character from O(n) to O (1), and doesn't increase the data size because all first characters in the set of named character references have the values 'a'-'z'/'A'-'Z', so a lookup array of exactly 52 elements can be used. The lookup table stores the cumulative "number" fields that would be calculated by a linear scan that matches a given node, thus allowing the unique index to be built-up as normal with a O(1) search instead of a linear scan. - The 'second layer' of nodes is also extracted out and searches of the second layer are done using a bit field of 52 bits (the set bits of the bit field depend on the first character's value), where each set bit corresponds to one of 'a'-'z'/'A'-'Z' (similar to the first layer, the second layer can only contain ASCII alphabetic characters). The bit field is then re-used (along with an offset) to get the index into the array of second layer nodes. This technique ultimately allows for storing the minimum number of nodes in the second layer, and therefore only increasing the size of the data by the size of the 'first to second layer link' info which is 52 * 8 = 416 bytes. - After the second layer, the rest of the data is stored using a mostly-normal DAFSA, but there are still a few differences: - The "number" field is cumulative, in the same way that the first/second layer store a cumulative "number" field. This cuts down slightly on the amount of work done during the search of a list of children, and we can get away with it because the cumulative "number" fields of the remaining nodes in the DAFSA (after the first and second layer nodes were extracted out) happens to require few enough bits that we can store the cumulative version while staying under our 32-bit budget. - Instead of storing a 'last sibling' flag to denote the end of a list of children, the length of each node's list of children is stored. Again, this is mostly done just because there are enough bits available to do so while keeping the DAFSA node within 32 bits. - Note: Together, these modifications open up the possibility of using a binary search instead of a linear search over the children, but due to the consistently small lengths of the lists of children in the remaining DAFSA, a linear search actually seems to be the better option. The new data size is 24,724 bytes, up from 24,412 bytes (+312, -104 from the 52 first layer nodes going from 4-bytes to 2-bytes, and +416 from the addition of the 'first to second layer link' data). In terms of raw matching speed (outside the context of the tokenizer), this provides about a 1.72x speedup. In very named-character-reference-heavy tokenizer benchmarks, this provides about a 1.05x speedup (the effect of named character reference matching speed is diluted when benchmarking the tokenizer). Additionally, fixes the size of the named character reference data when targeting Windows.
2025-07-09 00:05:28 -07:00
default:
VERIFY_NOT_REACHED();
}
}
}