Store CSS token payloads in a variant so each token only carries the
state needed by its type. Keep delimiter, number, hash, string, and
dimension data separate instead of storing every possible payload on
every token.
Use a smaller component-value token for function and block boundary
metadata. These component values only need token type, original source
text, and source positions, so avoid embedding full token payload
storage inside every Function and SimpleBlock.
Shrink CSS source positions to explicit 32-bit counters. Guard the C++
and Rust tokenizer paths against overflow. Add size assertions for the
hot Token and ComponentValue types so future growth is intentional.
Now that Badge can have multiple types, and a Badge of a derived class
can convert into a Badge of the superclass, we can simplify a few method
signatures and overloads.
test-css-tokenizer is updated to run both the C++ and Rust tokenizers
and compare their output, to ensure they behave identically. The Parser
still uses the C++ Tokenizer.
The LibWeb crate, FFI layer etc are all based on the existing ones for
other libraries.
This is a direct AI translation to get us started, and not idiomatic
Rust. Future work can be done to make it more sensible.
This brings us in line with the other numeric types (percentage and
dimension) and allows us to test the clamping behavior that will be
added in a future commit.
Add `create_foo()` static methods for the missing Token::Types, and use
them in the Tokenizer. This means we slightly deviate from the spec now:
it says "create foo token... set its bar to 32", but we now just wait
and construct the Token fully-formed. But those cases are short so it
should still be clear what we're doing.
This makes it possible to construct all kinds of Token elsewhere, such
as for testing purposes.
