mirror of
https://github.com/LadybirdBrowser/ladybird.git
synced 2025-12-08 06:09:58 +00:00
00182a2405
This ports the lexer to UTF-16 and deals with the immediate fallout up to the AST. The AST will be dealt with in upcoming commits. The lexer will still accept UTF-8 strings as input, and will transcode them to UTF-16 for lexing. This doesn't actually incur a new allocation, as we were already converting the input StringView to a ByteString for each lexer. One immediate logical benefit here is that we do not need to know off- hand how many UTF-8 bytes some special code points occupy. They all happen to be a single UTF-16 code unit. So instead of advancing the lexer by 3 positions in some cases, we can just always advance by 1. |
||
|---|---|---|
| .. | ||
| abench.cpp | ||
| CMakeLists.txt | ||
| dns.cpp | ||
| image.cpp | ||
| js.cpp | ||
| test262-runner.cpp | ||
| wasm.cpp | ||
| xml.cpp | ||