Stowage/ladybird
2
0
Fork 0
mirror of https://github.com/LadybirdBrowser/ladybird.git synced 2025-10-31 21:30:58 +00:00
Code Issues Projects Releases Packages Wiki Activity
ladybird/Libraries/LibJS/Parser.h

332 lines
13 KiB
C
Raw Normal View History

LibJS: Add Javascript lexer and parser This adds a basic Javascript lexer and parser. It can parse the currently existing demo programs. More work needs to be done to turn it into a complete parser than can parse arbitrary JS Code. The lexer outputs tokens with preceeding whitespace and comments in the trivia member. This should allow us to generate the exact source code by concatenating the generated tokens. The parser is written in a way that it always returns a complete syntax tree. Error conditions are represented as nodes in the tree. This simplifies the code and allows it to be used as an early stage parser, e.g for parsing JS documents in an IDE while editing the source code.:
2020-03-11 19:27:43 +01:00
/*
Everywhere: Use s.unverwerth@serenityos.org :^)
2021-05-29 12:38:28 +02:00
* Copyright (c) 2020, Stephan Unverwerth <s.unverwerth@serenityos.org>
LibJS: Make parsing import and export entries follow the spec The big changes are: - Allow strings as Module{Export, Import}Name - Properly track declarations in default export statements However, the spec is a little strange in that it allows function and class declarations without a name in default export statements. This is quite hard to fully implement without rewriting more of the parser so for now this behavior is emulated by faking things with function and class expressions. See the comments in parse_export_statement for details on the hacks and where it goes wrong.
2022-01-16 23:51:28 +01:00
* Copyright (c) 2021-2022, David Tuin <davidot@serenityos.org>
Meta: Update my e-mail address everywhere
2024-10-04 13:19:50 +02:00
* Copyright (c) 2023, Andreas Kling <andreas@ladybird.org>
LibJS: Add Javascript lexer and parser This adds a basic Javascript lexer and parser. It can parse the currently existing demo programs. More work needs to be done to turn it into a complete parser than can parse arbitrary JS Code. The lexer outputs tokens with preceeding whitespace and comments in the trivia member. This should allow us to generate the exact source code by concatenating the generated tokens. The parser is written in a way that it always returns a complete syntax tree. Error conditions are represented as nodes in the tree. This simplifies the code and allows it to be used as an early stage parser, e.g for parsing JS documents in an IDE while editing the source code.:
2020-03-11 19:27:43 +01:00
*
Everything: Move to SPDX license identifiers in all files. SPDX License Identifiers are a more compact / standardized way of representing file license information. See: https://spdx.dev/resources/use/#identifiers This was done with the `ambr` search and replace tool. ambr --no-parent-ignore --key-from-file --rep-from-file key.txt rep.txt *
2021-04-22 01:24:48 -07:00
* SPDX-License-Identifier: BSD-2-Clause
LibJS: Add Javascript lexer and parser This adds a basic Javascript lexer and parser. It can parse the currently existing demo programs. More work needs to be done to turn it into a complete parser than can parse arbitrary JS Code. The lexer outputs tokens with preceeding whitespace and comments in the trivia member. This should allow us to generate the exact source code by concatenating the generated tokens. The parser is written in a way that it always returns a complete syntax tree. Error conditions are represented as nodes in the tree. This simplifies the code and allows it to be used as an early stage parser, e.g for parsing JS documents in an IDE while editing the source code.:
2020-03-11 19:27:43 +01:00
*/
#pragma once
LibJS: Fix mixing of logical and coalescing operators The same expression is not allowed to contain both the logical && and || operators, and the coalescing ?? operator. This patch changes how "forbidden" tokens are handled, using a finite set instead of an Vector. This supports much more efficient merging of the forbidden tokens when propagating forward, and allowing the return of forbidden tokens to parent contexts.
2022-02-15 22:34:59 -08:00
#include <AK/Assertions.h>
LibJS: break or continue with nonexistent label is a syntax error
2020-10-08 10:49:08 -07:00
#include <AK/HashTable.h>
LibJS: Make the AST reference-counted This allows function objects to outlive the original parsed program without their ScopeNode disappearing.
2020-03-18 11:23:53 +01:00
#include <AK/NonnullRefPtr.h>
LibJS: Add strict mode Adds the ability for a scope (either a function or the entire program) to be in strict mode. Scopes default to non-strict mode. There are two ways to determine the strict-ness of the JS engine: 1. In the parser, this can be accessed with the parser_state variable m_is_strict_mode boolean. If true, the Parser is currently parsing in strict mode. This is done so that the Parser can generate syntax errors at parse time, which is required in some cases. 2. With Interpreter.is_strict_mode(). This allows strict mode checking at runtime as opposed to compile time. Additionally, in order to test this, a global isStrictMode() function has been added to the JS ReplObject under the test-mode flag.
2020-05-27 22:22:08 -07:00
#include <LibJS/AST.h>
LibJS: Enable EXPLICIT_SYMBOL_EXPORT and annotate minimum symbol set
2025-07-19 13:49:30 -07:00
#include <LibJS/Export.h>
LibJS: Add strict mode Adds the ability for a scope (either a function or the entire program) to be in strict mode. Scopes default to non-strict mode. There are two ways to determine the strict-ness of the JS engine: 1. In the parser, this can be accessed with the parser_state variable m_is_strict_mode boolean. If true, the Parser is currently parsing in strict mode. This is done so that the Parser can generate syntax errors at parse time, which is required in some cases. 2. With Interpreter.is_strict_mode(). This allows strict mode checking at runtime as opposed to compile time. Additionally, in order to test this, a global isStrictMode() function has been added to the JS ReplObject under the test-mode flag.
2020-05-27 22:22:08 -07:00
#include <LibJS/Lexer.h>
LibJS: Make Parser::Error a standalone ParserError class This allows us to forward declare it and reduce the number of things that need to include Parser.h.
2022-11-23 12:39:23 +01:00
#include <LibJS/ParserError.h>
LibJS: Track source positions all the way down to exceptions This makes exceptions have a trace of source positions too, which could probably be helpful in making fancier error tracebacks.
2020-12-28 20:45:22 +03:30
#include <LibJS/SourceRange.h>
LibJS: Fix mixing of logical and coalescing operators The same expression is not allowed to contain both the logical && and || operators, and the coalescing ?? operator. This patch changes how "forbidden" tokens are handled, using a finite set instead of an Vector. This supports much more efficient merging of the forbidden tokens when propagating forward, and allowing the return of forbidden tokens to parent contexts.
2022-02-15 22:34:59 -08:00
#include <LibJS/Token.h>
#include <initializer_list>
LibJS: Add Javascript lexer and parser This adds a basic Javascript lexer and parser. It can parse the currently existing demo programs. More work needs to be done to turn it into a complete parser than can parse arbitrary JS Code. The lexer outputs tokens with preceeding whitespace and comments in the trivia member. This should allow us to generate the exact source code by concatenating the generated tokens. The parser is written in a way that it always returns a complete syntax tree. Error conditions are represented as nodes in the tree. This simplifies the code and allows it to be used as an early stage parser, e.g for parsing JS documents in an IDE while editing the source code.:
2020-03-11 19:27:43 +01:00
namespace JS {
LibJS: Add operator precedence parsing Obey precedence and associativity rules when parsing expressions with chained operators.
2020-03-12 23:02:41 +01:00
enum class Associativity {
Left,
Right
};
LibJS: Refactor parse_function_node() bool parameters into bit flags I'm about to add even more options and a bunch of unnamed true/false arguments is really not helpful. Let's make this a single parse options parameter using bit flags.
2020-10-20 17:56:49 +01:00
struct FunctionNodeParseOptions {
LibJS: Allow anonymous functions as default exports This requires a special case with names as the default function is supposed to have a unique name ("*default*" in our case) but when checked should have name "default".
2022-09-02 00:46:37 +02:00
enum : u16 {
LibJS: Refactor parse_function_node() bool parameters into bit flags I'm about to add even more options and a bunch of unnamed true/false arguments is really not helpful. Let's make this a single parse options parameter using bit flags.
2020-10-20 17:56:49 +01:00
CheckForFunctionAndName = 1 << 0,
AllowSuperPropertyLookup = 1 << 1,
AllowSuperConstructorCall = 1 << 2,
LibJS: Move checks for invalid getter/setter params to parse_function_node This allows us to provide better error messages as we can point the syntax error location to the exact first invalid parameter instead of always the end of the function within a object literal or class definition. Before this change: const Foo = { set bar() {} } ^ Uncaught exception: [SyntaxError]: Object setter property must have one argument (line: 1, column: 28) class Foo { set bar() {} } ^ Uncaught exception: [SyntaxError]: Class setter method must have one argument (line: 1, column: 26) After this change: const Foo = { set bar() {} } ^ Uncaught exception: [SyntaxError]: Setter function must have one argument (line: 1, column: 23) class Foo { set bar() {} } ^ Uncaught exception: [SyntaxError]: Setter function must have one argument (line: 1, column: 21) The only possible downside of this change is that class getters/setters and functions in objects are not distinguished in the message anymore - I don't think that's important though, and classes are (mostly) just syntactic sugar anyway.
2020-10-20 18:32:51 +01:00
IsGetterFunction = 1 << 3,
IsSetterFunction = 1 << 4,
LibJS: Implement rules for duplicate function parameters - A regular function can have duplicate parameters except in strict mode or if its parameter list is not "simple" (has a default or rest parameter) - An arrow function can never have duplicate parameters Compared to other engines I opted for more useful syntax error messages than a generic "duplicate parameter name not allowed in this context": "use strict"; function test(foo, foo) {} ^ Uncaught exception: [SyntaxError]: Duplicate parameter 'foo' not allowed in strict mode (line: 1, column: 34) function test(foo, foo = 1) {} ^ Uncaught exception: [SyntaxError]: Duplicate parameter 'foo' not allowed in function with default parameter (line: 1, column: 20) function test(foo, ...foo) {} ^ Uncaught exception: [SyntaxError]: Duplicate parameter 'foo' not allowed in function with rest parameter (line: 1, column: 23) (foo, foo) => {} ^ Uncaught exception: [SyntaxError]: Duplicate parameter 'foo' not allowed in arrow function (line: 1, column: 7)
2020-10-25 11:14:04 +00:00
IsArrowFunction = 1 << 5,
LibJS: Parse generator functions in object literals Also add some parser tests
2021-06-14 14:52:59 +04:30
IsGeneratorFunction = 1 << 6,
LibJS: Add support for async functions This commit adds support for the most bare bones version of async functions, support for async generator functions, async arrow functions and await expressions are TODO.
2021-11-09 20:39:22 +02:00
IsAsyncFunction = 1 << 7,
LibJS: Allow anonymous functions as default exports This requires a special case with names as the default function is supposed to have a unique name ("*default*" in our case) but when checked should have name "default".
2022-09-02 00:46:37 +02:00
HasDefaultExportName = 1 << 8,
LibJS: Make JS parser emit accurate `this` insights for constructors This way we don't have to handle it when instantiating the constructor.
2025-04-07 21:04:45 +02:00
IsConstructor = 1 << 9,
LibJS: Refactor parse_function_node() bool parameters into bit flags I'm about to add even more options and a bunch of unnamed true/false arguments is really not helpful. Let's make this a single parse options parameter using bit flags.
2020-10-20 17:56:49 +01:00
};
};
LibJS: Make scoping follow the spec Before this we used an ad-hoc combination of references and 'variables' stored in a hashmap. This worked in most cases but is not spec like. Additionally hoisting, dynamically naming functions and scope analysis was not done properly. This patch fixes all of that by: - Implement BindingInitialization for destructuring assignment. - Implementing a new ScopePusher which tracks the lexical and var scoped declarations. This hoists functions to the top level if no lexical declaration name overlaps. Furthermore we do checking of redeclarations in the ScopePusher now requiring less checks all over the place. - Add methods for parsing the directives and statement lists instead of having that code duplicated in multiple places. This allows declarations to pushed to the appropriate scope more easily. - Remove the non spec way of storing 'variables' in DeclarativeEnvironment and make Reference follow the spec instead of checking both the bindings and 'variables'. - Remove all scoping related things from the Interpreter. And instead use environments as specified by the spec. This also includes fixing that NativeFunctions did not produce a valid FunctionEnvironment which could cause issues with callbacks and eval. All FunctionObjects now have a valid NewFunctionEnvironment implementation. - Remove execute_statements from Interpreter and instead use ASTNode::execute everywhere this simplifies AST.cpp as you no longer need to worry about which method to call. - Make ScopeNodes setup their own environment. This uses four different methods specified by the spec {Block, Function, Eval, Global}DeclarationInstantiation with the annexB extensions. - Implement and use NamedEvaluation where specified. Additionally there are fixes to things exposed by these changes to eval, {for, for-in, for-of} loops and assignment. Finally it also fixes some tests in test-js which where passing before but not now that we have correct behavior :^).
2021-09-22 12:44:56 +02:00
class ScopePusher;
LibJS: Enable EXPLICIT_SYMBOL_EXPORT and annotate minimum symbol set
2025-07-19 13:49:30 -07:00
class JS_API Parser {
LibJS: Add Javascript lexer and parser This adds a basic Javascript lexer and parser. It can parse the currently existing demo programs. More work needs to be done to turn it into a complete parser than can parse arbitrary JS Code. The lexer outputs tokens with preceeding whitespace and comments in the trivia member. This should allow us to generate the exact source code by concatenating the generated tokens. The parser is written in a way that it always returns a complete syntax tree. Error conditions are represented as nodes in the tree. This simplifies the code and allows it to be used as an early stage parser, e.g for parsing JS documents in an IDE while editing the source code.:
2020-03-11 19:27:43 +01:00
public:
LibJS: Add missing steps and spec comments to PerformEval While adding spec comments to PerformEval, I noticed we were missing multiple steps. Namely, these were: - Checking if the host will allow us to compile the string (allowing LibWeb to perform CSP for eval) - The parser's initial state depending on the environment around us on direct eval: - Allowing new.target via eval in functions - Allowing super calls and super properties via eval in classes - Disallowing the use of the arguments object in class field initializers at eval's parse time - Setting ScriptOrModule of eval's execution context The spec allows us to apply the additional parsing steps in any order. The method I have gone with is passing in a struct to the parser's constructor, which overrides the parser's initial state to (dis)allow the things stated above from the get-go.
2022-04-10 00:55:45 +01:00
struct EvalInitialState {
bool in_eval_function_context { false };
bool allow_super_property_lookup { false };
bool allow_super_constructor_call { false };
bool in_class_field_initializer { false };
};
explicit Parser(Lexer lexer, Program::Type program_type = Program::Type::Script, Optional<EvalInitialState> initial_state_for_eval = {});
LibJS: Add Javascript lexer and parser This adds a basic Javascript lexer and parser. It can parse the currently existing demo programs. More work needs to be done to turn it into a complete parser than can parse arbitrary JS Code. The lexer outputs tokens with preceeding whitespace and comments in the trivia member. This should allow us to generate the exact source code by concatenating the generated tokens. The parser is written in a way that it always returns a complete syntax tree. Error conditions are represented as nodes in the tree. This simplifies the code and allows it to be used as an early stage parser, e.g for parsing JS documents in an IDE while editing the source code.:
2020-03-11 19:27:43 +01:00
LibJS: Correct behaviour of direct vs. indirect eval eval only has direct access to the local scope when accessed through the name eval. This includes locals named eval, because of course it does.
2021-06-19 20:13:53 -07:00
NonnullRefPtr<Program> parse_program(bool starts_in_strict_mode = false);
LibJS: Add Javascript lexer and parser This adds a basic Javascript lexer and parser. It can parse the currently existing demo programs. More work needs to be done to turn it into a complete parser than can parse arbitrary JS Code. The lexer outputs tokens with preceeding whitespace and comments in the trivia member. This should allow us to generate the exact source code by concatenating the generated tokens. The parser is written in a way that it always returns a complete syntax tree. Error conditions are represented as nodes in the tree. This simplifies the code and allows it to be used as an early stage parser, e.g for parsing JS documents in an IDE while editing the source code.:
2020-03-11 19:27:43 +01:00
LibJS: Parse FunctionExpressions FunctionExpression is mostly like FunctionDeclaration, except the name is optional. Share the parsing logic in parse_function_node<NodeType>. This allows us to do nice things like: document.addEventListener("DOMContentLoaded", function() { alert("Hello friends!"); });
2020-03-19 11:52:56 +01:00
template<typename FunctionNodeType>
LibJS: Allow anonymous functions as default exports This requires a special case with names as the default function is supposed to have a unique name ("*default*" in our case) but when checked should have name "default".
2022-09-02 00:46:37 +02:00
NonnullRefPtr<FunctionNodeType> parse_function_node(u16 parse_options = FunctionNodeParseOptions::CheckForFunctionAndName, Optional<Position> const& function_start = {});
LibJS: Create FunctionParameters earlier in the parser This avoids making multiple copies of the Vector<FunctionParameter> in the parser.
2025-03-27 14:28:35 +00:00
NonnullRefPtr<FunctionParameters const> parse_formal_parameters(int& function_length, u16 parse_options = 0);
LibJS: Allow member expressions in binding patterns Also allows literal string and numbers as property names in object binding patterns.
2021-09-18 01:11:32 +02:00
enum class AllowDuplicates {
Yes,
No
};
enum class AllowMemberExpressions {
Yes,
No
};
LibJS: Make RefPtr and NonnullRefPtr usage const-correct This mainly affected the AST, which is now const throughout.
2023-02-19 22:07:52 +01:00
RefPtr<BindingPattern const> parse_binding_pattern(AllowDuplicates is_var_declaration = AllowDuplicates::No, AllowMemberExpressions allow_member_expressions = AllowMemberExpressions::No);
LibJS: Parse FunctionExpressions FunctionExpression is mostly like FunctionDeclaration, except the name is optional. Share the parsing logic in parse_function_node<NodeType>. This allows us to do nice things like: document.addEventListener("DOMContentLoaded", function() { alert("Hello friends!"); });
2020-03-19 11:52:56 +01:00
LibJS: Correctly parse yield-from expressions This commit implements parsing for `yield *expr`, and the multiple ways something can or can't be parsed like that. Also makes yield-from a TODO in the bytecode generator. Behold, the glory of javascript syntax: ```js // 'yield' = expression in generators. function* foo() { yield *bar; // <- Syntax error here, expression can't start with * } // 'yield' = identifier anywhere else. function foo() { yield *bar; // Perfectly fine, this is just `yield * bar` } ```
2021-06-14 15:46:41 +04:30
struct PrimaryExpressionParseResult {
LibJS: Make RefPtr and NonnullRefPtr usage const-correct This mainly affected the AST, which is now const throughout.
2023-02-19 22:07:52 +01:00
NonnullRefPtr<Expression const> result;
LibJS: Correctly parse yield-from expressions This commit implements parsing for `yield *expr`, and the multiple ways something can or can't be parsed like that. Also makes yield-from a TODO in the bytecode generator. Behold, the glory of javascript syntax: ```js // 'yield' = expression in generators. function* foo() { yield *bar; // <- Syntax error here, expression can't start with * } // 'yield' = identifier anywhere else. function foo() { yield *bar; // Perfectly fine, this is just `yield * bar` } ```
2021-06-14 15:46:41 +04:30
bool should_continue_parsing_as_expression { true };
};
LibJS: Make RefPtr and NonnullRefPtr usage const-correct This mainly affected the AST, which is now const throughout.
2023-02-19 22:07:52 +01:00
NonnullRefPtr<Declaration const> parse_declaration();
LibJS: Allow labelled functions in certain contexts Also disallow duplicated labels.
2021-07-25 01:01:22 +02:00
enum class AllowLabelledFunction {
No,
Yes
};
LibJS: Make RefPtr and NonnullRefPtr usage const-correct This mainly affected the AST, which is now const throughout.
2023-02-19 22:07:52 +01:00
NonnullRefPtr<Statement const> parse_statement(AllowLabelledFunction allow_labelled_function = AllowLabelledFunction::No);
NonnullRefPtr<BlockStatement const> parse_block_statement();
LibJS: Create FunctionParameters earlier in the parser This avoids making multiple copies of the Vector<FunctionParameter> in the parser.
2025-03-27 14:28:35 +00:00
NonnullRefPtr<FunctionBody const> parse_function_body(NonnullRefPtr<FunctionParameters const>, FunctionKind function_kind, FunctionParsingInsights&);
LibJS: Make RefPtr and NonnullRefPtr usage const-correct This mainly affected the AST, which is now const throughout.
2023-02-19 22:07:52 +01:00
NonnullRefPtr<ReturnStatement const> parse_return_statement();
LibJS: Add using declaration support, RAII like operation in js In this patch only top level and not the more complicated for loop using statements are supported. Also, as noted in the latest meeting of tc39 async parts of the spec are not stage 3 thus not included.
2022-12-20 22:09:57 +01:00
enum class IsForLoopVariableDeclaration {
No,
Yes
};
LibJS: Make RefPtr and NonnullRefPtr usage const-correct This mainly affected the AST, which is now const throughout.
2023-02-19 22:07:52 +01:00
NonnullRefPtr<VariableDeclaration const> parse_variable_declaration(IsForLoopVariableDeclaration is_for_loop_variable_declaration = IsForLoopVariableDeclaration::No);
LibJS: Allow `let` and `const` to create locals in global code This is actually safe everywhere but in the topmost program scope. For web compatibility reasons, we have to flush all top-level bindings to the environment, in case a subsequent separate <script> program comes looking for them.
2024-06-13 21:31:31 +02:00
[[nodiscard]] RefPtr<Identifier const> parse_lexical_binding(Optional<DeclarationKind> = {});
LibJS: Make RefPtr and NonnullRefPtr usage const-correct This mainly affected the AST, which is now const throughout.
2023-02-19 22:07:52 +01:00
NonnullRefPtr<UsingDeclaration const> parse_using_declaration(IsForLoopVariableDeclaration is_for_loop_variable_declaration = IsForLoopVariableDeclaration::No);
NonnullRefPtr<Statement const> parse_for_statement();
LibJS: Implement parsing and executing for-await-of loops
2021-11-23 16:09:28 +01:00
enum class IsForAwaitLoop {
No,
Yes
};
LibJS: Fix mixing of logical and coalescing operators The same expression is not allowed to contain both the logical && and || operators, and the coalescing ?? operator. This patch changes how "forbidden" tokens are handled, using a finite set instead of an Vector. This supports much more efficient merging of the forbidden tokens when propagating forward, and allowing the return of forbidden tokens to parent contexts.
2022-02-15 22:34:59 -08:00
struct ForbiddenTokens {
ForbiddenTokens(std::initializer_list<TokenType> const& forbidden);
ForbiddenTokens merge(ForbiddenTokens other) const;
bool allows(TokenType token) const;
ForbiddenTokens forbid(std::initializer_list<TokenType> const& forbidden) const;
private:
void forbid_tokens(std::initializer_list<TokenType> const& forbidden);
bool m_forbid_in_token : 1 { false };
bool m_forbid_logical_tokens : 1 { false };
bool m_forbid_coalesce_token : 1 { false };
bool m_forbid_paren_open : 1 { false };
bool m_forbid_question_mark_period : 1 { false };
bool m_forbid_equals : 1 { false };
};
struct ExpressionResult {
LibJS: Make RefPtr and NonnullRefPtr usage const-correct This mainly affected the AST, which is now const throughout.
2023-02-19 22:07:52 +01:00
template<typename T>
ExpressionResult(NonnullRefPtr<T const> expression, ForbiddenTokens forbidden = {})
: expression(move(expression))
, forbidden(forbidden)
{
}
LibJS: Fix mixing of logical and coalescing operators The same expression is not allowed to contain both the logical && and || operators, and the coalescing ?? operator. This patch changes how "forbidden" tokens are handled, using a finite set instead of an Vector. This supports much more efficient merging of the forbidden tokens when propagating forward, and allowing the return of forbidden tokens to parent contexts.
2022-02-15 22:34:59 -08:00
template<typename T>
ExpressionResult(NonnullRefPtr<T> expression, ForbiddenTokens forbidden = {})
LibJS: Make RefPtr and NonnullRefPtr usage const-correct This mainly affected the AST, which is now const throughout.
2023-02-19 22:07:52 +01:00
: expression(move(expression))
LibJS: Fix mixing of logical and coalescing operators The same expression is not allowed to contain both the logical && and || operators, and the coalescing ?? operator. This patch changes how "forbidden" tokens are handled, using a finite set instead of an Vector. This supports much more efficient merging of the forbidden tokens when propagating forward, and allowing the return of forbidden tokens to parent contexts.
2022-02-15 22:34:59 -08:00
, forbidden(forbidden)
{
}
LibJS: Make RefPtr and NonnullRefPtr usage const-correct This mainly affected the AST, which is now const throughout.
2023-02-19 22:07:52 +01:00
NonnullRefPtr<Expression const> expression;
LibJS: Fix mixing of logical and coalescing operators The same expression is not allowed to contain both the logical && and || operators, and the coalescing ?? operator. This patch changes how "forbidden" tokens are handled, using a finite set instead of an Vector. This supports much more efficient merging of the forbidden tokens when propagating forward, and allowing the return of forbidden tokens to parent contexts.
2022-02-15 22:34:59 -08:00
ForbiddenTokens forbidden;
};
LibJS: Make RefPtr and NonnullRefPtr usage const-correct This mainly affected the AST, which is now const throughout.
2023-02-19 22:07:52 +01:00
NonnullRefPtr<Statement const> parse_for_in_of_statement(NonnullRefPtr<ASTNode const> lhs, IsForAwaitLoop is_await);
NonnullRefPtr<IfStatement const> parse_if_statement();
NonnullRefPtr<ThrowStatement const> parse_throw_statement();
NonnullRefPtr<TryStatement const> parse_try_statement();
NonnullRefPtr<CatchClause const> parse_catch_clause();
NonnullRefPtr<SwitchStatement const> parse_switch_statement();
NonnullRefPtr<SwitchCase const> parse_switch_case();
NonnullRefPtr<BreakStatement const> parse_break_statement();
NonnullRefPtr<ContinueStatement const> parse_continue_statement();
NonnullRefPtr<DoWhileStatement const> parse_do_while_statement();
NonnullRefPtr<WhileStatement const> parse_while_statement();
NonnullRefPtr<WithStatement const> parse_with_statement();
NonnullRefPtr<DebuggerStatement const> parse_debugger_statement();
NonnullRefPtr<ConditionalExpression const> parse_conditional_expression(NonnullRefPtr<Expression const> test, ForbiddenTokens);
NonnullRefPtr<OptionalChain const> parse_optional_chain(NonnullRefPtr<Expression const> base);
NonnullRefPtr<Expression const> parse_expression(int min_precedence, Associativity associate = Associativity::Right, ForbiddenTokens forbidden = {});
LibJS: Correctly parse yield-from expressions This commit implements parsing for `yield *expr`, and the multiple ways something can or can't be parsed like that. Also makes yield-from a TODO in the bytecode generator. Behold, the glory of javascript syntax: ```js // 'yield' = expression in generators. function* foo() { yield *bar; // <- Syntax error here, expression can't start with * } // 'yield' = identifier anywhere else. function foo() { yield *bar; // Perfectly fine, this is just `yield * bar` } ```
2021-06-14 15:46:41 +04:30
PrimaryExpressionParseResult parse_primary_expression();
LibJS: Make RefPtr and NonnullRefPtr usage const-correct This mainly affected the AST, which is now const throughout.
2023-02-19 22:07:52 +01:00
NonnullRefPtr<Expression const> parse_unary_prefixed_expression();
NonnullRefPtr<RegExpLiteral const> parse_regexp_literal();
NonnullRefPtr<ObjectExpression const> parse_object_expression();
NonnullRefPtr<ArrayExpression const> parse_array_expression();
LibJS: Allow invalid string in tagged template literals Since tagged template literals can inspect the raw string it is not a syntax error to have invalid escapes. However the cooked value should be `undefined`. We accomplish this by tracking whether parse_string_literal fails and then using a NullLiteral (since UndefinedLiteral is not a thing) and finally converting null in tagged template execution to undefined.
2022-08-17 02:04:27 +02:00
enum class StringLiteralType {
Normal,
NonTaggedTemplate,
TaggedTemplate
};
LibJS: Make RefPtr and NonnullRefPtr usage const-correct This mainly affected the AST, which is now const throughout.
2023-02-19 22:07:52 +01:00
NonnullRefPtr<StringLiteral const> parse_string_literal(Token const& token, StringLiteralType string_literal_type = StringLiteralType::Normal, bool* contains_invalid_escape = nullptr);
NonnullRefPtr<TemplateLiteral const> parse_template_literal(bool is_tagged);
ExpressionResult parse_secondary_expression(NonnullRefPtr<Expression const>, int min_precedence, Associativity associate = Associativity::Right, ForbiddenTokens forbidden = {});
NonnullRefPtr<Expression const> parse_call_expression(NonnullRefPtr<Expression const>);
NonnullRefPtr<NewExpression const> parse_new_expression();
NonnullRefPtr<ClassDeclaration const> parse_class_declaration();
NonnullRefPtr<ClassExpression const> parse_class_expression(bool expect_class_name);
NonnullRefPtr<YieldExpression const> parse_yield_expression();
NonnullRefPtr<AwaitExpression const> parse_await_expression();
NonnullRefPtr<Expression const> parse_property_key();
NonnullRefPtr<AssignmentExpression const> parse_assignment_expression(AssignmentOp, NonnullRefPtr<Expression const> lhs, int min_precedence, Associativity, ForbiddenTokens forbidden = {});
NonnullRefPtr<Identifier const> parse_identifier();
NonnullRefPtr<ImportStatement const> parse_import_statement(Program& program);
NonnullRefPtr<ExportStatement const> parse_export_statement(Program& program);
RefPtr<FunctionExpression const> try_parse_arrow_function_expression(bool expect_parens, bool is_async = false);
RefPtr<LabelledStatement const> try_parse_labelled_statement(AllowLabelledFunction allow_function);
RefPtr<MetaProperty const> try_parse_new_target_expression();
RefPtr<MetaProperty const> try_parse_import_meta_expression();
NonnullRefPtr<ImportCall const> parse_import_call();
LibJS: Implement 'new.target' This adds a new MetaProperty AST node which will be used for 'new.target' and 'import.meta' meta properties. The parser now distinguishes between "in function context" and "in arrow function context" (which is required for this). When encountering TokenType::New we will attempt to parse it as meta property and resort to regular new expression parsing if that fails, much like the parsing of labelled statements.
2020-11-02 21:27:42 +00:00
LibJS: Implement parsing and execution of optional chains
2021-09-14 06:56:31 +04:30
Vector<CallExpression::Argument> parse_arguments();
LibJS: Rename Parser::m_parser_state => m_state Also remove the "m_" prefix from all the public data members.
2021-06-19 14:43:09 +02:00
bool has_errors() const { return m_state.errors.size(); }
LibJS: Make Parser::Error a standalone ParserError class This allows us to forward declare it and reduce the number of things that need to include Parser.h.
2022-11-23 12:39:23 +01:00
Vector<ParserError> const& errors() const { return m_state.errors; }
LibJS: Add Javascript lexer and parser This adds a basic Javascript lexer and parser. It can parse the currently existing demo programs. More work needs to be done to turn it into a complete parser than can parse arbitrary JS Code. The lexer outputs tokens with preceeding whitespace and comments in the trivia member. This should allow us to generate the exact source code by concatenating the generated tokens. The parser is written in a way that it always returns a complete syntax tree. Error conditions are represented as nodes in the tree. This simplifies the code and allows it to be used as an early stage parser, e.g for parsing JS documents in an IDE while editing the source code.:
2020-03-11 19:27:43 +01:00
LibJS: Memoize failed calls of try_parse_arrow_function_expression()
2021-04-11 22:41:51 +02:00
struct TokenMemoization {
bool try_parse_arrow_function_expression_failed;
};
LibJS: Create FunctionParameters earlier in the parser This avoids making multiple copies of the Vector<FunctionParameter> in the parser.
2025-03-27 14:28:35 +00:00
static Parser parse_function_body_from_string(ByteString const& body_string, u16 parse_options, NonnullRefPtr<FunctionParameters const>, FunctionKind kind, FunctionParsingInsights&);
LibJS: Fix scope detection for ids in default function params This change fixes an issue where identifiers used in default function parameters were being "registered" in the function's parent scope instead of its own scope. This bug resulted in incorrectly detected local variables. (Variables used in the default function parameter expression should be considered 'captured by nested function'.) To resolve this issue, the function scope is now created before parsing function parameters. Since function parameters can no longer be passed in the constructor, a setter function has been introduced to set them later, when they are ready.
2023-07-07 23:14:03 +02:00
LibJS: Add Javascript lexer and parser This adds a basic Javascript lexer and parser. It can parse the currently existing demo programs. More work needs to be done to turn it into a complete parser than can parse arbitrary JS Code. The lexer outputs tokens with preceeding whitespace and comments in the trivia member. This should allow us to generate the exact source code by concatenating the generated tokens. The parser is written in a way that it always returns a complete syntax tree. Error conditions are represented as nodes in the tree. This simplifies the code and allows it to be used as an early stage parser, e.g for parsing JS documents in an IDE while editing the source code.:
2020-03-11 19:27:43 +01:00
private:
LibJS: Hoist variable declarations to the nearest relevant scope "var" declarations are hoisted to the nearest function scope, while "let" and "const" are hoisted to the nearest block scope. This is done by the parser, which keeps two scope stacks, one stack for the current var scope and one for the current let/const scope. When the interpreter enters a scope, we walk all of the declarations and insert them into the variable environment. We don't support the temporal dead zone for let/const yet.
2020-04-13 16:42:54 +02:00
friend class ScopePusher;
LibJS: Split parsing program to script and module separately This allows us to only perform checks like export bindings existing only for modules. Also this makes it easier to set strict and other state variables with TemporaryChanges.
2021-11-27 00:01:23 +01:00
void parse_script(Program& program, bool starts_in_strict_mode);
void parse_module(Program& program);
LibJS: Hoist variable declarations to the nearest relevant scope "var" declarations are hoisted to the nearest function scope, while "let" and "const" are hoisted to the nearest block scope. This is done by the parser, which keeps two scope stacks, one stack for the current var scope and one for the current let/const scope. When the interpreter enters a scope, we walk all of the declarations and insert them into the variable environment. We don't support the temporal dead zone for let/const yet.
2020-04-13 16:42:54 +02:00
LibJS: Add operator precedence parsing Obey precedence and associativity rules when parsing expressions with chained operators.
2020-03-12 23:02:41 +01:00
Associativity operator_associativity(TokenType) const;
LibJS: Add Javascript lexer and parser This adds a basic Javascript lexer and parser. It can parse the currently existing demo programs. More work needs to be done to turn it into a complete parser than can parse arbitrary JS Code. The lexer outputs tokens with preceeding whitespace and comments in the trivia member. This should allow us to generate the exact source code by concatenating the generated tokens. The parser is written in a way that it always returns a complete syntax tree. Error conditions are represented as nodes in the tree. This simplifies the code and allows it to be used as an early stage parser, e.g for parsing JS documents in an IDE while editing the source code.:
2020-03-11 19:27:43 +01:00
bool match_expression() const;
LibJS/Parser: Implement the parsing of unary/prefixed update expressions
2020-03-14 20:45:51 +02:00
bool match_unary_prefixed_expression() const;
LibJS: Fix mixing of logical and coalescing operators The same expression is not allowed to contain both the logical && and || operators, and the coalescing ?? operator. This patch changes how "forbidden" tokens are handled, using a finite set instead of an Vector. This supports much more efficient merging of the forbidden tokens when propagating forward, and allowing the return of forbidden tokens to parent contexts.
2022-02-15 22:34:59 -08:00
bool match_secondary_expression(ForbiddenTokens forbidden = {}) const;
LibJS: Add Javascript lexer and parser This adds a basic Javascript lexer and parser. It can parse the currently existing demo programs. More work needs to be done to turn it into a complete parser than can parse arbitrary JS Code. The lexer outputs tokens with preceeding whitespace and comments in the trivia member. This should allow us to generate the exact source code by concatenating the generated tokens. The parser is written in a way that it always returns a complete syntax tree. Error conditions are represented as nodes in the tree. This simplifies the code and allows it to be used as an early stage parser, e.g for parsing JS documents in an IDE while editing the source code.:
2020-03-11 19:27:43 +01:00
bool match_statement() const;
LibJS: Parse and partially execute import and export statements We produce the import and export entries as per the spec. However we do not yet verify that named things that are exported are declared somewhere.
2021-08-14 17:42:30 +02:00
bool match_export_or_import() const;
LibJS: Add using declaration support, RAII like operation in js In this patch only top level and not the more complicated for loop using statements are supported. Also, as noted in the latest meeting of tc39 async parts of the spec are not stage 3 thus not included.
2022-12-20 22:09:57 +01:00
enum class AllowUsingDeclaration {
No,
Yes
};
bool match_declaration(AllowUsingDeclaration allow_using = AllowUsingDeclaration::No) const;
LibJS: Do not save state for peeking at the next token from the lexer This saves having to save and load the parser state. This could give an incorrect token in some cases where the parser communicates to the lexer. However this is not applicable in any of the current usages and this would require one to parse the current token as normal which is exactly what you don't want to do in that scenario.
2021-10-08 00:38:24 +02:00
bool try_match_let_declaration() const;
LibJS: Add using declaration support, RAII like operation in js In this patch only top level and not the more complicated for loop using statements are supported. Also, as noted in the latest meeting of tc39 async parts of the spec are not stage 3 thus not included.
2022-12-20 22:09:57 +01:00
bool try_match_using_declaration() const;
LibJS: Do not save state for peeking at the next token from the lexer This saves having to save and load the parser state. This could give an incorrect token in some cases where the parser communicates to the lexer. However this is not applicable in any of the current usages and this would require one to parse the current token as normal which is exactly what you don't want to do in that scenario.
2021-10-08 00:38:24 +02:00
bool match_variable_declaration() const;
LibJS: Rework Identifier parsing to match the spec more closely
2021-07-11 15:34:55 +04:30
bool match_identifier() const;
LibJS: Add using declaration support in for and for of loops The using declarations have kind of special behavior in for loops so this is seperated.
2022-12-23 01:45:29 +01:00
bool token_is_identifier(Token const&) const;
LibJS: Allow reserved words as keys in object expressions.
2020-04-18 20:31:27 +02:00
bool match_identifier_name() const;
LibJS: Initial class implementation; allow super expressions in object literal methods; add EnvrionmentRecord fields and methods to LexicalEnvironment Adding EnvrionmentRecord's fields and methods lets us throw an exception when |this| is not initialized, which occurs when the super constructor in a derived class has not yet been called, or when |this| has already been initialized (the super constructor was already called).
2020-06-08 13:31:21 -05:00
bool match_property_key() const;
LibJS: Add parsing and evaluation of private fields and methods
2021-10-12 22:45:52 +02:00
bool is_private_identifier_valid() const;
LibJS: Add Javascript lexer and parser This adds a basic Javascript lexer and parser. It can parse the currently existing demo programs. More work needs to be done to turn it into a complete parser than can parse arbitrary JS Code. The lexer outputs tokens with preceeding whitespace and comments in the trivia member. This should allow us to generate the exact source code by concatenating the generated tokens. The parser is written in a way that it always returns a complete syntax tree. Error conditions are represented as nodes in the tree. This simplifies the code and allows it to be used as an early stage parser, e.g for parsing JS documents in an IDE while editing the source code.:
2020-03-11 19:27:43 +01:00
bool match(TokenType type) const;
bool done() const;
Everywhere: Run clang-format
2022-04-01 20:58:27 +03:00
void expected(char const* what);
LibJS: Use FlyString in PropertyKey instead of DeprecatedFlyString This required dealing with *substantial* fallout.
2025-03-18 18:08:02 -05:00
void syntax_error(String const& message, Optional<Position> = {});
LibJS: Add Javascript lexer and parser This adds a basic Javascript lexer and parser. It can parse the currently existing demo programs. More work needs to be done to turn it into a complete parser than can parse arbitrary JS Code. The lexer outputs tokens with preceeding whitespace and comments in the trivia member. This should allow us to generate the exact source code by concatenating the generated tokens. The parser is written in a way that it always returns a complete syntax tree. Error conditions are represented as nodes in the tree. This simplifies the code and allows it to be used as an early stage parser, e.g for parsing JS documents in an IDE while editing the source code.:
2020-03-11 19:27:43 +01:00
Token consume();
LibJS: Parse slashes after reserved identifiers correctly Previously we were unable to parse code like `yield/2` because `/2` was parsed as a regex. At the same time `for (a in / b/)` was parsed as a division. This is solved by defaulting to division in the lexer, but calling `force_slash_as_regex()` from the parser whenever an IdentifierName is parsed as a ReservedWord.
2023-05-28 00:08:52 +02:00
Token consume_and_allow_division();
LibJS: Rework Identifier parsing to match the spec more closely
2021-07-11 15:34:55 +04:30
Token consume_identifier();
Token consume_identifier_reference();
LibJS: Add Javascript lexer and parser This adds a basic Javascript lexer and parser. It can parse the currently existing demo programs. More work needs to be done to turn it into a complete parser than can parse arbitrary JS Code. The lexer outputs tokens with preceeding whitespace and comments in the trivia member. This should allow us to generate the exact source code by concatenating the generated tokens. The parser is written in a way that it always returns a complete syntax tree. Error conditions are represented as nodes in the tree. This simplifies the code and allows it to be used as an early stage parser, e.g for parsing JS documents in an IDE while editing the source code.:
2020-03-11 19:27:43 +01:00
Token consume(TokenType type);
LibJS: Unprefixed octal numbers are a syntax error in strict mode
2020-10-19 18:01:28 +01:00
Token consume_and_validate_numeric_literal();
LibJS: Implement automatic semicolon insertion
2020-04-17 15:05:58 +02:00
void consume_or_insert_semicolon();
LibJS: Add Parser save_state() and load_state() functions These functions allow us to try to parse ambiguous expressions (such as arrow function arguments in parentheses), and rewind the state of the Parser if an expression candidate failed to parse.
2020-03-30 08:24:43 -05:00
void save_state();
void load_state();
LibJS: Don't save rule start positions along with the parser state This fixes #4617. Also fixes the small problem where some save states would be leaked.
2020-12-29 16:47:39 +03:30
void discard_saved_state();
LibJS: Replace 'size_t line, size_t column' with 'Optional<Position>' This is a bit nicer for two reasons: - The absence of line number/column information isn't based on 'values are zero' anymore but on Optional's value - When reporting syntax errors with position information other than the current token's position we had to store line and column ourselves, like this: auto foo_start_line = m_parser_state.m_current_token.line_number(); auto foo_start_column = m_parser_state.m_current_token.line_column(); ... syntax_error("...", foo_start_line, foo_start_column); Which now becomes: auto foo_start= position(); ... syntax_error("...", foo_start); This makes it easier to report correct positions for syntax errors that only emerge a few tokens later :^)
2020-11-02 21:03:19 +00:00
Position position() const;
LibJS: Add Javascript lexer and parser This adds a basic Javascript lexer and parser. It can parse the currently existing demo programs. More work needs to be done to turn it into a complete parser than can parse arbitrary JS Code. The lexer outputs tokens with preceeding whitespace and comments in the trivia member. This should allow us to generate the exact source code by concatenating the generated tokens. The parser is written in a way that it always returns a complete syntax tree. Error conditions are represented as nodes in the tree. This simplifies the code and allows it to be used as an early stage parser, e.g for parsing JS documents in an IDE while editing the source code.:
2020-03-11 19:27:43 +01:00
LibJS: Make RefPtr and NonnullRefPtr usage const-correct This mainly affected the AST, which is now const throughout.
2023-02-19 22:07:52 +01:00
RefPtr<BindingPattern const> synthesize_binding_pattern(Expression const& expression);
LibJS: Allow member expressions in binding patterns Also allows literal string and numbers as property names in object binding patterns.
2021-09-18 01:11:32 +02:00
LibJS: Keep the lookahead lexer alive after parsing its next token Currently, the lexer holds a ByteString, which is always heap-allocated. When we create a copy of the lexer for the lookahead token, that token will outlive the lexer copy. The token holds a couple of string views into the lexer's source string. This is fine for now, because the source string will be kept alive by the original lexer. But if the lexer were to hold a String or Utf16String, short strings will be stored on the stack due to SSO. Thus the token will hold views into released stack data. We need to keep the lookahead lexer alive to prevent UAF on views into its source string.
2025-08-07 12:35:16 -04:00
Token next_token() const;
LibJS: Fix small issues in parser - Fix some places where escaped keywords are (not) allowed. - Be more strict about parameters for functions with 'use strict'. - Fix that expressions statements allowed functions and classes. - Fix that class expressions were not allowed. - Added a new next_token() method for checking the look ahead. - Fix that continue labels could jump to non iterating targets. - Fix that generator functions cannot be declared in if statements.
2021-08-28 17:04:37 +02:00
LibJS: Port the JS lexer and parser to UTF-16 This ports the lexer to UTF-16 and deals with the immediate fallout up to the AST. The AST will be dealt with in upcoming commits. The lexer will still accept UTF-8 strings as input, and will transcode them to UTF-16 for lexing. This doesn't actually incur a new allocation, as we were already converting the input StringView to a ByteString for each lexer. One immediate logical benefit here is that we do not need to know off- hand how many UTF-8 bytes some special code points occupy. They all happen to be a single UTF-16 code unit. So instead of advancing the lexer by 3 positions in some cases, we can just always advance by 1.
2025-08-06 07:18:45 -04:00
void check_identifier_name_for_assignment_validity(Utf16FlyString const&, bool force_strict = false);
LibJS: Rework Identifier parsing to match the spec more closely
2021-07-11 15:34:55 +04:30
Everywhere: Run clang-format
2022-04-01 20:58:27 +03:00
bool try_parse_arrow_function_expression_failed_at_position(Position const&) const;
void set_try_parse_arrow_function_expression_failed_at_position(Position const&, bool);
LibJS: Memoize failed calls of try_parse_arrow_function_expression()
2021-04-11 22:41:51 +02:00
LibJS: Handle escaped keywords in more cases and handle 'await' labels
2021-09-18 23:02:50 +02:00
bool match_invalid_escaped_keyword() const;
LibJS: Make scoping follow the spec Before this we used an ad-hoc combination of references and 'variables' stored in a hashmap. This worked in most cases but is not spec like. Additionally hoisting, dynamically naming functions and scope analysis was not done properly. This patch fixes all of that by: - Implement BindingInitialization for destructuring assignment. - Implementing a new ScopePusher which tracks the lexical and var scoped declarations. This hoists functions to the top level if no lexical declaration name overlaps. Furthermore we do checking of redeclarations in the ScopePusher now requiring less checks all over the place. - Add methods for parsing the directives and statement lists instead of having that code duplicated in multiple places. This allows declarations to pushed to the appropriate scope more easily. - Remove the non spec way of storing 'variables' in DeclarativeEnvironment and make Reference follow the spec instead of checking both the bindings and 'variables'. - Remove all scoping related things from the Interpreter. And instead use environments as specified by the spec. This also includes fixing that NativeFunctions did not produce a valid FunctionEnvironment which could cause issues with callbacks and eval. All FunctionObjects now have a valid NewFunctionEnvironment implementation. - Remove execute_statements from Interpreter and instead use ASTNode::execute everywhere this simplifies AST.cpp as you no longer need to worry about which method to call. - Make ScopeNodes setup their own environment. This uses four different methods specified by the spec {Block, Function, Eval, Global}DeclarationInstantiation with the annexB extensions. - Implement and use NamedEvaluation where specified. Additionally there are fixes to things exposed by these changes to eval, {for, for-in, for-of} loops and assignment. Finally it also fixes some tests in test-js which where passing before but not now that we have correct behavior :^).
2021-09-22 12:44:56 +02:00
bool parse_directive(ScopeNode& body);
void parse_statement_list(ScopeNode& output_node, AllowLabelledFunction allow_labelled_functions = AllowLabelledFunction::No);
LibJS: Make parsing import and export entries follow the spec The big changes are: - Allow strings as Module{Export, Import}Name - Properly track declarations in default export statements However, the spec is a little strange in that it allows function and class declarations without a name in default export statements. This is quite hard to fully implement without rewriting more of the parser so for now this behavior is emulated by faking things with function and class expressions. See the comments in parse_export_statement for details on the hacks and where it goes wrong.
2022-01-16 23:51:28 +01:00
LibJS: Port the Identifier AST (and related) nodes to UTF-16 This eliminates quite a lot of UTF-8 / UTF-16 churn.
2025-08-06 11:12:58 -04:00
Utf16FlyString consume_string_value();
LibJS: Make parsing import and export entries follow the spec The big changes are: - Allow strings as Module{Export, Import}Name - Properly track declarations in default export statements However, the spec is a little strange in that it allows function and class declarations without a name in default export statements. This is quite hard to fully implement without rewriting more of the parser so for now this behavior is emulated by faking things with function and class expressions. See the comments in parse_export_statement for details on the hacks and where it goes wrong.
2022-01-16 23:51:28 +01:00
ModuleRequest parse_module_request();
LibJS: Make scoping follow the spec Before this we used an ad-hoc combination of references and 'variables' stored in a hashmap. This worked in most cases but is not spec like. Additionally hoisting, dynamically naming functions and scope analysis was not done properly. This patch fixes all of that by: - Implement BindingInitialization for destructuring assignment. - Implementing a new ScopePusher which tracks the lexical and var scoped declarations. This hoists functions to the top level if no lexical declaration name overlaps. Furthermore we do checking of redeclarations in the ScopePusher now requiring less checks all over the place. - Add methods for parsing the directives and statement lists instead of having that code duplicated in multiple places. This allows declarations to pushed to the appropriate scope more easily. - Remove the non spec way of storing 'variables' in DeclarativeEnvironment and make Reference follow the spec instead of checking both the bindings and 'variables'. - Remove all scoping related things from the Interpreter. And instead use environments as specified by the spec. This also includes fixing that NativeFunctions did not produce a valid FunctionEnvironment which could cause issues with callbacks and eval. All FunctionObjects now have a valid NewFunctionEnvironment implementation. - Remove execute_statements from Interpreter and instead use ASTNode::execute everywhere this simplifies AST.cpp as you no longer need to worry about which method to call. - Make ScopeNodes setup their own environment. This uses four different methods specified by the spec {Block, Function, Eval, Global}DeclarationInstantiation with the annexB extensions. - Implement and use NamedEvaluation where specified. Additionally there are fixes to things exposed by these changes to eval, {for, for-in, for-of} loops and assignment. Finally it also fixes some tests in test-js which where passing before but not now that we have correct behavior :^).
2021-09-22 12:44:56 +02:00
LibJS: Track source positions all the way down to exceptions This makes exceptions have a trace of source positions too, which could probably be helpful in making fancier error tracebacks.
2020-12-28 20:45:22 +03:30
struct RulePosition {
LibJS: `save_state()' before creating a RulePosition Fixes #4617.
2020-12-29 08:42:02 +03:30
AK_MAKE_NONCOPYABLE(RulePosition);
AK_MAKE_NONMOVABLE(RulePosition);
public:
LibJS: Track source positions all the way down to exceptions This makes exceptions have a trace of source positions too, which could probably be helpful in making fancier error tracebacks.
2020-12-28 20:45:22 +03:30
RulePosition(Parser& parser, Position position)
: m_parser(parser)
, m_position(position)
{
LibJS: Don't save rule start positions along with the parser state This fixes #4617. Also fixes the small problem where some save states would be leaked.
2020-12-29 16:47:39 +03:30
m_parser.m_rule_starts.append(position);
LibJS: Track source positions all the way down to exceptions This makes exceptions have a trace of source positions too, which could probably be helpful in making fancier error tracebacks.
2020-12-28 20:45:22 +03:30
}
~RulePosition()
{
LibJS: Don't save rule start positions along with the parser state This fixes #4617. Also fixes the small problem where some save states would be leaked.
2020-12-29 16:47:39 +03:30
auto last = m_parser.m_rule_starts.take_last();
Everywhere: Rename ASSERT => VERIFY (...and ASSERT_NOT_REACHED => VERIFY_NOT_REACHED) Since all of these checks are done in release builds as well, let's rename them to VERIFY to prevent confusion, as everyone is used to assertions being compiled out in release. We can introduce a new ASSERT macro that is specifically for debug checks, but I'm doing this wholesale conversion first since we've accumulated thousands of these already, and it's not immediately obvious which ones are suitable for ASSERT.
2021-02-23 20:42:32 +01:00
VERIFY(last.line == m_position.line);
VERIFY(last.column == m_position.column);
LibJS: Track source positions all the way down to exceptions This makes exceptions have a trace of source positions too, which could probably be helpful in making fancier error tracebacks.
2020-12-28 20:45:22 +03:30
}
Everywhere: Run clang-format
2022-04-01 20:58:27 +03:00
Position const& position() const { return m_position; }
LibJS: Track source positions all the way down to exceptions This makes exceptions have a trace of source positions too, which could probably be helpful in making fancier error tracebacks.
2020-12-28 20:45:22 +03:30
private:
Parser& m_parser;
Position m_position;
};
[[nodiscard]] RulePosition push_start() { return { *this, position() }; }
LibJS: Add Parser save_state() and load_state() functions These functions allow us to try to parse ambiguous expressions (such as arrow function arguments in parentheses), and rewind the state of the Parser if an expression candidate failed to parse.
2020-03-30 08:24:43 -05:00
struct ParserState {
LibJS: Rename Parser::m_parser_state => m_state Also remove the "m_" prefix from all the public data members.
2021-06-19 14:43:09 +02:00
Lexer lexer;
LibJS: Keep the lookahead lexer alive after parsing its next token Currently, the lexer holds a ByteString, which is always heap-allocated. When we create a copy of the lexer for the lookahead token, that token will outlive the lexer copy. The token holds a couple of string views into the lexer's source string. This is fine for now, because the source string will be kept alive by the original lexer. But if the lexer were to hold a String or Utf16String, short strings will be stored on the stack due to SSO. Thus the token will hold views into released stack data. We need to keep the lookahead lexer alive to prevent UAF on views into its source string.
2025-08-07 12:35:16 -04:00
mutable Optional<Lexer> lookahead_lexer;
LibJS: Rename Parser::m_parser_state => m_state Also remove the "m_" prefix from all the public data members.
2021-06-19 14:43:09 +02:00
Token current_token;
LibJS: Don't create `arguments` object due to `o.arguments` access When deciding whether we need to create a full-blown `arguments` object, we look at various things, starting as early as in the parser. Until now, if the parser saw the identifier `arguments`, we'd decide that it's enough of a clue that we should create the `arguments` object since somebody is obviously accessing it. However, that missed the case where someone is just accessing a property named `arguments` on some object. In such cases (`o.arguments`), we now hold off on creating an `arguments` object. ~11% speed-up on Octane/typescript.js :^)
2023-11-16 12:21:20 +01:00
bool previous_token_was_period { false };
LibJS: Make Parser::Error a standalone ParserError class This allows us to forward declare it and reduce the number of things that need to include Parser.h.
2022-11-23 12:39:23 +01:00
Vector<ParserError> errors;
LibJS: Add missing initializer for ParserState::m_current_scope_pusher
2021-10-08 01:55:24 +02:00
ScopePusher* current_scope_pusher { nullptr };
LibJS: Track which Identifier nodes refer to function arguments This patch adds an "argument index" field to Identifier AST nodes. If the Identifier refers to a function parameter in the currently open function scope, we stash the index of the parameter here. This will allow us to implement much faster direct access to function argument variables.
2021-06-14 09:30:43 +02:00
LibJS: Port the JS lexer and parser to UTF-16 This ports the lexer to UTF-16 and deals with the immediate fallout up to the AST. The AST will be dealt with in upcoming commits. The lexer will still accept UTF-8 strings as input, and will transcode them to UTF-16 for lexing. This doesn't actually incur a new allocation, as we were already converting the input StringView to a ByteString for each lexer. One immediate logical benefit here is that we do not need to know off- hand how many UTF-8 bytes some special code points occupy. They all happen to be a single UTF-16 code unit. So instead of advancing the lexer by 3 positions in some cases, we can just always advance by 1.
2025-08-06 07:18:45 -04:00
HashMap<Utf16FlyString, Optional<Position>> labels_in_scope;
LibJS: Remove m_first_invalid_property_range from ObjectExpression This was state only used by the parser to output an error with appropriate location. This shrinks the size of ObjectExpression from 120 bytes down to just 56. This saves roughly 2.5 MiB when loading twitter.
2022-11-27 02:24:38 +01:00
HashMap<size_t, Position> invalid_property_range_in_object_expression;
LibJS: Port the JS lexer and parser to UTF-16 This ports the lexer to UTF-16 and deals with the immediate fallout up to the AST. The AST will be dealt with in upcoming commits. The lexer will still accept UTF-8 strings as input, and will transcode them to UTF-16 for lexing. This doesn't actually incur a new allocation, as we were already converting the input StringView to a ByteString for each lexer. One immediate logical benefit here is that we do not need to know off- hand how many UTF-8 bytes some special code points occupy. They all happen to be a single UTF-16 code unit. So instead of advancing the lexer by 3 positions in some cases, we can just always advance by 1.
2025-08-06 07:18:45 -04:00
HashTable<Utf16FlyString>* referenced_private_names { nullptr };
LibJS: Add parsing and evaluation of private fields and methods
2021-10-12 22:45:52 +02:00
LibJS: Rename Parser::m_parser_state => m_state Also remove the "m_" prefix from all the public data members.
2021-06-19 14:43:09 +02:00
bool strict_mode { false };
bool allow_super_property_lookup { false };
bool allow_super_constructor_call { false };
bool in_function_context { false };
LibJS: Identify global variables during parsing Identifying global variables during parsing will make it possible to generate special optimized instruction to access them in upcoming changes.
2023-07-12 04:02:27 +02:00
bool initiated_by_eval { false };
LibJS: Add missing steps and spec comments to PerformEval While adding spec comments to PerformEval, I noticed we were missing multiple steps. Namely, these were: - Checking if the host will allow us to compile the string (allowing LibWeb to perform CSP for eval) - The parser's initial state depending on the environment around us on direct eval: - Allowing new.target via eval in functions - Allowing super calls and super properties via eval in classes - Disallowing the use of the arguments object in class field initializers at eval's parse time - Setting ScriptOrModule of eval's execution context The spec allows us to apply the additional parsing steps in any order. The method I have gone with is passing in a struct to the parser's constructor, which overrides the parser's initial state to (dis)allow the things stated above from the get-go.
2022-04-10 00:55:45 +01:00
bool in_eval_function_context { false }; // This controls if we allow new.target or not. Note that eval("return") is not allowed, so we have to have a separate state variable for eval.
LibJS: Disallow yield expression correctly in formal parameters And add ZERO WIDTH NO BREAK SPACE to valid whitespace.
2021-08-21 11:31:36 +02:00
bool in_formal_parameter_context { false };
LibJS: Rename Parser::m_parser_state => m_state Also remove the "m_" prefix from all the public data members.
2021-06-19 14:43:09 +02:00
bool in_generator_function_context { false };
LibJS: Rename in_async_function_context to await_expression_is_valid Since await can be valid in module code which is not an async function the old name is not really representative for the usage.
2021-11-26 23:50:32 +01:00
bool await_expression_is_valid { false };
LibJS: Rename Parser::m_parser_state => m_state Also remove the "m_" prefix from all the public data members.
2021-06-19 14:43:09 +02:00
bool in_arrow_function_context { false };
bool in_break_context { false };
bool in_continue_context { false };
bool string_legacy_octal_escape_sequence_in_scope { false };
LibJS: Add support for public fields in classes
2021-08-28 17:11:05 +02:00
bool in_class_field_initializer { false };
LibJS: Add support for await expressions
2021-11-09 22:52:21 +02:00
bool in_class_static_init_block { false };
LibJS: Add an optimization to avoid needless arguments object creation This gives FunctionNode a "might need arguments object" boolean flag and sets it based on the simplest possible heuristic for this: if we encounter an identifier called "arguments" or "eval" up to the next (nested) function declaration or expression, we won't need an arguments object. Otherwise, we *might* need one - the final decision is made in the FunctionDeclarationInstantiation AO. Now, this is obviously not perfect. Even if you avoid eval, something like `foo.arguments` will still trigger a false positive - but it's a start and already massively cuts down on needlessly allocated objects, especially in real-world code that is often minified, and so a full "arguments" identifier will be an actual arguments object more often than not. To illustrate the actual impact of this change, here's the number of allocated arguments objects during a full test-js run: Before: - Unmapped arguments objects: 78765 - Mapped arguments objects: 2455 After: - Unmapped arguments objects: 18 - Mapped arguments objects: 37 This results in a ~5% speedup of test-js on my Linux host machine, and about 3.5% on i686 Serenity in QEMU (warm runs, average of 5). The following microbenchmark (calling an empty function 1M times) runs 25% faster on Linux and 45% on Serenity: function foo() {} for (var i = 0; i < 1_000_000; ++i) foo(); test262 reports no changes in either direction, apart from a speedup :^)
2021-10-05 08:44:58 +01:00
bool function_might_need_arguments_object { false };
LibJS: Add Parser save_state() and load_state() functions These functions allow us to try to parse ambiguous expressions (such as arrow function arguments in parentheses), and rewind the state of the Parser if an expression candidate failed to parse.
2020-03-30 08:24:43 -05:00
LibJS: Add a mode to parse JS as a module In a module strict mode should be enabled at the start of parsing and we allow import and export statements.
2021-08-14 17:30:37 +02:00
ParserState(Lexer, Program::Type);
LibJS: Add Parser save_state() and load_state() functions These functions allow us to try to parse ambiguous expressions (such as arrow function arguments in parentheses), and rewind the state of the Parser if an expression candidate failed to parse.
2020-03-30 08:24:43 -05:00
};
LibJS: Port the Identifier AST (and related) nodes to UTF-16 This eliminates quite a lot of UTF-8 / UTF-16 churn.
2025-08-06 11:12:58 -04:00
[[nodiscard]] NonnullRefPtr<Identifier const> create_identifier_and_register_in_current_scope(SourceRange range, Utf16FlyString string, Optional<DeclarationKind> = {});
LibJS: Update parser to detect if identifier refer a "local" variable This modification enables the parser to determine whether an identifier used within a function refers to a local variable or not. In this context, a local identifier means that it is not captured by any nested function declaration which means it modified only from inside a function. The information about whether identifier is local is stored inside Identifier AST node and also contains information about the index of local variable inside a function and information about total number of local variables used by a function is stored in function nodes.
2023-07-05 00:14:41 +02:00
LibJS: Make RefPtr and NonnullRefPtr usage const-correct This mainly affected the AST, which is now const throughout.
2023-02-19 22:07:52 +01:00
NonnullRefPtr<SourceCode const> m_source_code;
LibJS: Don't save rule start positions along with the parser state This fixes #4617. Also fixes the small problem where some save states would be leaked.
2020-12-29 16:47:39 +03:30
Vector<Position> m_rule_starts;
LibJS: Rename Parser::m_parser_state => m_state Also remove the "m_" prefix from all the public data members.
2021-06-19 14:43:09 +02:00
ParserState m_state;
LibJS: Add function default arguments Adds the ability for function arguments to have default values. This works for standard functions as well as arrow functions. Default values are not printed in a <function>.toString() call, as nodes cannot print their source string representation.
2020-05-02 11:46:39 -07:00
Vector<ParserState> m_saved_state;
LibJS: Remember arrow function parsing failures by offset We don't need to remember these by (line, column, offset). Just the offset should be enough.
2024-09-02 11:42:48 +02:00
HashMap<size_t, TokenMemoization> m_token_memoizations;
LibJS: Add a mode to parse JS as a module In a module strict mode should be enabled at the start of parsing and we allow import and export statements.
2021-08-14 17:30:37 +02:00
Program::Type m_program_type;
LibJS: Add Javascript lexer and parser This adds a basic Javascript lexer and parser. It can parse the currently existing demo programs. More work needs to be done to turn it into a complete parser than can parse arbitrary JS Code. The lexer outputs tokens with preceeding whitespace and comments in the trivia member. This should allow us to generate the exact source code by concatenating the generated tokens. The parser is written in a way that it always returns a complete syntax tree. Error conditions are represented as nodes in the tree. This simplifies the code and allows it to be used as an early stage parser, e.g for parsing JS documents in an IDE while editing the source code.:
2020-03-11 19:27:43 +01:00
};
Meta: Enforce newlines around namespaces This has come up several times during code review, so let's just enforce it using a new clang-format 20 option.
2025-05-13 07:06:33 -04:00
LibJS: Add Javascript lexer and parser This adds a basic Javascript lexer and parser. It can parse the currently existing demo programs. More work needs to be done to turn it into a complete parser than can parse arbitrary JS Code. The lexer outputs tokens with preceeding whitespace and comments in the trivia member. This should allow us to generate the exact source code by concatenating the generated tokens. The parser is written in a way that it always returns a complete syntax tree. Error conditions are represented as nodes in the tree. This simplifies the code and allows it to be used as an early stage parser, e.g for parsing JS documents in an IDE while editing the source code.:
2020-03-11 19:27:43 +01:00
}
Reference in a new issue Copy permalink