Stowage/ladybird
2
0
Fork 0
mirror of https://github.com/LadybirdBrowser/ladybird.git synced 2025-10-19 23:53:20 +00:00
Code Issues Projects Releases Packages Wiki Activity
ladybird/Userland/Libraries/LibJS/Parser.h

354 lines
14 KiB
C
Raw Normal View History

LibJS: Add Javascript lexer and parser This adds a basic Javascript lexer and parser. It can parse the currently existing demo programs. More work needs to be done to turn it into a complete parser than can parse arbitrary JS Code. The lexer outputs tokens with preceeding whitespace and comments in the trivia member. This should allow us to generate the exact source code by concatenating the generated tokens. The parser is written in a way that it always returns a complete syntax tree. Error conditions are represented as nodes in the tree. This simplifies the code and allows it to be used as an early stage parser, e.g for parsing JS documents in an IDE while editing the source code.:
2020-03-11 19:27:43 +01:00
/*
Everywhere: Use s.unverwerth@serenityos.org :^)
2021-05-29 12:38:28 +02:00
* Copyright (c) 2020, Stephan Unverwerth <s.unverwerth@serenityos.org>
LibJS: Make parsing import and export entries follow the spec The big changes are: - Allow strings as Module{Export, Import}Name - Properly track declarations in default export statements However, the spec is a little strange in that it allows function and class declarations without a name in default export statements. This is quite hard to fully implement without rewriting more of the parser so for now this behavior is emulated by faking things with function and class expressions. See the comments in parse_export_statement for details on the hacks and where it goes wrong.
2022-01-16 23:51:28 +01:00
* Copyright (c) 2021-2022, David Tuin <davidot@serenityos.org>
LibJS: Add Javascript lexer and parser This adds a basic Javascript lexer and parser. It can parse the currently existing demo programs. More work needs to be done to turn it into a complete parser than can parse arbitrary JS Code. The lexer outputs tokens with preceeding whitespace and comments in the trivia member. This should allow us to generate the exact source code by concatenating the generated tokens. The parser is written in a way that it always returns a complete syntax tree. Error conditions are represented as nodes in the tree. This simplifies the code and allows it to be used as an early stage parser, e.g for parsing JS documents in an IDE while editing the source code.:
2020-03-11 19:27:43 +01:00
*
Everything: Move to SPDX license identifiers in all files. SPDX License Identifiers are a more compact / standardized way of representing file license information. See: https://spdx.dev/resources/use/#identifiers This was done with the `ambr` search and replace tool. ambr --no-parent-ignore --key-from-file --rep-from-file key.txt rep.txt *
2021-04-22 01:24:48 -07:00
* SPDX-License-Identifier: BSD-2-Clause
LibJS: Add Javascript lexer and parser This adds a basic Javascript lexer and parser. It can parse the currently existing demo programs. More work needs to be done to turn it into a complete parser than can parse arbitrary JS Code. The lexer outputs tokens with preceeding whitespace and comments in the trivia member. This should allow us to generate the exact source code by concatenating the generated tokens. The parser is written in a way that it always returns a complete syntax tree. Error conditions are represented as nodes in the tree. This simplifies the code and allows it to be used as an early stage parser, e.g for parsing JS documents in an IDE while editing the source code.:
2020-03-11 19:27:43 +01:00
*/
#pragma once
LibJS: Fix mixing of logical and coalescing operators The same expression is not allowed to contain both the logical && and || operators, and the coalescing ?? operator. This patch changes how "forbidden" tokens are handled, using a finite set instead of an Vector. This supports much more efficient merging of the forbidden tokens when propagating forward, and allowing the return of forbidden tokens to parent contexts.
2022-02-15 22:34:59 -08:00
#include <AK/Assertions.h>
LibJS: break or continue with nonexistent label is a syntax error
2020-10-08 10:49:08 -07:00
#include <AK/HashTable.h>
LibJS: Make the AST reference-counted This allows function objects to outlive the original parsed program without their ScopeNode disappearing.
2020-03-18 11:23:53 +01:00
#include <AK/NonnullRefPtr.h>
LibJS: Add Error::source_location_hint() This util function on the Error struct will take the source and then returns a string like this based on line and column information it has: foo bar ^ Which can be shown in the repl for syntax errors :^)
2020-05-26 12:57:50 +01:00
#include <AK/StringBuilder.h>
LibJS: Add strict mode Adds the ability for a scope (either a function or the entire program) to be in strict mode. Scopes default to non-strict mode. There are two ways to determine the strict-ness of the JS engine: 1. In the parser, this can be accessed with the parser_state variable m_is_strict_mode boolean. If true, the Parser is currently parsing in strict mode. This is done so that the Parser can generate syntax errors at parse time, which is required in some cases. 2. With Interpreter.is_strict_mode(). This allows strict mode checking at runtime as opposed to compile time. Additionally, in order to test this, a global isStrictMode() function has been added to the JS ReplObject under the test-mode flag.
2020-05-27 22:22:08 -07:00
#include <LibJS/AST.h>
#include <LibJS/Lexer.h>
LibJS: Implement create_dynamic_function() according to the spec The three major changes are: - Parsing parameters, the function body, and then the full assembled function source all separately. This is required by the spec, as function parameters and body must be valid each on their own, which cannot be guaranteed if we only ever parse the full function. - Returning an ECMAScriptFunctionObject instead of a FunctionExpression that needs to be evaluated separately. This vastly simplifies the {Async,AsyncGenerator,Generator,}Function constructor implementations. Drop '_node' from the function name accordingly. - The prototype is now determined via GetPrototypeFromConstructor and passed to OrdinaryFunctionCreate.
2022-01-15 17:26:06 +01:00
#include <LibJS/Runtime/FunctionConstructor.h>
LibJS: Track source positions all the way down to exceptions This makes exceptions have a trace of source positions too, which could probably be helpful in making fancier error tracebacks.
2020-12-28 20:45:22 +03:30
#include <LibJS/SourceRange.h>
LibJS: Fix mixing of logical and coalescing operators The same expression is not allowed to contain both the logical && and || operators, and the coalescing ?? operator. This patch changes how "forbidden" tokens are handled, using a finite set instead of an Vector. This supports much more efficient merging of the forbidden tokens when propagating forward, and allowing the return of forbidden tokens to parent contexts.
2022-02-15 22:34:59 -08:00
#include <LibJS/Token.h>
#include <initializer_list>
LibJS: Let parser keep track of errors Rather than printing them to stderr directly the parser now keeps a Vector<Error>, which allows the "owner" of the parser to consume them individually after parsing. The Error struct has a message, line number, column number and a to_string() helper function to format this information into a meaningful error message. The Function() constructor will now include an error message when throwing a SyntaxError.
2020-05-14 16:26:01 +01:00
#include <stdio.h>
LibJS: Add Javascript lexer and parser This adds a basic Javascript lexer and parser. It can parse the currently existing demo programs. More work needs to be done to turn it into a complete parser than can parse arbitrary JS Code. The lexer outputs tokens with preceeding whitespace and comments in the trivia member. This should allow us to generate the exact source code by concatenating the generated tokens. The parser is written in a way that it always returns a complete syntax tree. Error conditions are represented as nodes in the tree. This simplifies the code and allows it to be used as an early stage parser, e.g for parsing JS documents in an IDE while editing the source code.:
2020-03-11 19:27:43 +01:00
namespace JS {
LibJS: Add operator precedence parsing Obey precedence and associativity rules when parsing expressions with chained operators.
2020-03-12 23:02:41 +01:00
enum class Associativity {
Left,
Right
};
LibJS: Refactor parse_function_node() bool parameters into bit flags I'm about to add even more options and a bunch of unnamed true/false arguments is really not helpful. Let's make this a single parse options parameter using bit flags.
2020-10-20 17:56:49 +01:00
struct FunctionNodeParseOptions {
LibJS: Allow anonymous functions as default exports This requires a special case with names as the default function is supposed to have a unique name ("*default*" in our case) but when checked should have name "default".
2022-09-02 00:46:37 +02:00
enum : u16 {
LibJS: Refactor parse_function_node() bool parameters into bit flags I'm about to add even more options and a bunch of unnamed true/false arguments is really not helpful. Let's make this a single parse options parameter using bit flags.
2020-10-20 17:56:49 +01:00
CheckForFunctionAndName = 1 << 0,
AllowSuperPropertyLookup = 1 << 1,
AllowSuperConstructorCall = 1 << 2,
LibJS: Move checks for invalid getter/setter params to parse_function_node This allows us to provide better error messages as we can point the syntax error location to the exact first invalid parameter instead of always the end of the function within a object literal or class definition. Before this change: const Foo = { set bar() {} } ^ Uncaught exception: [SyntaxError]: Object setter property must have one argument (line: 1, column: 28) class Foo { set bar() {} } ^ Uncaught exception: [SyntaxError]: Class setter method must have one argument (line: 1, column: 26) After this change: const Foo = { set bar() {} } ^ Uncaught exception: [SyntaxError]: Setter function must have one argument (line: 1, column: 23) class Foo { set bar() {} } ^ Uncaught exception: [SyntaxError]: Setter function must have one argument (line: 1, column: 21) The only possible downside of this change is that class getters/setters and functions in objects are not distinguished in the message anymore - I don't think that's important though, and classes are (mostly) just syntactic sugar anyway.
2020-10-20 18:32:51 +01:00
IsGetterFunction = 1 << 3,
IsSetterFunction = 1 << 4,
LibJS: Implement rules for duplicate function parameters - A regular function can have duplicate parameters except in strict mode or if its parameter list is not "simple" (has a default or rest parameter) - An arrow function can never have duplicate parameters Compared to other engines I opted for more useful syntax error messages than a generic "duplicate parameter name not allowed in this context": "use strict"; function test(foo, foo) {} ^ Uncaught exception: [SyntaxError]: Duplicate parameter 'foo' not allowed in strict mode (line: 1, column: 34) function test(foo, foo = 1) {} ^ Uncaught exception: [SyntaxError]: Duplicate parameter 'foo' not allowed in function with default parameter (line: 1, column: 20) function test(foo, ...foo) {} ^ Uncaught exception: [SyntaxError]: Duplicate parameter 'foo' not allowed in function with rest parameter (line: 1, column: 23) (foo, foo) => {} ^ Uncaught exception: [SyntaxError]: Duplicate parameter 'foo' not allowed in arrow function (line: 1, column: 7)
2020-10-25 11:14:04 +00:00
IsArrowFunction = 1 << 5,
LibJS: Parse generator functions in object literals Also add some parser tests
2021-06-14 14:52:59 +04:30
IsGeneratorFunction = 1 << 6,
LibJS: Add support for async functions This commit adds support for the most bare bones version of async functions, support for async generator functions, async arrow functions and await expressions are TODO.
2021-11-09 20:39:22 +02:00
IsAsyncFunction = 1 << 7,
LibJS: Allow anonymous functions as default exports This requires a special case with names as the default function is supposed to have a unique name ("*default*" in our case) but when checked should have name "default".
2022-09-02 00:46:37 +02:00
HasDefaultExportName = 1 << 8,
LibJS: Refactor parse_function_node() bool parameters into bit flags I'm about to add even more options and a bunch of unnamed true/false arguments is really not helpful. Let's make this a single parse options parameter using bit flags.
2020-10-20 17:56:49 +01:00
};
};
LibJS: Make scoping follow the spec Before this we used an ad-hoc combination of references and 'variables' stored in a hashmap. This worked in most cases but is not spec like. Additionally hoisting, dynamically naming functions and scope analysis was not done properly. This patch fixes all of that by: - Implement BindingInitialization for destructuring assignment. - Implementing a new ScopePusher which tracks the lexical and var scoped declarations. This hoists functions to the top level if no lexical declaration name overlaps. Furthermore we do checking of redeclarations in the ScopePusher now requiring less checks all over the place. - Add methods for parsing the directives and statement lists instead of having that code duplicated in multiple places. This allows declarations to pushed to the appropriate scope more easily. - Remove the non spec way of storing 'variables' in DeclarativeEnvironment and make Reference follow the spec instead of checking both the bindings and 'variables'. - Remove all scoping related things from the Interpreter. And instead use environments as specified by the spec. This also includes fixing that NativeFunctions did not produce a valid FunctionEnvironment which could cause issues with callbacks and eval. All FunctionObjects now have a valid NewFunctionEnvironment implementation. - Remove execute_statements from Interpreter and instead use ASTNode::execute everywhere this simplifies AST.cpp as you no longer need to worry about which method to call. - Make ScopeNodes setup their own environment. This uses four different methods specified by the spec {Block, Function, Eval, Global}DeclarationInstantiation with the annexB extensions. - Implement and use NamedEvaluation where specified. Additionally there are fixes to things exposed by these changes to eval, {for, for-in, for-of} loops and assignment. Finally it also fixes some tests in test-js which where passing before but not now that we have correct behavior :^).
2021-09-22 12:44:56 +02:00
class ScopePusher;
LibJS: Add Javascript lexer and parser This adds a basic Javascript lexer and parser. It can parse the currently existing demo programs. More work needs to be done to turn it into a complete parser than can parse arbitrary JS Code. The lexer outputs tokens with preceeding whitespace and comments in the trivia member. This should allow us to generate the exact source code by concatenating the generated tokens. The parser is written in a way that it always returns a complete syntax tree. Error conditions are represented as nodes in the tree. This simplifies the code and allows it to be used as an early stage parser, e.g for parsing JS documents in an IDE while editing the source code.:
2020-03-11 19:27:43 +01:00
class Parser {
public:
LibJS: Add missing steps and spec comments to PerformEval While adding spec comments to PerformEval, I noticed we were missing multiple steps. Namely, these were: - Checking if the host will allow us to compile the string (allowing LibWeb to perform CSP for eval) - The parser's initial state depending on the environment around us on direct eval: - Allowing new.target via eval in functions - Allowing super calls and super properties via eval in classes - Disallowing the use of the arguments object in class field initializers at eval's parse time - Setting ScriptOrModule of eval's execution context The spec allows us to apply the additional parsing steps in any order. The method I have gone with is passing in a struct to the parser's constructor, which overrides the parser's initial state to (dis)allow the things stated above from the get-go.
2022-04-10 00:55:45 +01:00
struct EvalInitialState {
bool in_eval_function_context { false };
bool allow_super_property_lookup { false };
bool allow_super_constructor_call { false };
bool in_class_field_initializer { false };
};
explicit Parser(Lexer lexer, Program::Type program_type = Program::Type::Script, Optional<EvalInitialState> initial_state_for_eval = {});
LibJS: Add Javascript lexer and parser This adds a basic Javascript lexer and parser. It can parse the currently existing demo programs. More work needs to be done to turn it into a complete parser than can parse arbitrary JS Code. The lexer outputs tokens with preceeding whitespace and comments in the trivia member. This should allow us to generate the exact source code by concatenating the generated tokens. The parser is written in a way that it always returns a complete syntax tree. Error conditions are represented as nodes in the tree. This simplifies the code and allows it to be used as an early stage parser, e.g for parsing JS documents in an IDE while editing the source code.:
2020-03-11 19:27:43 +01:00
LibJS: Correct behaviour of direct vs. indirect eval eval only has direct access to the local scope when accessed through the name eval. This includes locals named eval, because of course it does.
2021-06-19 20:13:53 -07:00
NonnullRefPtr<Program> parse_program(bool starts_in_strict_mode = false);
LibJS: Add Javascript lexer and parser This adds a basic Javascript lexer and parser. It can parse the currently existing demo programs. More work needs to be done to turn it into a complete parser than can parse arbitrary JS Code. The lexer outputs tokens with preceeding whitespace and comments in the trivia member. This should allow us to generate the exact source code by concatenating the generated tokens. The parser is written in a way that it always returns a complete syntax tree. Error conditions are represented as nodes in the tree. This simplifies the code and allows it to be used as an early stage parser, e.g for parsing JS documents in an IDE while editing the source code.:
2020-03-11 19:27:43 +01:00
LibJS: Parse FunctionExpressions FunctionExpression is mostly like FunctionDeclaration, except the name is optional. Share the parsing logic in parse_function_node<NodeType>. This allows us to do nice things like: document.addEventListener("DOMContentLoaded", function() { alert("Hello friends!"); });
2020-03-19 11:52:56 +01:00
template<typename FunctionNodeType>
LibJS: Allow anonymous functions as default exports This requires a special case with names as the default function is supposed to have a unique name ("*default*" in our case) but when checked should have name "default".
2022-09-02 00:46:37 +02:00
NonnullRefPtr<FunctionNodeType> parse_function_node(u16 parse_options = FunctionNodeParseOptions::CheckForFunctionAndName, Optional<Position> const& function_start = {});
Vector<FunctionNode::Parameter> parse_formal_parameters(int& function_length, u16 parse_options = 0);
LibJS: Allow member expressions in binding patterns Also allows literal string and numbers as property names in object binding patterns.
2021-09-18 01:11:32 +02:00
enum class AllowDuplicates {
Yes,
No
};
enum class AllowMemberExpressions {
Yes,
No
};
RefPtr<BindingPattern> parse_binding_pattern(AllowDuplicates is_var_declaration = AllowDuplicates::No, AllowMemberExpressions allow_member_expressions = AllowMemberExpressions::No);
LibJS: Parse FunctionExpressions FunctionExpression is mostly like FunctionDeclaration, except the name is optional. Share the parsing logic in parse_function_node<NodeType>. This allows us to do nice things like: document.addEventListener("DOMContentLoaded", function() { alert("Hello friends!"); });
2020-03-19 11:52:56 +01:00
LibJS: Correctly parse yield-from expressions This commit implements parsing for `yield *expr`, and the multiple ways something can or can't be parsed like that. Also makes yield-from a TODO in the bytecode generator. Behold, the glory of javascript syntax: ```js // 'yield' = expression in generators. function* foo() { yield *bar; // <- Syntax error here, expression can't start with * } // 'yield' = identifier anywhere else. function foo() { yield *bar; // Perfectly fine, this is just `yield * bar` } ```
2021-06-14 15:46:41 +04:30
struct PrimaryExpressionParseResult {
NonnullRefPtr<Expression> result;
bool should_continue_parsing_as_expression { true };
};
LibJS: Distinguish between statement and declaration This separates matching/parsing of statements and declarations and fixes a few edge cases where the parser would incorrectly accept a declaration where only a statement is allowed - for example: if (foo) const a = 1; for (var bar;;) function b() {} while (baz) class c {}
2020-10-22 23:30:07 +01:00
NonnullRefPtr<Declaration> parse_declaration();
LibJS: Allow labelled functions in certain contexts Also disallow duplicated labels.
2021-07-25 01:01:22 +02:00
enum class AllowLabelledFunction {
No,
Yes
};
NonnullRefPtr<Statement> parse_statement(AllowLabelledFunction allow_labelled_function = AllowLabelledFunction::No);
LibJS: Make the AST reference-counted This allows function objects to outlive the original parsed program without their ScopeNode disappearing.
2020-03-18 11:23:53 +01:00
NonnullRefPtr<BlockStatement> parse_block_statement();
LibJS: Propagate "contains direct call to eval()" flag from parser We now propagate this flag to FunctionDeclaration, and then also into ECMAScriptFunctionObject. This will be used to disable optimizations that aren't safe in the presence of direct eval().
2021-10-08 12:43:38 +02:00
NonnullRefPtr<FunctionBody> parse_function_body(Vector<FunctionDeclaration::Parameter> const& parameters, FunctionKind function_kind, bool& contains_direct_call_to_eval);
LibJS: Make the AST reference-counted This allows function objects to outlive the original parsed program without their ScopeNode disappearing.
2020-03-18 11:23:53 +01:00
NonnullRefPtr<ReturnStatement> parse_return_statement();
LibJS: Require initializer for 'const' variable declaration
2020-10-30 19:08:45 +00:00
NonnullRefPtr<VariableDeclaration> parse_variable_declaration(bool for_loop_variable_declaration = false);
LibJS: Implement basic for..in and for..of loops
2020-04-21 19:21:26 +01:00
NonnullRefPtr<Statement> parse_for_statement();
LibJS: Implement parsing and executing for-await-of loops
2021-11-23 16:09:28 +01:00
enum class IsForAwaitLoop {
No,
Yes
};
LibJS: Fix mixing of logical and coalescing operators The same expression is not allowed to contain both the logical && and || operators, and the coalescing ?? operator. This patch changes how "forbidden" tokens are handled, using a finite set instead of an Vector. This supports much more efficient merging of the forbidden tokens when propagating forward, and allowing the return of forbidden tokens to parent contexts.
2022-02-15 22:34:59 -08:00
struct ForbiddenTokens {
ForbiddenTokens(std::initializer_list<TokenType> const& forbidden);
ForbiddenTokens merge(ForbiddenTokens other) const;
bool allows(TokenType token) const;
ForbiddenTokens forbid(std::initializer_list<TokenType> const& forbidden) const;
private:
void forbid_tokens(std::initializer_list<TokenType> const& forbidden);
bool m_forbid_in_token : 1 { false };
bool m_forbid_logical_tokens : 1 { false };
bool m_forbid_coalesce_token : 1 { false };
bool m_forbid_paren_open : 1 { false };
bool m_forbid_question_mark_period : 1 { false };
bool m_forbid_equals : 1 { false };
};
struct ExpressionResult {
template<typename T>
ExpressionResult(NonnullRefPtr<T> expression, ForbiddenTokens forbidden = {})
: expression(expression)
, forbidden(forbidden)
{
}
NonnullRefPtr<Expression> expression;
ForbiddenTokens forbidden;
};
LibJS: Implement parsing and executing for-await-of loops
2021-11-23 16:09:28 +01:00
NonnullRefPtr<Statement> parse_for_in_of_statement(NonnullRefPtr<ASTNode> lhs, IsForAwaitLoop is_await);
LibJS: Parse "if" statements This patch implements basic parsing of "if" statements. We don't yet support parsing "else", so I added a FIXME about that.
2020-03-21 18:40:17 +01:00
NonnullRefPtr<IfStatement> parse_if_statement();
LibJS: Implement "throw" You can now throw an expression to the nearest catcher! :^) To support throwing arbitrary values, I added an Exception class that sits as a wrapper around whatever is thrown. In the future it will be a logical place to store a call stack.
2020-03-24 22:03:50 +01:00
NonnullRefPtr<ThrowStatement> parse_throw_statement();
LibJS: Parse "try", "catch" and "finally" This is the first step towards support exceptions. :^)
2020-03-24 14:03:55 +01:00
NonnullRefPtr<TryStatement> parse_try_statement();
NonnullRefPtr<CatchClause> parse_catch_clause();
LibJS: Lexer and parser support for "switch" statements
2020-03-29 13:09:54 +02:00
NonnullRefPtr<SwitchStatement> parse_switch_statement();
NonnullRefPtr<SwitchCase> parse_switch_case();
NonnullRefPtr<BreakStatement> parse_break_statement();
LibJS: Add support for "continue" inside "for" statements :^)
2020-04-05 00:22:42 +02:00
NonnullRefPtr<ContinueStatement> parse_continue_statement();
LibJS: Add support for do..while statements
2020-04-04 21:29:23 +02:00
NonnullRefPtr<DoWhileStatement> parse_do_while_statement();
LibJS: Parse while statements
2020-04-21 19:27:57 +01:00
NonnullRefPtr<WhileStatement> parse_while_statement();
LibJS: Parse "with" statements :^)
2020-11-28 15:05:57 +01:00
NonnullRefPtr<WithStatement> parse_with_statement();
LibJS: Implement (no-op) debugger statement
2020-04-30 17:26:27 +01:00
NonnullRefPtr<DebuggerStatement> parse_debugger_statement();
LibJS: Fix mixing of logical and coalescing operators The same expression is not allowed to contain both the logical && and || operators, and the coalescing ?? operator. This patch changes how "forbidden" tokens are handled, using a finite set instead of an Vector. This supports much more efficient merging of the forbidden tokens when propagating forward, and allowing the return of forbidden tokens to parent contexts.
2022-02-15 22:34:59 -08:00
NonnullRefPtr<ConditionalExpression> parse_conditional_expression(NonnullRefPtr<Expression> test, ForbiddenTokens);
LibJS: Implement parsing and execution of optional chains
2021-09-14 06:56:31 +04:30
NonnullRefPtr<OptionalChain> parse_optional_chain(NonnullRefPtr<Expression> base);
LibJS: Fix mixing of logical and coalescing operators The same expression is not allowed to contain both the logical && and || operators, and the coalescing ?? operator. This patch changes how "forbidden" tokens are handled, using a finite set instead of an Vector. This supports much more efficient merging of the forbidden tokens when propagating forward, and allowing the return of forbidden tokens to parent contexts.
2022-02-15 22:34:59 -08:00
NonnullRefPtr<Expression> parse_expression(int min_precedence, Associativity associate = Associativity::Right, ForbiddenTokens forbidden = {});
LibJS: Correctly parse yield-from expressions This commit implements parsing for `yield *expr`, and the multiple ways something can or can't be parsed like that. Also makes yield-from a TODO in the bytecode generator. Behold, the glory of javascript syntax: ```js // 'yield' = expression in generators. function* foo() { yield *bar; // <- Syntax error here, expression can't start with * } // 'yield' = identifier anywhere else. function foo() { yield *bar; // Perfectly fine, this is just `yield * bar` } ```
2021-06-14 15:46:41 +04:30
PrimaryExpressionParseResult parse_primary_expression();
LibJS: Make the AST reference-counted This allows function objects to outlive the original parsed program without their ScopeNode disappearing.
2020-03-18 11:23:53 +01:00
NonnullRefPtr<Expression> parse_unary_prefixed_expression();
LibJS: Lex and parse regex literals, add RegExp objects This adds regex parsing/lexing, as well as a relatively empty RegExpObject. The purpose of this patch is to allow the engine to not get hung up on parsing regexes. This will aid in finding new syntax errors (say, from google or twitter) without having to replace all of their regexes first!
2020-06-03 16:05:49 -07:00
NonnullRefPtr<RegExpLiteral> parse_regexp_literal();
LibJS: Make the AST reference-counted This allows function objects to outlive the original parsed program without their ScopeNode disappearing.
2020-03-18 11:23:53 +01:00
NonnullRefPtr<ObjectExpression> parse_object_expression();
LibJS: Parse ArrayExpression and start implementing Array objects Note that property lookup is not functional yet.
2020-03-20 20:29:57 +01:00
NonnullRefPtr<ArrayExpression> parse_array_expression();
LibJS: Allow invalid string in tagged template literals Since tagged template literals can inspect the raw string it is not a syntax error to have invalid escapes. However the cooked value should be `undefined`. We accomplish this by tracking whether parse_string_literal fails and then using a NullLiteral (since UndefinedLiteral is not a thing) and finally converting null in tagged template execution to undefined.
2022-08-17 02:04:27 +02:00
enum class StringLiteralType {
Normal,
NonTaggedTemplate,
TaggedTemplate
};
NonnullRefPtr<StringLiteral> parse_string_literal(Token const& token, StringLiteralType string_literal_type = StringLiteralType::Normal, bool* contains_invalid_escape = nullptr);
LibJS: Add raw strings to tagged template literals When calling a function with a tagged template, the first array that is passed in now contains a "raw" property with the raw, escaped strings.
2020-05-06 16:34:14 -07:00
NonnullRefPtr<TemplateLiteral> parse_template_literal(bool is_tagged);
LibJS: Fix mixing of logical and coalescing operators The same expression is not allowed to contain both the logical && and || operators, and the coalescing ?? operator. This patch changes how "forbidden" tokens are handled, using a finite set instead of an Vector. This supports much more efficient merging of the forbidden tokens when propagating forward, and allowing the return of forbidden tokens to parent contexts.
2022-02-15 22:34:59 -08:00
ExpressionResult parse_secondary_expression(NonnullRefPtr<Expression>, int min_precedence, Associativity associate = Associativity::Right, ForbiddenTokens forbidden = {});
Everywhere: Prevent risky implicit casts of (Nonnull)RefPtr Our existing implementation did not check the element type of the other pointer in the constructors and move assignment operators. This meant that some operations that would require explicit casting on raw pointers were done implicitly, such as: - downcasting a base class to a derived class (e.g. `Kernel::Inode` => `Kernel::ProcFSDirectoryInode` in Kernel/ProcFS.cpp), - casting to an unrelated type (e.g. `Promise<bool>` => `Promise<Empty>` in LibIMAP/Client.cpp) This, of course, allows gross violations of the type system, and makes the need to type-check less obvious before downcasting. Luckily, while adding the `static_ptr_cast`s, only two truly incorrect usages were found; in the other instances, our casts just needed to be made explicit.
2021-09-03 19:11:51 +02:00
NonnullRefPtr<Expression> parse_call_expression(NonnullRefPtr<Expression>);
LibJS: Implement basic support for the "new" keyword NewExpression mostly piggybacks on the existing CallExpression. The big difference is that "new" creates a new Object and passes it as |this| to the callee.
2020-03-28 16:33:52 +01:00
NonnullRefPtr<NewExpression> parse_new_expression();
LibJS: Initial class implementation; allow super expressions in object literal methods; add EnvrionmentRecord fields and methods to LexicalEnvironment Adding EnvrionmentRecord's fields and methods lets us throw an exception when |this| is not initialized, which occurs when the super constructor in a derived class has not yet been called, or when |this| has already been initialized (the super constructor was already called).
2020-06-08 13:31:21 -05:00
NonnullRefPtr<ClassDeclaration> parse_class_declaration();
NonnullRefPtr<ClassExpression> parse_class_expression(bool expect_class_name);
LibJS: Implement generator functions (only in bytecode mode)
2021-06-11 01:38:30 +04:30
NonnullRefPtr<YieldExpression> parse_yield_expression();
LibJS: Add support for await expressions
2021-11-09 22:52:21 +02:00
NonnullRefPtr<AwaitExpression> parse_await_expression();
LibJS: Initial class implementation; allow super expressions in object literal methods; add EnvrionmentRecord fields and methods to LexicalEnvironment Adding EnvrionmentRecord's fields and methods lets us throw an exception when |this| is not initialized, which occurs when the super constructor in a derived class has not yet been called, or when |this| has already been initialized (the super constructor was already called).
2020-06-08 13:31:21 -05:00
NonnullRefPtr<Expression> parse_property_key();
LibJS: Fix mixing of logical and coalescing operators The same expression is not allowed to contain both the logical && and || operators, and the coalescing ?? operator. This patch changes how "forbidden" tokens are handled, using a finite set instead of an Vector. This supports much more efficient merging of the forbidden tokens when propagating forward, and allowing the return of forbidden tokens to parent contexts.
2022-02-15 22:34:59 -08:00
NonnullRefPtr<AssignmentExpression> parse_assignment_expression(AssignmentOp, NonnullRefPtr<Expression> lhs, int min_precedence, Associativity, ForbiddenTokens forbidden = {});
LibJS: Restructure and fully implement BindingPatterns
2021-06-12 18:04:28 -07:00
NonnullRefPtr<Identifier> parse_identifier();
LibJS: Parse and partially execute import and export statements We produce the import and export entries as per the spec. However we do not yet verify that named things that are exported are declared somewhere.
2021-08-14 17:42:30 +02:00
NonnullRefPtr<ImportStatement> parse_import_statement(Program& program);
NonnullRefPtr<ExportStatement> parse_export_statement(Program& program);
LibJS: Add Javascript lexer and parser This adds a basic Javascript lexer and parser. It can parse the currently existing demo programs. More work needs to be done to turn it into a complete parser than can parse arbitrary JS Code. The lexer outputs tokens with preceeding whitespace and comments in the trivia member. This should allow us to generate the exact source code by concatenating the generated tokens. The parser is written in a way that it always returns a complete syntax tree. Error conditions are represented as nodes in the tree. This simplifies the code and allows it to be used as an early stage parser, e.g for parsing JS documents in an IDE while editing the source code.:
2020-03-11 19:27:43 +01:00
LibJS: Parse async arrow functions
2021-11-15 00:47:16 +01:00
RefPtr<FunctionExpression> try_parse_arrow_function_expression(bool expect_parens, bool is_async = false);
LibJS: Replace the custom unwind mechanism with completions :^) This includes: - Parsing proper LabelledStatements with try_parse_labelled_statement() - Removing LabelableStatement - Implementing the LoopEvaluation semantics via loop_evaluation() in each IterationStatement subclass; and IterationStatement evaluation via {For,ForIn,ForOf,ForAwaitOf,While,DoWhile}Statement::execute() - Updating ReturnStatement, BreakStatement and ContinueStatement to return the appropriate completion types - Basically reimplementing TryStatement and SwitchStatement according to the spec, using completions - Honoring result completion types in AsyncBlockStart and OrdinaryCallEvaluateBody - Removing any uses of the VM unwind mechanism - most importantly, VM::throw_exception() now exclusively sets an exception and no longer triggers any unwinding mechanism. However, we already did a good job updating all of LibWeb and userland applications to not use it, and the few remaining uses elsewhere don't rely on unwinding AFAICT.
2022-01-05 19:11:16 +01:00
RefPtr<LabelledStatement> try_parse_labelled_statement(AllowLabelledFunction allow_function);
LibJS: Implement 'new.target' This adds a new MetaProperty AST node which will be used for 'new.target' and 'import.meta' meta properties. The parser now distinguishes between "in function context" and "in arrow function context" (which is required for this). When encountering TokenType::New we will attempt to parse it as meta property and resort to regular new expression parsing if that fails, much like the parsing of labelled statements.
2020-11-02 21:27:42 +00:00
RefPtr<MetaProperty> try_parse_new_target_expression();
LibJS: Parse dynamic import calls 'import()' and 'import.meta' For now both just throw when executing but this can be implemented when modules are implemented :^).
2021-11-26 23:45:10 +01:00
RefPtr<MetaProperty> try_parse_import_meta_expression();
NonnullRefPtr<ImportCall> parse_import_call();
LibJS: Implement 'new.target' This adds a new MetaProperty AST node which will be used for 'new.target' and 'import.meta' meta properties. The parser now distinguishes between "in function context" and "in arrow function context" (which is required for this). When encountering TokenType::New we will attempt to parse it as meta property and resort to regular new expression parsing if that fails, much like the parsing of labelled statements.
2020-11-02 21:27:42 +00:00
LibJS: Implement parsing and execution of optional chains
2021-09-14 06:56:31 +04:30
Vector<CallExpression::Argument> parse_arguments();
LibJS: Replace 'size_t line, size_t column' with 'Optional<Position>' This is a bit nicer for two reasons: - The absence of line number/column information isn't based on 'values are zero' anymore but on Optional's value - When reporting syntax errors with position information other than the current token's position we had to store line and column ourselves, like this: auto foo_start_line = m_parser_state.m_current_token.line_number(); auto foo_start_column = m_parser_state.m_current_token.line_column(); ... syntax_error("...", foo_start_line, foo_start_column); Which now becomes: auto foo_start= position(); ... syntax_error("...", foo_start); This makes it easier to report correct positions for syntax errors that only emerge a few tokens later :^)
2020-11-02 21:03:19 +00:00
struct Error {
String message;
Optional<Position> position;
LibJS: Let parser keep track of errors Rather than printing them to stderr directly the parser now keeps a Vector<Error>, which allows the "owner" of the parser to consume them individually after parsing. The Error struct has a message, line number, column number and a to_string() helper function to format this information into a meaningful error message. The Function() constructor will now include an error message when throwing a SyntaxError.
2020-05-14 16:26:01 +01:00
String to_string() const
{
LibJS: Replace 'size_t line, size_t column' with 'Optional<Position>' This is a bit nicer for two reasons: - The absence of line number/column information isn't based on 'values are zero' anymore but on Optional's value - When reporting syntax errors with position information other than the current token's position we had to store line and column ourselves, like this: auto foo_start_line = m_parser_state.m_current_token.line_number(); auto foo_start_column = m_parser_state.m_current_token.line_column(); ... syntax_error("...", foo_start_line, foo_start_column); Which now becomes: auto foo_start= position(); ... syntax_error("...", foo_start); This makes it easier to report correct positions for syntax errors that only emerge a few tokens later :^)
2020-11-02 21:03:19 +00:00
if (!position.has_value())
LibJS: Let parser keep track of errors Rather than printing them to stderr directly the parser now keeps a Vector<Error>, which allows the "owner" of the parser to consume them individually after parsing. The Error struct has a message, line number, column number and a to_string() helper function to format this information into a meaningful error message. The Function() constructor will now include an error message when throwing a SyntaxError.
2020-05-14 16:26:01 +01:00
return message;
LibJS: Replace 'size_t line, size_t column' with 'Optional<Position>' This is a bit nicer for two reasons: - The absence of line number/column information isn't based on 'values are zero' anymore but on Optional's value - When reporting syntax errors with position information other than the current token's position we had to store line and column ourselves, like this: auto foo_start_line = m_parser_state.m_current_token.line_number(); auto foo_start_column = m_parser_state.m_current_token.line_column(); ... syntax_error("...", foo_start_line, foo_start_column); Which now becomes: auto foo_start= position(); ... syntax_error("...", foo_start); This makes it easier to report correct positions for syntax errors that only emerge a few tokens later :^)
2020-11-02 21:03:19 +00:00
return String::formatted("{} (line: {}, column: {})", message, position.value().line, position.value().column);
LibJS: Let parser keep track of errors Rather than printing them to stderr directly the parser now keeps a Vector<Error>, which allows the "owner" of the parser to consume them individually after parsing. The Error struct has a message, line number, column number and a to_string() helper function to format this information into a meaningful error message. The Function() constructor will now include an error message when throwing a SyntaxError.
2020-05-14 16:26:01 +01:00
}
LibJS: Add Error::source_location_hint() This util function on the Error struct will take the source and then returns a string like this based on line and column information it has: foo bar ^ Which can be shown in the repl for syntax errors :^)
2020-05-26 12:57:50 +01:00
Everywhere: Run clang-format
2022-04-01 20:58:27 +03:00
String source_location_hint(StringView source, char const spacer = ' ', char const indicator = '^') const
LibJS: Add Error::source_location_hint() This util function on the Error struct will take the source and then returns a string like this based on line and column information it has: foo bar ^ Which can be shown in the repl for syntax errors :^)
2020-05-26 12:57:50 +01:00
{
LibJS: Replace 'size_t line, size_t column' with 'Optional<Position>' This is a bit nicer for two reasons: - The absence of line number/column information isn't based on 'values are zero' anymore but on Optional's value - When reporting syntax errors with position information other than the current token's position we had to store line and column ourselves, like this: auto foo_start_line = m_parser_state.m_current_token.line_number(); auto foo_start_column = m_parser_state.m_current_token.line_column(); ... syntax_error("...", foo_start_line, foo_start_column); Which now becomes: auto foo_start= position(); ... syntax_error("...", foo_start); This makes it easier to report correct positions for syntax errors that only emerge a few tokens later :^)
2020-11-02 21:03:19 +00:00
if (!position.has_value())
LibJS: Add Error::source_location_hint() This util function on the Error struct will take the source and then returns a string like this based on line and column information it has: foo bar ^ Which can be shown in the repl for syntax errors :^)
2020-05-26 12:57:50 +01:00
return {};
LibJS: Partially revert e3fa32b This was causing some syntactically wrong inputs to crash and subsequently broke CI (test262 parser tests).
2021-07-20 17:53:48 +01:00
// We need to modify the source to match what the lexer considers one line - normalizing
// line terminators to \n is easier than splitting using all different LT characters.
Everywhere: Add sv suffix to strings relying on StringView(char const*) Each of these strings would previously rely on StringView's char const* constructor overload, which would call __builtin_strlen on the string. Since we now have operator ""sv, we can replace these with much simpler versions. This opens the door to being able to remove StringView(char const*). No functional changes.
2022-07-11 17:32:29 +00:00
String source_string = source.replace("\r\n"sv, "\n"sv, ReplaceMode::All).replace("\r"sv, "\n"sv, ReplaceMode::All).replace(LINE_SEPARATOR_STRING, "\n"sv, ReplaceMode::All).replace(PARAGRAPH_SEPARATOR_STRING, "\n"sv, ReplaceMode::All);
LibJS: Partially revert e3fa32b This was causing some syntactically wrong inputs to crash and subsequently broke CI (test262 parser tests).
2021-07-20 17:53:48 +01:00
StringBuilder builder;
builder.append(source_string.split_view('\n', true)[position.value().line - 1]);
LibJS: Add Error::source_location_hint() This util function on the Error struct will take the source and then returns a string like this based on line and column information it has: foo bar ^ Which can be shown in the repl for syntax errors :^)
2020-05-26 12:57:50 +01:00
builder.append('\n');
LibJS: Replace 'size_t line, size_t column' with 'Optional<Position>' This is a bit nicer for two reasons: - The absence of line number/column information isn't based on 'values are zero' anymore but on Optional's value - When reporting syntax errors with position information other than the current token's position we had to store line and column ourselves, like this: auto foo_start_line = m_parser_state.m_current_token.line_number(); auto foo_start_column = m_parser_state.m_current_token.line_column(); ... syntax_error("...", foo_start_line, foo_start_column); Which now becomes: auto foo_start= position(); ... syntax_error("...", foo_start); This makes it easier to report correct positions for syntax errors that only emerge a few tokens later :^)
2020-11-02 21:03:19 +00:00
for (size_t i = 0; i < position.value().column - 1; ++i)
LibJS: Add Error::source_location_hint() This util function on the Error struct will take the source and then returns a string like this based on line and column information it has: foo bar ^ Which can be shown in the repl for syntax errors :^)
2020-05-26 12:57:50 +01:00
builder.append(spacer);
builder.append(indicator);
return builder.build();
}
LibJS: Let parser keep track of errors Rather than printing them to stderr directly the parser now keeps a Vector<Error>, which allows the "owner" of the parser to consume them individually after parsing. The Error struct has a message, line number, column number and a to_string() helper function to format this information into a meaningful error message. The Function() constructor will now include an error message when throwing a SyntaxError.
2020-05-14 16:26:01 +01:00
};
LibJS: Rename Parser::m_parser_state => m_state Also remove the "m_" prefix from all the public data members.
2021-06-19 14:43:09 +02:00
bool has_errors() const { return m_state.errors.size(); }
Everywhere: Run clang-format
2022-04-01 20:58:27 +03:00
Vector<Error> const& errors() const { return m_state.errors; }
LibJS: Optimize source_location_hint and add flag in print_errors This optimizes the algorithm used in source_location_hint and adds a flag to not print hints in print_errors.
2021-07-19 17:56:21 +02:00
void print_errors(bool print_hint = true) const
LibJS: Let parser keep track of errors Rather than printing them to stderr directly the parser now keeps a Vector<Error>, which allows the "owner" of the parser to consume them individually after parsing. The Error struct has a message, line number, column number and a to_string() helper function to format this information into a meaningful error message. The Function() constructor will now include an error message when throwing a SyntaxError.
2020-05-14 16:26:01 +01:00
{
LibJS: Rename Parser::m_parser_state => m_state Also remove the "m_" prefix from all the public data members.
2021-06-19 14:43:09 +02:00
for (auto& error : m_state.errors) {
LibJS: Optimize source_location_hint and add flag in print_errors This optimizes the algorithm used in source_location_hint and adds a flag to not print hints in print_errors.
2021-07-19 17:56:21 +02:00
if (print_hint) {
auto hint = error.source_location_hint(m_state.lexer.source());
if (!hint.is_empty())
warnln("{}", hint);
}
LibJS: Include source location hint in Parser::print_errors()
2020-12-06 14:50:39 +00:00
warnln("SyntaxError: {}", error.to_string());
}
LibJS: Let parser keep track of errors Rather than printing them to stderr directly the parser now keeps a Vector<Error>, which allows the "owner" of the parser to consume them individually after parsing. The Error struct has a message, line number, column number and a to_string() helper function to format this information into a meaningful error message. The Function() constructor will now include an error message when throwing a SyntaxError.
2020-05-14 16:26:01 +01:00
}
LibJS: Add Javascript lexer and parser This adds a basic Javascript lexer and parser. It can parse the currently existing demo programs. More work needs to be done to turn it into a complete parser than can parse arbitrary JS Code. The lexer outputs tokens with preceeding whitespace and comments in the trivia member. This should allow us to generate the exact source code by concatenating the generated tokens. The parser is written in a way that it always returns a complete syntax tree. Error conditions are represented as nodes in the tree. This simplifies the code and allows it to be used as an early stage parser, e.g for parsing JS documents in an IDE while editing the source code.:
2020-03-11 19:27:43 +01:00
LibJS: Memoize failed calls of try_parse_arrow_function_expression()
2021-04-11 22:41:51 +02:00
struct TokenMemoization {
bool try_parse_arrow_function_expression_failed;
};
LibJS: Implement create_dynamic_function() according to the spec The three major changes are: - Parsing parameters, the function body, and then the full assembled function source all separately. This is required by the spec, as function parameters and body must be valid each on their own, which cannot be guaranteed if we only ever parse the full function. - Returning an ECMAScriptFunctionObject instead of a FunctionExpression that needs to be evaluated separately. This vastly simplifies the {Async,AsyncGenerator,Generator,}Function constructor implementations. Drop '_node' from the function name accordingly. - The prototype is now determined via GetPrototypeFromConstructor and passed to OrdinaryFunctionCreate.
2022-01-15 17:26:06 +01:00
// Needs to mess with m_state, and we're not going to expose a non-const getter for that :^)
LibJS: Replace GlobalObject with VM in remaining AOs [Part 19/19]
2022-08-21 20:38:35 +01:00
friend ThrowCompletionOr<ECMAScriptFunctionObject*> FunctionConstructor::create_dynamic_function(VM&, FunctionObject&, FunctionObject*, FunctionKind, MarkedVector<Value> const&);
LibJS: Implement create_dynamic_function() according to the spec The three major changes are: - Parsing parameters, the function body, and then the full assembled function source all separately. This is required by the spec, as function parameters and body must be valid each on their own, which cannot be guaranteed if we only ever parse the full function. - Returning an ECMAScriptFunctionObject instead of a FunctionExpression that needs to be evaluated separately. This vastly simplifies the {Async,AsyncGenerator,Generator,}Function constructor implementations. Drop '_node' from the function name accordingly. - The prototype is now determined via GetPrototypeFromConstructor and passed to OrdinaryFunctionCreate.
2022-01-15 17:26:06 +01:00
LibJS: Add Javascript lexer and parser This adds a basic Javascript lexer and parser. It can parse the currently existing demo programs. More work needs to be done to turn it into a complete parser than can parse arbitrary JS Code. The lexer outputs tokens with preceeding whitespace and comments in the trivia member. This should allow us to generate the exact source code by concatenating the generated tokens. The parser is written in a way that it always returns a complete syntax tree. Error conditions are represented as nodes in the tree. This simplifies the code and allows it to be used as an early stage parser, e.g for parsing JS documents in an IDE while editing the source code.:
2020-03-11 19:27:43 +01:00
private:
LibJS: Hoist variable declarations to the nearest relevant scope "var" declarations are hoisted to the nearest function scope, while "let" and "const" are hoisted to the nearest block scope. This is done by the parser, which keeps two scope stacks, one stack for the current var scope and one for the current let/const scope. When the interpreter enters a scope, we walk all of the declarations and insert them into the variable environment. We don't support the temporal dead zone for let/const yet.
2020-04-13 16:42:54 +02:00
friend class ScopePusher;
LibJS: Split parsing program to script and module separately This allows us to only perform checks like export bindings existing only for modules. Also this makes it easier to set strict and other state variables with TemporaryChanges.
2021-11-27 00:01:23 +01:00
void parse_script(Program& program, bool starts_in_strict_mode);
void parse_module(Program& program);
LibJS: Hoist variable declarations to the nearest relevant scope "var" declarations are hoisted to the nearest function scope, while "let" and "const" are hoisted to the nearest block scope. This is done by the parser, which keeps two scope stacks, one stack for the current var scope and one for the current let/const scope. When the interpreter enters a scope, we walk all of the declarations and insert them into the variable environment. We don't support the temporal dead zone for let/const yet.
2020-04-13 16:42:54 +02:00
LibJS: Add operator precedence parsing Obey precedence and associativity rules when parsing expressions with chained operators.
2020-03-12 23:02:41 +01:00
Associativity operator_associativity(TokenType) const;
LibJS: Add Javascript lexer and parser This adds a basic Javascript lexer and parser. It can parse the currently existing demo programs. More work needs to be done to turn it into a complete parser than can parse arbitrary JS Code. The lexer outputs tokens with preceeding whitespace and comments in the trivia member. This should allow us to generate the exact source code by concatenating the generated tokens. The parser is written in a way that it always returns a complete syntax tree. Error conditions are represented as nodes in the tree. This simplifies the code and allows it to be used as an early stage parser, e.g for parsing JS documents in an IDE while editing the source code.:
2020-03-11 19:27:43 +01:00
bool match_expression() const;
LibJS/Parser: Implement the parsing of unary/prefixed update expressions
2020-03-14 20:45:51 +02:00
bool match_unary_prefixed_expression() const;
LibJS: Fix mixing of logical and coalescing operators The same expression is not allowed to contain both the logical && and || operators, and the coalescing ?? operator. This patch changes how "forbidden" tokens are handled, using a finite set instead of an Vector. This supports much more efficient merging of the forbidden tokens when propagating forward, and allowing the return of forbidden tokens to parent contexts.
2022-02-15 22:34:59 -08:00
bool match_secondary_expression(ForbiddenTokens forbidden = {}) const;
LibJS: Add Javascript lexer and parser This adds a basic Javascript lexer and parser. It can parse the currently existing demo programs. More work needs to be done to turn it into a complete parser than can parse arbitrary JS Code. The lexer outputs tokens with preceeding whitespace and comments in the trivia member. This should allow us to generate the exact source code by concatenating the generated tokens. The parser is written in a way that it always returns a complete syntax tree. Error conditions are represented as nodes in the tree. This simplifies the code and allows it to be used as an early stage parser, e.g for parsing JS documents in an IDE while editing the source code.:
2020-03-11 19:27:43 +01:00
bool match_statement() const;
LibJS: Parse and partially execute import and export statements We produce the import and export entries as per the spec. However we do not yet verify that named things that are exported are declared somewhere.
2021-08-14 17:42:30 +02:00
bool match_export_or_import() const;
LibJS: Parse assert clauses of in- and export statements Based on proposal: https://tc39.es/proposal-import-assertions Since imports are not supported yet this is not functional.
2021-12-20 15:29:25 +01:00
bool match_assert_clause() const;
LibJS: Do not save state for peeking at the next token from the lexer This saves having to save and load the parser state. This could give an incorrect token in some cases where the parser communicates to the lexer. However this is not applicable in any of the current usages and this would require one to parse the current token as normal which is exactly what you don't want to do in that scenario.
2021-10-08 00:38:24 +02:00
bool match_declaration() const;
bool try_match_let_declaration() const;
bool match_variable_declaration() const;
LibJS: Rework Identifier parsing to match the spec more closely
2021-07-11 15:34:55 +04:30
bool match_identifier() const;
LibJS: Allow reserved words as keys in object expressions.
2020-04-18 20:31:27 +02:00
bool match_identifier_name() const;
LibJS: Initial class implementation; allow super expressions in object literal methods; add EnvrionmentRecord fields and methods to LexicalEnvironment Adding EnvrionmentRecord's fields and methods lets us throw an exception when |this| is not initialized, which occurs when the super constructor in a derived class has not yet been called, or when |this| has already been initialized (the super constructor was already called).
2020-06-08 13:31:21 -05:00
bool match_property_key() const;
LibJS: Add parsing and evaluation of private fields and methods
2021-10-12 22:45:52 +02:00
bool is_private_identifier_valid() const;
LibJS: Add Javascript lexer and parser This adds a basic Javascript lexer and parser. It can parse the currently existing demo programs. More work needs to be done to turn it into a complete parser than can parse arbitrary JS Code. The lexer outputs tokens with preceeding whitespace and comments in the trivia member. This should allow us to generate the exact source code by concatenating the generated tokens. The parser is written in a way that it always returns a complete syntax tree. Error conditions are represented as nodes in the tree. This simplifies the code and allows it to be used as an early stage parser, e.g for parsing JS documents in an IDE while editing the source code.:
2020-03-11 19:27:43 +01:00
bool match(TokenType type) const;
bool done() const;
Everywhere: Run clang-format
2022-04-01 20:58:27 +03:00
void expected(char const* what);
void syntax_error(String const& message, Optional<Position> = {});
LibJS: Add Javascript lexer and parser This adds a basic Javascript lexer and parser. It can parse the currently existing demo programs. More work needs to be done to turn it into a complete parser than can parse arbitrary JS Code. The lexer outputs tokens with preceeding whitespace and comments in the trivia member. This should allow us to generate the exact source code by concatenating the generated tokens. The parser is written in a way that it always returns a complete syntax tree. Error conditions are represented as nodes in the tree. This simplifies the code and allows it to be used as an early stage parser, e.g for parsing JS documents in an IDE while editing the source code.:
2020-03-11 19:27:43 +01:00
Token consume();
LibJS: Rework Identifier parsing to match the spec more closely
2021-07-11 15:34:55 +04:30
Token consume_identifier();
Token consume_identifier_reference();
LibJS: Add Javascript lexer and parser This adds a basic Javascript lexer and parser. It can parse the currently existing demo programs. More work needs to be done to turn it into a complete parser than can parse arbitrary JS Code. The lexer outputs tokens with preceeding whitespace and comments in the trivia member. This should allow us to generate the exact source code by concatenating the generated tokens. The parser is written in a way that it always returns a complete syntax tree. Error conditions are represented as nodes in the tree. This simplifies the code and allows it to be used as an early stage parser, e.g for parsing JS documents in an IDE while editing the source code.:
2020-03-11 19:27:43 +01:00
Token consume(TokenType type);
LibJS: Unprefixed octal numbers are a syntax error in strict mode
2020-10-19 18:01:28 +01:00
Token consume_and_validate_numeric_literal();
LibJS: Implement automatic semicolon insertion
2020-04-17 15:05:58 +02:00
void consume_or_insert_semicolon();
LibJS: Add Parser save_state() and load_state() functions These functions allow us to try to parse ambiguous expressions (such as arrow function arguments in parentheses), and rewind the state of the Parser if an expression candidate failed to parse.
2020-03-30 08:24:43 -05:00
void save_state();
void load_state();
LibJS: Don't save rule start positions along with the parser state This fixes #4617. Also fixes the small problem where some save states would be leaked.
2020-12-29 16:47:39 +03:30
void discard_saved_state();
LibJS: Replace 'size_t line, size_t column' with 'Optional<Position>' This is a bit nicer for two reasons: - The absence of line number/column information isn't based on 'values are zero' anymore but on Optional's value - When reporting syntax errors with position information other than the current token's position we had to store line and column ourselves, like this: auto foo_start_line = m_parser_state.m_current_token.line_number(); auto foo_start_column = m_parser_state.m_current_token.line_column(); ... syntax_error("...", foo_start_line, foo_start_column); Which now becomes: auto foo_start= position(); ... syntax_error("...", foo_start); This makes it easier to report correct positions for syntax errors that only emerge a few tokens later :^)
2020-11-02 21:03:19 +00:00
Position position() const;
LibJS: Add Javascript lexer and parser This adds a basic Javascript lexer and parser. It can parse the currently existing demo programs. More work needs to be done to turn it into a complete parser than can parse arbitrary JS Code. The lexer outputs tokens with preceeding whitespace and comments in the trivia member. This should allow us to generate the exact source code by concatenating the generated tokens. The parser is written in a way that it always returns a complete syntax tree. Error conditions are represented as nodes in the tree. This simplifies the code and allows it to be used as an early stage parser, e.g for parsing JS documents in an IDE while editing the source code.:
2020-03-11 19:27:43 +01:00
LibJS: Allow member expressions in binding patterns Also allows literal string and numbers as property names in object binding patterns.
2021-09-18 01:11:32 +02:00
RefPtr<BindingPattern> synthesize_binding_pattern(Expression const& expression);
LibJS: Make parsing import and export entries follow the spec The big changes are: - Allow strings as Module{Export, Import}Name - Properly track declarations in default export statements However, the spec is a little strange in that it allows function and class declarations without a name in default export statements. This is quite hard to fully implement without rewriting more of the parser so for now this behavior is emulated by faking things with function and class expressions. See the comments in parse_export_statement for details on the hacks and where it goes wrong.
2022-01-16 23:51:28 +01:00
Token next_token(size_t steps = 1) const;
LibJS: Fix small issues in parser - Fix some places where escaped keywords are (not) allowed. - Be more strict about parameters for functions with 'use strict'. - Fix that expressions statements allowed functions and classes. - Fix that class expressions were not allowed. - Added a new next_token() method for checking the look ahead. - Fix that continue labels could jump to non iterating targets. - Fix that generator functions cannot be declared in if statements.
2021-08-28 17:04:37 +02:00
LibJS: Make more use of Token::flystring_value() This patch makes check_identifier_name_for_assignment_validity() take a FlyString instead of a StringView. We then exploit this by passing FlyString in more places via flystring_value(). This gives a ~1% speedup when parsing the largest Discord JS file.
2022-02-13 13:34:26 +01:00
void check_identifier_name_for_assignment_validity(FlyString const&, bool force_strict = false);
LibJS: Rework Identifier parsing to match the spec more closely
2021-07-11 15:34:55 +04:30
Everywhere: Run clang-format
2022-04-01 20:58:27 +03:00
bool try_parse_arrow_function_expression_failed_at_position(Position const&) const;
void set_try_parse_arrow_function_expression_failed_at_position(Position const&, bool);
LibJS: Memoize failed calls of try_parse_arrow_function_expression()
2021-04-11 22:41:51 +02:00
LibJS: Handle escaped keywords in more cases and handle 'await' labels
2021-09-18 23:02:50 +02:00
bool match_invalid_escaped_keyword() const;
LibJS: Make scoping follow the spec Before this we used an ad-hoc combination of references and 'variables' stored in a hashmap. This worked in most cases but is not spec like. Additionally hoisting, dynamically naming functions and scope analysis was not done properly. This patch fixes all of that by: - Implement BindingInitialization for destructuring assignment. - Implementing a new ScopePusher which tracks the lexical and var scoped declarations. This hoists functions to the top level if no lexical declaration name overlaps. Furthermore we do checking of redeclarations in the ScopePusher now requiring less checks all over the place. - Add methods for parsing the directives and statement lists instead of having that code duplicated in multiple places. This allows declarations to pushed to the appropriate scope more easily. - Remove the non spec way of storing 'variables' in DeclarativeEnvironment and make Reference follow the spec instead of checking both the bindings and 'variables'. - Remove all scoping related things from the Interpreter. And instead use environments as specified by the spec. This also includes fixing that NativeFunctions did not produce a valid FunctionEnvironment which could cause issues with callbacks and eval. All FunctionObjects now have a valid NewFunctionEnvironment implementation. - Remove execute_statements from Interpreter and instead use ASTNode::execute everywhere this simplifies AST.cpp as you no longer need to worry about which method to call. - Make ScopeNodes setup their own environment. This uses four different methods specified by the spec {Block, Function, Eval, Global}DeclarationInstantiation with the annexB extensions. - Implement and use NamedEvaluation where specified. Additionally there are fixes to things exposed by these changes to eval, {for, for-in, for-of} loops and assignment. Finally it also fixes some tests in test-js which where passing before but not now that we have correct behavior :^).
2021-09-22 12:44:56 +02:00
bool parse_directive(ScopeNode& body);
void parse_statement_list(ScopeNode& output_node, AllowLabelledFunction allow_labelled_functions = AllowLabelledFunction::No);
LibJS: Make parsing import and export entries follow the spec The big changes are: - Allow strings as Module{Export, Import}Name - Properly track declarations in default export statements However, the spec is a little strange in that it allows function and class declarations without a name in default export statements. This is quite hard to fully implement without rewriting more of the parser so for now this behavior is emulated by faking things with function and class expressions. See the comments in parse_export_statement for details on the hacks and where it goes wrong.
2022-01-16 23:51:28 +01:00
FlyString consume_string_value();
ModuleRequest parse_module_request();
LibJS: Make scoping follow the spec Before this we used an ad-hoc combination of references and 'variables' stored in a hashmap. This worked in most cases but is not spec like. Additionally hoisting, dynamically naming functions and scope analysis was not done properly. This patch fixes all of that by: - Implement BindingInitialization for destructuring assignment. - Implementing a new ScopePusher which tracks the lexical and var scoped declarations. This hoists functions to the top level if no lexical declaration name overlaps. Furthermore we do checking of redeclarations in the ScopePusher now requiring less checks all over the place. - Add methods for parsing the directives and statement lists instead of having that code duplicated in multiple places. This allows declarations to pushed to the appropriate scope more easily. - Remove the non spec way of storing 'variables' in DeclarativeEnvironment and make Reference follow the spec instead of checking both the bindings and 'variables'. - Remove all scoping related things from the Interpreter. And instead use environments as specified by the spec. This also includes fixing that NativeFunctions did not produce a valid FunctionEnvironment which could cause issues with callbacks and eval. All FunctionObjects now have a valid NewFunctionEnvironment implementation. - Remove execute_statements from Interpreter and instead use ASTNode::execute everywhere this simplifies AST.cpp as you no longer need to worry about which method to call. - Make ScopeNodes setup their own environment. This uses four different methods specified by the spec {Block, Function, Eval, Global}DeclarationInstantiation with the annexB extensions. - Implement and use NamedEvaluation where specified. Additionally there are fixes to things exposed by these changes to eval, {for, for-in, for-of} loops and assignment. Finally it also fixes some tests in test-js which where passing before but not now that we have correct behavior :^).
2021-09-22 12:44:56 +02:00
LibJS: Track source positions all the way down to exceptions This makes exceptions have a trace of source positions too, which could probably be helpful in making fancier error tracebacks.
2020-12-28 20:45:22 +03:30
struct RulePosition {
LibJS: `save_state()' before creating a RulePosition Fixes #4617.
2020-12-29 08:42:02 +03:30
AK_MAKE_NONCOPYABLE(RulePosition);
AK_MAKE_NONMOVABLE(RulePosition);
public:
LibJS: Track source positions all the way down to exceptions This makes exceptions have a trace of source positions too, which could probably be helpful in making fancier error tracebacks.
2020-12-28 20:45:22 +03:30
RulePosition(Parser& parser, Position position)
: m_parser(parser)
, m_position(position)
{
LibJS: Don't save rule start positions along with the parser state This fixes #4617. Also fixes the small problem where some save states would be leaked.
2020-12-29 16:47:39 +03:30
m_parser.m_rule_starts.append(position);
LibJS: Track source positions all the way down to exceptions This makes exceptions have a trace of source positions too, which could probably be helpful in making fancier error tracebacks.
2020-12-28 20:45:22 +03:30
}
~RulePosition()
{
LibJS: Don't save rule start positions along with the parser state This fixes #4617. Also fixes the small problem where some save states would be leaked.
2020-12-29 16:47:39 +03:30
auto last = m_parser.m_rule_starts.take_last();
Everywhere: Rename ASSERT => VERIFY (...and ASSERT_NOT_REACHED => VERIFY_NOT_REACHED) Since all of these checks are done in release builds as well, let's rename them to VERIFY to prevent confusion, as everyone is used to assertions being compiled out in release. We can introduce a new ASSERT macro that is specifically for debug checks, but I'm doing this wholesale conversion first since we've accumulated thousands of these already, and it's not immediately obvious which ones are suitable for ASSERT.
2021-02-23 20:42:32 +01:00
VERIFY(last.line == m_position.line);
VERIFY(last.column == m_position.column);
LibJS: Track source positions all the way down to exceptions This makes exceptions have a trace of source positions too, which could probably be helpful in making fancier error tracebacks.
2020-12-28 20:45:22 +03:30
}
Everywhere: Run clang-format
2022-04-01 20:58:27 +03:00
Position const& position() const { return m_position; }
LibJS: Track source positions all the way down to exceptions This makes exceptions have a trace of source positions too, which could probably be helpful in making fancier error tracebacks.
2020-12-28 20:45:22 +03:30
private:
Parser& m_parser;
Position m_position;
};
[[nodiscard]] RulePosition push_start() { return { *this, position() }; }
LibJS: Add Parser save_state() and load_state() functions These functions allow us to try to parse ambiguous expressions (such as arrow function arguments in parentheses), and rewind the state of the Parser if an expression candidate failed to parse.
2020-03-30 08:24:43 -05:00
struct ParserState {
LibJS: Rename Parser::m_parser_state => m_state Also remove the "m_" prefix from all the public data members.
2021-06-19 14:43:09 +02:00
Lexer lexer;
Token current_token;
Vector<Error> errors;
LibJS: Add missing initializer for ParserState::m_current_scope_pusher
2021-10-08 01:55:24 +02:00
ScopePusher* current_scope_pusher { nullptr };
LibJS: Track which Identifier nodes refer to function arguments This patch adds an "argument index" field to Identifier AST nodes. If the Identifier refers to a function parameter in the currently open function scope, we stash the index of the parameter here. This will allow us to implement much faster direct access to function argument variables.
2021-06-14 09:30:43 +02:00
LibJS: Allow multiple labels on the same statement Since there are only a number of statements where labels can actually be used we now also only store labels when necessary. Also now tracks the first continue usage of a label since this might not be valid but that can only be determined after we have parsed the statement. Also ensures the correct error does not get wiped by load_state.
2021-09-18 23:01:54 +02:00
HashMap<StringView, Optional<Position>> labels_in_scope;
LibJS: Add parsing and evaluation of private fields and methods
2021-10-12 22:45:52 +02:00
HashTable<StringView>* referenced_private_names { nullptr };
LibJS: Rename Parser::m_parser_state => m_state Also remove the "m_" prefix from all the public data members.
2021-06-19 14:43:09 +02:00
bool strict_mode { false };
bool allow_super_property_lookup { false };
bool allow_super_constructor_call { false };
bool in_function_context { false };
LibJS: Add missing steps and spec comments to PerformEval While adding spec comments to PerformEval, I noticed we were missing multiple steps. Namely, these were: - Checking if the host will allow us to compile the string (allowing LibWeb to perform CSP for eval) - The parser's initial state depending on the environment around us on direct eval: - Allowing new.target via eval in functions - Allowing super calls and super properties via eval in classes - Disallowing the use of the arguments object in class field initializers at eval's parse time - Setting ScriptOrModule of eval's execution context The spec allows us to apply the additional parsing steps in any order. The method I have gone with is passing in a struct to the parser's constructor, which overrides the parser's initial state to (dis)allow the things stated above from the get-go.
2022-04-10 00:55:45 +01:00
bool in_eval_function_context { false }; // This controls if we allow new.target or not. Note that eval("return") is not allowed, so we have to have a separate state variable for eval.
LibJS: Disallow yield expression correctly in formal parameters And add ZERO WIDTH NO BREAK SPACE to valid whitespace.
2021-08-21 11:31:36 +02:00
bool in_formal_parameter_context { false };
LibJS: Rename Parser::m_parser_state => m_state Also remove the "m_" prefix from all the public data members.
2021-06-19 14:43:09 +02:00
bool in_generator_function_context { false };
LibJS: Rename in_async_function_context to await_expression_is_valid Since await can be valid in module code which is not an async function the old name is not really representative for the usage.
2021-11-26 23:50:32 +01:00
bool await_expression_is_valid { false };
LibJS: Rename Parser::m_parser_state => m_state Also remove the "m_" prefix from all the public data members.
2021-06-19 14:43:09 +02:00
bool in_arrow_function_context { false };
bool in_break_context { false };
bool in_continue_context { false };
bool string_legacy_octal_escape_sequence_in_scope { false };
LibJS: Add support for public fields in classes
2021-08-28 17:11:05 +02:00
bool in_class_field_initializer { false };
LibJS: Add support for await expressions
2021-11-09 22:52:21 +02:00
bool in_class_static_init_block { false };
LibJS: Add an optimization to avoid needless arguments object creation This gives FunctionNode a "might need arguments object" boolean flag and sets it based on the simplest possible heuristic for this: if we encounter an identifier called "arguments" or "eval" up to the next (nested) function declaration or expression, we won't need an arguments object. Otherwise, we *might* need one - the final decision is made in the FunctionDeclarationInstantiation AO. Now, this is obviously not perfect. Even if you avoid eval, something like `foo.arguments` will still trigger a false positive - but it's a start and already massively cuts down on needlessly allocated objects, especially in real-world code that is often minified, and so a full "arguments" identifier will be an actual arguments object more often than not. To illustrate the actual impact of this change, here's the number of allocated arguments objects during a full test-js run: Before: - Unmapped arguments objects: 78765 - Mapped arguments objects: 2455 After: - Unmapped arguments objects: 18 - Mapped arguments objects: 37 This results in a ~5% speedup of test-js on my Linux host machine, and about 3.5% on i686 Serenity in QEMU (warm runs, average of 5). The following microbenchmark (calling an empty function 1M times) runs 25% faster on Linux and 45% on Serenity: function foo() {} for (var i = 0; i < 1_000_000; ++i) foo(); test262 reports no changes in either direction, apart from a speedup :^)
2021-10-05 08:44:58 +01:00
bool function_might_need_arguments_object { false };
LibJS: Add Parser save_state() and load_state() functions These functions allow us to try to parse ambiguous expressions (such as arrow function arguments in parentheses), and rewind the state of the Parser if an expression candidate failed to parse.
2020-03-30 08:24:43 -05:00
LibJS: Add a mode to parse JS as a module In a module strict mode should be enabled at the start of parsing and we allow import and export statements.
2021-08-14 17:30:37 +02:00
ParserState(Lexer, Program::Type);
LibJS: Add Parser save_state() and load_state() functions These functions allow us to try to parse ambiguous expressions (such as arrow function arguments in parentheses), and rewind the state of the Parser if an expression candidate failed to parse.
2020-03-30 08:24:43 -05:00
};
LibJS: Memoize failed calls of try_parse_arrow_function_expression()
2021-04-11 22:41:51 +02:00
class PositionKeyTraits {
public:
Everywhere: Run clang-format
2022-04-01 20:58:27 +03:00
static int hash(Position const& position)
LibJS: Memoize failed calls of try_parse_arrow_function_expression()
2021-04-11 22:41:51 +02:00
{
return int_hash(position.line) ^ int_hash(position.column);
}
Everywhere: Run clang-format
2022-04-01 20:58:27 +03:00
static bool equals(Position const& a, Position const& b)
LibJS: Memoize failed calls of try_parse_arrow_function_expression()
2021-04-11 22:41:51 +02:00
{
return a.column == b.column && a.line == b.line;
}
};
LibJS: Don't save rule start positions along with the parser state This fixes #4617. Also fixes the small problem where some save states would be leaked.
2020-12-29 16:47:39 +03:30
Vector<Position> m_rule_starts;
LibJS: Rename Parser::m_parser_state => m_state Also remove the "m_" prefix from all the public data members.
2021-06-19 14:43:09 +02:00
ParserState m_state;
LibJS: Keep track of file names, lines and columns inside the AST
2021-02-28 10:42:34 +01:00
FlyString m_filename;
LibJS: Add function default arguments Adds the ability for function arguments to have default values. This works for standard functions as well as arrow functions. Default values are not printed in a <function>.toString() call, as nodes cannot print their source string representation.
2020-05-02 11:46:39 -07:00
Vector<ParserState> m_saved_state;
LibJS: Memoize failed calls of try_parse_arrow_function_expression()
2021-04-11 22:41:51 +02:00
HashMap<Position, TokenMemoization, PositionKeyTraits> m_token_memoizations;
LibJS: Add a mode to parse JS as a module In a module strict mode should be enabled at the start of parsing and we allow import and export statements.
2021-08-14 17:30:37 +02:00
Program::Type m_program_type;
LibJS: Add Javascript lexer and parser This adds a basic Javascript lexer and parser. It can parse the currently existing demo programs. More work needs to be done to turn it into a complete parser than can parse arbitrary JS Code. The lexer outputs tokens with preceeding whitespace and comments in the trivia member. This should allow us to generate the exact source code by concatenating the generated tokens. The parser is written in a way that it always returns a complete syntax tree. Error conditions are represented as nodes in the tree. This simplifies the code and allows it to be used as an early stage parser, e.g for parsing JS documents in an IDE while editing the source code.:
2020-03-11 19:27:43 +01:00
};
}
Reference in a new issue Copy permalink