Stowage/ladybird
2
0
Fork 0
mirror of https://github.com/LadybirdBrowser/ladybird.git synced 2025-11-06 16:20:58 +00:00
Code Issues Projects Releases Packages Wiki Activity
ladybird/Userland/Libraries/LibSQL/AST/Lexer.h

62 lines
1.6 KiB
C
Raw Normal View History

LibSQL: Introduce a SQL library with a tokenizer LibSQL aims to be a SQLite clone for SerenityOS. Step 1 is creating a tokenizer to lex SQL tokens. This lexer is heavily influenced by the LibJS lexer.
2021-04-18 17:35:40 -04:00
/*
Everywhere: Update copyrights with my new serenityos.org e-mail :^)
2022-01-31 13:07:22 -05:00
* Copyright (c) 2021, Tim Flynn <trflynn89@serenityos.org>
LibSQL: Make lexer and parser more standard SQL compliant SQL was standardized before there was consensus on sane language syntax constructs had evolved. The language is mostly case-insensitive, with unquoted text converted to upper case. Identifiers can include lower case characters and other 'special' characters by enclosing the identifier with double quotes. A double quote is escaped by doubling it. Likewise, a single quote in a literal string is escaped by doubling it. All this means that the strategy used in the lexer, where a token's value is a StringView 'window' on the source string, does not work, because the value needs to be massaged before being handed to the parser. Therefore a token now has a String containing its value. Given the limited lifetime of a token, this is acceptable overhead. Not doing this means that for example quote removal and double quote escaping would need to be done in the parser or in AST node construction, which would spread lexing basically all over the place. Which would be suboptimal. There was some impact on the sql utility and SyntaxHighlighter component which was addressed by storing the token's end position together with the start position in order to properly highlight it. Finally, reviewing the tests for parsing numeric literals revealed an inconsistency in which tokens we accept or reject: `1a` is accepted but `1e` is rejected. Related to this is the fate of `0x`. Added a FIXME reminding us to address this.
2021-06-21 11:20:09 -04:00
* Copyright (c) 2021, Jan de Visser <jan@de-visser.net>
LibSQL: Introduce a SQL library with a tokenizer LibSQL aims to be a SQLite clone for SerenityOS. Step 1 is creating a tokenizer to lex SQL tokens. This lexer is heavily influenced by the LibJS lexer.
2021-04-18 17:35:40 -04:00
*
Everything: Move to SPDX license identifiers in all files. SPDX License Identifiers are a more compact / standardized way of representing file license information. See: https://spdx.dev/resources/use/#identifiers This was done with the `ambr` search and replace tool. ambr --no-parent-ignore --key-from-file --rep-from-file key.txt rep.txt *
2021-04-22 01:24:48 -07:00
* SPDX-License-Identifier: BSD-2-Clause
LibSQL: Introduce a SQL library with a tokenizer LibSQL aims to be a SQLite clone for SerenityOS. Step 1 is creating a tokenizer to lex SQL tokens. This lexer is heavily influenced by the LibJS lexer.
2021-04-18 17:35:40 -04:00
*/
#pragma once
#include "Token.h"
Everywhere: Rename {Deprecated => Byte}String This commit un-deprecates DeprecatedString, and repurposes it as a byte string. As the null state has already been removed, there are no other particularly hairy blockers in repurposing this type as a byte string (what it _really_ is). This commit is auto-generated: $ xs=$(ack -l \bDeprecatedString\b\|deprecated_string AK Userland \ Meta Ports Ladybird Tests Kernel) $ perl -pie 's/\bDeprecatedString\b/ByteString/g; s/deprecated_string/byte_string/g' $xs $ clang-format --style=file -i \ $(git diff --name-only | grep \.cpp\|\.h) $ gn format $(git ls-files '*.gn' '*.gni')
2023-12-16 17:49:34 +03:30
#include <AK/ByteString.h>
LibSQL: Introduce a SQL library with a tokenizer LibSQL aims to be a SQLite clone for SerenityOS. Step 1 is creating a tokenizer to lex SQL tokens. This lexer is heavily influenced by the LibJS lexer.
2021-04-18 17:35:40 -04:00
#include <AK/HashMap.h>
#include <AK/StringView.h>
LibSQL: Move Lexer and Parser machinery to AST directory The SQL engine is expected to be a fairly sizeable piece of software. Therefore we're starting to restructure the codebase for growth.
2021-06-21 10:57:44 -04:00
namespace SQL::AST {
LibSQL: Introduce a SQL library with a tokenizer LibSQL aims to be a SQLite clone for SerenityOS. Step 1 is creating a tokenizer to lex SQL tokens. This lexer is heavily influenced by the LibJS lexer.
2021-04-18 17:35:40 -04:00
class Lexer {
public:
explicit Lexer(StringView source);
Token next();
private:
LibSQL: Make lexer and parser more standard SQL compliant SQL was standardized before there was consensus on sane language syntax constructs had evolved. The language is mostly case-insensitive, with unquoted text converted to upper case. Identifiers can include lower case characters and other 'special' characters by enclosing the identifier with double quotes. A double quote is escaped by doubling it. Likewise, a single quote in a literal string is escaped by doubling it. All this means that the strategy used in the lexer, where a token's value is a StringView 'window' on the source string, does not work, because the value needs to be massaged before being handed to the parser. Therefore a token now has a String containing its value. Given the limited lifetime of a token, this is acceptable overhead. Not doing this means that for example quote removal and double quote escaping would need to be done in the parser or in AST node construction, which would spread lexing basically all over the place. Which would be suboptimal. There was some impact on the sql utility and SyntaxHighlighter component which was addressed by storing the token's end position together with the start position in order to properly highlight it. Finally, reviewing the tests for parsing numeric literals revealed an inconsistency in which tokens we accept or reject: `1a` is accepted but `1e` is rejected. Related to this is the fate of `0x`. Added a FIXME reminding us to address this.
2021-06-21 11:20:09 -04:00
void consume(StringBuilder* = nullptr);
LibSQL: Introduce a SQL library with a tokenizer LibSQL aims to be a SQLite clone for SerenityOS. Step 1 is creating a tokenizer to lex SQL tokens. This lexer is heavily influenced by the LibJS lexer.
2021-04-18 17:35:40 -04:00
bool consume_whitespace_and_comments();
LibSQL: Make lexer and parser more standard SQL compliant SQL was standardized before there was consensus on sane language syntax constructs had evolved. The language is mostly case-insensitive, with unquoted text converted to upper case. Identifiers can include lower case characters and other 'special' characters by enclosing the identifier with double quotes. A double quote is escaped by doubling it. Likewise, a single quote in a literal string is escaped by doubling it. All this means that the strategy used in the lexer, where a token's value is a StringView 'window' on the source string, does not work, because the value needs to be massaged before being handed to the parser. Therefore a token now has a String containing its value. Given the limited lifetime of a token, this is acceptable overhead. Not doing this means that for example quote removal and double quote escaping would need to be done in the parser or in AST node construction, which would spread lexing basically all over the place. Which would be suboptimal. There was some impact on the sql utility and SyntaxHighlighter component which was addressed by storing the token's end position together with the start position in order to properly highlight it. Finally, reviewing the tests for parsing numeric literals revealed an inconsistency in which tokens we accept or reject: `1a` is accepted but `1e` is rejected. Related to this is the fate of `0x`. Added a FIXME reminding us to address this.
2021-06-21 11:20:09 -04:00
bool consume_numeric_literal(StringBuilder&);
bool consume_string_literal(StringBuilder&);
bool consume_quoted_identifier(StringBuilder&);
bool consume_blob_literal(StringBuilder&);
bool consume_exponent(StringBuilder&);
bool consume_hexadecimal_number(StringBuilder&);
LibSQL: Introduce a SQL library with a tokenizer LibSQL aims to be a SQLite clone for SerenityOS. Step 1 is creating a tokenizer to lex SQL tokens. This lexer is heavily influenced by the LibJS lexer.
2021-04-18 17:35:40 -04:00
bool match(char a, char b) const;
bool is_identifier_start() const;
bool is_identifier_middle() const;
bool is_numeric_literal_start() const;
LibSQL: Lex string and blob literals Blob literals are the same as string literals except prefixed with an 'x' or 'X'.
2021-04-20 13:29:06 -04:00
bool is_string_literal_start() const;
bool is_string_literal_end() const;
LibSQL: Make lexer and parser more standard SQL compliant SQL was standardized before there was consensus on sane language syntax constructs had evolved. The language is mostly case-insensitive, with unquoted text converted to upper case. Identifiers can include lower case characters and other 'special' characters by enclosing the identifier with double quotes. A double quote is escaped by doubling it. Likewise, a single quote in a literal string is escaped by doubling it. All this means that the strategy used in the lexer, where a token's value is a StringView 'window' on the source string, does not work, because the value needs to be massaged before being handed to the parser. Therefore a token now has a String containing its value. Given the limited lifetime of a token, this is acceptable overhead. Not doing this means that for example quote removal and double quote escaping would need to be done in the parser or in AST node construction, which would spread lexing basically all over the place. Which would be suboptimal. There was some impact on the sql utility and SyntaxHighlighter component which was addressed by storing the token's end position together with the start position in order to properly highlight it. Finally, reviewing the tests for parsing numeric literals revealed an inconsistency in which tokens we accept or reject: `1a` is accepted but `1e` is rejected. Related to this is the fate of `0x`. Added a FIXME reminding us to address this.
2021-06-21 11:20:09 -04:00
bool is_quoted_identifier_start() const;
bool is_quoted_identifier_end() const;
LibSQL: Lex string and blob literals Blob literals are the same as string literals except prefixed with an 'x' or 'X'.
2021-04-20 13:29:06 -04:00
bool is_blob_literal_start() const;
LibSQL: Introduce a SQL library with a tokenizer LibSQL aims to be a SQLite clone for SerenityOS. Step 1 is creating a tokenizer to lex SQL tokens. This lexer is heavily influenced by the LibJS lexer.
2021-04-18 17:35:40 -04:00
bool is_line_comment_start() const;
bool is_block_comment_start() const;
bool is_block_comment_end() const;
bool is_line_break() const;
bool is_eof() const;
Everywhere: Rename {Deprecated => Byte}String This commit un-deprecates DeprecatedString, and repurposes it as a byte string. As the null state has already been removed, there are no other particularly hairy blockers in repurposing this type as a byte string (what it _really_ is). This commit is auto-generated: $ xs=$(ack -l \bDeprecatedString\b\|deprecated_string AK Userland \ Meta Ports Ladybird Tests Kernel) $ perl -pie 's/\bDeprecatedString\b/ByteString/g; s/deprecated_string/byte_string/g' $xs $ clang-format --style=file -i \ $(git diff --name-only | grep \.cpp\|\.h) $ gn format $(git ls-files '*.gn' '*.gni')
2023-12-16 17:49:34 +03:30
static HashMap<ByteString, TokenType> s_keywords;
LibSQL: Introduce a SQL library with a tokenizer LibSQL aims to be a SQLite clone for SerenityOS. Step 1 is creating a tokenizer to lex SQL tokens. This lexer is heavily influenced by the LibJS lexer.
2021-04-18 17:35:40 -04:00
static HashMap<char, TokenType> s_one_char_tokens;
Everywhere: Rename {Deprecated => Byte}String This commit un-deprecates DeprecatedString, and repurposes it as a byte string. As the null state has already been removed, there are no other particularly hairy blockers in repurposing this type as a byte string (what it _really_ is). This commit is auto-generated: $ xs=$(ack -l \bDeprecatedString\b\|deprecated_string AK Userland \ Meta Ports Ladybird Tests Kernel) $ perl -pie 's/\bDeprecatedString\b/ByteString/g; s/deprecated_string/byte_string/g' $xs $ clang-format --style=file -i \ $(git diff --name-only | grep \.cpp\|\.h) $ gn format $(git ls-files '*.gn' '*.gni')
2023-12-16 17:49:34 +03:30
static HashMap<ByteString, TokenType> s_two_char_tokens;
LibSQL: Introduce a SQL library with a tokenizer LibSQL aims to be a SQLite clone for SerenityOS. Step 1 is creating a tokenizer to lex SQL tokens. This lexer is heavily influenced by the LibJS lexer.
2021-04-18 17:35:40 -04:00
StringView m_source;
size_t m_line_number { 1 };
size_t m_line_column { 0 };
char m_current_char { 0 };
Userland: Allow building SerenityOS with -funsigned-char Some of the code assumed that chars were always signed while that is not the case on ARM hosts. Also, some of the code tried to use EOF (-1) in a way similar to what fgetc() does, however instead of storing the characters in an int variable a char was used. While this seemed to work it also meant that character 0xFF would be incorrectly seen as an end-of-file. Careful reading of fgetc() reveals that fgetc() stores character data in an int where valid characters are in the range of 0-255 and the EOF value is explicitly outside of that range (usually -1).
2021-06-13 09:15:00 +02:00
bool m_eof { false };
LibSQL: Introduce a SQL library with a tokenizer LibSQL aims to be a SQLite clone for SerenityOS. Step 1 is creating a tokenizer to lex SQL tokens. This lexer is heavily influenced by the LibJS lexer.
2021-04-18 17:35:40 -04:00
size_t m_position { 0 };
};
}
Reference in a new issue Copy permalink