Lysandros Nikolaou 01481f2dc1 gh-104169: Refactor tokenizer into lexer and wrappers (#110684) ... * The lexer, which include the actual lexeme producing logic, goes into the `lexer` directory. * The wrappers, one wrapper per input mode (file, string, utf-8, and readline), go into the `tokenizer` directory and include logic for creating a lexer instance and managing the buffer for different modes. --------- Co-authored-by: Pablo Galindo <pablogsal@gmail.com> Co-authored-by: blurb-it[bot] <43283697+blurb-it[bot]@users.noreply.github.com>		2023-10-11 15:14:44 +00:00
..
file_tokenizer.c	gh-104169: Refactor tokenizer into lexer and wrappers (#110684)	2023-10-11 15:14:44 +00:00
helpers.c	gh-104169: Refactor tokenizer into lexer and wrappers (#110684)	2023-10-11 15:14:44 +00:00
helpers.h	gh-104169: Refactor tokenizer into lexer and wrappers (#110684)	2023-10-11 15:14:44 +00:00
readline_tokenizer.c	gh-104169: Refactor tokenizer into lexer and wrappers (#110684)	2023-10-11 15:14:44 +00:00
string_tokenizer.c	gh-104169: Refactor tokenizer into lexer and wrappers (#110684)	2023-10-11 15:14:44 +00:00
tokenizer.h	gh-104169: Refactor tokenizer into lexer and wrappers (#110684)	2023-10-11 15:14:44 +00:00
utf8_tokenizer.c	gh-104169: Refactor tokenizer into lexer and wrappers (#110684)	2023-10-11 15:14:44 +00:00