cpython/Parser/lexer/lexer.h

#ifndef _PY_LEXER_LEXER_H_
#define _PY_LEXER_LEXER_H_

#include "state.h"

int _PyLexer_update_ftstring_expr(struct tok_state *tok, char cur);

int _PyTokenizer_Get(struct tok_state *, struct token *);

#endif
gh-104169: Refactor tokenizer into lexer and wrappers (#110684) * The lexer, which include the actual lexeme producing logic, goes into the `lexer` directory. * The wrappers, one wrapper per input mode (file, string, utf-8, and readline), go into the `tokenizer` directory and include logic for creating a lexer instance and managing the buffer for different modes. --------- Co-authored-by: Pablo Galindo <pablogsal@gmail.com> Co-authored-by: blurb-it[bot] <43283697+blurb-it[bot]@users.noreply.github.com> 2023-10-11 17:14:44 +02:00			`#ifndef _PY_LEXER_LEXER_H_`
			`#define _PY_LEXER_LEXER_H_`

			`#include "state.h"`

gh-132661: Implement PEP 750 (#132662) Co-authored-by: Lysandros Nikolaou <lisandrosnik@gmail.com> Co-authored-by: Bénédikt Tran <10796600+picnixz@users.noreply.github.com> Co-authored-by: Adam Turner <9087854+AA-Turner@users.noreply.github.com> Co-authored-by: Hugo van Kemenade <1324225+hugovk@users.noreply.github.com> Co-authored-by: Wingy <git@wingysam.xyz> Co-authored-by: Koudai Aono <koxudaxi@gmail.com> Co-authored-by: Dave Peck <davepeck@gmail.com> Co-authored-by: Terry Jan Reedy <tjreedy@udel.edu> Co-authored-by: Paul Everitt <pauleveritt@me.com> Co-authored-by: sobolevn <mail@sobolevn.me> 2025-04-30 11:46:41 +02:00			`int _PyLexer_update_ftstring_expr(struct tok_state *tok, char cur);`
gh-104169: Refactor tokenizer into lexer and wrappers (#110684) * The lexer, which include the actual lexeme producing logic, goes into the `lexer` directory. * The wrappers, one wrapper per input mode (file, string, utf-8, and readline), go into the `tokenizer` directory and include logic for creating a lexer instance and managing the buffer for different modes. --------- Co-authored-by: Pablo Galindo <pablogsal@gmail.com> Co-authored-by: blurb-it[bot] <43283697+blurb-it[bot]@users.noreply.github.com> 2023-10-11 17:14:44 +02:00
			`int _PyTokenizer_Get(struct tok_state , struct token );`

			`#endif`