3 commits

Author	SHA1	Message	Date
Ramin Farajpour Cami	e6110efd03	gh-144759: Fix undefined behavior from NULL pointer arithmetic in lexer (#144788) ... Guard against NULL pointer arithmetic in `_PyLexer_remember_fstring_buffers` and `_PyLexer_restore_fstring_buffers`. When `start` or `multi_line_start` are NULL (uninitialized in tok_mode_stack[0]), performing `NULL - tok->buf` is undefined behavior. Add explicit NULL checks to store -1 as sentinel and restore NULL accordingly. Add test_lexer_buffer_realloc_with_null_start to test_repl.py that exercises the code path where the lexer buffer is reallocated while tok_mode_stack[0] has NULL start/multi_line_start pointers. This triggers _PyLexer_remember_fstring_buffers and verifies the NULL checks prevent undefined behavior.	2026-02-15 14:39:57 +00:00
Lysandros Nikolaou	60202609a2	gh-132661: Implement PEP 750 (#132662) ... Co-authored-by: Lysandros Nikolaou <lisandrosnik@gmail.com> Co-authored-by: Bénédikt Tran <10796600+picnixz@users.noreply.github.com> Co-authored-by: Adam Turner <9087854+AA-Turner@users.noreply.github.com> Co-authored-by: Hugo van Kemenade <1324225+hugovk@users.noreply.github.com> Co-authored-by: Wingy <git@wingysam.xyz> Co-authored-by: Koudai Aono <koxudaxi@gmail.com> Co-authored-by: Dave Peck <davepeck@gmail.com> Co-authored-by: Terry Jan Reedy <tjreedy@udel.edu> Co-authored-by: Paul Everitt <pauleveritt@me.com> Co-authored-by: sobolevn <mail@sobolevn.me>	2025-04-30 11:46:41 +02:00
Lysandros Nikolaou	01481f2dc1	gh-104169: Refactor tokenizer into lexer and wrappers (#110684) ... * The lexer, which include the actual lexeme producing logic, goes into the `lexer` directory. * The wrappers, one wrapper per input mode (file, string, utf-8, and readline), go into the `tokenizer` directory and include logic for creating a lexer instance and managing the buffer for different modes. --------- Co-authored-by: Pablo Galindo <pablogsal@gmail.com> Co-authored-by: blurb-it[bot] <43283697+blurb-it[bot]@users.noreply.github.com>	2023-10-11 15:14:44 +00:00