5 commits

Author	SHA1	Message	Date
Miss Islington (bot)	6a40e1c8d0	[3.13] gh-150207: Raise MemoryError on tokenizer allocation failure instead of crashing (GH-150275) (#150998) ... gh-150207: Raise MemoryError on tokenizer allocation failure instead of crashing (GH-150275) (cherry picked from commit 262625fa30) Co-authored-by: Grant Herman <grantlouisherman041@gmail.com>	2026-06-06 03:47:00 +01:00
Stan Ulbrych	5577bd1298	[3.13] gh-144872: fix heap buffer overflow _PyTokenizer_ensure_utf8 (GH-144807) (#145441) ... (cherry picked from commit 3fc945df22) Co-authored-by: AdamKorcz <44787359+AdamKorcz@users.noreply.github.com>	2026-03-02 23:00:59 +00:00
Pablo Galindo Salgado	2daece9903	[3.13] gh-145234: Normalize decoded CR in string tokenizer (GH-145281) (#145312)	2026-02-28 01:38:36 +00:00
Filipe Laíns	23645420dc	GH-110749: fix unistd.h import in file_tokenizer.c (#110750)	2023-10-12 07:52:13 +02:00
Lysandros Nikolaou	01481f2dc1	gh-104169: Refactor tokenizer into lexer and wrappers (#110684) ... * The lexer, which include the actual lexeme producing logic, goes into the `lexer` directory. * The wrappers, one wrapper per input mode (file, string, utf-8, and readline), go into the `tokenizer` directory and include logic for creating a lexer instance and managing the buffer for different modes. --------- Co-authored-by: Pablo Galindo <pablogsal@gmail.com> Co-authored-by: blurb-it[bot] <43283697+blurb-it[bot]@users.noreply.github.com>	2023-10-11 15:14:44 +00:00