gh-139516: Fix lambda colon start format spec in f-string in tokenizer (GH-139657)
(cherry picked from commit 539461d9ec)
Co-authored-by: Tomasz Pytel <tompytel@gmail.com>
gh-134752: Improve speed of test_tokenize.StringPrefixTest.test_prefixes. (GH-134766)
(cherry picked from commit 579686d9fb)
Co-authored-by: Eric V. Smith <ericvsmith@users.noreply.github.com>
gh-134675: Add t-string prefixes to tokenizer module, lexical analysis doc, and add a test to make sure we catch this error in the future. (GH-134734)
* Add t-string prefixes to _all_string_prefixes, and add a test to make sure we catch this error in the future.
* Update lexical analysis docs for t-string prefixes.
(cherry picked from commit 08c78e02fa)
Co-authored-by: Eric V. Smith <ericvsmith@users.noreply.github.com>
gh-134582: Fix t-strings untokenize() roundtrip removing space between braces (GH-134603)
(cherry picked from commit 52509cc94b)
Co-authored-by: Loïc Simon <loic.pano@gmail.com>
* Implement C recursion protection with limit pointers for Linux, MacOS and Windows
* Remove calls to PyOS_CheckStack
* Add stack protection to parser
* Make tests more robust to low stacks
* Improve error messages for stack overflow
Revert "GH-91079: Implement C stack limits using addresses, not counters. (GH-130007)" for now
Unfortunatlely, the change broke some buildbots.
This reverts commit 2498c22fa0.
* Implement C recursion protection with limit pointers
* Remove calls to PyOS_CheckStack
* Add stack protection to parser
* Make tests more robust to low stacks
* Improve error messages for stack overflow
This adds a new command line argument, `--parallel-threads` to the
regression test runner to allow it to run individual tests in multiple
threads in parallel in order to find multithreading bugs.
Some tests pass when run with `--parallel-threads`, but there's still
more work before the entire suite passes.
This commit replaces the Python implementation of the tokenize module with an implementation
that reuses the real C tokenizer via a private extension module. The tokenize module now implements
a compatibility layer that transforms tokens from the C tokenizer into Python tokenize tokens for backward
compatibility.
As the C tokenizer does not emit some tokens that the Python tokenizer provides (such as comments and non-semantic newlines), a new special mode has been added to the C tokenizer mode that currently is only used via
the extension module that exposes it to the Python layer. This new mode forces the C tokenizer to emit these new extra tokens and add the appropriate metadata that is needed to match the old Python implementation.
Co-authored-by: Pablo Galindo <pablogsal@gmail.com>
