Commit graph

136 commits

Author SHA1 Message Date
Petr Viktorin
f0a0467c17
gh-143054: Disallow non-top-level Cut for now (GH-143622)
The behaviour of Cut in nested parentheses, Repeat, Opt, and similar
is somewhat chaotic. Apparently even the academic papers on PEG aren't
as clear as they could be.

And it doesn't really matter. Python only uses top-level cuts.
When that changes, we can clarify as much as necessary (and even
change the implementation to make sense for what we'll need).

Document that this is deliberately unspecified, and add a test to
make sure any decision is deliberate, tested and documented.
2026-01-13 13:21:59 +01:00
Victor Stinner
7fe1a18b77
gh-130396: Remove _Py_ReachedRecursionLimitWithMargin() function (#141951)
Move the private function to the internal C API (pycore_ceval.h).
2025-11-27 12:32:00 +01:00
Serhiy Storchaka
d8e6bdc0d0
gh-135801: Add the module parameter to compile() etc (GH-139652)
Many functions related to compiling or parsing Python code, such as
compile(), ast.parse(), symtable.symtable(),
and importlib.abc.InspectLoader.source_to_code() now allow to pass
the module name used when filtering syntax warnings.
2025-11-13 13:21:32 +02:00
Hugo van Kemenade
800d856bc7
gh-92266: Update pre-commit (#139411) 2025-10-01 08:40:54 +03:00
Pablo Galindo Salgado
0ce9fb7e3b
gh-138970: Add general metadata system to the peg generator (#138971) 2025-09-18 02:17:04 +01:00
sobolevn
06f8d7aa9f
gh-138281: Remove unused topsort and bump minimal version in peg_generator (#138487) 2025-09-04 00:27:14 +01:00
sobolevn
0d1f4e1639
gh-138281: Run ruff on Tools/peg_generator (#138282)
Co-authored-by: Adam Turner <9087854+AA-Turner@users.noreply.github.com>
2025-09-03 17:45:29 +00:00
chemelnucfin
7bca924fdb
gh-138279: Remove unused attr in RuleCollectorVisitor in parser_generator.py (#138280) 2025-08-31 14:23:40 +00:00
chemelnucfin
11217a883d
gh-138257: Fix internal RuleCollectorVisitor attribute name (#138208)
The internal `RuleCollectorVisitor` class had an attribute named "rulses"
instead of "rules" that other `GrammarVisitor` subclasses define.
2025-08-30 11:28:44 +02:00
chemelnucfin
f520f22927
parser_generator.py typo - keywods -> keywords (#135014) 2025-07-19 11:54:20 +03:00
Bénédikt Tran
754e7c9b51
gh-133157: remove usage of _Py_NO_SANITIZE_UNDEFINED in Parser/pegen.c (#134048) 2025-06-10 01:08:30 +01:00
Victor Stinner
6e80f11eb5
gh-135028: Increase parser MAXSTACK for nested parenthesis (#135031) 2025-06-03 08:40:45 +02:00
Steve Dower
986c367028
gh-133779: Revert Windows generation of pyconfig.h and go back to a static header. (GH-133966)
Extension builders must specify Py_GIL_DISABLED if they want to link to the free-threaded builds.
This was usually the case already, but this change guarantees it in all circumstances.
2025-05-19 11:35:22 +01:00
Alex Prengère
b48599b801
gh-133555: Allow regenerating the parser with Python < 3.14 (#133557)
Co-authored-by: Adam Turner <9087854+AA-Turner@users.noreply.github.com>
2025-05-08 02:28:20 +01:00
Victor Stinner
1a70f66ea8
gh-132921: Fix setuptools._distutils.dep_util deprecation (#132923)
Replace setuptools._distutils.dep_util with setuptools.modified in
peg_generator compile_c_extension().
2025-04-25 11:31:19 +00:00
Mark Shannon
014223649c
GH-130396: Use computed stack limits on linux (GH-130398)
* Implement C recursion protection with limit pointers for Linux, MacOS and Windows

* Remove calls to PyOS_CheckStack

* Add stack protection to parser

* Make tests more robust to low stacks

* Improve error messages for stack overflow
2025-02-25 09:24:48 +00:00
Petr Viktorin
ef29104f7d
GH-91079: Revert "GH-91079: Implement C stack limits using addresses, not counters. (GH-130007)" for now (GH130413)
Revert "GH-91079: Implement C stack limits using addresses, not counters. (GH-130007)" for now

Unfortunatlely, the change broke some buildbots.

This reverts commit 2498c22fa0.
2025-02-24 11:16:08 +01:00
Mark Shannon
2498c22fa0
GH-91079: Implement C stack limits using addresses, not counters. (GH-130007)
* Implement C recursion protection with limit pointers

* Remove calls to PyOS_CheckStack

* Add stack protection to parser

* Make tests more robust to low stacks

* Improve error messages for stack overflow
2025-02-19 11:44:57 +00:00
Pablo Galindo Salgado
47c5a0f307
gh-125588: Allow to regenerate the parser with Python < 3.12 (#127969)
Signed-off-by: Pablo Galindo <pablogsal@gmail.com>
2024-12-15 23:17:01 +00:00
Pablo Galindo Salgado
9dfef4e5f4
gh-125588: Teach the python PEG generator the new f-string tokens (#125589)
Signed-off-by: Pablo Galindo <pablogsal@gmail.com>
2024-10-29 23:40:12 +00:00
Mikhail Efimov
c1bdbe84c8
gh-124889: Rework Python generator cache (#125816) 2024-10-22 09:42:56 +01:00
efimov-mikhail
1f9025a4e7
gh-124889: Remove redundant artificial rules in PEG parser (#124893)
Cache in C PEG-generator reworked:
we save artificial rules in cache by Node string representation as a key instead of Node object itself.
As a result total count of artificial rules in parsers.c is lowered from 283 to 170.
More natural number ordering is used for the names of artificial rules.

Auxiliary method CCallMakerVisitor._generate_artificial_rule_call is added.
Its purpose is abstracting work with artificial rules cache.

Explicit using of "is_repeat1" kwarg is added to visit_Repeat0 and visit_Repeat1 methods.
Its slightly improve code readabitily.
2024-10-03 13:58:56 +01:00
Serhiy Storchaka
6c09b8de5c
gh-122270: Fix typos in the Py_DEBUG macro name (GH-122271) 2024-07-25 14:04:22 +03:00
Lysandros Nikolaou
348184845a
gh-120956: Avoid comparison of int to Py_ssize_t in parser (#120959) 2024-06-24 18:13:02 +02:00
yf-yang
ace2045ea6
Fix types in pegen parser generator (GH-120720) 2024-06-19 16:12:40 +02:00
Nikita Sobolev
1e5f615086
gh-116991: Improve pygen --help for python subparser (#116992) 2024-06-03 10:52:35 +03:00
Petr Viktorin
48f21b3631
gh-118235: Move RAISE_SYNTAX_ERROR actions to invalid rules and make sure they stay there (GH-119731)
The Full Grammar specification in the docs omits rule actions, so grammar rules that raise a syntax error looked like valid syntax.
This was solved in ef940de by hiding those rules in the custom syntax highlighter.

This moves all syntax-error alternatives to invalid rules, adds a validator that ensures that actions containing RAISE_SYNTAX_ERROR are in invalid rules, and reverts the syntax highlighter hack.
2024-05-30 09:27:32 +02:00
Xie Yanbo
bf08f0a5fe
Fix typos in comments (#119645) 2024-05-28 09:53:32 +02:00
trag1c
c4f9823be2
gh-118671: Updated dead ActiveState links (#118730)
Co-authored-by: blurb-it[bot] <43283697+blurb-it[bot]@users.noreply.github.com>
Co-authored-by: Kirill Podoprigora <kirill.bast9@mail.ru>
2024-05-08 01:06:38 -06:00
David Rubin
9b280ab0ab
gh-116988: Remove duplicates of annotated_rhs in the Grammar (#117004) 2024-04-24 18:16:06 +01:00
Sam Gross
5f1997896d
gh-112984: Fix link error on free-threaded Windows build (GH-114455)
The test_peg_generator test tried to link the python313_d.lib library,
which failed because the library is now named python313t_d.lib. The
underlying problem is that the "compiler" attribute was not set when
we call get_libraries() from distutils.
2024-01-23 18:05:15 +00:00
Serhiy Storchaka
b3d2427f22
gh-58032: Do not use argparse.FileType in module CLIs and scripts (GH-113649)
Open and close files manually. It prevents from leaking files,
preliminary creation of output files, and accidental closing of stdin
and stdout.
2024-01-10 15:07:19 +02:00
Donghee Na
e5cce70df7
gh-112532: Fix peg generator build.py for mimalloc build (gh-113492)
gh-112532: Fix peg generator for mimalloc build
2023-12-27 02:54:16 +09:00
Steve Dower
79dad03747
gh-111650: Ensure pyconfig.h includes Py_GIL_DISABLED on Windows (GH-112778) 2023-12-13 15:38:45 +00:00
Brett Cannon
56e59a49ae
GH-111807: Lower the parser stack depth under WASI debug builds (#112225) 2023-11-20 13:27:33 +00:00
Pablo Galindo Salgado
8316769d83
gh-110805: Fix test_peg_generator after the change in the parser for REPL tracebacks (#110814)
Fix test_peg_generator after the change in the parser for REPL tracebacks
2023-10-13 13:04:23 +02:00
Lysandros Nikolaou
17d65547df
gh-104169: Fix test_peg_generator after tokenizer refactoring (#110727)
* Fix test_peg_generator after tokenizer refactoring
* Remove references to tokenizer.c in comments etc.
2023-10-12 09:34:35 +02:00
Pablo Galindo Salgado
b28ffaa193
gh-109596: Ensure repeated rules in the grammar are not allowed and fix incorrect soft keywords (#109606) 2023-09-22 19:03:23 +01:00
Alex Waygood
697c9dcf8f
gh-108455: peg_generator: enable mypy's --warn-unreachable setting and redundant-expr error code (#109160) 2023-09-08 22:05:40 +01:00
Alex Waygood
b89b838ebc
gh-108455: peg_generator: use types-setuptools==68.1.0.1 in CI (#108697) 2023-08-31 08:41:25 +01:00
Alex Waygood
77e8f233ac
gh-108455: peg_generator: install two stubs packages before running mypy (#108637) 2023-08-29 20:14:08 +01:00
Nikita Sobolev
f8be2e262c
gh-108455: peg generator: Use strict_optional=True for grammar_parser (#108629) 2023-08-29 13:02:52 +01:00
Alex Waygood
0d140b8c5e
gh-108455: peg_generator: make the mypy config slightly stricter (#108627)
* Enable `--no-implicit-reexport`

* Enable the `truthy-bool` error code

* Enable the `ignore-without-code` error code

* Explicitly note that `--warn-unreachable` cannot yet be enabled
2023-08-29 11:23:22 +00:00
Alex Waygood
c879b9fddf
gh-108455: Simplify the peg_generator mypy config (#108620)
make it easier to see exactly which options from mypy's `--strict` mode can't currently be enabled
2023-08-29 10:24:06 +00:00
Nikita Sobolev
cf7ba83eb2
gh-108455: Run mypy on Tools/peg_generator (#108456)
Co-authored-by: Hugo van Kemenade <hugovk@users.noreply.github.com>
2023-08-28 21:04:12 +01:00
Dennis Sweeney
86617518c4
gh-108179: Add error message for parser stack overflows (#108256) 2023-08-22 08:41:50 +01:00
Pablo Galindo Salgado
da8f87b7ea
gh-107015: Remove async_hacks from the tokenizer (#107018) 2023-07-26 16:34:15 +01:00
Lysandros Nikolaou
a241003d04
gh-105191: Cleanup peg generator; keep only necessary files (#105197) 2023-06-01 17:24:15 +02:00
Lysandros Nikolaou
afa759fb80
gh-100238: Use setuptools in peg-generator and reenable tests (#104798) 2023-05-26 15:32:03 +02:00
Victor Stinner
ae00b810d1
gh-104780: Remove 2to3 program and lib2to3 module (#104781)
* Remove the Tools/scripts/2to3 script.
* Remove the Lib/test/test_lib2to3/ directory.
* Doc/tools/extensions/pyspecific.py: remove the "2to3fixer" object
  type.
* Makefile and PC/layout/main.py no longer compile lib2to3 grammar
  files.
* Update Makefile for 2to3 removal.
2023-05-23 19:40:02 +02:00