gh-131865: Properly apply exported CFLAGS for dtrace/systemtap builds (GH-131866)
When using --with-dtrace the resulting object file could be missing
specific CFLAGS exported by the build system due to the systemtap
script using specific defaults.
Exporting the CC and CFLAGS variables before the dtrace invocation
allows us to properly apply CFLAGS exported by the build system
even when cross-compiling.
The fix does not affect the dtrace invocation on Solaris/macOS.
(cherry picked from commit 0cd4befb02)
Co-authored-by: stratakis <cstratak@redhat.com>
gh-123297: Propagate LD_FLAGS to LDCXXSHARED in sysconfig (GH-123298)
(cherry picked from commit c535a49e92)
Co-authored-by: Pablo Galindo Salgado <Pablogsal@gmail.com>
xmltok_impl.c and xmltok_ns.c are _included_ in xmltok.c by the C
pre-processor.
(cherry picked from commit 606be66362)
Co-authored-by: Erlend E. Aasland <erlend@python.org>
Add tests for "import", pkgutil.resolve_name() and unittest.mock.path()
for cases when "import a.b as x" and "from a import b as x" give
different results.
(cherry picked from commit c0eaa232f6)
Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>
* gh-115421: Test that our Makefile has all needed test folders (GH-115813)
* Update the list of installed test subdirectories
---------
Co-authored-by: Nikita Sobolev <mail@sobolevn.me>
gh-108303: Move all doctest related files and tests to `Lib/test/test_doctest/` (GH-112109)
(cherry picked from commit 9c93350f58)
Co-authored-by: Nikita Sobolev <mail@sobolevn.me>
Co-authored-by: Brett Cannon <brett@python.org>
gh-112305: Fix check-clean-src to detect frozen_modules .h files. (GH-113344)
A typo left this check broken so many of us who do out-of-tree builds
were seeing strange failures due to bad `Python/frozen_modules/*.h`
files being picked up from the source tree and used at build time from
different Python versions leading to errors like:
`Fatal Python error: _PyImport_InitCore: failed to initialize importlib`
Or similar once our build got to an "invoke the interpreter"
bootstrapping step due to incorrect bytecode being embedded.
(cherry picked from commit 103c4ea274)
Co-authored-by: Gregory P. Smith <greg@krypto.org>
gh-112088: Run autoreconf in GHA check_generated_files (#112090)
The "Check if generated files are up to date" job of GitHub Actions
now runs the "autoreconf -ivf -Werror" command instead of the "make
regen-configure" command to avoid depending on the external quay.io
server.
Add Tools/build/regen-configure.sh script to regenerate the configure
with an Ubuntu container image. The
"quay.io/tiran/cpython_autoconf:271" container image
(https://github.com/tiran/cpython_autoconf) is no longer used.
(cherry picked from commit d9fd33a869)
gh-103053: Fix make check-clean-src: check "python" program (GH-110449)
"make check-clean-src" now also checks if the "python" program is
found in the source directory: fail with an error if it does exist.
(cherry picked from commit a155f9f342)
Co-authored-by: Victor Stinner <vstinner@python.org>
gh-110429: Fix race condition in "make regen-all" (GH-110433)
"make regen-pegen" now creates a temporary file called "parser.c.new"
instead of "parser.new.c". Previously, if "make clinic" was run in
parallel with "make regen-all", clinic may try but fail to open
"parser.new.c" if the temporay file was removed in the meanwhile.
(cherry picked from commit fb6c4ed2bb)
Co-authored-by: Victor Stinner <vstinner@python.org>
gh-109972: Split test_gdb.py into test_gdb package (#109977)
Split test_gdb.py file into a test_gdb package made of multiple
tests, so tests can now be run in parallel.
* Create Lib/test/test_gdb/ directory.
* Split test_gdb.py into multiple files in Lib/test/test_gdb/
directory.
* Move Lib/test/gdb_sample.py to Lib/test/test_gdb/ directory.
Update get_sample_script(): use __file__ to locate gdb_sample.py.
* Move gdb_has_frame_select() and HAS_PYUP_PYDOWN to test_misc.py.
* Explicitly skip test_gdb on Windows. Previously, test_gdb was
skipped even if gdb was available because of
gdb_has_frame_select().
(cherry picked from commit 8f324b7ecd)
gh-108303: Move `test_future` into its own test_future_stmt subdir (#109368)
(cherry picked from commit 82505dc351)
Co-authored-by: Nikita Sobolev <mail@sobolevn.me>
* gh-108303: Create Lib/test/test_dataclasses/ directory (GH-108978)
Move test_dataclasses.py and its "dataclass_*.py" modules into the
new Lib/test/test_dataclasses/ subdirectory.
(cherry picked from commit 14d6e197cc)
Co-authored-by: Victor Stinner <vstinner@python.org>
* Fix Lint job: update Lib/test/.ruff.toml
---------
Co-authored-by: Victor Stinner <vstinner@python.org>
gh-108740: Fix "make regen-all" race condition (#108741)
Fix a race condition in "make regen-all". The deepfreeze.c source and
files generated by Argument Clinic are now generated or updated
before generating "global objects". Previously, some identifiers may
miss depending on the order in which these files were generated.
* "make regen-global-objects": Make sure that deepfreeze.c is
generated and up to date, and always run "make clinic".
* "make clinic" no longer runs generate_global_objects.py script.
* "make regen-deepfreeze" now only updates deepfreeze.c (C file).
It doesn't build deepfreeze.o (object) anymore.
* Remove misleading messages in "make regen-global-objects" and
"make clinic". They are now outdated, these commands are now
safe to use.
* Document generates files in Doc/using/configure.rst.
Co-authored-by: Erlend E. Aasland <erlend@python.org>
(cherry picked from commit db1ee6a19a)
Add Modules/_testcapi/util.h header (GH-108774)
It contains common macros used in C API tests.
(cherry picked from commit 0e01fac315)
Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>
gh-108303: Add Lib/test/test_cppext/ sub-directory (GH-108325)
* Move test_cppext to its own directory
* Rename setup_testcppext.py to setup.py
* Rename _testcppext.cpp to extension.cpp
* The source (extension.cpp) is now also copied by the test.
(cherry picked from commit 21dda09600)
Co-authored-by: Victor Stinner <vstinner@python.org>
gh-106016: Add Lib/test/test_module/ directory (#108293)
* Move Python scripts related to test_module to this new directory:
good_getattr.py and bad_getattrX.py scripts.
* Move Lib/test/test_module.py to Lib/test/test_module/__init__.py.
(cherry picked from commit adfc118fda)
The _xxsubinterpreters module was meant to only use public API. Some internal C-API usage snuck in over the last few years (e.g. gh-28969). This fixes that.
(cherry picked from commit e6373c0d8b)
Co-authored-by: Eric Snow <ericsnowcurrently@gmail.com>
gh-75371: reformat Makefile.pre.in to accommodate for empty FRAMEWORKALTINSTALLLAST (GH-107035)
in the case of an empty FRAMEWORKALTINSTALLLAST, this patch prevents leaving
an astray linebreak and two tabs in the resulting Makefile.
Before change:
```
.PHONY: commoninstall
commoninstall: check-clean-src \
altbininstall libinstall inclinstall libainstall \
sharedinstall altmaninstall \
```
After change (with empty FRAMEWORKALTINSTALLLAST):
```
.PHONY: commoninstall
commoninstall: check-clean-src \
altbininstall libinstall inclinstall libainstall \
sharedinstall altmaninstall
```
(cherry picked from commit 9c38206925)
Co-authored-by: Moritz Neeb <nt4u@kpvn.de>
This ensures that `commoninstall` is completed before `bininstall` is
started when parallel builds are used (`make -j install`), and so the
`python3` symlink is only installed after all standard library modules
are installed.
(cherry picked from commit 990cb3676c)
gh-99108: Refresh HACL* (GH-104808)
Refresh HACL* from upstream to improve SHA2 performance and fix a 32-bit issue in SHA3.
(cherry picked from commit 160321e530)
Co-authored-by: Jonathan Protzenko <protz@microsoft.com>
Apply BOLT optimizations to libpython for shared builds. Most of the C
code is in libpython so it is critical to apply BOLT there fully realize
BOLT benefits.
This change also reworks how BOLT instrumentation is applied. It
effectively removes the readelf based logic added in gh-101525 and
replaces it with a mechanism that saves a copy of the pre-bolt binary
and restores that copy when necessary. This allows us to perform BOLT
optimizations without having to manually delete the output binary to
force a new bolt run.
Also:
- add a clean-bolt target for purging BOLT files and hook that up to the
clean target
- .gitignore BOLT related files
Before and after this refactor, `make` will no-op after a previous run.
Both versions should also share common make DAG deficiencies where
targets fail to trigger as often as they need to or can trigger
prematurely in certain scenarios. e.g. after this change you may need to
`rm profile-bolt-stamp` to force a BOLT run because there aren't
appropriate non-phony targets for BOLT's make target to depend on.
To make it easier to iterate on custom BOLT settings, the flags to pass
to instrumentation and application are now defined in configure and can
be overridden by passing BOLT_INSTRUMENT_FLAGS and BOLT_APPLY_FLAGS.
This implements PEP 695, Type Parameter Syntax. It adds support for:
- Generic functions (def func[T](): ...)
- Generic classes (class X[T](): ...)
- Type aliases (type X = ...)
- New scoping when the new syntax is used within a class body
- Compiler and interpreter changes to support the new syntax and scoping rules
Co-authored-by: Marc Mueller <30130371+cdce8p@users.noreply.github.com>
Co-authored-by: Eric Traut <eric@traut.com>
Co-authored-by: Larry Hastings <larry@hastings.org>
Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com>
By convention make targets that don't refer to a file have a dependency
on the fake .PHONY target/file. This ensures that these targets are
always evaluated because there is no rule to create a .PHONY file
and that will force make to think the rule is out of date and needs
to be rebuilt.
This commit consistently associates virtual targets with .PHONY by
declaring the .PHONY dependency immediately above the make rule. This
should avoid race conditions and avoidable rebuilds across multiple make
invocations.
Replaces our built-in SHA3 implementation with a verified one from the HACL* project.
This implementation is used when OpenSSL does not provide SHA3 or is not present.
3.11 shiped with a very slow tiny sha3 implementation to get off of the <=3.10 reference implementation that wound up having serious bugs. This brings us back to a reasonably performing built-in implementation consistent with what we've just replaced our other guaranteed available standard hash algorithms with: code from the HACL* project.
---------
Co-authored-by: Gregory P. Smith <greg@krypto.org>
Deep-frozen code objects are cannot be shared (currently) by
interpreters, due to how adaptive specialization can modify the
bytecodes. We work around this by only using the deep-frozen objects in
the main interpreter. This does incur a performance penalty for
subinterpreters, which we may be able to resolve later.
This avoids conflicting with the shebang of the called scripts as well
as avoiding hard errors on platforms where the called script runs a
failing unchecked command in the usual course of checking since
`SHELL=/bin/sh -e` as of a90863c.
Fixes gh-103776.
