The tests were flaky on slow machines because subprocesses could finish
before enough samples were collected. This adds synchronization similar
to test_external_inspection: test scripts now signal when they start
working, and the profiler waits for this signal before sampling.
Test scripts now run in infinite loops until killed rather than for
fixed iterations, ensuring the profiler always has active work to
sample regardless of machine speed.
This PR implements frame caching in the RemoteUnwinder class to significantly reduce memory reads when profiling remote processes with deep call stacks.
When cache_frames=True, the unwinder stores the frame chain from each sample and reuses unchanged portions in subsequent samples. Since most profiling samples capture similar call stacks (especially the parent frames), this optimization avoids repeatedly reading the same frame data from the target process.
The implementation adds a last_profiled_frame field to the thread state that tracks where the previous sample stopped. On the next sample, if the current frame chain reaches this marker, the cached frames from that point onward are reused instead of being re-read from remote memory.
The sampling profiler now enables frame caching by default.
Extend defect handling via policy to a couple of missed defects.
---------
Co-authored-by: Martin Panter <vadmium@users.noreply.github.com>
Co-authored-by: Ivo Bellin Salarin <ivo@nilleb.com>
The keyword typo suggestion mechanism in traceback would incorrectly
suggest replacements when the extracted source code was merely incomplete
rather than containing an actual typo. For example, when a missing comma
caused a syntax error, the system would suggest replacing 'print' with
'not' because the incomplete code snippet happened to pass validation.
The fix adds a validation step that first checks whether the original
extracted code raises a SyntaxError. If the code compiles successfully
or is simply incomplete (compile_command returns None), the function
returns early since there is no way to verify that a keyword replacement
would actually fix the problem.
RDM: This fixes a subtle folding error that showed up when a token exactly filled a line and was followed by whitespace and a token with no folding whitespace that was longer than a line. In this particular circumstance the whitespace after the first token got pushed on to the next line, and then stolen to go in front of the next unfoldable token...leaving a completely empty line in the line buffer. That line got turned in to a newline, which is RFC illegal, and the newish security check caught it. The fix is to just delete that empty line from the buffer.
Co-authored-by: blurb-it[bot] <43283697+blurb-it[bot]@users.noreply.github.com>
Enhanced the _find_lineno method in doctest to correctly identify and
report line numbers for doctests defined in __test__ dictionaries when
formatted as triple-quoted strings.
Finds a non-blank line in the test string and matches it in the source
file, verifying subsequent lines also match to handle duplicate lines.
Previously, doctest would report "line None" for __test__ dictionary
strings, making it difficult to debug failing tests.
Co-authored-by: Jurjen N.E. Bos <jneb@users.sourceforge.net>
Co-authored-by: R. David Murray <rdmurray@bitdance.com>
While CPython doesn't support `--enable-wasm-dynamic-linking`, external tools like componentize-py do and they have to patch around it. Since the flag is off by default, allowing the flag so external users can add/inject dynamic linking support seems acceptable.
It should make it easier when you need to rebuild just the e.g. host Python, but it requires ./configure to run.
Co-authored-by: Emma Smith <emma@emmatyping.dev>
Loading a small data which does not even involve arbitrary code execution
could consume arbitrary large amount of memory. There were three issues:
* PUT and LONG_BINPUT with large argument (the C implementation only).
Since the memo is implemented in C as a continuous dynamic array, a single
opcode can cause its resizing to arbitrary size. Now the sparsity of
memo indices is limited.
* BINBYTES, BINBYTES8 and BYTEARRAY8 with large argument. They allocated
the bytes or bytearray object of the specified size before reading into
it. Now they read very large data by chunks.
* BINSTRING, BINUNICODE, LONG4, BINUNICODE8 and FRAME with large
argument. They read the whole data by calling the read() method of
the underlying file object, which usually allocates the bytes object of
the specified size before reading into it. Now they read very large data
by chunks.
Also add comprehensive benchmark suite to measure performance and memory
impact of chunked reading optimization in PR #119204.
Features:
- Normal mode: benchmarks legitimate pickles (time/memory metrics)
- Antagonistic mode: tests malicious pickles (DoS protection)
- Baseline comparison: side-by-side comparison of two Python builds
- Support for truncated data and sparse memo attack vectors
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Co-authored-by: Gregory P. Smith <greg@krypto.org>
Add readline.get_pre_input_hook() to retrieve the current pre-input
hook. This allows applications to save and restore the hook without
overwriting user settings.
This fixes a regression introduced in gh-140558. The interpreter would
crash if we inserted a non `str` key into a split table that matches an
existing key.
The previous test_spawn_doesnt_hang test had a few problems:
* It would cause ENV CHANGED failures if other tests were running
concurrently due to stty changes
* Typing while the test was running could cause it to fail
* Factor out bodies of the largest uops, to reduce jit code size.
* Factor out common assert, also reducing jit code size.
* Limit size of jitted code for a single executor to 1MB.
Much of the information was duplicated in stdtypes.rst; this PR keeps lexical/syntactical details in Lexical Analysis and the evaluation & runtime behaviour in Standard types, with cross-references between the two.
Since the t-string section only listed differences from f-strings, and the grammar for the two is equivalent, that section was moved to Standard types almost entirely.
Co-authored-by: Blaise Pabon <blaise@gmail.com>
Co-authored-by: Adam Turner <9087854+AA-Turner@users.noreply.github.com>
Co-authored-by: Stan Ulbrych <89152624+StanFromIreland@users.noreply.github.com>
* Remove quadratic behavior in node ID cache clearing
Co-authored-by: Jacob Walls <38668450+jacobtylerwalls@users.noreply.github.com>
* Add news fragment
---------
Co-authored-by: Jacob Walls <38668450+jacobtylerwalls@users.noreply.github.com>
Reading a specially prepared small Plist file could cause OOM because file's
read(n) preallocates a bytes object for reading the specified amount of
data. Now plistlib reads large data by chunks, therefore the upper limit of
consumed memory is proportional to the size of the input file.
Reading the whole body of the HTTP response could cause OOM if
the Content-Length value is too large even if the server does not send
a large amount of data. Now the HTTP client reads large data by chunks,
therefore the amount of consumed memory is proportional to the amount
of sent data.
On Windows, Popen._communicate() previously wrote to stdin synchronously, which could block indefinitely if the subprocess didn't consume input= quickly and the pipe buffer filled up. The timeout= parameter was only checked when joining the reader threads, not during the stdin write.
This change moves the Windows stdin writing to a background thread (similar to how stdout/stderr are read in threads), allowing the timeout to be properly enforced. If timeout expires, TimeoutExpired is raised promptly and the writer thread continues in the background. Subsequent calls to communicate() will join the existing writer thread.
Adds test_communicate_timeout_large_input to verify that TimeoutExpired is raised promptly when communicate() is called with large input and a timeout, even when the subprocess doesn't consume stdin quickly.
This test already passed on POSIX (where select() is used) but failed on Windows where the stdin write blocks without checking the timeout.
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
Fix inconsistent subprocess.Popen.communicate() behavior between Windows
and POSIX when using memoryview objects with non-byte elements as input.
On POSIX systems, the code was incorrectly comparing bytes written against
element count instead of byte count, causing data truncation for large
inputs with non-byte element types.
Changes:
- Cast memoryview inputs to byte view when input is already a memoryview
- Fix progress tracking to use len(input_view) instead of len(self._input)
- Add comprehensive test coverage for memoryview inputs
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
* old-man-yells-at-ReST
* Update 2025-05-30-18-37-44.gh-issue-134453.kxkA-o.rst
* assertIsNone review feedback
* fix memoryview_nonbytes test to fail without our fix on main, and have a nicer error.
Thanks to Peter Bierma @ZeroIntensity for the code review.
* gh-141473: Fix subprocess.Popen.communicate to send input to stdin
* Docs: Clarify that `input` is one time only on `communicate()`
* NEWS entry
* Add a regression test.
---------
Co-authored-by: Gregory P. Smith <greg@krypto.org>
Removes a copy going from bytearray to bytes.
Co-authored-by: Victor Stinner <vstinner@python.org>
Co-authored-by: Bénédikt Tran <10796600+picnixz@users.noreply.github.com>
Removes a copy going from bytearray to bytes.
Co-authored-by: Victor Stinner <vstinner@python.org>
Co-authored-by: Bénédikt Tran <10796600+picnixz@users.noreply.github.com>
Added atomic operations to `scanner_begin()` and `scanner_end()` to prevent
race conditions on the `executing` flag in free-threaded builds. Also added
tests for concurrent usage of the `re` module.
Without the atomic operations, `test_scanner_concurrent_access()` triggers
`assert(self->executing)` failures, or a thread sanitizer run emits errors.
This simplifies the Lexical Analysis section on Names (but keeps it technically correct) by putting all the info about non-ASCII characters in a separate (and very technical) section.
It uses a mental model where the parser doesn't handle Unicode complexity “immediately”, but:
- parses any non-ASCII character (outside strings/comments) as part of a name, since these can't (yet) be e.g. operators
- normalizes the name
- validates the name, using the xid_start/xid_continue sets
Co-authored-by: Stan Ulbrych <89152624+StanFromIreland@users.noreply.github.com>
Co-authored-by: Blaise Pabon <blaise@gmail.com>
Co-authored-by: Micha Albert <info@micha.zone>
Co-authored-by: KeithTheEE <kmurrayis@gmail.com>
* gh-140550: PEP 793 reference documentation
Since the PEP calls for soft-deprecation of the existing initialization
function, this reorganizes the relevant docs to put the new way of
doing things first, and de-emphasize the old.
Some bits, like the tutorial, are left out of this patch. (See the
issue for a list.)
* Make Py_{SIZE,IS_TYPE,SET_SIZE} regular functions in stable ABI
Group them together with Py_TYPE & Py_SET_TYPE to cut down
on repetitive preprocessor macros.
Format repetitive definitions in object.c more concisely.
Py_SET_TYPE is still left out of the Limited API.
* test_hashlib: better handle support for SHA3
It's possible that the SSL library supports only SHA3 algo and doesn't
have SHAKE one.
The current test wrongly detect this and set both HASH and HASHXOF to
None expecting to have the extra SHA3 attributes present but this should
only be true for SHAKE algo.
To better handle this, move the HASH condition to a dedicated try-expect
condition and check if HASHXOF is None in the relevant code effectively
checking if SHA3 is supported by the SSL library but SHAKE algo needs to
use the sha3module one.
Signed-off-by: Christian Marangi <ansuelsmth@gmail.com>
* rework the conditional import for all its attrs
---------
Signed-off-by: Christian Marangi <ansuelsmth@gmail.com>
Co-authored-by: Gregory P. Smith <greg@krypto.org>
Most of the `self.assertTrue(self.called)` checks are flaky because
the worker threads may sometimes finish before the main thread calls
`self.during_threads()`.
Co-authored-by: Éric <merwok@netwok.org>
Co-authored-by: Hugo van Kemenade <1324225+hugovk@users.noreply.github.com>
Co-authored-by: Victor Stinner <vstinner@python.org>
Co-authored-by: Stan Ulbrych <89152624+StanFromIreland@users.noreply.github.com>
Co-authored-by: Hugo van Kemenade <1324225+hugovk@users.noreply.github.com>
Simplify preload regression test using `__main__`
With the fix for gh-126631 `__main__` modules can be preloaded and the regression
test for gh-135335 can be simplified to just use a self-contained script rather
than requiring a module.
Note this assumes and implicitly tests that `__main__` is preloaded by default.
If we overflowed the global version counter (i.e., after 2*24 calls to
`_PyMonitoring_SetEvents`), we bailed out after setting global monitoring
events but before instrumenting code objects, which led to assertion errors
later on.
Also add a `time.sleep()` to `test_free_threading.test_monitoring` to avoid
overflowing the global version counter.
Revert (unneeded, already done elsewhere) "gh-98552: flush std streams in the multiprocessing forkserver before fork (#141849)"
This reverts commit 58badb1711.
* gh-141801: Use accessors for ASN1_STRING fields
While ASN1_STRING is currently exposed, it is better to use the
accessors. See https://github.com/openssl/openssl/issues/29117 where, if
the type were opaque, OpenSSL's X509 objects could be much more
memory-efficient.
* Update Modules/_ssl.c
Co-authored-by: Bénédikt Tran <10796600+picnixz@users.noreply.github.com>
* Update Modules/_ssl.c
Co-authored-by: Bénédikt Tran <10796600+picnixz@users.noreply.github.com>
---------
Co-authored-by: Bénédikt Tran <10796600+picnixz@users.noreply.github.com>
Added a critical section to protect the states of `ReaderObj` and `WriterObj` in the free-threading build. Without the critical sections, both new free-threading tests were crashing.
* gh-138697: Fix inferring dest from a single-dash long option in argparse
If a short option and a single-dash long option are passed to add_argument(),
dest is now inferred from the single-dash long option.
* Make double-dash options taking priority over single-dash long options.
---------
Co-authored-by: Savannah Ostrowski <savannah@python.org>
This introduces a Wasmtime configuration file to get some CLI options out of the code for easier manipulation. It also allows for easier tweaking after the Makefile is generated.
As well, cut back on the flexibility of specifying HOSTRUNNER for simpler code. The flexibility was never used and so it didn't make sense to keep it around.
This commit updates CI and configuration from wasi-sdk-25 to wasi-sdk-29
which was released recently. This notably includes stubs for pthreads
which all return errors, so some adjustment in logic is necessary to
retain knowledge that WASI cannot yet spawn threads for example.
This additionally increases the wasm stack allowance to 32MiB from 16MiB
to accomodate the `test_recursive_pickle` test in the
`test_functools.py` file. It looks like the Clang/LLVM update that
happened in wasi-sdk-29 relative to wasi-sdk-25 is likely the cause of
this where presumably functions have more locals than before and/or a
slightly adjusted stack space requirement which overflows the stack.
In Fedora, we've been given a slightly incomplete reproducer for a problematic
Python 3.14 color-related change in argparse that leads to an exception when
Python is used from mod_wsgi: https://bugzilla.redhat.com/2414940
mod_wsgi replaces sys.stdout with a custom object that raises OSError on .fileno():
8460dbfcd5/src/server/wsgi_logger.c (L434-L440)
This should be supported, as the documentation of fileno explicitly says:
> An OSError is raised if the IO object does not use a file descriptor.
https://docs.python.org/3.14/library/io.html#io.IOBase.fileno
The previously expected exception inherits from OSError,
so it is still expected.
Fixes https://github.com/python/cpython/issues/141570
Co-authored-by: Cody Maloney <cmaloney@users.noreply.github.com>
Co-authored-by: Victor Stinner <vstinner@python.org>
Only raises if the stack pointer is both below the limit *and* above the stack base.
This prevents false positives for user-space threads, as the stack pointer will be outside those bounds
if the stack has been swapped.
The dataclasses `__init__` function is generated dynamically by a call to `exec()` and so doesn't have deferred reference counting enabled. Enable deferred reference counting on functions when assigned as an attribute to type objects to avoid reference count contention when creating dataclass instances.
* Promote _PyObject_Dump() as a public function.
* Keep _PyObject_Dump() alias to PyUnstable_Object_Dump()
for backward compatibility.
* Replace _PyObject_Dump() with PyUnstable_Object_Dump().
Co-authored-by: Peter Bierma <zintensitydev@gmail.com>
Co-authored-by: Kumar Aditya <kumaraditya@python.org>
Co-authored-by: Petr Viktorin <encukou@gmail.com>
* Add parameters nolinestop and strictlimits in the tkinter.Text.search() method.
* Add the tkinter.Text.search_all() method.
* Add more tests for tkinter.Text.search().
* stopindex is now only ignored if it is None.
Adapted from a patch for Python 3.14 submitted to the Debian BTS by John
https://bugs.debian.org/1105111#20
Co-authored-by: John David Anglin <dave.anglin@bell.net>
- Introduce a new field in the GC state to store the frame that initiated garbage collection.
- Update RemoteUnwinder to include options for including "<native>" and "<GC>" frames in the stack trace.
- Modify the sampling profiler to accept parameters for controlling the inclusion of native and GC frames.
- Enhance the stack collector to properly format and append these frames during profiling.
- Add tests to verify the correct behavior of the profiler with respect to native and GC frames, including options to exclude them.
Co-authored-by: Pablo Galindo Salgado <pablogsal@gmail.com>
Raise ValueError for infinite inputs to stdev/pstdev
---
Co-authored-by: Gregory P. Smith <68491+gpshead@users.noreply.github.com>
Co-authored-by: Bénédikt Tran <10796600+picnixz@users.noreply.github.com>
In the linecache module and in the Python implementation of the
warnings module, a DeprecationWarning is issued when
m.__loader__ differs from m.__spec__.loader (like in the C
implementation of the warnings module).
Replace code that directly accesses PyASCIIObject.hash with
PyUnstable_Unicode_GET_CACHED_HASH().
Remove redundant "assert(PyUnicode_Check(op))" from
PyUnstable_Unicode_GET_CACHED_HASH(), _PyASCIIObject_CAST() already
implements the check.
This needs a single bit, but was stored as a void* in the module
struct. This didn't matter due to packing, but now that there's
another bool in the struct, we can save a bit of memory by
making md_gil a bool.
Variables that changed type are renamed, to detect conflicts.
When iterparse() opens a file by filename and is not explicitly closed,
emit a ResourceWarning to alert developers of the resource leak.
Signed-off-by: Osama Abdelkader <osama.abdelkader@gmail.com>
Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>
This PR changes the current JIT model from trace projection to trace recording. Benchmarking: better pyperformance (about 1.7% overall) geomean versus current https://raw.githubusercontent.com/facebookexperimental/free-threading-benchmarking/refs/heads/main/results/bm-20251108-3.15.0a1%2B-7e2bc1d-JIT/bm-20251108-vultr-x86_64-Fidget%252dSpinner-tracing_jit-3.15.0a1%2B-7e2bc1d-vs-base.svg, 100% faster Richards on the most improved benchmark versus the current JIT. Slowdown of about 10-15% on the worst benchmark versus the current JIT. **Note: the fastest version isn't the one merged, as it relies on fixing bugs in the specializing interpreter, which is left to another PR**. The speedup in the merged version is about 1.1%. https://raw.githubusercontent.com/facebookexperimental/free-threading-benchmarking/refs/heads/main/results/bm-20251112-3.15.0a1%2B-f8a764a-JIT/bm-20251112-vultr-x86_64-Fidget%252dSpinner-tracing_jit-3.15.0a1%2B-f8a764a-vs-base.svg
Stats: 50% more uops executed, 30% more traces entered the last time we ran them. It also suggests our trace lengths for a real trace recording JIT are too short, as a lot of trace too long aborts https://github.com/facebookexperimental/free-threading-benchmarking/blob/main/results/bm-20251023-3.15.0a1%2B-eb73378-CLANG%2CJIT/bm-20251023-vultr-x86_64-Fidget%252dSpinner-tracing_jit-3.15.0a1%2B-eb73378-pystats-vs-base.md .
This new JIT frontend is already able to record/execute significantly more instructions than the previous JIT frontend. In this PR, we are now able to record through custom dunders, simple object creation, generators, etc. None of these were done by the old JIT frontend. Some custom dunders uops were discovered to be broken as part of this work gh-140277
The optimizer stack space check is disabled, as it's no longer valid to deal with underflow.
Pros:
* Ignoring the generated tracer code as it's automatically created, this is only additional 1k lines of code. The maintenance burden is handled by the DSL and code generator.
* `optimizer.c` is now significantly simpler, as we don't have to do strange things to recover the bytecode from a trace.
* The new JIT frontend is able to handle a lot more control-flow than the old one.
* Tracing is very low overhead. We use the tail calling interpreter/computed goto interpreter to switch between tracing mode and non-tracing mode. I call this mechanism dual dispatch, as we have two dispatch tables dispatching to each other. Specialization is still enabled while tracing.
* Better handling of polymorphism. We leverage the specializing interpreter for this.
Cons:
* (For now) requires tail calling interpreter or computed gotos. This means no Windows JIT for now :(. Not to fret, tail calling is coming soon to Windows though https://github.com/python/cpython/pull/139962
Design:
* After each instruction, the `record_previous_inst` function/label is executed. This does as the name suggests.
* The tracing interpreter lowers bytecode to uops directly so that it can obtain "fresh" values at the point of lowering.
* The tracing version behaves nearly identical to the normal interpreter, in fact it even has specialization! This allows it to run without much of a slowdown when tracing. The actual cost of tracing is only a function call and writes to memory.
* The tracing interpreter uses the specializing interpreter's deopt to naturally form the side exit chains. This allows it to side exit chain effectively, without repeating much code. We force a re-specializing when tracing a deopt.
* The tracing interpreter can even handle goto errors/exceptions, but I chose to disable them for now as it's not tested.
* Because we do not share interpreter dispatch, there is should be no significant slowdown to the original specializing interpreter on tailcall and computed got with JIT disabled. With JIT enabled, there might be a slowdown in the form of the JIT trying to trace.
* Things that could have dynamic instruction pointer effects are guarded on. The guard deopts to a new instruction --- `_DYNAMIC_EXIT`.
Add PyUnstable_ThreadState_SetStackProtection() and
PyUnstable_ThreadState_ResetStackProtection() functions
to set the stack base address and stack size of a Python
thread state.
Co-authored-by: Petr Viktorin <encukou@gmail.com>
Update `bytearray` to contain a `bytes` and provide a zero-copy path to
"extract" the `bytes`. This allows making several code paths more efficient.
This does not move any codepaths to make use of this new API. The documentation
changes include common code patterns which can be made more efficient with
this API.
---
When just changing `bytearray` to contain `bytes` I ran pyperformance on a
`--with-lto --enable-optimizations --with-static-libpython` build and don't see
any major speedups or slowdowns with this; all seems to be in the noise of
my machine (Generally changes under 5% or benchmarks that don't touch
bytes/bytearray).
Co-authored-by: Victor Stinner <vstinner@python.org>
Co-authored-by: Maurycy Pawłowski-Wieroński <5383+maurycy@users.noreply.github.com>
Split existing tests on smaller methods and move them to separate class.
Rename variable "content" to "it".
Use BytesIO instead of StringIO.
Add few more tests.
Many functions related to compiling or parsing Python code, such as
compile(), ast.parse(), symtable.symtable(),
and importlib.abc.InspectLoader.source_to_code() now allow to pass
the module name used when filtering syntax warnings.
* gh-137109: refactor warning about threads when forking
This splits the OS API specific functionality to get the number of threads out
from the fallback Python method and warning raising code itself. This way the
OS APIs can be queried before we've run
`os.register_at_fork(after_in_parent=...)` registered functions which
themselves may (re)start threads that would otherwise be detected.
This is best effort. If the OS APIs are either unavailable or fail, the
warning generating code still falls back to looking at the Python threading
state after the CPython interpreter world has been restarted and the
after_in_parent calls have been made. The common case for most Linux and macOS
environments should work today.
This also lines up with the existing TODO refactoring, we may choose to expose
this API to get the number of OS threads in the `os` module in the future.
* NEWS entry
* avoid "function-prototype" compiler warning?
Fix error in assertion which causes failure if pos is equal to PY_SSIZE_T_MAX.
Fix undefined behavior in read() and readinto() if pos is larger that the size
of the underlying buffer.
* Increase test coverage for csv.DictReader and csv.Sniffer
Previously there were no tests for the DictReader fieldnames
setter, the case where a StopIteration was encountered when trying
to determine the fieldnames from the content or the case where
Sniffer could not find a delimiter.
* Revert whitespace change to comment
* Add a test that csv.Sniffer.has_header checks up to 20 rows
* Replace name and age with letter and offset
Co-authored-by: Bénédikt Tran <10796600+picnixz@users.noreply.github.com>
* Address review comment
---------
Co-authored-by: Bénédikt Tran <10796600+picnixz@users.noreply.github.com>
Co-authored-by: Gregory P. Smith <68491+gpshead@users.noreply.github.com>
* checks 21, not 20
* Say "header" instead of "first row" to disambiguate per review.
---------
Co-authored-by: Stan Ulbrych <89152624+StanFromIreland@users.noreply.github.com>
Co-authored-by: Maurycy Pawłowski-Wieroński <maurycy@maurycy.com>
This also documents the previously undocumented default_port parameter.
Co-authored-by: Stan Ulbrych <89152624+StanFromIreland@users.noreply.github.com>
* chore: test_exec_set_nomemory_hang from 3.13
Signed-off-by: yihong0618 <zouzou0208@gmail.com>
* fix: apply comments
Signed-off-by: yihong0618 <zouzou0208@gmail.com>
* Update Lib/test/test_exceptions.py
Co-authored-by: Peter Bierma <zintensitydev@gmail.com>
* Update Lib/test/test_exceptions.py
Co-authored-by: Peter Bierma <zintensitydev@gmail.com>
* fix: windows too long name 60 times is enough
Signed-off-by: yihong0618 <zouzou0208@gmail.com>
---------
Signed-off-by: yihong0618 <zouzou0208@gmail.com>
Co-authored-by: Peter Bierma <zintensitydev@gmail.com>
* Doc: Remove sencence implying that concurrent.futures.ThreadPoolExecutor does not exist
Closes#140578
* Add NEWS.d entry for gh-140578
---------
Co-authored-by: Louis Paternault <spalax@gresille.org>
* Test passing unsupported Format values to call_annotate_function()
* Test call_evaluate_function with fake globals that raise errors
* Fix typo and comparison in test_fake_global_evaluation
* Test unsupported format in ForwardRef.evaluate()
* Test dict cell closure with multiple variables
* Test all options in ForwardRef repr
* Test ForwardRef being a final class
* Test `get_annotations(format=Format.VALUE)` for stringized annotations on custom objects
* Test `get_annotations(format=Format.VALUE)` for stringized annotations on wrapped partial functions
* Update test_stringized_annotations_with_star_unpack() to actually test stringized annotations
* Test __annotate__ returning a non-dict
* Test passing globals and locals to stringized `get_annotations()`
* Refine some wording in unittest partial mock doc
Some of the descriptions were addressed in first person,
but have now been changed to address the user reading the documentation instead.
Co-authored-by: C.A.M. Gerlach <CAM.Gerlach@Gerlach.CAM>
* Refer to bytes objects or bytes-like objects instead of strings.
* Remove backticks -- they do not have effect on formatting.
* Re-wrap lines to ensure the pydoc output fits in 80 coluimns.
* Remove references to the 1024 bytes limit.
* Document that ensure_ascii=True forces escaping not only non-ASCII, but also
non-printable characters (the only affected ASCII character is U+007F).
* Ensure that the help output for the json module does not exceed 80
columns (except one long line in an example and generated lines).
* Add more tests.
Adds documentation for each of the following:
- PyEnum_Type
- PyFilter_Type
- PyMap_Type
- PyReversed_Type
- PyZip_Type
In addition, PyRange_Type and PyRange_Check are also documented.
Co-authored-by: Stan Ulbrych <89152624+StanFromIreland@users.noreply.github.com>
Co-authored-by: Hugo van Kemenade <1324225+hugovk@users.noreply.github.com>
Co-authored-by: Emma Smith <emma@emmatyping.dev>
Co-authored-by: Author: Terry Jan Reedy <tjreedy@udel.edu>
Co-authored-by: Victor Stinner <vstinner@python.org>
Example of fixed warnings on 32-bit Windows:
Python\remote_debugging.c(24,53): warning C4244: 'function':
conversion from 'uint64_t' to 'uintptr_t', possible loss of data
Modules\_remote_debugging_module.c(789,44): warning C4244:
'function': conversion from 'uint64_t' to 'size_t', possible loss
of data
Clarify that:
- it takes parsing for an attack
- that some doors are closed by default
- only Expat version 2.7.2 has all the fixes
- use of the bundle depends on configuration
The test just before the cast ensures that the cast cannot overflow.
Fix the warning on 32-bit Windows:
Modules\_randommodule.c(525,28): warning C4244: '=': conversion
from 'uint64_t' to 'Py_ssize_t', possible loss of data
Update selected RFC 2822 references to RFC 5322
RFC 2822 was obsoleted by RFC 5322 in 2008. This updates references
to use the current standard in documentation, docstrings, and comments.
It preserves RFC 2822 references in legacy API components to maintain their
historical context.
RFC 822 → RFC 2822 → RFC 5322 progression is explained where relevant.
In some places specific sections of RFC are referenced where it seems helpful.
Scout rule was applied in some places and RFC mentions format was
normalized in doc strings and comments.
faulthandler now detects if a frame or a code object is invalid or
freed.
Add helper functions:
* _PyCode_SafeAddr2Line()
* _PyFrame_SafeGetCode()
* _PyFrame_SafeGetLasti()
_PyMem_IsPtrFreed() now detects pointers in [-0xff, 0xff] range
as freed.
Allow the --enable-pystats build option to be used with free-threading. The
stats are now stored on a per-interpreter basis, rather than process global.
For free-threaded builds, the stats structure is allocated per-thread and
then periodically merged into the per-interpreter stats structure (on thread
exit or when the reporting function is called). Most of the pystats related
code has be moved into the file Python/pystats.c.
Clean up What's New for 3.15
A bit early but I was reading through it and noticed some issues:
- A few improvements were listed in the removals section
- The "Porting to 3.15" section in the C API chapter had some
changes that aren't about the C API
- Some other typos and wording fixes
RDM: Like the change made in a earlier PR to the folder, we can/must use 'maxlen' as a stand in for 'unlimited' when computing line lengths when max_line_length is 0 or None; otherwise the computation results in a traceback.
Remove 'response_class' from getresponse docstring.
This variable is not documented as part of the API in the standard
library documentation; it should be considered as an implementation
detail and as such should not be included in the doc string.
Closes#57665.
* Try to match the module name pattern with module names constructed
starting from different parent directories of the filename.
E.g., for "/path/to/package/module" try to match with
"path.to.package.module", "to.package.module", "package.module" and
"module".
* Ignore trailing "/__init__.py".
* Ignore trailing ".pyw" on Windows.
* Keep matching with the full filename (without optional ".py" extension)
for compatibility.
* Only ignore the case of the ".py" extension on Windows.
Move the public PyUnicodeWriter API and the private _PyUnicodeWriter
API to a new Objects/unicode_writer.c file.
Rename a few helper functions to share them between unicodeobject.c
and unicode_writer.c, such as resize_compact() or unicode_result().
Expose `_PyUnicode_IsXidContinue/Start` in `unicodedata`:
add isxidstart() and isxidcontinue() functions.
Co-authored-by: Victor Stinner <vstinner@python.org>
libregrtest redirects test output to a file as part of its operation.
When `unittest` checks to see if it should colorize with
`isatty(sys.stdout)` that fails resulting in no colorizing of the
unittest output.
Update `libregrtest` to set `FORCE_COLOR=1` when redirecting test output
so that unittest will do color printing.
Co-authored-by: Victor Stinner <vstinner@python.org>
Exposes the GITHUB_ACTIONS environment variable to iOS simulator test runs, and
uses this variable to skip a Unix Datagram socketserver test that is unreliable
in the iOS GitHub Actions environment.
Co-authored-by: Hugo van Kemenade <1324225+hugovk@users.noreply.github.com>
Add _Py_DumpWideString() function to dump a wide string as ASCII. It
supports surrogate pairs.
Replace _Py_EncodeLocaleRaw() with _Py_DumpWideString()
in write_thread_name().
Python has required thread local support since 3.12 (see GH-103324). By assuming that thread locals are always supported, we can improve the performance of third-party extensions by allowing them to access the attached thread and interpreter states directly.
While `RawIOBase.readinto` should return a count of bytes between 0 and
the length of the given buffer, it is not required to. Add validation
inside RawIOBase.read() that the returned byte count is valid.
Co-authored-by: Shamil <ashm.tech@proton.me>
Co-authored-by: Victor Stinner <vstinner@python.org>
The methods are already wrapped with a lock, which makes them thread-safe in
free-threaded build. This replaces `PyThread_acquire_lock` with `PyMutex` and
removes some macros and allocation handling code.
Also add a test for free-threading to ensure we aren't getting data races and
that the locking is working.
Co-authored-by: Victor Stinner <vstinner@python.org>
Co-authored-by: Sergey B Kirpichev <skirpichev@gmail.com>
Co-authored-by: Hugo van Kemenade <1324225+hugovk@users.noreply.github.com>
Optimize `asyncio.all_tasks()` for the common case where the event loop is running in the current thread by avoiding stop-the-world pauses and locking.
This optimization is already present for `asyncio.current_task()` so we do the same for `asyncio.all_tasks()`.
* Set stx_mode to None if STATX_TYPE|STATX_MODE is missing from
stx_mask.
* Enhance os.statx() tests.
* statx_result structure: remove atime_sec, btime_sec, ctime_sec and
mtime_sec members. Compute them on demand when stx_atime,
stx_btime, stx_ctime and stx_mtime are read.
* Doc: fix statx members sorting.
* gh-140443:use fma in loghelper to improve accuracy of log for very large integers
Use fused multiply-add in log_helper() for huge ints.
Saving a rounding here is remarkably effective. Across some millions
of randomized test cases with ints up to a billion bits, on Windows
and using log10, the ULP error distribution was dramatically
flattened, and its range was nearly cut in half. In fact, the largest
error Tim saw was under 0.6 ULP.
---------
Co-authored-by: abhi210 <27881020+Abhi210@users.noreply.github.com>
Co-authored-by: Stan Ulbrych <89152624+StanFromIreland@users.noreply.github.com>
The `make_gen()` function creates and tracks generator/coro objects, but
doesn't fully initialize all the fields. At a minimum, we need to
initialize all the fields that may be accessed by gen_traverse because
the call to `compute_cr_origin()` can trigger a GC.
While looking at #140028, I found some unrelated test regressions in the
3.14 cycle. These seem to all come from #130317. From what I can tell,
that made Python more correct than it was before. According to [0], HP PA
RISC uses 1 for SNaN and thus a 0 for QNaN.
[0]: https://grouper.ieee.org/groups/1788/email/msg03272.html
Currently, there are a few places where tp_mro could theoretically
become NULL, but do not in practice. This commit adds defensive checks for
NULL values to ensure that any changes do not introduce a crash and that
state invariants are upheld.
The assertions added in this commit are all instances where a NULL value would get passed to something not expecting a NULL, so it is better to catch an assertion failure than crash later on.
There are a few cases where it is OK for the return of lookup_tp_mro to be NULL, such as when passed to is_subtype_with_mro, which handles this explicitly.
Some systems have the definitions of the mask bits without having the
corresponding members in struct statx. Add configure checks for members
added after Linux 4.11 (when statx itself was added).
Android has Linux's statx, but MACHDEP is "android" on Android, so
configure doesn't check for statx on Android. Base the check for statx
on ac_sys_system instead, which is "Linux-android" on Android, "Linux"
on other Linux distributions, and "AIX" on AIX (which has an
incompatible function named statx).
* Count number of actually tracked objects, instead of trackable objects. This ensures that untracking tuples has the desired effect of reducing GC overhead
* Do not track most untrackable tuples during creation. This prevents large numbers of small tuples causing execessive GCs.
* Support non-UTF-8 shebang and comments if non-UTF-8 encoding is specified.
* Detect decoding error for non-UTF-8 encoding.
* Detect null bytes in source code.
* Link to compression setting constants from compression functions
* De-duplicate descriptions of the constants
Co-authored-by: Victor Stinner <vstinner@python.org>
Fix memory leak in sub-interpreter creation caused by overwriting of the previously used `_malloced` field. Now the pointer is stored in the first word of the memory block to avoid it being overwritten accidentally.
Co-authored-by: Kumar Aditya <kumaraditya@python.org>
Merge `_Info`, `_StatResultInfo` and `_DirEntryInfo` into a single `_Info`
class. No other changes.
This will allow us to use a cached `os.stat()` result from our upcoming
`_Info.stat()` method even when we have a backing `os.DirEntry`.
In Python 3.13 (but not 3.12 or 3.14), pathlib classes are defined in
`pathlib._local` rather than `pathlib`. In hindsight this was a mistake,
but it was difficult to predict how the abstract/local split would pan out.
In this patch we re-introduce `pathlib._local` as a stub module that
re-exports the classes from `pathlib`. This allows path objects pickled in
3.13 to be unpicked in 3.14+
Use PyBytesWriter in action_helpers.c _build_concatenated_bytes().
3x faster bytes concat in the parser.
Co-authored-by: Victor Stinner <vstinner@python.org>
stx_atomic_write_unit_max_opt was added in Linux 6.16, but is controlled
by the STATX_WRITE_ATOMIC mask bit added in Linux 6.11. That's safe at
runtime because all kernels clear the reserved space in struct statx and
zero is a valid value for stx_atomic_write_unit_max_opt, and it avoids
allocating another mask bit, which are a limited resource. But it also
means the kernel headers don't provide a way to check whether
stx_atomic_write_unit_max_opt exists, so add a configure check.
On modern systems, the result of wcsxfrm() is much larger the size of
the input string (from 4+2*n on Windows to 4+5*n on Linux for simple
ASCII strings), so optimistic allocation of the buffer of the same size
never works.
The exception is if the locale is "C" (or unset), but in that case the `wcsxfrm`
call should be fast (and calling `locale.strxfrm()` doesn't make too much
sense in the first place).
This fixes a regression introduced by GH-136004, in which finalization would hang while executing atexit handlers if the system was out of memory.
---------
Signed-off-by: yihong0618 <zouzou0208@gmail.com>
Co-authored-by: Peter Bierma <zintensitydev@gmail.com>
Co-authored-by: Victor Stinner <vstinner@python.org>
The previous `Py_REFCNT(x) == 1` checks can have data races in the free
threaded build. `_PyObject_IsUniquelyReferenced(x)` is a more conservative
check that is safe in the free threaded build and is identical to
`Py_REFCNT(x) == 1` in the default GIL-enabled build.
Previously, the _BlocksOutputBuffer code creates a list of bytes objects to handle the output data from compression libraries. This ends up being slow due to the output buffer code needing to copy each bytes element of the list into the final bytes object buffer at the end of compression.
The new PyBytesWriter API introduced in PEP 782 is an ergonomic and fast method of writing data into a buffer that will later turn into a bytes object. Benchmarks show that using the PyBytesWriter API is 10-30% faster for decompression across a variety of settings. The performance gains are greatest when the decompressor is very performant, such as for Zstandard (and likely zlib-ng). Otherwise the decompressor can bottleneck decompression and the gains are more modest, but still sizable (e.g. 10% faster for zlib)!
Co-authored-by: Bénédikt Tran <10796600+picnixz@users.noreply.github.com>
Revert GH-131993.
Fix swallowing some syntax warnings in different modules if they accidentally
have the same message and are emitted from the same line.
Fix memory leak in sub-interpreter creation caused by overwriting of the previously used `_malloced` field. Now the pointer is stored in the first word of the memory block to avoid it being overwritten accidentally.
Co-authored-by: Kumar Aditya <kumaraditya@python.org>
This amends 4e7e2dd043 to catch errors
that `os.getlogin` can raise as specified by POSIX and Linux/glibc [1].
[1]: https://man7.org/linux/man-pages/man3/getlogin.3.html#ERRORS
---------
Signed-off-by: yihong0618 <zouzou0208@gmail.com>
Co-authored-by: Bénédikt Tran <10796600+picnixz@users.noreply.github.com>
Restore `JoinablePath.[is_]relative_to()`, which were deleted in
ef63cca494. These methods are too useful to
forgo. Restore old tests, and add new tests covering path classes with
non-overridden `__eq__()` and `__hash__()`.
Slightly simplify `PurePath.relative_to()` while we're in the area.
No change to public APIs, because the pathlib ABCs are still private.
Fix incorrect sharing of current task with the forked child process by clearing thread state's current task and current loop in `PyOS_AfterFork_Child`.
* Support non-UTF-8 shebang and comments if non-UTF-8 encoding is specified.
* Detect decoding error in comments for UTF-8 encoding.
* Include the decoding error position for default encoding in SyntaxError.
* Move PyUnicode_Format() implementation from unicodeobject.c
to unicode_format.c.
* Replace unicode_modifiable() with _PyUnicode_IsModifiable()
* Add empty lines to have two empty lines between functions.
Add 't' prefix to colorizer.py stringprefix regex to support Python 3.14 template strings.
Add t prefixes to test_colorizer.py source test text and adjust line numbers on test methods.
---------
Co-authored-by: blurb-it[bot] <43283697+blurb-it[bot]@users.noreply.github.com>
Co-authored-by: Łukasz Langa <lukasz@langa.pl>
Co-authored-by: Stan Ulbrych <89152624+StanFromIreland@users.noreply.github.com>
* Move Python/formatter_unicode.c to Objects/unicode_formatter.c.
* Move Objects/stringlib/localeutil.h content into
unicode_formatter.c. Remove localeutil.h.
* Move _PyUnicode_InsertThousandsGrouping() to unicode_formatter.c
and mark the function as static.
* Rename unicode_fill() to _PyUnicode_Fill() and export it in
pycore_unicodeobject.h.
* Move MAX_UNICODE to pycore_unicodeobject.h as _Py_MAX_UNICODE.
Fix a flaky test introduced in 13dc2fde8c.
After a single HTTP/0.9 request, both client and server are expected to
close the connection on their side. In particular, if a client sends two
requests with the same connection, only the first one should be handled.
In the tests, it might happen that checking for the second request to be
ignored did not take into account that the server may have already closed
the connection. This flaky behavior was first observed on macOS CI workers
but could not be reproduced locally on a Linux machine.
* Modules/pyexpat.c: Disallow collection of in-use parent parsers.
Within libexpat, a parser created via `XML_ExternalEntityParserCreate`
is relying on its parent parser throughout its entire lifetime.
Prior to this fix, is was possible for the parent parser to be
garbage-collected too early.
Currently, Fedora 42 uses a custom Linux Kernel 6.16.9 that backported an upstream change
from 6.17-rc7 [1,3] but not its subsequent fix [2]. Until the issue is resolved upstream,
we skip the failing test `test_socket.test_aead_aes_gcm` for kernel versions between 6.16
and 6.17.x.
[1] 1b34cbbf4f
[2] d0ca0df179.
[3] 45bcf60fe4
Fix a compiler warning `-Wunused-function` after f04bea44c3.
The `set_invalid_arg` function in `Modules/pyexpat.c` may be unused if the underlying Expat
version is less than 2.4.0.
* Fix potential infinite recursion.
* Fix a bug when reference can cross boundaries of substitutions, e.g.
a=$(
b=$(a)a)
* Fix potential quadratic complexity.
* Fix KeyError for undefined CFLAGS, LDFLAGS, or CPPFLAGS.
* Fix infinite recursion when keep_unresolved=False.
* Unify behavior with keep_unresolved=False for bogus $ occurred before
and after variable references.
* fix: add missing `build-details.json` step for building wasm
Signed-off-by: Ho Kim <ho.kim@ulagbulag.io>
* gh-138489: Add missing build-details.json step for building wasm
Signed-off-by: Ho Kim <ho.kim@ulagbulag.io>
* Update Makefile.pre.in
Co-authored-by: Adam Turner <9087854+AA-Turner@users.noreply.github.com>
---------
Signed-off-by: Ho Kim <ho.kim@ulagbulag.io>
Co-authored-by: Adam Turner <9087854+AA-Turner@users.noreply.github.com>
* Fix `hashlib.file_digest()` versionchanged description of `BlockingIOError`
The sentence was missing a negation and contradicted the other two
descriptions in the same commit. I believe code behaviour is correct.
* fixup! Fix `hashlib.file_digest()` versionchanged description of `BlockingIOError`
* Remove unncessary NEWS.d entry
Replace hardcoded 5 seconds with support.SHORT_TIMEOUT.
Fix the following error on slow CI such as GitHub Action UBSan:
test test_pyrepl failed -- Traceback (most recent call last):
File "Lib/test/test_pyrepl/test_unix_console.py", line 362, in test_repl_eio
_, err = proc.communicate(timeout=5) # sleep for pty to settle
~~~~~~~~~~~~~~~~^^^^^^^^^^^
File "Lib/subprocess.py", line 1219, in communicate
stdout, stderr = self._communicate(input, endtime, timeout)
~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^
File "Lib/subprocess.py", line 2126, in _communicate
self._check_timeout(endtime, orig_timeout, stdout, stderr)
~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "Lib/subprocess.py", line 1266, in _check_timeout
raise TimeoutExpired(
...<2 lines>...
stderr=b''.join(stderr_seq) if stderr_seq else None)
subprocess.TimeoutExpired: Command '[...]' timed out after 5 seconds
The last remaining set of tests to split out that are focused on one
specific implementation portion (`bufferedio.c`).
test_io.test_general is now largely tests around `io.open` and module
properties (ex. pickling, class hierarchy, module members, etc).
This closes#138013.
gh-138013: Split TextIO tests from test_general
These tests take 1.3 seconds on my dev machine, match fairly closely
with testing `textio.c` implementation only.
* gh-139116: tracemalloc: Detach thread state when acquiring tables_lock
This prevents a deadlock when:
- One thread is in `_PyTraceMalloc_Stop`, with `TABLES_LOCK` held, calling
`PyRefTracer_SetTracer` which wants to stop the world
- Another is thread in `PyTraceMalloc_Track`, just attached thread state, waiting
for `TABLES_LOCK`
Detaching the thread state while waiting for `TABLES_LOCK` allows
`PyRefTracer_SetTracer` to stop the world.
Co-authored-by: Peter Bierma <zintensitydev@gmail.com>
Passing a negative or zero size to `cursor.fetchmany()` made it fetch all rows
instead of none.
While this could be considered a security vulnerability, it was decided to treat
this issue as a regular bug as passing a non-sanitized *size* value in the first
place is not recommended.
Adds a PrivacyInfo.xcprivacy file when packaging `_hashlib` and `_ssl` modules
for iOS (based on the original OpenSSL sources); and adds handling to the build
script to allow any app to add `xcprivacy` handling for a binary module.
Expose the XML Expat 2.7.2 APIs to tune protections against
"billion laughs" [1] attacks.
The exposed APIs are available on Expat parsers, that is,
parsers created by `xml.parsers.expat.ParserCreate()`, as:
- `parser.SetBillionLaughsAttackProtectionActivationThreshold(threshold)`, and
- `parser.SetBillionLaughsAttackProtectionMaximumAmplification(max_factor)`.
This completes the work in f04bea44c3,
and improves the existing related documentation.
[1]: https://en.wikipedia.org/wiki/Billion_laughs_attack
* Remove generator type check in raise_SIGINT_then_send_None
In the Cinder JIT we use a different type for generators, which breaks
the test which uses this function.
In general I believe the intent with generators is they have the right
structure rather than type, so a failure to find the 'send()' method is arguably
more correct if the wrong object is used.
* Also stop using PyGenObject type
Expose the XML Expat 2.7.2 mitigation APIs to disallow use of
disproportional amounts of dynamic memory from within an Expat
parser (see CVE-2025-59375 for instance).
The exposed APIs are available on Expat parsers, that is,
parsers created by `xml.parsers.expat.ParserCreate()`, as:
- `parser.SetAllocTrackerActivationThreshold(threshold)`, and
- `parser.SetAllocTrackerMaximumAmplification(max_factor)`.
Replace PyBytes_FromStringAndSize() and _PyBytes_Resize() with the
PyBytesWriter API.
Add _PyBytesWriter_GetSize() and _PyBytesWriter_GetData() static
inline functions.
Merge `_WindowsPathInfo` and `_PosixPathInfo` classes into a new
`_StatResultInfo` class. On Windows, this means relying on `os.stat()`
rather than `os.path.isfile()` and friends, which is a little slower. But
there's value in making the code easier to maintain, and we're going to
need the stat result for implementing `size()`, `mode()` etc.
Also move the classes from `pathlib._os` to `pathlib` proper.
Functions that take timestamp or timeout arguments now accept any
real numbers (such as Decimal and Fraction), not only integers or floats,
although this does not improve precision.
On some macOS versions there was an off-by-one error in wcsxfrm() which
caused writing past the end of the array if its size was not calculated
by running wcsxfrm() first.
Co-authored-by: Ronald Oussoren <ronaldoussoren@mac.com>
Fix race condition in test_external_inspection thread status tests
The tests test_thread_status_detection and test_thread_status_gil_detection
had a race condition where the test could sample thread status between when
the sleeper thread sends its "ready" message and when it actually calls
time.sleep(). This caused intermittent test failures where the sleeper
thread would show as running (status=0) instead of idle (status=1 or 2).
The fix moves the thread status collection inside the retry loop and
specifically waits for the expected thread states before proceeding with
assertions. The retry loop now continues until:
- The sleeper thread shows as idle (status=1 for CPU mode, status=2 for GIL mode)
- The busy thread shows as running (status=0)
- Both thread IDs are found in the status collection
This ensures the test waits for threads to settle into their expected states
before making assertions, eliminating the race condition.
Increase parallelism by splitting out `SignalsTest` from test_general.
`SignalsTest` takes 24.2 seconds on my dev machine when fully enabled
making it the largest part of `test_io`. Code move done via copy/paste
then tweak imports.
After splitting `test_io.test_general` is down to 10.1 seconds on my dev
box with all parts enabled.
Fix a bug in the pydoc module that was hiding functions in a Python
module if they were implemented in an extension module and the module did
not have __all__.
Adds tooling to generate and test an iOS XCframework, in a way that will also facilitate
adding other XCframework targets for other Apple platforms (tvOS, watchOS, visionOS and
even macOS, potentially).
---------
Co-authored-by: Hugo van Kemenade <1324225+hugovk@users.noreply.github.com>
In the _interpreters module, we use PyEval_EvalCode() to run Python code in another interpreter. However, when the process receives a KeyboardInterrupt, PyEval_EvalCode() will jump straight to finalization rather than returning. This prevents us from cleaning up and marking the thread as "not running main", which triggers an assertion in PyThreadState_Clear() on debug builds. Since everything else works as intended, remove that assertion.
* bpo-36967: Eliminate unnecessary check in _strptime when determining AM/PM
* Pauls suggestion to refactor test
* Fix test
---------
Co-authored-by: Stan Ulbrych <89152624+StanFromIreland@users.noreply.github.com>
Co-authored-by: Paul Ganssle <1377457+pganssle@users.noreply.github.com>
During finalization, we need to mark all non-daemon threads as daemon to quickly shut down threads when sending CTRL^C to the process. This was a minor regression from GH-136004.
* fix: available_timezones is reporting an invalid IANA zone name
* 📜🤖 Added by blurb_it.
* correct rst format for backticks
---------
Co-authored-by: blurb-it[bot] <43283697+blurb-it[bot]@users.noreply.github.com>
Co-authored-by: Paul Ganssle <1377457+pganssle@users.noreply.github.com>
Adds a mention of binary releases to the Android documentation.
---------
Co-authored-by: Russell Keith-Magee <russell@keith-magee.com>
Co-authored-by: Adam Turner <9087854+AA-Turner@users.noreply.github.com>
Prior to 3.9, Py_AddPendingCall() would always run pending calls in the main interpreter, but then each interpreter got their own ceval state, and they were scheduled for any interpreter. In GH-104813, this was undone, so Py_AddPendingCall() would always schedule for the main interpreter.
Use multiphase initialization in the _testcapi module to allow loading in subinterpreters. The isolation here isn't perfect as there's still some use of globals, but _testcapi should generally work in other interpreters.
Android pipes stdout/stderr to the log, which means every write to the log
becomes a separate log line. As a result, most practical uses of stdout/stderr
should be buffered; but it doesn't hurt to preserve unbuffered handling in case
it's useful.
In `_io__Buffered_flush_impl` the macro `CHECK_CLOSED` is used to check
the `buffered*` is in a good state to be flushed. That differs slightly
from `buffered_closed`.
In some cases, that difference would result in `close()` thinking the
file needed to be flushed and closed while `flush()` thought the file
was already closed.
This could happen during GC and would result in an unraisable exception.
* gh-138813: Default `BaseProcess` `kwargs` to `None` (#138814)
Set `BaseProcess.__init__(..., kwargs=None)` and initialize `kwargs` with
`dict(kwargs) if kwargs else {}`. This avoids a shared mutable default and
matches threading.Thread behavior.
Co-authored-by: Dmitrii Chuprov <cheese@altlinux.org>
* DummyProcess kwargs=None (which threading.Thread accepts properly)
Co-authored-by: Gregory P. Smith <greg@krypto.org>
Replace PyBytes_FromStringAndSize(NULL, size) and _PyBytes_Resize()
with the new public PyBytesWriter API.
Change also 'read' variable type from int to Py_ssize_t.
With https://github.com/llvm/llvm-project/pull/150201 being merged, there is
now a better way to generate the Emscripten trampoline, instead of including
hand-generated binary WASM content. Requires Emscripten 4.0.12.
Without the ability to set required capabilities, the REPL cannot
function properly (syntax highlighting and multiline editing can't
work).
We refuse to work in this degraded state.
This issue appears specifically for TypedDicts because the TypedDict constructor
code converts string annotations to ForwardRef objects, and those are not evaluated
properly by the get_type_hints() stack because of other shenanigans with type
parameters.
This issue does not affect normal generic classes because their annotations are not
pre-converted to ForwardRefs.
The fix attempts to restore the pre- #137227 behavior in the narrow scenario where
the issue manifests. It mostly makes changes only in the paths accessible from get_type_hints(),
ensuring that newer APIs (such as evaluate_forward_ref() and annotationlib) are not affected
by get_type_hints()'s past odd choices. This PR does not fix issue #138949, an older issue I
discovered while playing around with this one; we'll need a separate and perhaps more
invasive fix for that, but it should wait until after 3.14.0.
This amends commit bf8bbe9a81 by
restricting `echo_char` in `getpass.getpass` to single printable
ASCII characters as it would be uncommon to use long strings or
multi-byte characters for keyboard feedback.
---------
Co-authored-by: Bénédikt Tran <10796600+picnixz@users.noreply.github.com>
Co-authored-by: Brian Schubert <brianm.schubert@gmail.com>
Rely on default test discovery.
Validation:
```bash
# Run before commit
./python -m test test_io -uall,walltime,largefile,cpu,extralargefile -M25G -o --fail-env-changed -j0 --list-cases | sort > old_cases.txt
# Run after commit
./python -m test test_io -uall,walltime,largefile,cpu,extralargefile -M25G -o --fail-env-changed -j0 --list-cases | sort > new_cases.txt
diff new_cases.txt old_cases.
# <outputs no changes in case list>
```
There was a deadlock originally seen by Memray when a daemon thread
enabled or disabled profiling while the interpreter was shutting down.
I think this could also happen with garbage collection, but I haven't
seen that in practice.
The daemon thread could be hung while trying acquire the global rwmutex
that prevents overlapping global and per-interpreter stop-the-world events.
Since it already held the main interpreter's stop-the-world lock, it
also deadlocked the main thread, which is trying to perform interpreter
finalization.
Swap the order of lock acquisition to prevent this deadlock.
Additionally, refactor `_PyParkingLot_Park` so that the global buckets
hashtable is left in a clean state if the thread is hung in
`PyEval_AcquireThread`.
Performance about the same, using the benchmark from gh-120754:
```
pyperf compare_to main.json pep782.json
read_file_small: Mean +- std dev: [main] 5.71 us +- 0.05 us -> [pep782] 5.68 us +- 0.05 us: 1.01x faster
read_file_large: Mean +- std dev: [main] 89.7 us +- 0.9 us -> [pep782] 86.9 us +- 0.8 us: 1.03x faster
read_all_rst_bytes: Mean +- std dev: [main] 926 us +- 8 us -> [pep782] 920 us +- 12 us: 1.01x faster
read_all_rst_text: Mean +- std dev: [main] 2.24 ms +- 0.02 ms -> [pep782] 2.17 ms +- 0.04 ms: 1.03x faster
Benchmark hidden because not significant (1): read_all_py_bytes
Geometric mean: 1.01x faster
```
* Add documentation links to datetime strftime/strptime docstrings
- Add links to format codes documentation for all strftime methods
- Add links to format codes documentation for all strptime methods
- Addresses issue #97517
* Update C extension docstrings with format codes documentation
* Regenerate clinic code for updated docstrings
* Add clinic-generated header file for updated docstrings
* Fix docstring spacing consistency in both Python and C files
* Update Lib/_pydatetime.py
Co-authored-by: Stan Ulbrych <89152624+StanFromIreland@users.noreply.github.com>
---------
Co-authored-by: Stan Ulbrych <89152624+StanFromIreland@users.noreply.github.com>
Replace PyBytes_FromStringAndSize(NULL, size) with the new public
PyBytesWriter API.
Don't build the fcntl with the limited C API anymore, since
the PyBytesWriter API is not part of the limited C API.
The stack collector base class keeps all frames until export() is
called, which causes significant unnecessary memory usage. Instead, we
can process the frames on the fly in the collect call by dispatching the
aggregation logic to the subclass through the process_frames method.
Co-authored-by: Pablo Galindo Salgado <pablogsal@gmail.com>
Use the new public PyBytesWriter API in:
* _PyBytes_FromHex()
* _PyBytes_FromBuffer()
* _PyBytes_FromList()
* _PyBytes_FromTuple()
* _PyBytes_FromIterator()
Add _PyBytesWriter_ResizeAndUpdatePointer() and
_PyBytesWriter_GetAllocated() helper functions.
Rename `pathlib._os.magic_open()` to `vfsopen()`. The new name is a bit
less abstract, and it aligns with the `vfspath()` method added in 5dbd27d.
Per discussion on discourse[^1], adjust `vfsopen()` so that the following
methods may be called:
- `__open_reader__()`
- `__open_writer__(mode)`
- `__open_updater__(mode)`
These three methods return readable, writable, and full duplex file objects
respectively. In the 'writer' method, *mode* is either 'a', 'w' or 'x'. In
the 'updater' method, *mode* is either 'r' or 'w'.
In the pathlib ABCs, replace `ReadablePath.__open_rb__()` with
`__open_reader__()`, and replace `WritablePath.__open_wb__()` with
`__open_writer__()`.
[^1]: https://discuss.python.org/t/open-able-objects/90238
Co-authored-by: Petr Viktorin <encukou@gmail.com>
The following types are now immutable:
* `_curses_panel.panel`,
* `[posix,nt].ScandirIterator`, `[posix,nt].DirEntry` (exposed in `os.py`),
* `_remote_debugging.RemoteUnwinder`,
* `_tkinter.Tcl_Obj`, `_tkinter.tkapp`, `_tkinter.tktimertoken`,
* `zlib.Compress`, and `zlib.Decompress`.
Remove dead code in code gen - codegen_check_annotation is only called if future annotations are enabled, and if future annotations are enabled it does nothing.
The codecs lookup function now performs only minimal normalization of
the encoding name before passing it to the search functions:
all ASCII letters are converted to lower case, spaces are replaced
with hyphens.
Excessive normalization broke third-party codecs providers, like
python-iconv.
Revert "bpo-37751: Fix codecs.lookup() normalization (GH-15092)"
This reverts commit 20f59fe1f7.
Follow-up refactoring after GH-133143, where we use `_Py_ID` for "big" and "little"
in `abi_info.byteorder`.
This uses `_Py_ID` for `sys.byteorder`, but also `float_repr_style` and a module name.
In bf8bbe9a81, `getpass.getpass` gained
the ability to provide keyboard feedback through `echo_char`.
On Unix, line editing shortcuts such as Ctrl+U were previously handled
as the terminal operates in canonical mode (see termios(3)). However,
since keyboard feedback requires to switch to noncanonical mode, this
now results in an inconsistency when `getpass.getpass` uses `echo_char`
as those shortcuts are no more supported. This limitation is specific
to Unix and does not affect Windows users where line editing shortcuts
were never supported.
While file timestamps can be anything the file system can store, most
lie between the recent past and the near future. Optimize fill_time()
for typical timestamps in three ways:
- When possible, convert to nanoseconds with C arithmetic.
- When using C arithmetic and the seconds member is not required (for
st_birthtime), avoid creating a long object.
- When using C arithmetic, reorder the code to avoid the null checks
implied in Py_XDECREF().
Co-authored-by: Victor Stinner <vstinner@python.org>
Adds a --no-randomize option to the CI runner, so that randomisation can be easily
disabled for --fast-ci and --slow-ci configurations on single-threaded testing platforms
like Android, iOS, and Emscripten.
---------
Co-authored-by: Malcolm Smith <smith@chaquo.com>
This makes information about the interpreter ABI more accessible.
Co-authored-by: Petr Viktorin <encukou@gmail.com>
Co-authored-by: Victor Stinner <vstinner@python.org>
Co-authored-by: Adam Turner <9087854+AA-Turner@users.noreply.github.com>
Move PYTHON_API_VERSION & PYTHON_ABI_VERSION to patchlevel.h
This allows removing #include "modsupport.h" from python_ver_rc.h,
which allows modsupport.h to use common helpers from Python.h --
specifically, `_Py_PACK_VERSION` for defining limited API.
* KeyError is not raised for defaultdict
* Fix relative paths on different drives on Windows
* Add a round-trip test
Co-authored-by: Itamar Oren <itamarost@gmail.com>
The signature algorithms allowed for certificate-based client authentication or
for the server to complete the TLS handshake can be defined on a SSL context via
`ctx.set_client_sigalgs()` and `ctx.set_server_sigalgs()`.
With OpenSSL 3.4 or later, the list of available TLS algorithms can be retrieved
by `ssl.get_sigalgs()`.
With OpenSSL 3.5 or later, the selected signature algorithms can be retrieved from
SSL sockets via `socket.client_sigalg()` and `socket.server_sigalg()`.
This commit also partially amends 377b787618
by using `PyUnicode_DecodeFSDefault` instead of `PyUnicode_DecodeASCII` in
`_ssl._SSLContext.get_groups`, so that functions consistently decode strings
obtained from OpenSSL.
---------
Co-authored-by: Bénédikt Tran <10796600+picnixz@users.noreply.github.com>
Some common tests in `test.list_tests.CommonTest` explicitly tested `list`
instead of testing the underlying list-like type defined in `type2test`.
---------
Co-authored-by: Devansh Baghla <devanshbaghla34@gmail.com>
gh-126631: gh-137996: fix pre-loading of `__main__`
The `main_path` parameter was renamed `init_main_from_name`, update the
forkserver code accordingly. This was leading to slower startup times when people
were trying to preload the main module.
---------
Co-authored-by: Gregory P. Smith <greg@krypto.org>
fix potential hang.
It can happen that the child crashes right in the beginning for whatever reason. In this case, the parent will hang when writing into the pipe, because the child fd is not closed yet.
The normal pattern is to close the child fds right after the child proc is forked/executed/spawned, so when the child dies, then also the pipes will be closed, and there will be no hang (the parent gets SIGPIPE instead).
Like cygwin, MUSL defaults to utf-8 if no variables are set. I have no
idea if the existing tests pass on cygwin, but I made the modifications
such that I shouldn't break it if is. The additional checks needed for
MUSL are guarded by DEFAULT_LOCALE_IS_C being False. Based on this
flag, we expect utf-8 for the encodings and no coercion message, as
long as LC_ALL is not set to C. (That looks like a bit of an issue with
the test structure, but I'm not going to attempt to "fix" that.)
DEFAULT_ENCODING is intentionally not given a default since it is only
used when DEFAULT_LOCALE_IS_C is False, and if you use the flag you'll
need to set it.
After reading through issue 30672, looking at the source, and running a
test on Android, I *think* the current situation is that coercion will
be done if the local is set to POSIX regardless of platform. However,
if the platform doesn't make POSIX equivalent to C, the encodings when
coercion is disabled will not be the same as for C (it is utf-8 on
android, for example). This means the tests would fail if POSIX were
added unconditionally to the EXPECTED_C_LOCALE_EQUIVALENTS as envisioned
in the issue. This *could* be fixed with another flag, but I'm not sure
it is worth the effort. I'm not even sure Python is behaving optimally
in this case (assuming my analysis is correct). So I just altered the
comment and add POSIX if and only if the platform is linux.
* Don't fail trying to parse weird patterns.
* Don't fail trying to decode non-UTF-8 "robots.txt" files.
* No longer ignore trailing "?" in patterns and URLs.
* Distinguish raw special characters "?", "=" and "&" from the
percent-encoded ones.
* Remove tests that do nothing.
It should interpret the result of wcsxfrm() as a sequence of abstract
integers, not a sequence of Unicode code points or using other encoding
scheme that does not preserve ordering.
GH-119186: Slightly speed up `os.walk(topdown=True)`
When `os.walk()` traverses into subdirectories in top-down mode, call
`os.path.join()` once to add a trailing slash, and use string concatenation
thereafter to generate child paths.
Reduce what happens in `load_tests` so that the next change,
moving the `Buffered*` tests to `test_bufferdio` is purely mechanical
movement and updating imports.
This adds two classes, one per I/O implementation, to act as dispatch to
the implementation-specific mocks as well as module members. Previously
the mappings CTestCase and PyTestCase provide were injected directly
during `load_tests`.
CTestCase and PyTestCase inherit from `unittest.TestCase` so when the
split happens default test discovery will work for the classes in
`test_bufferedio`. `test_general` keeps a manual test list for this
refactoring; some of the tests (ex. `ProtocolsTest`) aren't currently
run and fixing that + helpers to not be picked up is out of my current
scope.
CTestCase and PyTestCase have an `io` class member which points to the
implementation meaning that can be removed from individual test cases
which now inherit from them.
This code is picking up `MockRawIO` which is defined globally in the
module but these should use the mock specific to the I/O implementation
being tested.
Co-authored-by: Victor Stinner <vstinner@python.org>
Centralize `io` tests into the `test_io` module so they are easier to
find and work on. This will make it easier to split `test_general` which
takes 30+ seconds in a debug build on my machine.
This renames `test_bufio` to be `test_bufferedio` so that it matches
the implementation file name (`bufferedio.c`).
Validation performed:
Tests are run in parallel after change:
```bash
./python.exe -m test test_io -uall,largefile,extralargefile -M12G -j8
```
Docstring reformat in `test_io/__init__.py` looks reasonable:
```python
>>> import test.test_io
>>> help(test.test_io)
```
Co-authored-by: Adam Turner <9087854+AA-Turner@users.noreply.github.com>
Previously, the size would be returned on Windows and an OSError would
be raised on Unix.
Also, raise ValueError instead of OSError for trackfd=False.
Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>
Encode Solaris/Illumos thread names to ASCII, since
OpenIndiana does not support non-ASCII names.
Add tests for setting non-ASCII name for the main thread.
Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>
* Reapply "gh-132947: Apply changes from importlib_metadata 8.7 (#137885)" (#137924)
This reverts commit 3706ef66ef.
* Skip the triggering test on buildbots only.
Emscripten's libc is a hybrid of musl and llvm libc; but it reports that it is
"glibc". This modifies the return value of `platform.libc_ver()` to return
something that is Emscripten-specific.
Co-authored-by: Adam Turner <9087854+AA-Turner@users.noreply.github.com>
Co-authored-by: Peter Bierma <zintensitydev@gmail.com>
Co-authored-by: sobolevn <mail@sobolevn.me>
Co-authored-by: Stan Ulbrych <89152624+StanFromIreland@users.noreply.github.com>
The `Tools/ssl/multissltests.py` script may extract a possibly untrusted tarball.
Since the script does not necessarily use Python 3.14 or later (where the `"data"`
filter became the default `tarfile` extraction filter), the user may theoretically
suffer from a path traversal attack.
Although the script should not be used in production and usually relies on downloading
trusted sources, the `"data"` extraction filter is now explicitly used wherever relevant.
* Merge existing tests test_repr_str and test_to_str.
* Add more tests for non-printable and non-ASCII bytes.
* Add tests for special escape sequences ('\t\n\r').
* Add tests for slashes.
* Add more tests for quotes.
* Add tests for subclasses.
* Add test for non-ASCII class name.
* Only apply @check_bytes_warnings for str() tests.
Don't ignore errors raised by `PyErr_WarnFormat` in `warn_about_fork_with_threads`
Instead, ignore the warnings in all test code that forks. (That's a lot of functions.)
In `test_support`, make `ignore_warnings` a context manager (as well as decorator),
and add a `message` argument to it.
Also add a `ignore_fork_in_thread_deprecation_warnings` helper for the deadlock-in-fork
warning.
We already use an anonymous union for PyObject. This makes the workarounds available in all public headers:
- MSVC: `__pragma(warning(disable: 4201))` (with push/pop). Warning 4201 is specifically for anonymous unions, so let's disable for all of `<Python.h>`
- GCC/clang, pedantic old C standards: define `_Py_ANONYMOUS` as `__extension__`
- otherwise, define `_Py_ANONYMOUS` as nothing
(Note that this is only for public headers -- CPython internals use C11, which has anonymous structs/unions.)
C API WG vote: https://github.com/capi-workgroup/decisions/issues/74
This partially reverts commit d83e30cadd
by bringing back the CI job for testing OpenSSL 1.1.1w. Despite this
version being upstream EOL, the rationale for keeping it as follows:
- It most resembles other 1.1.1-work-a-like ssl APIs supported by important vendors.
- Python officially requires OpenSSL 1.1.1 or later, although OpenSSL 3.0 or later
is recommended for cryptographic modules. Since changing the build requirements
requires a transition period, we need to keep testing the allowed versions.
- The code base still contains calls to OpenSSL functions that are deprecated since
OpenSSL 3.0 as well as `ifdef` blocks constrained to OpenSSL 1.1.1.
The csv.register_dialect() docstring no longer imply that it returns a
dialect.
All functions have now signatures.
Co-authored-by: maurycy <5383+maurycy@users.noreply.github.com>
Specifically, clarify that ZipInfo.date_time pulls the datetime information from the central directory, and that times are interpreted as local time.
Co-authored-by: Emma Smith <emma@emmatyping.dev>
Co-authored-by: sobolevn <mail@sobolevn.me>
Co-authored-by: Ned Batchelder <ned@nedbatchelder.com>
Co-authored-by: Tomas R. <tomas.roun8@gmail.com>
Co-authored-by: Petr Viktorin <encukou@gmail.com>
Modifies the iOS testbed project to add a test plan. This simplifies the iOS
test runner, as we can now use the built-in log streaming to see test results.
It also allows for some other affordances, like providing a default LLDB config,
and using a standardized mechanism for specifying test arguments.
A runtime check is needed to support cross-compiling.
Remove the _Py_NORMALIZE_CENTURY macro.
Remove _pydatetime.py's _can_support_c99.
Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>
It is now a positive integer larger larger than any limited resource value.
This simplifies comparison of the resource values.
Previously, it could be negative, such as -1 or -3, depending on platform.
Deprecation warning is emitted if the old negative value is passed.
This partially reverts #137047, keeping the tests for GC collectability of the
original class that dataclass adds `__slots__` to.
The reference leaks solved there are instead solved by having the `__dict__` &
`__weakref__` descriptors not tied to (and referencing) their class.
Instead, they're shared between all classes that need them (within
an interpreter).
The `__objclass__` ol the descriptors is set to `object`, since these
descriptors work with *any* object. (The appropriate checks were already
made in the get/set code, so the `__objclass__` check was redundant.)
The repr of these descriptors (and any others whose `__objclass__` is `object`)
now doesn't mention the objclass.
This change required adjustment of introspection code that checks
`__objclass__` to determine an object's “own” (i.e. not inherited) `__dict__`.
Third-party code that does similar introspection of the internals will also
need adjusting.
Co-authored-by: Jelle Zijlstra <jelle.zijlstra@gmail.com>
* Sync with importlib_metadata 8.2.0
Removes deprecated behaviors, including support for `PackageMetadata.__getitem__` returning None for missing keys and Distribution subclasses not implementing abstract methods.
Prioritizes valid dists to invalid dists when retrieving by name (python/cpython/#120492). Adds SimplePath to `importlib.metadata.__all__`.
* Add blurb
The Python pickle module looks for "00" and "01" but _pickle only looked
for 2 characters that parsed to 0 or 1, meaning some payloads like "+0" or
" 0" would lead to different results in different implementations.
"] ]>" and "]] >" no longer end the CDATA section.
Make CDATA section parsing context depending.
Add private method HTMLParser._set_support_cdata() to change the context.
If called with True, "<[CDATA[" starts a CDATA section which ends with "]]>".
If called with False, "<[CDATA[" starts a bogus comments which ends with ">".
* Add OpenSSL 3.5.2 definitions to Modules/_ssl_data_35.h (moved from Modules/_ssl_data_34.h)
* Demote OpenSSL 3.1 to "old", remove it from CI
* Update all OpenSSL versions to latest patchlevel in CI config and multissltests defaults
* Add OpenSSL 3.5.2 to CI configuration and multissltests default list
* Fix a typo in the argument parser description of multissltests.py
There were a few thread-safety issues when profiling or tracing all
threads via PyEval_SetProfileAllThreads or PyEval_SetTraceAllThreads:
* The loop over thread states could crash if a thread exits concurrently
(in both the free threading and default build)
* The modification of `c_profilefunc` and `c_tracefunc` wasn't
thread-safe on the free threading build.
The previous behavior was copied from earlier typing code. It works around the way
typing.get_type_hints passes its namespaces, but I don't think the behavior is logical
or correct.
Some systems reportedly don't expand '%OB' and '%Ob'.
In this case (and similar theoretically possible ones, like expanding to empty
string or 'OB'), fall back to the month_name & month_abbr.
* Docs: Move Enum functions and add examples
When the `Enum` functions `_add_alias_` and `_add_value_alias_` were added in de6bca9564, the documentation for them was done under `EnumType` instead of `Enum`.
This change moves them to the docs of the `Enum` class and adds an example for each function.
---------
Co-authored-by: Ethan Furman <ethan@stoneleaf.us>
Remove the `__dict__` and `__weakref__` descriptors from the original class when creating a dataclass from it.
An interesting hack, but more localized in scope than gh-135230.
This may be a breaking change if people intentionally keep the original class around
when using `@dataclass(slots=True)`, and then use `__dict__` or `__weakref__` on the
original class.
Co-authored-by: Alyssa Coghlan <ncoghlan@gmail.com>
Co-authored-by: Petr Viktorin <encukou@gmail.com>
Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>
Enough other classes in `importlib.abc` inherit from the class and the deprecation was to redirect people to `TraversableResources`. The documentation now makes it clear the class only exists for backwards compatibility.
---------
Co-authored-by: Hugo van Kemenade <1324225+hugovk@users.noreply.github.com>
The `PyEval_SetProfileAllThreads` function and other related functions
had a race condition on `tstate->c_profilefunc` that could lead to a
crash when disable profiling or tracing on all threads while another
thread is starting to profile or trace a a call.
There are still potential crashes when threads exit concurrently with
profiling or tracing be enabled/disabled across all threads.
* - Add an explainer guide (aka HOWTO, not how-to) for asyncio.
* Fix linter errors.
* - Enforce max line length of roughly 79 chars.
- Start sentences on new lines to minimize disruption of diffs.
* Add reference to subinterpreters.
* - Significantly reduce article size. Remove both example sections & "Which concurrency do I want" section.
* Align section-header lengths with section names.
* - Remove reference to deleted section.
* - Fix a variety of rote style guide items like title-alignment, use of ie and $, and so forth.
- Add links to other parts of the docs for keywords and objects like await, coro, task, future, etc.
* - One last title alignment.
* - Style nit.
* - Rework a variety of I statements.
* Lint fix.
* - Firm up commentary on yield from in corotuines.
* Update language comparing await and yield from.
* - Remove await-ing Tasks and futures section
* Update Doc/howto/a-conceptual-overview-of-asyncio.rst
Co-authored-by: Peter Bierma <zintensitydev@gmail.com>
* Update Doc/howto/a-conceptual-overview-of-asyncio.rst
Co-authored-by: Peter Bierma <zintensitydev@gmail.com>
* - Address comments related to style & writing flow.
* per-thread event loop note.
* Add section describing coroutines roots in generators.
* Phrasing tweak.
* Use asyncio.create_task instead of asyncio.Task
* Update Doc/howto/a-conceptual-overview-of-asyncio.rst
Co-authored-by: Peter Bierma <zintensitydev@gmail.com>
* Update Doc/howto/a-conceptual-overview-of-asyncio.rst
Co-authored-by: Peter Bierma <zintensitydev@gmail.com>
* small phrasing.
* Update Doc/howto/a-conceptual-overview-of-asyncio.rst
Co-authored-by: Peter Bierma <zintensitydev@gmail.com>
* phrasing nit.
* style nits
* Update Doc/howto/a-conceptual-overview-of-asyncio.rst
Co-authored-by: Peter Bierma <zintensitydev@gmail.com>
* phrasing nit
* Fix misnaming of async generator.
* phrasing nits.
* Update Doc/howto/a-conceptual-overview-of-asyncio.rst
Co-authored-by: Peter Bierma <zintensitydev@gmail.com>
* Update Doc/howto/a-conceptual-overview-of-asyncio.rst
Co-authored-by: Peter Bierma <zintensitydev@gmail.com>
* consistent spacing
* Update Doc/howto/a-conceptual-overview-of-asyncio.rst
Co-authored-by: Peter Bierma <zintensitydev@gmail.com>
* phrasing nits
* Update Doc/howto/a-conceptual-overview-of-asyncio.rst
Co-authored-by: Peter Bierma <zintensitydev@gmail.com>
* Update Doc/howto/a-conceptual-overview-of-asyncio.rst
Co-authored-by: Peter Bierma <zintensitydev@gmail.com>
* Update Doc/howto/a-conceptual-overview-of-asyncio.rst
Co-authored-by: Peter Bierma <zintensitydev@gmail.com>
* Update Doc/howto/a-conceptual-overview-of-asyncio.rst
Co-authored-by: Peter Bierma <zintensitydev@gmail.com>
* add conclusion
* nits
* - Variety of style & grammar improvements thanks to ZeroIntensity's comments.
* - Make all directives start with a 3 space indent. Then 4 thereafter.
* - Use :linenos: instead of manually writing the line numbers.
* - Fix label typo for article.
* fix label link.
* Apply suggestions from code review
Co-authored-by: 🇺🇦 Sviatoslav Sydorenko (Святослав Сидоренко) <wk.cvs.github@sydorenko.org.ua>
Co-authored-by: Carol Willing <carolcode@willingconsulting.com>
Co-authored-by: Peter Bierma <zintensitydev@gmail.com>
* - introduce async-sleep name
* Phrasing
* nit
* ungendered octopus
* teammates
* jobs
* rework fella to penguin
* - remove byline; add seealso
* Change ref from asyncio to use seealso block.
* Remove typehints. Fix indentation in one code example.
* Slight rephrase for clarity.
* Make references point to asyncio. Wrap some long lines.
* - Variety of style/phrasing improvements based on PR feedback.
* phrasing.
* phrasing nit.
* Apply suggestions from code review
Co-authored-by: Peter Bierma <zintensitydev@gmail.com>
* Update Doc/howto/a-conceptual-overview-of-asyncio.rst
Co-authored-by: Carol Willing <carolcode@willingconsulting.com>
* nit
* Apply suggestions from code review
Co-authored-by: Carol Willing <carolcode@willingconsulting.com>
* fix backticks.
* nits
* nit
* add section on asyncio.run
* title change under the hood.
* modify task coro example.
* howtos article link.
* prefer await without backticks.
* phrasing tweak.
* Rework phrasing around how await tasks pauses and returns control in the await section.
* move code block to beforfe explanation in coroutine under the hood.
* phrasing.
* link to yield from.
* style nits
* nit
* - Modify language re: event-loop cycling endlessly.
- Discuss why await was designed to not yield for coros.
* - Add a note about debug=True on asyncio.run to await coro section.
* clarity nit
* - Add two other references in seealso block.
* nit
* Language simplification
* Apply suggestions from code review
Co-authored-by: Peter Bierma <zintensitydev@gmail.com>
* nit
* grammar fix.
* fix
* worker bees
* rework event loop paragraph to significantly deemphasize queues
* remove all references to queue besides the initial analogy.
* add note about garbage collection of tasks
* add practical note re: garbage collection
* phrasing nits
* re arrange note on task gc.
* line wrap nit
* Update Doc/howto/a-conceptual-overview-of-asyncio.rst
Co-authored-by: Kumar Aditya <kumaraditya@python.org>
* link to debug mode docs.
* readd part2 prefix.
* simplify title.
* fix titles. tihnk I messed this up earlier.
* avoid idiom in title.
* fix titles once agian.
* Apply suggestions from code review
Co-authored-by: Stan Ulbrych <89152624+StanFromIreland@users.noreply.github.com>
* rework task gc example.
* phrasing tweak.
* tewak.
* nit
* nit
* nit
* nit
---------
Co-authored-by: Peter Bierma <zintensitydev@gmail.com>
Co-authored-by: 🇺🇦 Sviatoslav Sydorenko (Святослав Сидоренко) <wk.cvs.github@sydorenko.org.ua>
Co-authored-by: Carol Willing <carolcode@willingconsulting.com>
Co-authored-by: Kumar Aditya <kumaraditya@python.org>
Co-authored-by: Stan Ulbrych <89152624+StanFromIreland@users.noreply.github.com>
* gh-106320: Remove private _PyInterpreterState functions (#106335)
Remove private _PyThreadState and _PyInterpreterState C API
functions: move them to the internal C API (pycore_pystate.h and
pycore_interp.h). Don't export most of these functions anymore, but
still export functions used by tests.
Remove _PyThreadState_Prealloc() and _PyThreadState_Init() from the C
API, but keep it in the stable API.
* Doc: minor change
* Revert "Doc: minor change"
This reverts commit ebfa0937c2.
* [Doc] Remove unnecessary quotes from typing (See Also section)
* [Doc] Remove unnecessary quotes from typing
---------
Co-authored-by: Victor Stinner <vstinner@python.org>
These are tests to ensure behaviour introduced by GH-136189 is working as expected.
Co-authored-by: Mikhail Borisov <43937008+fxeqxmulfx@users.noreply.github.com>
Co-authored-by: Kumar Aditya <kumaraditya@python.org>
Co-authored-by: Neil Schemenauer <nas-github@arctrix.com>
Fix a bug caused by the garbage collector clearing weakrefs too early. The
weakrefs in the ``tp_subclasses`` dictionary are needed in order to correctly
invalidate type caches (for example, by calling ``PyType_Modified()``).
Clearing weakrefs before calling finalizers causes the caches to not be
correctly invalidated. That can cause crashes since the caches can refer to
invalid objects. Defer the clearing of weakrefs without callbacks until after
finalizers are executed.
In GH-116206, the comment about moving reachable objects to next generation
got moved from its original place to a place where there is no code below
it. Put the comment back to where the actual movement of reachable objects
happens.
* Return large limit values as positive integers instead of negative integers
in resource.getrlimit().
* Accept large values and reject negative values (except RLIM_INFINITY)
for limits in resource.setrlimit().
Update `validate_abstract_methods` in `test_collections.py`
The test for missing abstract methods in `validate_abstract_methods` incorrectly attempted to instantiate the generated class `C` with an argument (`C(name)`), which always raises a `TypeError: C() takes no arguments`. Although the test originally passes, it passes for the wrong reason.
This change makes the test correctly validate the enforcement of abstract methods in ABCs.
Fix name of the Python encoding in Unicode errors of the code page
codec: use "cp65000" and "cp65001" instead of "CP_UTF7" and "CP_UTF8"
which are not valid Python code names.
* Update SQLite to 3.50.3 for binary releases.
* macOS and Windows news entries. what about Android?
* update sbom hash
* newline fix via regen-sbom
* news wording
* Update SQLite to 3.50.4 for binary releases.
* update 3.50.4.0.tar.gz hash in sbom & regen-sbom to fix whitespace
* Postpone to a separate PR the build-installer changes to support additional hash types
Co-authored-by: Erlend E. Aasland <erlend.aasland@protonmail.com>
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Ned Deily <nad@python.org>
The existing link works, but includes the legacy `/all` part of the path
which causes a scary-looking banner about a misconfiguration on the
`Home` page when in reality it's the link that includes a deprecated path.
X25519 is not a valid curve if OpenSSL is built with FIPS mode,
and ignoring unknown groups in `SSL_CTX_set1_groups_list()`
is only supported since OpenSSL 3.3, so we use two curves that
are known to be FIPS-compliant, namely P-256 and P-384.
Default implementation of sys.unraisablehook() now uses traceback._print_exception_bltin() to print exceptions with colorized text.
Co-authored-by: Bénédikt Tran <10796600+picnixz@users.noreply.github.com>
Co-authored-by: Victor Stinner <vstinner@python.org>
Move docs to the reference section & reduce the “tutorial” part to a quick intro & link.
Clarify what values are accepted.
Add macro/attribute equivalents.
Discourage _align_ values that aren't powers of two.
Remove feature coverage comment from msgfmt script's docstring
msgfmt handles plural forms since cb081b83.
The comment was incorrectly applied in 637a33b -- original patch was from before the cb081b83 commit.
Co-authored-by: Tomas R. <tomas.roun8@gmail.com>
The calendar module displays month names in some locales using the genitive case.
This is grammatically incorrect, as the nominative case should be used when the month
is named by itself. To address this issue, this change introduces new lists
`standalone_month_name` and `standalone_month_abbr` that contain month names in
the nominative case -- or more generally, in the form that should be used to
name the month itself, rather than form a date.
The module now uses the `%OB` format specifier to get month names in this form
where available.
The implementation does not create anymore local functions which reduces
the overhead for small inputs. Some other calls are inlined into a
single `_convert_literal` function.
We have a gain of 10-20% for small inputs and only 1-2% for bigger
inputs.
Call backtrace() once when installing the signal handler to ensure that
libgcc is dynamically loaded outside the signal handler.
This fixes a "signal-unsafe call inside of a signal" TSan error from
test_faulthandler.test_enable_fd.
Co-authored-by: Adam Turner <9087854+AA-Turner@users.noreply.github.com>
Co-authored-by: Hugo van Kemenade <1324225+hugovk@users.noreply.github.com>
Co-authored-by: Stan Ulbrych <89152624+StanFromIreland@users.noreply.github.com>
Add support for getting and setting groups used for key agreement.
* `ssl.SSLSocket.group()` returns the name of the group used
for the key agreement of the current session establishment.
This feature requires Python to be built with OpenSSL 3.2 or later.
* `ssl.SSLContext.get_groups()` returns the list of names of groups
that are compatible with the TLS version of the current context.
This feature requires Python to be built with OpenSSL 3.5 or later.
* `ssl.SSLContext.set_groups()` sets the groups allowed for key agreement
for sockets created with this context. This feature is always supported.
The `Modules/hashlib.h` helper file is now removed and split into multiple files:
* `Modules/_hashlib/hashlib_buffer.[ch]` -- Utilities for getting a buffer view and handling buffer inputs.
* `Modules/_hashlib/hashlib_fetch.h` -- Utilities used when fetching a message digest from a digest-like identifier.
Currently, this file only contains common error messages as the fetching API is not yet implemented.
* `Modules/_hashlib/hashlib_mutex.h` -- Utilities for managing the lock on cryptographic hash objects.
* Reword, expand, and clarify the limitation, highlighting the REPL case.
* Mention in the high level Process description.
* added a pointer to the GH issue from the doc note
---------
Co-authored-by: Gregory P. Smith <greg@krypto.org>
Adjust `pathlib._os.vfspath()` so that it doesn't try `os.fsdecode()`. I
don't know that supporting `os.PathLike` arguments is a good idea, so
it's best to leave it out for now.
The OpenSSL and HACL* implementations of HMAC single-shot
digest computation reject keys whose length exceeds `INT_MAX`
and `UINT32_MAX` respectively. The OpenSSL implementation
also rejects messages whose length exceed `INT_MAX`.
Using such keys in `hmac.digest` previously raised an `OverflowError`
which was propagated to the caller. This commit mitigates this case by
making `hmac.digest` fall back to HMAC's pure Python implementation
which accepts arbitrary large keys or messages.
This change only affects the top-level entrypoint `hmac.digest`, leaving
`_hashopenssl.hmac_digest` and `_hmac.compute_digest` untouched.
Previously, if OpenSSL was not present and built-in cryptographic extension modules
were disabled, requesting `hashlib.<name>` raised `AttributeError` and an ERROR log
message with the exception traceback is emitted when importing `hashlib`.
Now, the named constructor function will always be available but raises a `ValueError`
at runtime indicating that the algorithm is not supported. The log message has also
been reworded to be less verbose.
Previously, DocTest's lineno of functions and methods decorated with
functools.cache(), functools.lru_cache() and functools.cached_property()
was not properly returned (None was returned) because the
computation relied on inspect.isfunction() which does not consider the
decorated result as a function.
We now use the more generic inspect.isroutine(), as elsewhere
in doctest's logic.
Also, added a special case for functools.cached_property().
_testclinic.c mocks out PY_VERSION_HEX to 3.8 before including
_testclinic_depr.c.h to avoid the errors the preprocessor would
otherwise throw due to the deprecation feature it is testing.
Also partially revert 74e2acddf6:
this restores Modules/_testclinic.c to match the same file in the 3.14
branch.
De-instrumenting code objects modifies the thread local bytecode for all threads as such, holding the critical section on the code object is not sufficient and leads to data races. Now, the de-instrumentation is now performed under a stop the world pause as such no thread races with executing the thread local bytecode while it is being de-instrumented.
Basic support for pyrepl in Emscripten. Limitations:
* requires JSPI
* no signal handling implemented
As followup work, it would be nice to implement a webworker variant
for when JSPI is not available and proper signal handling.
Because it requires JSPI, it doesn't work in Safari. Firefox requires
setting an experimental flag. All the Chromiums have full support since
May. Until we make it work without JSPI, let's keep the original web_example
around.
Co-authored-by: Łukasz Langa <lukasz@langa.pl>
Co-authored-by: Éric <merwok@netwok.org>
Co-authored-by: Petr Viktorin <encukou@gmail.com>
Co-authored-by: Hugo van Kemenade <1324225+hugovk@users.noreply.github.com>
Co-authored-by: Loïc Simon <loic.pano@gmail.com>
Co-authored-by: pauleveritt <pauleveritt@me.com>
Modifies the test runner script to no longer export the the HOST environment
variable, and to allow for tests that produce no Python output (output from the
Android console is still expected and required). These changes stem from
knowledge gained during developing a PR for Android support in cibuildwheel.
An interesting hack, but more localized in scope than #135230.
This may be a breaking change if people intentionally keep the original class around
when using `@dataclass(slots=True)`, and then use `__dict__` or `__weakref__` on the
original class.
Co-authored-by: Alyssa Coghlan <ncoghlan@gmail.com>
* Revert "gh-84481: Make ZipFile.data_offset more robust (#132178)"
This reverts commit 6cd1d6c6b1.
* Revert "gh-84481: Add ZipFile.data_offset attribute (#132165)"
This reverts commit 0788948dcb.
---------
Co-authored-by: Gregory P. Smith <greg@krypto.org>
This makes the following APIs public:
* `Py_BEGIN_CRITICAL_SECTION_MUTEX(mutex),`
* `Py_BEGIN_CRITICAL_SECTION2_MUTEX(mutex1, mutex2)`
* `void PyCriticalSection_BeginMutex(PyCriticalSection *c, PyMutex *mutex)`
* `void PyCriticalSection2_BeginMutex(PyCriticalSection2 *c, PyMutex *mutex1, PyMutex *mutex2)`
The macros are identical to the corresponding `Py_BEGIN_CRITICAL_SECTION` and
`Py_BEGIN_CRITICAL_SECTION2` macros (e.g., they include braces), but they
accept a `PyMutex` instead of an object.
The new macros are still paired with the existing END macros
(`Py_END_CRITICAL_SECTION`, `Py_END_CRITICAL_SECTION2`).
In free-threading, multiple threads can be cleared concurrently as such the modifications on `sys_tracing_threads` should be done while holding the profile lock, otherwise it can race with other threads setting up profiling.
`_datetime` is a special module, because it's the only non-builtin C extension that contains static types. As such, it would initialize static types in the module's execution function, which can run concurrently. Since static type initialization is not thread-safe, this caused crashes. This fixes it by moving the initialization of `_datetime`'s static types to interpreter startup (where all other static types are initialized), which is already properly protected through other locks.
Make the setlogmask() function in the syslog module thread-safe. These changes are relevant for scenarios where the GIL is disabled or when using subinterpreters.
Fix the `test_generated_cases` to work with `-O` or `-OO` flags.
Previously, `test_generated_cases` was catching an `AssertionError` while `Tools/cases_generator/optimizer_generator.py` used an `assert` statement. This approach semantically incorrect, no one should trying to catch an `AssertionError`!
Now the `assert` statement has been replaced with an explicit `raise ValueError(...)` and the corresponding `self.assertRaisesRegex(AssertionError, ...)` has been updated to catch a `ValueError` instead.
UUIDv8 has been added in Python 3.14.0a2 and its construction time
has been improved in Python 3.14.0a4, but since those changes will
not be visible when comparing the latest Python 3.13 and 3.14 together,
we do not document them on the What's New page to avoid confusion.
Co-authored-by: Hugo van Kemenade <1324225+hugovk@users.noreply.github.com>
Co-authored-by: Łukasz Langa <lukasz@langa.pl>
Co-authored-by: Adam Turner <9087854+aa-turner@users.noreply.github.com>
gh-91349: Adjust default compression level to 6 (down from 9) in gzip and tarfile
It is the default level used by most compression tools and a
better tradeoff between speed and performance.
Co-authored-by: rmorotti <romain.morotti@man.com>
Co-authored-by: Pieter Eendebak <pieter.eendebak@gmail.com>
Some internal helper functions taking the module object to retrieve its state
under some conditions now directly take the module's state instead as those
conditions hold most of the time.
- Fix `hashlib_helper.block_algorithm` where the dummy functions were incorrectly defined.
- Rename `hashlib_helper.HashAPI` to `hashlib_helper.HashInfo` and add more helper methods.
- Simplify `hashlib_helper.requires_*()` functions.
- Rewrite some private helpers in `hashlib_helper`.
- Remove `find_{builtin,openssl}_hashdigest_constructor()` as they are no more needed and were
not meant to be public in the first place.
- Fix some tests in `test_hashlib` when FIPS mode is on.
This is useful for implementing proper `input()`. It requires the
JavaScript engine to support the wasm JSPI spec which is now stage 4.
It is supported on Chrome since version 137 and on Firefox and node
behind a flag.
We override the `__wasi_fd_read()` syscall with our own variant that
checks for a readAsync operation. If it has it, we use our own async
variant of `fd_read()`, otherwise we use the original `fd_read()`.
We also add a variant of `FS.createDevice()` called
`FS.createAsyncInputDevice()`.
Finally, if JSPI is available, we wrap the `main()` symbol with
`WebAssembly.promising()` so that we can stack switch from `fd_read()`.
If JSPI is not available, attempting to read from an AsyncInputDevice
will raise an `OSError`.
Add a copy of the text from SimpleQueue.close()
---------
Co-authored-by: saggarwal145 <saggarwal145@bloomberg.net>
Co-authored-by: Petr Viktorin <encukou@gmail.com>
* Doc/c-api/memory.rst: extend --without-pymalloc doc with ASan information
This commit extends the documentation for disabling pymalloc with the `--without-pymalloc` flag regarding why it is worth to use it when enabling AddressSanitizer for Python build (which is done, e.g., in CPython's CI builds).
I have tested the CPython latest main build with both ASan and pymalloc enabled and it seems to work just fine. I did run the `python -m test` suite which didn't uncover any ASan crashes (though, it detected some memory leaks, which I believe are irrelevant here).
I have discussed ASan and this flag with @encukou on the CPython Core sprint on EuroPython 2025. We initially thought that the `--without-pymalloc` flag is needed for ASan builds due to the fact pymalloc must hit the begining of page when determining if the memory to be freed comes from pymalloc or was allocated by the system malloc. In other words, we thought, that ASan would crash CPython during free of big objects (allocated by system malloc). It may be that this was the case in the past, but it is not the case anymore as the `address_in_range` function used by pymalloc is annotated to be skipped from the ASan instrumentation.
This code can be seen here:
acefb978dc/Objects/obmalloc.c (L2096-L2110)
While the annotation macro is defined here:
acefb978dc/Include/pyport.h (L582-L598)
And the corresponding attribute is documented in:
* for gcc: https://gcc.gnu.org/onlinedocs/gcc/Common-Function-Attributes.html#index-no_005fsanitize_005faddress-function-attribute
* for clang: https://clang.llvm.org/docs/AttributeReference.html#no-sanitize-address-no-address-safety-analysis
* Update Doc/c-api/memory.rst
* Improve --with-address-sanitizer and pymalloc docs
---------
Co-authored-by: Petr Viktorin <encukou@gmail.com>
Fixed-sized types, like ``c_int32``, are currently missing from the fundamental data types table
in the ``ctypes`` documentation. This commit adds them, and notes that ``c_[u]int8`` is an alias
of ``c_[u]byte``.
The current example `batched('ABCDEFG', n=3) → ABC DEF G` can confuse readers because both, the size of the tuples and the number of tuples are 3.
By using a batch size of n=2, it is clearer that the `n` argument refers to the size of the resulting tuples.
I.e. the new example is: `batched('ABCDEFG', n=2) → AB CD EF G`
Explicitly pass an `optimizer` parameter to the calls of `ast.parse/compile`, because if it is not provided, the interpreter will use its internal state, which can be modified using the `-O` or `-OO` flags.
Co-authored-by: Kirill Podoprigora <kirill.bast9@mail.ru>
Make the pwd module functions getpwuid(), getpwnam(), and getpwall() thread-safe. These changes apply to scenarios where the GIL is disabled or in subinterpreter use cases.
Clears the umask used during a test of pydoc.apropos when testing on
Emscripten. This is to work around a known issue in Emscripten; but it's not
clear if the chmod call that is causing the problem is actually testing
anything of significance.
Provide a stub implementation of umask that is enough to get some tests passing.
More work is needed upstream in Emscripten to make all umask tests to pass.
* Improved venv docs to note that isolation is the default.
* Insert "that" so that a sentence reads better.
* Improved wording.
---------
Co-authored-by: Vinay Sajip <vinay_sajip@yahoo.co.uk>
This helps catch double deallocation bugs and is similar to the
assertion in the GIL-enabled build. The call to `validate_refcounts`
is moved up to start of the GC because `queue_untracked_obj_decref()`
creates it own zero reference count garbage.
Users new to Python packaging often try to use pip from the REPL only to
be met with a confusing SyntaxError. If this happens, guide the user to
use a system terminal instead to invoke pip.
Closes#72327
---------
Co-authored-by: Tom Viner <tom@viner.tv>
Co-authored-by: Brian Schubert <brianm.schubert@gmail.com>
Co-authored-by: Hugo van Kemenade <1324225+hugovk@users.noreply.github.com>
Co-authored-by: Alyssa Coghlan <ncoghlan@gmail.com>
Fix "msvcrt" import warning on Linux when "_ctypes" is not available.
On Linux, compiling without "libffi" causes a
"No module named 'msvcrt'" warning when launching PyREPL.
Make grp module methods getgrgid() and getgrnam() thread-safe when the GIL is disabled and getgrgid_r()/getgrnam_r() C APIs are not available.
---------
Co-authored-by: Kumar Aditya <kumaraditya@python.org>
Previously, we assumed that instrumentation would happen for all copies of
the bytecode if the instrumentation version on the code object didn't match
the per-interpreter instrumentation version. That assumption was incorrect:
instrumentation will exit early if there are no new "events," even if there
is an instrumentation version mismatch.
To fix this, include the instrumented opcodes when creating new copies of
the bytecode, rather than replacing them with their uninstrumented variants.
I don't think we have to worry about races between instrumentation and creating
new copies of the bytecode: instrumentation and new bytecode creation cannot happen
concurrently. Instrumentation requires that either the world is stopped or the
code object's per-object lock is held and new bytecode creation requires holding
the code object's per-object lock.
Change the names of the symbol tables for lambda expressions and generator
expressions to "<lambda>" and "<genexpr>" respectively to avoid conflicts
with user-defined names.
For unsigned integer formats in the PyArg_Parse* functions,
accepting Python integers with value that is larger than
the maximal value the corresponding C type or less than
the minimal value for the corresponding signed integer type
is now deprecated.
Allow Py_LIMITED_API for (Py_GIL_DISABLED && _Py_OPAQUE_PYOBJECT)
API that's removed when _Py_OPAQUE_PYOBJECT is defined:
- PyObject_HEAD
- _PyObject_EXTRA_INIT
- PyObject_HEAD_INIT
- PyObject_VAR_HEAD
- struct _object (i.e. PyObject) (opaque)
- struct PyVarObject (opaque)
- Py_SIZE
- Py_SET_TYPE
- Py_SET_SIZE
- PyModuleDef_Base (opaque)
- PyModuleDef_HEAD_INIT
- PyModuleDef (opaque)
- _Py_IsImmortal
- _Py_IsStaticImmortal
Note that the `_Py_IsImmortal` removal (and a few other issues)
means _Py_OPAQUE_PYOBJECT only works with limited
API 3.14+ now.
Co-authored-by: Victor Stinner <vstinner@python.org>
Co-authored-by: Adam Turner <9087854+AA-Turner@users.noreply.github.com>
Co-authored-by: Bénédikt Tran <10796600+picnixz@users.noreply.github.com>
Co-authored-by: Hugo van Kemenade <1324225+hugovk@users.noreply.github.com>
Co-authored-by: Zachary Ware <zach@python.org>
This commit fixes the following problems:
* The x86_64 trampolines are not preserving frame pointers
* The hardcoded offsets to the code segment from the FDE only worked properly for x64_64
* The CIE data was not following conventions of aarch64
* The eh_frame for aarch64 was not fully correct
- Use %T format specifier instead of %s and Py_TYPE(x)->tp_name.
- Remove legacy %.200s format specifier for truncating type names.
Co-authored-by: Victor Stinner <vstinner@python.org>
* Fix flag mask inversion when unnamed flags exist.
For example:
class Flag(enum.Flag):
A = 0x01
B = 0x02
MASK = 0xff
~Flag.MASK is Flag(0)
* EJECT and KEEP flags (IntEnum is KEEP) use direct value.
* correct Flag inversion to only flip flag bits
IntFlag will flip all bits -- this only makes a difference in flag sets with
missing values.
* correct negative assigned values in flags
negative values are no longer used as-is, but become inverted; i.e.
class Y(self.enum_type):
A = auto()
B = auto()
C = ~A # aka ~1 aka 0b1 110 (from enum.bin()) aka 6
D = auto()
assert Y.C. is Y.B|Y.D
Implement a statistical sampling profiler that can profile external
Python processes by PID. Uses the _remote_debugging module and converts
the results to pstats-compatible format for analysis.
Co-authored-by: Pablo Galindo <pablogsal@gmail.com>
Weakrefs to unreachable garbage that are created during running of
finalizers need to be cleared. This avoids exposing objects that
have `tp_clear` called on them to Python-level code.
Fixes build errors encountered in python-greenlet/greenlet#450 when building greenlet on the free-threaded build.
---------
Co-authored-by: Peter Bierma <zintensitydev@gmail.com>
Co-authored-by: Victor Stinner <vstinner@python.org>
* gh-135913: Document ob_refcnt, ob_type, ob_size
In `typeobj.rst`, instead of `:c:member:` it would be better to
use `.. c:member::` with a `:no-index:` option, see:
See ref. https://www.sphinx-doc.org/en/master/usage/domains/index.html#basic-markup
However, `c:member` currently does not support `:no-index:`.
* Turn the __future__ table to list-table.
This'll make it easier to add entries that need longer markup
* Semantic markup for __future__ feature descriptions.
* Document CO_* C macros.
* Remove the table
* Replace warnings with notes
Latest releases of Python 3.9-3.15 include expat 2.7.1 which is not vulnerable.
expat 2.6.0 was released in February 2024.
This is still formally undefined behaviour, but we may as well
keep the *same* undefined behaviour as previous versions.
PEP 796 proposes a cleaner and more consistent replacement for 3.15+
This fixes the data races in typeobject.c in subinterpreters under free-threading. The type flags and slots are only modified in the main interpreter as all static types are first initialised in main interpreter.
* Whitespaces no longer accepted between `</` and the tag name.
E.g. `</ script>` does not end the script section.
* Vertical tabulation (`\v`) and non-ASCII whitespaces no longer recognized
as whitespaces. The only whitespaces are `\t\n\r\f `.
* Null character (U+0000) no longer ends the tag name.
* Attributes and slashes after the tag name in end tags are now ignored,
instead of terminating after the first `>` in quoted attribute value.
E.g. `</script/foo=">"/>`.
* Multiple slashes and whitespaces between the last attribute and closing `>`
are now ignored in both start and end tags. E.g. `<a foo=bar/ //>`.
* Multiple `=` between attribute name and value are no longer collapsed.
E.g. `<a foo==bar>` produces attribute "foo" with value "=bar".
* Whitespaces between the `=` separator and attribute name or value are no
longer ignored. E.g. `<a foo =bar>` produces two attributes "foo" and
"=bar", both with value None; `<a foo= bar>` produces two attributes:
"foo" with value "" and "bar" with value None.
* Fix Sphinx errors.
* Apply suggestions from code review
Co-authored-by: Ezio Melotti <ezio.melotti@gmail.com>
* Address review comments.
* Move to Security.
---------
Co-authored-by: Ezio Melotti <ezio.melotti@gmail.com>
Move PYOS_LOG2_STACK_MARGIN, PYOS_STACK_MARGIN,
PYOS_STACK_MARGIN_BYTES and PYOS_STACK_MARGIN_SHIFT macros to
pycore_pythonrun.h internal header. Add underscore (_) prefix to the
names to make them private. Rename _PYOS to _PyOS.
After Python finalization gets to the point where no other thread
can attach thread state, attempting to acquire a Python lock must hang.
Raise PythonFinalizationError instead of hanging.
This adds a "macro" to the optimizer DSL called "REPLACE_OPCODE_IF_EVALUATES_PURE", which allows automatically constant evaluating a bytecode body if certain inputs have no side effects upon evaluations (such as ints, strings, and floats).
Co-authored-by: Tomas R. <tomas.roun8@gmail.com>
* Add detection of decimal non-ASCII alt digits.
* Add support of non-decimal alt digits on locale lzh_TW.
* Accept only numbers in correct range if alt digits are known.
* Fix bug in detecting the position of the week day name on locales byn_ER and wal_ET.
* Fix support of single-digit hour on locales ar_SA and bg_BG.
* Add support for %T, %R, %r, %C, %OC.
* Prepare code to use nl_langinfo().
New scheme from Stefan Pochmann for picking minimum run lengths.
By allowing them to change a little from one run to the next, it's possible to
arrange for that all merges, at all levels, strongly tend to be as evenly balanced
as possible, for randomly ordered data. Meaning the number of initial runs is a
power of 2, and all merges involve runs whose lengths differ by no more than 1.
The iOS testbed now treats the app_packages folder as a site folder. This ensures it is
on the path, but also ensures any .pth files are processed on app startup.
0x8b correctly encodes to ‹, but 0x9b was mistakenly marked as a control character instead of ›.
---------
Co-authored-by: blurb-it[bot] <43283697+blurb-it[bot]@users.noreply.github.com>
Co-authored-by: Brian Schubert <brianm.schubert@gmail.com>
Co-authored-by: Terry Jan Reedy <tjreedy@udel.edu>
* gh-91555: add warning to docs about possibility of deadlock/infinite recursion
Attempt to clarify in the documentation that care must be taken when using
multiprocessing classes to implement logging since they have builtin internal
logging, and hence may cause deadlock/infinite recursion.
* Update Doc/library/logging.handlers.rst
Co-authored-by: Vinay Sajip <vinay_sajip@yahoo.co.uk>
* Change whitespace.
---------
Co-authored-by: Vinay Sajip <vinay_sajip@yahoo.co.uk>
The documentation incorrectly stated that generator.close() 'raises' a
GeneratorExit exception. This was misleading because the method doesn't
raise the exception to the caller - it sends the exception internally
to the generator and returns None.
Paragraph should not be under `slice.step`. It applies to the whole class.
---------
Co-authored-by: Rob Reynolds <13379223+reynoldsnlp@users.noreply.github.com>
The free threading build uses QSBR to delay the freeing of dictionary
keys and list arrays when the objects are accessed by multiple threads
in order to allow concurrent reads to proceed with holding the object
lock. The requests are processed in batches to reduce execution
overhead, but for large memory blocks this can lead to excess memory
usage.
Take into account the size of the memory block when deciding when to
process QSBR requests.
Also track the amount of memory being held by QSBR for mimalloc pages. Advance the write sequence if this memory exceeds a limit. Advancing the sequence will allow it to be freed more quickly.
Process the held QSBR items from the "eval breaker", rather than from `_PyMem_FreeDelayed()`. This gives a higher chance that the global read sequence has advanced enough so that items can be freed.
Co-authored-by: Sam Gross <colesbury@gmail.com>
Using Ubuntu 24.04 on the Windows Subsystem for Linux, perf will raise a
`PermissionError` instead of `FileNotFoundError`. This commit modifies
the tests to catch that.
* find defined "(fetch)" remotes with "python/cpython" in their URL
* if there is exactly one, use that remote name
* if there is one named "upstream", "origin", or "python",
use that remote (in that precedence order)
* otherwise report an error listing the defined remotes
- Add a fast path when the digest length is 0 to avoid calling useless functions.
- Directly allocate via `PyBytes_FromStringAndSize(NULL, length)` when possible.
Passing a negative digest length to `_hashilb.HASHXOF.[hex]digest()` now
raises a ValueError instead of a MemoryError or a SystemError. This makes
the behavior consistent with that of `_sha3.shake_{128,256}.[hex]digest`.
Temporarily skip test_os.test_mode on Emscripten; this fails consistently
on the buildbot, but not on other test configurations. Reported as #135783
for follow up.
We weren't handling non-positive maxsize values (including the default) properly
in Queue.full(). This change fixes that and adjusts an associated assert.
We also sync the docs for UUIDv1 and UUIDv6 concerning the node address and clock sequence.
---------
Co-authored-by: Bénédikt Tran <10796600+picnixz@users.noreply.github.com>
Most importantly, this resolves the issues with functions and types defined in __main__.
It also expands the number of supported objects and simplifies the implementation.
* Remove duplicated code. Tests for Random and SystemRandom now share
the code.
* Move implementation agnostic tests that was only run for SystemRandom,
so they are now run for Random too.
* Add tests for __index__() support.
* Add tests for randint().
If a preloaded module writes to stdout or stderr, and the stream is buffered,
child processes will inherit the buffered data after forking. Attempt to
prevent this by flushing the streams after preload.
Co-authored-by: Mikhail Efimov <efimov.mikhail@gmail.com>
Co-authored-by: Victor Stinner <vstinner@python.org>
The Emscripten path resolver uses the same mechanism for resolving `..`
at a file system root as for resolving symlinks. This is because
roots don't store their mountpoints. If the parent of a node is itself,
it is a root but it might be a mountpoint in some other file system.
If a path has enough `..`'s at the root, it will return ELOOP.
Enough turns out to be 49.
As noted in the new tests, there are a few situations we must carefully accommodate
for functions that get pickled during interp.call(). We do so by running the script
from the main interpreter's __main__ module in a hidden module in the other
interpreter. That hidden module is used as the function __globals__.
This PR adds a PyJitRef API to the JIT's optimizer that mimics the _PyStackRef API. This allows it to track references and their stack lifetimes properly. Thus opening up the doorway to refcount elimination in the JIT.
* gh-134632: Fix `build-details.json` to use `INCLUDEPY` path
Fix ``build-details.json`` generation to use ``INCLUDEPY``, in order to
reference the ``pythonX.Y`` subdirectory of the include directory, as
required in :pep:`739`, instead of the top-level include directory.
* test_build_details: Add tests for the c_api section
* test_build_details: Expect pkgconfig for CPython unconditionally
Remove `experimental` qualification for free-threading in the document text. Note that images included in the document will be updated later in the release cycle.
For several builtin functions, we now fall back to __main__.__dict__ for the globals
when there is no current frame and _PyInterpreterState_IsRunningMain() returns
true. This allows those functions to be run with Interpreter.call().
The affected builtins:
* exec()
* eval()
* globals()
* locals()
* vars()
* dir()
We take a similar approach with "stateless" functions, which don't use any
global variables.
Use `ma_used` instead of `ma_keys->dk_nentries` for modification check
so that we only check if the dictionary is modified, not if new keys are
added to a different dictionary that shared the same keys object.
* gh-67022: Document bytes/str inconsistency in email.header.decode_header()
This function's possible return types have been surprising and error-prone
for the entirety of its Python 3.x history. It can return either:
1. `typing.List[typing.Tuple[bytes, typing.Optional[str]]]` of length >1
2. or `typing.List[typing.Tuple[str, None]]`, of length exactly 1
This means that any user of this function must be prepared to accept either
`bytes` or `str` for the first member of the 2-tuples it returns, which is a
very surprising behavior in Python 3.x, particularly given that the second
member of the tuple is supposed to represent the charset/encoding of the
first member.
This patch documents the behavior of this function, and adds test cases
to demonstrate it.
As discussed in bpo-22833, this cannot be changed in a backwards-compatible
way, and some users of this function depend precisely on the existing
behavior.
Add warnings about obsolescence of 'email.header.decode_header' and 'email.header.make_header' functions.
Recommend use of `email.headerregistry.HeaderRegistry` instead, as suggested
in https://github.com/python/cpython/pull/92900#discussion_r1112472177
Prior to issue #120485 these servers did not allow port reuse, which
makes sense as the behavior of port reuse is surprising if you're not
expecting it. It's unclear to me why these services were switched to
allow port reuse, but I believe the desired behavior (unless subclasses
opt in) is to not allow port reuse.
See also: https://bugzilla.redhat.com/show_bug.cgi?id=2323170
* gh-135171: Update documentation for the generator expression
Document that the iterator for the leftmost "for" clause is created
immediately.
* Update Doc/reference/expressions.rst
Co-authored-by: Brian Skinn <brian.skinn@gmail.com>
---------
Co-authored-by: Brian Skinn <brian.skinn@gmail.com>
In this refactor we:
* move some code around
* make a couple of typedefs opaque
* decouple errors from session state
* improve tracebacks for propagated exceptions
This change helps simplify several upcoming changes.
Document behaviour of single-phase init. Call it "legacy".
Reorganize PyModule docs.
Move PyInit_modulename docs from the tutorial to reference documentation.
Move PyMODINIT_FUNC docs from generic macros to the new page.
Add doc stubs for `PYTHON_API_VERSION` & `PYTHON_ABI_VERSION`
Remove incorrect refcounts.dat entry for `PyModuleDef_Init`.
This removes the "Return value: Borrowed reference." note.
Instead, note that the function sometimes returns a borrowed reference,
sometimes as strong one.
(IMO, it's best to not think of `PyModuleDef` as a `PyObject` at all,
and act like it can't be reference-counted.)
Co-authored-by: Adam Turner <9087854+AA-Turner@users.noreply.github.com>
Co-authored-by: Stan Ulbrych <89152624+StanFromIreland@users.noreply.github.com>
Co-authored-by: Hugo van Kemenade <1324225+hugovk@users.noreply.github.com>
PEP-734 has been accepted (for 3.14).
(FTR, I'm opposed to putting this under the concurrent package, but
doing so is the SC condition under which the module can land in 3.14.)
* Replace _Py_ALIGN_AS(V) by _Py_ALIGNED_DEF(N, T)
This is now a common façade for the various `_Alignas` alternatives,
which behave in interesting ways -- see the source comment.
The new macro (and MSVC's `__declspec(align)`) should not be used
on a variable/member declaration that includes a struct declaraton.
A workaround is to separate the struct definition.
Do that for `PyASCIIObject.state`.
* Specify minimum PyGC_Head and PyObject alignment
As documented in InternalDocs/garbage_collector.md, the garbage collector
stores flags in the least significant two bits of the _gc_prev pointer
in struct PyGC_Head. Consequently, this pointer is only capable of storing
a location that's aligned to a 4-byte boundary.
Encode this requirement using _Py_ALIGNED_DEF.
This patch fixes a segfault in m68k, which was previously investigated
by Adrian Glaubitz here:
https://lists.debian.org/debian-68k/2024/11/msg00020.htmlhttps://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1087600
Original patch (using the GCC-only Py_ALIGNED) by Finn Thain.
Co-authored-by: Finn Thain <fthain@linux-m68k.org>
Co-authored-by: Victor Stinner <vstinner@python.org>
Co-authored-by: John Paul Adrian Glaubitz <glaubitz@physik.fu-berlin.de>
When shutdown is called with wait=False, the executor thread keeps running
even after the ProcessPoolExecutor's state is reset. The executor then tries
to replenish the worker processes pool resulting in an error and a potential hang
when it comes across a worker that has died. Fixed the issue by having
_adjust_process_count() return without doing anything if the ProcessPoolExecutor's
state has been reset.
Added unit tests to validate two scenarios:
max_workers < num_tasks (exception)
max_workers > num_tasks (exception + hang)
Use critical sections to make heapq methods that update the heap thread-safe when the GIL is disabled.
---------
Co-authored-by: mpage <mpage@meta.com>
Prepare the docs for using the notation used in the `python.gram`
file. If we want to sync the two, the meta-syntax should be the same.
Link the Full Grammar docs here; keep only a few extras.
Also, remove the distinction between lexical and syntactic rules,
except for whitespace handling.
With f- and t-strings, the line between the two is blurry.
Co-authored-by: Blaise Pabon <blaise@gmail.com>
Co-authored-by: Adam Turner <9087854+AA-Turner@users.noreply.github.com>
Co-authored-by: Lysandros Nikolaou <lisandrosnik@gmail.com>
Co-authored-by: Colin Marquardt <cmarqu42@gmail.com>
* Cleanup imports and update module docstring.
* Simplify detection of SIMD support.
* Correctly guard `update()` cases.
* Rewrite `py_blake2b_or_s_new` and rename it to `py_blake2_new`.
* Rewrite `blake2_blake2b_copy_locked` and `py_blake2_clear`.
* Refactor computations of `digest` and `hexdigest`.
* Simplify `py_blake2b_get_name` and `py_blake2b_get_block_size`.
* Add `hacl_get_blake2_info` to extract static BLAKE-2 information.
This new helper is used by `py_blake2b_get_digest_size`, but can
be later used to expose `key_length` more easily.
Append the defect to defects instead of to the parse tree.
Co-authored-by: Bénédikt Tran <10796600+picnixz@users.noreply.github.com>
Co-authored-by: Hugo van Kemenade <1324225+hugovk@users.noreply.github.com>
Fix Py_RETURN_NONE, Py_RETURN_TRUE and Py_RETURN_FALSE macros in the
limited C API 3.11 and older:
Don't treat Py_None, Py_True and Py_False as immortal.
Set sys.stdout encoder error handler to backslashreplace in regrtest
workers to avoid UnicodeEncodeError when printing a traceback
or any other non-encodable character.
Move the code from the Regrtest class to setup_process().
Call setup_process() earlier, before displaying regrtest headers.
Modifies the environment handling and execution arguments of the Android management
script to support the compilation of third-party binaries, and the use of the testbed to
invoke third-party test code.
Co-authored-by: Adam Turner <9087854+AA-Turner@users.noreply.github.com>
Co-authored-by: Russell Keith-Magee <russell@keith-magee.com>
We were incorrectly handling a few opcodes that leave their operands on the stack. Treat all of these conservatively; assume that they always leave operands on the stack.
The `test_ssl_in_multiple_threads` test failed because `test_check_hostname_idn()`
modified the global warnings filters via `warnings_helper.check_no_resource_warning()`.
Only check for warnings when the context aware warnings feature is enabled, which makes
the warnings filter context-local and thread-safe.
On Windows, the `_PyOS_SigintEvent()` event handle is used to interrupt
the main thread when Ctrl-C is pressed. Previously, we also waited on
the event from other threads, but ignored the result. However, this can
race with interpreter shutdown because the main thread closes the handle
in `_PySignal_Fini` and threads may still be running and using mutexes
during interpreter shtudown.
Only use `_PyOS_SigintEvent()` in the main thread in parking_lot.c, like
we do in other places in the CPython codebase.
On a fresh Xcode install (including some CI provider configurations), there is
no pre-existing testing set that can be used to identify simulator models. Use
the default device set to detect available models instead. Live testing
simulators are still created in the testing set.
When sanity checking against gettotalrefcount(), we exclude the blocks for
immortalized strings since their references are not tracked/reported. This
now matches refleak.py's book-keeping using the same functions.
Apply Intel Control-flow Technology for x86-64 on asm_trampoline.S.
Required for mitigation against return-oriented programming (ROP)
and Call or Jump Oriented Programming (COP/JOP) attacks.
Manual application is required for the assembly files.
See also: https://sourceware.org/annobin/annobin.html/Test-cf-protection.html
* Hard-cap max file descriptors in subprocess test fd_status
On some systems, `SC_OPEN_MAX` may return a very large value (i.e. 10**30), leading to the subprocess test timing out (or run forever).
Prevent this situation by applying a hard cap on how many file descriptors are checked.
* Fix typo in usage docstring
s/fd_stats/fd_status/
Run each example as a subtest in unit tests synthesized by
doctest.DocFileSuite() and doctest.DocTestSuite().
Add the doctest.DocTestRunner.report_skip() method.
OpenSSL and HACL*-based hash functions constructors now support both `data` and `string` parameters.
Previously these constructor functions inconsistently supported sometimes `data` and sometimes `string`,
while the documentation expected `data` to be given in all cases.
Reorder result tuple of parse_code_object
The standard followed by APIs like pstat.Stats is to take a file, line,
function triplet. The parse_code_object function (and callers exposing
this in Python like RemoteUnwinder.get_stack_trace) return function,
file, line triplets which requires the caller to reorder these when
using it in classes like pstat.Stats.
We add the following attributes on `_mi_assert_fail` to help IDE introspection:
* `__attribute__((__noreturn__))`
* `__attribute__((cold))`
* `__THROW` (GCC only)
In 121ed71f4e, mt_continue_should_break
was changed to be guarded by `Py_DEBUG`, but it's used in `compress_mt_continue_lock_held`
with just `assert`, so it needs to be available when `NDEBUG` is undefined
too.
`Py_DEBUG` implies `NDEBUG` is undefined, so we can check just that.
Fixes: 121ed71f4e
Contrary to the current docs, ast.Constant will never hold containers
such as frozenset or tuple; the Python parser only emits it for simple
literals.
For precision, add the exact list of types that may be contained in an
ast.Constant.
The problem we're fixing here is that we were using PyDict_Size() on "defaults",
which it is actually a tuple. We're also adding some explicit type checks.
This is a follow-up to gh-133221/gh-133528.
Replace most PyUnicodeWriter_WriteUTF8() calls with
PyUnicodeWriter_WriteASCII().
Unrelated change to please the linter: remove an unused
import in test_ctypes.
Co-authored-by: Peter Bierma <zintensitydev@gmail.com>
Co-authored-by: Bénédikt Tran <10796600+picnixz@users.noreply.github.com>
Use "backslashreplace" error handler to decode stdout and stderr.
Example:
vstinner@WIN C:\victor\python\main\build\test_python_worker_8360\x91>
"C:\victor\python\main\PCbuild\amd64\python_d.exe" -m test
--fast-ci --slow-ci --testdir
C:\Users\vstinner\AppData\Local\Temp\tmp0t59e8da
test_regrtest_noop1 test_regrtest_noop2 test_regrtest_noop3
test_regrtest_noop4
Notice the "\x91" byte at the end of the first line: it's the
non-ASCII U+00E6 character encoded to the OEM cp437 code page.
Add _Py_PACK_VERSION for CPython's own definitions
Py_PACK_VERSION was added to limited API in 3.14, so if
Py_LIMITED_API is lower, the macro can't be used.
Add a private version that can be used in CPython headers
for checks like `Py_LIMITED_API+0 >= _Py_PACK_VERSION(3, 14)`.
In the free-threaded build, avoid data races caused by updating type
slots or type flags after the type was initially created. For those
(typically rare) cases, use the stop-the-world mechanism. Remove the
use of atomics when reading or writing type flags.
This is the same underlying bug as gh-130519. The destructor may call
arbitrary code, changing the `tstate->qsbr pointer` and invalidating the
old `struct _qsbr_thread_state`.
[main] Update stdtypes.rst
- Added explicit mention of `del s[i]` (item deletion by index) to the Mutable Sequence Types section.
- Clarified that this operation removes the item at the specified index from the sequence.
- Addresses issue #134789.
Adds `_PyObject_GetMethodStackRef` which uses stackrefs and takes advantage of deferred reference counting in free-threading while calling method objects in vectorcall.
This is a small follow-up to gh-133481. There's a corner case
in the behavior of PyImport_ImportModuleAttrString(), where
it expects __builtins__ to be set if __globals__ is set.
* FOR_ITER now pushes either the iterator and NULL or leaves the iterable and pushes tagged zero
* NEXT_ITER uses the tagged int as the index into the sequence or, if TOS is NULL, iterates as before.
Some curses module-level functions and window methods now raise
a `curses.error` when a call to a C curses function fails:
- Module-level functions: assume_default_colors, baudrate, cbreak,
echo, longname, initscr, nl, raw, termattrs, termname, and unctrl.
- Window methods: addch, addnstr, addstr, border, box, chgat,
getbkgd, inch, insstr, and insnstr.
In addition, `curses.window.refresh` and `curses.window.noutrefresh`
now raise a `TypeError` instead of a `curses.error` when called with an
incorrect number of arguments for pads.
See also ee36db5500 for similar
changes.
* Add t-string prefixes to _all_string_prefixes, and add a test to make sure we catch this error in the future.
* Update lexical analysis docs for t-string prefixes.
Disable immortalization around Py_CompileString*().
The same approach as 332356b that fixed the refleaks in compile() and eval().
E: 09e72cf can pass test_capi, test_sys and test__interpchannels with this patch for me.
- Mention (again) that `type.__annotations__` is unsafe. It is now safe
when using only classes defined under PEP 649 semantics, but not with
classes defined using `from __future__ import annotations`.
- Mention that annotations on instances no longer work. There was already
an issue about this.
- Mention the general changes in the "Porting to Python 3.14" section.
- `annotationlib` was proposed by PEP-749, not PEP-649.
Co-authored-by: Emma Smith <emma@emmatyping.dev>
Co-authored-by: Carol Willing <carolcode@willingconsulting.com>
Rename components related to `_hashlib.{HASH,HASHXOF}` objects.
- The `EVPobject` structure is renamed `HASHobject`.
- Non-clinic `HASH` methods are now prefixed by `_hashlib_HASH_*`.
A similar change is made for non-clinic `HASHXOF` methods.
- Functions extracting information from `EVP_MD` objects and functions
constructing `EVP_MD` objects now include `openssl_evp_md` in their name.
This change allows us to avoid future ambiguities between the `EVP_MD`
and the `EVP_MAC` APIs (currently, we only use `EVP_MD` for hash functions
and rely on the legacy interface for HMAC instead of using `EVP_MAC`).
* gh-132876: workaround broken ldexp() on Windows 10
ldexp() fails to round subnormal results before Windows 11,
so hide their bug.
Co-authored-by: Tim Peters <tim.peters@gmail.com>
Fix an UnboundLocalError that can occur when parsing certain delimited constructs in headers (domain literals, quoted strings, comments). After the fix the _get_ptext_to_endchars returns an empty string if there is no content after the opening delimiter. The calling code is responsible for handling the lack of the trailing delimiter, which it already does; this edge case was the header ending immediately after the opening delimiter.
Completely refactor Modules/_remote_debugging_module.c with improved
code organization, replacing scattered reference counting and error
handling with centralized goto error paths. This cleanup improves
maintainability and reduces code duplication throughout the module while
preserving the same external API.
Implement memory page caching optimization in Python/remote_debug.h to
avoid repeated reads of the same memory regions during debugging
operations. The cache stores previously read memory pages and reuses
them for subsequent reads, significantly reducing system calls and
improving performance.
Add code object caching mechanism with a new code_object_generation
field in the interpreter state that tracks when code object caches need
invalidation. This allows efficient reuse of parsed code object metadata
and eliminates redundant processing of the same code objects across
debugging sessions.
Optimize memory operations by replacing multiple individual structure
copies with single bulk reads for the same data structures. This reduces
the number of memory operations and system calls required to gather
debugging information from the target process.
Update Makefile.pre.in to include Python/remote_debug.h in the headers
list, ensuring that changes to the remote debugging header force proper
recompilation of dependent modules and maintain build consistency across
the codebase.
Also, make the module compatible with the free threading build as an extra :)
Co-authored-by: Łukasz Langa <lukasz@langa.pl>
As explained in #133960, this removes most of the behavior differences with ForwardRef.evaluate.
The remaining difference is about recursive evaluation of forwardrefs; this is practically useful
in cases where an annotation refers to a type alias that itself is string-valued.
This also improves several edge cases that were previously not handled optimally. For example,
the function now takes advantage of the partial evaluation behavior of ForwardRef.evaluate() to
evaluate more ForwardRefs in the FORWARDREF format.
This also fixes#133959 as a side effect, because the buggy behavior in #133959 derives from
evaluate_forward_ref().
Doc/library/multiprocessing.rst: freeze_support: Change to specify spawn method instead of platform
Have multiprocessing.freeze_support() enable on spawn, not just win32.
---------
Co-authored-by: Gregory P. Smith <greg@krypto.org>
GH-128840: Limit the number of parts in IPv6 address parsing
Limit length of IP address string to 39
---------
Co-authored-by: Hugo van Kemenade <1324225+hugovk@users.noreply.github.com>
Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>
Co-authored-by: Gregory P. Smith <greg@krypto.org>
Previously the module would fail to load because the `alloca()` symbol
was undefined. Now we check for GCC/Clang builtins for systems who do
not define `alloca()` in headers.
Move from using critical sections to locks for the (de)compression methods.
Since the methods allow other threads to run, we should use a lock rather
than a critical section.
Removed special-casing for WASI when setting C stack depth limits. Since WASI has its own C stack checking this isn't a security risk.
Also disabled some tests that stopped passing. They all happened to have already been disabled under Emscripten.
This is mostly a refactor to clean things up a bit, most notably the "XI namespace" code.
Making the session opaque requires adding the following internal-only functions:
* _PyXI_NewSession()
* _PyXI_FreeSession()
* _PyXI_GetMainNamespace()
Set the LC_CTYPE locale to the LC_TIME locale even if
nl_langinfo(ALT_DIGITS) result is ASCII. The result is a list
separated by NUL characters and the code only checks the first list
item which can be ASCII whereas following items are non-ASCII.
Fix test__locale for the uk_UA locale on RHEL 7.
Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>
Fix `build-details.json` generation to use the correct `c_api.headers`
key as defined in PEP 739, instead of `c_api.include`.
Co-authored-by: Filipe Laíns 🇵🇸 <lains@riseup.net>
When inexperienced users create a PR from their default branch, all of the concurrency keys
collide as there is no namespacing. This becomes an issue at events with many new contributors,
where workflow runs are cancelled on other pull requests.
Disambiguate by adding the username of the relevant 'actor' to the concurrency key.
Co-authored-by: Adam Turner <9087854+AA-Turner@users.noreply.github.com>
Co-authored-by: Sviatoslav Sydorenko <sviat@redhat.com>
Co-authored-by: C.A.M. Gerlach <CAM.Gerlach@Gerlach.CAM>
Authored-by: Adam Turner <9087854+AA-Turner@users.noreply.github.com>
The `__main__` module imported in the `_pyrepl` module points to the `_pyrepl` module itself when the interpreter was launched without `-m` option and didn't execute a module,
while it's an unexpected behavior that `__main__` can be `_pyrepl` and relative imports such as `from . import *` works based on the `_pyrepl` module.
Co-authored-by: Łukasz Langa <lukasz@langa.pl>
Update `test_io` `_check_warn_on_dealloc` to use `self.` to dispatch to
different I/O implementations.
Update the `_pyio` implementation to match expected behavior, using the
same `_dealloc_warn` design as the C implementation uses to report the
topmost `__del__` object.
The FileIO one now matches all the others, so can use IOBase. There was
a missing check on closing (self._fd must be valid), add that check
Two special methods, __buffer__ and __release_buffer__ were added to
Python 3.12 by PEP 688. The C API Type Object documentation for slots
includes `tp_as_buffer`, and sub-slots `bf_getbuffer`, `bf_releasebuffer`
but does not refer to the Python Data Model version of those. Add the
missing references.
Add documentation for compression & compression.zstd.
🎉
---------
Co-authored-by: Adam Turner <9087854+AA-Turner@users.noreply.github.com>
Co-authored-by: Stan Ulbrych <89152624+StanFromIreland@users.noreply.github.com>
Co-authored-by: Hugo van Kemenade <1324225+hugovk@users.noreply.github.com>
Co-authored-by: Sumana Harihareswara <sh@changeset.nyc>
Co-authored-by: Martin Panter <vadmium@users.noreply.github.com>
It now supports a "full" fallback to _PyFunction_GetXIData() and then `_PyPickle_GetXIData()`. There's also room for other fallback modes if that later makes sense.
Switches over to a _Py_thread_local in place of autoTssKey, and also fixes a few other checks regarding PyGILState_Ensure after finalization.
Note that this doesn't fix concurrent use of PyGILState_Ensure with Py_Finalize; I'm pretty sure zapthreads doesn't work at all, and that needs to be fixed seperately.
bpo-28494: Improve zipfile.is_zipfile reliability
The zipfile.is_zipfile function would only search for the EndOfZipfile
section header. This failed to correctly identify non-zipfiles that
contained this header. Now the zipfile.is_zipfile function verifies
the first central directory entry.
Changes:
* Extended zipfile.is_zipfile to verify zipfile catalog
* Added tests to validate failure of binary non-zipfiles
* Reuse 'concat' handling for is_zipfile
Co-authored-by: John Jolly <john.jolly@gmail.com>
* Mention remote debugging via -p PID in usage text
Adds a brief note to the pdb help summary about attaching to a running
process using the -p option, making the remote debugging feature
more visible.
* Mention remote debugging in pdb.rst
* made curses buffer heap allocated instead of stack
* change docs to explicitly mention the max buffer size
* changing GetStr() function to behave similarly too
* Update Doc/library/curses.rst
* Update instr with proper return error handling
* Update Modules/_cursesmodule.c
* change to strlen and better memory safety
* change from const int to Py_ssize_t
* add mem allocation guard
* update versionchanged to mention it was an increase.
* explicitly use versionchanged 3.14 as that is its own branch now.
TESTED: `python -m test -u curses test_curses`
---------
Co-authored-by: blurb-it[bot] <43283697+blurb-it[bot]@users.noreply.github.com>
Co-authored-by: Gregory P. Smith <greg@krypto.org>
Co-authored-by: Bénédikt Tran <10796600+picnixz@users.noreply.github.com>
Co-authored-by: Stan Ulbrych <89152624+StanFromIreland@users.noreply.github.com>
Co-authored-by: Tomas R. <tomas.roun8@gmail.com>
Co-authored-by: Łukasz Langa <lukasz@langa.pl>
* Teach pprint about dict views with PrettyPrinter._pprint_dict_view and ._pprint_dict_items_view.
* Use _private names for _dict_*_view attributes of PrettyPrinter.
* Use explicit 'items' keyword when calling _pprint_dict_view from _pprint_dict_items_view.
* 📜🤖 Added by blurb_it.
* Improve tests
* Add tests for collections.abc.[Keys|Items|Mapping|Values]View support in pprint.
* Add support for collections.abc.[Keys|Items|Mapping|Values]View in pprint.
* Split _pprint_dict_view into _pprint_abc_view, so pretty-printing normal dict views and ABC views is handled in two simple methods.
* Simplify redundant code.
* Add collections.abc views to some existing pprint tests.
* Test that views from collection.UserDict are correctly formatted by pprint.
* Handle recursive dict and ABC views.
* Test that subclasses of ABC views work in pprint.
* Test dict views coming from collections.Counter.
* Test ABC views coming from collections.ChainMap.
* Test odict views coming from collections.OrderedDict.
* Rename _pprint_abc_view to _pprint_mapping_abc_view.
* Add pprint test for mapping ABC views where ._mapping has a custom __repr__ and fix ChainMap test.
* When a mapping ABC view has a ._mapping that defines a custom __repr__, dispatch pretty-printing it by that __repr__.
* Add tests for ABC mapping views subclasses that don't replace __repr__, also handling those that delete ._mapping on instances.
* Simplify the pretty printing of ABC mapping views.
* Add a test for depth handling when pretty printing dict views.
* Fix checking whether the view type is a subclass of an items view, add a test.
* Move construction of the views __repr__ set out of _safe_repr.
---------
Co-authored-by: blurb-it[bot] <43283697+blurb-it[bot]@users.noreply.github.com>
Co-authored-by: Éric <merwok@netwok.org>
Co-authored-by: Oleg Iarygin <oleg@arhadthedev.net>
Co-authored-by: Adam Turner <9087854+AA-Turner@users.noreply.github.com>
Co-authored-by: Gregory P. Smith <greg@krypto.org>
* Add "cyclic isolate" to the glossary.
* Add a new "Object Life Cycle" page.
* Improve docs for related API, with special focus on cross-references and warnings
Co-authored-by: Petr Viktorin <encukou@gmail.com>
Co-authored-by: Peter Bierma <zintensitydev@gmail.com>
While some `libcurses` functions are meant to return OK on success,
this is not always the case for all implementations. As such, we relax
the checks on the return values and allow any non-ERR value to be
considered equivalent to OK.
* Ensure that created files and dirs are always removed after test.
Now addCleanup() does not conflict with tearDown().
* Use os_helper.unlink() and os_helper.rmdir().
* Import TESTFN from os_helper.
Explain history of de-facto standard and how to pick between the two Base-85 encoding functions in the base-64 module.
---------
Co-authored-by: C.A.M. Gerlach <CAM.Gerlach@Gerlach.CAM>
The runtime behavior of `http.server` CLI is hard to test on an arbitrary platform.
As such, tests asserting the correctness of `python -m http.server` are temporarily
removed and will be rewritten later once a universal solution has been found.
It was previously possible to specify things like `+00:90:00` which would be equivalent to `+01:30:00`, but is not a valid ISO8601 string.
---------
Co-authored-by: Erlend E. Aasland <erlend.aasland@protonmail.com>
Co-authored-by: Paul Ganssle <1377457+pganssle@users.noreply.github.com>
Incidentally, this also fixed the warning not showing up if a subinterpreter wasn't
cleaned up via _interpreters.destroy. I had to update some of the tests as a result.
The 'help("topics")' output was misaligned due to "ASSIGNMENTEXPRESSIONS"
exceeding the implicit maximum default column width of 19 characters.
Reduced the number of columns from 4 to 3 in the listtopics()
function to allow more space for longer topic names.
Co-authored-by: Łukasz Langa <lukasz@langa.pl>
- Rename error helpers with a `curses_set_error_*` prefix instead of `PyCurses*`.
- Cleanly report both NULL and ERR cases.
- Raise `curses.error` in `is_linetouched` instead of a `TypeError`.
Extension builders must specify Py_GIL_DISABLED if they want to link to the free-threaded builds.
This was usually the case already, but this change guarantees it in all circumstances.
In `faulthandler_sigfpe()`, instead of using 1/0 arithmetic, we explicitly raise SIGFPE.
We also remove `faulthandler._read_null()` since reading from NULL is an undefined
behavior and `faulthandler` should not check for low-level C undefined behaviors.
* gh-134150: Clarify distinction between JSON objects and Python objects in json module docs
* Revert change to JSON introduction
* Clarify occurrences of "object literal" as JSON
Comments immediately preceding the object's source code are used
if the object has no docstring.
Comments that do not describe the object should be separated from
the following source code by an empty line.
Bumps the HACL* revision to include recent revisions that corrects issues
building with legacy/cross-platform macOS SDKs.
Signed-off-by: aeiouaeiouaeiouaeiouaeiouaeiou <aeioudev@outlook.com>
Fix race in `lru_cache` by acquiring critical section on the cache object itself and call the lock held variant of dict functions to modify the underlying dict.
In the abstract interface of `JoinablePath`, replace `__str__()` with
`__vfspath__()`. This frees user implementations of `JoinablePath` to
implement `__str__()` however they like (or not at all.)
Also add `pathlib._os.vfspath()`, which calls `__fspath__()` or
`__vfspath__()`.
If the error handler is used, a new bytes object is created to set as
the object attribute of UnicodeDecodeError, and that bytes object then
replaces the original data. A pointer to the decoded data will became invalid
after destroying that temporary bytes object. So we need other way to return
the first invalid escape from _PyUnicode_DecodeUnicodeEscapeInternal().
_PyBytes_DecodeEscape() does not have such issue, because it does not
use the error handlers registry, but it should be changed for compatibility
with _PyUnicode_DecodeUnicodeEscapeInternal().
* Fix TypeError when formatter_class is a custom subclass of
HelpFormatter.
* Fix TypeError when formatter_class is not a subclass of
HelpFormatter and non-standard prefix_char is used.
* Fix support of colorizing when formatter_class is not a subclass of
HelpFormatter.
* Remove the prefix_chars parameter of HelpFormatter.
This check is potentially problematic because it could force evaluation of
annotations unnecessarily. This doesn't trigger for builtin objects (functions,
classes, or modules) with annotations, but it could trigger for third-party objects.
The check was not particularly useful anyway, because it succeeds if ``__annotations__``
is a dict or None, so the only thing this did was guard against objects that have an
``__annotations__`` attribute that is of some other type. That doesn't seem particularly
useful, so I just removed the check.
The function `dict_set_fromkeys()` adds elements of a set to an existing
dictionary. The size of the expanded dictionary was estimated with
`PySet_GET_SIZE(iterable)`, which did not take into account the size of the
existing dictionary.
This converts functions, code, str, bytes, bytearray, and memoryview objects to PyCodeObject,
and ensure that the object looks like a script. That means no args, no return, and no closure.
_PyCode_GetPureScriptXIData() takes it a step further and ensures there are no globals.
We also add _PyObject_SupportedAsScript() to the internal C-API.
* All parameters of sqlite3.connect() except "database" are now keyword-only.
* The first three parameters of methods create_function() and
create_aggregate() are now positional-only.
* The first parameter of methods set_authorizer(), set_progress_handler()
and set_trace_callback() is now positional-only.
Prevent the possibility of re-entrancy leading to deadlock or infinite recursion (caused by logging triggered by logging), by disabling logging while the logger is handling log messages.
This reverts commit 3c73cf5 (gh-133497), which itself reverted
the original commit d270bb5 (gh-133221).
We reverted the original change due to failing android tests.
The checks in _PyCode_CheckNoInternalState() were too strict,
so we've relaxed them.
Also move up the explanation of insertion order preservation. Both paragraphs seemed out of place down where they were.
---------
Co-authored-by: Terry Jan Reedy <tjreedy@udel.edu>
Followup to 942673ed19 (GH-133588)
* Update configure for Python 3.15
* Update magic number for 3.15
* Remove deprecated 'check_home' argument from sysconfig.is_python_build
* Add warningignore entries for Modules/_sqlite/clinic/connection.c.h
* Work around c-analyzer complaints about _testclinic deprecation tests
---------
Co-authored-by: Hugo van Kemenade <1324225+hugovk@users.noreply.github.com>
