The GC for the free threaded build would get slower with each collection due
to effectively double counting objects freed by the GC.
(cherry picked from commit eb892868b3)
Co-authored-by: Kevin Wang <kevmo314@gmail.com>
If we overflowed the global version counter (i.e., after 2*24 calls to
`_PyMonitoring_SetEvents`), we bailed out after setting global monitoring
events but before instrumenting code objects, which led to assertion errors
later on.
Also add a `time.sleep()` to `test_free_threading.test_monitoring` to avoid
overflowing the global version counter.
(cherry picked from commit e457d60daa)
gh-140530: fix a reference leak in an error path for `raise exc from cause` (#140908)
Fix a reference leak in `raise E from T` when `T` is an exception
subtype for which `T.__new__` does not return an exception instance.
(cherry picked from commit 0c77e7c23b)
gh-140815: Fix faulthandler for invalid/freed frame (#140921)
faulthandler now detects if a frame or a code object is invalid or
freed.
Add helper functions:
* _PyCode_SafeAddr2Line()
* _PyFrame_SafeGetCode()
* _PyFrame_SafeGetLasti()
_PyMem_IsPtrFreed() now detects pointers in [-0xff, 0xff] range
as freed.
(cherry picked from commit a84181c31b)
Revert GH-131993.
Fix swallowing some syntax warnings in different modules if they accidentally
have the same message and are emitted from the same line.
Fix duplicated warnings in the "finally" block.
(cherry picked from commit 279db6bede)
There were a few thread-safety issues when profiling or tracing all
threads via PyEval_SetProfileAllThreads or PyEval_SetTraceAllThreads:
* The loop over thread states could crash if a thread exits concurrently
(in both the free threading and default build)
* The modification of `c_profilefunc` and `c_tracefunc` wasn't
thread-safe on the free threading build.
(cherry picked from commit a10152f8fd)
* [3.13] gh-139748: fix leaks in AC error paths when using unicode FS-based converters (GH-139765)
(cherry picked from commit b04a57deef)
Co-authored-by: Bénédikt Tran <10796600+picnixz@users.noreply.github.com>
fix comment reference from man 7 signal to man 7 signal-safety (GH-138554)
(cherry picked from commit 2f5ace780b)
docs: fix comment reference from man 7 signal to man 7 signal-safety
Co-authored-by: mqudah <mohghq@gmail.com>
* [3.13] gh-87135: Hang non-main threads that attempt to acquire the GIL during finalization (GH-105805)
Instead of surprise crashes and memory corruption, we now hang threads that attempt to re-enter the Python interpreter after Python runtime finalization has started. These are typically daemon threads (our long standing mis-feature) but could also be threads spawned by extension modules that then try to call into Python. This marks the `PyThread_exit_thread` public C API as deprecated as there is no plausible safe way to accomplish that on any supported platform in the face of things like C++ code with finalizers anywhere on a thread's stack. Doing this was the least bad option.
(cherry picked from commit 8cc5aa47ee)
Co-authored-by: Jeremy Maitin-Shepard <jeremy@jeremyms.com>
Co-authored-by: Gregory P. Smith <greg@krypto.org>
* state "3.13.7 and earlier"
* backport: do not add the deprecated marker
* fix Py_IsFinalizing doc ref
---------
Co-authored-by: Jeremy Maitin-Shepard <jeremy@jeremyms.com>
The `PyEval_SetProfileAllThreads` function and other related functions
had a race condition on `tstate->c_profilefunc` that could lead to a
crash when disable profiling or tracing on all threads while another
thread is starting to profile or trace a a call.
There are still potential crashes when threads exit concurrently with
profiling or tracing be enabled/disabled across all threads.
(cherry picked from commit 362692852f)
gh-137440: Update comment in Python/hamt.c on importing for testing (GH-137441)
Switch from `_testcapi` to `_testinternalcapi`.
(cherry picked from commit 7ab68cd506)
Co-authored-by: Bartosz Sławecki <bartosz@ilikepython.com>
gh-58124: Avoid CP_UTF8 in UnicodeDecodeError (GH-137415)
Fix name of the Python encoding in Unicode errors of the code page
codec: use "cp65000" and "cp65001" instead of "CP_UTF7" and "CP_UTF8"
which are not valid Python code names.
(cherry picked from commit ce1b747ff6)
Co-authored-by: Victor Stinner <vstinner@python.org>
The free threading build uses QSBR to delay the freeing of dictionary
keys and list arrays when the objects are accessed by multiple threads
in order to allow concurrent reads to proceed with holding the object
lock. The requests are processed in batches to reduce execution
overhead, but for large memory blocks this can lead to excess memory
usage.
Take into account the size of the memory block when deciding when to
process QSBR requests.
Also track the amount of memory being held by QSBR for mimalloc pages.
Advance the write sequence if this memory exceeds a limit. Advancing
the sequence will allow it to be freed more quickly.
Process the held QSBR items from the "eval breaker", rather than from
`_PyMem_FreeDelayed()`. This gives a higher chance that the global read
sequence has advanced enough so that items can be freed.
(cherry picked from commit 113de8545f)
Co-authored-by: Neil Schemenauer <nas-github@arctrix.com>
Co-authored-by: Sam Gross <colesbury@gmail.com>
gh-134411: assert `PyLong_FromLong(x) != NULL` when `x` is known to be small (GH-134415)
Since `PyLong_From Long(PY_MONITORING_DEBUGGER_ID)` falls to `small_int` case and can't return `NULL`. Added `assert`s for extra confidence.
https://github.com/python/cpython/issues/134411#issuecomment-2897653868
(cherry picked from commit cf19b6435d)
Co-authored-by: Sergey Muraviov <smurav@mail.ru>
This commit fixes the following problems:
* The x86_64 trampolines are not preserving frame pointers
* The hardcoded offsets to the code segment from the FDE only worked properly for x64_64
* The CIE data was not following conventions of aarch64
* The eh_frame for aarch64 was not fully correct
(cherry picked from commit 236f733d8f)
The free threading build could spin unnecessarily on `_Py_yield()` if the initial
compare and swap failed.
(cherry picked from commit cbfaf41caf)
Co-authored-by: Joseph Tibbertsma <josephtibbertsma@gmail.com>
Apply Intel Control-flow Technology for x86-64 on asm_trampoline.S.
Required for mitigation against return-oriented programming (ROP)
and Call or Jump Oriented Programming (COP/JOP) attacks.
Manual application is required for the assembly files.
See also: https://sourceware.org/annobin/annobin.html/Test-cf-protection.html
[3.14] gh-128605: Revert "Add branch protections for x86_64 in asm_tr…ampoline.S (GH-128606) (GH-135077)" (GH-135175)
This reverts commit 899cca6dbf,
which broke buildbots.
(cherry picked from commit b477e21d5c)
Co-authored-by: Petr Viktorin <encukou@gmail.com>
[3.14] gh-128605: Add branch protections for x86_64 in asm_trampoline.S (GH-128606) (GH-135077)
Apply Intel Control-flow Technology for x86-64 on asm_trampoline.S.
Required for mitigation against return-oriented programming (ROP)
and Call or Jump Oriented Programming (COP/JOP) attacks.
Manual application is required for the assembly files.
See also: https://sourceware.org/annobin/annobin.html/Test-cf-protection.html
(cherry picked from commit 899cca6dbf)
Co-authored-by: stratakis <cstratak@redhat.com>
gh-130070: Fix `exec(<string>, closure=<non-None>)` unexpected path (#130071)
Fixed an assertion error (so, it could be reproduced only in builds with assertions enabled)
for `exec` when the `source` argument is a string and the `closure` argument is not `None`.
Co-authored-by: sobolevn <mail@sobolevn.me>
(cherry picked from commit 954b2cf031)
(cherry picked from commit ac3c439cdf)
Co-authored-by: Peter Bierma <zintensitydev@gmail.com>
Co-authored-by: sobolevn <mail@sobolevn.me>
Co-authored-by: Victor Stinner <vstinner@python.org>
Co-authored-by: Mark Shannon <mark@hotpy.org>
gh-131988: Fix a multithreaded scaling regression
The 3.13 free threaded build immortalizes certain objects to avoid
reference count contention. In gh-127114 the condition was
unintentionally changed to happen when the first thread was created
instead of the first non-main thread. The `interp->gc.immortalize` field
is then cleared again during `_PyGC_Init()`.
Change the condition so that we check if we should immortalize objects
using deferred reference counting whenever a non-main thread is created.
CPython's pthread-based thread identifier relies on pthread_t being able
to be represented as an unsigned integer type.
This is true in most Linux libc implementations where it's defined as an
unsigned long, however musl typedefs it as a struct *.
If the pointer has the high bit set and is cast to PyThread_ident_t, the
resultant value can be sign-extended [0]. This can cause issues when
comparing against threading._MainThread's identifier. The main thread's
identifier value is retrieved via _get_main_thread_ident which is backed
by an unsigned long which truncates sign extended bits.
>>> hex(threading.main_thread().ident)
'0xb6f33f3c'
>>> hex(threading.current_thread().ident)
'0xffffffffb6f33f3c'
Work around this by conditionally compiling in some code for non-glibc
based Linux platforms that are at risk of sign-extension to return a
PyLong based on the main thread's unsigned long thread identifier if the
current thread is the main thread.
[0]: https://gcc.gnu.org/onlinedocs/gcc-14.2.0/gcc/Arrays-and-pointers-implementation.html
---------
(cherry picked from commit 72123063dd)
Signed-off-by: Vincent Fazio <vfazio@gmail.com>
Co-authored-by: Vincent Fazio <vfazio@gmail.com>
