* [3.13] gh-87135: Hang non-main threads that attempt to acquire the GIL during finalization (GH-105805)
Instead of surprise crashes and memory corruption, we now hang threads that attempt to re-enter the Python interpreter after Python runtime finalization has started. These are typically daemon threads (our long standing mis-feature) but could also be threads spawned by extension modules that then try to call into Python. This marks the `PyThread_exit_thread` public C API as deprecated as there is no plausible safe way to accomplish that on any supported platform in the face of things like C++ code with finalizers anywhere on a thread's stack. Doing this was the least bad option.
(cherry picked from commit 8cc5aa47ee)
Co-authored-by: Jeremy Maitin-Shepard <jeremy@jeremyms.com>
Co-authored-by: Gregory P. Smith <greg@krypto.org>
* state "3.13.7 and earlier"
* backport: do not add the deprecated marker
* fix Py_IsFinalizing doc ref
---------
Co-authored-by: Jeremy Maitin-Shepard <jeremy@jeremyms.com>
(cherry picked from commit 698bab5a40)
Co-authored-by: Weilin Du <108666168+LamentXU123@users.noreply.github.com>
Co-authored-by: Terry Jan Reedy <tjreedy@udel.edu>
The free threading build uses QSBR to delay the freeing of dictionary
keys and list arrays when the objects are accessed by multiple threads
in order to allow concurrent reads to proceed with holding the object
lock. The requests are processed in batches to reduce execution
overhead, but for large memory blocks this can lead to excess memory
usage.
Take into account the size of the memory block when deciding when to
process QSBR requests.
Also track the amount of memory being held by QSBR for mimalloc pages.
Advance the write sequence if this memory exceeds a limit. Advancing
the sequence will allow it to be freed more quickly.
Process the held QSBR items from the "eval breaker", rather than from
`_PyMem_FreeDelayed()`. This gives a higher chance that the global read
sequence has advanced enough so that items can be freed.
(cherry picked from commit 113de8545f)
Co-authored-by: Neil Schemenauer <nas-github@arctrix.com>
Co-authored-by: Sam Gross <colesbury@gmail.com>
gh-135607: remove null checking of weakref list in dealloc of extension modules and objects (#135614)
(cherry picked from commit b1056c2a44)
Co-authored-by: Xuanteng Huang <44627253+xuantengh@users.noreply.github.com>
Co-authored-by: Kumar Aditya <kumaraditya@python.org>
gh-134989: Fix Py_RETURN_NONE in the limited C API (GH-135165)
Fix Py_RETURN_NONE, Py_RETURN_TRUE and Py_RETURN_FALSE macros in the
limited C API 3.11 and older:
Don't treat Py_None, Py_True and Py_False as immortal.
(cherry picked from commit 9258f3da91)
gh-134875: Fix mimallc build error for the old compilers (gh-134994)
(cherry picked from commit b525e31b7f)
Co-authored-by: Donghee Na <donghee.na@python.org>
gh-129748: Update mimalloc to use atomic store for mi_block_set_nextx (GH-134238)
(cherry picked from commit 317c496223)
Co-authored-by: Donghee Na <donghee.na@python.org>
If the error handler is used, a new bytes object is created to set as
the object attribute of UnicodeDecodeError, and that bytes object then
replaces the original data. A pointer to the decoded data will became invalid
after destroying that temporary bytes object. So we need other way to return
the first invalid escape from _PyUnicode_DecodeUnicodeEscapeInternal().
_PyBytes_DecodeEscape() does not have such issue, because it does not
use the error handlers registry, but it should be changed for compatibility
with _PyUnicode_DecodeUnicodeEscapeInternal().
(cherry picked from commit 9f69a58623)
Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>
The X/Open curses specification[0] and ncurses documentation[1]
both state that subwindows must be deleted before the main window.
Deleting the windows in the wrong order causes a double-free with
NetBSD's curses implementation.
To fix this, keep track of the original window object in the subwindow
object, and keep a reference to the original for the lifetime of
the subwindow.
[0] https://pubs.opengroup.org/onlinepubs/7908799/xcurses/delwin.html
[1] https://invisible-island.net/ncurses/man/curs_window.3x.html
(cherry picked from commit 0af61fe2f4)
Co-authored-by: Michael Forney <mforney@mforney.org>
Use the standard `__ARM_ARCH` macro, which is supported by GCC and Clang.
The branching logic for of `__ARMEL__` has been removed so if the target
architecture supports v7+ instructions, a yield is emitted, otherwise a nop
is emitted. This covers both big and little endian scenarios.
(cherry picked from commit 03f6c8e239)
Signed-off-by: Vincent Fazio <vfazio@gmail.com>
Co-authored-by: Vincent Fazio <vfazio@gmail.com>
Removes ``PyConfig.use_system_logger``, resolving an ABI incompatibility introduced in
3.13.2.
Changes the default behavior of iOS to *always* direct stdout/stderr to the system log.
gh-129296: Fix `pyatomic.h` include paths (GH-129320)
Use relative includes in Include/cpython/pyatomic.h for
pyatomic_gcc.h, pyatomic_std.h and pyatomic_msc.h.
Do a similar change in Include/cpython/pythread.h for
pthread_stubs.h include.
(cherry picked from commit 3a974e39d5)
Co-authored-by: Zanie Blue <contact@zanie.dev>
The use of PySys_GetObject() and _PySys_GetAttr(), which return a borrowed
reference, has been replaced by using one of the following functions, which
return a strong reference and distinguish a missing attribute from an error:
_PySys_GetOptionalAttr(), _PySys_GetOptionalAttrString(),
_PySys_GetRequiredAttr(), and _PySys_GetRequiredAttrString().
(cherry picked from commit 0ef4ffeefd)
We should use a relaxed atomic load in the free threading build in
`PyType_Modified()` because that's called without the type lock held.
It's not necessary to use atomics in `type_modified_unlocked()`.
We should also use `FT_ATOMIC_STORE_UINT_RELAXED()` instead of the
`UINT32` variant because `tp_version_tag` is declared as `unsigned int`.
(cherry picked from commit 57f45ee2d8)
The MemoryError freelist was not thread-safe in the free threaded build.
Use a mutex to protect accesses to the freelist. Unlike other freelists,
the MemoryError freelist is not performance sensitive.
(cherry picked from commit 51b4edb1a4)
tracemalloc_alloc(), tracemalloc_realloc(), PyTraceMalloc_Track(),
PyTraceMalloc_Untrack() and _PyTraceMalloc_TraceRef() now check
tracemalloc_config.tracing after calling TABLES_LOCK().
_PyTraceMalloc_Stop() now protects more code with TABLES_LOCK(),
especially setting tracemalloc_config.tracing to 1.
Add a test using PyTraceMalloc_Track() to test tracemalloc.stop()
race condition.
Call _PyTraceMalloc_Init() at Python startup.
Previously, `_Py_RefcntAdd` hasn't called
`_Py_INCREF_STAT_INC/_Py_INCREF_IMMORTAL_STAT_INC` which is incorrect.
Now it has been fixed.
(cherry picked from commit ab05beb8ce)
- Add a helper to set an error from locale-encoded `char*`
- Use the helper for gdbm & dlerror messages
(cherry picked from commit 7303f06846)
Co-authored-by: Bénédikt Tran <10796600+picnixz@users.noreply.github.com>
Co-authored-by: Victor Stinner <vstinner@python.org>
gh-127906: Fix Py_BUILD_ASSERT_EXPR() on Windows
Change Py_BUILD_ASSERT_EXPR implementation on Windows to avoid a
compiler warning about an unnamed structure.
Add `extern "C"` around `PyTraceMalloc_` functions. (GH-127772)
Pretty much everything else exported by Python.h has an extern "C"
annotation, yet this header appears to be missing one.
(cherry picked from commit 2cdeb61b57)
Co-authored-by: Peter Hawkins <hawkinsp@cs.stanford.edu>
Adds a `use_system_log` config item to enable stdout/stderr redirection for
Apple platforms. This log streaming is then used by a new iOS test runner
script, allowing the display of test suite output at runtime. The iOS test
runner script can be used by any Python project, not just the CPython test
suite.
(cherry picked from commit 2041a95e68)
Objects may be temporarily "resurrected" in destructors when calling
finalizers or watcher callbacks. We previously undid the resurrection
by decrementing the reference count using `Py_SET_REFCNT`. This was not
thread-safe because other threads might be accessing the object
(modifying its reference count) if it was exposed by the finalizer,
watcher callback, or temporarily accessed by a racy dictionary or list
access.
This adds internal-only thread-safe functions for temporary object
resurrection during destructors.
(cherry picked from commit f4f530804b)
