gh-149816: fix thread safety of deletion of list slice (GH-149936)
(cherry picked from commit 00ea77613b)
Co-authored-by: Kumar Aditya <kumaraditya@python.org>
gh-149816: Fix a race condition in `_PyBytes_FromList` with free-threading (GH-149909)
(cherry picked from commit 46afba7b93)
Co-authored-by: sobolevn <mail@sobolevn.me>
In free-threaded builds, concurrent calls to PyDict_AddWatcher, PyDict_ClearWatcher, PyDict_Watch, and PyDict_Unwatch can race on the shared callback array and the per-dict watcher tags. This change adds a mutex to serialize watcher registration and removal, atomic operations for tag updates, and atomic acquire/release synchronization for callback dispatch in _PyDict_SendEvent.
(cherry picked from commit 8a4895985f)
Co-authored-by: Alper <alperyoney@fb.com>
* SEND specialization. Adds 2 new specialized instructions:
* SEND_VIRTUAL: for sends to virtual iterators e.g lists and tuples
* SEND_ASYNC_GEN: for sends to async generators
Tweak FOR_ITER_VIRTUAL so that SEND_VIRTUAL and FOR_ITER_VIRTUAL use equivalent guards
* Add Zd/Zf format support to array, memoryview and struct.
* ctypes: Replace F/D/G complex format with Zf/Zd/Zg.
* Modify array, ctypes and struct modules to support format strings
longer than 1 character (such as "Zd").
* Change array.typecodes type from str to tuple.
When someone adds a new type but doesn't increment
`_Py_MAX_MANAGED_STATIC_BUILTIN_TYPES` or
`_Py_MAX_MANAGED_STATIC_EXT_TYPES`, JIT tests fail,
because JIT builds define an extra type.
But the JIT tests don't necessarily run for the commit
that causes the failure.
As a workaround, use the same size for the array for all
builds, potentially with an empty spot.
Avoid racing with the owning thread's refcount operations when
immortalizing an interned string: if we don't own it and its refcount
isn't merged, intern a copy we own instead. Use atomic stores in
_Py_SetImmortalUntracked so concurrent atomic reads are race-free.
* Make the `PY_UNWIND` monitoring event available as a code-local
event to allow trapping on function exit events when an exception
bubbles up. This complements the PY_RETURN event by allowing to
catch any function exit event.
* Allow `PY_UNWIND` to be `DISABLE`d; disabling it disables the event for the whole code object.
* Do the above for `PY_THROW`, `RAISE`, `EXCEPTION_HANDLED`, and `RERAISE` events.
* Add FOR_ITER_VIRTUAL to specialize FOR_ITER for virtual iterators
* Add GET_ITER_SELF to specialize GET_ITER for iterators (including generators)
* Add GET_ITER_VIRTUAL to specialize GET_ITER for iterables as virtual iterators
* Add new (internal) _tp_iteritem function slot to PyTypeObject
* Put limited RESUME at start of genexpr for free-threading. Fix up exception handling in genexpr
There are newly documented restrictions on tp_traverse:
The traversal function must not have any side effects.
It must not modify the reference counts of any Python
objects nor create or destroy any Python objects.
* Add several functions that are guaranteed side-effect-free,
with a _DuringGC suffix.
* Use these in ctypes
* Consolidate tp_traverse docs in gcsupport.rst, moving unique
content from typeobj.rst there
Co-authored-by: Lysandros Nikolaou <lisandrosnik@gmail.com>
Co-authored-by: Victor Stinner <vstinner@python.org>
Fix huge page leak in datastack chunk allocator
The original fix rounded datastack chunk allocations in pystate.c so that
_PyObject_VirtualFree() would receive the full huge page mapping size.
Change direction and move that logic into _PyObject_VirtualAlloc() and
_PyObject_VirtualFree() instead. The key invariant is that munmap() must see
the full mapped size, so alloc and free now apply the same platform-specific
rounding in the allocator layer.
This keeps _PyStackChunk bookkeeping in requested-size units, avoids a
hardcoded 2 MB assumption, and also covers other small virtual-memory users
such as the JIT tracer state allocation in optimizer.c.
When Python is built in debug mode:
* long_alloc() now initializes digits with a pattern to detect usage of
uninitialized digits.
* _PyLong_CompactValue() now makes sure that the digit is zero when the
sign is zero.
* PyLongWriter_Finish() now raises SystemError if it detects uninitialized
digits
Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>
