gh-151613: Fix remote debugging frame cache ABA (#151614)
The remote debugging frame cache previously used only the last_profiled_frame address as its cache anchor. If a frame returned and a later frame reused the same _PyInterpreterFrame address, the profiler could accept a stale cache entry and splice parent frames from a different call chain into the current stack.
This adds a last_profiled_frame_seq counter next to last_profiled_frame, increments it when the anchor advances, stores it in frame cache entries, and validates cache hits against both the frame address and the sequence. Cache miss walks now copy stack chunks before storing new cache entries so stored continuations come from a stable snapshot. The new regression test exercises alternating call chains and checks that cached stacks never contain frames from both branches.
(cherry picked from commit 8cda6ae2f1)
This mimics an optimization already present for the single-mutex critical section.
(cherry picked from commit c2ca7724af)
Co-authored-by: Daniele Parmeggiani <8658291+dpdani@users.noreply.github.com>
In free-threaded builds, concurrent calls to PyDict_AddWatcher, PyDict_ClearWatcher, PyDict_Watch, and PyDict_Unwatch can race on the shared callback array and the per-dict watcher tags. This change adds a mutex to serialize watcher registration and removal, atomic operations for tag updates, and atomic acquire/release synchronization for callback dispatch in _PyDict_SendEvent.
(cherry picked from commit 8a4895985f)
Co-authored-by: Alper <alperyoney@fb.com>
* SEND specialization. Adds 2 new specialized instructions:
* SEND_VIRTUAL: for sends to virtual iterators e.g lists and tuples
* SEND_ASYNC_GEN: for sends to async generators
Tweak FOR_ITER_VIRTUAL so that SEND_VIRTUAL and FOR_ITER_VIRTUAL use equivalent guards
When someone adds a new type but doesn't increment
`_Py_MAX_MANAGED_STATIC_BUILTIN_TYPES` or
`_Py_MAX_MANAGED_STATIC_EXT_TYPES`, JIT tests fail,
because JIT builds define an extra type.
But the JIT tests don't necessarily run for the commit
that causes the failure.
As a workaround, use the same size for the array for all
builds, potentially with an empty spot.
The replaces the incremental GC with a forward port (from 3.13) of the generational GC.
Co-Authored-By: Neil Schemenauer <nas@arctrix.com>
Co-Authored-By: Zanie Blue <contact@zanie.dev>
Co-Authored-By: Sergey Miryanov <sergey.miryanov@gmail.com>
Co-authored-by: Hugo van Kemenade <1324225+hugovk@users.noreply.github.com>
Add a keyword-only `max_threads` argument to `dump_traceback()` and
`dump_traceback_later()`, defaulting to 100 to preserve existing
behavior. Allows server processes with many worker threads to dump
beyond the historical 100-thread cap (previously a hardcoded
`MAX_NTHREADS = 100` in `Python/traceback.c`).
The cap matters in practice: tstates are prepended to the
PyInterpreterState linked list, so the dump walks newest-first. With
more than 100 threads alive, the main thread (oldest, at the tail) is
silently elided from watchdog dumps -- exactly the thread that's
usually wanted.
The hardcoded value is moved to a new internal macro
`_Py_TRACEBACK_MAX_NTHREADS` in `pycore_traceback.h` so the in-tree
fatal-signal callers all reference one source of truth.
* Records the same objects for each member of family before execution
* Records derived values when recording the trace
* This makes sure that specialization, or deoptimization, does not cause invalid values to be recorded
Add `canonical=False` keyword argument to `a2b_base64`, `a2b_base32`, `a2b_base85`, and `a2b_ascii85` (and their `base64` module wrappers). When `canonical=True`, non-canonical encodings are rejected per [RFC 4648 section 3.5](https://datatracker.ietf.org/doc/html/rfc4648.html#section-3.5).
This is independent of `strict_mode`.
For base85/ascii85, the check also rejects single-character final groups (never produced by a conforming encoder) and verifies partial group padding matches what the encoder would produce.
Co-authored-by: Serhiy Storchaka via lots of great code review!
* Replaces ad-hoc logic for ending traces with a simple inequality: `fitness < exit_quality`
* Fitness starts high and is reduced for branches, backward edges, calls and trace length
* Exit quality reflect how good a spot that instruction is to end a trace. Closing a loop is very, specializable instructions are very low and the others in between.
* Make the `PY_UNWIND` monitoring event available as a code-local
event to allow trapping on function exit events when an exception
bubbles up. This complements the PY_RETURN event by allowing to
catch any function exit event.
* Allow `PY_UNWIND` to be `DISABLE`d; disabling it disables the event for the whole code object.
* Do the above for `PY_THROW`, `RAISE`, `EXCEPTION_HANDLED`, and `RERAISE` events.
If _BlocksOutputBuffer_Finish() fails (memory allocation failure),
PyBytesWriter_Discard() is called on the writer. Then if
_BlocksOutputBuffer_OnError() is called, it calls again
PyBytesWriter_Discard() causing a double free.
Fix _BlocksOutputBuffer_Finish() by setting buffer->writer to NULL,
so _BlocksOutputBuffer_OnError() does nothing instead of calling
PyBytesWriter_Discard() again.
* Add FOR_ITER_VIRTUAL to specialize FOR_ITER for virtual iterators
* Add GET_ITER_SELF to specialize GET_ITER for iterators (including generators)
* Add GET_ITER_VIRTUAL to specialize GET_ITER for iterables as virtual iterators
* Add new (internal) _tp_iteritem function slot to PyTypeObject
* Put limited RESUME at start of genexpr for free-threading. Fix up exception handling in genexpr
