The PyThreadState field gains a reference count field to avoid
issues with PyThreadState being a dangling pointer to freed memory.
The refcount starts with a value of two: one reference is owned by the
interpreter's linked list of thread states and one reference is owned by
the OS thread. The reference count is decremented when the thread state
is removed from the interpreter's linked list and before the OS thread
calls `PyThread_hang_thread()`. The thread that decrements it to zero
frees the `PyThreadState` memory.
The `holds_gil` field is moved out of the `_status` bit field, to avoid
a data race where on thread calls `PyThreadState_Clear()`, modifying the
`_status` bit field while the OS thread reads `holds_gil` when
attempting to acquire the GIL.
The `PyThreadState.state` field now has `_Py_THREAD_SHUTTING_DOWN` as a
possible value. This corresponds to the `_PyThreadState_MustExit()`
check. This avoids race conditions in the free threading build when
checking `_PyThreadState_MustExit()`.
Move deprecated PyUnicode API docs to new section
Move Py_UNICODE to a new "Deprecated API" section.
Formally soft-deprecate PyUnicode_READY, and move it
Document and soft-deprecate PyUnicode_IS_READY, and move it
Document PyUnicode_IS_ASCII, PyUnicode_CHECK_INTERNED
PyUnicode_New docs: Clarify requirements for "fresh" strings
PyUnicodeWriter_DecodeUTF8Stateful: Link "error-handlers"
Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>
This fixes a fairly subtle bug involving finalizers and resurrection in
debug free threaded builds: if `_PyObject_ResurrectEnd` returns `1`
(i.e., the object was resurrected by a finalizer), it's not safe to
access the object because it might still be deallocated. For example:
* The finalizer may have exposed the object to another thread. That
thread may hold the last reference and concurrently deallocate it any
time after `_PyObject_ResurrectEnd()` returns `1`.
* `_PyObject_ResurrectEnd()` may call `_Py_brc_queue_object()`, which
may internally deallocate the object immediately if the owning thread
is dead.
Therefore, it's important not to access the object after it's
resurrected. We only violate this in two cases, and only in debug
builds:
* We assert that the object is tracked appropriately. This is now moved
up betewen the finalizer and the `_PyObject_ResurrectEnd()` call.
* The `--with-trace-refs` builds may need to remember the object if
it's resurrected. This is now handled by `_PyObject_ResurrectStart()`
and `_PyObject_ResurrectEnd()`.
Note that `--with-trace-refs` is currently disabled in `--disable-gil`
builds because the refchain hash table isn't thread-safe, but this
refactoring avoids an additional thread-safety issue.
* Implement C recursion protection with limit pointers for Linux, MacOS and Windows
* Remove calls to PyOS_CheckStack
* Add stack protection to parser
* Make tests more robust to low stacks
* Improve error messages for stack overflow
Revert "GH-91079: Implement C stack limits using addresses, not counters. (GH-130007)" for now
Unfortunatlely, the change broke some buildbots.
This reverts commit 2498c22fa0.
Deprecate private C API functions:
* _PyUnicodeWriter_Init()
* _PyUnicodeWriter_Finish()
* _PyUnicodeWriter_Dealloc()
* _PyUnicodeWriter_WriteChar()
* _PyUnicodeWriter_WriteStr()
* _PyUnicodeWriter_WriteSubstring()
* _PyUnicodeWriter_WriteASCIIString()
* _PyUnicodeWriter_WriteLatin1String()
These functions are not deprecated in the internal C API (if the
Py_BUILD_CORE macro is defined).
* Implement C recursion protection with limit pointers
* Remove calls to PyOS_CheckStack
* Add stack protection to parser
* Make tests more robust to low stacks
* Improve error messages for stack overflow
Implement PyUnicode_KIND() and PyUnicode_DATA() as function, in
addition to the macros with the same names. The macros rely on C bit
fields which have compiler-specific layout.
Use relative includes in Include/cpython/pyatomic.h for
pyatomic_gcc.h, pyatomic_std.h and pyatomic_msc.h.
Do a similar change in Include/cpython/pythread.h for
pthread_stubs.h include.
This exposes `_Py_TryIncref` as `PyUnstable_TryIncref()` and the helper
function `_PyObject_SetMaybeWeakref` as `PyUnstable_EnableTryIncRef`.
These are helpers for dealing with unowned references in a safe way,
particularly in the free threading build.
Co-authored-by: Petr Viktorin <encukou@gmail.com>
- Unify `get_unicode` and `get_string` in a single function.
- Allow to retrieve the underlying `object` attribute, its
size, and the adjusted 'start' and 'end', all at once.
Add a new `_PyUnicodeError_GetParams` internal function for this.
(In `exceptions.c`, it's somewhat common to not need all the attributes,
but the compiler has opportunity to inline the function and optimize
unneeded work away. Outside that file, we'll usually need all or
most of them at once.)
- Use a common implementation for the following functions:
- `PyUnicode{Decode,Encode}Error_GetEncoding`
- `PyUnicode{Decode,Encode,Translate}Error_GetObject`
- `PyUnicode{Decode,Encode,Translate}Error_{Get,Set}Reason`
- `PyUnicode{Decode,Encode,Translate}Error_{Get,Set}{Start,End}`
Methods (functions defined in class scope) are likely to be cleaned
up by the GC anyway.
Add a new code flag, `CO_METHOD`, that is set for functions defined
in a class scope. Use that when deciding to defer functions.
The `PyWeakref_IsDead()` function tests if a weak reference is dead
without any side effects. Although you can also detect if a weak
reference is dead using `PyWeakref_GetRef()`, that function returns a
strong reference that must be `Py_DECREF()`'d, which can introduce side
effects if the last reference is concurrently dropped (at least in the
free threading build).
Adds a `use_system_log` config item to enable stdout/stderr redirection for
Apple platforms. This log streaming is then used by a new iOS test runner
script, allowing the display of test suite output at runtime. The iOS test
runner script can be used by any Python project, not just the CPython test
suite.
* Replace uses of `PyCell_GET` and `PyCell_SET`. These macros are not
safe to use in the free-threaded build. Use `PyCell_GetRef()` and
`PyCell_SetTakeRef()` instead.
* Since `PyCell_GetRef()` returns a strong rather than borrowed ref, some
code restructuring was required, e.g. `frame_get_var()` returns a strong
ref now.
* Add critical sections to `PyCell_GET` and `PyCell_SET`.
* Move critical_section.h earlier in the Python.h file.
* Add `PyCell_GET` to the free-threading howto table of APIs that return
borrowed refs.
* Add additional unit tests for free-threading.
* Mark almost all reachable objects before doing collection phase
* Add stats for objects marked
* Visit new frames before each increment
* Update docs
* Clearer calculation of work to do.
"Generally, mixed-mode arithmetic combining real and complex variables should
be performed directly, not by first coercing the real to complex, lest the sign
of zero be rendered uninformative; the same goes for combinations of pure
imaginary quantities with complex variables." (c) Kahan, W: Branch cuts for
complex elementary functions.
This patch implements mixed-mode arithmetic rules, combining real and
complex variables as specified by C standards since C99 (in particular,
there is no special version for the true division with real lhs
operand). Most C compilers implementing C99+ Annex G have only these
special rules (without support for imaginary type, which is going to be
deprecated in C2y).
* Mark almost all reachable objects before doing collection phase
* Add stats for objects marked
* Visit new frames before each increment
* Remove lazy dict tracking
* Update docs
* Clearer calculation of work to do.
