gh-140815: Fix faulthandler for invalid/freed frame (#140921)
faulthandler now detects if a frame or a code object is invalid or
freed.
Add helper functions:
* _PyCode_SafeAddr2Line()
* _PyFrame_SafeGetCode()
* _PyFrame_SafeGetLasti()
_PyMem_IsPtrFreed() now detects pointers in [-0xff, 0xff] range
as freed.
(cherry picked from commit a84181c31b)
Currently, there are a few places where tp_mro could theoretically
become NULL, but do not in practice. This commit adds defensive checks for
NULL values to ensure that any changes do not introduce a crash and that
state invariants are upheld.
The assertions added in this commit are all instances where a NULL value would get passed to something not expecting a NULL, so it is better to catch an assertion failure than crash later on.
There are a few cases where it is OK for the return of lookup_tp_mro to be NULL, such as when passed to is_subtype_with_mro, which handles this explicitly.
(cherry picked from commit a8edca62fc)
Co-authored-by: Emma Smith <emma@emmatyping.dev>
GH-140058: Clear key and value if `PyTuple_New` fails in `dictiter_iternextitem` (GH-140059)
(cherry picked from commit ded59f7e8e)
Co-authored-by: Sergey Miryanov <sergey.miryanov@gmail.com>
gh-71810: Fix _PyLong_AsByteArray() undefined behavior (GH-138873)
Don't read p[-1] when p is an empty string: when n==0.
(cherry picked from commit 8b5ce31c2b)
Co-authored-by: Victor Stinner <vstinner@python.org>
* gh-138479: Ensure that `__typing_subst__` returns a tuple (GH-138482)
Raise an exception if __typing_subst__ returns a non-tuple object.
(cherry picked from commit 1da989be74)
Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>
gh-58124: Avoid CP_UTF8 in UnicodeDecodeError (GH-137415)
Fix name of the Python encoding in Unicode errors of the code page
codec: use "cp65000" and "cp65001" instead of "CP_UTF7" and "CP_UTF8"
which are not valid Python code names.
(cherry picked from commit ce1b747ff6)
Co-authored-by: Victor Stinner <vstinner@python.org>
The free threading build uses QSBR to delay the freeing of dictionary
keys and list arrays when the objects are accessed by multiple threads
in order to allow concurrent reads to proceed with holding the object
lock. The requests are processed in batches to reduce execution
overhead, but for large memory blocks this can lead to excess memory
usage.
Take into account the size of the memory block when deciding when to
process QSBR requests.
Also track the amount of memory being held by QSBR for mimalloc pages.
Advance the write sequence if this memory exceeds a limit. Advancing
the sequence will allow it to be freed more quickly.
Process the held QSBR items from the "eval breaker", rather than from
`_PyMem_FreeDelayed()`. This gives a higher chance that the global read
sequence has advanced enough so that items can be freed.
(cherry picked from commit 113de8545f)
Co-authored-by: Neil Schemenauer <nas-github@arctrix.com>
Co-authored-by: Sam Gross <colesbury@gmail.com>
gh-135607: remove null checking of weakref list in dealloc of extension modules and objects (#135614)
(cherry picked from commit b1056c2a44)
Co-authored-by: Xuanteng Huang <44627253+xuantengh@users.noreply.github.com>
Co-authored-by: Kumar Aditya <kumaraditya@python.org>
Previous error message suggested to use cls.__new__(), which
obviously does not work. Now the error message is the same as for
cls(...).
(cherry picked from commit c45f4f3ebe)
Use `ma_used` instead of `ma_keys->dk_nentries` for modification check
so that we only check if the dictionary is modified, not if new keys are
added to a different dictionary that shared the same keys object.
(cherry picked from commit d8994b0a77)
random.getrandbits() can now generate more that 2**31 bits.
random.randbytes() can now generate more that 256 MiB.
(cherry picked from commit 68784fed78)
If the error handler is used, a new bytes object is created to set as
the object attribute of UnicodeDecodeError, and that bytes object then
replaces the original data. A pointer to the decoded data will became invalid
after destroying that temporary bytes object. So we need other way to return
the first invalid escape from _PyUnicode_DecodeUnicodeEscapeInternal().
_PyBytes_DecodeEscape() does not have such issue, because it does not
use the error handlers registry, but it should be changed for compatibility
with _PyUnicode_DecodeUnicodeEscapeInternal().
(cherry picked from commit 9f69a58623)
Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>
This fixes a crash in `_PyObject_TryGetInstanceAttribute` due to the use
of `_PyDictKeys_StringLookup` on an unlocked dictionary that may be
concurrently modified.
The underlying bug was already fixed in 3.14 and the main branch.
(partially cherry picked from commit 1b15c89a17)
The function `dict_set_fromkeys()` adds elements of a set to an existing
dictionary. The size of the expanded dictionary was estimated with
`PySet_GET_SIZE(iterable)`, which did not take into account the size of the
existing dictionary.
(cherry picked from commit 421ba589d0)
Co-authored-by: Angela Liss <59097311+angela-tarantula@users.noreply.github.com>
Two races related to the type lookup cache, when used in the
free-threaded build. This caused test_opcache to sometimes fail (as
well as other hard to re-produce failures).
gh-132713: Fix typing.Union[index] race condition (GH-132802)
Add union_init_parameters() helper function. Use a critical section
to initialize the 'parameters' member.
(cherry picked from commit dc3e9638c2)
Co-authored-by: Victor Stinner <vstinner@python.org>
gh-132176: Fix crash on `type()` when `tuple` subclass passed as `bases` (GH-132212)
(cherry picked from commit b6c552f9e6)
Co-authored-by: sobolevn <mail@sobolevn.me>
Co-authored-by: Victor Stinner <vstinner@python.org>
The clearing of the key, hash, and value need to use atomic operations
to avoid a data race with concurrent read operations.
(cherry picked from commit c00ac57824)
Co-authored-by: Sam Gross <colesbury@gmail.com>
The free-threading build interns and immortalizes most constants
generated by the bytecode compiler. However, users can construct their
own code objects with arbitrary constants. We should not intern or
immortalize these objects if they are not of a type that we know how to
handle.
This change fixes a reference leak failure in the recently added
`test_code.test_unusual_constants` test. It also addresses a potential
crash that could occur when attempting to destroy an immortalized
object during interpreter shutdown.
(cherry picked from commit 12db45211d)
gh-130932: Fix incorrect exception handling in _PyModule_IsPossiblyShadowing (GH-130934)
I chose to not raise an exception here because I think it would be
confusing for module attribute access to start raising something other
than AttributeError if e.g. the cwd goes away
Without the change in moduleobject.c
```
./python.exe -m unittest test.test_import.ImportTests.test_script_shadowing_stdlib_cwd_failure
...
Assertion failed: (PyErr_Occurred()), function _PyObject_SetAttributeErrorContext, file object.c, line 1253.
```
(cherry picked from commit 0a9ae5ed48)
Co-authored-by: Shantanu <12621235+hauntsaninja@users.noreply.github.com>
The bytecode compiler only generates a few different types of constants,
like str, int, tuple, slices, etc. Users can construct code objects with
various unusual constants, including ones that are not hashable or not
even constant.
The free threaded build previously crashed with a fatal error when
confronted with these constants. Instead, treat distinct objects of
otherwise unhandled types as not equal for the purposes of deduplication.
(cherry picked from commit 2905690a91)
Co-authored-by: Sam Gross <colesbury@gmail.com>
gh-130547: Fix race between dict_dealloc and split_keys_entry_added (gh-130778)
(cherry picked from commit 80e6d3ec49)
Co-authored-by: Donghee Na <donghee.na@python.org>
This avoids a case where the interpreter's queue of memory to be freed
could grow rapidly if there are many short lived threads.
(cherry picked from commit 2f6e0e9f70)
Fix a few thread-safety bugs to enable test_opcache when run with TSAN:
* Use relaxed atomics when clearing `ht->_spec_cache.getitem`
(gh-115999)
* Add temporary suppression for type slot modifications (gh-127266)
* Use atomic load when reading `*dictptr`
(cherry picked from commit f151d27159)
The use of PySys_GetObject() and _PySys_GetAttr(), which return a borrowed
reference, has been replaced by using one of the following functions, which
return a strong reference and distinguish a missing attribute from an error:
_PySys_GetOptionalAttr(), _PySys_GetOptionalAttrString(),
_PySys_GetRequiredAttr(), and _PySys_GetRequiredAttrString().
(cherry picked from commit 0ef4ffeefd)
* gh-128714: Fix function object races in `__annotate__`, `__annotations__` and `__type_params__` in free-threading build (#129016)
(cherry picked from commit 55f17b77c3)
---------
Co-authored-by: Hugo van Kemenade <1324225+hugovk@users.noreply.github.com>
