The standard says that a call to `memcpy` must pass a valid source and
destination pointer even if the size is 0, so we must avoid calling
`memcpy` when our source pointer is NULL. If we don't, an optimizing
compiler can decide that the pointer must be non-NULL based on the
presence of UB, and optimize out checks for null pointers.
Specifically, note that the standard says:
Where an argument declared as size_t n specifies the length of the
array for a function, n can have the value zero on a call to that
function. Unless explicitly stated otherwise in the description of
a particular function in this subclause, pointer arguments on such
a call shall still have valid values, as described in 7.1.4.
And section 7.1.4 says:
If an argument to a function has an invalid value (such as a value
outside the domain of the function, or a pointer outside the address
space of the program, or a null pointer, or a pointer to
non-modifiable storage when the corresponding parameter is not
const-qualified) or a type (after default argument promotion) not
expected by a function with a variable number of arguments, the
behavior is undefined.
The specification for `memcpy` doesn't state that it's allowed to be
called with null pointers, and Linux's `/usr/include/string.h` declares
`memcpy` as `__nonnull ((1, 2))`.
Append parsed values to the result list with _PyList_AppendTakeRef and
insert key/value pairs with _PyDict_SetItem_Take2, which take ownership of
the references instead of incref-ing on insert and then decref-ing the
local. This removes a reference-count round-trip per element (and, on the
free-threaded build, a per-append lock).
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Fix crashes in socket.sendmsg() and socket.recvmsg_into() that could
occur if buffer sequences are mutated re-entrantly during argument
parsing via __buffer__ protocol callbacks.
The bug occurs because:
1. PySequence_Fast() returns the original list object when the input
is already a list (not a copy).
2. During iteration, PyObject_GetBuffer() triggers __buffer__
callbacks which may clear the list.
3. Subsequent iterations access invalid memory (heap OOB read).
The fix replaces PySequence_Fast() with PySequence_Tuple() which
always creates a new tuple, ensuring the sequence cannot be mutated
during iteration.
Co-authored-by: tonghuaroot <23011166+tonghuaroot@users.noreply.github.com>
TextIOWrapper keeps its underlying stream in a member called
`self->buffer`. That stream can be detached by user code, such as custom
`.flush` implementations resulting in `self->buffer` being set to NULL.
The implementation often checked at the start of functions if
`self->buffer` is in a good state, but did not always recheck after
other Python code was called which could modify `self->buffer`.
The cases which need to be re-checked are hard to spot so rather than
rely on reviewer effort create better safety by making all self->buffer
access go through helper functions.
Thank you yihong0618 for the test, NEWS and initial implementation in
gh-143041.
Co-authored-by: yihong0618 <zouzou0208@gmail.com>
Co-authored-by: Victor Stinner <vstinner@python.org>
Scan the nfc_first/nfc_last reindex tables comparing only .start, range-check
the candidate once, and terminate on a sentinel above every codepoint, so each
entry costs a single comparison. ~2x faster on non-Latin and combining-heavy
NFC/NFKC input; no new data tables.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Replace all documentation which says "See PEP 585"
The following classes in the stdlib get simple updates:
- array.array
- asyncio.Future
- asyncio.Task
- collections.defaultdict
- collections.deque
- contextvars.ContextVar
- contextvars.Token
- ctypes.Array
- os.DirEntry
- re.Match
- re.Pattern
- string.templatelib.Interpolation
- string.templatelib.Template
- types.MappingProxyType
- queue.SimpleQueue
- weakref.ref
The following classes are documented publicly as functions, and are
therefore updated internally (`__class_getitem__.__doc__`) but not in the
public docs:
- functools.partial
- itertools.chain
The following builtin types have updates to `__class_getitem__.__doc__`
but not to any documentation pages:
- BaseExceptionGroup
- coroutines (from generators)
- dict
- enumerate
- frozendict
- frozenset
- generators (and async generators)
- list
- memoryview
- set
- slice
- tuple
Special cases:
- union objects are now documented as "supporting class-level []",
rather than anything to do with generics.
- Templates might be generic over a single type (union, in theory) or
over a TypeVarTuple. As this is not currently fully settled, it is
marked with a comment and a mild hint that it is a single type is used
(namely, "type" is singular rather than "types", plural)
* Apply suggestions from code review
Co-authored-by: Jelle Zijlstra <jelle.zijlstra@gmail.com>
* Correct several class getitem docs
And expand the text for tuples.
Co-authored-by: Jelle Zijlstra <906600+JelleZijlstra@users.noreply.github.com>
* Add notes on generic typing of builtins
* Fix typo in tuple.__class_getitem__ docstring
* Typo fix: malformed refs
Fix `generic` links which weren't marked as `:ref:`.
* Strike unnecessary docs on generic-ness
Co-authored-by: Jelle Zijlstra <906600+JelleZijlstra@users.noreply.github.com>
* Apply suggestions from code review
These are applied at both the originally indicated locations and in the
corresponding docstring definitions.
Co-authored-by: Alex Waygood <66076021+AlexWaygood@users.noreply.github.com>
* Update Doc/library/re.rst
Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com>
* Update Objects/enumobject.c
Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com>
* Remove tuple generic doc in 'stdtypes' page
This is covered in more detail in the cross-linked typing documentation.
The other copy of this documentation -- in the docstring for
`tuple.__class_getitem__` -- is left in place.
* Fix whitespace around new doc of generics
Per review, do not introduce or remove whitespace such that section
breaks are altered by the introduction of doc on various generic types.
In most cases, this is a removal of an extra line.
In one case (Arrays), it is the reintroduction of a line.
Additionally, two other minor fixes are included:
- incorrect indent on 'defaultdicts'
- make `mappingproxy.__class_getitem__.__doc__` consistent with other
mapping type generic docs
Co-authored-by: Bénédikt Tran <10796600+picnixz@users.noreply.github.com>
* Move placement of memoryview generic note
Previous placement was at the end of the main docstring, which is
consistent with other types but places it after a section on various
methods (which makes it read somewhat inconsistently). Moving it up
helps resolve.
Co-authored-by: Bénédikt Tran <10796600+picnixz@users.noreply.github.com>
* Ensure sphinxdoc does not start sentences lowercase
Lowercase class names at the start of sentences are marked out with the
`class` role. In the case of `deque`, documentation already refers to
these as `Deques`, so this form is preferred.
* Apply suggestions from code review
Co-authored-by: Bénédikt Tran <10796600+picnixz@users.noreply.github.com>
* Fix line endings and wrap more tightly
Line endings fixed by pre-commit ; also re-wrapped the MappingProxyType
text which was too long.
* Use 'ContextVars' style in sphinx doc
---------
Co-authored-by: Jelle Zijlstra <jelle.zijlstra@gmail.com>
Co-authored-by: Jelle Zijlstra <906600+JelleZijlstra@users.noreply.github.com>
Co-authored-by: Alex Waygood <66076021+AlexWaygood@users.noreply.github.com>
Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com>
Co-authored-by: Bénédikt Tran <10796600+picnixz@users.noreply.github.com>
Replace the insertion sort used for canonical ordering of combining
characters with a hybrid approach: insertion sort for short runs (< 20)
and counting sort for longer runs, reducing worst-case complexity from
O(n^2) to O(n). This prevents denial of service via crafted Unicode
strings with many combining characters in alternating CCC order.
Co-authored-by: ch4n3-yoon <ch4n3.yoon@gmail.com>
Co-authored-by: Seokchan Yoon <13852925+ch4n3-yoon@users.noreply.github.com>
Co-authored-by: Stan Ulbrych <stan@python.org>
Co-authored-by: Bénédikt Tran <10796600+picnixz@users.noreply.github.com>
Co-authored-by: Petr Viktorin <encukou@gmail.com>
Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>
Co-authored-by: Maurycy Pawłowski-Wieroński <maurycy@maurycy.com>
state_init() always did PyMem_New(state->mark, groups*2), which for a
pattern with no capturing groups is PyMem_Malloc(0) -- a real allocation
(plus matching free) on every match/search/fullmatch call, for an array
that is never read: groupless patterns emit no MARK opcodes and group 0's
span is taken from state->start/ptr.
Guard the allocation with `if (pattern->groups)`. state->mark stays NULL
(set by the preceding memset), and both the error path and state_fini
already PyMem_Free(NULL) safely.
Supported encodings: "cp932", "cp949", "cp950", "Big5","EUC-JP",
"GB2312", "GBK", "johab", and "Shift_JIS".
Partially supported encodings (only BMP characters): "Big5-HKSCS",
"EUC_JIS-2004", "EUC_JISX0213", "Shift_JIS-2004", "Shift_JISX0213",
"utf-8-sig" and non-standard aliases like "UTF8" (without hyphen).
The parser now raises ValueError for known unsupported
multi-byte encodings such us "ISO-2022-JP" or "raw-unicode-escape"
instead of failing later, when encounter non-ASCII data.
