Stowage/cpython
2
0
Fork 0
mirror of https://github.com/python/cpython.git synced 2025-12-08 06:10:17 +00:00
Code Issues Projects Releases Packages Wiki Activity

Merge remote-tracking branch 'upstream/main' into tachyon-opcodes

Browse source
This commit is contained in:
Pablo Galindo Salgado 2025-12-07 04:38:30 +00:00
commit 8129e3d7f4
154 changed files with 10330 additions and 4926 deletions
Download patch file Download diff file Expand all files Collapse all files

3
.github/CODEOWNERS vendored
View file

@ -126,6 +126,9 @@ Doc/howto/clinic.rst @erlend-aasland @AA-Turner
# C Analyser
Tools/c-analyzer/ @ericsnowcurrently
# C API Documentation Checks
Tools/check-c-api-docs/ @ZeroIntensity
# Fuzzing
Modules/_xxtestfuzz/ @ammaraskar

3
.github/ISSUE_TEMPLATE/config.yml vendored
View file

@ -5,3 +5,6 @@ contact_links:
- name: "Proposing new features"
about: "Submit major feature proposal (e.g. syntax changes) to an ideas forum first."
url: "https://discuss.python.org/c/ideas/6"
- name: "Python Install Manager issues"
about: "Report issues with the Python Install Manager (for Windows)"
url: "https://github.com/python/pymanager/issues"

3
.github/workflows/build.yml vendored
View file

@ -142,6 +142,9 @@ jobs:
- name: Check for unsupported C global variables
if: github.event_name == 'pull_request' # $GITHUB_EVENT_NAME
run: make check-c-globals
- name: Check for undocumented C APIs
run: make check-c-api-docs
build-windows:
name: >-

39
Android/android.py
View file

@ -29,6 +29,7 @@
ANDROID_DIR.name == "Android" and (PYTHON_DIR / "pyconfig.h.in").exists()
)
ENV_SCRIPT = ANDROID_DIR / "android-env.sh"
TESTBED_DIR = ANDROID_DIR / "testbed"
CROSS_BUILD_DIR = PYTHON_DIR / "cross-build"
@ -129,12 +130,11 @@ def android_env(host):
sysconfig_filename = next(sysconfig_files).name
host = re.fullmatch(r"_sysconfigdata__android_(.+).py", sysconfig_filename)[1]
env_script = ANDROID_DIR / "android-env.sh"
env_output = subprocess.run(
f"set -eu; "
f"HOST={host}; "
f"PREFIX={prefix}; "
f". {env_script}; "
f". {ENV_SCRIPT}; "
f"export",
check=True, shell=True, capture_output=True, encoding='utf-8',
).stdout
@ -151,7 +151,7 @@ def android_env(host):
env[key] = value
if not env:
raise ValueError(f"Found no variables in {env_script.name} output:\n"
raise ValueError(f"Found no variables in {ENV_SCRIPT.name} output:\n"
+ env_output)
return env
@ -281,15 +281,30 @@ def clean_all(context):
def setup_ci():
# https://github.blog/changelog/2024-04-02-github-actions-hardware-accelerated-android-virtualization-now-available/
if "GITHUB_ACTIONS" in os.environ and platform.system() == "Linux":
run(
["sudo", "tee", "/etc/udev/rules.d/99-kvm4all.rules"],
input='KERNEL=="kvm", GROUP="kvm", MODE="0666", OPTIONS+="static_node=kvm"\n',
text=True,
)
run(["sudo", "udevadm", "control", "--reload-rules"])
run(["sudo", "udevadm", "trigger", "--name-match=kvm"])
if "GITHUB_ACTIONS" in os.environ:
# Enable emulator hardware acceleration
# (https://github.blog/changelog/2024-04-02-github-actions-hardware-accelerated-android-virtualization-now-available/).
if platform.system() == "Linux":
run(
["sudo", "tee", "/etc/udev/rules.d/99-kvm4all.rules"],
input='KERNEL=="kvm", GROUP="kvm", MODE="0666", OPTIONS+="static_node=kvm"\n',
text=True,
)
run(["sudo", "udevadm", "control", "--reload-rules"])
run(["sudo", "udevadm", "trigger", "--name-match=kvm"])
# Free up disk space by deleting unused versions of the NDK
# (https://github.com/freakboy3742/pyspamsum/pull/108).
for line in ENV_SCRIPT.read_text().splitlines():
if match := re.fullmatch(r"ndk_version=(.+)", line):
ndk_version = match[1]
break
else:
raise ValueError(f"Failed to find NDK version in {ENV_SCRIPT.name}")
for item in (android_home / "ndk").iterdir():
if item.name[0].isdigit() and item.name != ndk_version:
delete_glob(item)
def setup_sdk():

4
Android/testbed/app/build.gradle.kts
View file

@ -79,7 +79,7 @@ android {
val androidEnvFile = file("../../android-env.sh").absoluteFile
namespace = "org.python.testbed"
compileSdk = 34
compileSdk = 35
defaultConfig {
applicationId = "org.python.testbed"
@ -92,7 +92,7 @@ android {
}
throw GradleException("Failed to find API level in $androidEnvFile")
}
targetSdk = 34
targetSdk = 35
versionCode = 1
versionName = "1.0"

2
Doc/c-api/set.rst
View file

@ -147,7 +147,7 @@ subtypes but not for instances of :class:`frozenset` or its subtypes.
Return ``1`` if found and removed, ``0`` if not found (no action taken), and ``-1`` if an
error is encountered. Does not raise :exc:`KeyError` for missing keys. Raise a
:exc:`TypeError` if the *key* is unhashable. Unlike the Python :meth:`~frozenset.discard`
:exc:`TypeError` if the *key* is unhashable. Unlike the Python :meth:`~set.discard`
method, this function does not automatically convert unhashable sets into
temporary frozensets. Raise :exc:`SystemError` if *set* is not an
instance of :class:`set` or its subtype.

9
Doc/library/readline.rst
View file

@ -246,6 +246,15 @@ Startup hooks
if Python was compiled for a version of the library that supports it.
.. function:: get_pre_input_hook()
Get the current pre-input hook function, or ``None`` if no pre-input hook
function has been set. This function only exists if Python was compiled
for a version of the library that supports it.
.. versionadded:: next
.. _readline-completion:
Completion

402
Doc/library/stdtypes.rst
View file

@ -2656,6 +2656,8 @@ expression support in the :mod:`re` module).
single: : (colon); in formatted string literal
single: = (equals); for help in debugging using string literals
.. _stdtypes-fstrings:
Formatted String Literals (f-strings)
-------------------------------------
@ -2664,123 +2666,147 @@ Formatted String Literals (f-strings)
The :keyword:`await` and :keyword:`async for` can be used in expressions
within f-strings.
.. versionchanged:: 3.8
Added the debugging operator (``=``)
Added the debug specifier (``=``)
.. versionchanged:: 3.12
Many restrictions on expressions within f-strings have been removed.
Notably, nested strings, comments, and backslashes are now permitted.
An :dfn:`f-string` (formally a :dfn:`formatted string literal`) is
a string literal that is prefixed with ``f`` or ``F``.
This type of string literal allows embedding arbitrary Python expressions
within *replacement fields*, which are delimited by curly brackets (``{}``).
These expressions are evaluated at runtime, similarly to :meth:`str.format`,
and are converted into regular :class:`str` objects.
For example:
This type of string literal allows embedding the results of arbitrary Python
expressions within *replacement fields*, which are delimited by curly
brackets (``{}``).
Each replacement field must contain an expression, optionally followed by:
.. doctest::
* a *debug specifier* -- an equal sign (``=``);
* a *conversion specifier* -- ``!s``, ``!r`` or ``!a``; and/or
* a *format specifier* prefixed with a colon (``:``).
>>> who = 'nobody'
>>> nationality = 'Spanish'
>>> f'{who.title()} expects the {nationality} Inquisition!'
'Nobody expects the Spanish Inquisition!'
See the :ref:`Lexical Analysis section on f-strings <f-strings>` for details
on the syntax of these fields.
It is also possible to use a multi line f-string:
Debug specifier
^^^^^^^^^^^^^^^
.. doctest::
.. versionadded:: 3.8
>>> f'''This is a string
... on two lines'''
'This is a string\non two lines'
If a debug specifier -- an equal sign (``=``) -- appears after the replacement
field expression, the resulting f-string will contain the expression's source,
the equal sign, and the value of the expression.
This is often useful for debugging::
A single opening curly bracket, ``'{'``, marks a *replacement field* that
can contain any Python expression:
>>> number = 14.3
>>> f'{number=}'
'number=14.3'
.. doctest::
Whitespace before, inside and after the expression, as well as whitespace
after the equal sign, is significant --- it is retained in the result::
>>> nationality = 'Spanish'
>>> f'The {nationality} Inquisition!'
'The Spanish Inquisition!'
>>> f'{ number - 4 = }'
' number - 4 = 10.3'
To include a literal ``{`` or ``}``, use a double bracket:
.. doctest::
Conversion specifier
^^^^^^^^^^^^^^^^^^^^
>>> x = 42
>>> f'{{x}} is {x}'
'{x} is 42'
Functions can also be used, and :ref:`format specifiers <formatstrings>`:
.. doctest::
>>> from math import sqrt
>>> f'√2 \N{ALMOST EQUAL TO} {sqrt(2):.5f}'
'√2 ≈ 1.41421'
Any non-string expression is converted using :func:`str`, by default:
.. doctest::
By default, the value of a replacement field expression is converted to
a string using :func:`str`::
>>> from fractions import Fraction
>>> f'{Fraction(1, 3)}'
>>> one_third = Fraction(1, 3)
>>> f'{one_third}'
'1/3'
To use an explicit conversion, use the ``!`` (exclamation mark) operator,
followed by any of the valid formats, which are:
When a debug specifier but no format specifier is used, the default conversion
instead uses :func:`repr`::
========== ==============
Conversion Meaning
========== ==============
``!a`` :func:`ascii`
``!r`` :func:`repr`
``!s`` :func:`str`
========== ==============
>>> f'{one_third = }'
'one_third = Fraction(1, 3)'
For example:
The conversion can be specified explicitly using one of these specifiers:
.. doctest::
* ``!s`` for :func:`str`
* ``!r`` for :func:`repr`
* ``!a`` for :func:`ascii`
>>> from fractions import Fraction
>>> f'{Fraction(1, 3)!s}'
For example::
>>> str(one_third)
'1/3'
>>> f'{Fraction(1, 3)!r}'
>>> repr(one_third)
'Fraction(1, 3)'
>>> question = '¿Dónde está el Presidente?'
>>> print(f'{question!a}')
'\xbfD\xf3nde est\xe1 el Presidente?'
While debugging it may be helpful to see both the expression and its value,
by using the equals sign (``=``) after the expression.
This preserves spaces within the brackets, and can be used with a converter.
By default, the debugging operator uses the :func:`repr` (``!r``) conversion.
For example:
>>> f'{one_third!s} is {one_third!r}'
'1/3 is Fraction(1, 3)'
.. doctest::
>>> string = "¡kočka 😸!"
>>> ascii(string)
"'\\xa1ko\\u010dka \\U0001f638!'"
>>> f'{string = !a}'
"string = '\\xa1ko\\u010dka \\U0001f638!'"
Format specifier
^^^^^^^^^^^^^^^^
After the expression has been evaluated, and possibly converted using an
explicit conversion specifier, it is formatted using the :func:`format` function.
If the replacement field includes a *format specifier* introduced by a colon
(``:``), the specifier is passed to :func:`!format` as the second argument.
The result of :func:`!format` is then used as the final value for the
replacement field. For example::
>>> from fractions import Fraction
>>> calculation = Fraction(1, 3)
>>> f'{calculation=}'
'calculation=Fraction(1, 3)'
>>> f'{calculation = }'
'calculation = Fraction(1, 3)'
>>> f'{calculation = !s}'
'calculation = 1/3'
>>> one_third = Fraction(1, 3)
>>> f'{one_third:.6f}'
'0.333333'
>>> f'{one_third:_^+10}'
'___+1/3___'
>>> >>> f'{one_third!r:_^20}'
'___Fraction(1, 3)___'
>>> f'{one_third = :~>10}~'
'one_third = ~~~~~~~1/3~'
Once the output has been evaluated, it can be formatted using a
:ref:`format specifier <formatstrings>` following a colon (``':'``).
After the expression has been evaluated, and possibly converted to a string,
the :meth:`!__format__` method of the result is called with the format specifier,
or the empty string if no format specifier is given.
The formatted result is then used as the final value for the replacement field.
For example:
.. _stdtypes-tstrings:
.. doctest::
Template String Literals (t-strings)
------------------------------------
>>> from fractions import Fraction
>>> f'{Fraction(1, 7):.6f}'
'0.142857'
>>> f'{Fraction(1, 7):_^+10}'
'___+1/7___'
An :dfn:`t-string` (formally a :dfn:`template string literal`) is
a string literal that is prefixed with ``t`` or ``T``.
These strings follow the same syntax and evaluation rules as
:ref:`formatted string literals <stdtypes-fstrings>`,
with for the following differences:
* Rather than evaluating to a ``str`` object, template string literals evaluate
to a :class:`string.templatelib.Template` object.
* The :func:`format` protocol is not used.
Instead, the format specifier and conversions (if any) are passed to
a new :class:`~string.templatelib.Interpolation` object that is created
for each evaluated expression.
It is up to code that processes the resulting :class:`~string.templatelib.Template`
object to decide how to handle format specifiers and conversions.
* Format specifiers containing nested replacement fields are evaluated eagerly,
prior to being passed to the :class:`~string.templatelib.Interpolation` object.
For instance, an interpolation of the form ``{amount:.{precision}f}`` will
evaluate the inner expression ``{precision}`` to determine the value of the
``format_spec`` attribute.
If ``precision`` were to be ``2``, the resulting format specifier
would be ``'.2f'``.
* When the equals sign ``'='`` is provided in an interpolation expression,
the text of the expression is appended to the literal string that precedes
the relevant interpolation.
This includes the equals sign and any surrounding whitespace.
The :class:`!Interpolation` instance for the expression will be created as
normal, except that :attr:`~string.templatelib.Interpolation.conversion` will
be set to '``r``' (:func:`repr`) by default.
If an explicit conversion or format specifier are provided,
this will override the default behaviour.
.. _old-string-formatting:
@ -4800,7 +4826,7 @@ other sequence-like behavior.
There are currently two built-in set types, :class:`set` and :class:`frozenset`.
The :class:`set` type is mutable --- the contents can be changed using methods
like :meth:`add <frozenset.add>` and :meth:`remove <frozenset.add>`.
like :meth:`~set.add` and :meth:`~set.remove`.
Since it is mutable, it has no hash value and cannot be used as
either a dictionary key or as an element of another set.
The :class:`frozenset` type is immutable and :term:`hashable` ---
@ -4822,164 +4848,172 @@ The constructors for both classes work the same:
objects. If *iterable* is not specified, a new empty set is
returned.
Sets can be created by several means:
Sets can be created by several means:
* Use a comma-separated list of elements within braces: ``{'jack', 'sjoerd'}``
* Use a set comprehension: ``{c for c in 'abracadabra' if c not in 'abc'}``
* Use the type constructor: ``set()``, ``set('foobar')``, ``set(['a', 'b', 'foo'])``
* Use a comma-separated list of elements within braces: ``{'jack', 'sjoerd'}``
* Use a set comprehension: ``{c for c in 'abracadabra' if c not in 'abc'}``
* Use the type constructor: ``set()``, ``set('foobar')``, ``set(['a', 'b', 'foo'])``
Instances of :class:`set` and :class:`frozenset` provide the following
operations:
Instances of :class:`set` and :class:`frozenset` provide the following
operations:
.. describe:: len(s)
.. describe:: len(s)
Return the number of elements in set *s* (cardinality of *s*).
Return the number of elements in set *s* (cardinality of *s*).
.. describe:: x in s
.. describe:: x in s
Test *x* for membership in *s*.
Test *x* for membership in *s*.
.. describe:: x not in s
.. describe:: x not in s
Test *x* for non-membership in *s*.
Test *x* for non-membership in *s*.
.. method:: isdisjoint(other, /)
.. method:: frozenset.isdisjoint(other, /)
set.isdisjoint(other, /)
Return ``True`` if the set has no elements in common with *other*. Sets are
disjoint if and only if their intersection is the empty set.
Return ``True`` if the set has no elements in common with *other*. Sets are
disjoint if and only if their intersection is the empty set.
.. method:: issubset(other, /)
set <= other
.. method:: frozenset.issubset(other, /)
set.issubset(other, /)
.. describe:: set <= other
Test whether every element in the set is in *other*.
Test whether every element in the set is in *other*.
.. method:: set < other
.. describe:: set < other
Test whether the set is a proper subset of *other*, that is,
``set <= other and set != other``.
Test whether the set is a proper subset of *other*, that is,
``set <= other and set != other``.
.. method:: issuperset(other, /)
set >= other
.. method:: frozenset.issuperset(other, /)
set.issuperset(other, /)
.. describe:: set >= other
Test whether every element in *other* is in the set.
Test whether every element in *other* is in the set.
.. method:: set > other
.. describe:: set > other
Test whether the set is a proper superset of *other*, that is, ``set >=
other and set != other``.
Test whether the set is a proper superset of *other*, that is, ``set >=
other and set != other``.
.. method:: union(*others)
set | other | ...
.. method:: frozenset.union(*others)
set.union(*others)
.. describe:: set | other | ...
Return a new set with elements from the set and all others.
Return a new set with elements from the set and all others.
.. method:: intersection(*others)
set & other & ...
.. method:: frozenset.intersection(*others)
set.intersection(*others)
.. describe:: set & other & ...
Return a new set with elements common to the set and all others.
Return a new set with elements common to the set and all others.
.. method:: difference(*others)
set - other - ...
.. method:: frozenset.difference(*others)
set.difference(*others)
.. describe:: set - other - ...
Return a new set with elements in the set that are not in the others.
Return a new set with elements in the set that are not in the others.
.. method:: symmetric_difference(other, /)
set ^ other
.. method:: frozenset.symmetric_difference(other, /)
set.symmetric_difference(other, /)
.. describe:: set ^ other
Return a new set with elements in either the set or *other* but not both.
Return a new set with elements in either the set or *other* but not both.
.. method:: copy()
.. method:: frozenset.copy()
set.copy()
Return a shallow copy of the set.
Return a shallow copy of the set.
Note, the non-operator versions of :meth:`union`, :meth:`intersection`,
:meth:`difference`, :meth:`symmetric_difference`, :meth:`issubset`, and
:meth:`issuperset` methods will accept any iterable as an argument. In
contrast, their operator based counterparts require their arguments to be
sets. This precludes error-prone constructions like ``set('abc') & 'cbs'``
in favor of the more readable ``set('abc').intersection('cbs')``.
Note, the non-operator versions of :meth:`~frozenset.union`,
:meth:`~frozenset.intersection`, :meth:`~frozenset.difference`, :meth:`~frozenset.symmetric_difference`, :meth:`~frozenset.issubset`, and
:meth:`~frozenset.issuperset` methods will accept any iterable as an argument. In
contrast, their operator based counterparts require their arguments to be
sets. This precludes error-prone constructions like ``set('abc') & 'cbs'``
in favor of the more readable ``set('abc').intersection('cbs')``.
Both :class:`set` and :class:`frozenset` support set to set comparisons. Two
sets are equal if and only if every element of each set is contained in the
other (each is a subset of the other). A set is less than another set if and
only if the first set is a proper subset of the second set (is a subset, but
is not equal). A set is greater than another set if and only if the first set
is a proper superset of the second set (is a superset, but is not equal).
Both :class:`set` and :class:`frozenset` support set to set comparisons. Two
sets are equal if and only if every element of each set is contained in the
other (each is a subset of the other). A set is less than another set if and
only if the first set is a proper subset of the second set (is a subset, but
is not equal). A set is greater than another set if and only if the first set
is a proper superset of the second set (is a superset, but is not equal).
Instances of :class:`set` are compared to instances of :class:`frozenset`
based on their members. For example, ``set('abc') == frozenset('abc')``
returns ``True`` and so does ``set('abc') in set([frozenset('abc')])``.
Instances of :class:`set` are compared to instances of :class:`frozenset`
based on their members. For example, ``set('abc') == frozenset('abc')``
returns ``True`` and so does ``set('abc') in set([frozenset('abc')])``.
The subset and equality comparisons do not generalize to a total ordering
function. For example, any two nonempty disjoint sets are not equal and are not
subsets of each other, so *all* of the following return ``False``: ``a<b``,
``a==b``, or ``a>b``.
The subset and equality comparisons do not generalize to a total ordering
function. For example, any two nonempty disjoint sets are not equal and are not
subsets of each other, so *all* of the following return ``False``: ``a<b``,
``a==b``, or ``a>b``.
Since sets only define partial ordering (subset relationships), the output of
the :meth:`list.sort` method is undefined for lists of sets.
Since sets only define partial ordering (subset relationships), the output of
the :meth:`list.sort` method is undefined for lists of sets.
Set elements, like dictionary keys, must be :term:`hashable`.
Set elements, like dictionary keys, must be :term:`hashable`.
Binary operations that mix :class:`set` instances with :class:`frozenset`
return the type of the first operand. For example: ``frozenset('ab') |
set('bc')`` returns an instance of :class:`frozenset`.
Binary operations that mix :class:`set` instances with :class:`frozenset`
return the type of the first operand. For example: ``frozenset('ab') |
set('bc')`` returns an instance of :class:`frozenset`.
The following table lists operations available for :class:`set` that do not
apply to immutable instances of :class:`frozenset`:
The following table lists operations available for :class:`set` that do not
apply to immutable instances of :class:`frozenset`:
.. method:: update(*others)
set |= other | ...
.. method:: set.update(*others)
.. describe:: set |= other | ...
Update the set, adding elements from all others.
Update the set, adding elements from all others.
.. method:: intersection_update(*others)
set &= other & ...
.. method:: set.intersection_update(*others)
.. describe:: set &= other & ...
Update the set, keeping only elements found in it and all others.
Update the set, keeping only elements found in it and all others.
.. method:: difference_update(*others)
set -= other | ...
.. method:: set.difference_update(*others)
.. describe:: set -= other | ...
Update the set, removing elements found in others.
Update the set, removing elements found in others.
.. method:: symmetric_difference_update(other, /)
set ^= other
.. method:: set.symmetric_difference_update(other, /)
.. describe:: set ^= other
Update the set, keeping only elements found in either set, but not in both.
Update the set, keeping only elements found in either set, but not in both.
.. method:: add(elem, /)
.. method:: set.add(elem, /)
Add element *elem* to the set.
Add element *elem* to the set.
.. method:: remove(elem, /)
.. method:: set.remove(elem, /)
Remove element *elem* from the set. Raises :exc:`KeyError` if *elem* is
not contained in the set.
Remove element *elem* from the set. Raises :exc:`KeyError` if *elem* is
not contained in the set.
.. method:: discard(elem, /)
.. method:: set.discard(elem, /)
Remove element *elem* from the set if it is present.
Remove element *elem* from the set if it is present.
.. method:: pop()
.. method:: set.pop()
Remove and return an arbitrary element from the set. Raises
:exc:`KeyError` if the set is empty.
Remove and return an arbitrary element from the set. Raises
:exc:`KeyError` if the set is empty.
.. method:: clear()
.. method:: set.clear()
Remove all elements from the set.
Remove all elements from the set.
Note, the non-operator versions of the :meth:`update`,
:meth:`intersection_update`, :meth:`difference_update`, and
:meth:`symmetric_difference_update` methods will accept any iterable as an
argument.
Note, the non-operator versions of the :meth:`~set.update`,
:meth:`~set.intersection_update`, :meth:`~set.difference_update`, and
:meth:`~set.symmetric_difference_update` methods will accept any iterable as an
argument.
Note, the *elem* argument to the :meth:`~object.__contains__`,
:meth:`remove`, and
:meth:`discard` methods may be a set. To support searching for an equivalent
frozenset, a temporary one is created from *elem*.
Note, the *elem* argument to the :meth:`~object.__contains__`,
:meth:`~set.remove`, and
:meth:`~set.discard` methods may be a set. To support searching for an equivalent
frozenset, a temporary one is created from *elem*.
.. _typesmapping:

2
Doc/reference/datamodel.rst
View file

@ -449,7 +449,7 @@ Sets
These represent a mutable set. They are created by the built-in :func:`set`
constructor and can be modified afterwards by several methods, such as
:meth:`add <frozenset.add>`.
:meth:`~set.add`.
Frozen sets

2
Doc/reference/expressions.rst
View file

@ -174,7 +174,7 @@ Formally:
.. grammar-snippet::
:group: python-grammar
strings: ( `STRING` | fstring)+ | tstring+
strings: ( `STRING` | `fstring`)+ | `tstring`+
This feature is defined at the syntactical level, so it only works with literals.
To concatenate string expressions at run time, the '+' operator may be used::

363
Doc/reference/lexical_analysis.rst
View file

@ -345,7 +345,15 @@ Whitespace between tokens
Except at the beginning of a logical line or in string literals, the whitespace
characters space, tab and formfeed can be used interchangeably to separate
tokens. Whitespace is needed between two tokens only if their concatenation
tokens:
.. grammar-snippet::
:group: python-grammar
whitespace: ' ' | tab | formfeed
Whitespace is needed between two tokens only if their concatenation
could otherwise be interpreted as a different token. For example, ``ab`` is one
token, but ``a b`` is two tokens. However, ``+a`` and ``+ a`` both produce
two tokens, ``+`` and ``a``, as ``+a`` is not a valid token.
@ -1032,124 +1040,59 @@ f-strings
---------
.. versionadded:: 3.6
.. versionchanged:: 3.7
The :keyword:`await` and :keyword:`async for` can be used in expressions
within f-strings.
.. versionchanged:: 3.8
Added the debug specifier (``=``)
.. versionchanged:: 3.12
Many restrictions on expressions within f-strings have been removed.
Notably, nested strings, comments, and backslashes are now permitted.
A :dfn:`formatted string literal` or :dfn:`f-string` is a string literal
that is prefixed with '``f``' or '``F``'. These strings may contain
replacement fields, which are expressions delimited by curly braces ``{}``.
While other string literals always have a constant value, formatted strings
are really expressions evaluated at run time.
that is prefixed with '``f``' or '``F``'.
Unlike other string literals, f-strings do not have a constant value.
They may contain *replacement fields* delimited by curly braces ``{}``.
Replacement fields contain expressions which are evaluated at run time.
For example::
Escape sequences are decoded like in ordinary string literals (except when
a literal is also marked as a raw string). After decoding, the grammar
for the contents of the string is:
>>> who = 'nobody'
>>> nationality = 'Spanish'
>>> f'{who.title()} expects the {nationality} Inquisition!'
'Nobody expects the Spanish Inquisition!'
.. productionlist:: python-grammar
f_string: (`literal_char` | "{{" | "}}" | `replacement_field`)*
replacement_field: "{" `f_expression` ["="] ["!" `conversion`] [":" `format_spec`] "}"
f_expression: (`conditional_expression` | "*" `or_expr`)
: ("," `conditional_expression` | "," "*" `or_expr`)* [","]
: | `yield_expression`
conversion: "s" | "r" | "a"
format_spec: (`literal_char` | `replacement_field`)*
literal_char: <any code point except "{", "}" or NULL>
Any doubled curly braces (``{{`` or ``}}``) outside replacement fields
are replaced with the corresponding single curly brace::
The parts of the string outside curly braces are treated literally,
except that any doubled curly braces ``'{{'`` or ``'}}'`` are replaced
with the corresponding single curly brace. A single opening curly
bracket ``'{'`` marks a replacement field, which starts with a
Python expression. To display both the expression text and its value after
evaluation, (useful in debugging), an equal sign ``'='`` may be added after the
expression. A conversion field, introduced by an exclamation point ``'!'`` may
follow. A format specifier may also be appended, introduced by a colon ``':'``.
A replacement field ends with a closing curly bracket ``'}'``.
>>> print(f'{{...}}')
{...}
Other characters outside replacement fields are treated like in ordinary
string literals.
This means that escape sequences are decoded (except when a literal is
also marked as a raw string), and newlines are possible in triple-quoted
f-strings::
>>> name = 'Galahad'
>>> favorite_color = 'blue'
>>> print(f'{name}:\t{favorite_color}')
Galahad: blue
>>> print(rf"C:\Users\{name}")
C:\Users\Galahad
>>> print(f'''Three shall be the number of the counting
... and the number of the counting shall be three.''')
Three shall be the number of the counting
and the number of the counting shall be three.
Expressions in formatted string literals are treated like regular
Python expressions surrounded by parentheses, with a few exceptions.
An empty expression is not allowed, and both :keyword:`lambda` and
assignment expressions ``:=`` must be surrounded by explicit parentheses.
Python expressions.
Each expression is evaluated in the context where the formatted string literal
appears, in order from left to right. Replacement expressions can contain
newlines in both single-quoted and triple-quoted f-strings and they can contain
comments. Everything that comes after a ``#`` inside a replacement field
is a comment (even closing braces and quotes). In that case, replacement fields
must be closed in a different line.
.. code-block:: text
>>> f"abc{a # This is a comment }"
... + 3}"
'abc5'
.. versionchanged:: 3.7
Prior to Python 3.7, an :keyword:`await` expression and comprehensions
containing an :keyword:`async for` clause were illegal in the expressions
in formatted string literals due to a problem with the implementation.
.. versionchanged:: 3.12
Prior to Python 3.12, comments were not allowed inside f-string replacement
fields.
When the equal sign ``'='`` is provided, the output will have the expression
text, the ``'='`` and the evaluated value. Spaces after the opening brace
``'{'``, within the expression and after the ``'='`` are all retained in the
output. By default, the ``'='`` causes the :func:`repr` of the expression to be
provided, unless there is a format specified. When a format is specified it
defaults to the :func:`str` of the expression unless a conversion ``'!r'`` is
declared.
.. versionadded:: 3.8
The equal sign ``'='``.
If a conversion is specified, the result of evaluating the expression
is converted before formatting. Conversion ``'!s'`` calls :func:`str` on
the result, ``'!r'`` calls :func:`repr`, and ``'!a'`` calls :func:`ascii`.
The result is then formatted using the :func:`format` protocol. The
format specifier is passed to the :meth:`~object.__format__` method of the
expression or conversion result. An empty string is passed when the
format specifier is omitted. The formatted result is then included in
the final value of the whole string.
Top-level format specifiers may include nested replacement fields. These nested
fields may include their own conversion fields and :ref:`format specifiers
<formatspec>`, but may not include more deeply nested replacement fields. The
:ref:`format specifier mini-language <formatspec>` is the same as that used by
the :meth:`str.format` method.
Formatted string literals may be concatenated, but replacement fields
cannot be split across literals.
Some examples of formatted string literals::
>>> name = "Fred"
>>> f"He said his name is {name!r}."
"He said his name is 'Fred'."
>>> f"He said his name is {repr(name)}." # repr() is equivalent to !r
"He said his name is 'Fred'."
>>> width = 10
>>> precision = 4
>>> value = decimal.Decimal("12.34567")
>>> f"result: {value:{width}.{precision}}" # nested fields
'result: 12.35'
>>> today = datetime(year=2017, month=1, day=27)
>>> f"{today:%B %d, %Y}" # using date format specifier
'January 27, 2017'
>>> f"{today=:%B %d, %Y}" # using date format specifier and debugging
'today=January 27, 2017'
>>> number = 1024
>>> f"{number:#0x}" # using integer format specifier
'0x400'
>>> foo = "bar"
>>> f"{ foo = }" # preserves whitespace
" foo = 'bar'"
>>> line = "The mill's closed"
>>> f"{line = }"
'line = "The mill\'s closed"'
>>> f"{line = :20}"
"line = The mill's closed "
>>> f"{line = !r:20}"
'line = "The mill\'s closed" '
appears, in order from left to right.
An empty expression is not allowed, and both :keyword:`lambda` and
assignment expressions ``:=`` must be surrounded by explicit parentheses::
>>> f'{(half := 1/2)}, {half * 42}'
'0.5, 21.0'
Reusing the outer f-string quoting type inside a replacement field is
permitted::
@ -1158,10 +1101,6 @@ permitted::
>>> f"abc {a["x"]} def"
'abc 2 def'
.. versionchanged:: 3.12
Prior to Python 3.12, reuse of the same quoting type of the outer f-string
inside a replacement field was not possible.
Backslashes are also allowed in replacement fields and are evaluated the same
way as in any other context::
@ -1172,23 +1111,84 @@ way as in any other context::
b
c
.. versionchanged:: 3.12
Prior to Python 3.12, backslashes were not permitted inside an f-string
replacement field.
It is possible to nest f-strings::
Formatted string literals cannot be used as docstrings, even if they do not
include expressions.
>>> name = 'world'
>>> f'Repeated:{f' hello {name}' * 3}'
'Repeated: hello world hello world hello world'
::
Portable Python programs should not use more than 5 levels of nesting.
.. impl-detail::
CPython does not limit nesting of f-strings.
Replacement expressions can contain newlines in both single-quoted and
triple-quoted f-strings and they can contain comments.
Everything that comes after a ``#`` inside a replacement field
is a comment (even closing braces and quotes).
This means that replacement fields with comments must be closed in a
different line:
.. code-block:: text
>>> a = 2
>>> f"abc{a # This comment }" continues until the end of the line
... + 3}"
'abc5'
After the expression, replacement fields may optionally contain:
* a *debug specifier* -- an equal sign (``=``), optionally surrounded by
whitespace on one or both sides;
* a *conversion specifier* -- ``!s``, ``!r`` or ``!a``; and/or
* a *format specifier* prefixed with a colon (``:``).
See the :ref:`Standard Library section on f-strings <stdtypes-fstrings>`
for details on how these fields are evaluated.
As that section explains, *format specifiers* are passed as the second argument
to the :func:`format` function to format a replacement field value.
For example, they can be used to specify a field width and padding characters
using the :ref:`Format Specification Mini-Language <formatspec>`::
>>> number = 14.3
>>> f'{number:20.7f}'
' 14.3000000'
Top-level format specifiers may include nested replacement fields::
>>> field_size = 20
>>> precision = 7
>>> f'{number:{field_size}.{precision}f}'
' 14.3000000'
These nested fields may include their own conversion fields and
:ref:`format specifiers <formatspec>`::
>>> number = 3
>>> f'{number:{field_size}}'
' 3'
>>> f'{number:{field_size:05}}'
'00000000000000000003'
However, these nested fields may not include more deeply nested replacement
fields.
Formatted string literals cannot be used as :term:`docstrings <docstring>`,
even if they do not include expressions::
>>> def foo():
... f"Not a docstring"
...
>>> foo.__doc__ is None
True
>>> print(foo.__doc__)
None
See also :pep:`498` for the proposal that added formatted string literals,
and :meth:`str.format`, which uses a related format string mechanism.
.. seealso::
* :pep:`498` -- Literal String Interpolation
* :pep:`701` -- Syntactic formalization of f-strings
* :meth:`str.format`, which uses a related format string mechanism.
.. _t-strings:
@ -1201,36 +1201,99 @@ t-strings
A :dfn:`template string literal` or :dfn:`t-string` is a string literal
that is prefixed with '``t``' or '``T``'.
These strings follow the same syntax and evaluation rules as
:ref:`formatted string literals <f-strings>`, with the following differences:
These strings follow the same syntax rules as
:ref:`formatted string literals <f-strings>`.
For differences in evaluation rules, see the
:ref:`Standard Library section on t-strings <stdtypes-tstrings>`
* Rather than evaluating to a ``str`` object, template string literals evaluate
to a :class:`string.templatelib.Template` object.
* The :func:`format` protocol is not used.
Instead, the format specifier and conversions (if any) are passed to
a new :class:`~string.templatelib.Interpolation` object that is created
for each evaluated expression.
It is up to code that processes the resulting :class:`~string.templatelib.Template`
object to decide how to handle format specifiers and conversions.
Formal grammar for f-strings
----------------------------
* Format specifiers containing nested replacement fields are evaluated eagerly,
prior to being passed to the :class:`~string.templatelib.Interpolation` object.
For instance, an interpolation of the form ``{amount:.{precision}f}`` will
evaluate the inner expression ``{precision}`` to determine the value of the
``format_spec`` attribute.
If ``precision`` were to be ``2``, the resulting format specifier
would be ``'.2f'``.
F-strings are handled partly by the :term:`lexical analyzer`, which produces the
tokens :py:data:`~token.FSTRING_START`, :py:data:`~token.FSTRING_MIDDLE`
and :py:data:`~token.FSTRING_END`, and partly by the parser, which handles
expressions in the replacement field.
The exact way the work is split is a CPython implementation detail.
* When the equals sign ``'='`` is provided in an interpolation expression,
the text of the expression is appended to the literal string that precedes
the relevant interpolation.
This includes the equals sign and any surrounding whitespace.
The :class:`!Interpolation` instance for the expression will be created as
normal, except that :attr:`~string.templatelib.Interpolation.conversion` will
be set to '``r``' (:func:`repr`) by default.
If an explicit conversion or format specifier are provided,
this will override the default behaviour.
Correspondingly, the f-string grammar is a mix of
:ref:`lexical and syntactic definitions <notation-lexical-vs-syntactic>`.
Whitespace is significant in these situations:
* There may be no whitespace in :py:data:`~token.FSTRING_START` (between
the prefix and quote).
* Whitespace in :py:data:`~token.FSTRING_MIDDLE` is part of the literal
string contents.
* In ``fstring_replacement_field``, if ``f_debug_specifier`` is present,
all whitespace after the opening brace until the ``f_debug_specifier``,
as well as whitespace immediatelly following ``f_debug_specifier``,
is retained as part of the expression.
.. impl-detail::
The expression is not handled in the tokenization phase; it is
retrieved from the source code using locations of the ``{`` token
and the token after ``=``.
The ``FSTRING_MIDDLE`` definition uses
:ref:`negative lookaheads <lexical-lookaheads>` (``!``)
to indicate special characters (backslash, newline, ``{``, ``}``) and
sequences (``f_quote``).
.. grammar-snippet::
:group: python-grammar
fstring: `FSTRING_START` `fstring_middle`* `FSTRING_END`
FSTRING_START: `fstringprefix` ("'" | '"' | "'''" | '"""')
FSTRING_END: `f_quote`
fstringprefix: <("f" | "fr" | "rf"), case-insensitive>
f_debug_specifier: '='
f_quote: <the quote character(s) used in FSTRING_START>
fstring_middle:
| `fstring_replacement_field`
| `FSTRING_MIDDLE`
FSTRING_MIDDLE:
| (!"\" !`newline` !'{' !'}' !`f_quote`) `source_character`
| `stringescapeseq`
| "{{"
| "}}"
| <newline, in triple-quoted f-strings only>
fstring_replacement_field:
| '{' `f_expression` [`f_debug_specifier`] [`fstring_conversion`]
[`fstring_full_format_spec`] '}'
fstring_conversion:
| "!" ("s" | "r" | "a")
fstring_full_format_spec:
| ':' `fstring_format_spec`*
fstring_format_spec:
| `FSTRING_MIDDLE`
| `fstring_replacement_field`
f_expression:
| ','.(`conditional_expression` | "*" `or_expr`)+ [","]
| `yield_expression`
.. note::
In the above grammar snippet, the ``f_quote`` and ``FSTRING_MIDDLE`` rules
are context-sensitive -- they depend on the contents of ``FSTRING_START``
of the nearest enclosing ``fstring``.
Constructing a more traditional formal grammar from this template is left
as an exercise for the reader.
The grammar for t-strings is identical to the one for f-strings, with *t*
instead of *f* at the beginning of rule and token names and in the prefix.
.. grammar-snippet::
:group: python-grammar
tstring: TSTRING_START tstring_middle* TSTRING_END
<rest of the t-string grammar is omitted; see above>
.. _numbers:

4
Doc/whatsnew/2.3.rst
View file

@ -66,7 +66,7 @@ Here's a simple example::
The union and intersection of sets can be computed with the :meth:`~frozenset.union` and
:meth:`~frozenset.intersection` methods; an alternative notation uses the bitwise operators
``&`` and ``|``. Mutable sets also have in-place versions of these methods,
:meth:`!union_update` and :meth:`~frozenset.intersection_update`. ::
:meth:`!union_update` and :meth:`~set.intersection_update`. ::
>>> S1 = sets.Set([1,2,3])
>>> S2 = sets.Set([4,5,6])
@ -87,7 +87,7 @@ It's also possible to take the symmetric difference of two sets. This is the
set of all elements in the union that aren't in the intersection. Another way
of putting it is that the symmetric difference contains all elements that are in
exactly one set. Again, there's an alternative notation (``^``), and an
in-place version with the ungainly name :meth:`~frozenset.symmetric_difference_update`. ::
in-place version with the ungainly name :meth:`~set.symmetric_difference_update`. ::
>>> S1 = sets.Set([1,2,3,4])
>>> S2 = sets.Set([3,4,5,6])

151
Doc/whatsnew/3.15.rst
View file

@ -66,7 +66,7 @@ Summary -- Release highlights
.. PEP-sized items next.
* :pep:`799`: :ref:`A dedicated profiling package for organizing Python
profiling tools <whatsnew315-sampling-profiler>`
profiling tools <whatsnew315-profiling-package>`
* :pep:`686`: :ref:`Python now uses UTF-8 as the default encoding
<whatsnew315-utf8-default>`
* :pep:`782`: :ref:`A new PyBytesWriter C API to create a Python bytes object
@ -77,12 +77,32 @@ Summary -- Release highlights
New features
============
.. _whatsnew315-profiling-package:
:pep:`799`: A dedicated profiling package
-----------------------------------------
A new :mod:`!profiling` module has been added to organize Python's built-in
profiling tools under a single, coherent namespace. This module contains:
* :mod:`!profiling.tracing`: deterministic function-call tracing (relocated from
:mod:`cProfile`).
* :mod:`!profiling.sampling`: a new statistical sampling profiler (named Tachyon).
The :mod:`cProfile` module remains as an alias for backwards compatibility.
The :mod:`profile` module is deprecated and will be removed in Python 3.17.
.. seealso:: :pep:`799` for further details.
(Contributed by Pablo Galindo and László Kiss Kollár in :gh:`138122`.)
.. _whatsnew315-sampling-profiler:
:pep:`799`: High frequency statistical sampling profiler
--------------------------------------------------------
Tachyon: High frequency statistical sampling profiler
-----------------------------------------------------
A new statistical sampling profiler has been added to the new :mod:`!profiling` module as
A new statistical sampling profiler (Tachyon) has been added as
:mod:`!profiling.sampling`. This profiler enables low-overhead performance analysis of
running Python processes without requiring code modification or process restart.
@ -91,101 +111,64 @@ every function call, the sampling profiler periodically captures stack traces fr
running processes. This approach provides virtually zero overhead while achieving
sampling rates of **up to 1,000,000 Hz**, making it the fastest sampling profiler
available for Python (at the time of its contribution) and ideal for debugging
performance issues in production environments.
performance issues in production environments. This capability is particularly
valuable for debugging performance issues in production systems where traditional
profiling approaches would be too intrusive.
Key features include:
* **Zero-overhead profiling**: Attach to any running Python process without
affecting its performance
* **No code modification required**: Profile existing applications without restart
* **Real-time statistics**: Monitor sampling quality during data collection
* **Multiple output formats**: Generate both detailed statistics and flamegraph data
* **Thread-aware profiling**: Option to profile all threads or just the main thread
affecting its performance. Ideal for production debugging where you can't afford
to restart or slow down your application.
Profile process 1234 for 10 seconds with default settings:
* **No code modification required**: Profile existing applications without restart.
Simply point the profiler at a running process by PID and start collecting data.
.. code-block:: shell
* **Flexible target modes**:
python -m profiling.sampling 1234
* Profile running processes by PID (``attach``) - attach to already-running applications
* Run and profile scripts directly (``run``) - profile from the very start of execution
* Execute and profile modules (``run -m``) - profile packages run as ``python -m module``
Profile with custom interval and duration, save to file:
* **Multiple profiling modes**: Choose what to measure based on your performance investigation:
.. code-block:: shell
* **Wall-clock time** (``--mode wall``, default): Measures real elapsed time including I/O,
network waits, and blocking operations. Use this to understand where your program spends
calendar time, including when waiting for external resources.
* **CPU time** (``--mode cpu``): Measures only active CPU execution time, excluding I/O waits
and blocking. Use this to identify CPU-bound bottlenecks and optimize computational work.
* **GIL-holding time** (``--mode gil``): Measures time spent holding Python's Global Interpreter
Lock. Use this to identify which threads dominate GIL usage in multi-threaded applications.
python -m profiling.sampling -i 50 -d 30 -o profile.stats 1234
* **Thread-aware profiling**: Option to profile all threads (``-a``) or just the main thread,
essential for understanding multi-threaded application behavior.
Generate collapsed stacks for flamegraph:
* **Multiple output formats**: Choose the visualization that best fits your workflow:
.. code-block:: shell
* ``--pstats``: Detailed tabular statistics compatible with :mod:`pstats`. Shows function-level
timing with direct and cumulative samples. Best for detailed analysis and integration with
existing Python profiling tools.
* ``--collapsed``: Generates collapsed stack traces (one line per stack). This format is
specifically designed for creating flamegraphs with external tools like Brendan Gregg's
FlameGraph scripts or speedscope.
* ``--flamegraph``: Generates a self-contained interactive HTML flamegraph using D3.js.
Opens directly in your browser for immediate visual analysis. Flamegraphs show the call
hierarchy where width represents time spent, making it easy to spot bottlenecks at a glance.
* ``--gecko``: Generates Gecko Profiler format compatible with Firefox Profiler
(https://profiler.firefox.com). Upload the output to Firefox Profiler for advanced
timeline-based analysis with features like stack charts, markers, and network activity.
* ``--heatmap``: Generates an interactive HTML heatmap visualization with line-level sample
counts. Creates a directory with per-file heatmaps showing exactly where time is spent
at the source code level.
python -m profiling.sampling --collapsed 1234
* **Live interactive mode**: Real-time TUI profiler with a top-like interface (``--live``).
Monitor performance as your application runs with interactive sorting and filtering.
Profile all threads and sort by total time:
* **Async-aware profiling**: Profile async/await code with task-based stack reconstruction
(``--async-aware``). See which coroutines are consuming time, with options to show only
running tasks or all tasks including those waiting.
.. code-block:: shell
python -m profiling.sampling -a --sort-tottime 1234
The profiler generates statistical estimates of where time is spent:
.. code-block:: text
Real-time sampling stats: Mean: 100261.5Hz (9.97µs) Min: 86333.4Hz (11.58µs) Max: 118807.2Hz (8.42µs) Samples: 400001
Captured 498841 samples in 5.00 seconds
Sample rate: 99768.04 samples/sec
Error rate: 0.72%
Profile Stats:
nsamples sample% tottime (s) cumul% cumtime (s) filename:lineno(function)
43/418858 0.0 0.000 87.9 4.189 case.py:667(TestCase.run)
3293/418812 0.7 0.033 87.9 4.188 case.py:613(TestCase._callTestMethod)
158562/158562 33.3 1.586 33.3 1.586 test_compile.py:725(TestSpecifics.test_compiler_recursion_limit.<locals>.check_limit)
129553/129553 27.2 1.296 27.2 1.296 ast.py:46(parse)
0/128129 0.0 0.000 26.9 1.281 test_ast.py:884(AST_Tests.test_ast_recursion_limit.<locals>.check_limit)
7/67446 0.0 0.000 14.2 0.674 test_compile.py:729(TestSpecifics.test_compiler_recursion_limit)
6/60380 0.0 0.000 12.7 0.604 test_ast.py:888(AST_Tests.test_ast_recursion_limit)
3/50020 0.0 0.000 10.5 0.500 test_compile.py:727(TestSpecifics.test_compiler_recursion_limit)
1/38011 0.0 0.000 8.0 0.380 test_ast.py:886(AST_Tests.test_ast_recursion_limit)
1/25076 0.0 0.000 5.3 0.251 test_compile.py:728(TestSpecifics.test_compiler_recursion_limit)
22361/22362 4.7 0.224 4.7 0.224 test_compile.py:1368(TestSpecifics.test_big_dict_literal)
4/18008 0.0 0.000 3.8 0.180 test_ast.py:889(AST_Tests.test_ast_recursion_limit)
11/17696 0.0 0.000 3.7 0.177 subprocess.py:1038(Popen.__init__)
16968/16968 3.6 0.170 3.6 0.170 subprocess.py:1900(Popen._execute_child)
2/16941 0.0 0.000 3.6 0.169 test_compile.py:730(TestSpecifics.test_compiler_recursion_limit)
Legend:
nsamples: Direct/Cumulative samples (direct executing / on call stack)
sample%: Percentage of total samples this function was directly executing
tottime: Estimated total time spent directly in this function
cumul%: Percentage of total samples when this function was on the call stack
cumtime: Estimated cumulative time (including time in called functions)
filename:lineno(function): Function location and name
Summary of Interesting Functions:
Functions with Highest Direct/Cumulative Ratio (Hot Spots):
1.000 direct/cumulative ratio, 33.3% direct samples: test_compile.py:(TestSpecifics.test_compiler_recursion_limit.<locals>.check_limit)
1.000 direct/cumulative ratio, 27.2% direct samples: ast.py:(parse)
1.000 direct/cumulative ratio, 3.6% direct samples: subprocess.py:(Popen._execute_child)
Functions with Highest Call Frequency (Indirect Calls):
418815 indirect calls, 87.9% total stack presence: case.py:(TestCase.run)
415519 indirect calls, 87.9% total stack presence: case.py:(TestCase._callTestMethod)
159470 indirect calls, 33.5% total stack presence: test_compile.py:(TestSpecifics.test_compiler_recursion_limit)
Functions with Highest Call Magnification (Cumulative/Direct):
12267.9x call magnification, 159470 indirect calls from 13 direct: test_compile.py:(TestSpecifics.test_compiler_recursion_limit)
10581.7x call magnification, 116388 indirect calls from 11 direct: test_ast.py:(AST_Tests.test_ast_recursion_limit)
9740.9x call magnification, 418815 indirect calls from 43 direct: case.py:(TestCase.run)
The profiler automatically identifies performance bottlenecks through statistical
analysis, highlighting functions with high CPU usage and call frequency patterns.
This capability is particularly valuable for debugging performance issues in
production systems where traditional profiling approaches would be too intrusive.
.. seealso:: :pep:`799` for further details.
(Contributed by Pablo Galindo and László Kiss Kollár in :gh:`135953`.)
(Contributed by Pablo Galindo and László Kiss Kollár in :gh:`135953` and :gh:`138122`.)
.. _whatsnew315-improved-error-messages:

2
Include/cpython/pystate.h
View file

@ -135,6 +135,8 @@ struct _ts {
/* Pointer to currently executing frame. */
struct _PyInterpreterFrame *current_frame;
struct _PyInterpreterFrame *last_profiled_frame;
Py_tracefunc c_profilefunc;
Py_tracefunc c_tracefunc;
PyObject *c_profileobj;

33
Include/internal/pycore_call.h
View file

@ -64,39 +64,6 @@ PyAPI_FUNC(PyObject*) _PyObject_CallMethod(
PyObject *name,
const char *format, ...);
extern PyObject* _PyObject_CallMethodIdObjArgs(
PyObject *obj,
_Py_Identifier *name,
...);
static inline PyObject *
_PyObject_VectorcallMethodId(
_Py_Identifier *name, PyObject *const *args,
size_t nargsf, PyObject *kwnames)
{
PyObject *oname = _PyUnicode_FromId(name); /* borrowed */
if (!oname) {
return _Py_NULL;
}
return PyObject_VectorcallMethod(oname, args, nargsf, kwnames);
}
static inline PyObject *
_PyObject_CallMethodIdNoArgs(PyObject *self, _Py_Identifier *name)
{
size_t nargsf = 1 | PY_VECTORCALL_ARGUMENTS_OFFSET;
return _PyObject_VectorcallMethodId(name, &self, nargsf, _Py_NULL);
}
static inline PyObject *
_PyObject_CallMethodIdOneArg(PyObject *self, _Py_Identifier *name, PyObject *arg)
{
PyObject *args[2] = {self, arg};
size_t nargsf = 2 | PY_VECTORCALL_ARGUMENTS_OFFSET;
assert(arg != NULL);
return _PyObject_VectorcallMethodId(name, args, nargsf, _Py_NULL);
}
/* === Vectorcall protocol (PEP 590) ============================= */

60
Include/internal/pycore_ceval.h
View file

@ -33,8 +33,6 @@ extern int _PyEval_SetOpcodeTrace(PyFrameObject *f, bool enable);
// Export for 'array' shared extension
PyAPI_FUNC(PyObject*) _PyEval_GetBuiltin(PyObject *);
extern PyObject* _PyEval_GetBuiltinId(_Py_Identifier *);
extern void _PyEval_SetSwitchInterval(unsigned long microseconds);
extern unsigned long _PyEval_GetSwitchInterval(void);
@ -410,6 +408,64 @@ _PyForIter_VirtualIteratorNext(PyThreadState* tstate, struct _PyInterpreterFrame
PyAPI_DATA(const _Py_CODEUNIT *) _Py_INTERPRETER_TRAMPOLINE_INSTRUCTIONS_PTR;
/* Helper functions for large uops */
PyAPI_FUNC(PyObject *)
_Py_VectorCall_StackRefSteal(
_PyStackRef callable,
_PyStackRef *arguments,
int total_args,
_PyStackRef kwnames);
PyAPI_FUNC(PyObject *)
_Py_BuiltinCallFast_StackRefSteal(
_PyStackRef callable,
_PyStackRef *arguments,
int total_args);
PyAPI_FUNC(PyObject *)
_Py_BuiltinCallFastWithKeywords_StackRefSteal(
_PyStackRef callable,
_PyStackRef *arguments,
int total_args);
PyAPI_FUNC(PyObject *)
_PyCallMethodDescriptorFast_StackRefSteal(
_PyStackRef callable,
PyMethodDef *meth,
PyObject *self,
_PyStackRef *arguments,
int total_args);
PyAPI_FUNC(PyObject *)
_PyCallMethodDescriptorFastWithKeywords_StackRefSteal(
_PyStackRef callable,
PyMethodDef *meth,
PyObject *self,
_PyStackRef *arguments,
int total_args);
PyAPI_FUNC(PyObject *)
_Py_CallBuiltinClass_StackRefSteal(
_PyStackRef callable,
_PyStackRef *arguments,
int total_args);
PyAPI_FUNC(PyObject *)
_Py_BuildString_StackRefSteal(
_PyStackRef *arguments,
int total_args);
PyAPI_FUNC(PyObject *)
_Py_BuildMap_StackRefSteal(
_PyStackRef *arguments,
int half_args);
PyAPI_FUNC(void)
_Py_assert_within_stack_bounds(
_PyInterpreterFrame *frame, _PyStackRef *stack_pointer,
const char *filename, int lineno);
#ifdef __cplusplus
}
#endif

2
Include/internal/pycore_debug_offsets.h
View file

@ -102,6 +102,7 @@ typedef struct _Py_DebugOffsets {
uint64_t next;
uint64_t interp;
uint64_t current_frame;
uint64_t last_profiled_frame;
uint64_t thread_id;
uint64_t native_thread_id;
uint64_t datastack_chunk;
@ -272,6 +273,7 @@ typedef struct _Py_DebugOffsets {
.next = offsetof(PyThreadState, next), \
.interp = offsetof(PyThreadState, interp), \
.current_frame = offsetof(PyThreadState, current_frame), \
.last_profiled_frame = offsetof(PyThreadState, last_profiled_frame), \
.thread_id = offsetof(PyThreadState, thread_id), \
.native_thread_id = offsetof(PyThreadState, native_thread_id), \
.datastack_chunk = offsetof(PyThreadState, datastack_chunk), \

7
Include/internal/pycore_dict.h
View file

@ -36,13 +36,6 @@ extern int _PyDict_DelItem_KnownHash_LockHeld(PyObject *mp, PyObject *key,
extern int _PyDict_Contains_KnownHash(PyObject *, PyObject *, Py_hash_t);
// "Id" variants
extern PyObject* _PyDict_GetItemIdWithError(PyObject *dp,
_Py_Identifier *key);
extern int _PyDict_ContainsId(PyObject *, _Py_Identifier *);
extern int _PyDict_SetItemId(PyObject *dp, _Py_Identifier *key, PyObject *item);
extern int _PyDict_DelItemId(PyObject *mp, _Py_Identifier *key);
extern int _PyDict_Next(
PyObject *mp, Py_ssize_t *pos, PyObject **key, PyObject **value, Py_hash_t *hash);

2
Include/internal/pycore_global_objects_fini_generated.h generated
View file

@ -1609,6 +1609,7 @@ _PyStaticObjects_CheckRefcnt(PyInterpreterState *interp) {
_PyStaticObject_CheckRefcnt((PyObject *)&_Py_ID(c_parameter_type));
_PyStaticObject_CheckRefcnt((PyObject *)&_Py_ID(c_return));
_PyStaticObject_CheckRefcnt((PyObject *)&_Py_ID(c_stack));
_PyStaticObject_CheckRefcnt((PyObject *)&_Py_ID(cache_frames));
_PyStaticObject_CheckRefcnt((PyObject *)&_Py_ID(cached_datetime_module));
_PyStaticObject_CheckRefcnt((PyObject *)&_Py_ID(cached_statements));
_PyStaticObject_CheckRefcnt((PyObject *)&_Py_ID(cadata));
@ -2053,6 +2054,7 @@ _PyStaticObjects_CheckRefcnt(PyInterpreterState *interp) {
_PyStaticObject_CheckRefcnt((PyObject *)&_Py_ID(stacklevel));
_PyStaticObject_CheckRefcnt((PyObject *)&_Py_ID(start));
_PyStaticObject_CheckRefcnt((PyObject *)&_Py_ID(statement));
_PyStaticObject_CheckRefcnt((PyObject *)&_Py_ID(stats));
_PyStaticObject_CheckRefcnt((PyObject *)&_Py_ID(status));
_PyStaticObject_CheckRefcnt((PyObject *)&_Py_ID(stderr));
_PyStaticObject_CheckRefcnt((PyObject *)&_Py_ID(stdin));

2
Include/internal/pycore_global_strings.h
View file

@ -332,6 +332,7 @@ struct _Py_global_strings {
STRUCT_FOR_ID(c_parameter_type)
STRUCT_FOR_ID(c_return)
STRUCT_FOR_ID(c_stack)
STRUCT_FOR_ID(cache_frames)
STRUCT_FOR_ID(cached_datetime_module)
STRUCT_FOR_ID(cached_statements)
STRUCT_FOR_ID(cadata)
@ -776,6 +777,7 @@ struct _Py_global_strings {
STRUCT_FOR_ID(stacklevel)
STRUCT_FOR_ID(start)
STRUCT_FOR_ID(statement)
STRUCT_FOR_ID(stats)
STRUCT_FOR_ID(status)
STRUCT_FOR_ID(stderr)
STRUCT_FOR_ID(stdin)

3
Include/internal/pycore_jit.h
View file

@ -13,6 +13,9 @@ extern "C" {
# error "this header requires Py_BUILD_CORE define"
#endif
/* To be able to reason about code layout and branches, keep code size below 1 MB */
#define PY_MAX_JIT_CODE_SIZE ((1 << 20)-1)
#ifdef _Py_JIT
typedef _Py_CODEUNIT *(*jit_func)(_PyInterpreterFrame *frame, _PyStackRef *stack_pointer, PyThreadState *tstate);

2
Include/internal/pycore_runtime_init_generated.h generated
View file

@ -1607,6 +1607,7 @@ extern "C" {
INIT_ID(c_parameter_type), \
INIT_ID(c_return), \
INIT_ID(c_stack), \
INIT_ID(cache_frames), \
INIT_ID(cached_datetime_module), \
INIT_ID(cached_statements), \
INIT_ID(cadata), \
@ -2051,6 +2052,7 @@ extern "C" {
INIT_ID(stacklevel), \
INIT_ID(start), \
INIT_ID(statement), \
INIT_ID(stats), \
INIT_ID(status), \
INIT_ID(stderr), \
INIT_ID(stdin), \

8
Include/internal/pycore_unicodeobject.h
View file

@ -307,14 +307,6 @@ PyAPI_FUNC(PyObject*) _PyUnicode_JoinArray(
Py_ssize_t seqlen
);
/* Test whether a unicode is equal to ASCII identifier. Return 1 if true,
0 otherwise. The right argument must be ASCII identifier.
Any error occurs inside will be cleared before return. */
extern int _PyUnicode_EqualToASCIIId(
PyObject *left, /* Left string */
_Py_Identifier *right /* Right identifier */
);
// Test whether a unicode is equal to ASCII string. Return 1 if true,
// 0 otherwise. The right argument must be ASCII-encoded string.
// Any error occurs inside will be cleared before return.

8
Include/internal/pycore_unicodeobject_generated.h generated
View file

@ -1108,6 +1108,10 @@ _PyUnicode_InitStaticStrings(PyInterpreterState *interp) {
_PyUnicode_InternStatic(interp, &string);
assert(_PyUnicode_CheckConsistency(string, 1));
assert(PyUnicode_GET_LENGTH(string) != 1);
string = &_Py_ID(cache_frames);
_PyUnicode_InternStatic(interp, &string);
assert(_PyUnicode_CheckConsistency(string, 1));
assert(PyUnicode_GET_LENGTH(string) != 1);
string = &_Py_ID(cached_datetime_module);
_PyUnicode_InternStatic(interp, &string);
assert(_PyUnicode_CheckConsistency(string, 1));
@ -2884,6 +2888,10 @@ _PyUnicode_InitStaticStrings(PyInterpreterState *interp) {
_PyUnicode_InternStatic(interp, &string);
assert(_PyUnicode_CheckConsistency(string, 1));
assert(PyUnicode_GET_LENGTH(string) != 1);
string = &_Py_ID(stats);
_PyUnicode_InternStatic(interp, &string);
assert(_PyUnicode_CheckConsistency(string, 1));
assert(PyUnicode_GET_LENGTH(string) != 1);
string = &_Py_ID(status);
_PyUnicode_InternStatic(interp, &string);
assert(_PyUnicode_CheckConsistency(string, 1));

5
Include/internal/pycore_uop.h
View file

@ -36,7 +36,12 @@ typedef struct _PyUOpInstruction{
} _PyUOpInstruction;
// This is the length of the trace we translate initially.
#ifdef Py_DEBUG
// With asserts, the stencils are a lot larger
#define UOP_MAX_TRACE_LENGTH 1000
#else
#define UOP_MAX_TRACE_LENGTH 3000
#endif
#define UOP_BUFFER_SIZE (UOP_MAX_TRACE_LENGTH * sizeof(_PyUOpInstruction))
/* Bloom filter with m = 256

2
Include/modsupport.h
View file

@ -132,7 +132,7 @@ PyAPI_FUNC(int) PyABIInfo_Check(PyABIInfo *info, const char *module_name);
) \
/////////////////////////////////////////////////////////
#define _PyABIInfo_DEFAULT() { \
#define _PyABIInfo_DEFAULT { \
1, 0, \
PyABIInfo_DEFAULT_FLAGS, \
PY_VERSION_HEX, \

6
Include/pymacro.h
View file

@ -116,6 +116,12 @@
/* Absolute value of the number x */
#define Py_ABS(x) ((x) < 0 ? -(x) : (x))
/* Safer implementation that avoids an undefined behavior for the minimal
value of the signed integer type if its absolute value is larger than
the maximal value of the signed integer type (in the two's complement
representations, which is common).
*/
#define _Py_ABS_CAST(T, x) ((x) >= 0 ? ((T) (x)) : ((T) (((T) -((x) + 1)) + 1u)))
#define _Py_XSTRINGIFY(x) #x

20
InternalDocs/frames.md
View file

@ -111,6 +111,26 @@ ### Shim frames
instruction which cleans up the shim frame and returns.
### Remote Profiling Frame Cache
The `last_profiled_frame` field in `PyThreadState` supports an optimization for
remote profilers that sample call stacks from external processes. When a remote
profiler reads the call stack, it writes the current frame address to this field.
The eval loop then keeps this pointer valid by updating it to the parent frame
whenever a frame returns (in `_PyEval_FrameClearAndPop`).
This creates a "high-water mark" that always points to a frame still on the stack.
On subsequent samples, the profiler can walk from `current_frame` until it reaches
`last_profiled_frame`, knowing that frames from that point downward are unchanged
and can be retrieved from a cache. This significantly reduces the amount of remote
memory reads needed when call stacks are deep and stable at their base.
The update in `_PyEval_FrameClearAndPop` is guarded: it only writes when
`last_profiled_frame` is non-NULL AND matches the frame being popped. This
prevents transient frames (called and returned between profiler samples) from
corrupting the cache pointer, while avoiding any overhead when profiling is inactive.
### The Instruction Pointer
`_PyInterpreterFrame` has two fields which are used to maintain the instruction

67
Lib/argparse.py
View file

@ -166,7 +166,6 @@ def __init__(
indent_increment=2,
max_help_position=24,
width=None,
color=True,
):
# default setting for width
if width is None:
@ -174,7 +173,6 @@ def __init__(
width = shutil.get_terminal_size().columns
width -= 2
self._set_color(color)
self._prog = prog
self._indent_increment = indent_increment
self._max_help_position = min(max_help_position,
@ -355,8 +353,14 @@ def _format_usage(self, usage, actions, groups, prefix):
if len(prefix) + len(self._decolor(usage)) > text_width:
# break usage into wrappable parts
opt_parts = self._get_actions_usage_parts(optionals, groups)
pos_parts = self._get_actions_usage_parts(positionals, groups)
# keep optionals and positionals together to preserve
# mutually exclusive group formatting (gh-75949)
all_actions = optionals + positionals
parts, pos_start = self._get_actions_usage_parts_with_split(
all_actions, groups, len(optionals)
)
opt_parts = parts[:pos_start]
pos_parts = parts[pos_start:]
# helper for wrapping lines
def get_lines(parts, indent, prefix=None):
@ -420,6 +424,17 @@ def _is_long_option(self, string):
return len(string) > 2
def _get_actions_usage_parts(self, actions, groups):
parts, _ = self._get_actions_usage_parts_with_split(actions, groups)
return parts
def _get_actions_usage_parts_with_split(self, actions, groups, opt_count=None):
"""Get usage parts with split index for optionals/positionals.
Returns (parts, pos_start) where pos_start is the index in parts
where positionals begin. When opt_count is None, pos_start is None.
This preserves mutually exclusive group formatting across the
optionals/positionals boundary (gh-75949).
"""
# find group indices and identify actions in groups
group_actions = set()
inserts = {}
@ -452,16 +467,12 @@ def _get_actions_usage_parts(self, actions, groups):
# produce all arg strings
elif not action.option_strings:
default = self._get_default_metavar_for_positional(action)
part = (
t.summary_action
+ self._format_args(action, default)
+ t.reset
)
part = self._format_args(action, default)
# if it's in a group, strip the outer []
if action in group_actions:
if part[0] == '[' and part[-1] == ']':
part = part[1:-1]
part = t.summary_action + part + t.reset
# produce the first way to invoke the option in brackets
else:
@ -515,8 +526,16 @@ def _get_actions_usage_parts(self, actions, groups):
for i in range(start + group_size, end):
parts[i] = None
# return the usage parts
return [item for item in parts if item is not None]
# if opt_count is provided, calculate where positionals start in
# the final parts list (for wrapping onto separate lines).
# Count before filtering None entries since indices shift after.
if opt_count is not None:
pos_start = sum(1 for p in parts[:opt_count] if p is not None)
else:
pos_start = None
# return the usage parts and split point (gh-75949)
return [item for item in parts if item is not None], pos_start
def _format_text(self, text):
if '%(prog)' in text:
@ -1570,8 +1589,8 @@ def add_argument(self, *args, **kwargs):
f'instance of it must be passed')
# raise an error if the metavar does not match the type
if hasattr(self, "_get_formatter"):
formatter = self._get_formatter()
if hasattr(self, "_get_validation_formatter"):
formatter = self._get_validation_formatter()
try:
formatter._format_args(action, None)
except TypeError:
@ -1765,8 +1784,8 @@ def _handle_conflict_resolve(self, action, conflicting_actions):
action.container._remove_action(action)
def _check_help(self, action):
if action.help and hasattr(self, "_get_formatter"):
formatter = self._get_formatter()
if action.help and hasattr(self, "_get_validation_formatter"):
formatter = self._get_validation_formatter()
try:
formatter._expand_help(action)
except (ValueError, TypeError, KeyError) as exc:
@ -1921,6 +1940,9 @@ def __init__(self,
self.suggest_on_error = suggest_on_error
self.color = color
# Cached formatter for validation (avoids repeated _set_color calls)
self._cached_formatter = None
add_group = self.add_argument_group
self._positionals = add_group(_('positional arguments'))
self._optionals = add_group(_('options'))
@ -1982,14 +2004,16 @@ def add_subparsers(self, **kwargs):
self._subparsers = self._positionals
# prog defaults to the usage message of this parser, skipping
# optional arguments and with no "usage:" prefix
# non-required optional arguments and with no "usage:" prefix
if kwargs.get('prog') is None:
# Create formatter without color to avoid storing ANSI codes in prog
formatter = self.formatter_class(prog=self.prog)
formatter._set_color(False)
positionals = self._get_positional_actions()
required_optionals = [action for action in self._get_optional_actions()
if action.required]
groups = self._mutually_exclusive_groups
formatter.add_usage(None, positionals, groups, '')
formatter.add_usage(None, required_optionals + positionals, groups, '')
kwargs['prog'] = formatter.format_help().strip()
# create the parsers action and add it to the positionals list
@ -2752,6 +2776,13 @@ def _get_formatter(self):
formatter._set_color(self.color)
return formatter
def _get_validation_formatter(self):
# Return cached formatter for read-only validation operations
# (_expand_help and _format_args). Avoids repeated slow _set_color calls.
if self._cached_formatter is None:
self._cached_formatter = self._get_formatter()
return self._cached_formatter
# =====================
# Help-printing methods
# =====================

4
Lib/asyncio/futures.py
View file

@ -389,7 +389,7 @@ def _set_state(future, other):
def _call_check_cancel(destination):
if destination.cancelled():
if source_loop is None or source_loop is dest_loop:
if source_loop is None or source_loop is events._get_running_loop():
source.cancel()
else:
source_loop.call_soon_threadsafe(source.cancel)
@ -398,7 +398,7 @@ def _call_set_state(source):
if (destination.cancelled() and
dest_loop is not None and dest_loop.is_closed()):
return
if dest_loop is None or dest_loop is source_loop:
if dest_loop is None or dest_loop is events._get_running_loop():
_set_state(destination, source)
else:
if dest_loop.is_closed():

1
Lib/cProfile.py
View file

@ -9,6 +9,5 @@
__all__ = ["run", "runctx", "Profile"]
if __name__ == "__main__":
import sys
from profiling.tracing.__main__ import main
main()

14
Lib/dataclasses.py
View file

@ -550,7 +550,12 @@ def __annotate__(format, /):
new_annotations = {}
for k in annotation_fields:
new_annotations[k] = cls_annotations[k]
# gh-142214: The annotation may be missing in unusual dynamic cases.
# If so, just skip it.
try:
new_annotations[k] = cls_annotations[k]
except KeyError:
pass
if return_type is not MISSING:
if format == Format.STRING:
@ -1399,9 +1404,10 @@ def _add_slots(cls, is_frozen, weakref_slot, defined_fields):
f.type = ann
# Fix the class reference in the __annotate__ method
init_annotate = newcls.__init__.__annotate__
if getattr(init_annotate, "__generated_by_dataclasses__", False):
_update_func_cell_for__class__(init_annotate, cls, newcls)
init = newcls.__init__
if init_annotate := getattr(init, "__annotate__", None):
if getattr(init_annotate, "__generated_by_dataclasses__", False):
_update_func_cell_for__class__(init_annotate, cls, newcls)
return newcls

26
Lib/doctest.py
View file

@ -1167,6 +1167,32 @@ def _find_lineno(self, obj, source_lines):
if pat.match(source_lines[lineno]):
return lineno
# Handle __test__ string doctests formatted as triple-quoted
# strings. Find a non-blank line in the test string and match it
# in the source, verifying subsequent lines also match to handle
# duplicate lines.
if isinstance(obj, str) and source_lines is not None:
obj_lines = obj.splitlines(keepends=True)
# Skip the first line (may be on same line as opening quotes)
# and any blank lines to find a meaningful line to match.
start_index = 1
while (start_index < len(obj_lines)
and not obj_lines[start_index].strip()):
start_index += 1
if start_index < len(obj_lines):
target_line = obj_lines[start_index]
for lineno, source_line in enumerate(source_lines):
if source_line == target_line:
# Verify subsequent lines also match
for i in range(start_index + 1, len(obj_lines) - 1):
source_idx = lineno + i - start_index
if source_idx >= len(source_lines):
break
if obj_lines[i] != source_lines[source_idx]:
break
else:
return lineno - start_index
# We couldn't find the line number.
return None

3
Lib/email/_header_value_parser.py
View file

@ -2792,6 +2792,9 @@ def _steal_trailing_WSP_if_exists(lines):
if lines and lines[-1] and lines[-1][-1] in WSP:
wsp = lines[-1][-1]
lines[-1] = lines[-1][:-1]
# gh-142006: if the line is now empty, remove it entirely.
if not lines[-1]:
lines.pop()
return wsp
def _refold_parse_tree(parse_tree, *, policy):

7
Lib/email/feedparser.py
View file

@ -504,10 +504,9 @@ def _parse_headers(self, lines):
self._input.unreadline(line)
return
else:
# Weirdly placed unix-from line. Note this as a defect
# and ignore it.
# Weirdly placed unix-from line.
defect = errors.MisplacedEnvelopeHeaderDefect(line)
self._cur.defects.append(defect)
self.policy.handle_defect(self._cur, defect)
continue
# Split the line on the colon separating field name from value.
# There will always be a colon, because if there wasn't the part of
@ -519,7 +518,7 @@ def _parse_headers(self, lines):
# message. Track the error but keep going.
if i == 0:
defect = errors.InvalidHeaderDefect("Missing header name.")
self._cur.defects.append(defect)
self.policy.handle_defect(self._cur, defect)
continue
assert i>0, "_parse_headers fed line with no : and no leading WS"

63
Lib/multiprocessing/resource_tracker.py
View file

@ -68,6 +68,13 @@ def __init__(self):
self._exitcode = None
self._reentrant_messages = deque()
# True to use colon-separated lines, rather than JSON lines,
# for internal communication. (Mainly for testing).
# Filenames not supported by the simple format will always be sent
# using JSON.
# The reader should understand all formats.
self._use_simple_format = False
def _reentrant_call_error(self):
# gh-109629: this happens if an explicit call to the ResourceTracker
# gets interrupted by a garbage collection, invoking a finalizer (*)
@ -200,7 +207,9 @@ def _launch(self):
os.close(r)
def _make_probe_message(self):
"""Return a JSON-encoded probe message."""
"""Return a probe message."""
if self._use_simple_format:
return b'PROBE:0:noop\n'
return (
json.dumps(
{"cmd": "PROBE", "rtype": "noop"},
@ -267,6 +276,15 @@ def _write(self, msg):
assert nbytes == len(msg), f"{nbytes=} != {len(msg)=}"
def _send(self, cmd, name, rtype):
if self._use_simple_format and '\n' not in name:
msg = f"{cmd}:{name}:{rtype}\n".encode("ascii")
if len(msg) > 512:
# posix guarantees that writes to a pipe of less than PIPE_BUF
# bytes are atomic, and that PIPE_BUF >= 512
raise ValueError('msg too long')
self._ensure_running_and_write(msg)
return
# POSIX guarantees that writes to a pipe of less than PIPE_BUF (512 on Linux)
# bytes are atomic. Therefore, we want the message to be shorter than 512 bytes.
# POSIX shm_open() and sem_open() require the name, including its leading slash,
@ -286,6 +304,7 @@ def _send(self, cmd, name, rtype):
# The entire JSON message is guaranteed < PIPE_BUF (512 bytes) by construction.
assert len(msg) <= 512, f"internal error: message too long ({len(msg)} bytes)"
assert msg.startswith(b'{')
self._ensure_running_and_write(msg)
@ -296,6 +315,30 @@ def _send(self, cmd, name, rtype):
getfd = _resource_tracker.getfd
def _decode_message(line):
if line.startswith(b'{'):
try:
obj = json.loads(line.decode('ascii'))
except Exception as e:
raise ValueError("malformed resource_tracker message: %r" % (line,)) from e
cmd = obj["cmd"]
rtype = obj["rtype"]
b64 = obj.get("base64_name", "")
if not isinstance(cmd, str) or not isinstance(rtype, str) or not isinstance(b64, str):
raise ValueError("malformed resource_tracker fields: %r" % (obj,))
try:
name = base64.urlsafe_b64decode(b64).decode('utf-8', 'surrogateescape')
except ValueError as e:
raise ValueError("malformed resource_tracker base64_name: %r" % (b64,)) from e
else:
cmd, rest = line.strip().decode('ascii').split(':', maxsplit=1)
name, rtype = rest.rsplit(':', maxsplit=1)
return cmd, rtype, name
def main(fd):
'''Run resource tracker.'''
# protect the process from ^C and "killall python" etc
@ -318,23 +361,7 @@ def main(fd):
with open(fd, 'rb') as f:
for line in f:
try:
try:
obj = json.loads(line.decode('ascii'))
except Exception as e:
raise ValueError("malformed resource_tracker message: %r" % (line,)) from e
cmd = obj["cmd"]
rtype = obj["rtype"]
b64 = obj.get("base64_name", "")
if not isinstance(cmd, str) or not isinstance(rtype, str) or not isinstance(b64, str):
raise ValueError("malformed resource_tracker fields: %r" % (obj,))
try:
name = base64.urlsafe_b64decode(b64).decode('utf-8', 'surrogateescape')
except ValueError as e:
raise ValueError("malformed resource_tracker base64_name: %r" % (b64,)) from e
cmd, rtype, name = _decode_message(line)
cleanup_func = _CLEANUP_FUNCS.get(rtype, None)
if cleanup_func is None:
raise ValueError(

34
Lib/pickle.py
View file

@ -189,6 +189,11 @@ def __init__(self, value):
__all__.extend(x for x in dir() if x.isupper() and not x.startswith('_'))
# Data larger than this will be read in chunks, to prevent extreme
# overallocation.
_MIN_READ_BUF_SIZE = (1 << 20)
class _Framer:
_FRAME_SIZE_MIN = 4
@ -287,7 +292,7 @@ def read(self, n):
"pickle exhausted before end of frame")
return data
else:
return self.file_read(n)
return self._chunked_file_read(n)
def readline(self):
if self.current_frame:
@ -302,11 +307,23 @@ def readline(self):
else:
return self.file_readline()
def _chunked_file_read(self, size):
cursize = min(size, _MIN_READ_BUF_SIZE)
b = self.file_read(cursize)
while cursize < size and len(b) == cursize:
delta = min(cursize, size - cursize)
b += self.file_read(delta)
cursize += delta
return b
def load_frame(self, frame_size):
if self.current_frame and self.current_frame.read() != b'':
raise UnpicklingError(
"beginning of a new frame before end of current frame")
self.current_frame = io.BytesIO(self.file_read(frame_size))
data = self._chunked_file_read(frame_size)
if len(data) < frame_size:
raise EOFError
self.current_frame = io.BytesIO(data)
# Tools used for pickling.
@ -1496,12 +1513,17 @@ def load_binbytes8(self):
dispatch[BINBYTES8[0]] = load_binbytes8
def load_bytearray8(self):
len, = unpack('<Q', self.read(8))
if len > maxsize:
size, = unpack('<Q', self.read(8))
if size > maxsize:
raise UnpicklingError("BYTEARRAY8 exceeds system's maximum size "
"of %d bytes" % maxsize)
b = bytearray(len)
self.readinto(b)
cursize = min(size, _MIN_READ_BUF_SIZE)
b = bytearray(cursize)
if self.readinto(b) == cursize:
while cursize < size and len(b) == cursize:
delta = min(cursize, size - cursize)
b += self.read(delta)
cursize += delta
self.append(b)
dispatch[BYTEARRAY8[0]] = load_bytearray8

20
Lib/profiling/sampling/_heatmap_assets/heatmap.css
View file

@ -1108,18 +1108,34 @@ #scroll_marker .marker {
}
#scroll_marker .marker.cold {
background: var(--heat-1);
}
#scroll_marker .marker.cool {
background: var(--heat-2);
}
#scroll_marker .marker.mild {
background: var(--heat-3);
}
#scroll_marker .marker.warm {
background: var(--heat-5);
background: var(--heat-4);
}
#scroll_marker .marker.hot {
background: var(--heat-5);
}
#scroll_marker .marker.very-hot {
background: var(--heat-6);
}
#scroll_marker .marker.intense {
background: var(--heat-7);
}
#scroll_marker .marker.vhot {
#scroll_marker .marker.extreme {
background: var(--heat-8);
}

49
Lib/profiling/sampling/_heatmap_assets/heatmap.js
View file

@ -26,6 +26,7 @@ function toggleTheme() {
if (btn) {
btn.innerHTML = next === 'dark' ? '&#9788;' : '&#9790;'; // sun or moon
}
applyLineColors();
// Rebuild scroll marker with new theme colors
buildScrollMarker();
@ -160,13 +161,6 @@ function getSampleCount(line) {
return parseInt(text) || 0;
}
function getIntensityClass(ratio) {
if (ratio > 0.75) return 'vhot';
if (ratio > 0.5) return 'hot';
if (ratio > 0.25) return 'warm';
return 'cold';
}
// ============================================================================
// Scroll Minimap
// ============================================================================
@ -194,7 +188,7 @@ function buildScrollMarker() {
const lineTop = Math.floor(line.offsetTop * markerScale);
const lineNumber = index + 1;
const intensityClass = maxSamples > 0 ? getIntensityClass(samples / maxSamples) : 'cold';
const intensityClass = maxSamples > 0 ? (intensityToClass(samples / maxSamples) || 'cold') : 'cold';
if (lineNumber === prevLine + 1 && lastMark?.classList.contains(intensityClass)) {
lastMark.style.height = `${lineTop + lineHeight - lastTop}px`;
@ -212,6 +206,21 @@ function buildScrollMarker() {
document.body.appendChild(scrollMarker);
}
function applyLineColors() {
const lines = document.querySelectorAll('.code-line');
lines.forEach(line => {
let intensity;
if (colorMode === 'self') {
intensity = parseFloat(line.getAttribute('data-self-intensity')) || 0;
} else {
intensity = parseFloat(line.getAttribute('data-cumulative-intensity')) || 0;
}
const color = intensityToColor(intensity);
line.style.background = color;
});
}
// ============================================================================
// Toggle Controls
// ============================================================================
@ -264,20 +273,7 @@ function applyHotFilter() {
function toggleColorMode() {
colorMode = colorMode === 'self' ? 'cumulative' : 'self';
const lines = document.querySelectorAll('.code-line');
lines.forEach(line => {
let bgColor;
if (colorMode === 'self') {
bgColor = line.getAttribute('data-self-color');
} else {
bgColor = line.getAttribute('data-cumulative-color');
}
if (bgColor) {
line.style.background = bgColor;
}
});
applyLineColors();
updateToggleUI('toggle-color-mode', colorMode === 'cumulative');
@ -294,14 +290,7 @@ function toggleColorMode() {
document.addEventListener('DOMContentLoaded', function() {
restoreUIState();
// Apply background colors
document.querySelectorAll('.code-line[data-bg-color]').forEach(line => {
const bgColor = line.getAttribute('data-bg-color');
if (bgColor) {
line.style.background = bgColor;
}
});
applyLineColors();
// Initialize navigation buttons
document.querySelectorAll('.nav-btn').forEach(button => {

16
Lib/profiling/sampling/_heatmap_assets/heatmap_index.js
View file

@ -1,6 +1,19 @@
// Tachyon Profiler - Heatmap Index JavaScript
// Index page specific functionality
// ============================================================================
// Heatmap Bar Coloring
// ============================================================================
function applyHeatmapBarColors() {
const bars = document.querySelectorAll('.heatmap-bar[data-intensity]');
bars.forEach(bar => {
const intensity = parseFloat(bar.getAttribute('data-intensity')) || 0;
const color = intensityToColor(intensity);
bar.style.backgroundColor = color;
});
}
// ============================================================================
// Theme Support
// ============================================================================
@ -17,6 +30,8 @@ function toggleTheme() {
if (btn) {
btn.innerHTML = next === 'dark' ? '&#9788;' : '&#9790;'; // sun or moon
}
applyHeatmapBarColors();
}
function restoreUIState() {
@ -108,4 +123,5 @@ function collapseAll() {
document.addEventListener('DOMContentLoaded', function() {
restoreUIState();
applyHeatmapBarColors();
});

40
Lib/profiling/sampling/_heatmap_assets/heatmap_shared.js Normal file
View file

@ -0,0 +1,40 @@
// Tachyon Profiler - Shared Heatmap JavaScript
// Common utilities shared between index and file views
// ============================================================================
// Heat Level Mapping (Single source of truth for intensity thresholds)
// ============================================================================
// Maps intensity (0-1) to heat level (0-8). Level 0 = no heat, 1-8 = heat levels.
function intensityToHeatLevel(intensity) {
if (intensity <= 0) return 0;
if (intensity <= 0.125) return 1;
if (intensity <= 0.25) return 2;
if (intensity <= 0.375) return 3;
if (intensity <= 0.5) return 4;
if (intensity <= 0.625) return 5;
if (intensity <= 0.75) return 6;
if (intensity <= 0.875) return 7;
return 8;
}
// Class names corresponding to heat levels 1-8 (used by scroll marker)
const HEAT_CLASS_NAMES = ['cold', 'cool', 'mild', 'warm', 'hot', 'very-hot', 'intense', 'extreme'];
function intensityToClass(intensity) {
const level = intensityToHeatLevel(intensity);
return level === 0 ? null : HEAT_CLASS_NAMES[level - 1];
}
// ============================================================================
// Color Mapping (Intensity to Heat Color)
// ============================================================================
function intensityToColor(intensity) {
const level = intensityToHeatLevel(intensity);
if (level === 0) {
return 'transparent';
}
const rootStyle = getComputedStyle(document.documentElement);
return rootStyle.getPropertyValue(`--heat-${level}`).trim();
}

14
Lib/profiling/sampling/_shared_assets/base.css
View file

@ -62,9 +62,9 @@ :root, [data-theme="light"] {
--header-gradient: linear-gradient(135deg, #3776ab 0%, #4584bb 100%);
/* Light mode heat palette - blue to yellow to orange to red (cold to hot) */
--heat-1: #d6e9f8;
--heat-1: #7ba3d1;
--heat-2: #a8d0ef;
--heat-3: #7ba3d1;
--heat-3: #d6e9f8;
--heat-4: #ffe6a8;
--heat-5: #ffd43b;
--heat-6: #ffb84d;
@ -120,11 +120,11 @@ [data-theme="dark"] {
--header-gradient: linear-gradient(135deg, #21262d 0%, #30363d 100%);
/* Dark mode heat palette - dark blue to teal to yellow to orange (cold to hot) */
--heat-1: #1e3a5f;
--heat-2: #2d5580;
--heat-3: #4a7ba7;
--heat-4: #5a9fa8;
--heat-5: #7ec488;
--heat-1: #4a7ba7;
--heat-2: #5a9fa8;
--heat-3: #6ab5b5;
--heat-4: #7ec488;
--heat-5: #a0d878;
--heat-6: #c4de6a;
--heat-7: #f4d44d;
--heat-8: #ff6b35;

41
Lib/profiling/sampling/cli.py
View file

@ -201,6 +201,11 @@ def _add_sampling_options(parser):
help="Gather bytecode opcode information for instruction-level profiling "
"(shows which bytecode instructions are executing, including specializations).",
)
sampling_group.add_argument(
"--async-aware",
action="store_true",
help="Enable async-aware profiling (uses task-based stack reconstruction)",
)
def _add_mode_options(parser):
@ -211,7 +216,14 @@ def _add_mode_options(parser):
choices=["wall", "cpu", "gil"],
default="wall",
help="Sampling mode: wall (all samples), cpu (only samples when thread is on CPU), "
"gil (only samples when thread holds the GIL)",
"gil (only samples when thread holds the GIL). Incompatible with --async-aware",
)
mode_group.add_argument(
"--async-mode",
choices=["running", "all"],
default="running",
help='Async profiling mode: "running" (only running task) '
'or "all" (all tasks including waiting). Requires --async-aware',
)
@ -392,6 +404,27 @@ def _validate_args(args, parser):
"Live mode requires the curses module, which is not available."
)
# Async-aware mode is incompatible with --native, --no-gc, --mode, and --all-threads
if args.async_aware:
issues = []
if args.native:
issues.append("--native")
if not args.gc:
issues.append("--no-gc")
if hasattr(args, 'mode') and args.mode != "wall":
issues.append(f"--mode={args.mode}")
if hasattr(args, 'all_threads') and args.all_threads:
issues.append("--all-threads")
if issues:
parser.error(
f"Options {', '.join(issues)} are incompatible with --async-aware. "
"Async-aware profiling uses task-based stack reconstruction."
)
# --async-mode requires --async-aware
if hasattr(args, 'async_mode') and args.async_mode != "running" and not args.async_aware:
parser.error("--async-mode requires --async-aware to be enabled.")
# Live mode is incompatible with format options
if hasattr(args, 'live') and args.live:
if args.format != "pstats":
@ -587,6 +620,7 @@ def _handle_attach(args):
all_threads=args.all_threads,
realtime_stats=args.realtime_stats,
mode=mode,
async_aware=args.async_mode if args.async_aware else None,
native=args.native,
gc=args.gc,
opcodes=args.opcodes,
@ -636,6 +670,7 @@ def _handle_run(args):
all_threads=args.all_threads,
realtime_stats=args.realtime_stats,
mode=mode,
async_aware=args.async_mode if args.async_aware else None,
native=args.native,
gc=args.gc,
opcodes=args.opcodes,
@ -670,6 +705,7 @@ def _handle_live_attach(args, pid):
pid=pid,
mode=mode,
opcodes=args.opcodes,
async_aware=args.async_mode if args.async_aware else None,
)
# Sample in live mode
@ -680,6 +716,7 @@ def _handle_live_attach(args, pid):
all_threads=args.all_threads,
realtime_stats=args.realtime_stats,
mode=mode,
async_aware=args.async_mode if args.async_aware else None,
native=args.native,
gc=args.gc,
opcodes=args.opcodes,
@ -711,6 +748,7 @@ def _handle_live_run(args):
pid=process.pid,
mode=mode,
opcodes=args.opcodes,
async_aware=args.async_mode if args.async_aware else None,
)
# Profile the subprocess in live mode
@ -722,6 +760,7 @@ def _handle_live_run(args):
all_threads=args.all_threads,
realtime_stats=args.realtime_stats,
mode=mode,
async_aware=args.async_mode if args.async_aware else None,
native=args.native,
gc=args.gc,
opcodes=args.opcodes,

97
Lib/profiling/sampling/collector.py
View file

@ -3,10 +3,16 @@
DEFAULT_LOCATION,
THREAD_STATUS_HAS_GIL,
THREAD_STATUS_ON_CPU,
THREAD_STATUS_UNKNOWN,
THREAD_STATUS_GIL_REQUESTED,
THREAD_STATUS_UNKNOWN,
)
try:
from _remote_debugging import FrameInfo
except ImportError:
# Fallback definition if _remote_debugging is not available
FrameInfo = None
def normalize_location(location):
"""Normalize location to a 4-tuple format.
@ -62,6 +68,95 @@ def _iter_all_frames(self, stack_frames, skip_idle=False):
if frames:
yield frames, thread_info.thread_id
def _iter_async_frames(self, awaited_info_list):
# Phase 1: Index tasks and build parent relationships with pre-computed selection
task_map, child_to_parent, all_task_ids, all_parent_ids = self._build_task_graph(awaited_info_list)
# Phase 2: Find leaf tasks (tasks not awaited by anyone)
leaf_task_ids = self._find_leaf_tasks(all_task_ids, all_parent_ids)
# Phase 3: Build linear stacks from each leaf to root (optimized - no sorting!)
yield from self._build_linear_stacks(leaf_task_ids, task_map, child_to_parent)
def _build_task_graph(self, awaited_info_list):
task_map = {}
child_to_parent = {} # Maps child_id -> (selected_parent_id, parent_count)
all_task_ids = set()
all_parent_ids = set() # Track ALL parent IDs for leaf detection
for awaited_info in awaited_info_list:
thread_id = awaited_info.thread_id
for task_info in awaited_info.awaited_by:
task_id = task_info.task_id
task_map[task_id] = (task_info, thread_id)
all_task_ids.add(task_id)
# Pre-compute selected parent and count for optimization
if task_info.awaited_by:
parent_ids = [p.task_name for p in task_info.awaited_by]
parent_count = len(parent_ids)
# Track ALL parents for leaf detection
all_parent_ids.update(parent_ids)
# Use min() for O(n) instead of sorted()[0] which is O(n log n)
selected_parent = min(parent_ids) if parent_count > 1 else parent_ids[0]
child_to_parent[task_id] = (selected_parent, parent_count)
return task_map, child_to_parent, all_task_ids, all_parent_ids
def _find_leaf_tasks(self, all_task_ids, all_parent_ids):
# Leaves are tasks that are not parents of any other task
return all_task_ids - all_parent_ids
def _build_linear_stacks(self, leaf_task_ids, task_map, child_to_parent):
for leaf_id in leaf_task_ids:
frames = []
visited = set()
current_id = leaf_id
thread_id = None
# Follow the single parent chain from leaf to root
while current_id is not None:
# Cycle detection
if current_id in visited:
break
visited.add(current_id)
# Check if task exists in task_map
if current_id not in task_map:
break
task_info, tid = task_map[current_id]
# Set thread_id from first task
if thread_id is None:
thread_id = tid
# Add all frames from all coroutines in this task
if task_info.coroutine_stack:
for coro_info in task_info.coroutine_stack:
for frame in coro_info.call_stack:
frames.append(frame)
# Get pre-computed parent info (no sorting needed!)
parent_info = child_to_parent.get(current_id)
# Add task boundary marker with parent count annotation if multiple parents
task_name = task_info.task_name or "Task-" + str(task_info.task_id)
if parent_info:
selected_parent, parent_count = parent_info
if parent_count > 1:
task_name = f"{task_name} ({parent_count} parents)"
frames.append(FrameInfo(("<task>", None, task_name, None)))
current_id = selected_parent
else:
# Root task - no parent
frames.append(FrameInfo(("<task>", None, task_name, None)))
current_id = None
# Yield the complete stack if we collected any frames
if frames and thread_id is not None:
yield frames, thread_id, leaf_id
def _is_gc_frame(self, frame):
if isinstance(frame, tuple):
funcname = frame[2] if len(frame) >= 3 else ""

141
Lib/profiling/sampling/heatmap_collector.py
View file

@ -5,13 +5,14 @@
import html
import importlib.resources
import json
import math
import os
import platform
import site
import sys
from dataclasses import dataclass, field
from pathlib import Path
from typing import Dict, List, Tuple, Optional, Any
from typing import Dict, List, Tuple
from ._css_utils import get_combined_css
from .collector import normalize_location, extract_lineno
@ -45,31 +46,6 @@ class TreeNode:
children: Dict[str, 'TreeNode'] = field(default_factory=dict)
@dataclass
class ColorGradient:
"""Configuration for heatmap color gradient calculations."""
# Color stops thresholds
stop_1: float = 0.2 # Blue to cyan transition
stop_2: float = 0.4 # Cyan to green transition
stop_3: float = 0.6 # Green to yellow transition
stop_4: float = 0.8 # Yellow to orange transition
stop_5: float = 1.0 # Orange to red transition
# Alpha (opacity) values
alpha_very_cold: float = 0.3
alpha_cold: float = 0.4
alpha_medium: float = 0.5
alpha_warm: float = 0.6
alpha_hot_base: float = 0.7
alpha_hot_range: float = 0.15
# Gradient multiplier
multiplier: int = 5
# Cache for calculated colors
cache: Dict[float, Tuple[int, int, int, float]] = field(default_factory=dict)
# ============================================================================
# Module Path Analysis
# ============================================================================
@ -225,8 +201,9 @@ def _load_templates(self):
self.file_css = css_content
# Load JS
self.index_js = (assets_dir / "heatmap_index.js").read_text(encoding="utf-8")
self.file_js = (assets_dir / "heatmap.js").read_text(encoding="utf-8")
shared_js = (assets_dir / "heatmap_shared.js").read_text(encoding="utf-8")
self.index_js = f"{shared_js}\n{(assets_dir / 'heatmap_index.js').read_text(encoding='utf-8')}"
self.file_js = f"{shared_js}\n{(assets_dir / 'heatmap.js').read_text(encoding='utf-8')}"
# Load Python logo
logo_dir = template_dir / "_assets"
@ -322,18 +299,13 @@ def _calculate_node_stats(node: TreeNode) -> Tuple[int, int]:
class _HtmlRenderer:
"""Renders hierarchical tree structures as HTML."""
def __init__(self, file_index: Dict[str, str], color_gradient: ColorGradient,
calculate_intensity_color_func):
"""Initialize renderer with file index and color calculation function.
def __init__(self, file_index: Dict[str, str]):
"""Initialize renderer with file index.
Args:
file_index: Mapping from filenames to HTML file names
color_gradient: ColorGradient configuration
calculate_intensity_color_func: Function to calculate colors
"""
self.file_index = file_index
self.color_gradient = color_gradient
self.calculate_intensity_color = calculate_intensity_color_func
self.heatmap_bar_height = 16
def render_hierarchical_html(self, trees: Dict[str, TreeNode]) -> str:
@ -451,8 +423,6 @@ def _render_file_item(self, stat: FileStats, indent: str = '') -> str:
module_name = html.escape(stat.module_name)
intensity = stat.percentage / 100.0
r, g, b, alpha = self.calculate_intensity_color(intensity)
bg_color = f"rgba({r}, {g}, {b}, {alpha})"
bar_width = min(stat.percentage, 100)
html_file = self.file_index[stat.filename]
@ -460,7 +430,7 @@ def _render_file_item(self, stat: FileStats, indent: str = '') -> str:
return (f'{indent}<div class="file-item">\n'
f'{indent} <a href="{html_file}" class="file-link" title="{full_path}">📄 {module_name}</a>\n'
f'{indent} <span class="file-samples">{stat.total_samples:,} samples</span>\n'
f'{indent} <div class="heatmap-bar-container"><div class="heatmap-bar" style="width: {bar_width}px; background-color: {bg_color}; height: {self.heatmap_bar_height}px;"></div></div>\n'
f'{indent} <div class="heatmap-bar-container"><div class="heatmap-bar" style="width: {bar_width}px; height: {self.heatmap_bar_height}px;" data-intensity="{intensity:.3f}"></div></div>\n'
f'{indent}</div>\n')
@ -507,20 +477,12 @@ def __init__(self, *args, **kwargs):
self._path_info = get_python_path_info()
self.stats = {}
# Color gradient configuration
self._color_gradient = ColorGradient()
# Template loader (loads all templates once)
self._template_loader = _TemplateLoader()
# File index (populated during export)
self.file_index = {}
@property
def _color_cache(self):
"""Compatibility property for accessing color cache."""
return self._color_gradient.cache
def set_stats(self, sample_interval_usec, duration_sec, sample_rate, error_rate=None, missed_samples=None, **kwargs):
"""Set profiling statistics to include in heatmap output.
@ -832,8 +794,7 @@ def _generate_index_html(self, index_path: Path, file_stats: List[FileStats]):
tree = _TreeBuilder.build_file_tree(file_stats)
# Render tree as HTML
renderer = _HtmlRenderer(self.file_index, self._color_gradient,
self._calculate_intensity_color)
renderer = _HtmlRenderer(self.file_index)
sections_html = renderer.render_hierarchical_html(tree)
# Format error rate and missed samples with bar classes
@ -895,56 +856,6 @@ def _generate_index_html(self, index_path: Path, file_stats: List[FileStats]):
except (IOError, OSError) as e:
raise RuntimeError(f"Failed to write index file {index_path}: {e}") from e
def _calculate_intensity_color(self, intensity: float) -> Tuple[int, int, int, float]:
"""Calculate RGB color and alpha for given intensity (0-1 range).
Returns (r, g, b, alpha) tuple representing the heatmap color gradient:
blue -> green -> yellow -> orange -> red
Results are cached to improve performance.
"""
# Round to 3 decimal places for cache key
cache_key = round(intensity, 3)
if cache_key in self._color_gradient.cache:
return self._color_gradient.cache[cache_key]
gradient = self._color_gradient
m = gradient.multiplier
# Color stops with (threshold, rgb_func, alpha_func)
stops = [
(gradient.stop_1,
lambda i: (0, int(150 * i * m), 255),
lambda i: gradient.alpha_very_cold),
(gradient.stop_2,
lambda i: (0, 255, int(255 * (1 - (i - gradient.stop_1) * m))),
lambda i: gradient.alpha_cold),
(gradient.stop_3,
lambda i: (int(255 * (i - gradient.stop_2) * m), 255, 0),
lambda i: gradient.alpha_medium),
(gradient.stop_4,
lambda i: (255, int(200 - 100 * (i - gradient.stop_3) * m), 0),
lambda i: gradient.alpha_warm),
(gradient.stop_5,
lambda i: (255, int(100 * (1 - (i - gradient.stop_4) * m)), 0),
lambda i: gradient.alpha_hot_base + gradient.alpha_hot_range * (i - gradient.stop_4) * m),
]
result = None
for threshold, rgb_func, alpha_func in stops:
if intensity < threshold or threshold == gradient.stop_5:
r, g, b = rgb_func(intensity)
result = (r, g, b, alpha_func(intensity))
break
# Fallback
if result is None:
result = (255, 0, 0, 0.75)
# Cache the result
self._color_gradient.cache[cache_key] = result
return result
def _generate_file_html(self, output_path: Path, filename: str,
line_counts: Dict[int, int], self_counts: Dict[int, int],
file_stat: FileStats):
@ -999,25 +910,23 @@ def _build_line_html(self, line_num: int, line_content: str,
# Calculate colors for both self and cumulative modes
if cumulative_samples > 0:
cumulative_intensity = cumulative_samples / max_samples if max_samples > 0 else 0
self_intensity = self_samples / max_self_samples if max_self_samples > 0 and self_samples > 0 else 0
log_cumulative = math.log(cumulative_samples + 1)
log_max = math.log(max_samples + 1)
cumulative_intensity = log_cumulative / log_max if log_max > 0 else 0
# Default to self-based coloring
intensity = self_intensity if self_samples > 0 else cumulative_intensity
r, g, b, alpha = self._calculate_intensity_color(intensity)
bg_color = f"rgba({r}, {g}, {b}, {alpha})"
# Pre-calculate colors for both modes (for JS toggle)
self_bg_color = self._format_color_for_intensity(self_intensity) if self_samples > 0 else "transparent"
cumulative_bg_color = self._format_color_for_intensity(cumulative_intensity)
if self_samples > 0 and max_self_samples > 0:
log_self = math.log(self_samples + 1)
log_max_self = math.log(max_self_samples + 1)
self_intensity = log_self / log_max_self if log_max_self > 0 else 0
else:
self_intensity = 0
self_display = f"{self_samples:,}" if self_samples > 0 else ""
cumulative_display = f"{cumulative_samples:,}"
tooltip = f"Self: {self_samples:,}, Total: {cumulative_samples:,}"
else:
bg_color = "transparent"
self_bg_color = "transparent"
cumulative_bg_color = "transparent"
cumulative_intensity = 0
self_intensity = 0
self_display = ""
cumulative_display = ""
tooltip = ""
@ -1059,8 +968,9 @@ def _build_line_html(self, line_num: int, line_content: str,
spec_color_attr = f'data-spec-color="{spec_color}" '
return (
f' <div class="code-line" data-bg-color="{bg_color}" '
f'data-self-color="{self_bg_color}" data-cumulative-color="{cumulative_bg_color}" '
f' <div class="code-line" '
f'data-self-intensity="{self_intensity:.3f}" '
f'data-cumulative-intensity="{cumulative_intensity:.3f}" '
f'{spec_color_attr}'
f'id="line-{line_num}"{title_attr}>\n'
f' <div class="line-number">{line_num}</div>\n'
@ -1167,11 +1077,6 @@ def flush_span():
flush_span()
return ''.join(result)
def _format_color_for_intensity(self, intensity: float) -> str:
"""Format color as rgba() string for given intensity."""
r, g, b, alpha = self._calculate_intensity_color(intensity)
return f"rgba({r}, {g}, {b}, {alpha})"
def _format_specialization_color(self, spec_pct: int) -> str:
"""Format specialization color based on percentage.

110
Lib/profiling/sampling/live_collector/collector.py
View file

@ -109,6 +109,7 @@ def __init__(
display=None,
mode=None,
opcodes=False,
async_aware=None,
):
"""
Initialize the live stats collector.
@ -122,6 +123,7 @@ def __init__(
display: DisplayInterface implementation (None means curses will be used)
mode: Profiling mode ('cpu', 'gil', etc.) - affects what stats are shown
opcodes: Whether to show opcode panel (requires --opcodes flag)
async_aware: Async tracing mode - None (sync only), "all" or "running"
"""
self.result = collections.defaultdict(
lambda: dict(total_rec_calls=0, direct_calls=0, cumulative_calls=0)
@ -140,6 +142,9 @@ def __init__(
self.running = True
self.pid = pid
self.mode = mode # Profiling mode
self.async_aware = async_aware # Async tracing mode
# Pre-select frame iterator method to avoid per-call dispatch overhead
self._get_frame_iterator = self._get_async_frame_iterator if async_aware else self._get_sync_frame_iterator
self._saved_stdout = None
self._saved_stderr = None
self._devnull = None
@ -319,6 +324,15 @@ def process_frames(self, frames, thread_id=None):
if thread_data:
thread_data.opcode_stats[top_location][opcode] += 1
def _get_sync_frame_iterator(self, stack_frames):
"""Iterator for sync frames."""
return self._iter_all_frames(stack_frames, skip_idle=self.skip_idle)
def _get_async_frame_iterator(self, stack_frames):
"""Iterator for async frames, yielding (frames, thread_id) tuples."""
for frames, thread_id, task_id in self._iter_async_frames(stack_frames):
yield frames, thread_id
def collect_failed_sample(self):
self.failed_samples += 1
self.total_samples += 1
@ -329,78 +343,40 @@ def collect(self, stack_frames):
self.start_time = time.perf_counter()
self._last_display_update = self.start_time
# Thread status counts for this sample
temp_status_counts = {
"has_gil": 0,
"on_cpu": 0,
"gil_requested": 0,
"unknown": 0,
"total": 0,
}
has_gc_frame = False
# Always collect data, even when paused
# Track thread status flags and GC frames
for interpreter_info in stack_frames:
threads = getattr(interpreter_info, "threads", [])
for thread_info in threads:
temp_status_counts["total"] += 1
# Collect thread status stats (only available in sync mode)
if not self.async_aware:
status_counts, sample_has_gc, per_thread_stats = self._collect_thread_status_stats(stack_frames)
for key, count in status_counts.items():
self.thread_status_counts[key] += count
if sample_has_gc:
has_gc_frame = True
# Track thread status using bit flags
status_flags = getattr(thread_info, "status", 0)
thread_id = getattr(thread_info, "thread_id", None)
for thread_id, stats in per_thread_stats.items():
thread_data = self._get_or_create_thread_data(thread_id)
thread_data.has_gil += stats.get("has_gil", 0)
thread_data.on_cpu += stats.get("on_cpu", 0)
thread_data.gil_requested += stats.get("gil_requested", 0)
thread_data.unknown += stats.get("unknown", 0)
thread_data.total += stats.get("total", 0)
if stats.get("gc_samples", 0):
thread_data.gc_frame_samples += stats["gc_samples"]
# Update aggregated counts
if status_flags & THREAD_STATUS_HAS_GIL:
temp_status_counts["has_gil"] += 1
if status_flags & THREAD_STATUS_ON_CPU:
temp_status_counts["on_cpu"] += 1
if status_flags & THREAD_STATUS_GIL_REQUESTED:
temp_status_counts["gil_requested"] += 1
if status_flags & THREAD_STATUS_UNKNOWN:
temp_status_counts["unknown"] += 1
# Process frames using pre-selected iterator
for frames, thread_id in self._get_frame_iterator(stack_frames):
if not frames:
continue
# Update per-thread status counts
if thread_id is not None:
thread_data = self._get_or_create_thread_data(thread_id)
thread_data.increment_status_flag(status_flags)
self.process_frames(frames, thread_id=thread_id)
# Process frames (respecting skip_idle)
if self.skip_idle:
has_gil = bool(status_flags & THREAD_STATUS_HAS_GIL)
on_cpu = bool(status_flags & THREAD_STATUS_ON_CPU)
if not (has_gil or on_cpu):
continue
# Track thread IDs
if thread_id is not None and thread_id not in self.thread_ids:
self.thread_ids.append(thread_id)
frames = getattr(thread_info, "frame_info", None)
if frames:
self.process_frames(frames, thread_id=thread_id)
# Track thread IDs only for threads that actually have samples
if (
thread_id is not None
and thread_id not in self.thread_ids
):
self.thread_ids.append(thread_id)
# Increment per-thread sample count and check for GC frames
thread_has_gc_frame = False
for frame in frames:
funcname = getattr(frame, "funcname", "")
if "<GC>" in funcname or "gc_collect" in funcname:
has_gc_frame = True
thread_has_gc_frame = True
break
if thread_id is not None:
thread_data = self._get_or_create_thread_data(thread_id)
thread_data.sample_count += 1
if thread_has_gc_frame:
thread_data.gc_frame_samples += 1
# Update cumulative thread status counts
for key, count in temp_status_counts.items():
self.thread_status_counts[key] += count
if thread_id is not None:
thread_data = self._get_or_create_thread_data(thread_id)
thread_data.sample_count += 1
if has_gc_frame:
self.gc_frame_samples += 1
@ -893,10 +869,12 @@ def _handle_input(self):
# Handle help toggle keys
if ch == ord("h") or ch == ord("H") or ch == ord("?"):
self.show_help = not self.show_help
return
# If showing help, any other key closes it
elif self.show_help and ch != -1:
if self.show_help and ch != -1:
self.show_help = False
return
# Handle regular commands
if ch == ord("q") or ch == ord("Q"):

10
Lib/profiling/sampling/pstats_collector.py
View file

@ -47,8 +47,14 @@ def _process_frames(self, frames):
self.callers[callee][caller] += 1
def collect(self, stack_frames):
for frames, thread_id in self._iter_all_frames(stack_frames, skip_idle=self.skip_idle):
self._process_frames(frames)
if stack_frames and hasattr(stack_frames[0], "awaited_by"):
# Async frame processing
for frames, thread_id, task_id in self._iter_async_frames(stack_frames):
self._process_frames(frames)
else:
# Regular frame processing
for frames, thread_id in self._iter_all_frames(stack_frames, skip_idle=self.skip_idle):
self._process_frames(frames)
def export(self, filename):
self.create_stats()

127
Lib/profiling/sampling/sample.py
View file

@ -27,28 +27,31 @@
class SampleProfiler:
def __init__(self, pid, sample_interval_usec, all_threads, *, mode=PROFILING_MODE_WALL, native=False, gc=True, opcodes=False, skip_non_matching_threads=True):
def __init__(self, pid, sample_interval_usec, all_threads, *, mode=PROFILING_MODE_WALL, native=False, gc=True, opcodes=False, skip_non_matching_threads=True, collect_stats=False):
self.pid = pid
self.sample_interval_usec = sample_interval_usec
self.all_threads = all_threads
self.mode = mode # Store mode for later use
self.collect_stats = collect_stats
if _FREE_THREADED_BUILD:
self.unwinder = _remote_debugging.RemoteUnwinder(
self.pid, all_threads=self.all_threads, mode=mode, native=native, gc=gc,
opcodes=opcodes, skip_non_matching_threads=skip_non_matching_threads
opcodes=opcodes, skip_non_matching_threads=skip_non_matching_threads,
cache_frames=True, stats=collect_stats
)
else:
only_active_threads = bool(self.all_threads)
self.unwinder = _remote_debugging.RemoteUnwinder(
self.pid, only_active_thread=only_active_threads, mode=mode, native=native, gc=gc,
opcodes=opcodes, skip_non_matching_threads=skip_non_matching_threads
opcodes=opcodes, skip_non_matching_threads=skip_non_matching_threads,
cache_frames=True, stats=collect_stats
)
# Track sample intervals and total sample count
self.sample_intervals = deque(maxlen=100)
self.total_samples = 0
self.realtime_stats = False
def sample(self, collector, duration_sec=10):
def sample(self, collector, duration_sec=10, *, async_aware=False):
sample_interval_sec = self.sample_interval_usec / 1_000_000
running_time = 0
num_samples = 0
@ -68,7 +71,12 @@ def sample(self, collector, duration_sec=10):
current_time = time.perf_counter()
if next_time < current_time:
try:
stack_frames = self.unwinder.get_stack_trace()
if async_aware == "all":
stack_frames = self.unwinder.get_all_awaited_by()
elif async_aware == "running":
stack_frames = self.unwinder.get_async_stack_trace()
else:
stack_frames = self.unwinder.get_stack_trace()
collector.collect(stack_frames)
except ProcessLookupError:
duration_sec = current_time - start_time
@ -124,6 +132,10 @@ def sample(self, collector, duration_sec=10):
print(f"Sample rate: {sample_rate:.2f} samples/sec")
print(f"Error rate: {error_rate:.2f}%")
# Print unwinder stats if stats collection is enabled
if self.collect_stats:
self._print_unwinder_stats()
# Pass stats to flamegraph collector if it's the right type
if hasattr(collector, 'set_stats'):
collector.set_stats(self.sample_interval_usec, running_time, sample_rate, error_rate, missed_samples, mode=self.mode)
@ -171,17 +183,100 @@ def _print_realtime_stats(self):
(1.0 / min_hz) * 1_000_000 if min_hz > 0 else 0
) # Max time = Min Hz
# Build cache stats string if stats collection is enabled
cache_stats_str = ""
if self.collect_stats:
try:
stats = self.unwinder.get_stats()
hits = stats.get('frame_cache_hits', 0)
partial = stats.get('frame_cache_partial_hits', 0)
misses = stats.get('frame_cache_misses', 0)
total = hits + partial + misses
if total > 0:
hit_pct = (hits + partial) / total * 100
cache_stats_str = f" {ANSIColors.MAGENTA}Cache: {hit_pct:.1f}% ({hits}+{partial}/{misses}){ANSIColors.RESET}"
except RuntimeError:
pass
# Clear line and print stats
print(
f"\r\033[K{ANSIColors.BOLD_BLUE}Real-time sampling stats:{ANSIColors.RESET} "
f"{ANSIColors.YELLOW}Mean: {mean_hz:.1f}Hz ({mean_us_per_sample:.2f}µs){ANSIColors.RESET} "
f"{ANSIColors.GREEN}Min: {min_hz:.1f}Hz ({max_us_per_sample:.2f}µs){ANSIColors.RESET} "
f"{ANSIColors.RED}Max: {max_hz:.1f}Hz ({min_us_per_sample:.2f}µs){ANSIColors.RESET} "
f"{ANSIColors.CYAN}Samples: {self.total_samples}{ANSIColors.RESET}",
f"\r\033[K{ANSIColors.BOLD_BLUE}Stats:{ANSIColors.RESET} "
f"{ANSIColors.YELLOW}{mean_hz:.1f}Hz ({mean_us_per_sample:.1f}µs){ANSIColors.RESET} "
f"{ANSIColors.GREEN}Min: {min_hz:.1f}Hz{ANSIColors.RESET} "
f"{ANSIColors.RED}Max: {max_hz:.1f}Hz{ANSIColors.RESET} "
f"{ANSIColors.CYAN}N={self.total_samples}{ANSIColors.RESET}"
f"{cache_stats_str}",
end="",
flush=True,
)
def _print_unwinder_stats(self):
"""Print unwinder statistics including cache performance."""
try:
stats = self.unwinder.get_stats()
except RuntimeError:
return # Stats not enabled
print(f"\n{ANSIColors.BOLD_BLUE}{'='*50}{ANSIColors.RESET}")
print(f"{ANSIColors.BOLD_BLUE}Unwinder Statistics:{ANSIColors.RESET}")
# Frame cache stats
total_samples = stats.get('total_samples', 0)
frame_cache_hits = stats.get('frame_cache_hits', 0)
frame_cache_partial_hits = stats.get('frame_cache_partial_hits', 0)
frame_cache_misses = stats.get('frame_cache_misses', 0)
total_lookups = frame_cache_hits + frame_cache_partial_hits + frame_cache_misses
# Calculate percentages
hits_pct = (frame_cache_hits / total_lookups * 100) if total_lookups > 0 else 0
partial_pct = (frame_cache_partial_hits / total_lookups * 100) if total_lookups > 0 else 0
misses_pct = (frame_cache_misses / total_lookups * 100) if total_lookups > 0 else 0
print(f" {ANSIColors.CYAN}Frame Cache:{ANSIColors.RESET}")
print(f" Total samples: {total_samples:,}")
print(f" Full hits: {frame_cache_hits:,} ({ANSIColors.GREEN}{hits_pct:.1f}%{ANSIColors.RESET})")
print(f" Partial hits: {frame_cache_partial_hits:,} ({ANSIColors.YELLOW}{partial_pct:.1f}%{ANSIColors.RESET})")
print(f" Misses: {frame_cache_misses:,} ({ANSIColors.RED}{misses_pct:.1f}%{ANSIColors.RESET})")
# Frame read stats
frames_from_cache = stats.get('frames_read_from_cache', 0)
frames_from_memory = stats.get('frames_read_from_memory', 0)
total_frames = frames_from_cache + frames_from_memory
cache_frame_pct = (frames_from_cache / total_frames * 100) if total_frames > 0 else 0
memory_frame_pct = (frames_from_memory / total_frames * 100) if total_frames > 0 else 0
print(f" {ANSIColors.CYAN}Frame Reads:{ANSIColors.RESET}")
print(f" From cache: {frames_from_cache:,} ({ANSIColors.GREEN}{cache_frame_pct:.1f}%{ANSIColors.RESET})")
print(f" From memory: {frames_from_memory:,} ({ANSIColors.RED}{memory_frame_pct:.1f}%{ANSIColors.RESET})")
# Code object cache stats
code_hits = stats.get('code_object_cache_hits', 0)
code_misses = stats.get('code_object_cache_misses', 0)
total_code = code_hits + code_misses
code_hits_pct = (code_hits / total_code * 100) if total_code > 0 else 0
code_misses_pct = (code_misses / total_code * 100) if total_code > 0 else 0
print(f" {ANSIColors.CYAN}Code Object Cache:{ANSIColors.RESET}")
print(f" Hits: {code_hits:,} ({ANSIColors.GREEN}{code_hits_pct:.1f}%{ANSIColors.RESET})")
print(f" Misses: {code_misses:,} ({ANSIColors.RED}{code_misses_pct:.1f}%{ANSIColors.RESET})")
# Memory operations
memory_reads = stats.get('memory_reads', 0)
memory_bytes = stats.get('memory_bytes_read', 0)
if memory_bytes >= 1024 * 1024:
memory_str = f"{memory_bytes / (1024 * 1024):.1f} MB"
elif memory_bytes >= 1024:
memory_str = f"{memory_bytes / 1024:.1f} KB"
else:
memory_str = f"{memory_bytes} B"
print(f" {ANSIColors.CYAN}Memory:{ANSIColors.RESET}")
print(f" Read operations: {memory_reads:,} ({memory_str})")
# Stale invalidations
stale_invalidations = stats.get('stale_cache_invalidations', 0)
if stale_invalidations > 0:
print(f" {ANSIColors.YELLOW}Stale cache invalidations: {stale_invalidations}{ANSIColors.RESET}")
def sample(
pid,
@ -191,6 +286,7 @@ def sample(
all_threads=False,
realtime_stats=False,
mode=PROFILING_MODE_WALL,
async_aware=None,
native=False,
gc=True,
opcodes=False,
@ -231,12 +327,13 @@ def sample(
native=native,
gc=gc,
opcodes=opcodes,
skip_non_matching_threads=skip_non_matching_threads
skip_non_matching_threads=skip_non_matching_threads,
collect_stats=realtime_stats,
)
profiler.realtime_stats = realtime_stats
# Run the sampling
profiler.sample(collector, duration_sec)
profiler.sample(collector, duration_sec, async_aware=async_aware)
return collector
@ -249,6 +346,7 @@ def sample_live(
all_threads=False,
realtime_stats=False,
mode=PROFILING_MODE_WALL,
async_aware=None,
native=False,
gc=True,
opcodes=False,
@ -289,14 +387,15 @@ def sample_live(
native=native,
gc=gc,
opcodes=opcodes,
skip_non_matching_threads=skip_non_matching_threads
skip_non_matching_threads=skip_non_matching_threads,
collect_stats=realtime_stats,
)
profiler.realtime_stats = realtime_stats
def curses_wrapper_func(stdscr):
collector.init_curses(stdscr)
try:
profiler.sample(collector, duration_sec)
profiler.sample(collector, duration_sec, async_aware=async_aware)
# Mark as finished and keep the TUI running until user presses 'q'
collector.mark_finished()
# Keep processing input until user quits

16
Lib/profiling/sampling/stack_collector.py
View file

@ -18,10 +18,18 @@ def __init__(self, sample_interval_usec, *, skip_idle=False):
self.skip_idle = skip_idle
def collect(self, stack_frames, skip_idle=False):
for frames, thread_id in self._iter_all_frames(stack_frames, skip_idle=skip_idle):
if not frames:
continue
self.process_frames(frames, thread_id)
if stack_frames and hasattr(stack_frames[0], "awaited_by"):
# Async-aware mode: process async task frames
for frames, thread_id, task_id in self._iter_async_frames(stack_frames):
if not frames:
continue
self.process_frames(frames, thread_id)
else:
# Sync-only mode
for frames, thread_id in self._iter_all_frames(stack_frames, skip_idle=skip_idle):
if not frames:
continue
self.process_frames(frames, thread_id)
def process_frames(self, frames, thread_id):
pass

2
Lib/ssl.py
View file

@ -185,7 +185,7 @@ class _TLSContentType:
class _TLSAlertType:
"""Alert types for TLSContentType.ALERT messages
See RFC 8466, section B.2
See RFC 8446, section B.2
"""
CLOSE_NOTIFY = 0
UNEXPECTED_MESSAGE = 10

26
Lib/test/_test_multiprocessing.py
View file

@ -39,6 +39,7 @@
from test.support import socket_helper
from test.support import threading_helper
from test.support import warnings_helper
from test.support import subTests
from test.support.script_helper import assert_python_failure, assert_python_ok
# Skip tests if _multiprocessing wasn't built.
@ -4383,6 +4384,19 @@ def test_copy(self):
self.assertEqual(bar.z, 2 ** 33)
def resource_tracker_format_subtests(func):
"""Run given test using both resource tracker communication formats"""
def _inner(self, *args, **kwargs):
tracker = resource_tracker._resource_tracker
for use_simple_format in False, True:
with (
self.subTest(use_simple_format=use_simple_format),
unittest.mock.patch.object(
tracker, '_use_simple_format', use_simple_format)
):
func(self, *args, **kwargs)
return _inner
@unittest.skipUnless(HAS_SHMEM, "requires multiprocessing.shared_memory")
@hashlib_helper.requires_hashdigest('sha256')
class _TestSharedMemory(BaseTestCase):
@ -4662,6 +4676,7 @@ def test_shared_memory_SharedMemoryServer_ignores_sigint(self):
smm.shutdown()
@unittest.skipIf(os.name != "posix", "resource_tracker is posix only")
@resource_tracker_format_subtests
def test_shared_memory_SharedMemoryManager_reuses_resource_tracker(self):
# bpo-36867: test that a SharedMemoryManager uses the
# same resource_tracker process as its parent.
@ -4913,6 +4928,7 @@ def test_shared_memory_cleaned_after_process_termination(self):
"shared_memory objects to clean up at shutdown", err)
@unittest.skipIf(os.name != "posix", "resource_tracker is posix only")
@resource_tracker_format_subtests
def test_shared_memory_untracking(self):
# gh-82300: When a separate Python process accesses shared memory
# with track=False, it must not cause the memory to be deleted
@ -4940,6 +4956,7 @@ def test_shared_memory_untracking(self):
mem.close()
@unittest.skipIf(os.name != "posix", "resource_tracker is posix only")
@resource_tracker_format_subtests
def test_shared_memory_tracking(self):
# gh-82300: When a separate Python process accesses shared memory
# with track=True, it must cause the memory to be deleted when
@ -7353,13 +7370,18 @@ def test_forkpty(self):
@unittest.skipUnless(HAS_SHMEM, "requires multiprocessing.shared_memory")
class TestSharedMemoryNames(unittest.TestCase):
def test_that_shared_memory_name_with_colons_has_no_resource_tracker_errors(self):
@subTests('use_simple_format', (True, False))
def test_that_shared_memory_name_with_colons_has_no_resource_tracker_errors(
self, use_simple_format):
# Test script that creates and cleans up shared memory with colon in name
test_script = textwrap.dedent("""
import sys
from multiprocessing import shared_memory
from multiprocessing import resource_tracker
import time
resource_tracker._resource_tracker._use_simple_format = %s
# Test various patterns of colons in names
test_names = [
"a:b",
@ -7387,7 +7409,7 @@ def test_that_shared_memory_name_with_colons_has_no_resource_tracker_errors(self
sys.exit(1)
print("SUCCESS")
""")
""" % use_simple_format)
rc, out, err = assert_python_ok("-c", test_script)
self.assertIn(b"SUCCESS", out)

14
Lib/test/datetimetester.py
View file

@ -6300,21 +6300,21 @@ def test_vilnius_1941_fromutc(self):
gdt = datetime(1941, 6, 23, 20, 59, 59, tzinfo=timezone.utc)
ldt = gdt.astimezone(Vilnius)
self.assertEqual(ldt.strftime("%c %Z%z"),
self.assertEqual(ldt.strftime("%a %b %d %H:%M:%S %Y %Z%z"),
'Mon Jun 23 23:59:59 1941 MSK+0300')
self.assertEqual(ldt.fold, 0)
self.assertFalse(ldt.dst())
gdt = datetime(1941, 6, 23, 21, tzinfo=timezone.utc)
ldt = gdt.astimezone(Vilnius)
self.assertEqual(ldt.strftime("%c %Z%z"),
self.assertEqual(ldt.strftime("%a %b %d %H:%M:%S %Y %Z%z"),
'Mon Jun 23 23:00:00 1941 CEST+0200')
self.assertEqual(ldt.fold, 1)
self.assertTrue(ldt.dst())
gdt = datetime(1941, 6, 23, 22, tzinfo=timezone.utc)
ldt = gdt.astimezone(Vilnius)
self.assertEqual(ldt.strftime("%c %Z%z"),
self.assertEqual(ldt.strftime("%a %b %d %H:%M:%S %Y %Z%z"),
'Tue Jun 24 00:00:00 1941 CEST+0200')
self.assertEqual(ldt.fold, 0)
self.assertTrue(ldt.dst())
@ -6324,22 +6324,22 @@ def test_vilnius_1941_toutc(self):
ldt = datetime(1941, 6, 23, 22, 59, 59, tzinfo=Vilnius)
gdt = ldt.astimezone(timezone.utc)
self.assertEqual(gdt.strftime("%c %Z"),
self.assertEqual(gdt.strftime("%a %b %d %H:%M:%S %Y %Z"),
'Mon Jun 23 19:59:59 1941 UTC')
ldt = datetime(1941, 6, 23, 23, 59, 59, tzinfo=Vilnius)
gdt = ldt.astimezone(timezone.utc)
self.assertEqual(gdt.strftime("%c %Z"),
self.assertEqual(gdt.strftime("%a %b %d %H:%M:%S %Y %Z"),
'Mon Jun 23 20:59:59 1941 UTC')
ldt = datetime(1941, 6, 23, 23, 59, 59, tzinfo=Vilnius, fold=1)
gdt = ldt.astimezone(timezone.utc)
self.assertEqual(gdt.strftime("%c %Z"),
self.assertEqual(gdt.strftime("%a %b %d %H:%M:%S %Y %Z"),
'Mon Jun 23 21:59:59 1941 UTC')
ldt = datetime(1941, 6, 24, 0, tzinfo=Vilnius)
gdt = ldt.astimezone(timezone.utc)
self.assertEqual(gdt.strftime("%c %Z"),
self.assertEqual(gdt.strftime("%a %b %d %H:%M:%S %Y %Z"),
'Mon Jun 23 22:00:00 1941 UTC')
def test_constructors(self):

167
Lib/test/pickletester.py
View file

@ -74,6 +74,15 @@ def count_opcode(code, pickle):
def identity(x):
return x
def itersize(start, stop):
# Produce geometrical increasing sequence from start to stop
# (inclusively) for tests.
size = start
while size < stop:
yield size
size <<= 1
yield stop
class UnseekableIO(io.BytesIO):
def peek(self, *args):
@ -853,9 +862,8 @@ def assert_is_copy(self, obj, objcopy, msg=None):
self.assertEqual(getattr(obj, slot, None),
getattr(objcopy, slot, None), msg=msg)
def check_unpickling_error(self, errors, data):
with self.subTest(data=data), \
self.assertRaises(errors):
def check_unpickling_error_strict(self, errors, data):
with self.assertRaises(errors):
try:
self.loads(data)
except BaseException as exc:
@ -864,6 +872,10 @@ def check_unpickling_error(self, errors, data):
(data, exc.__class__.__name__, exc))
raise
def check_unpickling_error(self, errors, data):
with self.subTest(data=data):
self.check_unpickling_error_strict(errors, data)
def test_load_from_data0(self):
self.assert_is_copy(self._testdata, self.loads(DATA0))
@ -1150,6 +1162,155 @@ def test_negative_32b_binput(self):
dumped = b'\x80\x03X\x01\x00\x00\x00ar\xff\xff\xff\xff.'
self.check_unpickling_error(ValueError, dumped)
def test_too_large_put(self):
# Test that PUT with large id does not cause allocation of
# too large memo table. The C implementation uses a dict-based memo
# for sparse indices (when idx > memo_len * 2) instead of allocating
# a massive array. This test verifies large sparse indices work without
# causing memory exhaustion.
#
# The following simple pickle creates an empty list, memoizes it
# using a large index, then loads it back on the stack, builds
# a tuple containing 2 identical empty lists and returns it.
data = lambda n: (b'((lp' + str(n).encode() + b'\n' +
b'g' + str(n).encode() + b'\nt.')
# 0: ( MARK
# 1: ( MARK
# 2: l LIST (MARK at 1)
# 3: p PUT 1000000000000
# 18: g GET 1000000000000
# 33: t TUPLE (MARK at 0)
# 34: . STOP
for idx in [10**6, 10**9, 10**12]:
if idx > sys.maxsize:
continue
self.assertEqual(self.loads(data(idx)), ([],)*2)
def test_too_large_long_binput(self):
# Test that LONG_BINPUT with large id does not cause allocation of
# too large memo table. The C implementation uses a dict-based memo
# for sparse indices (when idx > memo_len * 2) instead of allocating
# a massive array. This test verifies large sparse indices work without
# causing memory exhaustion.
#
# The following simple pickle creates an empty list, memoizes it
# using a large index, then loads it back on the stack, builds
# a tuple containing 2 identical empty lists and returns it.
data = lambda n: (b'(]r' + struct.pack('<I', n) +
b'j' + struct.pack('<I', n) + b't.')
# 0: ( MARK
# 1: ] EMPTY_LIST
# 2: r LONG_BINPUT 4294967295
# 7: j LONG_BINGET 4294967295
# 12: t TUPLE (MARK at 0)
# 13: . STOP
for idx in itersize(1 << 20, min(sys.maxsize, (1 << 32) - 1)):
self.assertEqual(self.loads(data(idx)), ([],)*2)
def _test_truncated_data(self, dumped, expected_error=None):
# Test that instructions to read large data without providing
# such amount of data do not cause large memory usage.
if expected_error is None:
expected_error = self.truncated_data_error
# BytesIO
with self.assertRaisesRegex(*expected_error):
self.loads(dumped)
if hasattr(self, 'unpickler'):
try:
with open(TESTFN, 'wb') as f:
f.write(dumped)
# buffered file
with open(TESTFN, 'rb') as f:
u = self.unpickler(f)
with self.assertRaisesRegex(*expected_error):
u.load()
# unbuffered file
with open(TESTFN, 'rb', buffering=0) as f:
u = self.unpickler(f)
with self.assertRaisesRegex(*expected_error):
u.load()
finally:
os_helper.unlink(TESTFN)
def test_truncated_large_binstring(self):
data = lambda size: b'T' + struct.pack('<I', size) + b'.' * 5
# 0: T BINSTRING '....'
# 9: . STOP
self.assertEqual(self.loads(data(4)), '....') # self-testing
for size in itersize(1 << 10, min(sys.maxsize - 5, (1 << 31) - 1)):
self._test_truncated_data(data(size))
self._test_truncated_data(data(1 << 31),
(pickle.UnpicklingError, 'truncated|exceeds|negative byte count'))
def test_truncated_large_binunicode(self):
data = lambda size: b'X' + struct.pack('<I', size) + b'.' * 5
# 0: X BINUNICODE '....'
# 9: . STOP
self.assertEqual(self.loads(data(4)), '....') # self-testing
for size in itersize(1 << 10, min(sys.maxsize - 5, (1 << 32) - 1)):
self._test_truncated_data(data(size))
def test_truncated_large_binbytes(self):
data = lambda size: b'B' + struct.pack('<I', size) + b'.' * 5
# 0: B BINBYTES b'....'
# 9: . STOP
self.assertEqual(self.loads(data(4)), b'....') # self-testing
for size in itersize(1 << 10, min(sys.maxsize, 1 << 31)):
self._test_truncated_data(data(size))
def test_truncated_large_long4(self):
data = lambda size: b'\x8b' + struct.pack('<I', size) + b'.' * 5
# 0: \x8b LONG4 0x2e2e2e2e
# 9: . STOP
self.assertEqual(self.loads(data(4)), 0x2e2e2e2e) # self-testing
for size in itersize(1 << 10, min(sys.maxsize - 5, (1 << 31) - 1)):
self._test_truncated_data(data(size))
self._test_truncated_data(data(1 << 31),
(pickle.UnpicklingError, 'LONG pickle has negative byte count'))
def test_truncated_large_frame(self):
data = lambda size: b'\x95' + struct.pack('<Q', size) + b'N.'
# 0: \x95 FRAME 2
# 9: N NONE
# 10: . STOP
self.assertIsNone(self.loads(data(2))) # self-testing
for size in itersize(1 << 10, sys.maxsize - 9):
self._test_truncated_data(data(size))
if sys.maxsize + 1 < 1 << 64:
self._test_truncated_data(data(sys.maxsize + 1),
((OverflowError, ValueError),
'FRAME length exceeds|frame size > sys.maxsize'))
def test_truncated_large_binunicode8(self):
data = lambda size: b'\x8d' + struct.pack('<Q', size) + b'.' * 5
# 0: \x8d BINUNICODE8 '....'
# 13: . STOP
self.assertEqual(self.loads(data(4)), '....') # self-testing
for size in itersize(1 << 10, sys.maxsize - 9):
self._test_truncated_data(data(size))
if sys.maxsize + 1 < 1 << 64:
self._test_truncated_data(data(sys.maxsize + 1), self.size_overflow_error)
def test_truncated_large_binbytes8(self):
data = lambda size: b'\x8e' + struct.pack('<Q', size) + b'.' * 5
# 0: \x8e BINBYTES8 b'....'
# 13: . STOP
self.assertEqual(self.loads(data(4)), b'....') # self-testing
for size in itersize(1 << 10, sys.maxsize):
self._test_truncated_data(data(size))
if sys.maxsize + 1 < 1 << 64:
self._test_truncated_data(data(sys.maxsize + 1), self.size_overflow_error)
def test_truncated_large_bytearray8(self):
data = lambda size: b'\x96' + struct.pack('<Q', size) + b'.' * 5
# 0: \x96 BYTEARRAY8 bytearray(b'....')
# 13: . STOP
self.assertEqual(self.loads(data(4)), bytearray(b'....')) # self-testing
for size in itersize(1 << 10, sys.maxsize):
self._test_truncated_data(data(size))
if sys.maxsize + 1 < 1 << 64:
self._test_truncated_data(data(sys.maxsize + 1), self.size_overflow_error)
def test_badly_escaped_string(self):
self.check_unpickling_error(ValueError, b"S'\\'\n.")

52
Lib/test/test_argparse.py
View file

@ -2770,6 +2770,16 @@ def test_optional_subparsers(self):
ret = parser.parse_args(())
self.assertIsNone(ret.command)
def test_subparser_help_with_parent_required_optional(self):
parser = ErrorRaisingArgumentParser(prog='PROG')
parser.add_argument('--foo', required=True)
parser.add_argument('--bar')
subparsers = parser.add_subparsers()
parser_sub = subparsers.add_parser('sub')
parser_sub.add_argument('arg')
self.assertEqual(parser_sub.format_usage(),
'usage: PROG --foo FOO sub [-h] arg\n')
def test_help(self):
self.assertEqual(self.parser.format_usage(),
'usage: PROG [-h] [--foo] bar {1,2,3} ...\n')
@ -4966,6 +4976,25 @@ def test_long_mutex_groups_wrap(self):
''')
self.assertEqual(parser.format_usage(), usage)
def test_mutex_groups_with_mixed_optionals_positionals_wrap(self):
# https://github.com/python/cpython/issues/75949
# Mutually exclusive groups containing both optionals and positionals
# should preserve pipe separators when the usage line wraps.
parser = argparse.ArgumentParser(prog='PROG')
g = parser.add_mutually_exclusive_group()
g.add_argument('-v', '--verbose', action='store_true')
g.add_argument('-q', '--quiet', action='store_true')
g.add_argument('-x', '--extra-long-option-name', nargs='?')
g.add_argument('-y', '--yet-another-long-option', nargs='?')
g.add_argument('positional', nargs='?')
usage = textwrap.dedent('''\
usage: PROG [-h] [-v | -q | -x [EXTRA_LONG_OPTION_NAME] |
-y [YET_ANOTHER_LONG_OPTION] |
positional]
''')
self.assertEqual(parser.format_usage(), usage)
class TestHelpVariableExpansion(HelpTestCase):
"""Test that variables are expanded properly in help messages"""
@ -7342,7 +7371,28 @@ def test_argparse_color(self):
),
)
def test_argparse_color_usage(self):
def test_argparse_color_mutually_exclusive_group_usage(self):
parser = argparse.ArgumentParser(color=True, prog="PROG")
group = parser.add_mutually_exclusive_group()
group.add_argument('--foo', action='store_true', help='FOO')
group.add_argument('--spam', help='SPAM')
group.add_argument('badger', nargs='*', help='BADGER')
prog = self.theme.prog
heading = self.theme.heading
long = self.theme.summary_long_option
short = self.theme.summary_short_option
label = self.theme.summary_label
pos = self.theme.summary_action
reset = self.theme.reset
self.assertEqual(parser.format_usage(),
f"{heading}usage: {reset}{prog}PROG{reset} [{short}-h{reset}] "
f"[{long}--foo{reset} | "
f"{long}--spam {label}SPAM{reset} | "
f"{pos}badger ...{reset}]\n")
def test_argparse_color_custom_usage(self):
# Arrange
parser = argparse.ArgumentParser(
add_help=False,

24
Lib/test/test_asyncio/test_tasks.py
View file

@ -3680,6 +3680,30 @@ def task_factory(loop, coro):
(loop, context), kwargs = callback.call_args
self.assertEqual(context['exception'], exc_context.exception)
def test_run_coroutine_threadsafe_and_cancel(self):
task = None
thread_future = None
# Use a custom task factory to capture the created Task
def task_factory(loop, coro):
nonlocal task
task = asyncio.Task(coro, loop=loop)
return task
self.addCleanup(self.loop.set_task_factory,
self.loop.get_task_factory())
async def target():
nonlocal thread_future
self.loop.set_task_factory(task_factory)
thread_future = asyncio.run_coroutine_threadsafe(asyncio.sleep(10), self.loop)
await asyncio.sleep(0)
thread_future.cancel()
self.loop.run_until_complete(target())
self.assertTrue(task.cancelled())
self.assertTrue(thread_future.cancelled())
class SleepTests(test_utils.TestCase):
def setUp(self):

11
Lib/test/test_bytes.py
View file

@ -549,6 +549,17 @@ def test_hex_separator_basics(self):
self.assertEqual(three_bytes.hex(':', 2), 'b9:01ef')
self.assertEqual(three_bytes.hex(':', 1), 'b9:01:ef')
self.assertEqual(three_bytes.hex('*', -2), 'b901*ef')
self.assertEqual(three_bytes.hex(sep=':', bytes_per_sep=2), 'b9:01ef')
self.assertEqual(three_bytes.hex(sep='*', bytes_per_sep=-2), 'b901*ef')
for bytes_per_sep in 3, -3, 2**31-1, -(2**31-1):
with self.subTest(bytes_per_sep=bytes_per_sep):
self.assertEqual(three_bytes.hex(':', bytes_per_sep), 'b901ef')
for bytes_per_sep in 2**31, -2**31, 2**1000, -2**1000:
with self.subTest(bytes_per_sep=bytes_per_sep):
try:
self.assertEqual(three_bytes.hex(':', bytes_per_sep), 'b901ef')
except OverflowError:
pass
value = b'{s\005\000\000\000worldi\002\000\000\000s\005\000\000\000helloi\001\000\000\0000'
self.assertEqual(value.hex('.', 8), '7b7305000000776f.726c646902000000.730500000068656c.6c6f690100000030')

28
Lib/test/test_dataclasses/__init__.py
View file

@ -927,6 +927,20 @@ class C:
validate_class(C)
def test_incomplete_annotations(self):
# gh-142214
@dataclass
class C:
"doc" # needed because otherwise we fetch the annotations at the wrong time
x: int
C.__annotate__ = lambda _: {}
self.assertEqual(
annotationlib.get_annotations(C.__init__),
{"return": None}
)
def test_missing_default(self):
# Test that MISSING works the same as a default not being
# specified.
@ -2578,6 +2592,20 @@ def __init__(self, x: int) -> None:
self.assertFalse(hasattr(E.__init__.__annotate__, "__generated_by_dataclasses__"))
def test_slots_true_init_false(self):
# Test that slots=True and init=False work together and
# that __annotate__ is not added to __init__.
@dataclass(slots=True, init=False)
class F:
x: int
f = F()
f.x = 10
self.assertEqual(f.x, 10)
self.assertFalse(hasattr(F.__init__, "__annotate__"))
def test_init_false_forwardref(self):
# Test forward references in fields not required for __init__ annotations.

8
Lib/test/test_dict.py
View file

@ -1621,6 +1621,14 @@ def __eq__(self, other):
self.assertEqual(len(d), 1)
def test_split_table_update_with_str_subclass(self):
class MyStr(str): pass
class MyClass: pass
obj = MyClass()
obj.attr = 1
obj.__dict__[MyStr('attr')] = 2
self.assertEqual(obj.attr, 2)
class CAPITest(unittest.TestCase):

122
Lib/test/test_doctest/test_doctest.py
View file

@ -833,6 +833,118 @@ def test_empty_namespace_package(self):
self.assertEqual(len(include_empty_finder.find(mod)), 1)
self.assertEqual(len(exclude_empty_finder.find(mod)), 0)
def test_lineno_of_test_dict_strings(self):
"""Test line numbers are found for __test__ dict strings."""
module_content = '''\
"""Module docstring."""
def dummy_function():
"""Dummy function docstring."""
pass
__test__ = {
'test_string': """
This is a test string.
>>> 1 + 1
2
""",
}
'''
with tempfile.TemporaryDirectory() as tmpdir:
module_path = os.path.join(tmpdir, 'test_module_lineno.py')
with open(module_path, 'w') as f:
f.write(module_content)
sys.path.insert(0, tmpdir)
try:
import test_module_lineno
finder = doctest.DocTestFinder()
tests = finder.find(test_module_lineno)
test_dict_test = None
for test in tests:
if '__test__' in test.name:
test_dict_test = test
break
self.assertIsNotNone(
test_dict_test,
"__test__ dict test not found"
)
# gh-69113: line number should not be None for __test__ strings
self.assertIsNotNone(
test_dict_test.lineno,
"Line number should not be None for __test__ dict strings"
)
self.assertGreater(
test_dict_test.lineno,
0,
"Line number should be positive"
)
finally:
if 'test_module_lineno' in sys.modules:
del sys.modules['test_module_lineno']
sys.path.pop(0)
def test_lineno_multiline_matching(self):
"""Test multi-line matching when no unique line exists."""
# gh-69113: test that line numbers are found even when lines
# appear multiple times (e.g., ">>> x = 1" in both test entries)
module_content = '''\
"""Module docstring."""
__test__ = {
'test_one': """
>>> x = 1
>>> x
1
""",
'test_two': """
>>> x = 1
>>> x
2
""",
}
'''
with tempfile.TemporaryDirectory() as tmpdir:
module_path = os.path.join(tmpdir, 'test_module_multiline.py')
with open(module_path, 'w') as f:
f.write(module_content)
sys.path.insert(0, tmpdir)
try:
import test_module_multiline
finder = doctest.DocTestFinder()
tests = finder.find(test_module_multiline)
test_one = None
test_two = None
for test in tests:
if 'test_one' in test.name:
test_one = test
elif 'test_two' in test.name:
test_two = test
self.assertIsNotNone(test_one, "test_one not found")
self.assertIsNotNone(test_two, "test_two not found")
self.assertIsNotNone(
test_one.lineno,
"Line number should not be None for test_one"
)
self.assertIsNotNone(
test_two.lineno,
"Line number should not be None for test_two"
)
self.assertNotEqual(
test_one.lineno,
test_two.lineno,
"test_one and test_two should have different line numbers"
)
finally:
if 'test_module_multiline' in sys.modules:
del sys.modules['test_module_multiline']
sys.path.pop(0)
def test_DocTestParser(): r"""
Unit tests for the `DocTestParser` class.
@ -2434,7 +2546,8 @@ def test_DocTestSuite_errors():
<BLANKLINE>
>>> print(result.failures[1][1]) # doctest: +ELLIPSIS
Traceback (most recent call last):
File "...sample_doctest_errors.py", line None, in test.test_doctest.sample_doctest_errors.__test__.bad
File "...sample_doctest_errors.py", line 37, in test.test_doctest.sample_doctest_errors.__test__.bad
>...>> 2 + 2
AssertionError: Failed example:
2 + 2
Expected:
@ -2464,7 +2577,8 @@ def test_DocTestSuite_errors():
<BLANKLINE>
>>> print(result.errors[1][1]) # doctest: +ELLIPSIS
Traceback (most recent call last):
File "...sample_doctest_errors.py", line None, in test.test_doctest.sample_doctest_errors.__test__.bad
File "...sample_doctest_errors.py", line 39, in test.test_doctest.sample_doctest_errors.__test__.bad
>...>> 1/0
File "<doctest test.test_doctest.sample_doctest_errors.__test__.bad[1]>", line 1, in <module>
1/0
~^~
@ -3256,7 +3370,7 @@ def test_testmod_errors(): r"""
~^~
ZeroDivisionError: division by zero
**********************************************************************
File "...sample_doctest_errors.py", line ?, in test.test_doctest.sample_doctest_errors.__test__.bad
File "...sample_doctest_errors.py", line 37, in test.test_doctest.sample_doctest_errors.__test__.bad
Failed example:
2 + 2
Expected:
@ -3264,7 +3378,7 @@ def test_testmod_errors(): r"""
Got:
4
**********************************************************************
File "...sample_doctest_errors.py", line ?, in test.test_doctest.sample_doctest_errors.__test__.bad
File "...sample_doctest_errors.py", line 39, in test.test_doctest.sample_doctest_errors.__test__.bad
Failed example:
1/0
Exception raised:

10
Lib/test/test_email/test__header_value_parser.py
View file

@ -3255,5 +3255,15 @@ def test_long_filename_attachment(self):
" filename*1*=_TEST_TES.txt\n",
)
def test_fold_unfoldable_element_stealing_whitespace(self):
# gh-142006: When an element is too long to fit on the current line
# the previous line's trailing whitespace should not trigger a double newline.
policy = self.policy.clone(max_line_length=10)
# The non-whitespace text needs to exactly fill the max_line_length (10).
text = ("a" * 9) + ", " + ("b" * 20)
expected = ("a" * 9) + ",\n " + ("b" * 20) + "\n"
token = parser.get_address_list(text)[0]
self._test(token, expected, policy=policy)
if __name__ == '__main__':
unittest.main()

46
Lib/test/test_email/test_defect_handling.py
View file

@ -126,12 +126,10 @@ def test_multipart_invalid_cte(self):
errors.InvalidMultipartContentTransferEncodingDefect)
def test_multipart_no_cte_no_defect(self):
if self.raise_expected: return
msg = self._str_msg(self.multipart_msg.format(''))
self.assertEqual(len(self.get_defects(msg)), 0)
def test_multipart_valid_cte_no_defect(self):
if self.raise_expected: return
for cte in ('7bit', '8bit', 'BINary'):
msg = self._str_msg(
self.multipart_msg.format("\nContent-Transfer-Encoding: "+cte))
@ -300,6 +298,47 @@ def test_missing_ending_boundary(self):
self.assertDefectsEqual(self.get_defects(msg),
[errors.CloseBoundaryNotFoundDefect])
def test_line_beginning_colon(self):
string = (
"Subject: Dummy subject\r\n: faulty header line\r\n\r\nbody\r\n"
)
with self._raise_point(errors.InvalidHeaderDefect):
msg = self._str_msg(string)
self.assertEqual(len(self.get_defects(msg)), 1)
self.assertDefectsEqual(
self.get_defects(msg), [errors.InvalidHeaderDefect]
)
if msg:
self.assertEqual(msg.items(), [("Subject", "Dummy subject")])
self.assertEqual(msg.get_payload(), "body\r\n")
def test_misplaced_envelope(self):
string = (
"Subject: Dummy subject\r\nFrom wtf\r\nTo: abc\r\n\r\nbody\r\n"
)
with self._raise_point(errors.MisplacedEnvelopeHeaderDefect):
msg = self._str_msg(string)
self.assertEqual(len(self.get_defects(msg)), 1)
self.assertDefectsEqual(
self.get_defects(msg), [errors.MisplacedEnvelopeHeaderDefect]
)
if msg:
headers = [("Subject", "Dummy subject"), ("To", "abc")]
self.assertEqual(msg.items(), headers)
self.assertEqual(msg.get_payload(), "body\r\n")
class TestCompat32(TestDefectsBase, TestEmailBase):
policy = policy.compat32
def get_defects(self, obj):
return obj.defects
class TestDefectDetection(TestDefectsBase, TestEmailBase):
@ -332,6 +371,9 @@ def _raise_point(self, defect):
with self.assertRaises(defect):
yield
def get_defects(self, obj):
return obj.defects
if __name__ == '__main__':
unittest.main()

88
Lib/test/test_email/test_email.py
View file

@ -2263,70 +2263,6 @@ def test_parse_missing_minor_type(self):
eq(msg.get_content_maintype(), 'text')
eq(msg.get_content_subtype(), 'plain')
# test_defect_handling
def test_same_boundary_inner_outer(self):
msg = self._msgobj('msg_15.txt')
# XXX We can probably eventually do better
inner = msg.get_payload(0)
self.assertHasAttr(inner, 'defects')
self.assertEqual(len(inner.defects), 1)
self.assertIsInstance(inner.defects[0],
errors.StartBoundaryNotFoundDefect)
# test_defect_handling
def test_multipart_no_boundary(self):
msg = self._msgobj('msg_25.txt')
self.assertIsInstance(msg.get_payload(), str)
self.assertEqual(len(msg.defects), 2)
self.assertIsInstance(msg.defects[0],
errors.NoBoundaryInMultipartDefect)
self.assertIsInstance(msg.defects[1],
errors.MultipartInvariantViolationDefect)
multipart_msg = textwrap.dedent("""\
Date: Wed, 14 Nov 2007 12:56:23 GMT
From: foo@bar.invalid
To: foo@bar.invalid
Subject: Content-Transfer-Encoding: base64 and multipart
MIME-Version: 1.0
Content-Type: multipart/mixed;
boundary="===============3344438784458119861=="{}
--===============3344438784458119861==
Content-Type: text/plain
Test message
--===============3344438784458119861==
Content-Type: application/octet-stream
Content-Transfer-Encoding: base64
YWJj
--===============3344438784458119861==--
""")
# test_defect_handling
def test_multipart_invalid_cte(self):
msg = self._str_msg(
self.multipart_msg.format("\nContent-Transfer-Encoding: base64"))
self.assertEqual(len(msg.defects), 1)
self.assertIsInstance(msg.defects[0],
errors.InvalidMultipartContentTransferEncodingDefect)
# test_defect_handling
def test_multipart_no_cte_no_defect(self):
msg = self._str_msg(self.multipart_msg.format(''))
self.assertEqual(len(msg.defects), 0)
# test_defect_handling
def test_multipart_valid_cte_no_defect(self):
for cte in ('7bit', '8bit', 'BINary'):
msg = self._str_msg(
self.multipart_msg.format(
"\nContent-Transfer-Encoding: {}".format(cte)))
self.assertEqual(len(msg.defects), 0)
# test_headerregistry.TestContentTypeHeader invalid_1 and invalid_2.
def test_invalid_content_type(self):
eq = self.assertEqual
@ -2403,30 +2339,6 @@ def test_missing_start_boundary(self):
self.assertIsInstance(bad.defects[0],
errors.StartBoundaryNotFoundDefect)
# test_defect_handling
def test_first_line_is_continuation_header(self):
eq = self.assertEqual
m = ' Line 1\nSubject: test\n\nbody'
msg = email.message_from_string(m)
eq(msg.keys(), ['Subject'])
eq(msg.get_payload(), 'body')
eq(len(msg.defects), 1)
self.assertDefectsEqual(msg.defects,
[errors.FirstHeaderLineIsContinuationDefect])
eq(msg.defects[0].line, ' Line 1\n')
# test_defect_handling
def test_missing_header_body_separator(self):
# Our heuristic if we see a line that doesn't look like a header (no
# leading whitespace but no ':') is to assume that the blank line that
# separates the header from the body is missing, and to stop parsing
# headers and start parsing the body.
msg = self._str_msg('Subject: test\nnot a header\nTo: abc\n\nb\n')
self.assertEqual(msg.keys(), ['Subject'])
self.assertEqual(msg.get_payload(), 'not a header\nTo: abc\n\nb\n')
self.assertDefectsEqual(msg.defects,
[errors.MissingHeaderBodySeparatorDefect])
def test_string_payload_with_extra_space_after_cte(self):
# https://github.com/python/cpython/issues/98188
cte = "base64 "

4301
Lib/test/test_external_inspection.py
View file

File diff suppressed because it is too large Load diff

58
Lib/test/test_generated_cases.py
View file

@ -165,7 +165,7 @@ def test_inst_one_pop(self):
value = stack_pointer[-1];
SPAM(value);
stack_pointer += -1;
assert(WITHIN_STACK_BOUNDS());
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
DISPATCH();
}
"""
@ -190,7 +190,7 @@ def test_inst_one_push(self):
res = SPAM();
stack_pointer[0] = res;
stack_pointer += 1;
assert(WITHIN_STACK_BOUNDS());
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
DISPATCH();
}
"""
@ -247,7 +247,7 @@ def test_binary_op(self):
res = SPAM(left, right);
stack_pointer[-2] = res;
stack_pointer += -1;
assert(WITHIN_STACK_BOUNDS());
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
DISPATCH();
}
"""
@ -366,14 +366,14 @@ def test_sync_sp(self):
_PyStackRef res;
arg = stack_pointer[-1];
stack_pointer += -1;
assert(WITHIN_STACK_BOUNDS());
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
_PyFrame_SetStackPointer(frame, stack_pointer);
escaping_call();
stack_pointer = _PyFrame_GetStackPointer(frame);
res = Py_None;
stack_pointer[0] = res;
stack_pointer += 1;
assert(WITHIN_STACK_BOUNDS());
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
DISPATCH();
}
@ -489,7 +489,7 @@ def test_error_if_pop(self):
res = 0;
stack_pointer[-2] = res;
stack_pointer += -1;
assert(WITHIN_STACK_BOUNDS());
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
DISPATCH();
}
"""
@ -523,7 +523,7 @@ def test_error_if_pop_with_result(self):
}
stack_pointer[-2] = res;
stack_pointer += -1;
assert(WITHIN_STACK_BOUNDS());
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
DISPATCH();
}
"""
@ -553,7 +553,7 @@ def test_cache_effect(self):
uint32_t extra = read_u32(&this_instr[2].cache);
(void)extra;
stack_pointer += -1;
assert(WITHIN_STACK_BOUNDS());
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
DISPATCH();
}
"""
@ -640,7 +640,7 @@ def test_macro_instruction(self):
}
stack_pointer[-3] = res;
stack_pointer += -2;
assert(WITHIN_STACK_BOUNDS());
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
DISPATCH();
}
@ -688,7 +688,7 @@ def test_macro_instruction(self):
stack_pointer = _PyFrame_GetStackPointer(frame);
stack_pointer[-3] = res;
stack_pointer += -2;
assert(WITHIN_STACK_BOUNDS());
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
DISPATCH();
}
"""
@ -827,7 +827,7 @@ def test_array_input(self):
below = stack_pointer[-2 - oparg*2];
SPAM(values, oparg);
stack_pointer += -2 - oparg*2;
assert(WITHIN_STACK_BOUNDS());
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
DISPATCH();
}
"""
@ -860,7 +860,7 @@ def test_array_output(self):
stack_pointer[-2] = below;
stack_pointer[-1 + oparg*3] = above;
stack_pointer += oparg*3;
assert(WITHIN_STACK_BOUNDS());
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
DISPATCH();
}
"""
@ -889,7 +889,7 @@ def test_array_input_output(self):
above = 0;
stack_pointer[0] = above;
stack_pointer += 1;
assert(WITHIN_STACK_BOUNDS());
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
DISPATCH();
}
"""
@ -918,11 +918,11 @@ def test_array_error_if(self):
extra = stack_pointer[-1 - oparg];
if (oparg == 0) {
stack_pointer += -1 - oparg;
assert(WITHIN_STACK_BOUNDS());
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
JUMP_TO_LABEL(error);
}
stack_pointer += -1 - oparg;
assert(WITHIN_STACK_BOUNDS());
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
DISPATCH();
}
"""
@ -960,7 +960,7 @@ def test_macro_push_push(self):
stack_pointer[0] = val1;
stack_pointer[1] = val2;
stack_pointer += 2;
assert(WITHIN_STACK_BOUNDS());
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
DISPATCH();
}
"""
@ -1263,13 +1263,13 @@ def test_flush(self):
stack_pointer[0] = a;
stack_pointer[1] = b;
stack_pointer += 2;
assert(WITHIN_STACK_BOUNDS());
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
// SECOND
{
USE(a, b);
}
stack_pointer += -2;
assert(WITHIN_STACK_BOUNDS());
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
DISPATCH();
}
"""
@ -1325,7 +1325,7 @@ def test_pop_on_error_peeks(self):
}
}
stack_pointer += -2;
assert(WITHIN_STACK_BOUNDS());
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
DISPATCH();
}
"""
@ -1368,14 +1368,14 @@ def test_push_then_error(self):
stack_pointer[0] = a;
stack_pointer[1] = b;
stack_pointer += 2;
assert(WITHIN_STACK_BOUNDS());
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
JUMP_TO_LABEL(error);
}
}
stack_pointer[0] = a;
stack_pointer[1] = b;
stack_pointer += 2;
assert(WITHIN_STACK_BOUNDS());
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
DISPATCH();
}
"""
@ -1661,7 +1661,7 @@ def test_pystackref_frompyobject_new_next_to_cmacro(self):
stack_pointer[0] = out1;
stack_pointer[1] = out2;
stack_pointer += 2;
assert(WITHIN_STACK_BOUNDS());
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
DISPATCH();
}
"""
@ -1881,7 +1881,7 @@ def test_reassigning_dead_inputs(self):
stack_pointer = _PyFrame_GetStackPointer(frame);
in = temp;
stack_pointer += -1;
assert(WITHIN_STACK_BOUNDS());
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
_PyFrame_SetStackPointer(frame, stack_pointer);
PyStackRef_CLOSE(in);
stack_pointer = _PyFrame_GetStackPointer(frame);
@ -2115,8 +2115,9 @@ def test_validate_uop_unused_input(self):
"""
output = """
case OP: {
CHECK_STACK_BOUNDS(-1);
stack_pointer += -1;
assert(WITHIN_STACK_BOUNDS());
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
break;
}
"""
@ -2132,8 +2133,9 @@ def test_validate_uop_unused_input(self):
"""
output = """
case OP: {
CHECK_STACK_BOUNDS(-1);
stack_pointer += -1;
assert(WITHIN_STACK_BOUNDS());
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
break;
}
"""
@ -2153,9 +2155,10 @@ def test_validate_uop_unused_output(self):
case OP: {
JitOptRef foo;
foo = NULL;
CHECK_STACK_BOUNDS(1);
stack_pointer[0] = foo;
stack_pointer += 1;
assert(WITHIN_STACK_BOUNDS());
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
break;
}
"""
@ -2172,8 +2175,9 @@ def test_validate_uop_unused_output(self):
"""
output = """
case OP: {
CHECK_STACK_BOUNDS(1);
stack_pointer += 1;
assert(WITHIN_STACK_BOUNDS());
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
break;
}
"""

5
Lib/test/test_marshal.py
View file

@ -43,6 +43,11 @@ def test_ints(self):
for expected in (-n, n):
self.helper(expected)
n = n >> 1
n = 1 << 100
while n:
for expected in (-n, -n+1, n-1, n):
self.helper(expected)
n = n >> 1
def test_int64(self):
# Simulate int marshaling with TYPE_INT64.

19
Lib/test/test_memoryview.py
View file

@ -600,6 +600,25 @@ def test_memoryview_hex(self):
m2 = m1[::-1]
self.assertEqual(m2.hex(), '30' * 200000)
def test_memoryview_hex_separator(self):
x = bytes(range(97, 102))
m1 = memoryview(x)
m2 = m1[::-1]
self.assertEqual(m2.hex(':'), '65:64:63:62:61')
self.assertEqual(m2.hex(':', 2), '65:6463:6261')
self.assertEqual(m2.hex(':', -2), '6564:6362:61')
self.assertEqual(m2.hex(sep=':', bytes_per_sep=2), '65:6463:6261')
self.assertEqual(m2.hex(sep=':', bytes_per_sep=-2), '6564:6362:61')
for bytes_per_sep in 5, -5, 2**31-1, -(2**31-1):
with self.subTest(bytes_per_sep=bytes_per_sep):
self.assertEqual(m2.hex(':', bytes_per_sep), '6564636261')
for bytes_per_sep in 2**31, -2**31, 2**1000, -2**1000:
with self.subTest(bytes_per_sep=bytes_per_sep):
try:
self.assertEqual(m2.hex(':', bytes_per_sep), '6564636261')
except OverflowError:
pass
def test_copy(self):
m = memoryview(b'abc')
with self.assertRaises(TypeError):

18
Lib/test/test_minidom.py
View file

@ -2,6 +2,7 @@
import copy
import pickle
import time
import io
from test import support
import unittest
@ -173,6 +174,23 @@ def testAppendChild(self):
self.assertEqual(dom.documentElement.childNodes[-1].data, "Hello")
dom.unlink()
def testAppendChildNoQuadraticComplexity(self):
impl = getDOMImplementation()
newdoc = impl.createDocument(None, "some_tag", None)
top_element = newdoc.documentElement
children = [newdoc.createElement(f"child-{i}") for i in range(1, 2 ** 15 + 1)]
element = top_element
start = time.time()
for child in children:
element.appendChild(child)
element = child
end = time.time()
# This example used to take at least 30 seconds.
self.assertLess(end - start, 1)
def testAppendChildFragment(self):
dom, orig, c1, c2, c3, frag = self._create_fragment_test_nodes()
dom.documentElement.appendChild(frag)

8
Lib/test/test_pickle.py
View file

@ -59,6 +59,8 @@ class PyUnpicklerTests(AbstractUnpickleTests, unittest.TestCase):
truncated_errors = (pickle.UnpicklingError, EOFError,
AttributeError, ValueError,
struct.error, IndexError, ImportError)
truncated_data_error = (EOFError, '')
size_overflow_error = (pickle.UnpicklingError, 'exceeds')
def loads(self, buf, **kwds):
f = io.BytesIO(buf)
@ -103,6 +105,8 @@ class InMemoryPickleTests(AbstractPickleTests, AbstractUnpickleTests,
truncated_errors = (pickle.UnpicklingError, EOFError,
AttributeError, ValueError,
struct.error, IndexError, ImportError)
truncated_data_error = ((pickle.UnpicklingError, EOFError), '')
size_overflow_error = ((OverflowError, pickle.UnpicklingError), 'exceeds')
def dumps(self, arg, protocol=None, **kwargs):
return pickle.dumps(arg, protocol, **kwargs)
@ -375,6 +379,8 @@ class CUnpicklerTests(PyUnpicklerTests):
unpickler = _pickle.Unpickler
bad_stack_errors = (pickle.UnpicklingError,)
truncated_errors = (pickle.UnpicklingError,)
truncated_data_error = (pickle.UnpicklingError, 'truncated')
size_overflow_error = (OverflowError, 'exceeds')
class CPicklingErrorTests(PyPicklingErrorTests):
pickler = _pickle.Pickler
@ -478,7 +484,7 @@ def test_pickler(self):
0) # Write buffer is cleared after every dump().
def test_unpickler(self):
basesize = support.calcobjsize('2P2n2P 2P2n2i5P 2P3n8P2n2i')
basesize = support.calcobjsize('2P2n3P 2P2n2i5P 2P3n8P2n2i')
unpickler = _pickle.Unpickler
P = struct.calcsize('P') # Size of memo table entry.
n = struct.calcsize('n') # Size of mark table entry.

6
Lib/test/test_profiling/test_heatmap.py
View file

@ -151,12 +151,6 @@ def test_init_sets_total_samples_to_zero(self):
collector = HeatmapCollector(sample_interval_usec=100)
self.assertEqual(collector._total_samples, 0)
def test_init_creates_color_cache(self):
"""Test that color cache is initialized."""
collector = HeatmapCollector(sample_interval_usec=100)
self.assertIsInstance(collector._color_cache, dict)
self.assertEqual(len(collector._color_cache), 0)
def test_init_gets_path_info(self):
"""Test that path info is retrieved during init."""
collector = HeatmapCollector(sample_interval_usec=100)

35
Lib/test/test_profiling/test_sampling_profiler/mocks.py
View file

@ -56,3 +56,38 @@ def __init__(self, interpreter_id, threads):
def __repr__(self):
return f"MockInterpreterInfo(interpreter_id={self.interpreter_id}, threads={self.threads})"
class MockCoroInfo:
"""Mock CoroInfo for testing async tasks."""
def __init__(self, task_name, call_stack):
self.task_name = task_name # In reality, this is the parent task ID
self.call_stack = call_stack
def __repr__(self):
return f"MockCoroInfo(task_name={self.task_name}, call_stack={self.call_stack})"
class MockTaskInfo:
"""Mock TaskInfo for testing async tasks."""
def __init__(self, task_id, task_name, coroutine_stack, awaited_by=None):
self.task_id = task_id
self.task_name = task_name
self.coroutine_stack = coroutine_stack # List of CoroInfo objects
self.awaited_by = awaited_by or [] # List of CoroInfo objects (parents)
def __repr__(self):
return f"MockTaskInfo(task_id={self.task_id}, task_name={self.task_name})"
class MockAwaitedInfo:
"""Mock AwaitedInfo for testing async tasks."""
def __init__(self, thread_id, awaited_by):
self.thread_id = thread_id
self.awaited_by = awaited_by # List of TaskInfo objects
def __repr__(self):
return f"MockAwaitedInfo(thread_id={self.thread_id}, awaited_by={len(self.awaited_by)} tasks)"

799
Lib/test/test_profiling/test_sampling_profiler/test_async.py Normal file
View file

@ -0,0 +1,799 @@
"""Tests for async stack reconstruction in the sampling profiler.
Each test covers a distinct algorithm path or edge case:
1. Graph building: _build_task_graph()
2. Leaf identification: _find_leaf_tasks()
3. Stack traversal: _build_linear_stacks() with BFS
"""
import unittest
try:
import _remote_debugging # noqa: F401
from profiling.sampling.pstats_collector import PstatsCollector
except ImportError:
raise unittest.SkipTest(
"Test only runs when _remote_debugging is available"
)
from .mocks import MockFrameInfo, MockCoroInfo, MockTaskInfo, MockAwaitedInfo
class TestAsyncStackReconstruction(unittest.TestCase):
"""Test async task tree linear stack reconstruction algorithm."""
def test_empty_input(self):
"""Test _build_task_graph with empty awaited_info_list."""
collector = PstatsCollector(sample_interval_usec=1000)
stacks = list(collector._iter_async_frames([]))
self.assertEqual(len(stacks), 0)
def test_single_root_task(self):
"""Test _find_leaf_tasks: root task with no parents is its own leaf."""
collector = PstatsCollector(sample_interval_usec=1000)
root = MockTaskInfo(
task_id=123,
task_name="Task-1",
coroutine_stack=[
MockCoroInfo(
task_name="Task-1",
call_stack=[MockFrameInfo("main.py", 10, "main")]
)
],
awaited_by=[]
)
awaited_info_list = [MockAwaitedInfo(thread_id=100, awaited_by=[root])]
stacks = list(collector._iter_async_frames(awaited_info_list))
# Single root is both leaf and root
self.assertEqual(len(stacks), 1)
frames, thread_id, leaf_id = stacks[0]
self.assertEqual(leaf_id, 123)
self.assertEqual(thread_id, 100)
def test_parent_child_chain(self):
"""Test _build_linear_stacks: BFS follows parent links from leaf to root.
Task graph:
Parent (id=1)
|
Child (id=2)
"""
collector = PstatsCollector(sample_interval_usec=1000)
child = MockTaskInfo(
task_id=2,
task_name="Child",
coroutine_stack=[
MockCoroInfo(task_name="Child", call_stack=[MockFrameInfo("c.py", 5, "child_fn")])
],
awaited_by=[
MockCoroInfo(task_name=1, call_stack=[MockFrameInfo("p.py", 10, "parent_await")])
]
)
parent = MockTaskInfo(
task_id=1,
task_name="Parent",
coroutine_stack=[
MockCoroInfo(task_name="Parent", call_stack=[MockFrameInfo("p.py", 15, "parent_fn")])
],
awaited_by=[]
)
awaited_info_list = [MockAwaitedInfo(thread_id=200, awaited_by=[child, parent])]
stacks = list(collector._iter_async_frames(awaited_info_list))
# Leaf is child, traverses to parent
self.assertEqual(len(stacks), 1)
frames, thread_id, leaf_id = stacks[0]
self.assertEqual(leaf_id, 2)
# Verify both child and parent frames present
func_names = [f.funcname for f in frames]
self.assertIn("child_fn", func_names)
self.assertIn("parent_fn", func_names)
def test_multiple_leaf_tasks(self):
"""Test _find_leaf_tasks: identifies multiple leaves correctly.
Task graph (fan-out from root):
Root (id=1)
/ \
Leaf1 (id=10) Leaf2 (id=20)
Expected: 2 stacks (one for each leaf).
"""
collector = PstatsCollector(sample_interval_usec=1000)
leaf1 = MockTaskInfo(
task_id=10,
task_name="Leaf1",
coroutine_stack=[MockCoroInfo(task_name="Leaf1", call_stack=[MockFrameInfo("l1.py", 1, "f1")])],
awaited_by=[MockCoroInfo(task_name=1, call_stack=[MockFrameInfo("r.py", 5, "root")])]
)
leaf2 = MockTaskInfo(
task_id=20,
task_name="Leaf2",
coroutine_stack=[MockCoroInfo(task_name="Leaf2", call_stack=[MockFrameInfo("l2.py", 2, "f2")])],
awaited_by=[MockCoroInfo(task_name=1, call_stack=[MockFrameInfo("r.py", 5, "root")])]
)
root = MockTaskInfo(
task_id=1,
task_name="Root",
coroutine_stack=[MockCoroInfo(task_name="Root", call_stack=[MockFrameInfo("r.py", 10, "main")])],
awaited_by=[]
)
awaited_info_list = [MockAwaitedInfo(thread_id=300, awaited_by=[leaf1, leaf2, root])]
stacks = list(collector._iter_async_frames(awaited_info_list))
# Two leaves = two stacks
self.assertEqual(len(stacks), 2)
leaf_ids = {leaf_id for _, _, leaf_id in stacks}
self.assertEqual(leaf_ids, {10, 20})
def test_cycle_detection(self):
"""Test _build_linear_stacks: cycle detection prevents infinite loops.
Task graph (cyclic dependency):
A (id=1) <---> B (id=2)
Neither task is a leaf (both have parents), so no stacks are produced.
"""
collector = PstatsCollector(sample_interval_usec=1000)
task_a = MockTaskInfo(
task_id=1,
task_name="A",
coroutine_stack=[MockCoroInfo(task_name="A", call_stack=[MockFrameInfo("a.py", 1, "a")])],
awaited_by=[MockCoroInfo(task_name=2, call_stack=[MockFrameInfo("b.py", 5, "b")])]
)
task_b = MockTaskInfo(
task_id=2,
task_name="B",
coroutine_stack=[MockCoroInfo(task_name="B", call_stack=[MockFrameInfo("b.py", 10, "b")])],
awaited_by=[MockCoroInfo(task_name=1, call_stack=[MockFrameInfo("a.py", 15, "a")])]
)
awaited_info_list = [MockAwaitedInfo(thread_id=400, awaited_by=[task_a, task_b])]
stacks = list(collector._iter_async_frames(awaited_info_list))
# No leaves (both have parents), should return empty
self.assertEqual(len(stacks), 0)
def test_orphaned_parent_reference(self):
"""Test _build_linear_stacks: handles parent ID not in task_map."""
collector = PstatsCollector(sample_interval_usec=1000)
# Task references non-existent parent
orphan = MockTaskInfo(
task_id=5,
task_name="Orphan",
coroutine_stack=[MockCoroInfo(task_name="Orphan", call_stack=[MockFrameInfo("o.py", 1, "orphan")])],
awaited_by=[MockCoroInfo(task_name=999, call_stack=[])] # 999 doesn't exist
)
awaited_info_list = [MockAwaitedInfo(thread_id=500, awaited_by=[orphan])]
stacks = list(collector._iter_async_frames(awaited_info_list))
# Stops at missing parent, yields what it has
self.assertEqual(len(stacks), 1)
frames, _, leaf_id = stacks[0]
self.assertEqual(leaf_id, 5)
def test_multiple_coroutines_per_task(self):
"""Test _build_linear_stacks: collects frames from all coroutines in task."""
collector = PstatsCollector(sample_interval_usec=1000)
# Task with multiple coroutines (e.g., nested async generators)
task = MockTaskInfo(
task_id=7,
task_name="Multi",
coroutine_stack=[
MockCoroInfo(task_name="Multi", call_stack=[MockFrameInfo("g.py", 5, "gen1")]),
MockCoroInfo(task_name="Multi", call_stack=[MockFrameInfo("g.py", 10, "gen2")]),
],
awaited_by=[]
)
awaited_info_list = [MockAwaitedInfo(thread_id=600, awaited_by=[task])]
stacks = list(collector._iter_async_frames(awaited_info_list))
self.assertEqual(len(stacks), 1)
frames, _, _ = stacks[0]
# Both coroutine frames should be present
func_names = [f.funcname for f in frames]
self.assertIn("gen1", func_names)
self.assertIn("gen2", func_names)
def test_multiple_threads(self):
"""Test _build_task_graph: handles multiple AwaitedInfo (different threads)."""
collector = PstatsCollector(sample_interval_usec=1000)
# Two threads with separate task trees
thread1_task = MockTaskInfo(
task_id=100,
task_name="T1",
coroutine_stack=[MockCoroInfo(task_name="T1", call_stack=[MockFrameInfo("t1.py", 1, "t1")])],
awaited_by=[]
)
thread2_task = MockTaskInfo(
task_id=200,
task_name="T2",
coroutine_stack=[MockCoroInfo(task_name="T2", call_stack=[MockFrameInfo("t2.py", 1, "t2")])],
awaited_by=[]
)
awaited_info_list = [
MockAwaitedInfo(thread_id=1, awaited_by=[thread1_task]),
MockAwaitedInfo(thread_id=2, awaited_by=[thread2_task]),
]
stacks = list(collector._iter_async_frames(awaited_info_list))
# Two threads = two stacks
self.assertEqual(len(stacks), 2)
# Verify thread IDs preserved
thread_ids = {thread_id for _, thread_id, _ in stacks}
self.assertEqual(thread_ids, {1, 2})
def test_collect_public_interface(self):
"""Test collect() method correctly routes to async frame processing."""
collector = PstatsCollector(sample_interval_usec=1000)
child = MockTaskInfo(
task_id=50,
task_name="Child",
coroutine_stack=[MockCoroInfo(task_name="Child", call_stack=[MockFrameInfo("c.py", 1, "child")])],
awaited_by=[MockCoroInfo(task_name=51, call_stack=[])]
)
parent = MockTaskInfo(
task_id=51,
task_name="Parent",
coroutine_stack=[MockCoroInfo(task_name="Parent", call_stack=[MockFrameInfo("p.py", 1, "parent")])],
awaited_by=[]
)
awaited_info_list = [MockAwaitedInfo(thread_id=999, awaited_by=[child, parent])]
# Public interface: collect()
collector.collect(awaited_info_list)
# Verify stats collected
self.assertGreater(len(collector.result), 0)
func_names = [loc[2] for loc in collector.result.keys()]
self.assertIn("child", func_names)
self.assertIn("parent", func_names)
def test_diamond_pattern_multiple_parents(self):
"""Test _build_linear_stacks: task with 2+ parents picks one deterministically.
CRITICAL: Tests that when a task has multiple parents, we pick one parent
deterministically (sorted, first one) and annotate the task name with parent count.
"""
collector = PstatsCollector(sample_interval_usec=1000)
# Diamond pattern: Root spawns A and B, both await Child
#
# Root (id=1)
# / \
# A (id=2) B (id=3)
# \ /
# Child (id=4)
#
child = MockTaskInfo(
task_id=4,
task_name="Child",
coroutine_stack=[MockCoroInfo(task_name="Child", call_stack=[MockFrameInfo("c.py", 1, "child_work")])],
awaited_by=[
MockCoroInfo(task_name=2, call_stack=[MockFrameInfo("a.py", 5, "a_await")]), # Parent A
MockCoroInfo(task_name=3, call_stack=[MockFrameInfo("b.py", 5, "b_await")]), # Parent B
]
)
parent_a = MockTaskInfo(
task_id=2,
task_name="A",
coroutine_stack=[MockCoroInfo(task_name="A", call_stack=[MockFrameInfo("a.py", 10, "a_work")])],
awaited_by=[MockCoroInfo(task_name=1, call_stack=[MockFrameInfo("root.py", 5, "root_spawn")])]
)
parent_b = MockTaskInfo(
task_id=3,
task_name="B",
coroutine_stack=[MockCoroInfo(task_name="B", call_stack=[MockFrameInfo("b.py", 10, "b_work")])],
awaited_by=[MockCoroInfo(task_name=1, call_stack=[MockFrameInfo("root.py", 5, "root_spawn")])]
)
root = MockTaskInfo(
task_id=1,
task_name="Root",
coroutine_stack=[MockCoroInfo(task_name="Root", call_stack=[MockFrameInfo("root.py", 20, "main")])],
awaited_by=[]
)
awaited_info_list = [MockAwaitedInfo(thread_id=777, awaited_by=[child, parent_a, parent_b, root])]
stacks = list(collector._iter_async_frames(awaited_info_list))
# Should get 1 stack: Child->A->Root (picks parent with lowest ID: 2)
self.assertEqual(len(stacks), 1, "Diamond should create only 1 path, picking first sorted parent")
# Verify the single stack
frames, thread_id, leaf_id = stacks[0]
self.assertEqual(leaf_id, 4)
self.assertEqual(thread_id, 777)
func_names = [f.funcname for f in frames]
# Stack should contain child, parent A (id=2, first when sorted), and root
self.assertIn("child_work", func_names)
self.assertIn("a_work", func_names, "Should use parent A (id=2, first when sorted)")
self.assertNotIn("b_work", func_names, "Should not include parent B")
self.assertIn("main", func_names)
# Verify Child task is annotated with parent count
self.assertIn("Child (2 parents)", func_names, "Child task should be annotated with parent count")
def test_empty_coroutine_stack(self):
"""Test _build_linear_stacks: handles empty coroutine_stack (line 109 condition false)."""
collector = PstatsCollector(sample_interval_usec=1000)
# Task with no coroutine_stack
task = MockTaskInfo(
task_id=99,
task_name="EmptyStack",
coroutine_stack=[], # Empty!
awaited_by=[]
)
awaited_info_list = [MockAwaitedInfo(thread_id=111, awaited_by=[task])]
stacks = list(collector._iter_async_frames(awaited_info_list))
self.assertEqual(len(stacks), 1)
frames, _, _ = stacks[0]
# Should only have task marker, no function frames
func_names = [f.funcname for f in frames]
self.assertEqual(len(func_names), 1, "Should only have task marker")
self.assertIn("EmptyStack", func_names)
def test_orphaned_parent_with_no_frames_collected(self):
"""Test _build_linear_stacks: orphaned parent at start with empty frames (line 94-96)."""
collector = PstatsCollector(sample_interval_usec=1000)
# Leaf that doesn't exist in task_map (should not happen normally, but test robustness)
# We'll create a scenario where the leaf_id is present but empty
# Task references non-existent parent, and has no coroutine_stack
orphan = MockTaskInfo(
task_id=88,
task_name="Orphan",
coroutine_stack=[], # No frames
awaited_by=[MockCoroInfo(task_name=999, call_stack=[])] # Parent doesn't exist
)
awaited_info_list = [MockAwaitedInfo(thread_id=222, awaited_by=[orphan])]
stacks = list(collector._iter_async_frames(awaited_info_list))
# Should yield because we have the task marker even with no function frames
self.assertEqual(len(stacks), 1)
frames, _, leaf_id = stacks[0]
self.assertEqual(leaf_id, 88)
# Has task marker but no function frames
self.assertGreater(len(frames), 0, "Should have at least task marker")
def test_frame_ordering(self):
"""Test _build_linear_stacks: frames are collected in correct order (leaf->root).
Task graph (3-level chain):
Root (id=1) <- root_bottom, root_top
|
Middle (id=2) <- mid_bottom, mid_top
|
Leaf (id=3) <- leaf_bottom, leaf_top
Expected frame order: leaf_bottom, leaf_top, mid_bottom, mid_top, root_bottom, root_top
(stack is built bottom-up: leaf frames first, then parent frames).
"""
collector = PstatsCollector(sample_interval_usec=1000)
leaf = MockTaskInfo(
task_id=3,
task_name="Leaf",
coroutine_stack=[
MockCoroInfo(task_name="Leaf", call_stack=[
MockFrameInfo("leaf.py", 1, "leaf_bottom"),
MockFrameInfo("leaf.py", 2, "leaf_top"),
])
],
awaited_by=[MockCoroInfo(task_name=2, call_stack=[])]
)
middle = MockTaskInfo(
task_id=2,
task_name="Middle",
coroutine_stack=[
MockCoroInfo(task_name="Middle", call_stack=[
MockFrameInfo("mid.py", 1, "mid_bottom"),
MockFrameInfo("mid.py", 2, "mid_top"),
])
],
awaited_by=[MockCoroInfo(task_name=1, call_stack=[])]
)
root = MockTaskInfo(
task_id=1,
task_name="Root",
coroutine_stack=[
MockCoroInfo(task_name="Root", call_stack=[
MockFrameInfo("root.py", 1, "root_bottom"),
MockFrameInfo("root.py", 2, "root_top"),
])
],
awaited_by=[]
)
awaited_info_list = [MockAwaitedInfo(thread_id=333, awaited_by=[leaf, middle, root])]
stacks = list(collector._iter_async_frames(awaited_info_list))
self.assertEqual(len(stacks), 1)
frames, _, _ = stacks[0]
func_names = [f.funcname for f in frames]
# Order should be: leaf frames, leaf marker, middle frames, middle marker, root frames, root marker
leaf_bottom_idx = func_names.index("leaf_bottom")
leaf_top_idx = func_names.index("leaf_top")
mid_bottom_idx = func_names.index("mid_bottom")
root_bottom_idx = func_names.index("root_bottom")
# Verify leaf comes before middle comes before root
self.assertLess(leaf_bottom_idx, leaf_top_idx, "Leaf frames in order")
self.assertLess(leaf_top_idx, mid_bottom_idx, "Leaf before middle")
self.assertLess(mid_bottom_idx, root_bottom_idx, "Middle before root")
def test_complex_multi_parent_convergence(self):
"""Test _build_linear_stacks: multiple leaves with same parents pick deterministically.
Tests that when multiple leaves have multiple parents, each leaf picks the same
parent (sorted, first one) and all leaves are annotated with parent count.
Task graph structure (both leaves awaited by both A and B)::
Root (id=1)
/ \\
A (id=2) B (id=3)
| \\ / |
| \\ / |
| \\/ |
| /\\ |
| / \\ |
LeafX (id=4) LeafY (id=5)
Expected behavior: Both leaves pick parent A (lowest id=2) for their stack path.
Result: 2 stacks, both going through A -> Root (B is skipped).
"""
collector = PstatsCollector(sample_interval_usec=1000)
leaf_x = MockTaskInfo(
task_id=4,
task_name="LeafX",
coroutine_stack=[MockCoroInfo(task_name="LeafX", call_stack=[MockFrameInfo("x.py", 1, "x")])],
awaited_by=[
MockCoroInfo(task_name=2, call_stack=[]),
MockCoroInfo(task_name=3, call_stack=[]),
]
)
leaf_y = MockTaskInfo(
task_id=5,
task_name="LeafY",
coroutine_stack=[MockCoroInfo(task_name="LeafY", call_stack=[MockFrameInfo("y.py", 1, "y")])],
awaited_by=[
MockCoroInfo(task_name=2, call_stack=[]),
MockCoroInfo(task_name=3, call_stack=[]),
]
)
parent_a = MockTaskInfo(
task_id=2,
task_name="A",
coroutine_stack=[MockCoroInfo(task_name="A", call_stack=[MockFrameInfo("a.py", 1, "a")])],
awaited_by=[MockCoroInfo(task_name=1, call_stack=[])]
)
parent_b = MockTaskInfo(
task_id=3,
task_name="B",
coroutine_stack=[MockCoroInfo(task_name="B", call_stack=[MockFrameInfo("b.py", 1, "b")])],
awaited_by=[MockCoroInfo(task_name=1, call_stack=[])]
)
root = MockTaskInfo(
task_id=1,
task_name="Root",
coroutine_stack=[MockCoroInfo(task_name="Root", call_stack=[MockFrameInfo("r.py", 1, "root")])],
awaited_by=[]
)
awaited_info_list = [MockAwaitedInfo(thread_id=444, awaited_by=[leaf_x, leaf_y, parent_a, parent_b, root])]
stacks = list(collector._iter_async_frames(awaited_info_list))
# 2 leaves, each picks same parent (A, id=2) = 2 paths
self.assertEqual(len(stacks), 2, "Should create 2 paths: X->A->Root, Y->A->Root")
# Verify both leaves pick parent A (id=2, first when sorted)
leaf_ids_seen = set()
for frames, _, leaf_id in stacks:
leaf_ids_seen.add(leaf_id)
func_names = [f.funcname for f in frames]
# Both stacks should go through parent A only
self.assertIn("a", func_names, "Should use parent A (id=2, first when sorted)")
self.assertNotIn("b", func_names, "Should not include parent B")
self.assertIn("root", func_names, "Should reach root")
# Check for parent count annotation on the leaf
if leaf_id == 4:
self.assertIn("x", func_names)
self.assertIn("LeafX (2 parents)", func_names, "LeafX should be annotated with parent count")
elif leaf_id == 5:
self.assertIn("y", func_names)
self.assertIn("LeafY (2 parents)", func_names, "LeafY should be annotated with parent count")
# Both leaves should be represented
self.assertEqual(leaf_ids_seen, {4, 5}, "Both LeafX and LeafY should have paths")
class TestFlamegraphCollectorAsync(unittest.TestCase):
"""Test FlamegraphCollector with async frames."""
def test_flamegraph_with_async_frames(self):
"""Test FlamegraphCollector correctly processes async task frames."""
from profiling.sampling.stack_collector import FlamegraphCollector
collector = FlamegraphCollector(sample_interval_usec=1000)
# Build async task tree: Root -> Child
child = MockTaskInfo(
task_id=2,
task_name="ChildTask",
coroutine_stack=[
MockCoroInfo(
task_name="ChildTask",
call_stack=[MockFrameInfo("child.py", 10, "child_work")]
)
],
awaited_by=[MockCoroInfo(task_name=1, call_stack=[])]
)
root = MockTaskInfo(
task_id=1,
task_name="RootTask",
coroutine_stack=[
MockCoroInfo(
task_name="RootTask",
call_stack=[MockFrameInfo("root.py", 20, "root_work")]
)
],
awaited_by=[]
)
awaited_info_list = [MockAwaitedInfo(thread_id=100, awaited_by=[child, root])]
# Collect async frames
collector.collect(awaited_info_list)
# Verify samples were collected
self.assertGreater(collector._total_samples, 0)
# Verify the flamegraph tree structure contains our functions
root_node = collector._root
self.assertGreater(root_node["samples"], 0)
# Check that thread ID was tracked
self.assertIn(100, collector._all_threads)
def test_flamegraph_with_task_markers(self):
"""Test FlamegraphCollector includes <task> boundary markers."""
from profiling.sampling.stack_collector import FlamegraphCollector
collector = FlamegraphCollector(sample_interval_usec=1000)
task = MockTaskInfo(
task_id=42,
task_name="MyTask",
coroutine_stack=[
MockCoroInfo(
task_name="MyTask",
call_stack=[MockFrameInfo("work.py", 5, "do_work")]
)
],
awaited_by=[]
)
awaited_info_list = [MockAwaitedInfo(thread_id=200, awaited_by=[task])]
collector.collect(awaited_info_list)
# Find <task> marker in the tree
def find_task_marker(node, depth=0):
for func, child in node.get("children", {}).items():
if func[0] == "<task>":
return func
result = find_task_marker(child, depth + 1)
if result:
return result
return None
task_marker = find_task_marker(collector._root)
self.assertIsNotNone(task_marker, "Should have <task> marker in tree")
self.assertEqual(task_marker[0], "<task>")
self.assertIn("MyTask", task_marker[2])
def test_flamegraph_multiple_async_samples(self):
"""Test FlamegraphCollector aggregates multiple async samples correctly."""
from profiling.sampling.stack_collector import FlamegraphCollector
collector = FlamegraphCollector(sample_interval_usec=1000)
task = MockTaskInfo(
task_id=1,
task_name="Task",
coroutine_stack=[
MockCoroInfo(
task_name="Task",
call_stack=[MockFrameInfo("work.py", 10, "work")]
)
],
awaited_by=[]
)
awaited_info_list = [MockAwaitedInfo(thread_id=300, awaited_by=[task])]
# Collect multiple samples
for _ in range(5):
collector.collect(awaited_info_list)
# Verify sample count
self.assertEqual(collector._sample_count, 5)
self.assertEqual(collector._total_samples, 5)
class TestAsyncAwareParameterFlow(unittest.TestCase):
"""Integration tests for async_aware parameter flow from CLI to unwinder."""
def test_sample_function_accepts_async_aware(self):
"""Test that sample() function accepts async_aware parameter."""
from profiling.sampling.sample import sample
import inspect
sig = inspect.signature(sample)
self.assertIn("async_aware", sig.parameters)
def test_sample_live_function_accepts_async_aware(self):
"""Test that sample_live() function accepts async_aware parameter."""
from profiling.sampling.sample import sample_live
import inspect
sig = inspect.signature(sample_live)
self.assertIn("async_aware", sig.parameters)
def test_sample_profiler_sample_accepts_async_aware(self):
"""Test that SampleProfiler.sample() accepts async_aware parameter."""
from profiling.sampling.sample import SampleProfiler
import inspect
sig = inspect.signature(SampleProfiler.sample)
self.assertIn("async_aware", sig.parameters)
def test_async_aware_all_sees_sleeping_and_running_tasks(self):
"""Test async_aware='all' captures both sleeping and CPU-running tasks."""
# Sleeping task (awaiting)
sleeping_task = MockTaskInfo(
task_id=1,
task_name="SleepingTask",
coroutine_stack=[
MockCoroInfo(
task_name="SleepingTask",
call_stack=[MockFrameInfo("sleeper.py", 10, "sleep_work")]
)
],
awaited_by=[]
)
# CPU-running task (active)
running_task = MockTaskInfo(
task_id=2,
task_name="RunningTask",
coroutine_stack=[
MockCoroInfo(
task_name="RunningTask",
call_stack=[MockFrameInfo("runner.py", 20, "cpu_work")]
)
],
awaited_by=[]
)
# Both tasks returned by get_all_awaited_by
awaited_info_list = [MockAwaitedInfo(thread_id=100, awaited_by=[sleeping_task, running_task])]
collector = PstatsCollector(sample_interval_usec=1000)
collector.collect(awaited_info_list)
collector.create_stats()
# Both tasks should be visible
sleeping_key = ("sleeper.py", 10, "sleep_work")
running_key = ("runner.py", 20, "cpu_work")
self.assertIn(sleeping_key, collector.stats)
self.assertIn(running_key, collector.stats)
# Task markers should also be present
task_keys = [k for k in collector.stats if k[0] == "<task>"]
self.assertGreater(len(task_keys), 0, "Should have <task> markers in stats")
# Verify task names are in the markers
task_names = [k[2] for k in task_keys]
self.assertTrue(
any("SleepingTask" in name for name in task_names),
"SleepingTask should be in task markers"
)
self.assertTrue(
any("RunningTask" in name for name in task_names),
"RunningTask should be in task markers"
)
def test_async_aware_running_sees_only_running_task(self):
"""Test async_aware='running' only shows the currently running task stack."""
# Only the running task's stack is returned by get_async_stack_trace
running_task = MockTaskInfo(
task_id=2,
task_name="RunningTask",
coroutine_stack=[
MockCoroInfo(
task_name="RunningTask",
call_stack=[MockFrameInfo("runner.py", 20, "cpu_work")]
)
],
awaited_by=[]
)
# get_async_stack_trace only returns the running task
awaited_info_list = [MockAwaitedInfo(thread_id=100, awaited_by=[running_task])]
collector = PstatsCollector(sample_interval_usec=1000)
collector.collect(awaited_info_list)
collector.create_stats()
# Only running task should be visible
running_key = ("runner.py", 20, "cpu_work")
self.assertIn(running_key, collector.stats)
# Verify we don't see the sleeping task (it wasn't in the input)
sleeping_key = ("sleeper.py", 10, "sleep_work")
self.assertNotIn(sleeping_key, collector.stats)
# Task marker for running task should be present
task_keys = [k for k in collector.stats if k[0] == "<task>"]
self.assertGreater(len(task_keys), 0, "Should have <task> markers in stats")
task_names = [k[2] for k in task_keys]
self.assertTrue(
any("RunningTask" in name for name in task_names),
"RunningTask should be in task markers"
)
if __name__ == "__main__":
unittest.main()

162
Lib/test/test_profiling/test_sampling_profiler/test_cli.py
View file

@ -547,3 +547,165 @@ def test_sort_options(self):
mock_sample.assert_called_once()
mock_sample.reset_mock()
def test_async_aware_flag_defaults_to_running(self):
"""Test --async-aware flag enables async profiling with default 'running' mode."""
test_args = ["profiling.sampling.cli", "attach", "12345", "--async-aware"]
with (
mock.patch("sys.argv", test_args),
mock.patch("profiling.sampling.cli.sample") as mock_sample,
):
from profiling.sampling.cli import main
main()
mock_sample.assert_called_once()
# Verify async_aware was passed with default "running" mode
call_kwargs = mock_sample.call_args[1]
self.assertEqual(call_kwargs.get("async_aware"), "running")
def test_async_aware_with_async_mode_all(self):
"""Test --async-aware with --async-mode all."""
test_args = ["profiling.sampling.cli", "attach", "12345", "--async-aware", "--async-mode", "all"]
with (
mock.patch("sys.argv", test_args),
mock.patch("profiling.sampling.cli.sample") as mock_sample,
):
from profiling.sampling.cli import main
main()
mock_sample.assert_called_once()
call_kwargs = mock_sample.call_args[1]
self.assertEqual(call_kwargs.get("async_aware"), "all")
def test_async_aware_default_is_none(self):
"""Test async_aware defaults to None when --async-aware not specified."""
test_args = ["profiling.sampling.cli", "attach", "12345"]
with (
mock.patch("sys.argv", test_args),
mock.patch("profiling.sampling.cli.sample") as mock_sample,
):
from profiling.sampling.cli import main
main()
mock_sample.assert_called_once()
call_kwargs = mock_sample.call_args[1]
self.assertIsNone(call_kwargs.get("async_aware"))
def test_async_mode_invalid_choice(self):
"""Test --async-mode with invalid choice raises error."""
test_args = ["profiling.sampling.cli", "attach", "12345", "--async-aware", "--async-mode", "invalid"]
with (
mock.patch("sys.argv", test_args),
mock.patch("sys.stderr", io.StringIO()),
self.assertRaises(SystemExit) as cm,
):
from profiling.sampling.cli import main
main()
self.assertEqual(cm.exception.code, 2) # argparse error
def test_async_mode_requires_async_aware(self):
"""Test --async-mode without --async-aware raises error."""
test_args = ["profiling.sampling.cli", "attach", "12345", "--async-mode", "all"]
with (
mock.patch("sys.argv", test_args),
mock.patch("sys.stderr", io.StringIO()) as mock_stderr,
self.assertRaises(SystemExit) as cm,
):
from profiling.sampling.cli import main
main()
self.assertEqual(cm.exception.code, 2) # argparse error
error_msg = mock_stderr.getvalue()
self.assertIn("--async-mode requires --async-aware", error_msg)
def test_async_aware_incompatible_with_native(self):
"""Test --async-aware is incompatible with --native."""
test_args = ["profiling.sampling.cli", "attach", "12345", "--async-aware", "--native"]
with (
mock.patch("sys.argv", test_args),
mock.patch("sys.stderr", io.StringIO()) as mock_stderr,
self.assertRaises(SystemExit) as cm,
):
from profiling.sampling.cli import main
main()
self.assertEqual(cm.exception.code, 2) # argparse error
error_msg = mock_stderr.getvalue()
self.assertIn("--native", error_msg)
self.assertIn("incompatible with --async-aware", error_msg)
def test_async_aware_incompatible_with_no_gc(self):
"""Test --async-aware is incompatible with --no-gc."""
test_args = ["profiling.sampling.cli", "attach", "12345", "--async-aware", "--no-gc"]
with (
mock.patch("sys.argv", test_args),
mock.patch("sys.stderr", io.StringIO()) as mock_stderr,
self.assertRaises(SystemExit) as cm,
):
from profiling.sampling.cli import main
main()
self.assertEqual(cm.exception.code, 2) # argparse error
error_msg = mock_stderr.getvalue()
self.assertIn("--no-gc", error_msg)
self.assertIn("incompatible with --async-aware", error_msg)
def test_async_aware_incompatible_with_both_native_and_no_gc(self):
"""Test --async-aware is incompatible with both --native and --no-gc."""
test_args = ["profiling.sampling.cli", "attach", "12345", "--async-aware", "--native", "--no-gc"]
with (
mock.patch("sys.argv", test_args),
mock.patch("sys.stderr", io.StringIO()) as mock_stderr,
self.assertRaises(SystemExit) as cm,
):
from profiling.sampling.cli import main
main()
self.assertEqual(cm.exception.code, 2) # argparse error
error_msg = mock_stderr.getvalue()
self.assertIn("--native", error_msg)
self.assertIn("--no-gc", error_msg)
self.assertIn("incompatible with --async-aware", error_msg)
def test_async_aware_incompatible_with_mode(self):
"""Test --async-aware is incompatible with --mode (non-wall)."""
test_args = ["profiling.sampling.cli", "attach", "12345", "--async-aware", "--mode", "cpu"]
with (
mock.patch("sys.argv", test_args),
mock.patch("sys.stderr", io.StringIO()) as mock_stderr,
self.assertRaises(SystemExit) as cm,
):
from profiling.sampling.cli import main
main()
self.assertEqual(cm.exception.code, 2) # argparse error
error_msg = mock_stderr.getvalue()
self.assertIn("--mode=cpu", error_msg)
self.assertIn("incompatible with --async-aware", error_msg)
def test_async_aware_incompatible_with_all_threads(self):
"""Test --async-aware is incompatible with --all-threads."""
test_args = ["profiling.sampling.cli", "attach", "12345", "--async-aware", "--all-threads"]
with (
mock.patch("sys.argv", test_args),
mock.patch("sys.stderr", io.StringIO()) as mock_stderr,
self.assertRaises(SystemExit) as cm,
):
from profiling.sampling.cli import main
main()
self.assertEqual(cm.exception.code, 2) # argparse error
error_msg = mock_stderr.getvalue()
self.assertIn("--all-threads", error_msg)
self.assertIn("incompatible with --async-aware", error_msg)

125
Lib/test/test_profiling/test_sampling_profiler/test_integration.py
View file

@ -776,3 +776,128 @@ def test_live_incompatible_with_pstats_default_values(self):
from profiling.sampling.cli import main
main()
self.assertNotEqual(cm.exception.code, 0)
@requires_subprocess()
@skip_if_not_supported
@unittest.skipIf(
sys.platform == "linux" and not PROCESS_VM_READV_SUPPORTED,
"Test only runs on Linux with process_vm_readv support",
)
class TestAsyncAwareProfilingIntegration(unittest.TestCase):
"""Integration tests for async-aware profiling mode."""
@classmethod
def setUpClass(cls):
cls.async_script = '''
import asyncio
async def sleeping_leaf():
"""Leaf task that just sleeps - visible in 'all' mode."""
for _ in range(50):
await asyncio.sleep(0.02)
async def cpu_leaf():
"""Leaf task that does CPU work - visible in both modes."""
total = 0
for _ in range(200):
for i in range(10000):
total += i * i
await asyncio.sleep(0)
return total
async def supervisor():
"""Middle layer that spawns leaf tasks."""
tasks = [
asyncio.create_task(sleeping_leaf(), name="Sleeper-0"),
asyncio.create_task(sleeping_leaf(), name="Sleeper-1"),
asyncio.create_task(sleeping_leaf(), name="Sleeper-2"),
asyncio.create_task(cpu_leaf(), name="Worker"),
]
await asyncio.gather(*tasks)
async def main():
await supervisor()
if __name__ == "__main__":
asyncio.run(main())
'''
def _collect_async_samples(self, async_aware_mode):
"""Helper to collect samples and count function occurrences.
Returns a dict mapping function names to their sample counts.
"""
with test_subprocess(self.async_script) as subproc:
try:
collector = CollapsedStackCollector(1000, skip_idle=False)
profiling.sampling.sample.sample(
subproc.process.pid,
collector,
duration_sec=SHORT_TIMEOUT,
async_aware=async_aware_mode,
)
except PermissionError:
self.skipTest("Insufficient permissions for remote profiling")
# Count samples per function from collapsed stacks
# stack_counter keys are (call_tree, thread_id) where call_tree
# is a tuple of (file, line, func) tuples
func_samples = {}
total = 0
for (call_tree, _thread_id), count in collector.stack_counter.items():
total += count
for _file, _line, func in call_tree:
func_samples[func] = func_samples.get(func, 0) + count
func_samples["_total"] = total
return func_samples
def test_async_aware_all_sees_sleeping_and_running_tasks(self):
"""Test that async_aware='all' captures both sleeping and CPU-running tasks.
Task tree structure:
main
└── supervisor
├── Sleeper-0 (sleeping_leaf)
├── Sleeper-1 (sleeping_leaf)
├── Sleeper-2 (sleeping_leaf)
└── Worker (cpu_leaf)
async_aware='all' should see ALL 4 leaf tasks in the output.
"""
samples = self._collect_async_samples("all")
self.assertGreater(samples["_total"], 0, "Should have collected samples")
self.assertIn("sleeping_leaf", samples)
self.assertIn("cpu_leaf", samples)
self.assertIn("supervisor", samples)
def test_async_aware_running_sees_only_cpu_task(self):
"""Test that async_aware='running' only captures the actively running task.
Task tree structure:
main
└── supervisor
├── Sleeper-0 (sleeping_leaf) - NOT visible in 'running'
├── Sleeper-1 (sleeping_leaf) - NOT visible in 'running'
├── Sleeper-2 (sleeping_leaf) - NOT visible in 'running'
└── Worker (cpu_leaf) - VISIBLE in 'running'
async_aware='running' should only see the Worker task doing CPU work.
"""
samples = self._collect_async_samples("running")
total = samples["_total"]
cpu_leaf_samples = samples.get("cpu_leaf", 0)
self.assertGreater(total, 0, "Should have collected some samples")
self.assertGreater(cpu_leaf_samples, 0, "cpu_leaf should appear in samples")
# cpu_leaf should have at least 90% of samples (typically 99%+)
# sleeping_leaf may occasionally appear with very few samples (< 1%)
# when tasks briefly wake up to check sleep timers
cpu_percentage = (cpu_leaf_samples / total) * 100
self.assertGreater(cpu_percentage, 90.0,
f"cpu_leaf should dominate samples in 'running' mode, "
f"got {cpu_percentage:.1f}% ({cpu_leaf_samples}/{total})")

13
Lib/test/test_profiling/test_sampling_profiler/test_live_collector_interaction.py
View file

@ -173,6 +173,19 @@ def test_help_with_question_mark(self):
self.assertTrue(self.collector.show_help)
def test_help_dismiss_with_q_does_not_quit(self):
"""Test that pressing 'q' while help is shown only closes help, not quit"""
self.assertFalse(self.collector.show_help)
self.display.simulate_input(ord("h"))
self.collector._handle_input()
self.assertTrue(self.collector.show_help)
self.display.simulate_input(ord("q"))
self.collector._handle_input()
self.assertFalse(self.collector.show_help)
self.assertTrue(self.collector.running)
def test_filter_clear(self):
"""Test clearing filter."""
self.collector.filter_pattern = "test"

40
Lib/test/test_pty.py
View file

@ -3,7 +3,6 @@
is_android, is_apple_mobile, is_wasm32, reap_children, verbose, warnings_helper
)
from test.support.import_helper import import_module
from test.support.os_helper import TESTFN, unlink
# Skip these tests if termios is not available
import_module('termios')
@ -299,26 +298,27 @@ def test_master_read(self):
@warnings_helper.ignore_fork_in_thread_deprecation_warnings()
def test_spawn_doesnt_hang(self):
self.addCleanup(unlink, TESTFN)
with open(TESTFN, 'wb') as f:
STDOUT_FILENO = 1
dup_stdout = os.dup(STDOUT_FILENO)
os.dup2(f.fileno(), STDOUT_FILENO)
buf = b''
def master_read(fd):
nonlocal buf
data = os.read(fd, 1024)
buf += data
return data
# gh-140482: Do the test in a pty.fork() child to avoid messing
# with the interactive test runner's terminal settings.
pid, fd = pty.fork()
if pid == pty.CHILD:
pty.spawn([sys.executable, '-c', 'print("hi there")'])
os._exit(0)
try:
buf = bytearray()
try:
pty.spawn([sys.executable, '-c', 'print("hi there")'],
master_read)
finally:
os.dup2(dup_stdout, STDOUT_FILENO)
os.close(dup_stdout)
self.assertEqual(buf, b'hi there\r\n')
with open(TESTFN, 'rb') as f:
self.assertEqual(f.read(), b'hi there\r\n')
while (data := os.read(fd, 1024)) != b'':
buf.extend(data)
except OSError as e:
if e.errno != errno.EIO:
raise
(pid, status) = os.waitpid(pid, 0)
self.assertEqual(status, 0)
self.assertEqual(buf.take_bytes(), b"hi there\r\n")
finally:
os.close(fd)
class SmallPtyTests(unittest.TestCase):
"""These tests don't spawn children or hang."""

18
Lib/test/test_readline.py
View file

@ -413,6 +413,24 @@ def test_write_read_limited_history(self):
# So, we've only tested that the read did not fail.
# See TestHistoryManipulation for the full test.
@unittest.skipUnless(hasattr(readline, "get_pre_input_hook"),
"get_pre_input_hook not available")
def test_get_pre_input_hook(self):
# Save and restore the original hook to avoid side effects
original_hook = readline.get_pre_input_hook()
self.addCleanup(readline.set_pre_input_hook, original_hook)
# Test that get_pre_input_hook returns None when no hook is set
readline.set_pre_input_hook(None)
self.assertIsNone(readline.get_pre_input_hook())
# Set a hook and verify we can retrieve it
def my_hook():
pass
readline.set_pre_input_hook(my_hook)
self.assertIs(readline.get_pre_input_hook(), my_hook)
@unittest.skipUnless(support.Py_GIL_DISABLED, 'these tests can only possibly fail with GIL disabled')
class FreeThreadingTest(unittest.TestCase):

2
Lib/test/test_ssl.py
View file

@ -5613,7 +5613,7 @@ def test_tlsalerttype(self):
class Checked_TLSAlertType(enum.IntEnum):
"""Alert types for TLSContentType.ALERT messages
See RFC 8466, section B.2
See RFC 8446, section B.2
"""
CLOSE_NOTIFY = 0
UNEXPECTED_MESSAGE = 10

17
Lib/test/test_traceback.py
View file

@ -1784,6 +1784,23 @@ def test_keyword_suggestions_from_command_string(self):
stderr_text = stderr.decode('utf-8')
self.assertIn(f"Did you mean '{expected_kw}'", stderr_text)
def test_no_keyword_suggestion_for_comma_errors(self):
# When the parser identifies a missing comma, don't suggest
# bogus keyword replacements like 'print' -> 'not'
code = '''\
import sys
print(
"line1"
"line2"
file=sys.stderr
)
'''
source = textwrap.dedent(code).strip()
rc, stdout, stderr = assert_python_failure('-c', source)
stderr_text = stderr.decode('utf-8')
self.assertIn("Perhaps you forgot a comma", stderr_text)
self.assertNotIn("Did you mean", stderr_text)
@requires_debug_ranges()
@force_not_colorized_test_class
class PurePythonTracebackErrorCaretTests(

17
Lib/test/test_zipfile/test_core.py
View file

@ -2531,6 +2531,10 @@ def test_decompress_without_3rd_party_library(self):
@requires_zlib()
def test_full_overlap_different_names(self):
# The ZIP file contains two central directory entries with
# different names which refer to the same local header.
# The name of the local header matches the name of the first
# central directory entry.
data = (
b'PK\x03\x04\x14\x00\x00\x00\x08\x00\xa0lH\x05\xe2\x1e'
b'8\xbb\x10\x00\x00\x00\t\x04\x00\x00\x01\x00\x00\x00b\xed'
@ -2560,6 +2564,10 @@ def test_full_overlap_different_names(self):
@requires_zlib()
def test_full_overlap_different_names2(self):
# The ZIP file contains two central directory entries with
# different names which refer to the same local header.
# The name of the local header matches the name of the second
# central directory entry.
data = (
b'PK\x03\x04\x14\x00\x00\x00\x08\x00\xa0lH\x05\xe2\x1e'
b'8\xbb\x10\x00\x00\x00\t\x04\x00\x00\x01\x00\x00\x00a\xed'
@ -2591,6 +2599,8 @@ def test_full_overlap_different_names2(self):
@requires_zlib()
def test_full_overlap_same_name(self):
# The ZIP file contains two central directory entries with
# the same name which refer to the same local header.
data = (
b'PK\x03\x04\x14\x00\x00\x00\x08\x00\xa0lH\x05\xe2\x1e'
b'8\xbb\x10\x00\x00\x00\t\x04\x00\x00\x01\x00\x00\x00a\xed'
@ -2623,6 +2633,8 @@ def test_full_overlap_same_name(self):
@requires_zlib()
def test_quoted_overlap(self):
# The ZIP file contains two files. The second local header
# is contained in the range of the first file.
data = (
b'PK\x03\x04\x14\x00\x00\x00\x08\x00\xa0lH\x05Y\xfc'
b'8\x044\x00\x00\x00(\x04\x00\x00\x01\x00\x00\x00a\x00'
@ -2654,6 +2666,7 @@ def test_quoted_overlap(self):
@requires_zlib()
def test_overlap_with_central_dir(self):
# The local header offset is equal to the central directory offset.
data = (
b'PK\x01\x02\x14\x03\x14\x00\x00\x00\x08\x00G_|Z'
b'\xe2\x1e8\xbb\x0b\x00\x00\x00\t\x04\x00\x00\x01\x00\x00\x00'
@ -2668,11 +2681,15 @@ def test_overlap_with_central_dir(self):
self.assertEqual(zi.header_offset, 0)
self.assertEqual(zi.compress_size, 11)
self.assertEqual(zi.file_size, 1033)
# Found central directory signature PK\x01\x02 instead of
# local header signature PK\x03\x04.
with self.assertRaisesRegex(zipfile.BadZipFile, 'Bad magic number'):
zipf.read('a')
@requires_zlib()
def test_overlap_with_archive_comment(self):
# The local header is written after the central directory,
# in the archive comment.
data = (
b'PK\x01\x02\x14\x03\x14\x00\x00\x00\x08\x00G_|Z'
b'\xe2\x1e8\xbb\x0b\x00\x00\x00\t\x04\x00\x00\x01\x00\x00\x00'

9
Lib/traceback.py
View file

@ -1340,6 +1340,15 @@ def _find_keyword_typos(self):
if len(error_code) > 1024:
return
# If the original code doesn't raise SyntaxError, we can't validate
# that a keyword replacement actually fixes anything
try:
codeop.compile_command(error_code, symbol="exec", flags=codeop.PyCF_ONLY_AST)
except SyntaxError:
pass # Good - the original code has a syntax error we might fix
else:
return # Original code compiles or is incomplete - can't validate fixes
error_lines = error_code.splitlines()
tokens = tokenize.generate_tokens(io.StringIO(error_code).readline)
tokens_left_to_process = 10

9
Lib/xml/dom/minidom.py
View file

@ -292,13 +292,6 @@ def _append_child(self, node):
childNodes.append(node)
node.parentNode = self
def _in_document(node):
# return True iff node is part of a document tree
while node is not None:
if node.nodeType == Node.DOCUMENT_NODE:
return True
node = node.parentNode
return False
def _write_data(writer, text, attr):
"Writes datachars to writer."
@ -1555,7 +1548,7 @@ def _clear_id_cache(node):
if node.nodeType == Node.DOCUMENT_NODE:
node._id_cache.clear()
node._id_search_stack = None
elif _in_document(node):
elif node.ownerDocument:
node.ownerDocument._id_cache.clear()
node.ownerDocument._id_search_stack= None

5
Makefile.pre.in
View file

@ -3322,6 +3322,11 @@ check-c-globals:
--format summary \
--traceback
# Check for undocumented C APIs.
.PHONY: check-c-api-docs
check-c-api-docs:
$(PYTHON_FOR_REGEN) $(srcdir)/Tools/check-c-api-docs/main.py
# Find files with funny names
.PHONY: funny
funny:

1
Misc/ACKS
View file

@ -2119,6 +2119,7 @@ Xiang Zhang
Robert Xiao
Florent Xicluna
Yanbo, Xie
Kaisheng Xu
Xinhang Xu
Arnon Yaari
Alakshendra Yadav

4
Misc/NEWS.d/next/Build/2025-11-25-13-17-47.gh-issue-141926.KmuM2h.rst Normal file
View file

@ -0,0 +1,4 @@
``RUNSHARED`` is no longer cleared when cross-compiling. Previously,
``RUNSHARED`` was cleared when cross-compiling, which breaks PGO when using
``--enabled-shared`` on systems where the cross-compiled CPython is otherwise
executable (e.g., via transparent emulation).

1
Misc/NEWS.d/next/Build/2025-11-28-21-43-07.gh-issue-142050.PFi4tv.rst Normal file
View file

@ -0,0 +1 @@
Fixed a bug where JIT stencils produced on Windows contained debug data. Patch by Chris Eibl.

3
Misc/NEWS.d/next/Build/2025-12-03-10-44-42.gh-issue-142234.i1kaFb.rst Normal file
View file

@ -0,0 +1,3 @@
Allow ``--enable-wasm-dynamic-linking`` for WASI. While CPython doesn't
directly support it so external/downstream users do not have to patch in
support for the flag.

1
Misc/NEWS.d/next/C_API/2025-12-03-16-35-24.gh-issue-142225.vmCJoo.rst Normal file
View file

@ -0,0 +1 @@
Fixed the :c:macro:`PyABIInfo_VAR` macro.

2
Misc/NEWS.d/next/Core_and_Builtins/2025-12-01-10-03-08.gh-issue-116738.972YsG.rst Normal file
View file

@ -0,0 +1,2 @@
Fix :mod:`cmath` data race when initializing trigonometric tables with
subinterpreters.

1
Misc/NEWS.d/next/Core_and_Builtins/2025-12-02-21-11-46.gh-issue-141976.yu7pDV.rst Normal file
View file

@ -0,0 +1 @@
Check against abstract stack overflow in the JIT optimizer.

2
Misc/NEWS.d/next/Core_and_Builtins/2025-12-03-11-03-35.gh-issue-142218.44Fq_J.rst Normal file
View file

@ -0,0 +1,2 @@
Fix crash when inserting into a split table dictionary with a non
:class:`str` key that matches an existing key.

4
Misc/NEWS.d/next/Core_and_Builtins/2025-12-06-00-16-43.gh-issue-142236.m3EF9E.rst Normal file
View file

@ -0,0 +1,4 @@
Fix incorrect keyword suggestions for syntax errors in :mod:`traceback`. The
keyword typo suggestion mechanism would incorrectly suggest replacements when
the extracted source code was incomplete rather than containing an actual typo.
Patch by Pablo Galindo.

7
Misc/NEWS.d/next/Library/2024-05-20-12-35-52.gh-issue-115952.J6n_Kf.rst Normal file
View file

@ -0,0 +1,7 @@
Fix a potential memory denial of service in the :mod:`pickle` module.
When reading a pickled data received from untrusted source, it could cause
an arbitrary amount of memory to be allocated, even if the code that is
allowed to execute is restricted by overriding the
:meth:`~pickle.Unpickler.find_class` method.
This could have led to symptoms including a :exc:`MemoryError`, swapping, out
of memory (OOM) killed processes or containers, or even system crashes.

1
Misc/NEWS.d/next/Library/2025-11-14-18-00-41.gh-issue-141565.Ap2bhJ.rst Normal file
View file

@ -0,0 +1 @@
Add async-aware profiling to the Tachyon sampling profiler. The profiler now reconstructs and displays async task hierarchies in flamegraphs, making the output more actionable for users. Patch by Savannah Ostrowski and Pablo Galindo Salgado.

3
Misc/NEWS.d/next/Library/2025-11-15-11-10-16.gh-issue-48752.aB3xYz.rst Normal file
View file

@ -0,0 +1,3 @@
Add :func:`readline.get_pre_input_hook` function to retrieve the current
pre-input hook. This allows applications to save and restore the hook
without overwriting user settings. Patch by Sanyam Khurana.

1
Misc/NEWS.d/next/Library/2025-11-16-04-40-06.gh-issue-69113.Xy7Fmn.rst Normal file
View file

@ -0,0 +1 @@
Fix :mod:`doctest` to correctly report line numbers for doctests in ``__test__`` dictionary when formatted as triple-quoted strings by finding unique lines in the string and matching them in the source file.

2
Misc/NEWS.d/next/Library/2025-11-18-15-48-13.gh-issue-105836.sbUw24.rst Normal file
View file

@ -0,0 +1,2 @@
Fix :meth:`asyncio.run_coroutine_threadsafe` leaving underlying cancelled
asyncio task running.

1
Misc/NEWS.d/next/Library/2025-11-27-10-49-13.gh-issue-142006.nzJDG5.rst Normal file
View file

@ -0,0 +1 @@
Fix a bug in the :mod:`email.policy.default` folding algorithm which incorrectly resulted in a doubled newline when a line ending at exactly max_line_length was followed by an unfoldable token.

5
Misc/NEWS.d/next/Library/2025-12-01-14-43-58.gh-issue-138122.nRm3ic.rst Normal file
View file

@ -0,0 +1,5 @@
The ``_remote_debugging`` module now implements frame caching in the
``RemoteUnwinder`` class to reduce memory reads when profiling remote
processes. When ``cache_frames=True``, unchanged portions of the call stack
are reused from previous samples, significantly improving profiling
performance for deep call stacks.

12
Misc/NEWS.d/next/Library/2025-12-03-06-12-39.gh-issue-142214.appYNZ.rst Normal file
View file

@ -0,0 +1,12 @@
Fix two regressions in :mod:`dataclasses` in Python 3.14.1 related to
annotations.
* An exception is no longer raised if ``slots=True`` is used and the
``__init__`` method does not have an ``__annotate__`` attribute
(likely because ``init=False`` was used).
* An exception is no longer raised if annotations are requested on the
``__init__`` method and one of the fields is not present in the class
annotations. This can occur in certain dynamic scenarios.
Patch by Jelle Zijlstra.

4
Misc/NEWS.d/next/Library/2025-12-03-09-36-29.gh-issue-142206.ilwegH.rst Normal file
View file

@ -0,0 +1,4 @@
The resource tracker in the :mod:`multiprocessing` module can now understand
messages from older versions of itself. This avoids issues with upgrading
Python while it is running. (Note that such 'in-place' upgrades are not
tested.)

Some files were not shown because too many files have changed in this diff Show more