Stowage/cpython
2
0
Fork 0
mirror of https://github.com/python/cpython.git synced 2026-05-04 09:31:02 +00:00
Code Issues Projects Releases Packages Wiki Activity

Merge branch 'main' into hyperkai-patch-1

Browse source
This commit is contained in:
Jelle Zijlstra 2026-05-03 16:00:14 -07:00 • committed by GitHub
commit eb0b7abd69
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
1393 changed files with 78213 additions and 23288 deletions
Download patch file Download diff file Expand all files Collapse all files

4
.gitattributes vendored
View file

@ -34,6 +34,9 @@ Lib/test/xmltestdata/* noeol
Lib/venv/scripts/common/activate text eol=lf
Lib/venv/scripts/posix/* text eol=lf
# Prevent GitHub's web conflict editor from converting LF to CRLF
*.rst text eol=lf
# CRLF files
[attr]dos text eol=crlf
@ -112,3 +115,4 @@ Tools/peg_generator/pegen/grammar_parser.py generated
aclocal.m4 generated
configure generated
*.min.js generated
package-lock.json generated

32
.github/CODEOWNERS vendored
View file

@ -100,12 +100,12 @@ Lib/test/test_build_details.py @FFY00
InternalDocs/ @AA-Turner
# Tools, Configuration, etc
Doc/Makefile @AA-Turner @hugovk
Doc/_static/ @AA-Turner @hugovk
Doc/conf.py @AA-Turner @hugovk
Doc/make.bat @AA-Turner @hugovk
Doc/requirements.txt @AA-Turner @hugovk
Doc/tools/ @AA-Turner @hugovk
Doc/Makefile @AA-Turner @hugovk @StanFromIreland
Doc/_static/ @AA-Turner @hugovk @StanFromIreland
Doc/conf.py @AA-Turner @hugovk @StanFromIreland
Doc/make.bat @AA-Turner @hugovk @StanFromIreland
Doc/requirements.txt @AA-Turner @hugovk @StanFromIreland
Doc/tools/ @AA-Turner @hugovk @StanFromIreland
# PR Previews
.readthedocs.yml @AA-Turner
@ -132,7 +132,9 @@ Tools/c-analyzer/ @ericsnowcurrently
Tools/check-c-api-docs/ @ZeroIntensity
# Fuzzing
Modules/_xxtestfuzz/ @ammaraskar
Modules/_xxtestfuzz/ @python/fuzzers
Lib/test/test_xxtestfuzz.py @python/fuzzers
.github/workflows/reusable-cifuzz.yml @python/fuzzers
# Limited C API & Stable ABI
Doc/c-api/stable.rst @encukou
@ -425,19 +427,19 @@ Lib/dataclasses.py @ericvsmith
Lib/test/test_dataclasses/ @ericvsmith
# Dates and times
Doc/**/*time.rst @pganssle @abalkin @StanFromIreland
Doc/**/*time.rst @pganssle @StanFromIreland
Doc/library/datetime-* @pganssle @StanFromIreland
Doc/library/zoneinfo.rst @pganssle @StanFromIreland
Include/datetime.h @pganssle @abalkin @StanFromIreland
Include/internal/pycore_time.h @pganssle @abalkin @StanFromIreland
Include/datetime.h @pganssle @StanFromIreland
Include/internal/pycore_time.h @pganssle @StanFromIreland
Lib/test/test_zoneinfo/ @pganssle @StanFromIreland
Lib/zoneinfo/ @pganssle @StanFromIreland
Lib/*time.py @pganssle @abalkin @StanFromIreland
Lib/test/datetimetester.py @pganssle @abalkin @StanFromIreland
Lib/test/test_*time.py @pganssle @abalkin @StanFromIreland
Lib/*time.py @pganssle @StanFromIreland
Lib/test/datetimetester.py @pganssle @StanFromIreland
Lib/test/test_*time.py @pganssle @StanFromIreland
Modules/*zoneinfo* @pganssle @StanFromIreland
Modules/*time* @pganssle @abalkin @StanFromIreland
Python/pytime.c @pganssle @abalkin @StanFromIreland
Modules/*time* @pganssle @StanFromIreland
Python/pytime.c @pganssle @StanFromIreland
# Dbm
Doc/library/dbm.rst @corona10 @erlend-aasland @serhiy-storchaka

4
.github/actionlint.yaml vendored
View file

@ -1,7 +1,3 @@
self-hosted-runner:
# Pending https://github.com/rhysd/actionlint/pull/615
labels: ["windows-2025-vs2026"]
config-variables: null
paths:

8
.github/dependabot.yml vendored
View file

@ -3,7 +3,7 @@ updates:
- package-ecosystem: "github-actions"
directory: "/"
schedule:
interval: "monthly"
interval: "quarterly"
labels:
- "skip issue"
- "skip news"
@ -12,6 +12,10 @@ updates:
update-types:
- "version-update:semver-minor"
- "version-update:semver-patch"
groups:
actions:
patterns:
- "*"
cooldown:
# https://blog.yossarian.net/2025/11/21/We-should-all-be-using-dependency-cooldowns
# Cooldowns protect against supply chain attacks by avoiding the
@ -20,7 +24,7 @@ updates:
- package-ecosystem: "pip"
directory: "/Tools/"
schedule:
interval: "monthly"
interval: "quarterly"
labels:
- "skip issue"
- "skip news"

4
.github/workflows/add-issue-header.yml vendored
View file

@ -12,6 +12,8 @@ on:
# Only ever run once
- opened
permissions:
contents: read
jobs:
add-header:
@ -20,7 +22,7 @@ jobs:
issues: write
timeout-minutes: 5
steps:
- uses: actions/github-script@v8
- uses: actions/github-script@ed597411d8f924073f98dfc5c65a23a2325f34cd # v8.0.0
with:
# language=JavaScript
script: |

74
.github/workflows/build.yml vendored
View file

@ -64,7 +64,7 @@ jobs:
run: |
apt update && apt install git -yq
git config --global --add safe.directory "$GITHUB_WORKSPACE"
- uses: actions/checkout@v6
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
with:
fetch-depth: 1
persist-credentials: false
@ -101,10 +101,10 @@ jobs:
needs: build-context
if: needs.build-context.outputs.run-tests == 'true'
steps:
- uses: actions/checkout@v6
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
with:
persist-credentials: false
- uses: actions/setup-python@v6
- uses: actions/setup-python@a309ff8b426b58ec0e2a45f0f869d46889d02405 # v6.2.0
with:
python-version: '3.x'
- name: Runner image version
@ -165,13 +165,21 @@ jobs:
free-threading:
- false
- true
interpreter:
- switch-case
exclude:
# Skip Win32 on free-threaded builds
- { arch: Win32, free-threading: true }
include:
# msvc::musttail is currently only supported on x64,
# and only supported on 3.15+.
- { arch: x64, free-threading: false, interpreter: tail-call }
- { arch: x64, free-threading: true, interpreter: tail-call }
uses: ./.github/workflows/reusable-windows.yml
with:
arch: ${{ matrix.arch }}
free-threading: ${{ matrix.free-threading }}
interpreter: ${{ matrix.interpreter }}
build-windows-msi:
# ${{ '' } is a hack to nest jobs under the same sidebar category.
@ -198,10 +206,10 @@ jobs:
strategy:
fail-fast: false
matrix:
# macos-14 is M1, macos-15-intel is Intel.
# macos-26 is Apple Silicon, macos-15-intel is Intel.
# macos-15-intel only runs tests against the GIL-enabled CPython.
os:
- macos-14
- macos-26
- macos-15-intel
free-threading:
- false
@ -270,20 +278,20 @@ jobs:
# unsupported as it most resembles other 1.1.1-work-a-like ssl APIs
# supported by important vendors such as AWS-LC.
- { name: openssl, version: 1.1.1w }
- { name: openssl, version: 3.0.19 }
- { name: openssl, version: 3.3.6 }
- { name: openssl, version: 3.4.4 }
- { name: openssl, version: 3.5.5 }
- { name: openssl, version: 3.6.1 }
- { name: openssl, version: 3.0.20 }
- { name: openssl, version: 3.3.7 }
- { name: openssl, version: 3.4.5 }
- { name: openssl, version: 3.5.6 }
- { name: openssl, version: 3.6.2 }
## AWS-LC
- { name: aws-lc, version: 1.68.0 }
- { name: aws-lc, version: 1.72.1 }
env:
SSLLIB_VER: ${{ matrix.ssllib.version }}
MULTISSL_DIR: ${{ github.workspace }}/multissl
SSLLIB_DIR: ${{ github.workspace }}/multissl/${{ matrix.ssllib.name }}/${{ matrix.ssllib.version }}
LD_LIBRARY_PATH: ${{ github.workspace }}/multissl/${{ matrix.ssllib.name }}/${{ matrix.ssllib.version }}/lib
steps:
- uses: actions/checkout@v6
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
with:
persist-credentials: false
- name: Runner image version
@ -294,7 +302,7 @@ jobs:
run: sudo ./.github/workflows/posix-deps-apt.sh
- name: 'Restore SSL library build'
id: cache-ssl-lib
uses: actions/cache@v5
uses: actions/cache@668228422ae6a00e4ad889ee87cd7109ec5666a7 # v5.0.4
with:
path: ./multissl/${{ matrix.ssllib.name }}/${{ matrix.ssllib.version }}
key: ${{ matrix.os }}-multissl-${{ matrix.ssllib.name }}-${{ matrix.ssllib.version }}
@ -336,17 +344,17 @@ jobs:
matrix:
include:
- arch: aarch64
runs-on: macos-14
runs-on: macos-26
- arch: x86_64
runs-on: ubuntu-24.04
runs-on: ${{ matrix.runs-on }}
steps:
- uses: actions/checkout@v6
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
with:
persist-credentials: false
- name: Build and test
run: ./Android/android.py ci --fast-ci ${{ matrix.arch }}-linux-android
run: python3 Platforms/Android ci --fast-ci ${{ matrix.arch }}-linux-android
build-ios:
name: iOS
@ -355,7 +363,7 @@ jobs:
timeout-minutes: 60
runs-on: macos-14
steps:
- uses: actions/checkout@v6
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
with:
persist-credentials: false
@ -369,7 +377,13 @@ jobs:
sudo xcode-select --switch /Applications/Xcode_15.4.app
- name: Build and test
run: python3 Apple ci iOS --fast-ci --simulator 'iPhone SE (3rd generation),OS=17.5'
run: python3 Platforms/Apple ci iOS --fast-ci --simulator 'iPhone SE (3rd generation),OS=17.5'
build-emscripten:
name: 'Emscripten'
needs: build-context
if: needs.build-context.outputs.run-emscripten == 'true'
uses: ./.github/workflows/reusable-emscripten.yml
build-wasi:
name: 'WASI'
@ -384,10 +398,10 @@ jobs:
needs: build-context
if: needs.build-context.outputs.run-ubuntu == 'true'
env:
OPENSSL_VER: 3.5.5
OPENSSL_VER: 3.5.6
PYTHONSTRICTEXTENSIONBUILD: 1
steps:
- uses: actions/checkout@v6
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
with:
persist-credentials: false
- name: Register gcc problem matcher
@ -401,7 +415,7 @@ jobs:
echo "LD_LIBRARY_PATH=${GITHUB_WORKSPACE}/multissl/openssl/${OPENSSL_VER}/lib" >> "$GITHUB_ENV"
- name: 'Restore OpenSSL build'
id: cache-openssl
uses: actions/cache@v5
uses: actions/cache@668228422ae6a00e4ad889ee87cd7109ec5666a7 # v5.0.4
with:
path: ./multissl/openssl/${{ env.OPENSSL_VER }}
key: ${{ runner.os }}-multissl-openssl-${{ env.OPENSSL_VER }}
@ -448,7 +462,7 @@ jobs:
./python -m venv "$VENV_LOC" && "$VENV_PYTHON" -m pip install -r "${GITHUB_WORKSPACE}/Tools/requirements-hypothesis.txt"
- name: 'Restore Hypothesis database'
id: cache-hypothesis-database
uses: actions/cache@v5
uses: actions/cache@668228422ae6a00e4ad889ee87cd7109ec5666a7 # v5.0.4
with:
path: ${{ env.CPYTHON_BUILDDIR }}/.hypothesis/
key: hypothesis-database-${{ github.head_ref || github.run_id }}
@ -475,7 +489,7 @@ jobs:
-x test_subprocess \
-x test_signal \
-x test_sysconfig
- uses: actions/upload-artifact@v6
- uses: actions/upload-artifact@bbbca2ddaa5d8feaa63e36b76fdaad77386f024f # v7.0.0
if: always()
with:
name: hypothesis-example-db
@ -492,11 +506,11 @@ jobs:
matrix:
os: [ubuntu-24.04]
env:
OPENSSL_VER: 3.5.5
OPENSSL_VER: 3.5.6
PYTHONSTRICTEXTENSIONBUILD: 1
ASAN_OPTIONS: detect_leaks=0:allocator_may_return_null=1:handle_segv=0
steps:
- uses: actions/checkout@v6
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
with:
persist-credentials: false
- name: Runner image version
@ -506,7 +520,7 @@ jobs:
- name: Install dependencies
run: sudo ./.github/workflows/posix-deps-apt.sh
- name: Set up GCC-10 for ASAN
uses: egor-tensin/setup-gcc@v2
uses: egor-tensin/setup-gcc@a2861a8b8538f49cf2850980acccf6b05a1b2ae4 # v2.0
with:
version: 10
- name: Configure OpenSSL env vars
@ -516,7 +530,7 @@ jobs:
echo "LD_LIBRARY_PATH=${GITHUB_WORKSPACE}/multissl/openssl/${OPENSSL_VER}/lib" >> "$GITHUB_ENV"
- name: 'Restore OpenSSL build'
id: cache-openssl
uses: actions/cache@v5
uses: actions/cache@668228422ae6a00e4ad889ee87cd7109ec5666a7 # v5.0.4
with:
path: ./multissl/openssl/${{ env.OPENSSL_VER }}
key: ${{ matrix.os }}-multissl-openssl-${{ env.OPENSSL_VER }}
@ -563,7 +577,7 @@ jobs:
needs: build-context
if: needs.build-context.outputs.run-ubuntu == 'true'
steps:
- uses: actions/checkout@v6
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
with:
persist-credentials: false
- name: Runner image version
@ -599,6 +613,7 @@ jobs:
needs.build-context.outputs.run-ci-fuzz == 'true'
|| needs.build-context.outputs.run-ci-fuzz-stdlib == 'true'
permissions:
contents: read
security-events: write
strategy:
fail-fast: false
@ -650,6 +665,7 @@ jobs:
- build-ubuntu
- build-ubuntu-ssltests
- build-ios
- build-emscripten
- build-wasi
- test-hypothesis
- build-asan
@ -664,6 +680,7 @@ jobs:
with:
allowed-failures: >-
build-android,
build-emscripten,
build-windows-msi,
build-ubuntu-ssltests,
test-hypothesis,
@ -706,5 +723,6 @@ jobs:
}}
${{ !fromJSON(needs.build-context.outputs.run-android) && 'build-android,' || '' }}
${{ !fromJSON(needs.build-context.outputs.run-ios) && 'build-ios,' || '' }}
${{ !fromJSON(needs.build-context.outputs.run-emscripten) && 'build-emscripten,' || '' }}
${{ !fromJSON(needs.build-context.outputs.run-wasi) && 'build-wasi,' || '' }}
jobs: ${{ toJSON(needs) }}

28
.github/workflows/documentation-links.yml vendored
View file

@ -1,28 +0,0 @@
name: Read the Docs PR preview
# Automatically edits a pull request's descriptions with a link
# to the documentation's preview on Read the Docs.
on:
pull_request_target:
types:
- opened
paths:
- 'Doc/**'
- '.github/workflows/doc.yml'
concurrency:
group: ${{ github.workflow }}-${{ github.head_ref || github.run_id }}
cancel-in-progress: true
jobs:
documentation-links:
runs-on: ubuntu-latest
permissions:
pull-requests: write
timeout-minutes: 5
steps:
- uses: readthedocs/actions/preview@v1
with:
project-slug: "cpython-previews"
single-version: "true"

20
.github/workflows/jit.yml vendored
View file

@ -32,7 +32,7 @@ jobs:
runs-on: ubuntu-24.04
timeout-minutes: 60
steps:
- uses: actions/checkout@v6
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
with:
persist-credentials: false
- name: Build tier two interpreter
@ -69,10 +69,10 @@ jobs:
architecture: ARM64
runner: windows-11-arm
steps:
- uses: actions/checkout@v6
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
with:
persist-credentials: false
- uses: actions/setup-python@v6
- uses: actions/setup-python@a309ff8b426b58ec0e2a45f0f869d46889d02405 # v6.2.0
with:
python-version: '3.11'
# PCbuild downloads LLVM automatically:
@ -101,12 +101,12 @@ jobs:
- target: x86_64-apple-darwin/clang
runner: macos-15-intel
- target: aarch64-apple-darwin/clang
runner: macos-14
runner: macos-15
steps:
- uses: actions/checkout@v6
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
with:
persist-credentials: false
- uses: actions/setup-python@v6
- uses: actions/setup-python@a309ff8b426b58ec0e2a45f0f869d46889d02405 # v6.2.0
with:
python-version: '3.11'
- name: Install LLVM
@ -146,10 +146,10 @@ jobs:
- target: aarch64-unknown-linux-gnu/gcc
runner: ubuntu-24.04-arm
steps:
- uses: actions/checkout@v6
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
with:
persist-credentials: false
- uses: actions/setup-python@v6
- uses: actions/setup-python@a309ff8b426b58ec0e2a45f0f869d46889d02405 # v6.2.0
with:
python-version: '3.11'
- name: Build
@ -182,10 +182,10 @@ jobs:
use_clang: true
run_tests: false
steps:
- uses: actions/checkout@v6
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
with:
persist-credentials: false
- uses: actions/setup-python@v6
- uses: actions/setup-python@a309ff8b426b58ec0e2a45f0f869d46889d02405 # v6.2.0
with:
python-version: '3.11'
- name: Build

4
.github/workflows/lint.yml vendored
View file

@ -19,7 +19,7 @@ jobs:
timeout-minutes: 10
steps:
- uses: actions/checkout@v6
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
with:
persist-credentials: false
- uses: j178/prek-action@v1
- uses: j178/prek-action@0bb87d7f00b0c99306c8bcb8b8beba1eb581c037 # v1.1.1

5
.github/workflows/mypy.yml vendored
View file

@ -19,6 +19,7 @@ on:
- "Tools/build/consts_getter.py"
- "Tools/build/deepfreeze.py"
- "Tools/build/generate-build-details.py"
- "Tools/build/generate_levenshtein_examples.py"
- "Tools/build/generate_sbom.py"
- "Tools/build/generate_stdlib_module_names.py"
- "Tools/build/mypy.ini"
@ -65,10 +66,10 @@ jobs:
"Tools/peg_generator",
]
steps:
- uses: actions/checkout@v6
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
with:
persist-credentials: false
- uses: actions/setup-python@v6
- uses: actions/setup-python@a309ff8b426b58ec0e2a45f0f869d46889d02405 # v6.2.0
with:
python-version: "3.13"
cache: pip

10
.github/workflows/new-bugs-announce-notifier.yml vendored
View file

@ -6,19 +6,21 @@ on:
- opened
permissions:
issues: read
contents: read
jobs:
notify-new-bugs-announce:
runs-on: ubuntu-latest
permissions:
issues: read
timeout-minutes: 10
steps:
- uses: actions/setup-node@v6
- uses: actions/setup-node@53b83947a5a98c8d113130e565377fae1a50d02f # v6.3.0
with:
node-version: 20
- run: npm install mailgun.js form-data
- name: Send notification
uses: actions/github-script@v8
uses: actions/github-script@ed597411d8f924073f98dfc5c65a23a2325f34cd # v8.0.0
env:
MAILGUN_API_KEY: ${{ secrets.MAILGUN_PYTHON_ORG_MAILGUN_KEY }}
with:
@ -44,7 +46,7 @@ jobs:
// We need to truncate the body size, because the max size for
// the whole payload is 16kb. We want to be safe and assume that
// body can take up to ~8kb of space.
body : issue.data.body.substring(0, 8000)
body : (issue.data.body || "").substring(0, 8000)
};
const data = {

19
.github/workflows/posix-deps-apt.sh vendored
View file

@ -26,9 +26,16 @@ apt-get -yq --no-install-recommends install \
xvfb \
zlib1g-dev
# Workaround missing libmpdec-dev on ubuntu 24.04:
# https://launchpad.net/~ondrej/+archive/ubuntu/php
# https://deb.sury.org/
sudo add-apt-repository ppa:ondrej/php
apt-get update
apt-get -yq --no-install-recommends install libmpdec-dev
# Workaround missing libmpdec-dev on ubuntu 24.04 by building mpdecimal
# from source. ppa:ondrej/php (launchpad.net) are unreliable
# (https://status.canonical.com) so fetch the tarball directly
# from the upstream host.
# https://www.bytereef.org/mpdecimal/
MPDECIMAL_VERSION=4.0.1
curl -fsSL "https://www.bytereef.org/software/mpdecimal/releases/mpdecimal-${MPDECIMAL_VERSION}.tar.gz" \
| tar -xz -C /tmp
(cd "/tmp/mpdecimal-${MPDECIMAL_VERSION}" \
&& ./configure --prefix=/usr/local \
&& make -j"$(nproc)" \
&& make install)
ldconfig

11
.github/workflows/require-pr-label.yml vendored
View file

@ -4,6 +4,9 @@ on:
pull_request:
types: [opened, reopened, labeled, unlabeled, synchronize]
permissions:
contents: read
jobs:
label-dnm:
name: DO-NOT-MERGE
@ -15,7 +18,7 @@ jobs:
steps:
- name: Check there's no DO-NOT-MERGE
uses: mheap/github-action-required-labels@v5
uses: mheap/github-action-required-labels@0ac283b4e65c1fb28ce6079dea5546ceca98ccbe # v5.5.2
with:
mode: exactly
count: 0
@ -33,7 +36,7 @@ jobs:
steps:
# Check that the PR is not awaiting changes from the author due to previous review.
- name: Check there's no required changes
uses: mheap/github-action-required-labels@v5
uses: mheap/github-action-required-labels@0ac283b4e65c1fb28ce6079dea5546ceca98ccbe # v5.5.2
with:
mode: exactly
count: 0
@ -42,7 +45,7 @@ jobs:
awaiting change review
- id: is-feature
name: Check whether this PR is a feature (contains a "type-feature" label)
uses: mheap/github-action-required-labels@v5
uses: mheap/github-action-required-labels@0ac283b4e65c1fb28ce6079dea5546ceca98ccbe # v5.5.2
with:
mode: exactly
count: 1
@ -53,7 +56,7 @@ jobs:
- id: awaiting-merge
if: steps.is-feature.outputs.status == 'success'
name: Check for complete review
uses: mheap/github-action-required-labels@v5
uses: mheap/github-action-required-labels@0ac283b4e65c1fb28ce6079dea5546ceca98ccbe # v5.5.2
with:
mode: exactly
count: 1

4
.github/workflows/reusable-check-c-api-docs.yml vendored
View file

@ -15,10 +15,10 @@ jobs:
runs-on: ubuntu-latest
timeout-minutes: 5
steps:
- uses: actions/checkout@v4
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
with:
persist-credentials: false
- uses: actions/setup-python@v5
- uses: actions/setup-python@a309ff8b426b58ec0e2a45f0f869d46889d02405 # v6.2.0
with:
python-version: '3.x'
- name: Check for undocumented C APIs

67
.github/workflows/reusable-check-html-ids.yml vendored Normal file
View file

@ -0,0 +1,67 @@
name: Reusable check HTML IDs
on:
workflow_call:
permissions:
contents: read
env:
FORCE_COLOR: 1
jobs:
check-html-ids:
name: 'Check for removed HTML IDs'
runs-on: ubuntu-latest
timeout-minutes: 30
steps:
- name: 'Check out PR head'
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
with:
persist-credentials: false
ref: ${{ github.event.pull_request.head.sha }}
- name: 'Find merge base'
id: merge-base
run: |
BASE="${{ github.event.pull_request.base.sha }}"
HEAD="${{ github.event.pull_request.head.sha }}"
git fetch --depth=$((${{ github.event.pull_request.commits }} + 10)) --no-tags origin "$BASE" "$HEAD"
if ! MERGE_BASE=$(git merge-base "$BASE" "$HEAD" 2>/dev/null); then
git fetch --deepen=1 --no-tags origin "$BASE" "$HEAD"
OLDEST=$(git rev-list --reflog --max-parents=0 --reverse "${BASE}^" "${HEAD}^" | head -1)
TIMESTAMP=$(git show --format=%at --no-patch "$OLDEST")
git fetch --shallow-since="$TIMESTAMP" --no-tags origin "$BASE" "$HEAD"
MERGE_BASE=$(git merge-base "$BASE" "$HEAD")
fi
echo "sha=$MERGE_BASE" >> "$GITHUB_OUTPUT"
- name: 'Create worktree at merge base'
env:
MERGE_BASE: ${{ steps.merge-base.outputs.sha }}
run: git worktree add /tmp/merge-base "$MERGE_BASE" --detach
- name: 'Set up Python'
uses: actions/setup-python@a309ff8b426b58ec0e2a45f0f869d46889d02405 # v6.2.0
with:
python-version: '3'
cache: 'pip'
cache-dependency-path: 'Doc/requirements.txt'
- name: 'Install build dependencies'
run: make -C /tmp/merge-base/Doc/ venv
- name: 'Build HTML documentation'
run: make -C /tmp/merge-base/Doc/ SPHINXOPTS="--quiet" html
- name: 'Collect HTML IDs'
run: python Doc/tools/check-html-ids.py collect /tmp/merge-base/Doc/build/html -o /tmp/html-ids-base.json.gz
- name: 'Download PR head HTML IDs'
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
with:
name: html-ids-head.json.gz
path: /tmp
- name: 'Check for removed HTML IDs'
run: |
# shellcheck disable=SC2046
python Doc/tools/check-html-ids.py -v check \
/tmp/html-ids-base.json.gz /tmp/html-ids-head.json.gz \
$([ -f Doc/tools/removed-ids.txt ] && echo "--exclude-file Doc/tools/removed-ids.txt")

11
.github/workflows/reusable-cifuzz.yml vendored
View file

@ -13,6 +13,9 @@ on:
required: true
type: string
permissions:
contents: read
jobs:
cifuzz:
name: ${{ inputs.oss-fuzz-project-name }} (${{ inputs.sanitizer }})
@ -21,12 +24,12 @@ jobs:
steps:
- name: Build fuzzers (${{ inputs.sanitizer }})
id: build
uses: google/oss-fuzz/infra/cifuzz/actions/build_fuzzers@master
uses: google/oss-fuzz/infra/cifuzz/actions/build_fuzzers@ed23f8af80ff82b25ca67cd9b101e690b8897b3f # master
with:
oss-fuzz-project-name: ${{ inputs.oss-fuzz-project-name }}
sanitizer: ${{ inputs.sanitizer }}
- name: Run fuzzers (${{ inputs.sanitizer }})
uses: google/oss-fuzz/infra/cifuzz/actions/run_fuzzers@master
uses: google/oss-fuzz/infra/cifuzz/actions/run_fuzzers@ed23f8af80ff82b25ca67cd9b101e690b8897b3f # master
with:
fuzz-seconds: 600
oss-fuzz-project-name: ${{ inputs.oss-fuzz-project-name }}
@ -34,13 +37,13 @@ jobs:
sanitizer: ${{ inputs.sanitizer }}
- name: Upload crash
if: failure() && steps.build.outcome == 'success'
uses: actions/upload-artifact@v6
uses: actions/upload-artifact@bbbca2ddaa5d8feaa63e36b76fdaad77386f024f # v7.0.0
with:
name: ${{ inputs.sanitizer }}-artifacts
path: ./out/artifacts
- name: Upload SARIF
if: always() && steps.build.outcome == 'success'
uses: github/codeql-action/upload-sarif@v4
uses: github/codeql-action/upload-sarif@38697555549f1db7851b81482ff19f1fa5c4fedc # v4.34.1
with:
sarif_file: cifuzz-sarif/results.sarif
checkout_path: cifuzz-sarif

11
.github/workflows/reusable-context.yml vendored
View file

@ -41,6 +41,9 @@ on: # yamllint disable-line rule:truthy
run-ubuntu:
description: Whether to run the Ubuntu tests
value: ${{ jobs.compute-changes.outputs.run-ubuntu }} # bool
run-emscripten:
description: Whether to run the Emscripten tests
value: ${{ jobs.compute-changes.outputs.run-emscripten }} # bool
run-wasi:
description: Whether to run the WASI tests
value: ${{ jobs.compute-changes.outputs.run-wasi }} # bool
@ -51,6 +54,9 @@ on: # yamllint disable-line rule:truthy
description: Whether to run the Windows tests
value: ${{ jobs.compute-changes.outputs.run-windows-tests }} # bool
permissions:
contents: read
jobs:
compute-changes:
name: Create context from changed files
@ -65,19 +71,20 @@ jobs:
run-macos: ${{ steps.changes.outputs.run-macos }}
run-tests: ${{ steps.changes.outputs.run-tests }}
run-ubuntu: ${{ steps.changes.outputs.run-ubuntu }}
run-emscripten: ${{ steps.changes.outputs.run-emscripten }}
run-wasi: ${{ steps.changes.outputs.run-wasi }}
run-windows-msi: ${{ steps.changes.outputs.run-windows-msi }}
run-windows-tests: ${{ steps.changes.outputs.run-windows-tests }}
steps:
- name: Set up Python
uses: actions/setup-python@v6
uses: actions/setup-python@a309ff8b426b58ec0e2a45f0f869d46889d02405 # v6.2.0
with:
python-version: "3"
- run: >-
echo '${{ github.event_name }}'
- uses: actions/checkout@v6
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
with:
persist-credentials: false
ref: >-

30
.github/workflows/reusable-docs.yml vendored
View file

@ -27,7 +27,7 @@ jobs:
refspec_pr: '+${{ github.event.pull_request.head.sha }}:remotes/origin/${{ github.event.pull_request.head.ref }}'
steps:
- name: 'Check out latest PR branch commit'
uses: actions/checkout@v6
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
with:
persist-credentials: false
ref: >-
@ -52,11 +52,11 @@ jobs:
git fetch origin "${refspec_base}" --shallow-since="${DATE}" \
--no-tags --prune --no-recurse-submodules
- name: 'Set up Python'
uses: actions/setup-python@v6
uses: actions/setup-python@a309ff8b426b58ec0e2a45f0f869d46889d02405 # v6.2.0
with:
python-version: '3'
cache: 'pip'
cache-dependency-path: 'Doc/requirements.txt'
cache-dependency-path: 'Doc/pylock.toml'
- name: 'Install build dependencies'
run: make -C Doc/ venv
@ -75,6 +75,22 @@ jobs:
--fail-if-regression \
--fail-if-improved \
--fail-if-new-news-nit
- name: 'Collect HTML IDs'
if: github.event_name == 'pull_request'
run: python Doc/tools/check-html-ids.py collect Doc/build/html -o Doc/build/html-ids-head.json.gz
- name: 'Upload HTML IDs'
if: github.event_name == 'pull_request'
uses: actions/upload-artifact@bbbca2ddaa5d8feaa63e36b76fdaad77386f024f # v7.0.0
with:
name: html-ids-head
path: Doc/build/html-ids-head.json.gz
archive: false
check-html-ids:
name: 'Check for removed HTML IDs'
needs: build-doc
if: github.event_name == 'pull_request'
uses: ./.github/workflows/reusable-check-html-ids.yml
# Run "doctest" on HEAD as new syntax doesn't exist in the latest stable release
doctest:
@ -82,10 +98,10 @@ jobs:
runs-on: ubuntu-24.04
timeout-minutes: 60
steps:
- uses: actions/checkout@v6
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
with:
persist-credentials: false
- uses: actions/cache@v5
- uses: actions/cache@668228422ae6a00e4ad889ee87cd7109ec5666a7 # v5.0.4
with:
path: ~/.cache/pip
key: ubuntu-doc-${{ hashFiles('Doc/requirements.txt') }}
@ -108,11 +124,11 @@ jobs:
runs-on: ubuntu-latest
timeout-minutes: 30
steps:
- uses: actions/checkout@v6
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
with:
persist-credentials: false
- name: 'Set up Python'
uses: actions/setup-python@v6
uses: actions/setup-python@a309ff8b426b58ec0e2a45f0f869d46889d02405 # v6.2.0
with:
python-version: '3'
cache: 'pip'

77
.github/workflows/reusable-emscripten.yml vendored Normal file
View file

@ -0,0 +1,77 @@
name: Reusable Emscripten
on:
workflow_call:
permissions:
contents: read
env:
FORCE_COLOR: 1
jobs:
build-emscripten-reusable:
name: 'build and test'
runs-on: ubuntu-24.04
timeout-minutes: 40
steps:
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
with:
persist-credentials: false
- name: "Read Emscripten config"
id: emscripten-config
shell: python
run: |
import hashlib
import json
import os
import tomllib
from pathlib import Path
config = tomllib.loads(Path("Platforms/emscripten/config.toml").read_text())
h = hashlib.sha256()
h.update(json.dumps(config["dependencies"], sort_keys=True).encode())
h.update(Path("Platforms/emscripten/make_libffi.sh").read_bytes())
h.update(b'1') # Update to explicitly bust cache
emsdk_cache = Path(os.environ["RUNNER_TEMP"]) / "emsdk-cache"
with open(os.environ["GITHUB_OUTPUT"], "a") as f:
f.write(f"emscripten-version={config['emscripten-version']}\n")
f.write(f"node-version={config['node-version']}\n")
f.write(f"deps-hash={h.hexdigest()}\n")
with open(os.environ["GITHUB_ENV"], "a") as f:
f.write(f"EMSDK_CACHE={emsdk_cache}\n")
- name: "Install Node.js"
uses: actions/setup-node@53b83947a5a98c8d113130e565377fae1a50d02f # v6.3.0
with:
node-version: ${{ steps.emscripten-config.outputs.node-version }}
- name: "Cache Emscripten SDK"
id: emsdk-cache
uses: actions/cache@668228422ae6a00e4ad889ee87cd7109ec5666a7 # v5.0.4
with:
path: ${{ env.EMSDK_CACHE }}
key: emsdk-${{ steps.emscripten-config.outputs.emscripten-version }}-${{ steps.emscripten-config.outputs.deps-hash }}
restore-keys: emsdk-${{ steps.emscripten-config.outputs.emscripten-version }}
- name: "Install Python"
uses: actions/setup-python@a309ff8b426b58ec0e2a45f0f869d46889d02405 # v6.2.0
with:
python-version: '3.x'
- name: "Runner image version"
run: echo "IMAGE_OS_VERSION=${ImageOS}-${ImageVersion}" >> "$GITHUB_ENV"
- name: "Install Emscripten"
run: python3 Platforms/emscripten install-emscripten
- name: "Configure build Python"
run: python3 Platforms/emscripten configure-build-python -- --config-cache --with-pydebug
- name: "Make build Python"
run: python3 Platforms/emscripten make-build-python
- name: "Make dependencies"
run: >-
python3 Platforms/emscripten make-dependencies
${{ steps.emsdk-cache.outputs.cache-hit == 'true' && '--check-up-to-date' || '' }}
- name: "Configure host Python"
run: python3 Platforms/emscripten configure-host --host-runner node -- --config-cache
- name: "Make host Python"
run: python3 Platforms/emscripten make-host
- name: "Display build info"
run: python3 Platforms/emscripten run --pythoninfo
- name: "Test"
run: python3 Platforms/emscripten run --test

5
.github/workflows/reusable-macos.yml vendored
View file

@ -12,6 +12,9 @@ on:
required: true
type: string
permissions:
contents: read
env:
FORCE_COLOR: 1
@ -28,7 +31,7 @@ jobs:
PYTHONSTRICTEXTENSIONBUILD: 1
TERM: linux
steps:
- uses: actions/checkout@v6
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
with:
persist-credentials: false
- name: Runner image version

26
.github/workflows/reusable-san.yml vendored
View file

@ -12,6 +12,9 @@ on:
type: boolean
default: false
permissions:
contents: read
env:
FORCE_COLOR: 1
@ -26,7 +29,7 @@ jobs:
runs-on: ubuntu-24.04
timeout-minutes: 60
steps:
- uses: actions/checkout@v6
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
with:
persist-credentials: false
- name: Runner image version
@ -37,17 +40,15 @@ jobs:
# Install clang
wget https://apt.llvm.org/llvm.sh
chmod +x llvm.sh
sudo ./llvm.sh 20
sudo update-alternatives --install /usr/bin/clang clang /usr/bin/clang-20 100
sudo update-alternatives --set clang /usr/bin/clang-20
sudo update-alternatives --install /usr/bin/clang++ clang++ /usr/bin/clang++-20 100
sudo update-alternatives --set clang++ /usr/bin/clang++-20
if [ "${SANITIZER}" = "TSan" ]; then
sudo ./llvm.sh 17 # gh-121946: llvm-18 package is temporarily broken
sudo update-alternatives --install /usr/bin/clang clang /usr/bin/clang-17 100
sudo update-alternatives --set clang /usr/bin/clang-17
sudo update-alternatives --install /usr/bin/clang++ clang++ /usr/bin/clang++-17 100
sudo update-alternatives --set clang++ /usr/bin/clang++-17
# Reduce ASLR to avoid TSan crashing
sudo sysctl -w vm.mmap_rnd_bits=28
else
sudo ./llvm.sh 20
fi
- name: Sanitizer option setup
@ -59,7 +60,7 @@ jobs:
|| ''
}}.txt handle_segv=0" >> "$GITHUB_ENV"
else
echo "UBSAN_OPTIONS=${SAN_LOG_OPTION}" >> "$GITHUB_ENV"
echo "UBSAN_OPTIONS=${SAN_LOG_OPTION} halt_on_error=1 suppressions=${GITHUB_WORKSPACE}/Tools/ubsan/suppressions.txt" >> "$GITHUB_ENV"
fi
echo "CC=clang" >> "$GITHUB_ENV"
echo "CXX=clang++" >> "$GITHUB_ENV"
@ -73,7 +74,7 @@ jobs:
${{
inputs.sanitizer == 'TSan'
&& '--with-thread-sanitizer'
|| '--with-undefined-behavior-sanitizer'
|| '--with-undefined-behavior-sanitizer --with-strict-overflow'
}}
--with-pydebug
${{ fromJSON(inputs.free-threading) && '--disable-gil' || '' }}
@ -81,10 +82,13 @@ jobs:
run: make -j4
- name: Display build info
run: make pythoninfo
# test_{capi,faulthandler} are skipped under UBSan because
# they raise signals that UBSan with halt_on_error=1 intercepts.
- name: Tests
run: >-
./python -m test
${{ inputs.sanitizer == 'TSan' && '--tsan' || '' }}
${{ inputs.sanitizer == 'UBSan' && '-x test_capi -x test_faulthandler' || '' }}
-j4
- name: Parallel tests
if: >-
@ -96,7 +100,7 @@ jobs:
run: find "${GITHUB_WORKSPACE}" -name 'san_log.*' | xargs head -n 1000
- name: Archive logs
if: always()
uses: actions/upload-artifact@v6
uses: actions/upload-artifact@bbbca2ddaa5d8feaa63e36b76fdaad77386f024f # v7.0.0
with:
name: >-
${{ inputs.sanitizer }}-logs-${{

9
.github/workflows/reusable-ubuntu.yml vendored
View file

@ -23,6 +23,9 @@ on:
type: string
default: ''
permissions:
contents: read
env:
FORCE_COLOR: 1
@ -32,11 +35,11 @@ jobs:
runs-on: ${{ inputs.os }}
timeout-minutes: 60
env:
OPENSSL_VER: 3.5.5
OPENSSL_VER: 3.5.6
PYTHONSTRICTEXTENSIONBUILD: 1
TERM: linux
steps:
- uses: actions/checkout@v6
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
with:
persist-credentials: false
- name: Register gcc problem matcher
@ -56,7 +59,7 @@ jobs:
echo "LD_LIBRARY_PATH=${GITHUB_WORKSPACE}/multissl/openssl/${OPENSSL_VER}/lib" >> "$GITHUB_ENV"
- name: 'Restore OpenSSL build'
id: cache-openssl
uses: actions/cache@v5
uses: actions/cache@668228422ae6a00e4ad889ee87cd7109ec5666a7 # v5.0.4
with:
path: ./multissl/openssl/${{ env.OPENSSL_VER }}
key: ${{ inputs.os }}-multissl-openssl-${{ env.OPENSSL_VER }}

9
.github/workflows/reusable-wasi.yml vendored
View file

@ -3,6 +3,9 @@ name: Reusable WASI
on:
workflow_call:
permissions:
contents: read
env:
FORCE_COLOR: 1
@ -16,12 +19,12 @@ jobs:
CROSS_BUILD_PYTHON: cross-build/build
CROSS_BUILD_WASI: cross-build/wasm32-wasip1
steps:
- uses: actions/checkout@v6
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
with:
persist-credentials: false
# No problem resolver registered as one doesn't currently exist for Clang.
- name: "Install wasmtime"
uses: bytecodealliance/actions/wasmtime/setup@v1
uses: bytecodealliance/actions/wasmtime/setup@9152e710e9f7182e4c29ad218e4f335a7b203613 # v1.1.3
with:
version: ${{ env.WASMTIME_VERSION }}
- name: "Read WASI SDK version"
@ -42,7 +45,7 @@ jobs:
version: ${{ steps.wasi-sdk-version.outputs.version }}
add-to-path: false
- name: "Install Python"
uses: actions/setup-python@v6
uses: actions/setup-python@a309ff8b426b58ec0e2a45f0f869d46889d02405 # v6.2.0
with:
python-version: '3.x'
- name: "Runner image version"

2
.github/workflows/reusable-windows-msi.yml vendored
View file

@ -23,7 +23,7 @@ jobs:
ARCH: ${{ inputs.arch }}
IncludeFreethreaded: true
steps:
- uses: actions/checkout@v6
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
with:
persist-credentials: false
- name: Build CPython installer

19
.github/workflows/reusable-windows.yml vendored
View file

@ -12,6 +12,13 @@ on:
required: false
type: boolean
default: false
interpreter:
description: Which interpreter to build (switch-case or tail-call)
required: true
type: string
permissions:
contents: read
env:
FORCE_COLOR: 1
@ -20,22 +27,25 @@ env:
jobs:
build:
name: Build and test (${{ inputs.arch }})
name: Build and test (${{ inputs.arch }}, ${{ inputs.interpreter }})
runs-on: ${{ inputs.arch == 'arm64' && 'windows-11-arm' || 'windows-2025-vs2026' }}
timeout-minutes: 60
env:
ARCH: ${{ inputs.arch }}
steps:
- uses: actions/checkout@v6
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
with:
persist-credentials: false
- name: Register MSVC problem matcher
if: inputs.arch != 'Win32'
run: echo "::add-matcher::.github/problem-matchers/msvc.json"
- name: Build CPython
# msvc::musttail is not supported for debug builds, so we have to
# switch to release.
run: >-
.\\PCbuild\\build.bat
-e -d -v
-e -v
${{ inputs.interpreter == 'switch-case' && '-d' || '--tail-call-interp -c Release' }}
-p "${ARCH}"
${{ fromJSON(inputs.free-threading) && '--disable-gil' || '' }}
shell: bash
@ -45,6 +55,7 @@ jobs:
run: >-
.\\PCbuild\\rt.bat
-p "${ARCH}"
-d -q --fast-ci
-q --fast-ci
${{ inputs.interpreter == 'switch-case' && '-d' || '' }}
${{ fromJSON(inputs.free-threading) && '--disable-gil' || '' }}
shell: bash

6
.github/workflows/stale.yml vendored
View file

@ -4,17 +4,21 @@ on:
schedule:
- cron: "0 */6 * * *"
permissions:
contents: read
jobs:
stale:
if: github.repository_owner == 'python'
runs-on: ubuntu-latest
permissions:
actions: write
pull-requests: write
timeout-minutes: 10
steps:
- name: "Check PRs"
uses: actions/stale@v9
uses: actions/stale@b5d41d4e1d5dceea10e7104786b73624c18a190f # v10.2.0
with:
repo-token: ${{ secrets.GITHUB_TOKEN }}
stale-pr-message: 'This PR is stale because it has been open for 30 days with no activity.'

45
.github/workflows/tail-call.yml vendored
View file

@ -23,41 +23,6 @@ env:
LLVM_VERSION: 21
jobs:
windows:
name: ${{ matrix.target }}
runs-on: ${{ matrix.runner }}
timeout-minutes: 60
strategy:
fail-fast: false
matrix:
include:
- target: x86_64-pc-windows-msvc/msvc
architecture: x64
runner: windows-2025-vs2026
build_flags: ""
run_tests: true
- target: x86_64-pc-windows-msvc/msvc-free-threading
architecture: x64
runner: windows-2025-vs2026
build_flags: --disable-gil
run_tests: false
steps:
- uses: actions/checkout@v6
with:
persist-credentials: false
- uses: actions/setup-python@v6
with:
python-version: '3.11'
- name: Build
shell: pwsh
run: |
./PCbuild/build.bat --tail-call-interp ${{ matrix.build_flags }} -c Release -p ${{ matrix.architecture }}
- name: Test
if: matrix.run_tests
shell: pwsh
run: |
./PCbuild/rt.bat -p ${{ matrix.architecture }} -q --multiprocess 0 --timeout 4500 --verbose2 --verbose3
macos:
name: ${{ matrix.target }}
runs-on: ${{ matrix.runner }}
@ -69,12 +34,12 @@ jobs:
- target: x86_64-apple-darwin/clang
runner: macos-15-intel
- target: aarch64-apple-darwin/clang
runner: macos-14
runner: macos-15
steps:
- uses: actions/checkout@v6
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
with:
persist-credentials: false
- uses: actions/setup-python@v6
- uses: actions/setup-python@a309ff8b426b58ec0e2a45f0f869d46889d02405 # v6.2.0
with:
python-version: '3.11'
- name: Install dependencies
@ -110,10 +75,10 @@ jobs:
runner: ubuntu-24.04-arm
configure_flags: --with-pydebug
steps:
- uses: actions/checkout@v6
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
with:
persist-credentials: false
- uses: actions/setup-python@v6
- uses: actions/setup-python@a309ff8b426b58ec0e2a45f0f869d46889d02405 # v6.2.0
with:
python-version: '3.11'
- name: Build

4
.github/workflows/verify-ensurepip-wheels.yml vendored
View file

@ -25,10 +25,10 @@ jobs:
runs-on: ubuntu-latest
timeout-minutes: 10
steps:
- uses: actions/checkout@v6
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
with:
persist-credentials: false
- uses: actions/setup-python@v6
- uses: actions/setup-python@a309ff8b426b58ec0e2a45f0f869d46889d02405 # v6.2.0
with:
python-version: '3'
- name: Compare checksum of bundled wheels to the ones published on PyPI

2
.github/workflows/verify-expat.yml vendored
View file

@ -23,7 +23,7 @@ jobs:
runs-on: ubuntu-latest
timeout-minutes: 5
steps:
- uses: actions/checkout@v6
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
with:
persist-credentials: false
- name: Download and verify bundled libexpat files

6
.github/zizmor.yml vendored
View file

@ -1,10 +1,6 @@
# Configuration for the zizmor static analysis tool, run via prek in CI
# https://woodruffw.github.io/zizmor/configuration/
# https://docs.zizmor.sh/configuration/
rules:
dangerous-triggers:
ignore:
- documentation-links.yml
unpinned-uses:
config:
policies:
"*": ref-pin

4
.gitignore vendored
View file

@ -5,6 +5,7 @@
*.cover
*.iml
*.o
*.o.tmp
*.lto
*.a
*.so
@ -137,8 +138,9 @@ Tools/unicode/data/
/config.status
/config.status.lineno
/.ccache
/cross-build/
/cross-build*/
/jit_stencils*.h
/jit_unwind_info*.h
/platform
/profile-clean-stamp
/profile-run-stamp

33
.pre-commit-config.yaml
View file

@ -1,11 +1,11 @@
repos:
- repo: https://github.com/astral-sh/ruff-pre-commit
rev: v0.15.0
rev: e05c5c0818279e5ac248ac9e954431ba58865e61 # frozen: v0.15.7
hooks:
- id: ruff-check
name: Run Ruff (lint) on Apple/
args: [--exit-non-zero-on-fix, --config=Apple/.ruff.toml]
files: ^Apple/
name: Run Ruff (lint) on Platforms/Apple/
args: [--exit-non-zero-on-fix, --config=Platforms/Apple/.ruff.toml]
files: ^Platforms/Apple/
- id: ruff-check
name: Run Ruff (lint) on Doc/
args: [--exit-non-zero-on-fix]
@ -39,9 +39,9 @@ repos:
args: [--exit-non-zero-on-fix, --config=Tools/wasm/.ruff.toml]
files: ^Tools/wasm/
- id: ruff-format
name: Run Ruff (format) on Apple/
args: [--exit-non-zero-on-fix, --config=Apple/.ruff.toml]
files: ^Apple
name: Run Ruff (format) on Platforms/Apple/
args: [--exit-non-zero-on-fix, --config=Platforms/Apple/.ruff.toml]
files: ^Platforms/Apple/
- id: ruff-format
name: Run Ruff (format) on Doc/
args: [--exit-non-zero-on-fix]
@ -60,20 +60,20 @@ repos:
files: ^Tools/wasm/
- repo: https://github.com/psf/black-pre-commit-mirror
rev: 26.1.0
rev: ea488cebbfd88a5f50b8bd95d5c829d0bb76feb8 # frozen: 26.1.0
hooks:
- id: black
name: Run Black on Tools/jit/
files: ^Tools/jit/
- repo: https://github.com/Lucas-C/pre-commit-hooks
rev: v1.5.6
rev: ad1b27d73581aa16cca06fc4a0761fc563ffe8e8 # frozen: v1.5.6
hooks:
- id: remove-tabs
types: [python]
- repo: https://github.com/pre-commit/pre-commit-hooks
rev: v6.0.0
rev: 3e8a8703264a2f4a69428a0aa4dcb512790b2c8c # frozen: v6.0.0
hooks:
- id: check-case-conflict
- id: check-merge-conflict
@ -85,30 +85,33 @@ repos:
exclude: Lib/test/tokenizedata/coding20731.py
- id: end-of-file-fixer
files: '^\.github/CODEOWNERS$'
- id: mixed-line-ending
args: [--fix=auto]
exclude: '^Lib/test/.*data/'
- id: trailing-whitespace
types_or: [c, inc, python, rst, yaml]
- id: trailing-whitespace
files: '^\.github/CODEOWNERS|\.(gram)$'
- repo: https://github.com/python-jsonschema/check-jsonschema
rev: 0.36.1
rev: 9f48a48aa91a6040d749ad68ec70907d907a5a7f # frozen: 0.37.0
hooks:
- id: check-dependabot
- id: check-github-workflows
- id: check-readthedocs
- repo: https://github.com/rhysd/actionlint
rev: v1.7.10
rev: 393031adb9afb225ee52ae2ccd7a5af5525e03e8 # frozen: v1.7.11
hooks:
- id: actionlint
- repo: https://github.com/woodruffw/zizmor-pre-commit
rev: v1.22.0
- repo: https://github.com/zizmorcore/zizmor-pre-commit
rev: b546b77c44c466a54a42af5499dcc0dcc1a3193f # frozen: v1.22.0
hooks:
- id: zizmor
- repo: https://github.com/sphinx-contrib/sphinx-lint
rev: v1.0.2
rev: c883505f64b59c3c5c9375191e4ad9f98e727ccd # frozen: v1.0.2
hooks:
- id: sphinx-lint
args: [--enable=default-role]

9
Doc/Makefile
View file

@ -13,7 +13,7 @@ JOBS = auto
PAPER =
SOURCES =
DISTVERSION = $(shell $(PYTHON) tools/extensions/patchlevel.py)
REQUIREMENTS = requirements.txt
REQUIREMENTS = pylock.toml
SPHINXERRORHANDLING = --fail-on-warning
# Internal variables.
@ -88,6 +88,7 @@ htmlhelp: build
"build/htmlhelp/pydoc.hhp project file."
.PHONY: latex
latex: _ensure-sphinxcontrib-svg2pdfconverter
latex: BUILDER = latex
latex: build
@echo "Build finished; the LaTeX files are in build/latex."
@ -231,7 +232,7 @@ dist-text:
@echo "Build finished and archived!"
.PHONY: dist-pdf
dist-pdf:
dist-pdf: _ensure-sphinxcontrib-svg2pdfconverter
# archive the A4 latex
@echo "Building LaTeX (A4 paper)..."
mkdir -p dist
@ -292,6 +293,10 @@ _ensure-pre-commit:
_ensure-sphinx-autobuild:
$(MAKE) _ensure-package PACKAGE=sphinx-autobuild
.PHONY: _ensure-sphinxcontrib-svg2pdfconverter
_ensure-sphinxcontrib-svg2pdfconverter:
$(MAKE) _ensure-package PACKAGE=sphinxcontrib-svg2pdfconverter
.PHONY: check
check: _ensure-pre-commit
$(VENVDIR)/bin/python3 -m pre_commit run --all-files

12
Doc/c-api/allocation.rst
View file

@ -2,7 +2,7 @@
.. _allocating-objects:
Allocating Objects on the Heap
Allocating objects on the heap
==============================
@ -153,10 +153,12 @@ Allocating Objects on the Heap
To allocate and create extension modules.
Deprecated aliases
^^^^^^^^^^^^^^^^^^
Soft-deprecated aliases
^^^^^^^^^^^^^^^^^^^^^^^
These are :term:`soft deprecated` aliases to existing functions and macros.
.. soft-deprecated:: 3.15
These are aliases to existing functions and macros.
They exist solely for backwards compatibility.
@ -164,7 +166,7 @@ They exist solely for backwards compatibility.
:widths: auto
:header-rows: 1
* * Deprecated alias
* * Soft-deprecated alias
* Function
* * .. c:macro:: PyObject_NEW(type, typeobj)
* :c:macro:`PyObject_New`

22
Doc/c-api/arg.rst
View file

@ -516,6 +516,28 @@ API Functions
}
.. c:function:: int PyArg_ParseArray(PyObject *const *args, Py_ssize_t nargs, const char *format, ...)
Parse the parameters of a function that takes only array parameters into
local variables (that is, a function using the :c:macro:`METH_FASTCALL`
calling convention).
Returns true on success; on failure, it returns false and raises the
appropriate exception.
.. versionadded:: 3.15
.. c:function:: int PyArg_ParseArrayAndKeywords(PyObject *const *args, Py_ssize_t nargs, PyObject *kwnames, const char *format, const char * const *kwlist, ...)
Parse the parameters of a function that takes both array and keyword
parameters into local variables (that is, a function using the
:c:macro:`METH_FASTCALL` ``|`` :c:macro:`METH_KEYWORDS` calling convention).
Returns true on success; on failure, it returns false and raises the
appropriate exception.
.. versionadded:: 3.15
.. c:function:: int PyArg_UnpackTuple(PyObject *args, const char *name, Py_ssize_t min, Py_ssize_t max, ...)
A simpler form of parameter retrieval which does not use a format string to

9
Doc/c-api/buffer.rst
View file

@ -258,7 +258,9 @@ readonly, format
.. c:macro:: PyBUF_WRITEABLE
This is a :term:`soft deprecated` alias to :c:macro:`PyBUF_WRITABLE`.
This is an alias to :c:macro:`PyBUF_WRITABLE`.
.. soft-deprecated:: 3.13
.. c:macro:: PyBUF_FORMAT
@ -500,10 +502,11 @@ Buffer-related functions
*indices* must point to an array of ``view->ndim`` indices.
.. c:function:: int PyBuffer_FromContiguous(const Py_buffer *view, const void *buf, Py_ssize_t len, char fort)
.. c:function:: int PyBuffer_FromContiguous(const Py_buffer *view, const void *buf, Py_ssize_t len, char order)
Copy contiguous *len* bytes from *buf* to *view*.
*fort* can be ``'C'`` or ``'F'`` (for C-style or Fortran-style ordering).
*order* can be ``'C'`` or ``'F'`` or ``'A'`` (for C-style or Fortran-style
ordering or either one).
``0`` is returned on success, ``-1`` on error.

14
Doc/c-api/bytearray.rst
View file

@ -44,6 +44,10 @@ Direct API functions
On failure, return ``NULL`` with an exception set.
.. note::
If the object implements the buffer protocol, then the buffer
must not be mutated while the bytearray object is being created.
.. c:function:: PyObject* PyByteArray_FromStringAndSize(const char *string, Py_ssize_t len)
@ -58,6 +62,10 @@ Direct API functions
On failure, return ``NULL`` with an exception set.
.. note::
If the object implements the buffer protocol, then the buffer
must not be mutated while the bytearray object is being created.
.. c:function:: Py_ssize_t PyByteArray_Size(PyObject *bytearray)
@ -70,6 +78,9 @@ Direct API functions
``NULL`` pointer. The returned array always has an extra
null byte appended.
.. note::
It is not thread-safe to mutate the bytearray object while using the returned char array.
.. c:function:: int PyByteArray_Resize(PyObject *bytearray, Py_ssize_t len)
@ -89,6 +100,9 @@ These macros trade safety for speed and they don't check pointers.
Similar to :c:func:`PyByteArray_AsString`, but without error checking.
.. note::
It is not thread-safe to mutate the bytearray object while using the returned char array.
.. c:function:: Py_ssize_t PyByteArray_GET_SIZE(PyObject *bytearray)

29
Doc/c-api/bytes.rst
View file

@ -47,9 +47,9 @@ called with a non-bytes parameter.
*len* on success, and ``NULL`` on failure. If *v* is ``NULL``, the contents of
the bytes object are uninitialized.
.. deprecated:: 3.15
``PyBytes_FromStringAndSize(NULL, len)`` is :term:`soft deprecated`,
use the :c:type:`PyBytesWriter` API instead.
.. soft-deprecated:: 3.15
Use the :c:type:`PyBytesWriter` API instead of
``PyBytes_FromStringAndSize(NULL, len)``.
.. c:function:: PyObject* PyBytes_FromFormat(const char *format, ...)
@ -127,6 +127,10 @@ called with a non-bytes parameter.
Return the bytes representation of object *o* that implements the buffer
protocol.
.. note::
If the object implements the buffer protocol, then the buffer
must not be mutated while the bytes object is being created.
.. c:function:: Py_ssize_t PyBytes_Size(PyObject *o)
@ -185,6 +189,9 @@ called with a non-bytes parameter.
created, the old reference to *bytes* will still be discarded and the value
of *\*bytes* will be set to ``NULL``; the appropriate exception will be set.
.. note::
If *newpart* implements the buffer protocol, then the buffer
must not be mutated while the new bytes object is being created.
.. c:function:: void PyBytes_ConcatAndDel(PyObject **bytes, PyObject *newpart)
@ -192,6 +199,10 @@ called with a non-bytes parameter.
appended to *bytes*. This version releases the :term:`strong reference`
to *newpart* (i.e. decrements its reference count).
.. note::
If *newpart* implements the buffer protocol, then the buffer
must not be mutated while the new bytes object is being created.
.. c:function:: PyObject* PyBytes_Join(PyObject *sep, PyObject *iterable)
@ -210,6 +221,9 @@ called with a non-bytes parameter.
.. versionadded:: 3.14
.. note::
If *iterable* objects implement the buffer protocol, then the buffers
must not be mutated while the new bytes object is being created.
.. c:function:: int _PyBytes_Resize(PyObject **bytes, Py_ssize_t newsize)
@ -224,9 +238,8 @@ called with a non-bytes parameter.
*\*bytes* is set to ``NULL``, :exc:`MemoryError` is set, and ``-1`` is
returned.
.. deprecated:: 3.15
The function is :term:`soft deprecated`,
use the :c:type:`PyBytesWriter` API instead.
.. soft-deprecated:: 3.15
Use the :c:type:`PyBytesWriter` API instead.
.. c:function:: PyObject *PyBytes_Repr(PyObject *bytes, int smartquotes)
@ -371,6 +384,8 @@ Getters
Get the writer size.
The function cannot fail.
.. c:function:: void* PyBytesWriter_GetData(PyBytesWriter *writer)
Get the writer data: start of the internal buffer.
@ -378,6 +393,8 @@ Getters
The pointer is valid until :c:func:`PyBytesWriter_Finish` or
:c:func:`PyBytesWriter_Discard` is called on *writer*.
The function cannot fail.
Low-level API
^^^^^^^^^^^^^

4
Doc/c-api/code.rst
View file

@ -212,7 +212,7 @@ bound into a function.
.. c:function:: PyObject *PyCode_Optimize(PyObject *code, PyObject *consts, PyObject *names, PyObject *lnotab_obj)
This is a :term:`soft deprecated` function that does nothing.
This is a function that does nothing.
Prior to Python 3.10, this function would perform basic optimizations to a
code object.
@ -220,6 +220,8 @@ bound into a function.
.. versionchanged:: 3.10
This function now does nothing.
.. soft-deprecated:: 3.13
.. _c_codeobject_flags:

1
Doc/c-api/concrete.rst
View file

@ -112,6 +112,7 @@ Other Objects
picklebuffer.rst
weakref.rst
capsule.rst
sentinel.rst
frame.rst
gen.rst
coro.rst

93
Doc/c-api/descriptor.rst
View file

@ -8,13 +8,31 @@ Descriptor Objects
"Descriptors" are objects that describe some attribute of an object. They are
found in the dictionary of type objects.
.. XXX document these!
.. c:function:: PyObject* PyDescr_NewGetSet(PyTypeObject *type, struct PyGetSetDef *getset)
Create a new get-set descriptor for extension type *type* from the
:c:type:`PyGetSetDef` structure *getset*.
.. c:function:: PyObject* PyDescr_NewMember(PyTypeObject *type, struct PyMemberDef *meth)
Get-set descriptors expose attributes implemented by C getter and setter
functions rather than stored directly in the instance. This is the same kind
of descriptor created for entries in :c:member:`~PyTypeObject.tp_getset`, and
it appears in Python as a :class:`types.GetSetDescriptorType` object.
On success, return a :term:`strong reference` to the descriptor. Return
``NULL`` with an exception set on failure.
.. c:function:: PyObject* PyDescr_NewMember(PyTypeObject *type, struct PyMemberDef *member)
Create a new member descriptor for extension type *type* from the
:c:type:`PyMemberDef` structure *member*.
Member descriptors expose fields in the type's C struct as Python
attributes. This is the same kind of descriptor created for entries in
:c:member:`~PyTypeObject.tp_members`, and it appears in Python as a
:class:`types.MemberDescriptorType` object.
On success, return a :term:`strong reference` to the descriptor. Return
``NULL`` with an exception set on failure.
.. c:var:: PyTypeObject PyMemberDescr_Type
@ -30,22 +48,53 @@ found in the dictionary of type objects.
The type object for get/set descriptor objects created from
:c:type:`PyGetSetDef` structures. These descriptors implement attributes
whose value is computed by C getter and setter functions, and are used
for many built-in type attributes.
for many built-in type attributes. They correspond to
:class:`types.GetSetDescriptorType` objects in Python.
.. c:function:: PyObject* PyDescr_NewMethod(PyTypeObject *type, struct PyMethodDef *meth)
Create a new method descriptor for extension type *type* from the
:c:type:`PyMethodDef` structure *meth*.
Method descriptors expose C functions as methods on a type. This is the same
kind of descriptor created for entries in
:c:member:`~PyTypeObject.tp_methods`, and it appears in Python as a
:class:`types.MethodDescriptorType` object.
On success, return a :term:`strong reference` to the descriptor. Return
``NULL`` with an exception set on failure.
.. c:var:: PyTypeObject PyMethodDescr_Type
The type object for method descriptor objects created from
:c:type:`PyMethodDef` structures. These descriptors expose C functions as
methods on a type, and correspond to :class:`types.MemberDescriptorType`
methods on a type, and correspond to :class:`types.MethodDescriptorType`
objects in Python.
.. c:function:: PyObject* PyDescr_NewWrapper(PyTypeObject *type, struct wrapperbase *wrapper, void *wrapped)
.. c:struct:: wrapperbase
Describes a slot wrapper used by :c:func:`PyDescr_NewWrapper`.
Each ``wrapperbase`` record stores the Python-visible name and metadata for a
special method implemented by a type slot, together with the wrapper
function used to adapt that slot to Python's calling convention.
.. c:function:: PyObject* PyDescr_NewWrapper(PyTypeObject *type, struct wrapperbase *base, void *wrapped)
Create a new wrapper descriptor for extension type *type* from the
:c:struct:`wrapperbase` structure *base* and the wrapped slot function
pointer
*wrapped*.
Wrapper descriptors expose special methods implemented by type slots. This
is the same kind of descriptor that CPython creates for slot-based special
methods such as ``__repr__`` or ``__add__``, and it appears in Python as a
:class:`types.WrapperDescriptorType` object.
On success, return a :term:`strong reference` to the descriptor. Return
``NULL`` with an exception set on failure.
.. c:var:: PyTypeObject PyWrapperDescr_Type
@ -58,6 +107,16 @@ found in the dictionary of type objects.
.. c:function:: PyObject* PyDescr_NewClassMethod(PyTypeObject *type, PyMethodDef *method)
Create a new class method descriptor for extension type *type* from the
:c:type:`PyMethodDef` structure *method*.
Class method descriptors expose C methods that receive the class rather than
an instance when accessed. This is the same kind of descriptor created for
``METH_CLASS`` entries in :c:member:`~PyTypeObject.tp_methods`, and it
appears in Python as a :class:`types.ClassMethodDescriptorType` object.
On success, return a :term:`strong reference` to the descriptor. Return
``NULL`` with an exception set on failure.
.. c:function:: int PyDescr_IsData(PyObject *descr)
@ -66,12 +125,22 @@ found in the dictionary of type objects.
no error checking.
.. c:function:: PyObject* PyWrapper_New(PyObject *, PyObject *)
.. c:function:: PyObject* PyWrapper_New(PyObject *d, PyObject *self)
Create a new bound wrapper object from the wrapper descriptor *d* and the
instance *self*.
This is the bound form of a wrapper descriptor created by
:c:func:`PyDescr_NewWrapper`. CPython creates these objects when a slot
wrapper is accessed through an instance, and they appear in Python as
:class:`types.MethodWrapperType` objects.
On success, return a :term:`strong reference` to the wrapper object. Return
``NULL`` with an exception set on failure.
.. c:macro:: PyDescr_COMMON
This is a :term:`soft deprecated` macro including the common fields for a
This is a macro including the common fields for a
descriptor object.
This was included in Python's C API by mistake; do not use it in extensions.
@ -79,6 +148,8 @@ found in the dictionary of type objects.
descriptor protocol (:c:member:`~PyTypeObject.tp_descr_get` and
:c:member:`~PyTypeObject.tp_descr_set`).
.. soft-deprecated:: 3.15
Built-in descriptors
^^^^^^^^^^^^^^^^^^^^
@ -104,9 +175,9 @@ Built-in descriptors
.. c:var:: PyTypeObject PyClassMethodDescr_Type
The type object for C-level class method descriptor objects.
This is the type of the descriptors created for :func:`classmethod` defined in
C extension types, and is the same object as :class:`classmethod`
in Python.
This is the type of the descriptors created for :func:`classmethod` defined
in C extension types, and corresponds to
:class:`types.ClassMethodDescriptorType` objects in Python.
.. c:function:: PyObject *PyClassMethod_New(PyObject *callable)

167
Doc/c-api/dict.rst
View file

@ -42,6 +42,12 @@ Dictionary objects
enforces read-only behavior. This is normally used to create a view to
prevent modification of the dictionary for non-dynamic class types.
The first argument can be a :class:`dict`, a :class:`frozendict`, or a
mapping.
.. versionchanged:: 3.15
Also accept :class:`frozendict`.
.. c:var:: PyTypeObject PyDictProxy_Type
@ -68,6 +74,16 @@ Dictionary objects
*key*, return ``1``, otherwise return ``0``. On error, return ``-1``.
This is equivalent to the Python expression ``key in p``.
The first argument can be a :class:`dict` or a :class:`frozendict`.
.. note::
The operation is atomic on :term:`free threading <free-threaded build>`
when *key* is :class:`str`, :class:`int`, :class:`float`, :class:`bool` or :class:`bytes`.
.. versionchanged:: 3.15
Also accept :class:`frozendict`.
.. c:function:: int PyDict_ContainsString(PyObject *p, const char *key)
@ -75,17 +91,18 @@ Dictionary objects
:c:expr:`const char*` UTF-8 encoded bytes string, rather than a
:c:expr:`PyObject*`.
The first argument can be a :class:`dict` or a :class:`frozendict`.
.. versionadded:: 3.13
.. versionchanged:: 3.15
Also accept :class:`frozendict`.
.. c:function:: PyObject* PyDict_Copy(PyObject *p)
Return a new dictionary that contains the same key-value pairs as *p*.
.. versionchanged:: next
If *p* is a subclass of :class:`frozendict`, the result will be a
:class:`frozendict` instance instead of a :class:`dict` instance.
.. c:function:: int PyDict_SetItem(PyObject *p, PyObject *key, PyObject *val)
Insert *val* into the dictionary *p* with a key of *key*. *key* must be
@ -93,6 +110,11 @@ Dictionary objects
``0`` on success or ``-1`` on failure. This function *does not* steal a
reference to *val*.
.. note::
The operation is atomic on :term:`free threading <free-threaded build>`
when *key* is :class:`str`, :class:`int`, :class:`float`, :class:`bool` or :class:`bytes`.
.. c:function:: int PyDict_SetItemString(PyObject *p, const char *key, PyObject *val)
@ -108,6 +130,11 @@ Dictionary objects
If *key* is not in the dictionary, :exc:`KeyError` is raised.
Return ``0`` on success or ``-1`` on failure.
.. note::
The operation is atomic on :term:`free threading <free-threaded build>`
when *key* is :class:`str`, :class:`int`, :class:`float`, :class:`bool` or :class:`bytes`.
.. c:function:: int PyDict_DelItemString(PyObject *p, const char *key)
@ -126,8 +153,18 @@ Dictionary objects
* If the key is missing, set *\*result* to ``NULL`` and return ``0``.
* On error, raise an exception and return ``-1``.
The first argument can be a :class:`dict` or a :class:`frozendict`.
.. note::
The operation is atomic on :term:`free threading <free-threaded build>`
when *key* is :class:`str`, :class:`int`, :class:`float`, :class:`bool` or :class:`bytes`.
.. versionadded:: 3.13
.. versionchanged:: 3.15
Also accept :class:`frozendict`.
See also the :c:func:`PyObject_GetItem` function.
@ -137,16 +174,28 @@ Dictionary objects
has a key *key*. Return ``NULL`` if the key *key* is missing *without*
setting an exception.
The first argument can be a :class:`dict` or a :class:`frozendict`.
.. note::
Exceptions that occur while this calls :meth:`~object.__hash__` and
:meth:`~object.__eq__` methods are silently ignored.
Prefer the :c:func:`PyDict_GetItemWithError` function instead.
.. note::
In the :term:`free-threaded build`, the returned
:term:`borrowed reference` may become invalid if another thread modifies
the dictionary concurrently. Prefer :c:func:`PyDict_GetItemRef`, which
returns a :term:`strong reference`.
.. versionchanged:: 3.10
Calling this API without an :term:`attached thread state` had been allowed for historical
reason. It is no longer allowed.
.. versionchanged:: 3.15
Also accept :class:`frozendict`.
.. c:function:: PyObject* PyDict_GetItemWithError(PyObject *p, PyObject *key)
@ -155,6 +204,16 @@ Dictionary objects
occurred. Return ``NULL`` **without** an exception set if the key
wasn't present.
.. note::
In the :term:`free-threaded build`, the returned
:term:`borrowed reference` may become invalid if another thread modifies
the dictionary concurrently. Prefer :c:func:`PyDict_GetItemRef`, which
returns a :term:`strong reference`.
.. versionchanged:: 3.15
Also accept :class:`frozendict`.
.. c:function:: PyObject* PyDict_GetItemString(PyObject *p, const char *key)
@ -170,6 +229,16 @@ Dictionary objects
Prefer using the :c:func:`PyDict_GetItemWithError` function with your own
:c:func:`PyUnicode_FromString` *key* instead.
.. note::
In the :term:`free-threaded build`, the returned
:term:`borrowed reference` may become invalid if another thread modifies
the dictionary concurrently. Prefer :c:func:`PyDict_GetItemStringRef`,
which returns a :term:`strong reference`.
.. versionchanged:: 3.15
Also accept :class:`frozendict`.
.. c:function:: int PyDict_GetItemStringRef(PyObject *p, const char *key, PyObject **result)
@ -179,6 +248,9 @@ Dictionary objects
.. versionadded:: 3.13
.. versionchanged:: 3.15
Also accept :class:`frozendict`.
.. c:function:: PyObject* PyDict_SetDefault(PyObject *p, PyObject *key, PyObject *defaultobj)
@ -190,6 +262,14 @@ Dictionary objects
.. versionadded:: 3.4
.. note::
In the :term:`free-threaded build`, the returned
:term:`borrowed reference` may become invalid if another thread modifies
the dictionary concurrently. Prefer :c:func:`PyDict_SetDefaultRef`,
which returns a :term:`strong reference`.
.. c:function:: int PyDict_SetDefaultRef(PyObject *p, PyObject *key, PyObject *default_value, PyObject **result)
@ -209,6 +289,11 @@ Dictionary objects
These may refer to the same object: in that case you hold two separate
references to it.
.. note::
The operation is atomic on :term:`free threading <free-threaded build>`
when *key* is :class:`str`, :class:`int`, :class:`float`, :class:`bool` or :class:`bytes`.
.. versionadded:: 3.13
@ -226,6 +311,11 @@ Dictionary objects
Similar to :meth:`dict.pop`, but without the default value and
not raising :exc:`KeyError` if the key is missing.
.. note::
The operation is atomic on :term:`free threading <free-threaded build>`
when *key* is :class:`str`, :class:`int`, :class:`float`, :class:`bool` or :class:`bytes`.
.. versionadded:: 3.13
@ -242,17 +332,32 @@ Dictionary objects
Return a :c:type:`PyListObject` containing all the items from the dictionary.
The first argument can be a :class:`dict` or a :class:`frozendict`.
.. versionchanged:: 3.15
Also accept :class:`frozendict`.
.. c:function:: PyObject* PyDict_Keys(PyObject *p)
Return a :c:type:`PyListObject` containing all the keys from the dictionary.
The first argument can be a :class:`dict` or a :class:`frozendict`.
.. versionchanged:: 3.15
Also accept :class:`frozendict`.
.. c:function:: PyObject* PyDict_Values(PyObject *p)
Return a :c:type:`PyListObject` containing all the values from the dictionary
*p*.
The first argument can be a :class:`dict` or a :class:`frozendict`.
.. versionchanged:: 3.15
Also accept :class:`frozendict`.
.. c:function:: Py_ssize_t PyDict_Size(PyObject *p)
@ -261,11 +366,19 @@ Dictionary objects
Return the number of items in the dictionary. This is equivalent to
``len(p)`` on a dictionary.
The argument can be a :class:`dict` or a :class:`frozendict`.
.. versionchanged:: 3.15
Also accept :class:`frozendict`.
.. c:function:: Py_ssize_t PyDict_GET_SIZE(PyObject *p)
Similar to :c:func:`PyDict_Size`, but without error checking.
.. versionchanged:: 3.15
Also accept :class:`frozendict`.
.. c:function:: int PyDict_Next(PyObject *p, Py_ssize_t *ppos, PyObject **pkey, PyObject **pvalue)
@ -280,6 +393,8 @@ Dictionary objects
value represents offsets within the internal dictionary structure, and
since the structure is sparse, the offsets are not consecutive.
The first argument can be a :class:`dict` or a :class:`frozendict`.
For example::
PyObject *key, *value;
@ -313,7 +428,7 @@ Dictionary objects
}
The function is not thread-safe in the :term:`free-threaded <free threading>`
build without external synchronization. You can use
build without external synchronization for a mutable :class:`dict`. You can use
:c:macro:`Py_BEGIN_CRITICAL_SECTION` to lock the dictionary while iterating
over it::
@ -323,6 +438,8 @@ Dictionary objects
}
Py_END_CRITICAL_SECTION();
The function is thread-safe on a :class:`frozendict`.
.. note::
On the free-threaded build, this function can be used safely inside a
@ -333,6 +450,9 @@ Dictionary objects
:term:`strong reference <strong reference>` (for example, using
:c:func:`Py_NewRef`).
.. versionchanged:: 3.15
Also accept :class:`frozendict`.
.. c:function:: int PyDict_Merge(PyObject *a, PyObject *b, int override)
Iterate over mapping object *b* adding key-value pairs to dictionary *a*.
@ -342,6 +462,13 @@ Dictionary objects
only be added if there is not a matching key in *a*. Return ``0`` on
success or ``-1`` if an exception was raised.
.. note::
In the :term:`free-threaded build`, when *b* is a
:class:`dict` (with the standard iterator), both *a* and *b* are locked
for the duration of the operation. When *b* is a non-dict mapping, only
*a* is locked; *b* may be concurrently modified by another thread.
.. c:function:: int PyDict_Update(PyObject *a, PyObject *b)
@ -351,6 +478,13 @@ Dictionary objects
argument has no "keys" attribute. Return ``0`` on success or ``-1`` if an
exception was raised.
.. note::
In the :term:`free-threaded build`, when *b* is a
:class:`dict` (with the standard iterator), both *a* and *b* are locked
for the duration of the operation. When *b* is a non-dict mapping, only
*a* is locked; *b* may be concurrently modified by another thread.
.. c:function:: int PyDict_MergeFromSeq2(PyObject *a, PyObject *seq2, int override)
@ -366,6 +500,13 @@ Dictionary objects
if override or key not in a:
a[key] = value
.. note::
In the :term:`free-threaded <free threading>` build, only *a* is locked.
The iteration over *seq2* is not synchronized; *seq2* may be concurrently
modified by another thread.
.. c:function:: int PyDict_AddWatcher(PyDict_WatchCallback callback)
Register *callback* as a dictionary watcher. Return a non-negative integer
@ -373,6 +514,13 @@ Dictionary objects
of error (e.g. no more watcher IDs available), return ``-1`` and set an
exception.
.. note::
This function is not internally synchronized. In the
:term:`free-threaded <free threading>` build, callers should ensure no
concurrent calls to :c:func:`PyDict_AddWatcher` or
:c:func:`PyDict_ClearWatcher` are in progress.
.. versionadded:: 3.12
.. c:function:: int PyDict_ClearWatcher(int watcher_id)
@ -381,6 +529,13 @@ Dictionary objects
:c:func:`PyDict_AddWatcher`. Return ``0`` on success, ``-1`` on error (e.g.
if the given *watcher_id* was never registered.)
.. note::
This function is not internally synchronized. In the
:term:`free-threaded <free threading>` build, callers should ensure no
concurrent calls to :c:func:`PyDict_AddWatcher` or
:c:func:`PyDict_ClearWatcher` are in progress.
.. versionadded:: 3.12
.. c:function:: int PyDict_Watch(int watcher_id, PyObject *dict)
@ -499,7 +654,7 @@ Dictionary view objects
Frozen dictionary objects
^^^^^^^^^^^^^^^^^^^^^^^^^
.. versionadded:: next
.. versionadded:: 3.15
.. c:var:: PyTypeObject PyFrozenDict_Type

6
Doc/c-api/exceptions.rst
View file

@ -716,7 +716,7 @@ Signal Handling
This function may now execute a remote debugger script, if remote
debugging is enabled.
.. versionchanged:: next
.. versionchanged:: 3.15
The exception set by :c:func:`PyThreadState_SetAsyncExc` is now raised.
@ -818,7 +818,7 @@ Exception Classes
.. c:macro:: PyException_HEAD
This is a :term:`soft deprecated` macro including the base fields for an
This is a macro including the base fields for an
exception object.
This was included in Python's C API by mistake and is not designed for use
@ -826,6 +826,8 @@ Exception Classes
:c:func:`PyErr_NewException` or otherwise create a class inheriting from
:c:data:`PyExc_BaseException`.
.. soft-deprecated:: 3.15
Exception Objects
=================

11
Doc/c-api/extension-modules.rst
View file

@ -191,10 +191,10 @@ the :c:data:`Py_mod_multiple_interpreters` slot.
``PyInit`` function
...................
.. deprecated:: 3.15
.. soft-deprecated:: 3.15
This functionality is :term:`soft deprecated`.
It will not get new features, but there are no plans to remove it.
This functionality will not get new features,
but there are no plans to remove it.
Instead of :c:func:`PyModExport_modulename`, an extension module can define
an older-style :dfn:`initialization function` with the signature:
@ -272,10 +272,9 @@ For example, a module called ``spam`` would be defined like this::
Legacy single-phase initialization
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
.. deprecated:: 3.15
.. soft-deprecated:: 3.15
Single-phase initialization is :term:`soft deprecated`.
It is a legacy mechanism to initialize extension
Single-phase initialization is a legacy mechanism to initialize extension
modules, with known drawbacks and design flaws. Extension module authors
are encouraged to use multi-phase initialization instead.

16
Doc/c-api/file.rst
View file

@ -2,7 +2,7 @@
.. _fileobjects:
File Objects
File objects
------------
.. index:: pair: object; file
@ -123,9 +123,12 @@ the :mod:`io` APIs instead.
Write object *obj* to file object *p*. The only supported flag for *flags* is
:c:macro:`Py_PRINT_RAW`; if given, the :func:`str` of the object is written
instead of the :func:`repr`. Return ``0`` on success or ``-1`` on failure; the
appropriate exception will be set.
instead of the :func:`repr`.
If *obj* is ``NULL``, write the string ``"<NULL>"``.
Return ``0`` on success or ``-1`` on failure; the
appropriate exception will be set.
.. c:function:: int PyFile_WriteString(const char *s, PyObject *p)
@ -133,11 +136,12 @@ the :mod:`io` APIs instead.
failure; the appropriate exception will be set.
Deprecated API
^^^^^^^^^^^^^^
Soft-deprecated API
^^^^^^^^^^^^^^^^^^^
.. soft-deprecated:: 3.15
These are :term:`soft deprecated` APIs that were included in Python's C API
These are APIs that were included in Python's C API
by mistake. They are documented solely for completeness; use other
``PyFile*`` APIs instead.

67
Doc/c-api/float.rst
View file

@ -86,8 +86,7 @@ Floating-Point Objects
It is equivalent to the :c:macro:`!INFINITY` macro from the C11 standard
``<math.h>`` header.
.. deprecated:: 3.15
The macro is :term:`soft deprecated`.
.. soft-deprecated:: 3.15
.. c:macro:: Py_NAN
@ -103,8 +102,7 @@ Floating-Point Objects
Equivalent to :c:macro:`!INFINITY`.
.. deprecated:: 3.14
The macro is :term:`soft deprecated`.
.. soft-deprecated:: 3.14
.. c:macro:: Py_MATH_E
@ -161,8 +159,8 @@ Floating-Point Objects
that is, it is normal, subnormal or zero, but not infinite or NaN.
Return ``0`` otherwise.
.. deprecated:: 3.14
The macro is :term:`soft deprecated`. Use :c:macro:`!isfinite` instead.
.. soft-deprecated:: 3.14
Use :c:macro:`!isfinite` instead.
.. c:macro:: Py_IS_INFINITY(X)
@ -170,8 +168,8 @@ Floating-Point Objects
Return ``1`` if the given floating-point number *X* is positive or negative
infinity. Return ``0`` otherwise.
.. deprecated:: 3.14
The macro is :term:`soft deprecated`. Use :c:macro:`!isinf` instead.
.. soft-deprecated:: 3.14
Use :c:macro:`!isinf` instead.
.. c:macro:: Py_IS_NAN(X)
@ -179,8 +177,8 @@ Floating-Point Objects
Return ``1`` if the given floating-point number *X* is a not-a-number (NaN)
value. Return ``0`` otherwise.
.. deprecated:: 3.14
The macro is :term:`soft deprecated`. Use :c:macro:`!isnan` instead.
.. soft-deprecated:: 3.14
Use :c:macro:`!isnan` instead.
Pack and Unpack functions
@ -190,24 +188,23 @@ The pack and unpack functions provide an efficient platform-independent way to
store floating-point values as byte strings. The Pack routines produce a bytes
string from a C :c:expr:`double`, and the Unpack routines produce a C
:c:expr:`double` from such a bytes string. The suffix (2, 4 or 8) specifies the
number of bytes in the bytes string.
number of bytes in the bytes string:
On platforms that appear to use IEEE 754 formats these functions work by
copying bits. On other platforms, the 2-byte format is identical to the IEEE
754 binary16 half-precision format, the 4-byte format (32-bit) is identical to
the IEEE 754 binary32 single precision format, and the 8-byte format to the
IEEE 754 binary64 double precision format, although the packing of INFs and
NaNs (if such things exist on the platform) isn't handled correctly, and
attempting to unpack a bytes string containing an IEEE INF or NaN will raise an
exception.
* The 2-byte format is the IEEE 754 binary16 half-precision format.
* The 4-byte format is the IEEE 754 binary32 single-precision format.
* The 8-byte format is the IEEE 754 binary64 double-precision format.
Note that NaNs type may not be preserved on IEEE platforms (signaling NaN become
quiet NaN), for example on x86 systems in 32-bit mode.
The NaN type may not be preserved on some platforms while unpacking (signaling
NaNs become quiet NaNs), for example on x86 systems in 32-bit mode.
It's assumed that the :c:expr:`double` type has the IEEE 754 binary64 double
precision format. What happens if it's not true is partly accidental (alas).
On non-IEEE platforms with more precision, or larger dynamic range, than IEEE
754 supports, not all values can be packed; on non-IEEE platforms with less
precision, or smaller dynamic range, not all values can be unpacked. What
happens in such cases is partly accidental (alas).
precision, or smaller dynamic range, not all values can be unpacked. The
packing of special numbers like INFs and NaNs (if such things exist on the
platform) may not be handled correctly, and attempting to unpack a bytes string
containing an IEEE INF or NaN may raise an exception.
.. versionadded:: 3.11
@ -216,19 +213,14 @@ Pack functions
The pack routines write 2, 4 or 8 bytes, starting at *p*. *le* is an
:c:expr:`int` argument, non-zero if you want the bytes string in little-endian
format (exponent last, at ``p+1``, ``p+3``, or ``p+6`` ``p+7``), zero if you
want big-endian format (exponent first, at *p*). The :c:macro:`PY_BIG_ENDIAN`
constant can be used to use the native endian: it is equal to ``1`` on big
endian processor, or ``0`` on little endian processor.
format (exponent last, at ``p+1``, ``p+3``, or ``p+6`` and ``p+7``), zero if you
want big-endian format (exponent first, at *p*). Use the :c:macro:`!PY_LITTLE_ENDIAN`
constant to select the native endian: it is equal to ``0`` on big
endian processor, or ``1`` on little endian processor.
Return value: ``0`` if all is OK, ``-1`` if error (and an exception is set,
most likely :exc:`OverflowError`).
There are two problems on non-IEEE platforms:
* What this does is undefined if *x* is a NaN or infinity.
* ``-0.0`` and ``+0.0`` produce the same bytes string.
.. c:function:: int PyFloat_Pack2(double x, char *p, int le)
Pack a C double as the IEEE 754 binary16 half-precision format.
@ -241,6 +233,9 @@ There are two problems on non-IEEE platforms:
Pack a C double as the IEEE 754 binary64 double precision format.
.. impl-detail::
This function always succeeds in CPython.
Unpack functions
^^^^^^^^^^^^^^^^
@ -248,16 +243,16 @@ Unpack functions
The unpack routines read 2, 4 or 8 bytes, starting at *p*. *le* is an
:c:expr:`int` argument, non-zero if the bytes string is in little-endian format
(exponent last, at ``p+1``, ``p+3`` or ``p+6`` and ``p+7``), zero if big-endian
(exponent first, at *p*). The :c:macro:`PY_BIG_ENDIAN` constant can be used to
use the native endian: it is equal to ``1`` on big endian processor, or ``0``
(exponent first, at *p*). Use the :c:macro:`!PY_LITTLE_ENDIAN` constant to
select the native endian: it is equal to ``0`` on big endian processor, or ``1``
on little endian processor.
Return value: The unpacked double. On error, this is ``-1.0`` and
:c:func:`PyErr_Occurred` is true (and an exception is set, most likely
:exc:`OverflowError`).
Note that on a non-IEEE platform this will refuse to unpack a bytes string that
represents a NaN or infinity.
.. impl-detail::
These functions always succeed in CPython.
.. c:function:: double PyFloat_Unpack2(const char *p, int le)

22
Doc/c-api/frame.rst
View file

@ -1,6 +1,6 @@
.. highlight:: c
Frame Objects
Frame objects
-------------
.. c:type:: PyFrameObject
@ -147,7 +147,7 @@ See also :ref:`Reflection <reflection>`.
Return the line number that *frame* is currently executing.
Frame Locals Proxies
Frame locals proxies
^^^^^^^^^^^^^^^^^^^^
.. versionadded:: 3.13
@ -169,7 +169,7 @@ See :pep:`667` for more information.
Return non-zero if *obj* is a frame :func:`locals` proxy.
Legacy Local Variable APIs
Legacy local variable APIs
^^^^^^^^^^^^^^^^^^^^^^^^^^
These APIs are :term:`soft deprecated`. As of Python 3.13, they do nothing.
@ -178,40 +178,34 @@ They exist solely for backwards compatibility.
.. c:function:: void PyFrame_LocalsToFast(PyFrameObject *f, int clear)
This function is :term:`soft deprecated` and does nothing.
Prior to Python 3.13, this function would copy the :attr:`~frame.f_locals`
attribute of *f* to the internal "fast" array of local variables, allowing
changes in frame objects to be visible to the interpreter. If *clear* was
true, this function would process variables that were unset in the locals
dictionary.
.. versionchanged:: 3.13
.. soft-deprecated:: 3.13
This function now does nothing.
.. c:function:: void PyFrame_FastToLocals(PyFrameObject *f)
This function is :term:`soft deprecated` and does nothing.
Prior to Python 3.13, this function would copy the internal "fast" array
of local variables (which is used by the interpreter) to the
:attr:`~frame.f_locals` attribute of *f*, allowing changes in local
variables to be visible to frame objects.
.. versionchanged:: 3.13
.. soft-deprecated:: 3.13
This function now does nothing.
.. c:function:: int PyFrame_FastToLocalsWithError(PyFrameObject *f)
This function is :term:`soft deprecated` and does nothing.
Prior to Python 3.13, this function was similar to
:c:func:`PyFrame_FastToLocals`, but would return ``0`` on success, and
``-1`` with an exception set on failure.
.. versionchanged:: 3.13
.. soft-deprecated:: 3.13
This function now does nothing.
@ -219,7 +213,7 @@ They exist solely for backwards compatibility.
:pep:`667`
Internal Frames
Internal frames
^^^^^^^^^^^^^^^
Unless using :pep:`523`, you will not need this.
@ -249,5 +243,3 @@ Unless using :pep:`523`, you will not need this.
Return the currently executing line number, or -1 if there is no line number.
.. versionadded:: 3.12

255
Doc/c-api/gcsupport.rst
View file

@ -220,42 +220,6 @@ The :c:member:`~PyTypeObject.tp_traverse` handler accepts a function parameter o
detection; it's not expected that users will need to write their own
visitor functions.
The :c:member:`~PyTypeObject.tp_traverse` handler must have the following type:
.. c:type:: int (*traverseproc)(PyObject *self, visitproc visit, void *arg)
Traversal function for a container object. Implementations must call the
*visit* function for each object directly contained by *self*, with the
parameters to *visit* being the contained object and the *arg* value passed
to the handler. The *visit* function must not be called with a ``NULL``
object argument. If *visit* returns a non-zero value that value should be
returned immediately.
The traversal function must not have any side effects. Implementations
may not modify the reference counts of any Python objects nor create or
destroy any Python objects.
To simplify writing :c:member:`~PyTypeObject.tp_traverse` handlers, a :c:func:`Py_VISIT` macro is
provided. In order to use this macro, the :c:member:`~PyTypeObject.tp_traverse` implementation
must name its arguments exactly *visit* and *arg*:
.. c:macro:: Py_VISIT(o)
If the :c:expr:`PyObject *` *o* is not ``NULL``, call the *visit* callback, with arguments *o*
and *arg*. If *visit* returns a non-zero value, then return it.
Using this macro, :c:member:`~PyTypeObject.tp_traverse` handlers
look like::
static int
my_traverse(Noddy *self, visitproc visit, void *arg)
{
Py_VISIT(self->foo);
Py_VISIT(self->bar);
return 0;
}
The :c:member:`~PyTypeObject.tp_clear` handler must be of the :c:type:`inquiry` type, or ``NULL``
if the object is immutable.
@ -270,6 +234,225 @@ if the object is immutable.
in a reference cycle.
.. _gc-traversal:
Traversal
---------
The :c:member:`~PyTypeObject.tp_traverse` handler must have the following type:
.. c:type:: int (*traverseproc)(PyObject *self, visitproc visit, void *arg)
Traversal function for a garbage-collected object, used by the garbage
collector to detect reference cycles.
Implementations must call the
*visit* function for each object directly contained by *self*, with the
parameters to *visit* being the contained object and the *arg* value passed
to the handler. The *visit* function must not be called with a ``NULL``
object argument. If *visit* returns a non-zero value, that value should be
returned immediately.
A typical :c:member:`!tp_traverse` function calls the :c:func:`Py_VISIT`
convenience macro on each of the instance's members that are Python
objects that the instance owns.
For example, this is a (slightly outdated) traversal function for
the :py:class:`threading.local` class::
static int
local_traverse(PyObject *op, visitproc visit, void *arg)
{
localobject *self = (localobject *) op;
Py_VISIT(Py_TYPE(self));
Py_VISIT(self->args);
Py_VISIT(self->kw);
Py_VISIT(self->dict);
return 0;
}
.. note::
:c:func:`Py_VISIT` requires the *visit* and *arg* parameters to
:c:func:`!local_traverse` to have these specific names; don't name them just
anything.
Instances of :ref:`heap-allocated types <heap-types>` hold a reference to
their type. Their traversal function must therefore visit the type::
Py_VISIT(Py_TYPE(self));
Alternately, the type may delegate this responsibility by
calling ``tp_traverse`` of a heap-allocated superclass (or another
heap-allocated type, if applicable).
If they do not, the type object may not be garbage-collected.
If the :c:macro:`Py_TPFLAGS_MANAGED_DICT` bit is set in the
:c:member:`~PyTypeObject.tp_flags` field, the traverse function must call
:c:func:`PyObject_VisitManagedDict` like this::
int err = PyObject_VisitManagedDict((PyObject*)self, visit, arg);
if (err) {
return err;
}
Only the members that the instance *owns* (by having
:term:`strong references <strong reference>` to them) must be
visited. For instance, if an object supports weak references via the
:c:member:`~PyTypeObject.tp_weaklist` slot, the pointer supporting
the linked list (what *tp_weaklist* points to) must **not** be
visited as the instance does not directly own the weak references to itself.
The traversal function has a limitation:
.. warning::
The traversal function must not have any side effects. Implementations
may not modify the reference counts of any Python objects nor create or
destroy any Python objects, directly or indirectly.
This means that *most* Python C API functions may not be used, since
they can raise a new exception, return a new reference to a result object,
have internal logic that uses side effects.
Also, unless documented otherwise, functions that happen to not have side
effects may start having them in future versions, without warning.
For a list of safe functions, see a
:ref:`separate section <duringgc-functions>` below.
.. note::
The :c:func:`Py_VISIT` call may be skipped for those members that provably
cannot participate in reference cycles.
In the ``local_traverse`` example above, there is also a ``self->key``
member, but it can only be ``NULL`` or a Python string and therefore
cannot be part of a reference cycle.
On the other hand, even if you know a member can never be part of a cycle,
as a debugging aid you may want to visit it anyway just so the :mod:`gc`
module's :func:`~gc.get_referents` function will include it.
.. note::
The :c:member:`~PyTypeObject.tp_traverse` function can be called from any
thread.
.. impl-detail::
Garbage collection is a "stop-the-world" operation:
even in :term:`free threading` builds, only one thread state is
:term:`attached <attached thread state>` when :c:member:`!tp_traverse`
handlers run.
.. versionchanged:: 3.9
Heap-allocated types are expected to visit ``Py_TYPE(self)`` in
``tp_traverse``. In earlier versions of Python, due to
`bug 40217 <https://bugs.python.org/issue40217>`_, doing this
may lead to crashes in subclasses.
To simplify writing :c:member:`~PyTypeObject.tp_traverse` handlers,
a :c:func:`Py_VISIT` macro is provided.
In order to use this macro, the :c:member:`~PyTypeObject.tp_traverse`
implementation must name its arguments exactly *visit* and *arg*:
.. c:macro:: Py_VISIT(o)
If the :c:expr:`PyObject *` *o* is not ``NULL``, call the *visit*
callback, with arguments *o* and *arg*.
If *visit* returns a non-zero value, then return it.
This corresponds roughly to::
#define Py_VISIT(o) \
if (op) { \
int visit_result = visit(o, arg); \
if (visit_result != 0) { \
return visit_result; \
} \
}
Traversal-safe functions
^^^^^^^^^^^^^^^^^^^^^^^^
The following functions and macros are safe to use in a
:c:member:`~PyTypeObject.tp_traverse` handler:
* the *visit* function passed to ``tp_traverse``
* :c:func:`Py_VISIT`
* :c:func:`Py_SIZE`
* :c:func:`Py_TYPE`: if called from a :c:member:`!tp_traverse` handler,
:c:func:`!Py_TYPE`'s result will be valid for the duration of the handler call
* :c:func:`PyObject_VisitManagedDict`
* :c:func:`PyObject_TypeCheck`, :c:func:`PyType_IsSubtype`,
:c:func:`PyType_HasFeature`
* :samp:`Py{<type>}_Check` and :samp:`Py{<type>}_CheckExact` -- for example,
:c:func:`PyTuple_Check`
* :ref:`duringgc-functions`
.. _duringgc-functions:
"DuringGC" functions
^^^^^^^^^^^^^^^^^^^^
The following functions should *only* be used in a
:c:member:`~PyTypeObject.tp_traverse` handler; calling them in other
contexts may have unintended consequences.
These functions act like their counterparts without the ``_DuringGC`` suffix,
but they are guaranteed to not have side effects, they do not set an exception
on failure, and they return/set :term:`borrowed references <borrowed reference>`
as detailed in the individual documentation.
Note that these functions may fail (return ``NULL`` or ``-1``),
but as they do not set an exception, no error information is available.
In some cases, failure is not distinguishable from a successful ``NULL`` result.
.. c:function:: void *PyObject_GetTypeData_DuringGC(PyObject *o, PyTypeObject *cls)
void *PyObject_GetItemData_DuringGC(PyObject *o)
void *PyType_GetModuleState_DuringGC(PyTypeObject *type)
void *PyModule_GetState_DuringGC(PyObject *module)
int PyModule_GetToken_DuringGC(PyObject *module, void** result)
See :ref:`duringgc-functions` for common information.
.. versionadded:: next
.. seealso::
:c:func:`PyObject_GetTypeData`,
:c:func:`PyObject_GetItemData`,
:c:func:`PyType_GetModuleState`,
:c:func:`PyModule_GetState`,
:c:func:`PyModule_GetToken`,
:c:func:`PyType_GetBaseByToken`
.. c:function:: int PyType_GetBaseByToken_DuringGC(PyTypeObject *type, void *tp_token, PyTypeObject **result)
See :ref:`duringgc-functions` for common information.
Sets *\*result* to a :term:`borrowed reference` rather than a strong one.
The reference is valid for the duration
of the :c:member:`!tp_traverse` handler call.
.. versionadded:: next
.. seealso:: :c:func:`PyType_GetBaseByToken`
.. c:function:: PyObject* PyType_GetModule_DuringGC(PyTypeObject *type)
PyObject* PyType_GetModuleByToken_DuringGC(PyTypeObject *type, const void *mod_token)
See :ref:`duringgc-functions` for common information.
These functions return a :term:`borrowed reference`, which is
valid for the duration of the :c:member:`!tp_traverse` handler call.
.. versionadded:: next
.. seealso::
:c:func:`PyType_GetModule`,
:c:func:`PyType_GetModuleByToken`
Controlling the Garbage Collector State
---------------------------------------

4
Doc/c-api/gen.rst
View file

@ -90,7 +90,9 @@ Deprecated API
.. c:macro:: PyAsyncGenASend_CheckExact(op)
This is a :term:`soft deprecated` API that was included in Python's C API
This is an API that was included in Python's C API
by mistake.
It is solely here for completeness; do not use this API.
.. soft-deprecated:: 3.14

16
Doc/c-api/import.rst
View file

@ -350,14 +350,14 @@ Importing Modules
Gets the current lazy imports mode.
.. versionadded:: next
.. versionadded:: 3.15
.. c:function:: PyObject* PyImport_GetLazyImportsFilter()
Return a :term:`strong reference` to the current lazy imports filter,
or ``NULL`` if none exists. This function always succeeds.
.. versionadded:: next
.. versionadded:: 3.15
.. c:function:: int PyImport_SetLazyImportsMode(PyImport_LazyImportsMode mode)
@ -366,18 +366,20 @@ Importing Modules
This function always returns ``0``.
.. versionadded:: next
.. versionadded:: 3.15
.. c:function:: int PyImport_SetLazyImportsFilter(PyObject *filter)
Sets the current lazy imports filter. The *filter* should be a callable that
will receive ``(importing_module_name, imported_module_name, [fromlist])``
when an import can potentially be lazy and that must return ``True`` if
the import should be lazy and ``False`` otherwise.
when an import can potentially be lazy. The ``imported_module_name`` value
is the resolved module name, so ``lazy from .spam import eggs`` passes
``package.spam``. The callable must return ``True`` if the import should be
lazy and ``False`` otherwise.
Return ``0`` on success and ``-1`` with an exception set otherwise.
.. versionadded:: next
.. versionadded:: 3.15
.. c:type:: PyImport_LazyImportsMode
@ -396,7 +398,7 @@ Importing Modules
Disable lazy imports entirely. Even explicit ``lazy`` statements become
eager imports.
.. versionadded:: next
.. versionadded:: 3.15
.. c:function:: PyObject* PyImport_CreateModuleFromInitfunc(PyObject *spec, PyObject* (*initfunc)(void))

6
Doc/c-api/init_config.rst
View file

@ -1807,10 +1807,10 @@ PyConfig
.. c:member:: wchar_t* run_presite
``package.module`` path to module that should be imported before
``site.py`` is run.
``module`` or ``module:func`` entry point that should be executed before
the :mod:`site` module is imported.
Set by the :option:`-X presite=package.module <-X>` command-line
Set by the :option:`-X presite=module:func <-X>` command-line
option and the :envvar:`PYTHON_PRESITE` environment variable.
The command-line option takes precedence.

5
Doc/c-api/interp-lifecycle.rst
View file

@ -410,6 +410,11 @@ Initializing and finalizing the interpreter
(zero) if not. After :c:func:`Py_FinalizeEx` is called, this returns false until
:c:func:`Py_Initialize` is called again.
.. versionchanged:: next
This function no longer returns true until initialization has fully
completed, including import of the :mod:`site` module. Previously it
could return true while :c:func:`Py_Initialize` was still running.
.. c:function:: int Py_IsFinalizing()

76
Doc/c-api/intro.rst
View file

@ -526,14 +526,24 @@ to the C language.
Outdated macros
---------------
The following macros have been used to features that have been standardized
in C11.
The following :term:`soft deprecated` macros have been used to features that
have been standardized in C11 (or previous standards).
.. c:macro:: Py_ALIGNED(num)
Specify alignment to *num* bytes on compilers that support it.
On some GCC-like compilers, specify alignment to *num* bytes.
This does nothing on other compilers.
Consider using the C11 standard ``_Alignas`` specifier over this macro.
Use the standard ``alignas`` specifier rather than this macro.
.. soft-deprecated:: 3.15
.. c:macro:: PY_FORMAT_SIZE_T
The :c:func:`printf` formatting modifier for :c:type:`size_t`.
Use ``"z"`` directly instead.
.. soft-deprecated:: 3.15
.. c:macro:: Py_LL(number)
Py_ULL(number)
@ -546,24 +556,70 @@ in C11.
Consider using the C99 standard suffixes ``LL`` and ``LLU`` directly.
.. soft-deprecated:: 3.15
.. c:macro:: PY_LONG_LONG
PY_INT32_T
PY_UINT32_T
PY_INT64_T
PY_UINT64_T
Aliases for the types :c:type:`!long long`, :c:type:`!int32_t`,
:c:type:`!uint32_t`. :c:type:`!int64_t` and :c:type:`!uint64_t`,
respectively.
Historically, these types needed compiler-specific extensions.
.. soft-deprecated:: 3.15
.. c:macro:: PY_LLONG_MIN
PY_LLONG_MAX
PY_ULLONG_MAX
PY_SIZE_MAX
Aliases for the values :c:macro:`!LLONG_MIN`, :c:macro:`!LLONG_MAX`,
:c:macro:`!ULLONG_MAX`, and :c:macro:`!SIZE_MAX`, respectively.
Use these standard names instead.
The required header, ``<limits.h>``,
:ref:`is included <capi-system-includes>` in ``Python.h``.
.. soft-deprecated:: 3.15
.. c:macro:: Py_MEMCPY(dest, src, n)
This is a :term:`soft deprecated` alias to :c:func:`!memcpy`.
Use :c:func:`!memcpy` directly instead.
This is an alias to :c:func:`!memcpy`.
.. deprecated:: 3.14
The macro is :term:`soft deprecated`.
.. soft-deprecated:: 3.14
Use :c:func:`!memcpy` directly instead.
.. c:macro:: Py_UNICODE_SIZE
Size of the :c:type:`!wchar_t` type.
Use ``sizeof(wchar_t)`` or ``WCHAR_WIDTH/8`` instead.
The required header for the latter, ``<limits.h>``,
:ref:`is included <capi-system-includes>` in ``Python.h``.
.. soft-deprecated:: 3.15
.. c:macro:: Py_UNICODE_WIDE
Defined if ``wchar_t`` can hold a Unicode character (UCS-4).
Use ``sizeof(wchar_t) >= 4`` instead
.. soft-deprecated:: 3.15
.. c:macro:: Py_VA_COPY
This is a :term:`soft deprecated` alias to the C99-standard ``va_copy``
function.
This is an alias to the C99-standard ``va_copy`` function.
Historically, this would use a compiler-specific method to copy a ``va_list``.
.. versionchanged:: 3.6
This is now an alias to ``va_copy``.
.. soft-deprecated:: 3.15
.. _api-objects:

44
Doc/c-api/list.rst
View file

@ -74,11 +74,25 @@ List Objects
Like :c:func:`PyList_GetItemRef`, but returns a
:term:`borrowed reference` instead of a :term:`strong reference`.
.. note::
In the :term:`free-threaded build`, the returned
:term:`borrowed reference` may become invalid if another thread modifies
the list concurrently. Prefer :c:func:`PyList_GetItemRef`, which returns
a :term:`strong reference`.
.. c:function:: PyObject* PyList_GET_ITEM(PyObject *list, Py_ssize_t i)
Similar to :c:func:`PyList_GetItem`, but without error checking.
.. note::
In the :term:`free-threaded build`, the returned
:term:`borrowed reference` may become invalid if another thread modifies
the list concurrently. Prefer :c:func:`PyList_GetItemRef`, which returns
a :term:`strong reference`.
.. c:function:: int PyList_SetItem(PyObject *list, Py_ssize_t index, PyObject *item)
@ -108,6 +122,14 @@ List Objects
is being replaced; any reference in *list* at position *i* will be
leaked.
.. note::
In the :term:`free-threaded build`, this macro has no internal
synchronization. It is normally only used to fill in new lists where no
other thread has a reference to the list. If the list may be shared,
use :c:func:`PyList_SetItem` instead, which uses a :term:`per-object
lock`.
.. c:function:: int PyList_Insert(PyObject *list, Py_ssize_t index, PyObject *item)
@ -138,6 +160,12 @@ List Objects
Return ``0`` on success, ``-1`` on failure. Indexing from the end of the
list is not supported.
.. note::
In the :term:`free-threaded build`, when *itemlist* is a :class:`list`,
both *list* and *itemlist* are locked for the duration of the operation.
For other iterables (or ``NULL``), only *list* is locked.
.. c:function:: int PyList_Extend(PyObject *list, PyObject *iterable)
@ -150,6 +178,14 @@ List Objects
.. versionadded:: 3.13
.. note::
In the :term:`free-threaded build`, when *iterable* is a :class:`list`,
:class:`set`, :class:`dict`, or dict view, both *list* and *iterable*
(or its underlying dict) are locked for the duration of the operation.
For other iterables, only *list* is locked; *iterable* may be
concurrently modified by another thread.
.. c:function:: int PyList_Clear(PyObject *list)
@ -168,6 +204,14 @@ List Objects
Sort the items of *list* in place. Return ``0`` on success, ``-1`` on
failure. This is equivalent to ``list.sort()``.
.. note::
In the :term:`free-threaded build`, element comparison via
:meth:`~object.__lt__` can execute arbitrary Python code, during which
the :term:`per-object lock` may be temporarily released. For built-in
types (:class:`str`, :class:`int`, :class:`float`), the lock is not
released during comparison.
.. c:function:: int PyList_Reverse(PyObject *list)

4
Doc/c-api/long.rst
View file

@ -197,12 +197,10 @@ distinguished from a number. Use :c:func:`PyErr_Occurred` to disambiguate.
.. c:function:: long PyLong_AS_LONG(PyObject *obj)
A :term:`soft deprecated` alias.
Exactly equivalent to the preferred ``PyLong_AsLong``. In particular,
it can fail with :exc:`OverflowError` or another exception.
.. deprecated:: 3.14
The function is soft deprecated.
.. soft-deprecated:: 3.14
.. c:function:: int PyLong_AsInt(PyObject *obj)

61
Doc/c-api/memory.rst
View file

@ -204,8 +204,11 @@ The following function sets, modeled after the ANSI C standard, but specifying
behavior when requesting zero bytes, are available for allocating and releasing
memory from the Python heap.
The :ref:`default memory allocator <default-memory-allocators>` uses the
:ref:`pymalloc memory allocator <pymalloc>`.
In the GIL-enabled build (default build) the
:ref:`default memory allocator <default-memory-allocators>` uses the
:ref:`pymalloc memory allocator <pymalloc>`, whereas in the
:term:`free-threaded build`, the default is the
:ref:`mimalloc memory allocator <mimalloc>` instead.
.. warning::
@ -215,6 +218,11 @@ The :ref:`default memory allocator <default-memory-allocators>` uses the
The default allocator is now pymalloc instead of system :c:func:`malloc`.
.. versionchanged:: 3.13
In the :term:`free-threaded <free threading>` build, the default allocator
is now :ref:`mimalloc <mimalloc>`.
.. c:function:: void* PyMem_Malloc(size_t n)
Allocates *n* bytes and returns a pointer of type :c:expr:`void*` to the
@ -340,7 +348,9 @@ memory from the Python heap.
the :ref:`Customize Memory Allocators <customize-memory-allocators>` section.
The :ref:`default object allocator <default-memory-allocators>` uses the
:ref:`pymalloc memory allocator <pymalloc>`.
:ref:`pymalloc memory allocator <pymalloc>`. In the
:term:`free-threaded <free threading>` build, the default is the
:ref:`mimalloc memory allocator <mimalloc>` instead.
.. warning::
@ -420,14 +430,16 @@ Default Memory Allocators
Default memory allocators:
=============================== ==================== ================== ===================== ====================
Configuration Name PyMem_RawMalloc PyMem_Malloc PyObject_Malloc
=============================== ==================== ================== ===================== ====================
Release build ``"pymalloc"`` ``malloc`` ``pymalloc`` ``pymalloc``
Debug build ``"pymalloc_debug"`` ``malloc`` + debug ``pymalloc`` + debug ``pymalloc`` + debug
Release build, without pymalloc ``"malloc"`` ``malloc`` ``malloc`` ``malloc``
Debug build, without pymalloc ``"malloc_debug"`` ``malloc`` + debug ``malloc`` + debug ``malloc`` + debug
=============================== ==================== ================== ===================== ====================
=================================== ======================= ==================== ====================== ======================
Configuration Name PyMem_RawMalloc PyMem_Malloc PyObject_Malloc
=================================== ======================= ==================== ====================== ======================
Release build ``"pymalloc"`` ``malloc`` ``pymalloc`` ``pymalloc``
Debug build ``"pymalloc_debug"`` ``malloc`` + debug ``pymalloc`` + debug ``pymalloc`` + debug
Release build, without pymalloc ``"malloc"`` ``malloc`` ``malloc`` ``malloc``
Debug build, without pymalloc ``"malloc_debug"`` ``malloc`` + debug ``malloc`` + debug ``malloc`` + debug
Free-threaded build ``"mimalloc"`` ``mimalloc`` ``mimalloc`` ``mimalloc``
Free-threaded debug build ``"mimalloc_debug"`` ``mimalloc`` + debug ``mimalloc`` + debug ``mimalloc`` + debug
=================================== ======================= ==================== ====================== ======================
Legend:
@ -435,8 +447,7 @@ Legend:
* ``malloc``: system allocators from the standard C library, C functions:
:c:func:`malloc`, :c:func:`calloc`, :c:func:`realloc` and :c:func:`free`.
* ``pymalloc``: :ref:`pymalloc memory allocator <pymalloc>`.
* ``mimalloc``: :ref:`mimalloc memory allocator <mimalloc>`. The pymalloc
allocator will be used if mimalloc support isn't available.
* ``mimalloc``: :ref:`mimalloc memory allocator <mimalloc>`.
* "+ debug": with :ref:`debug hooks on the Python memory allocators
<pymem-debug-hooks>`.
* "Debug build": :ref:`Python build in debug mode <debug-build>`.
@ -733,9 +744,27 @@ The mimalloc allocator
.. versionadded:: 3.13
Python supports the mimalloc allocator when the underlying platform support is available.
mimalloc "is a general purpose allocator with excellent performance characteristics.
Initially developed by Daan Leijen for the runtime systems of the Koka and Lean languages."
Python supports the `mimalloc <https://github.com/microsoft/mimalloc/>`__
allocator when the underlying platform support is available.
mimalloc is a general purpose allocator with excellent performance
characteristics, initially developed by Daan Leijen for the runtime systems
of the Koka and Lean languages.
Unlike :ref:`pymalloc <pymalloc>`, which is optimized for small objects (512
bytes or fewer), mimalloc handles allocations of any size.
In the :term:`free-threaded <free threading>` build, mimalloc is the default
and **required** allocator for the :c:macro:`PYMEM_DOMAIN_MEM` and
:c:macro:`PYMEM_DOMAIN_OBJ` domains. It cannot be disabled in free-threaded
builds. The free-threaded build uses per-thread mimalloc heaps, which allows
allocation and deallocation to proceed without locking in most cases.
In the default (non-free-threaded) build, mimalloc is available but not the
default allocator. It can be selected at runtime using
:envvar:`PYTHONMALLOC`\ ``=mimalloc`` (or ``mimalloc_debug`` to include
:ref:`debug hooks <pymem-debug-hooks>`). It can be disabled at build time
using the :option:`--without-mimalloc` configure option, but this option
cannot be combined with :option:`--disable-gil`.
tracemalloc C API
=================

22
Doc/c-api/module.rst
View file

@ -230,6 +230,9 @@ Feature slots
When creating a module, Python checks the value of this slot
using :c:func:`PyABIInfo_Check`.
This slot is required, except for modules created from
:c:struct:`PyModuleDef`.
.. versionadded:: 3.15
.. c:macro:: Py_mod_multiple_interpreters
@ -620,9 +623,9 @@ rather than from an extension's :ref:`export hook <extension-export-hook>`.
and the :py:class:`~importlib.machinery.ModuleSpec` *spec*.
The *slots* argument must point to an array of :c:type:`PyModuleDef_Slot`
structures, terminated by an entry slot with slot ID of 0
structures, terminated by an entry with slot ID of 0
(typically written as ``{0}`` or ``{0, NULL}`` in C).
The *slots* argument may not be ``NULL``.
The array must include a :c:data:`Py_mod_abi` entry.
The *spec* argument may be any ``ModuleSpec``-like object, as described
in :c:macro:`Py_mod_create` documentation.
@ -682,6 +685,12 @@ remove it.
Usually, there is only one variable of this type for each extension module
defined this way.
The struct, including all members, is part of the
:ref:`Stable ABI <stable-abi>` for non-free-threaded builds (``abi3``).
In the Stable ABI for free-threaded builds (``abi3t``),
this struct is opaque, and unusable in practice; see :ref:`pymoduledef_slot`
for a replacement.
.. c:member:: PyModuleDef_Base m_base
Always initialize this member to :c:macro:`PyModuleDef_HEAD_INIT`:
@ -692,6 +701,11 @@ remove it.
The type of :c:member:`!PyModuleDef.m_base`.
The struct is part of the :ref:`Stable ABI <stable-abi>` for
non-free-threaded builds (``abi3``).
In the Stable ABI for Free-Threaded Builds
(``abi3t``), this struct is opaque, and unusable in practice.
.. c:macro:: PyModuleDef_HEAD_INIT
The required initial value for :c:member:`!PyModuleDef.m_base`.
@ -951,9 +965,7 @@ or code that creates modules dynamically.
// PyModule_AddObject() stole a reference to obj:
// Py_XDECREF(obj) is not needed here.
.. deprecated:: 3.13
:c:func:`PyModule_AddObject` is :term:`soft deprecated`.
.. soft-deprecated:: 3.13
.. c:function:: int PyModule_AddIntConstant(PyObject *module, const char *name, long value)

4
Doc/c-api/monitoring.rst
View file

@ -205,6 +205,4 @@ would typically correspond to a Python function.
.. versionadded:: 3.13
.. deprecated:: 3.14
This function is :term:`soft deprecated`.
.. soft-deprecated:: 3.14

10
Doc/c-api/object.rst
View file

@ -363,6 +363,8 @@ Object Protocol
representation on success, ``NULL`` on failure. This is the equivalent of the
Python expression ``repr(o)``. Called by the :func:`repr` built-in function.
If argument is ``NULL``, return the string ``'<NULL>'``.
.. versionchanged:: 3.4
This function now includes a debug assertion to help ensure that it
does not silently discard an active exception.
@ -377,6 +379,8 @@ Object Protocol
a string similar to that returned by :c:func:`PyObject_Repr` in Python 2.
Called by the :func:`ascii` built-in function.
If argument is ``NULL``, return the string ``'<NULL>'``.
.. index:: string; PyObject_Str (C function)
@ -387,6 +391,8 @@ Object Protocol
Python expression ``str(o)``. Called by the :func:`str` built-in function
and, therefore, by the :func:`print` function.
If argument is ``NULL``, return the string ``'<NULL>'``.
.. versionchanged:: 3.4
This function now includes a debug assertion to help ensure that it
does not silently discard an active exception.
@ -402,6 +408,8 @@ Object Protocol
a TypeError is raised when *o* is an integer instead of a zero-initialized
bytes object.
If argument is ``NULL``, return the :class:`bytes` object ``b'<NULL>'``.
.. c:function:: int PyObject_IsSubclass(PyObject *derived, PyObject *cls)
@ -817,4 +825,4 @@ Object Protocol
Returns 1 if the object was made immortal and returns 0 if it was not.
This function cannot fail.
.. versionadded:: next
.. versionadded:: 3.15

2
Doc/c-api/perfmaps.rst
View file

@ -31,7 +31,7 @@ Note that holding an :term:`attached thread state` is not required for these API
or ``-2`` on failure to create a lock. Check ``errno`` for more information
about the cause of a failure.
.. c:function:: int PyUnstable_WritePerfMapEntry(const void *code_addr, unsigned int code_size, const char *entry_name)
.. c:function:: int PyUnstable_WritePerfMapEntry(const void *code_addr, size_t code_size, const char *entry_name)
Write one single entry to the ``/tmp/perf-$pid.map`` file. This function is
thread safe. Here is what an example entry looks like::

35
Doc/c-api/sentinel.rst Normal file
View file

@ -0,0 +1,35 @@
.. highlight:: c
.. _sentinelobjects:
Sentinel objects
----------------
.. c:var:: PyTypeObject PySentinel_Type
This instance of :c:type:`PyTypeObject` represents the Python
:class:`sentinel` type. This is the same object as :class:`sentinel`.
.. versionadded:: next
.. c:function:: int PySentinel_Check(PyObject *o)
Return true if *o* is a :class:`sentinel` object. The :class:`sentinel` type
does not allow subclasses, so this check is exact.
.. versionadded:: next
.. c:function:: PyObject* PySentinel_New(const char *name, const char *module_name)
Return a new :class:`sentinel` object with :attr:`~sentinel.__name__` set to
*name* and :attr:`~sentinel.__module__` set to *module_name*.
*name* must not be ``NULL``. If *module_name* is ``NULL``, :attr:`~sentinel.__module__`
is set to ``None``.
Return ``NULL`` with an exception set on failure.
For pickling to work, *module_name* must be the name of an importable
module, and the sentinel must be accessible from that module under a
path matching *name*. Pickle treats *name* as a global variable name
in *module_name* (see :meth:`object.__reduce__`).
.. versionadded:: next

5
Doc/c-api/sequence.rst
View file

@ -109,9 +109,8 @@ Sequence Protocol
Alias for :c:func:`PySequence_Contains`.
.. deprecated:: 3.14
The function is :term:`soft deprecated` and should no longer be used to
write new code.
.. soft-deprecated:: 3.14
The function should no longer be used to write new code.
.. c:function:: Py_ssize_t PySequence_Index(PyObject *o, PyObject *value)

35
Doc/c-api/set.rst
View file

@ -89,6 +89,11 @@ the constructor functions work with any iterable Python object.
actually iterable. The constructor is also useful for copying a set
(``c=set(s)``).
.. note::
The operation is atomic on :term:`free threading <free-threaded build>`
when *iterable* is a :class:`set`, :class:`frozenset`, :class:`dict` or :class:`frozendict`.
.. c:function:: PyObject* PyFrozenSet_New(PyObject *iterable)
@ -97,6 +102,11 @@ the constructor functions work with any iterable Python object.
set on success or ``NULL`` on failure. Raise :exc:`TypeError` if *iterable* is
not actually iterable.
.. note::
The operation is atomic on :term:`free threading <free-threaded build>`
when *iterable* is a :class:`set`, :class:`frozenset`, :class:`dict` or :class:`frozendict`.
The following functions and macros are available for instances of :class:`set`
or :class:`frozenset` or instances of their subtypes.
@ -124,6 +134,10 @@ or :class:`frozenset` or instances of their subtypes.
the *key* is unhashable. Raise :exc:`SystemError` if *anyset* is not a
:class:`set`, :class:`frozenset`, or an instance of a subtype.
.. note::
The operation is atomic on :term:`free threading <free-threaded build>`
when *key* is :class:`str`, :class:`int`, :class:`float`, :class:`bool` or :class:`bytes`.
.. c:function:: int PySet_Add(PyObject *set, PyObject *key)
@ -135,6 +149,12 @@ or :class:`frozenset` or instances of their subtypes.
:exc:`SystemError` if *set* is not an instance of :class:`set` or its
subtype.
.. note::
The operation is atomic on :term:`free threading <free-threaded build>`
when *key* is :class:`str`, :class:`int`, :class:`float`, :class:`bool` or :class:`bytes`.
The following functions are available for instances of :class:`set` or its
subtypes but not for instances of :class:`frozenset` or its subtypes.
@ -149,6 +169,11 @@ subtypes but not for instances of :class:`frozenset` or its subtypes.
temporary frozensets. Raise :exc:`SystemError` if *set* is not an
instance of :class:`set` or its subtype.
.. note::
The operation is atomic on :term:`free threading <free-threaded build>`
when *key* is :class:`str`, :class:`int`, :class:`float`, :class:`bool` or :class:`bytes`.
.. c:function:: PyObject* PySet_Pop(PyObject *set)
@ -164,13 +189,19 @@ subtypes but not for instances of :class:`frozenset` or its subtypes.
success. Return ``-1`` and raise :exc:`SystemError` if *set* is not an instance of
:class:`set` or its subtype.
.. note::
In the :term:`free-threaded build`, the set is emptied before its entries
are cleared, so other threads will observe an empty set rather than
intermediate states.
Deprecated API
^^^^^^^^^^^^^^
.. c:macro:: PySet_MINSIZE
A :term:`soft deprecated` constant representing the size of an internal
A constant representing the size of an internal
preallocated table inside :c:type:`PySetObject` instances.
This is documented solely for completeness, as there are no guarantees
@ -180,3 +211,5 @@ Deprecated API
:c:macro:`!PySet_MINSIZE` can be replaced with a small constant like ``8``.
If looking for the size of a set, use :c:func:`PySet_Size` instead.
.. soft-deprecated:: 3.14

284
Doc/c-api/stable.rst
View file

@ -51,135 +51,212 @@ It is generally intended for specialized, low-level tools like debuggers.
Projects that use this API are expected to follow
CPython development and spend extra effort adjusting to changes.
.. _stable-abi:
.. _stable-application-binary-interface:
Stable Application Binary Interface
===================================
Stable Application Binary Interfaces
====================================
For simplicity, this document talks about *extensions*, but the Limited API
and Stable ABI work the same way for all uses of the API – for example,
embedding Python.
.. _limited-c-api:
Limited C API
-------------
Python 3.2 introduced the *Limited API*, a subset of Python's C API.
Extensions that only use the Limited API can be
compiled once and be loaded on multiple versions of Python.
Contents of the Limited API are :ref:`listed below <limited-api-list>`.
.. c:macro:: Py_LIMITED_API
Define this macro before including ``Python.h`` to opt in to only use
the Limited API, and to select the Limited API version.
Define ``Py_LIMITED_API`` to the value of :c:macro:`PY_VERSION_HEX`
corresponding to the lowest Python version your extension supports.
The extension will be ABI-compatible with all Python 3 releases
from the specified one onward, and can use Limited API introduced up to that
version.
Rather than using the ``PY_VERSION_HEX`` macro directly, hardcode a minimum
minor version (e.g. ``0x030A0000`` for Python 3.10) for stability when
compiling with future Python versions.
You can also define ``Py_LIMITED_API`` to ``3``. This works the same as
``0x03020000`` (Python 3.2, the version that introduced Limited API).
.. _stable-abi:
Stable ABI
----------
To enable this, Python provides a *Stable ABI*: a set of symbols that will
remain ABI-compatible across Python 3.x versions.
Python's :dfn:`Stable ABI` allows extensions to be compatible with multiple
versions of Python, without recompilation.
.. note::
The Stable ABI prevents ABI issues, like linker errors due to missing
symbols or data corruption due to changes in structure layouts or function
signatures.
However, other changes in Python can change the *behavior* of extensions.
See Python's Backwards Compatibility Policy (:pep:`387`) for details.
For simplicity, this document talks about *extensions*, but Stable ABI
works the same way for all uses of the API – for example, embedding Python.
The Stable ABI contains symbols exposed in the :ref:`Limited API
<limited-c-api>`, but also other ones – for example, functions necessary to
support older versions of the Limited API.
There are two Stable ABIs:
On Windows, extensions that use the Stable ABI should be linked against
- ``abi3``, introduced in Python 3.2, is compatible with
**non**-:term:`free-threaded <free-threaded build>` builds of CPython.
- ``abi3t``, introduced in Python 3.15, is compatible with
:term:`free-threaded <free-threaded build>` builds of CPython.
It has stricter API limitations than ``abi3``.
.. versionadded:: next
``abi3t`` was added in :pep:`803`
It is possible for an extension to be compiled for *both* ``abi3`` and
``abi3t`` at the same time; the result will be compatible with
both free-threaded and non-free-threaded builds of Python.
Currently, this has no downsides compared to compiling for ``abi3t`` only.
Each Stable ABI is versioned using the first two numbers of the Python version.
For example, Stable ABI 3.14 corresponds to Python 3.14.
An extension compiled for Stable ABI 3.x is ABI-compatible with Python 3.x
and above.
Extensions that target a stable ABI must only use a limited subset of
the C API. This subset is known as the :dfn:`Limited API`; its contents
are :ref:`listed below <limited-api-list>`.
On Windows, extensions that use a Stable ABI should be linked against
``python3.dll`` rather than a version-specific library such as
``python39.dll``.
This library only exposes the relevant symbols.
On some platforms, Python will look for and load shared library files named
with the ``abi3`` tag (e.g. ``mymodule.abi3.so``).
It does not check if such extensions conform to a Stable ABI.
The user (or their packaging tools) need to ensure that, for example,
extensions built with the 3.10+ Limited API are not installed for lower
with the ``abi3`` or ``abi3t`` tag (for example, ``mymodule.abi3.so``).
:term:`Free-threaded <free-threaded build>` interpreters only recognize the
``abi3t`` tag, while non-free-threaded ones will prefer ``abi3`` but fall back
to ``abi3t``.
Thus, extensions compatible with both ABIs should use the ``abi3t`` tag.
Python does not necessarily check that extensions it loads
have compatible ABI.
Extension authors are encouraged to add a check using the :c:macro:`Py_mod_abi`
slot or the :c:func:`PyABIInfo_Check` function, but the user
(or their packaging tool) is ultimately responsible for ensuring that,
for example, extensions built for Stable ABI 3.10 are not installed for lower
versions of Python.
All functions in the Stable ABI are present as functions in Python's shared
library, not solely as macros. This makes them usable from languages that don't
use the C preprocessor.
All functions in Stable ABI are present as functions in Python's shared
library, not solely as macros.
This makes them usable are usable from languages that don't use the C
preprocessor, including Python's :py:mod:`ctypes`.
Limited API Scope and Performance
---------------------------------
.. _abi3-compiling:
The goal for the Limited API is to allow everything that is possible with the
Compiling for Stable ABI
------------------------
.. note::
Build tools (such as, for example, meson-python, scikit-build-core,
or Setuptools) often have a mechanism for setting macros and synchronizing
them with extension filenames and other metadata.
Prefer using such a mechanism, if it exists, over defining the
macros manually.
The rest of this section is mainly relevant for tool authors, and for
people who compile extensions manually.
.. seealso:: `list of recommended tools`_ in the Python Packaging User Guide
.. _list of recommended tools: https://packaging.python.org/en/latest/guides/tool-recommendations/#build-backends-for-extension-modules
To compile for a Stable ABI, define one or both of the following macros
to the lowest Python version your extension should support, in
:c:macro:`Py_PACK_VERSION` format.
Typically, you should choose a specific value rather than the version of
the Python headers you are compiling against.
The macros must be defined before including ``Python.h``.
Since :c:macro:`Py_PACK_VERSION` is not available at this point, you
will need to use the numeric value directly.
For reference, the values for a few recent Python versions are:
.. version-hex-cheatsheet::
When one of the macros is defined, ``Python.h`` will only expose API that is
compatible with the given Stable ABI -- that is, the
:ref:`Limited API <limited-api-list>` plus some definitions that need to be
visible to the compiler but should not be used directly.
When both are defined, ``Python.h`` will only expose API compatible with
both Stable ABIs.
.. c:macro:: Py_LIMITED_API
Target ``abi3``, that is,
non-:term:`free-threaded builds <free-threaded build>` of CPython.
See :ref:`above <abi3-compiling>` for common information.
.. c:macro:: Py_TARGET_ABI3T
Target ``abi3t``, that is,
:term:`free-threaded builds <free-threaded build>` of CPython.
See :ref:`above <abi3-compiling>` for common information.
.. versionadded:: next
Both macros specify a target ABI; the different naming style is due to
backwards compatibility.
.. admonition:: Historical note
You can also define ``Py_LIMITED_API`` as ``3``. This works the same as
``0x03020000`` (Python 3.2, the version that introduced Stable ABI).
When both are defined, ``Python.h`` may, or may not, redefine
:c:macro:`!Py_LIMITED_API` to match :c:macro:`!Py_TARGET_ABI3T`.
On a :term:`free-threaded build` -- that is, when
:c:macro:`Py_GIL_DISABLED` is defined -- :c:macro:`!Py_TARGET_ABI3T`
defaults to the value of :c:macro:`!Py_LIMITED_API`.
This means that there are two ways to build for both ``abi3`` and ``abi3t``:
- define both :c:macro:`!Py_LIMITED_API` and :c:macro:`!Py_TARGET_ABI3T`, or
- define only :c:macro:`!Py_LIMITED_API` and:
- on Windows, define :c:macro:`!Py_GIL_DISABLED`;
- on other systems, use the headers of free-threaded build of Python.
.. _limited-api-scope-and-performance:
Stable ABI Scope and Performance
--------------------------------
The goal for Stable ABI is to allow everything that is possible with the
full C API, but possibly with a performance penalty.
Generally, compatibility with Stable ABI will require some changes to an
extension's source code.
For example, while :c:func:`PyList_GetItem` is available, its “unsafe” macro
For example, while :c:func:`PyList_GetItem` is available, its "unsafe" macro
variant :c:func:`PyList_GET_ITEM` is not.
The macro can be faster because it can rely on version-specific implementation
details of the list object.
Without ``Py_LIMITED_API`` defined, some C API functions are inlined or
replaced by macros.
Defining ``Py_LIMITED_API`` disables this inlining, allowing stability as
For another example, when *not* compiling for Stable ABI, some C API
functions are inlined or replaced by macros.
Compiling for Stable ABI disables this inlining, allowing stability as
Python's data structures are improved, but possibly reducing performance.
By leaving out the ``Py_LIMITED_API`` definition, it is possible to compile
a Limited API extension with a version-specific ABI. This can improve
performance for that Python version, but will limit compatibility.
Compiling with ``Py_LIMITED_API`` will then yield an extension that can be
distributed where a version-specific one is not available – for example,
for prereleases of an upcoming Python version.
By leaving out the :c:macro:`!Py_LIMITED_API` or :c:macro:`!Py_TARGET_ABI3T`
definition, it is possible to compile Stable-ABI-compatible source
for a version-specific ABI.
A potentially faster version-specific extension can then be distributed
alongside a version compiled for Stable ABI -- a slower but more compatible
fallback.
Limited API Caveats
-------------------
.. _limited-api-caveats:
Note that compiling with ``Py_LIMITED_API`` is *not* a complete guarantee that
code conforms to the :ref:`Limited API <limited-c-api>` or the :ref:`Stable ABI
<stable-abi>`. ``Py_LIMITED_API`` only covers definitions, but an API also
includes other issues, such as expected semantics.
Stable ABI Caveats
------------------
One issue that ``Py_LIMITED_API`` does not guard against is calling a function
with arguments that are invalid in a lower Python version.
Note that compiling for Stable ABI is *not* a complete guarantee that code will
be compatible with the expected Python versions.
Stable ABI prevents *ABI* issues, like linker errors due to missing
symbols or data corruption due to changes in structure layouts or function
signatures.
However, other changes in Python can change the *behavior* of extensions.
One issue that the :c:macro:`Py_TARGET_ABI3T` and :c:macro:`Py_LIMITED_API`
macros do not guard against is calling a function with arguments that are
invalid in a lower Python version.
For example, consider a function that starts accepting ``NULL`` for an
argument. In Python 3.9, ``NULL`` now selects a default behavior, but in
Python 3.8, the argument will be used directly, causing a ``NULL`` dereference
and crash. A similar argument works for fields of structs.
Another issue is that some struct fields are currently not hidden when
``Py_LIMITED_API`` is defined, even though they're part of the Limited API.
For these reasons, we recommend testing an extension with *all* minor Python
versions it supports, and preferably to build with the *lowest* such version.
versions it supports.
We also recommend reviewing documentation of all used API to check
if it is explicitly part of the Limited API. Even with ``Py_LIMITED_API``
defined, a few private declarations are exposed for technical reasons (or
even unintentionally, as bugs).
Also note that the Limited API is not necessarily stable: compiling with
``Py_LIMITED_API`` with Python 3.8 means that the extension will
run with Python 3.12, but it will not necessarily *compile* with Python 3.12.
In particular, parts of the Limited API may be deprecated and removed,
provided that the Stable ABI stays stable.
Also note that while compiling with ``Py_LIMITED_API`` 3.8 means that the
extension should *load* on Python 3.12, and *compile* with Python 3.12,
the same source will not necessarily compile with ``Py_LIMITED_API``
set to 3.12.
In general, parts of the Limited API may be deprecated and removed,
provided that Stable ABI stays stable.
.. _stable-abi-platform:
@ -189,12 +266,12 @@ Platform Considerations
ABI stability depends not only on Python, but also on the compiler used,
lower-level libraries and compiler options. For the purposes of
the :ref:`Stable ABI <stable-abi>`, these details define a “platform”. They
the :ref:`Stable ABIs <stable-abi>`, these details define a “platform”. They
usually depend on the OS type and processor architecture
It is the responsibility of each particular distributor of Python
to ensure that all Python versions on a particular platform are built
in a way that does not break the Stable ABI.
in a way that does not break the Stable ABIs, or the version-specific ABIs.
This is the case with Windows and macOS releases from ``python.org`` and many
third-party distributors.
@ -302,7 +379,7 @@ The full API is described below for advanced use cases.
.. c:macro:: PyABIInfo_STABLE
Specifies that the stable ABI is used.
Specifies that Stable ABI is used.
.. c:macro:: PyABIInfo_INTERNAL
@ -313,15 +390,22 @@ The full API is described below for advanced use cases.
.. c:macro:: PyABIInfo_FREETHREADED
Specifies ABI compatible with free-threading builds of CPython.
Specifies ABI compatible with :term:`free-threaded builds
<free-threaded build>` of CPython.
(That is, ones compiled with :option:`--disable-gil`; with ``t``
in :py:data:`sys.abiflags`)
.. c:macro:: PyABIInfo_GIL
Specifies ABI compatible with non-free-threading builds of CPython
Specifies ABI compatible with non-free-threaded builds of CPython
(ones compiled *without* :option:`--disable-gil`).
.. c:macro:: PyABIInfo_FREETHREADING_AGNOSTIC
Specifies ABI compatible with both free-threaded and
non-free-threaded builds of CPython, that is, both
``abi3`` and ``abi3t``.
.. c:member:: uint32_t build_version
The version of the Python headers used to build the code, in the format
@ -335,10 +419,11 @@ The full API is described below for advanced use cases.
The ABI version.
For the Stable ABI, this field should be the value of
:c:macro:`Py_LIMITED_API`
(except if :c:macro:`Py_LIMITED_API` is ``3``; use
:c:expr:`Py_PACK_VERSION(3, 2)` in that case).
For Stable ABI, this field should be the value of
:c:macro:`Py_LIMITED_API` or :c:macro:`Py_TARGET_ABI3T`.
If both are defined, use the smaller value.
(If :c:macro:`Py_LIMITED_API` is ``3``; use
:c:expr:`Py_PACK_VERSION(3, 2)` instead of ``3``.)
Otherwise, it should be set to :c:macro:`PY_VERSION_HEX`.
@ -355,12 +440,13 @@ The full API is described below for advanced use cases.
.. versionadded:: 3.15
.. _limited-c-api:
.. _limited-api-list:
Contents of Limited API
=======================
Currently, the :ref:`Limited API <limited-c-api>` includes the following items:
This is the definitive list of :ref:`Limited API <limited-c-api>` for
Python |version|:
.. limited-api-list::

33
Doc/c-api/structures.rst
View file

@ -33,6 +33,13 @@ under :ref:`reference counting <countingrefs>`.
The members must not be accessed directly; instead use macros such as
:c:macro:`Py_REFCNT` and :c:macro:`Py_TYPE`.
In the :ref:`Stable ABI <stable-abi>` for Free-Threaded Builds (``abi3t``),
this struct is opaque; its size and layout may change between
Python versions.
In Stable ABI for non-free-threaded builds (``abi3``), the
:c:member:`!ob_refcnt` and :c:member:`!ob_type` fields are available,
but using them directly is discouraged.
.. c:member:: Py_ssize_t ob_refcnt
The object's reference count, as returned by :c:macro:`Py_REFCNT`.
@ -48,6 +55,19 @@ under :ref:`reference counting <countingrefs>`.
Do not use this field directly; use :c:macro:`Py_TYPE` and
:c:func:`Py_SET_TYPE` instead.
.. c:member:: PyMutex ob_mutex
A :ref:`per-object lock <per-object-locks>`, present only in the :term:`free-threaded <free threading>`
build (when :c:macro:`Py_GIL_DISABLED` is defined).
This field is **reserved for use by the critical section API**
(:c:macro:`Py_BEGIN_CRITICAL_SECTION` / :c:macro:`Py_END_CRITICAL_SECTION`).
Do **not** lock it directly with ``PyMutex_Lock``; doing so can cause
deadlocks. If you need your own lock, add a separate :c:type:`PyMutex`
field to your object struct.
.. versionadded:: 3.13
.. c:type:: PyVarObject
@ -59,6 +79,19 @@ under :ref:`reference counting <countingrefs>`.
instead use macros such as :c:macro:`Py_SIZE`, :c:macro:`Py_REFCNT` and
:c:macro:`Py_TYPE`.
In the :ref:`Stable ABI <stable-abi>` for Free-Threaded Builds (``abi3t``),
this struct is opaque; its size and layout may change between
Python versions.
In Stable ABI for non-free-threaded builds (``abi3``), the
:c:member:`!ob_base` and :c:member:`!ob_size` fields are available,
but using them directly is discouraged.
.. c:member:: PyObject ob_base
Common object header.
Typically, this field is not accessed directly; instead
:c:type:`!PyVarObject` can be cast to :c:type:`PyObject`.
.. c:member:: Py_ssize_t ob_size
A size field, whose contents should be considered an object's internal

21
Doc/c-api/subinterpreters.rst
View file

@ -399,6 +399,27 @@ High-level APIs
.. versionadded:: 3.9
.. c:function:: void _PyInterpreterState_SetEvalFrameAllowSpecialization(PyInterpreterState *interp, int allow_specialization)
Enables or disables specialization why a custom frame evaluator is in place.
If *allow_specialization* is non-zero, the adaptive specializer will
continue to specialize bytecodes even though a custom eval frame function
is set. When *allow_specialization* is zero, setting a custom eval frame
disables specialization. The standard interpreter loop will continue to deopt
while a frame evaluation API is in place - the frame evaluation function needs
to handle the specialized opcodes to take advantage of this.
.. versionadded:: 3.15
.. c:function:: int _PyInterpreterState_IsSpecializationEnabled(PyInterpreterState *interp)
Return non-zero if adaptive specialization is enabled for the interpreter.
Specialization is enabled when no custom eval frame function is set, or
when one is set with *allow_specialization* enabled.
.. versionadded:: 3.15
Low-level APIs
--------------

218
Doc/c-api/threads.rst
View file

@ -10,43 +10,63 @@ Thread states and the global interpreter lock
single: interpreter lock
single: lock, interpreter
Unless on a :term:`free-threaded <free threading>` build of :term:`CPython`,
the Python interpreter is not fully thread-safe. In order to support
Unless on a :term:`free-threaded build` of :term:`CPython`,
the Python interpreter is generally not thread-safe. In order to support
multi-threaded Python programs, there's a global lock, called the :term:`global
interpreter lock` or :term:`GIL`, that must be held by the current thread before
it can safely access Python objects. Without the lock, even the simplest
operations could cause problems in a multi-threaded program: for example, when
interpreter lock` or :term:`GIL`, that must be held by a thread before
accessing Python objects. Without the lock, even the simplest operations
could cause problems in a multi-threaded program: for example, when
two threads simultaneously increment the reference count of the same object, the
reference count could end up being incremented only once instead of twice.
As such, only a thread that holds the GIL may operate on Python objects or
invoke Python's C API.
.. index:: single: setswitchinterval (in module sys)
Therefore, the rule exists that only the thread that has acquired the
:term:`GIL` may operate on Python objects or call Python/C API functions.
In order to emulate concurrency of execution, the interpreter regularly
tries to switch threads (see :func:`sys.setswitchinterval`). The lock is also
released around potentially blocking I/O operations like reading or writing
a file, so that other Python threads can run in the meantime.
In order to emulate concurrency, the interpreter regularly tries to switch
threads between bytecode instructions (see :func:`sys.setswitchinterval`).
This is why locks are also necessary for thread-safety in pure-Python code.
Additionally, the global interpreter lock is released around blocking I/O
operations, such as reading or writing to a file. From the C API, this is done
by :ref:`detaching the thread state <detaching-thread-state>`.
.. index::
single: PyThreadState (C type)
The Python interpreter keeps some thread-specific bookkeeping information
inside a data structure called :c:type:`PyThreadState`, known as a :term:`thread state`.
Each OS thread has a thread-local pointer to a :c:type:`PyThreadState`; a thread state
The Python interpreter keeps some thread-local information inside
a data structure called :c:type:`PyThreadState`, known as a :term:`thread state`.
Each thread has a thread-local pointer to a :c:type:`PyThreadState`; a thread state
referenced by this pointer is considered to be :term:`attached <attached thread state>`.
A thread can only have one :term:`attached thread state` at a time. An attached
thread state is typically analogous with holding the :term:`GIL`, except on
:term:`free-threaded <free threading>` builds. On builds with the :term:`GIL` enabled,
:term:`attaching <attached thread state>` a thread state will block until the :term:`GIL`
can be acquired. However, even on builds with the :term:`GIL` disabled, it is still required
to have an attached thread state to call most of the C API.
thread state is typically analogous with holding the GIL, except on
free-threaded builds. On builds with the GIL enabled, attaching a thread state
will block until the GIL can be acquired. However, even on builds with the GIL
disabled, it is still required to have an attached thread state, as the interpreter
needs to keep track of which threads may access Python objects.
In general, there will always be an :term:`attached thread state` when using Python's C API.
Only in some specific cases (such as in a :c:macro:`Py_BEGIN_ALLOW_THREADS` block) will the
thread not have an attached thread state. If uncertain, check if :c:func:`PyThreadState_GetUnchecked` returns
``NULL``.
.. note::
Even on the free-threaded build, attaching a thread state may block, as the
GIL can be re-enabled or threads might be temporarily suspended (such as during
a garbage collection).
Generally, there will always be an attached thread state when using Python's
C API, including during embedding and when implementing methods, so it's uncommon
to need to set up a thread state on your own. Only in some specific cases, such
as in a :c:macro:`Py_BEGIN_ALLOW_THREADS` block or in a fresh thread, will the
thread not have an attached thread state.
If uncertain, check if :c:func:`PyThreadState_GetUnchecked` returns ``NULL``.
If it turns out that you do need to create a thread state, call :c:func:`PyThreadState_New`
followed by :c:func:`PyThreadState_Swap`, or use the dangerous
:c:func:`PyGILState_Ensure` function.
.. _detaching-thread-state:
Detaching the thread state from extension code
----------------------------------------------
@ -86,28 +106,37 @@ The block above expands to the following code::
Here is how these functions work:
The :term:`attached thread state` holds the :term:`GIL` for the entire interpreter. When detaching
the :term:`attached thread state`, the :term:`GIL` is released, allowing other threads to attach
a thread state to their own thread, thus getting the :term:`GIL` and can start executing.
The pointer to the prior :term:`attached thread state` is stored as a local variable.
Upon reaching :c:macro:`Py_END_ALLOW_THREADS`, the thread state that was
previously :term:`attached <attached thread state>` is passed to :c:func:`PyEval_RestoreThread`.
This function will block until another releases its :term:`thread state <attached thread state>`,
thus allowing the old :term:`thread state <attached thread state>` to get re-attached and the
C API can be called again.
The attached thread state implies that the GIL is held for the interpreter.
To detach it, :c:func:`PyEval_SaveThread` is called and the result is stored
in a local variable.
For :term:`free-threaded <free threading>` builds, the :term:`GIL` is normally
out of the question, but detaching the :term:`thread state <attached thread state>` is still required
for blocking I/O and long operations. The difference is that threads don't have to wait for the :term:`GIL`
to be released to attach their thread state, allowing true multi-core parallelism.
By detaching the thread state, the GIL is released, which allows other threads
to attach to the interpreter and execute while the current thread performs
blocking I/O. When the I/O operation is complete, the old thread state is
reattached by calling :c:func:`PyEval_RestoreThread`, which will wait until
the GIL can be acquired.
.. note::
Calling system I/O functions is the most common use case for detaching
the :term:`thread state <attached thread state>`, but it can also be useful before calling
long-running computations which don't need access to Python objects, such
as compression or cryptographic functions operating over memory buffers.
Performing blocking I/O is the most common use case for detaching
the thread state, but it is also useful to call it over long-running
native code that doesn't need access to Python objects or Python's C API.
For example, the standard :mod:`zlib` and :mod:`hashlib` modules detach the
:term:`thread state <attached thread state>` when compressing or hashing data.
:term:`thread state <attached thread state>` when compressing or hashing
data.
On a :term:`free-threaded build`, the :term:`GIL` is usually out of the question,
but **detaching the thread state is still required**, because the interpreter
periodically needs to block all threads to get a consistent view of Python objects
without the risk of race conditions.
For example, CPython currently suspends all threads for a short period of time
while running the garbage collector.
.. warning::
Detaching the thread state can lead to unexpected behavior during interpreter
finalization. See :ref:`cautions-regarding-runtime-finalization` for more
details.
APIs
^^^^
@ -149,28 +178,74 @@ example usage in the Python source distribution.
declaration.
.. _gilstate:
Non-Python created threads
--------------------------
When threads are created using the dedicated Python APIs (such as the
:mod:`threading` module), a thread state is automatically associated to them
and the code shown above is therefore correct. However, when threads are
created from C (for example by a third-party library with its own thread
management), they don't hold the :term:`GIL`, because they don't have an
:term:`attached thread state`.
:mod:`threading` module), a thread state is automatically associated with them,
However, when a thread is created from native code (for example, by a
third-party library with its own thread management), it doesn't hold an
attached thread state.
If you need to call Python code from these threads (often this will be part
of a callback API provided by the aforementioned third-party library),
you must first register these threads with the interpreter by
creating an :term:`attached thread state` before you can start using the Python/C
API. When you are done, you should detach the :term:`thread state <attached thread state>`, and
finally free it.
creating a new thread state and attaching it.
The :c:func:`PyGILState_Ensure` and :c:func:`PyGILState_Release` functions do
all of the above automatically. The typical idiom for calling into Python
from a C thread is::
The most robust way to do this is through :c:func:`PyThreadState_New` followed
by :c:func:`PyThreadState_Swap`.
.. note::
``PyThreadState_New`` requires an argument pointing to the desired
interpreter; such a pointer can be acquired via a call to
:c:func:`PyInterpreterState_Get` from the code where the thread was
created.
For example::
/* The return value of PyInterpreterState_Get() from the
function that created this thread. */
PyInterpreterState *interp = thread_data->interp;
/* Create a new thread state for the interpreter. It does not start out
attached. */
PyThreadState *tstate = PyThreadState_New(interp);
/* Attach the thread state, which will acquire the GIL. */
PyThreadState_Swap(tstate);
/* Perform Python actions here. */
result = CallSomeFunction();
/* evaluate result or handle exception */
/* Destroy the thread state. No Python API allowed beyond this point. */
PyThreadState_Clear(tstate);
PyThreadState_DeleteCurrent();
.. warning::
If the interpreter finalized before ``PyThreadState_Swap`` was called, then
``interp`` will be a dangling pointer!
.. _gilstate:
Legacy API
----------
Another common pattern to call Python code from a non-Python thread is to use
:c:func:`PyGILState_Ensure` followed by a call to :c:func:`PyGILState_Release`.
These functions do not work well when multiple interpreters exist in the Python
process. If no Python interpreter has ever been used in the current thread (which
is common for threads created outside Python), ``PyGILState_Ensure`` will create
and attach a thread state for the "main" interpreter (the first interpreter in
the Python process).
Additionally, these functions have thread-safety issues during interpreter
finalization. Using ``PyGILState_Ensure`` during finalization will likely
crash the process.
Usage of these functions look like such::
PyGILState_STATE gstate;
gstate = PyGILState_Ensure();
@ -182,41 +257,6 @@ from a C thread is::
/* Release the thread. No Python API allowed beyond this point. */
PyGILState_Release(gstate);
Note that the ``PyGILState_*`` functions assume there is only one global
interpreter (created automatically by :c:func:`Py_Initialize`). Python
supports the creation of additional interpreters (using
:c:func:`Py_NewInterpreter`), but mixing multiple interpreters and the
``PyGILState_*`` API is unsupported. This is because :c:func:`PyGILState_Ensure`
and similar functions default to :term:`attaching <attached thread state>` a
:term:`thread state` for the main interpreter, meaning that the thread can't safely
interact with the calling subinterpreter.
Supporting subinterpreters in non-Python threads
------------------------------------------------
If you would like to support subinterpreters with non-Python created threads, you
must use the ``PyThreadState_*`` API instead of the traditional ``PyGILState_*``
API.
In particular, you must store the interpreter state from the calling
function and pass it to :c:func:`PyThreadState_New`, which will ensure that
the :term:`thread state` is targeting the correct interpreter::
/* The return value of PyInterpreterState_Get() from the
function that created this thread. */
PyInterpreterState *interp = ThreadData->interp;
PyThreadState *tstate = PyThreadState_New(interp);
PyThreadState_Swap(tstate);
/* GIL of the subinterpreter is now held.
Perform Python actions here. */
result = CallSomeFunction();
/* evaluate result or handle exception */
/* Destroy the thread state. No Python API allowed beyond this point. */
PyThreadState_Clear(tstate);
PyThreadState_DeleteCurrent();
.. _fork-and-threads:

7
Doc/c-api/tuple.rst
View file

@ -99,7 +99,8 @@ Tuple Objects
Insert a reference to object *o* at position *pos* of the tuple pointed to by
*p*. Return ``0`` on success. If *pos* is out of bounds, return ``-1``
and set an :exc:`IndexError` exception.
and set an :exc:`IndexError` exception. This function should only be used to fill in brand new tuples;
using it on an existing tuple is thread-unsafe.
.. note::
@ -110,7 +111,7 @@ Tuple Objects
.. c:function:: void PyTuple_SET_ITEM(PyObject *p, Py_ssize_t pos, PyObject *o)
Like :c:func:`PyTuple_SetItem`, but does no error checking, and should *only* be
used to fill in brand new tuples.
used to fill in brand new tuples, using it on an existing tuple is thread-unsafe.
Bounds checking is performed as an assertion if Python is built in
:ref:`debug mode <debug-build>` or :option:`with assertions <--with-assertions>`.
@ -236,6 +237,8 @@ type.
.. c:function:: PyObject* PyStructSequence_GetItem(PyObject *p, Py_ssize_t pos)
Return the object at position *pos* in the struct sequence pointed to by *p*.
The returned reference is borrowed from the struct sequence *p*
(that is: it is only valid as long as you hold a reference to *p*).
Bounds checking is performed as an assertion if Python is built in
:ref:`debug mode <debug-build>` or :option:`with assertions <--with-assertions>`.

4
Doc/c-api/type.rst
View file

@ -274,6 +274,10 @@ Type Objects
Return the module object associated with the given type when the type was
created using :c:func:`PyType_FromModuleAndSpec`.
The returned reference is :term:`borrowed <borrowed reference>` from *type*,
and will be valid as long as you hold a reference to *type*.
Do not release it with :c:func:`Py_DECREF` or similar.
If no module is associated with the given type, sets :py:class:`TypeError`
and returns ``NULL``.

124
Doc/c-api/typeobj.rst
View file

@ -1391,8 +1391,8 @@ and :c:data:`PyType_Type` effectively act as defaults.)
.. versionchanged:: 3.9
Renamed to the current name, without the leading underscore.
The old provisional name is :term:`soft deprecated`.
Renamed to the current name, without the leading underscore.
The old provisional name is :term:`soft deprecated`.
.. versionchanged:: 3.12
@ -1501,11 +1501,13 @@ and :c:data:`PyType_Type` effectively act as defaults.)
.. c:macro:: Py_TPFLAGS_HAVE_VERSION_TAG
This is a :term:`soft deprecated` macro that does nothing.
This macro does nothing.
Historically, this would indicate that the
:c:member:`~PyTypeObject.tp_version_tag` field was available and
initialized.
.. soft-deprecated:: 3.13
.. c:macro:: Py_TPFLAGS_INLINE_VALUES
@ -1563,93 +1565,9 @@ and :c:data:`PyType_Type` effectively act as defaults.)
.. corresponding-type-slot:: Py_tp_traverse
An optional pointer to a traversal function for the garbage collector. This is
only used if the :c:macro:`Py_TPFLAGS_HAVE_GC` flag bit is set. The signature is::
only used if the :c:macro:`Py_TPFLAGS_HAVE_GC` flag bit is set.
int tp_traverse(PyObject *self, visitproc visit, void *arg);
More information about Python's garbage collection scheme can be found
in section :ref:`supporting-cycle-detection`.
The :c:member:`~PyTypeObject.tp_traverse` pointer is used by the garbage collector to detect
reference cycles. A typical implementation of a :c:member:`~PyTypeObject.tp_traverse` function
simply calls :c:func:`Py_VISIT` on each of the instance's members that are Python
objects that the instance owns. For example, this is function :c:func:`!local_traverse` from the
:mod:`!_thread` extension module::
static int
local_traverse(PyObject *op, visitproc visit, void *arg)
{
localobject *self = (localobject *) op;
Py_VISIT(self->args);
Py_VISIT(self->kw);
Py_VISIT(self->dict);
return 0;
}
Note that :c:func:`Py_VISIT` is called only on those members that can participate
in reference cycles. Although there is also a ``self->key`` member, it can only
be ``NULL`` or a Python string and therefore cannot be part of a reference cycle.
On the other hand, even if you know a member can never be part of a cycle, as a
debugging aid you may want to visit it anyway just so the :mod:`gc` module's
:func:`~gc.get_referents` function will include it.
Heap types (:c:macro:`Py_TPFLAGS_HEAPTYPE`) must visit their type with::
Py_VISIT(Py_TYPE(self));
It is only needed since Python 3.9. To support Python 3.8 and older, this
line must be conditional::
#if PY_VERSION_HEX >= 0x03090000
Py_VISIT(Py_TYPE(self));
#endif
If the :c:macro:`Py_TPFLAGS_MANAGED_DICT` bit is set in the
:c:member:`~PyTypeObject.tp_flags` field, the traverse function must call
:c:func:`PyObject_VisitManagedDict` like this::
PyObject_VisitManagedDict((PyObject*)self, visit, arg);
.. warning::
When implementing :c:member:`~PyTypeObject.tp_traverse`, only the
members that the instance *owns* (by having :term:`strong references
<strong reference>` to them) must be
visited. For instance, if an object supports weak references via the
:c:member:`~PyTypeObject.tp_weaklist` slot, the pointer supporting
the linked list (what *tp_weaklist* points to) must **not** be
visited as the instance does not directly own the weak references to itself
(the weakreference list is there to support the weak reference machinery,
but the instance has no strong reference to the elements inside it, as they
are allowed to be removed even if the instance is still alive).
.. warning::
The traversal function must not have any side effects. It must not
modify the reference counts of any Python objects nor create or destroy
any Python objects.
Note that :c:func:`Py_VISIT` requires the *visit* and *arg* parameters to
:c:func:`!local_traverse` to have these specific names; don't name them just
anything.
Instances of :ref:`heap-allocated types <heap-types>` hold a reference to
their type. Their traversal function must therefore either visit
:c:func:`Py_TYPE(self) <Py_TYPE>`, or delegate this responsibility by
calling ``tp_traverse`` of another heap-allocated type (such as a
heap-allocated superclass).
If they do not, the type object may not be garbage-collected.
.. note::
The :c:member:`~PyTypeObject.tp_traverse` function can be called from any
thread.
.. versionchanged:: 3.9
Heap-allocated types are expected to visit ``Py_TYPE(self)`` in
``tp_traverse``. In earlier versions of Python, due to
`bug 40217 <https://bugs.python.org/issue40217>`_, doing this
may lead to crashes in subclasses.
See :ref:`gc-traversal` for documentation.
**Inheritance:**
@ -3057,6 +2975,24 @@ Buffer Object Structures
(5) Return ``0``.
**Thread safety:**
In the :term:`free-threaded build`, implementations must ensure:
* The export counter increment in step (3) is atomic.
* The underlying buffer data remains valid and at a stable memory
location for the lifetime of all exports.
* For objects that support resizing or reallocation (such as
:class:`bytearray`), the export counter is checked atomically before
such operations, and :exc:`BufferError` is raised if exports exist.
* The function is safe to call concurrently from multiple threads.
See also :ref:`thread-safety-memoryview` for the Python-level
thread safety guarantees of :class:`memoryview` objects.
If *exporter* is part of a chain or tree of buffer providers, two main
schemes can be used:
@ -3102,6 +3038,16 @@ Buffer Object Structures
(2) If the counter is ``0``, free all memory associated with *view*.
**Thread safety:**
In the :term:`free-threaded build`:
* The export counter decrement in step (1) must be atomic.
* Resource cleanup when the counter reaches zero must be done atomically,
as the final release may race with concurrent releases from other
threads and dellocation must only happen once.
The exporter MUST use the :c:member:`~Py_buffer.internal` field to keep
track of buffer-specific resources. This field is guaranteed to remain
constant, while a consumer MAY pass a copy of the original buffer as the

14
Doc/c-api/unicode.rst
View file

@ -1855,8 +1855,6 @@ object.
On success, return ``0``.
On error, set an exception, leave the writer unchanged, and return ``-1``.
.. versionadded:: 3.14
.. c:function:: int PyUnicodeWriter_WriteWideChar(PyUnicodeWriter *writer, const wchar_t *str, Py_ssize_t size)
Write the wide string *str* into *writer*.
@ -1867,7 +1865,7 @@ object.
On success, return ``0``.
On error, set an exception, leave the writer unchanged, and return ``-1``.
.. c:function:: int PyUnicodeWriter_WriteUCS4(PyUnicodeWriter *writer, Py_UCS4 *str, Py_ssize_t size)
.. c:function:: int PyUnicodeWriter_WriteUCS4(PyUnicodeWriter *writer, const Py_UCS4 *str, Py_ssize_t size)
Writer the UCS4 string *str* into *writer*.
@ -1883,13 +1881,23 @@ object.
On success, return ``0``.
On error, set an exception, leave the writer unchanged, and return ``-1``.
To write a :class:`str` subclass which overrides the :meth:`~object.__str__`
method, :c:func:`PyUnicode_FromObject` can be used to get the original
string.
.. c:function:: int PyUnicodeWriter_WriteRepr(PyUnicodeWriter *writer, PyObject *obj)
Call :c:func:`PyObject_Repr` on *obj* and write the output into *writer*.
If *obj* is ``NULL``, write the string ``"<NULL>"`` into *writer*.
On success, return ``0``.
On error, set an exception, leave the writer unchanged, and return ``-1``.
.. versionchanged:: 3.14.4
Added support for ``NULL``.
.. c:function:: int PyUnicodeWriter_WriteSubstring(PyUnicodeWriter *writer, PyObject *str, Py_ssize_t start, Py_ssize_t end)
Write the substring ``str[start:end]`` into *writer*.

45
Doc/conf.py
View file

@ -46,6 +46,7 @@
'linklint.ext',
'notfound.extension',
'sphinxext.opengraph',
'sphinxcontrib.rsvgconverter',
)
for optional_ext in _OPTIONAL_EXTENSIONS:
try:
@ -72,6 +73,7 @@
# General substitutions.
project = 'Python'
copyright = "2001 Python Software Foundation"
_doc_authors = 'Python documentation authors'
# We look for the Include/patchlevel.h file in the current Python source tree
# and replace the values accordingly.
@ -176,6 +178,7 @@
('c:type', '__int64'),
('c:type', 'unsigned __int64'),
('c:type', 'double'),
('c:type', '_Float16'),
# Standard C structures
('c:struct', 'in6_addr'),
('c:struct', 'in_addr'),
@ -359,69 +362,74 @@
# Grouping the document tree into LaTeX files. List of tuples
# (source start file, target name, title, author, document class [howto/manual]).
_stdauthor = 'The Python development team'
latex_documents = [
('c-api/index', 'c-api.tex', 'The Python/C API', _stdauthor, 'manual'),
('c-api/index', 'c-api.tex', 'The Python/C API', _doc_authors, 'manual'),
(
'extending/index',
'extending.tex',
'Extending and Embedding Python',
_stdauthor,
_doc_authors,
'manual',
),
(
'installing/index',
'installing.tex',
'Installing Python Modules',
_stdauthor,
_doc_authors,
'manual',
),
(
'library/index',
'library.tex',
'The Python Library Reference',
_stdauthor,
_doc_authors,
'manual',
),
(
'reference/index',
'reference.tex',
'The Python Language Reference',
_stdauthor,
_doc_authors,
'manual',
),
(
'tutorial/index',
'tutorial.tex',
'Python Tutorial',
_stdauthor,
_doc_authors,
'manual',
),
(
'using/index',
'using.tex',
'Python Setup and Usage',
_stdauthor,
_doc_authors,
'manual',
),
(
'faq/index',
'faq.tex',
'Python Frequently Asked Questions',
_stdauthor,
_doc_authors,
'manual',
),
(
'whatsnew/' + version,
'whatsnew.tex',
'What\'s New in Python',
'A. M. Kuchling',
_doc_authors,
'howto',
),
]
# Collect all HOWTOs individually
latex_documents.extend(
('howto/' + fn[:-4], 'howto-' + fn[:-4] + '.tex', '', _stdauthor, 'howto')
(
'howto/' + fn[:-4],
'howto-' + fn[:-4] + '.tex',
'',
_doc_authors,
'howto',
)
for fn in os.listdir('howto')
if fn.endswith('.rst') and fn != 'index.rst'
)
@ -432,7 +440,7 @@
# Options for Epub output
# -----------------------
epub_author = 'Python Documentation Authors'
epub_author = _doc_authors
epub_publisher = 'Python Software Foundation'
epub_exclude_files = (
'index.xhtml',
@ -556,6 +564,7 @@
# mapping unique short aliases to a base URL and a prefix.
# https://www.sphinx-doc.org/en/master/usage/extensions/extlinks.html
extlinks = {
"oss-fuzz": ("https://issues.oss-fuzz.com/issues/%s", "#%s"),
"pypi": ("https://pypi.org/project/%s/", "%s"),
"source": (SOURCE_URI, "%s"),
}
@ -567,6 +576,18 @@
# Relative filename of the data files
refcount_file = 'data/refcounts.dat'
stable_abi_file = 'data/stable_abi.dat'
threadsafety_file = 'data/threadsafety.dat'
# Options for notfound.extension
# -------------------------------
if not os.getenv("READTHEDOCS"):
if language_code:
notfound_urls_prefix = (
f'/{language_code.replace("_", "-").lower()}/{version}/'
)
else:
notfound_urls_prefix = f'/{version}/'
# Options for sphinxext-opengraph
# -------------------------------

14
Doc/data/refcounts.dat
View file

@ -2037,6 +2037,10 @@ PySeqIter_Check:PyObject *:op:0:
PySeqIter_New:PyObject*::+1:
PySeqIter_New:PyObject*:seq:0:
PySentinel_New:PyObject*::+1:
PySentinel_New:const char*:name::
PySentinel_New:const char*:module_name::
PySequence_Check:int:::
PySequence_Check:PyObject*:o:0:
@ -2427,10 +2431,20 @@ PyType_GetFlags:PyTypeObject*:type:0:
PyType_GetName:PyObject*::+1:
PyType_GetName:PyTypeObject*:type:0:
PyType_GetModule:PyObject*::0:
PyType_GetModule:PyTypeObject*:type:0:
PyType_GetModule_DuringGC:PyObject*::0:
PyType_GetModule_DuringGC:PyTypeObject*:type:0:
PyType_GetModuleByToken:PyObject*::+1:
PyType_GetModuleByToken:PyTypeObject*:type:0:
PyType_GetModuleByToken:PyModuleDef*:def::
PyType_GetModuleByToken_DuringGC:PyObject*::0:
PyType_GetModuleByToken_DuringGC:PyTypeObject*:type:0:
PyType_GetModuleByToken_DuringGC:PyModuleDef*:mod_token::
PyType_GetModuleByDef:PyObject*::0:
PyType_GetModuleByDef:PyTypeObject*:type:0:
PyType_GetModuleByDef:PyModuleDef*:def::

11
Doc/data/stable_abi.dat generated
View file

@ -470,8 +470,8 @@ func,PyMemoryView_GetContiguous,3.2,,
data,PyMemoryView_Type,3.2,,
type,PyMethodDef,3.2,,full-abi
data,PyMethodDescr_Type,3.2,,
type,PyModuleDef,3.2,,full-abi
type,PyModuleDef_Base,3.2,,full-abi
type,PyModuleDef,3.2,,abi3t-opaque
type,PyModuleDef_Base,3.2,,abi3t-opaque
func,PyModuleDef_Init,3.5,,
type,PyModuleDef_Slot,3.5,,full-abi
data,PyModuleDef_Type,3.5,,
@ -495,7 +495,9 @@ func,PyModule_GetName,3.2,,
func,PyModule_GetNameObject,3.7,,
func,PyModule_GetState,3.2,,
func,PyModule_GetStateSize,3.15,,
func,PyModule_GetState_DuringGC,3.15,,
func,PyModule_GetToken,3.15,,
func,PyModule_GetToken_DuringGC,3.15,,
func,PyModule_New,3.2,,
func,PyModule_NewObject,3.7,,
func,PyModule_SetDocString,3.7,,
@ -598,6 +600,7 @@ func,PyObject_GetIter,3.2,,
func,PyObject_GetOptionalAttr,3.13,,
func,PyObject_GetOptionalAttrString,3.13,,
func,PyObject_GetTypeData,3.12,,
func,PyObject_GetTypeData_DuringGC,3.15,,
func,PyObject_HasAttr,3.2,,
func,PyObject_HasAttrString,3.2,,
func,PyObject_HasAttrStringWithError,3.13,,
@ -750,13 +753,17 @@ func,PyType_FromSpecWithBases,3.3,,
func,PyType_GenericAlloc,3.2,,
func,PyType_GenericNew,3.2,,
func,PyType_GetBaseByToken,3.14,,
func,PyType_GetBaseByToken_DuringGC,3.15,,
func,PyType_GetFlags,3.2,,
func,PyType_GetFullyQualifiedName,3.13,,
func,PyType_GetModule,3.10,,
func,PyType_GetModuleByDef,3.13,,
func,PyType_GetModuleByToken,3.15,,
func,PyType_GetModuleByToken_DuringGC,3.15,,
func,PyType_GetModuleName,3.13,,
func,PyType_GetModuleState,3.10,,
func,PyType_GetModuleState_DuringGC,3.15,,
func,PyType_GetModule_DuringGC,3.15,,
func,PyType_GetName,3.11,,
func,PyType_GetQualName,3.11,,
func,PyType_GetSlot,3.4,,

284
Doc/data/threadsafety.dat Normal file
View file

@ -0,0 +1,284 @@
# Thread safety annotations for C API functions.
#
# Each line has the form:
# function_name : level
#
# Where level is one of:
# incompatible -- not safe even with external locking
# compatible -- safe if the caller serializes all access with external locks
# distinct -- safe on distinct objects without external synchronization
# shared -- safe for concurrent use on the same object
# atomic -- atomic
#
# Lines beginning with '#' are ignored.
# The function name must match the C domain identifier used in the documentation.
# Synchronization primitives (Doc/c-api/synchronization.rst)
PyMutex_Lock:atomic:
PyMutex_Unlock:atomic:
PyMutex_IsLocked:atomic:
# Dictionary objects (Doc/c-api/dict.rst)
# Type checks - read ob_type pointer, always safe
PyDict_Check:atomic:
PyDict_CheckExact:atomic:
# Creation - pure allocation, no shared state
PyDict_New:atomic:
# Lock-free lookups - use _Py_dict_lookup_threadsafe(), no locking.
# Atomic with simple types.
PyDict_Contains:shared:
PyDict_ContainsString:atomic:
PyDict_GetItemRef:shared:
PyDict_GetItemStringRef:atomic:
PyDict_Size:atomic:
PyDict_GET_SIZE:atomic:
# Borrowed-reference lookups - lock-free dict access but returned
# borrowed reference is unsafe in free-threaded builds without
# external synchronization
PyDict_GetItem:compatible:
PyDict_GetItemWithError:compatible:
PyDict_GetItemString:compatible:
PyDict_SetDefault:compatible:
# Iteration - no locking; returns borrowed refs
PyDict_Next:compatible:
# Single-item mutations - protected by per-object critical section
PyDict_SetItem:shared:
PyDict_SetItemString:atomic:
PyDict_DelItem:shared:
PyDict_DelItemString:atomic:
PyDict_SetDefaultRef:shared:
PyDict_Pop:shared:
PyDict_PopString:atomic:
# Bulk reads - hold per-object lock for duration
PyDict_Clear:atomic:
PyDict_Copy:atomic:
PyDict_Keys:atomic:
PyDict_Values:atomic:
PyDict_Items:atomic:
# Merge/update - lock target dict; also lock source when it is a dict
PyDict_Update:shared:
PyDict_Merge:shared:
PyDict_MergeFromSeq2:shared:
# Watcher registration - no synchronization on interpreter state
PyDict_AddWatcher:compatible:
PyDict_ClearWatcher:compatible:
# Per-dict watcher tags - non-atomic RMW on _ma_watcher_tag;
# safe on distinct dicts only
PyDict_Watch:distinct:
PyDict_Unwatch:distinct:
# List objects (Doc/c-api/list.rst)
# Type checks - read ob_type pointer, always safe
PyList_Check:atomic:
PyList_CheckExact:atomic:
# Creation - pure allocation, no shared state
PyList_New:atomic:
# Size - uses atomic load on free-threaded builds
PyList_Size:atomic:
PyList_GET_SIZE:atomic:
# Strong-reference lookup - lock-free with atomic ops
PyList_GetItemRef:atomic:
# Borrowed-reference lookups - no locking; returned borrowed
# reference is unsafe in free-threaded builds without
# external synchronization
PyList_GetItem:compatible:
PyList_GET_ITEM:compatible:
# Single-item mutations - hold per-object lock for duration;
# appear atomic to lock-free readers
PyList_SetItem:atomic:
PyList_Append:atomic:
# Insert - protected by per-object critical section; shifts
# elements so lock-free readers may observe intermediate states
PyList_Insert:shared:
# Initialization macro - no synchronization; normally only used
# to fill in new lists where there is no previous content
PyList_SET_ITEM:compatible:
# Bulk operations - hold per-object lock for duration
PyList_GetSlice:atomic:
PyList_AsTuple:atomic:
PyList_Clear:atomic:
# Reverse - protected by per-object critical section; swaps
# elements so lock-free readers may observe intermediate states
PyList_Reverse:shared:
# Slice assignment - lock target list; also lock source when it
# is a list
PyList_SetSlice:shared:
# Sort - per-object lock held; the list is emptied before sorting
# so other threads may observe an empty list, but they won't see the
# intermediate states of the sort
PyList_Sort:shared:
# Extend - lock target list; also lock source when it is a
# list, set, or dict
PyList_Extend:shared:
# Creation - pure allocation, no shared state
PyBytes_FromString:atomic:
PyBytes_FromStringAndSize:atomic:
PyBytes_DecodeEscape:atomic:
# Creation from formatting C primitives - pure allocation, no shared state
PyBytes_FromFormat:atomic:
PyBytes_FromFormatV:atomic:
# Creation from object - uses buffer protocol so may call arbitrary code;
# safe as long as the buffer is not mutated by another thread during the operation
PyBytes_FromObject:shared:
# Size - uses atomic load on free-threaded builds
PyBytes_Size:atomic:
PyBytes_GET_SIZE:atomic:
# Raw data - no locking; mutating it is unsafe if the bytes object is shared between threads
PyBytes_AsString:compatible:
PyBytes_AS_STRING:compatible:
PyBytes_AsStringAndSize:compatible:
# Concatenation - uses buffer protocol; safe as long as buffer is not mutated by another thread during the operation
PyBytes_Concat:shared:
PyBytes_ConcatAndDel:shared:
PyBytes_Join:shared:
# Resizing - safe if the object is unique
_PyBytes_Resize:distinct:
# Repr - atomic as bytes are immutable
PyBytes_Repr:atomic:
# Creation from object - may call arbitrary code
PyByteArray_FromObject:shared:
# Creation - pure allocation, no shared state
PyByteArray_FromStringAndSize:atomic:
# Concatenation - uses buffer protocol; safe as long as buffer is not mutated by another thread during the operation
PyByteArray_Concat:shared:
# Size - uses atomic load on free-threaded builds
PyByteArray_Size:atomic:
PyByteArray_GET_SIZE:atomic:
# Raw data - no locking; mutating it is unsafe if the bytearray object is shared between threads
PyByteArray_AsString:compatible:
PyByteArray_AS_STRING:compatible:
# Creation - may iterate the iterable argument, calling arbitrary code.
# Atomic for sets, frozensets, dicts, and frozendicts.
PySet_New:shared:
PyFrozenSet_New:shared:
# Size - uses atomic load on free-threaded builds
PySet_Size:atomic:
PySet_GET_SIZE:atomic:
# Contains - lock-free, atomic with simple types
PySet_Contains:shared:
# Mutations - hold per-object lock for duration
# atomic with simple types
PySet_Add:shared:
PySet_Discard:shared:
# Pop - hold per-object lock for duration
PySet_Pop:atomic:
# Clear - empties the set before clearing
PySet_Clear:atomic:
# Capsule objects (Doc/c-api/capsule.rst)
# Type check - read ob_type pointer, always safe
PyCapsule_CheckExact:atomic:
# Creation - pure allocation, no shared state
PyCapsule_New:atomic:
# Validation - reads pointer and name fields; safe on distinct objects
PyCapsule_IsValid:distinct:
# Getters - read struct fields; safe on distinct objects but
# concurrent access to the same capsule requires external synchronization
PyCapsule_GetPointer:distinct:
PyCapsule_GetName:distinct:
PyCapsule_GetDestructor:distinct:
PyCapsule_GetContext:distinct:
# Setters - write struct fields; safe on distinct objects but
# concurrent access to the same capsule requires external synchronization
PyCapsule_SetPointer:distinct:
PyCapsule_SetName:distinct:
PyCapsule_SetDestructor:distinct:
PyCapsule_SetContext:distinct:
# Import - looks up a capsule from a module attribute and
# calls PyCapsule_GetPointer; may call arbitrary code
PyCapsule_Import:compatible:
# Tuple objects
# Creation - pure allocation, no shared state
PyTuple_New:atomic:
PyTuple_FromArray:atomic:
PyTuple_Pack:atomic:
# Size - tuples are immutable so size never changes
PyTuple_Size:atomic:
PyTuple_GET_SIZE:atomic:
# Borrowed-reference lookups - tuples are immutable so items
# never change, however the tuple must be kept alive while using the borrowed reference
PyTuple_GetItem:compatible:
PyTuple_GET_ITEM:compatible:
# Slice - creates a new tuple from an existing tuple
PyTuple_GetSlice:atomic:
# SetItem - only usable on tuples with refcount 1
PyTuple_SetItem:compatible:
PyTuple_SET_ITEM:compatible:
# Resize - only usable on tuples with refcount 1
_PyTuple_Resize:compatible:
# Struct Sequence objects
# Creation
PyStructSequence_NewType:atomic:
PyStructSequence_New:atomic:
# Initialization - modifies the type object in place
PyStructSequence_InitType:distinct:
PyStructSequence_InitType2:distinct:
# Borrowed-reference lookups - same as tuple items
PyStructSequence_GetItem:compatible:
PyStructSequence_GET_ITEM:compatible:
# SetItem - only for filling in brand new instances
PyStructSequence_SetItem:compatible:
PyStructSequence_SET_ITEM:compatible:

2
Doc/deprecations/c-api-pending-removal-in-3.15.rst
View file

@ -7,8 +7,6 @@ Pending removal in Python 3.15
Use :c:func:`PyWeakref_GetRef` instead. The `pythoncapi-compat project
<https://github.com/python/pythoncapi-compat/>`__ can be used to get
:c:func:`PyWeakref_GetRef` on Python 3.12 and older.
* :c:type:`Py_UNICODE` type and the :c:macro:`!Py_UNICODE_WIDE` macro:
Use :c:type:`wchar_t` instead.
* :c:func:`!PyUnicode_AsDecodedObject`:
Use :c:func:`PyCodec_Decode` instead.
* :c:func:`!PyUnicode_AsDecodedUnicode`:

2
Doc/deprecations/index.rst
View file

@ -15,6 +15,8 @@ Deprecations
.. include:: pending-removal-in-future.rst
.. include:: soft-deprecations.rst
C API deprecations
------------------

2
Doc/deprecations/pending-removal-in-3.15.rst
View file

@ -60,7 +60,7 @@ Pending removal in Python 3.15
* :mod:`types`:
* :class:`types.CodeType`: Accessing :attr:`~codeobject.co_lnotab` was
* :class:`types.CodeType`: Accessing :attr:`!codeobject.co_lnotab` was
deprecated in :pep:`626`
since 3.10 and was planned to be removed in 3.12,
but it only got a proper :exc:`DeprecationWarning` in 3.12.

10
Doc/deprecations/pending-removal-in-3.17.rst
View file

@ -1,6 +1,14 @@
Pending removal in Python 3.17
------------------------------
* :mod:`datetime`:
* :meth:`~datetime.datetime.strptime` calls using a format string containing
``%e`` (day of month) without a year.
This has been deprecated since Python 3.15.
(Contributed by Stan Ulbrych in :gh:`70647`.)
* :mod:`collections.abc`:
- :class:`collections.abc.ByteString` is scheduled for removal in Python 3.17.
@ -27,7 +35,7 @@ Pending removal in Python 3.17
- Passing non-ascii *encoding* names to :func:`encodings.normalize_encoding`
is deprecated and scheduled for removal in Python 3.17.
(Contributed by Stan Ulbrych in :gh:`136702`)
(Contributed by Stan Ulbrych in :gh:`136702`.)
* :mod:`typing`:

9
Doc/deprecations/pending-removal-in-3.18.rst
View file

@ -1,9 +1,18 @@
Pending removal in Python 3.18
------------------------------
* No longer accept a boolean value when a file descriptor is expected.
(Contributed by Serhiy Storchaka in :gh:`82626`.)
* :mod:`decimal`:
* The non-standard and undocumented :class:`~decimal.Decimal` format
specifier ``'N'``, which is only supported in the :mod:`!decimal` module's
C implementation, has been deprecated since Python 3.13.
(Contributed by Serhiy Storchaka in :gh:`89902`.)
* Deprecations defined by :pep:`829`:
* ``import`` lines in :file:`{name}.pth` files are silently ignored.
(Contributed by Barry Warsaw in :gh:`148641`.)

22
Doc/deprecations/pending-removal-in-3.20.rst
View file

@ -1,6 +1,13 @@
Pending removal in Python 3.20
------------------------------
* Calling the ``__new__()`` method of :class:`struct.Struct` without the
*format* argument is deprecated and will be removed in Python 3.20. Calling
:meth:`~object.__init__` method on initialized :class:`~struct.Struct`
objects is deprecated and will be removed in Python 3.20.
(Contributed by Sergey B Kirpichev and Serhiy Storchaka in :gh:`143715`.)
* The ``__version__``, ``version`` and ``VERSION`` attributes have been
deprecated in these standard library modules and will be removed in
Python 3.20. Use :py:data:`sys.version_info` instead.
@ -31,3 +38,18 @@ Pending removal in Python 3.20
- :mod:`zlib`
(Contributed by Hugo van Kemenade and Stan Ulbrych in :gh:`76007`.)
* Deprecations defined by :pep:`829`:
* Warnings are produced for ``import`` lines found in :file:`{name}.pth`
files.
* :file:`{name}.pth` files are no longer decoded in the locale encoding by
default. They **MUST** be encoded in ``utf-8-sig``.
(Contributed by Barry Warsaw in :gh:`148641`.)
* :mod:`ast`:
* Creating instances of abstract AST nodes (such as :class:`ast.AST`
or :class:`!ast.expr`) is deprecated and will raise an error in Python 3.20.

2
Doc/deprecations/pending-removal-in-future.rst
View file

@ -47,7 +47,7 @@ although there is currently no date scheduled for their removal.
* :mod:`codecs`: use :func:`open` instead of :func:`codecs.open`. (:gh:`133038`)
* :attr:`codeobject.co_lnotab`: use the :meth:`codeobject.co_lines` method
* :attr:`!codeobject.co_lnotab`: use the :meth:`codeobject.co_lines` method
instead.
* :mod:`datetime`:

21
Doc/deprecations/soft-deprecations.rst Normal file
View file

@ -0,0 +1,21 @@
Soft deprecations
-----------------
There are no plans to remove :term:`soft deprecated` APIs.
* :func:`re.match` and :meth:`re.Pattern.match` are now
:term:`soft deprecated` in favor of the new :func:`re.prefixmatch` and
:meth:`re.Pattern.prefixmatch` APIs, which have been added as alternate,
more explicit names. These are intended to be used to alleviate confusion
around what *match* means by following the Zen of Python's *"Explicit is
better than implicit"* mantra. Most other language regular expression
libraries use an API named *match* to mean what Python has always called
*search*.
We **do not** plan to remove the older :func:`!match` name, as it has been
used in code for over 30 years. Code supporting older versions of Python
should continue to use :func:`!match`, while new code should prefer
:func:`!prefixmatch`. See :ref:`prefixmatch-vs-match`.
(Contributed by Gregory P. Smith in :gh:`86519` and
Hugo van Kemenade in :gh:`148100`.)

7
Doc/extending/first-extension-module.rst
View file

@ -265,12 +265,19 @@ Define this array just before your export hook:
.. code-block:: c
PyABIInfo_VAR(abi_info);
static PyModuleDef_Slot spam_slots[] = {
{Py_mod_abi, &abi_info},
{Py_mod_name, "spam"},
{Py_mod_doc, "A wonderful module with an example function"},
{0, NULL}
};
The ``PyABIInfo_VAR(abi_info);`` macro and the :c:data:`Py_mod_abi` slot
are a bit of boilerplate that helps prevent extensions compiled for
a different version of Python from crashing the interpreter.
For both :c:data:`Py_mod_name` and :c:data:`Py_mod_doc`, the values are C
strings -- that is, NUL-terminated, UTF-8 encoded byte arrays.

132
Doc/faq/programming.rst
View file

@ -8,11 +8,11 @@ Programming FAQ
.. contents::
General Questions
General questions
=================
Is there a source code level debugger with breakpoints, single-stepping, etc.?
------------------------------------------------------------------------------
Is there a source code-level debugger with breakpoints and single-stepping?
---------------------------------------------------------------------------
Yes.
@ -25,8 +25,7 @@ Reference Manual <pdb>`. You can also write your own debugger by using the code
for pdb as an example.
The IDLE interactive development environment, which is part of the standard
Python distribution (normally available as
`Tools/scripts/idle3 <https://github.com/python/cpython/blob/main/Tools/scripts/idle3>`_),
Python distribution (normally available as :mod:`idlelib`),
includes a graphical debugger.
PythonWin is a Python IDE that includes a GUI debugger based on pdb. The
@ -48,7 +47,6 @@ There are a number of commercial Python IDEs that include graphical debuggers.
They include:
* `Wing IDE <https://wingware.com/>`_
* `Komodo IDE <https://www.activestate.com/products/komodo-ide/>`_
* `PyCharm <https://www.jetbrains.com/pycharm/>`_
@ -57,13 +55,15 @@ Are there tools to help find bugs or perform static analysis?
Yes.
`Pylint <https://pylint.pycqa.org/en/latest/index.html>`_ and
`Pyflakes <https://github.com/PyCQA/pyflakes>`_ do basic checking that will
`Ruff <https://docs.astral.sh/ruff/>`__,
`Pylint <https://pylint.readthedocs.io/>`__ and
`Pyflakes <https://github.com/PyCQA/pyflakes>`__ do basic checking that will
help you catch bugs sooner.
Static type checkers such as `Mypy <https://mypy-lang.org/>`_,
`Pyre <https://pyre-check.org/>`_, and
`Pytype <https://github.com/google/pytype>`_ can check type hints in Python
Static type checkers such as `mypy <https://mypy-lang.org/>`__,
`ty <https://docs.astral.sh/ty/>`__,
`Pyrefly <https://pyrefly.org/>`__, and
`pytype <https://github.com/google/pytype>`__ can check type hints in Python
source code.
@ -79,7 +79,7 @@ set of modules required by a program and bind these modules together with a
Python binary to produce a single executable.
One is to use the freeze tool, which is included in the Python source tree as
`Tools/freeze <https://github.com/python/cpython/tree/main/Tools/freeze>`_.
:source:`Tools/freeze`.
It converts Python byte code to C arrays; with a C compiler you can
embed all your modules into a new program, which is then linked with the
standard Python modules.
@ -103,6 +103,7 @@ executables:
* `py2app <https://github.com/ronaldoussoren/py2app>`_ (macOS only)
* `py2exe <https://www.py2exe.org/>`_ (Windows only)
Are there coding standards or a style guide for Python programs?
----------------------------------------------------------------
@ -110,7 +111,7 @@ Yes. The coding style required for standard library modules is documented as
:pep:`8`.
Core Language
Core language
=============
.. _faq-unboundlocalerror:
@ -143,7 +144,7 @@ results in an :exc:`!UnboundLocalError`:
>>> foo()
Traceback (most recent call last):
...
UnboundLocalError: local variable 'x' referenced before assignment
UnboundLocalError: cannot access local variable 'x' where it is not associated with a value
This is because when you make an assignment to a variable in a scope, that
variable becomes local to that scope and shadows any similarly named variable
@ -208,7 +209,7 @@ Why do lambdas defined in a loop with different values all return the same resul
----------------------------------------------------------------------------------
Assume you use a for loop to define a few different lambdas (or even plain
functions), e.g.::
functions), for example::
>>> squares = []
>>> for x in range(5):
@ -227,7 +228,7 @@ they all return ``16``::
This happens because ``x`` is not local to the lambdas, but is defined in
the outer scope, and it is accessed when the lambda is called --- not when it
is defined. At the end of the loop, the value of ``x`` is ``4``, so all the
functions now return ``4**2``, i.e. ``16``. You can also verify this by
functions now return ``4**2``, that is ``16``. You can also verify this by
changing the value of ``x`` and see how the results of the lambdas change::
>>> x = 8
@ -298,9 +299,9 @@ using multiple imports per line uses less screen space.
It's good practice if you import modules in the following order:
1. standard library modules -- e.g. :mod:`sys`, :mod:`os`, :mod:`argparse`, :mod:`re`
1. standard library modules -- such as :mod:`sys`, :mod:`os`, :mod:`argparse`, :mod:`re`
2. third-party library modules (anything installed in Python's site-packages
directory) -- e.g. :mod:`!dateutil`, :mod:`!requests`, :mod:`!PIL.Image`
directory) -- such as :pypi:`dateutil`, :pypi:`requests`, :pypi:`tzdata`
3. locally developed modules
It is sometimes necessary to move imports to a function or class to avoid
@ -494,11 +495,11 @@ new objects).
In other words:
* If we have a mutable object (:class:`list`, :class:`dict`, :class:`set`,
etc.), we can use some specific operations to mutate it and all the variables
* If we have a mutable object (such as :class:`list`, :class:`dict`, :class:`set`),
we can use some specific operations to mutate it and all the variables
that refer to it will see the change.
* If we have an immutable object (:class:`str`, :class:`int`, :class:`tuple`,
etc.), all the variables that refer to it will always see the same value,
* If we have an immutable object (such as :class:`str`, :class:`int`, :class:`tuple`),
all the variables that refer to it will always see the same value,
but operations that transform that value into a new value always return a new
object.
@ -511,7 +512,7 @@ How do I write a function with output parameters (call by reference)?
Remember that arguments are passed by assignment in Python. Since assignment
just creates references to objects, there's no alias between an argument name in
the caller and callee, and so no call-by-reference per se. You can achieve the
the caller and callee, and consequently no call-by-reference. You can achieve the
desired effect in a number of ways.
1) By returning a tuple of the results::
@ -714,8 +715,8 @@ not::
"a" in ("b", "a")
The same is true of the various assignment operators (``=``, ``+=`` etc). They
are not truly operators but syntactic delimiters in assignment statements.
The same is true of the various assignment operators (``=``, ``+=``, and so on).
They are not truly operators but syntactic delimiters in assignment statements.
Is there an equivalent of C's "?:" ternary operator?
@ -868,9 +869,9 @@ with either a space or parentheses.
How do I convert a string to a number?
--------------------------------------
For integers, use the built-in :func:`int` type constructor, e.g. ``int('144')
For integers, use the built-in :func:`int` type constructor, for example, ``int('144')
== 144``. Similarly, :func:`float` converts to a floating-point number,
e.g. ``float('144') == 144.0``.
for example, ``float('144') == 144.0``.
By default, these interpret the number as decimal, so that ``int('0144') ==
144`` holds true, and ``int('0x144')`` raises :exc:`ValueError`. ``int(string,
@ -887,18 +888,18 @@ unwanted side effects. For example, someone could pass
directory.
:func:`eval` also has the effect of interpreting numbers as Python expressions,
so that e.g. ``eval('09')`` gives a syntax error because Python does not allow
so that, for example, ``eval('09')`` gives a syntax error because Python does not allow
leading '0' in a decimal number (except '0').
How do I convert a number to a string?
--------------------------------------
To convert, e.g., the number ``144`` to the string ``'144'``, use the built-in type
For example, to convert the number ``144`` to the string ``'144'``, use the built-in type
constructor :func:`str`. If you want a hexadecimal or octal representation, use
the built-in functions :func:`hex` or :func:`oct`. For fancy formatting, see
the :ref:`f-strings` and :ref:`formatstrings` sections,
e.g. ``"{:04d}".format(144)`` yields
the :ref:`f-strings` and :ref:`formatstrings` sections.
For example, ``"{:04d}".format(144)`` yields
``'0144'`` and ``"{:.3f}".format(1.0/3.0)`` yields ``'0.333'``.
@ -908,7 +909,7 @@ How do I modify a string in place?
You can't, because strings are immutable. In most situations, you should
simply construct a new string from the various parts you want to assemble
it from. However, if you need an object with the ability to modify in-place
unicode data, try using an :class:`io.StringIO` object or the :mod:`array`
Unicode data, try using an :class:`io.StringIO` object or the :mod:`array`
module::
>>> import io
@ -1066,13 +1067,14 @@ the raw string::
Also see the specification in the :ref:`language reference <strings>`.
Performance
===========
My program is too slow. How do I speed it up?
---------------------------------------------
That's a tough one, in general. First, here are a list of things to
That's a tough one, in general. First, here is a list of things to
remember before diving further:
* Performance characteristics vary across Python implementations. This FAQ
@ -1125,6 +1127,7 @@ yourself.
The wiki page devoted to `performance tips
<https://wiki.python.org/moin/PythonSpeed/PerformanceTips>`_.
.. _efficient_string_concatenation:
What is the most efficient way to concatenate many strings together?
@ -1143,7 +1146,7 @@ them into a list and call :meth:`str.join` at the end::
chunks.append(s)
result = ''.join(chunks)
(another reasonably efficient idiom is to use :class:`io.StringIO`)
(Another reasonably efficient idiom is to use :class:`io.StringIO`.)
To accumulate many :class:`bytes` objects, the recommended idiom is to extend
a :class:`bytearray` object using in-place concatenation (the ``+=`` operator)::
@ -1153,7 +1156,7 @@ a :class:`bytearray` object using in-place concatenation (the ``+=`` operator)::
result += b
Sequences (Tuples/Lists)
Sequences (tuples/lists)
========================
How do I convert between tuples and lists?
@ -1217,8 +1220,8 @@ list, deleting duplicates as you go::
else:
last = mylist[i]
If all elements of the list may be used as set keys (i.e. they are all
:term:`hashable`) this is often faster ::
If all elements of the list may be used as set keys (that is, they are all
:term:`hashable`) this is often faster::
mylist = list(set(mylist))
@ -1254,7 +1257,7 @@ difference is that a Python list can contain objects of many different types.
The ``array`` module also provides methods for creating arrays of fixed types
with compact representations, but they are slower to index than lists. Also
note that `NumPy <https://numpy.org/>`_
and other third party packages define array-like structures with
and other third-party packages define array-like structures with
various characteristics as well.
To get Lisp-style linked lists, you can emulate *cons cells* using tuples::
@ -1324,7 +1327,7 @@ Or, you can use an extension that provides a matrix datatype; `NumPy
How do I apply a method or function to a sequence of objects?
-------------------------------------------------------------
To call a method or function and accumulate the return values is a list,
To call a method or function and accumulate the return values in a list,
a :term:`list comprehension` is an elegant solution::
result = [obj.method() for obj in mylist]
@ -1340,6 +1343,7 @@ a plain :keyword:`for` loop will suffice::
for obj in mylist:
function(obj)
.. _faq-augmented-assignment-tuple-error:
Why does a_tuple[i] += ['item'] raise an exception when the addition works?
@ -1444,7 +1448,7 @@ How can I sort one list by values from another list?
----------------------------------------------------
Merge them into an iterator of tuples, sort the resulting list, and then pick
out the element you want. ::
out the element you want.
>>> list1 = ["what", "I'm", "sorting", "by"]
>>> list2 = ["something", "else", "to", "sort"]
@ -1504,14 +1508,15 @@ How do I check if an object is an instance of a given class or of a subclass of
Use the built-in function :func:`isinstance(obj, cls) <isinstance>`. You can
check if an object
is an instance of any of a number of classes by providing a tuple instead of a
single class, e.g. ``isinstance(obj, (class1, class2, ...))``, and can also
check whether an object is one of Python's built-in types, e.g.
single class, for example, ``isinstance(obj, (class1, class2, ...))``, and can also
check whether an object is one of Python's built-in types, for example,
``isinstance(obj, str)`` or ``isinstance(obj, (int, float, complex))``.
Note that :func:`isinstance` also checks for virtual inheritance from an
:term:`abstract base class`. So, the test will return ``True`` for a
registered class even if hasn't directly or indirectly inherited from it. To
test for "true inheritance", scan the :term:`MRO` of the class:
test for "true inheritance", scan the :term:`method resolution order` (MRO) of
the class:
.. testcode::
@ -1574,7 +1579,7 @@ call it::
What is delegation?
-------------------
Delegation is an object oriented technique (also called a design pattern).
Delegation is an object-oriented technique (also called a design pattern).
Let's say you have an object ``x`` and want to change the behaviour of just one
of its methods. You can create a new class that provides a new implementation
of the method you're interested in changing and delegates all other methods to
@ -1645,7 +1650,7 @@ How can I organize my code to make it easier to change the base class?
You could assign the base class to an alias and derive from the alias. Then all
you have to change is the value assigned to the alias. Incidentally, this trick
is also handy if you want to decide dynamically (e.g. depending on availability
is also handy if you want to decide dynamically (such as depending on availability
of resources) which base class to use. Example::
class Base:
@ -1710,9 +1715,9 @@ How can I overload constructors (or methods) in Python?
This answer actually applies to all methods, but the question usually comes up
first in the context of constructors.
In C++ you'd write
In C++ you'd write:
.. code-block:: c
.. code-block:: c++
class C {
C() { cout << "No arguments\n"; }
@ -1731,7 +1736,7 @@ default arguments. For example::
This is not entirely equivalent, but close enough in practice.
You could also try a variable-length argument list, e.g. ::
You could also try a variable-length argument list, for example::
def __init__(self, *args):
...
@ -1774,6 +1779,7 @@ to use private variable names at all.
The :ref:`private name mangling specifications <private-name-mangling>`
for details and special cases.
My class defines __del__ but it is not called when I delete the object.
-----------------------------------------------------------------------
@ -1783,7 +1789,7 @@ The :keyword:`del` statement does not necessarily call :meth:`~object.__del__` -
decrements the object's reference count, and if this reaches zero
:meth:`!__del__` is called.
If your data structures contain circular links (e.g. a tree where each child has
If your data structures contain circular links (for example, a tree where each child has
a parent reference and each parent has a list of children) the reference counts
will never go back to zero. Once in a while Python runs an algorithm to detect
such cycles, but the garbage collector might run some time after the last
@ -1885,9 +1891,9 @@ are preferred. In particular, identity tests should not be used to check
constants such as :class:`int` and :class:`str` which aren't guaranteed to be
singletons::
>>> a = 1000
>>> b = 500
>>> c = b + 500
>>> a = 10_000_000
>>> b = 5_000_000
>>> c = b + 5_000_000
>>> a is c
False
@ -1918,7 +1924,7 @@ correctly using identity tests:
.. code-block:: python
_sentinel = object()
_sentinel = sentinel('_sentinel')
def pop(self, key, default=_sentinel):
if key in self:
@ -1956,9 +1962,9 @@ parent class:
.. testcode::
from datetime import date
import datetime as dt
class FirstOfMonthDate(date):
class FirstOfMonthDate(dt.date):
"Always choose the first day of the month"
def __new__(cls, year, month, day):
return super().__new__(cls, year, month, 1)
@ -2001,7 +2007,7 @@ The two principal tools for caching methods are
former stores results at the instance level and the latter at the class
level.
The *cached_property* approach only works with methods that do not take
The ``cached_property`` approach only works with methods that do not take
any arguments. It does not create a reference to the instance. The
cached method result will be kept only as long as the instance is alive.
@ -2010,7 +2016,7 @@ method result will be released right away. The disadvantage is that if
instances accumulate, so too will the accumulated method results. They
can grow without bound.
The *lru_cache* approach works with methods that have :term:`hashable`
The ``lru_cache`` approach works with methods that have :term:`hashable`
arguments. It creates a reference to the instance unless special
efforts are made to pass in weak references.
@ -2044,11 +2050,11 @@ This example shows the various techniques::
# Depends on the station_id, date, and units.
The above example assumes that the *station_id* never changes. If the
relevant instance attributes are mutable, the *cached_property* approach
relevant instance attributes are mutable, the ``cached_property`` approach
can't be made to work because it cannot detect changes to the
attributes.
To make the *lru_cache* approach work when the *station_id* is mutable,
To make the ``lru_cache`` approach work when the *station_id* is mutable,
the class needs to define the :meth:`~object.__eq__` and :meth:`~object.__hash__`
methods so that the cache can detect relevant attribute updates::
@ -2094,10 +2100,10 @@ one user but run as another, such as if you are testing with a web server.
Unless the :envvar:`PYTHONDONTWRITEBYTECODE` environment variable is set,
creation of a .pyc file is automatic if you're importing a module and Python
has the ability (permissions, free space, etc...) to create a ``__pycache__``
has the ability (permissions, free space, and so on) to create a ``__pycache__``
subdirectory and write the compiled module to that subdirectory.
Running Python on a top level script is not considered an import and no
Running Python on a top-level script is not considered an import and no
``.pyc`` will be created. For example, if you have a top-level module
``foo.py`` that imports another module ``xyz.py``, when you run ``foo`` (by
typing ``python foo.py`` as a shell command), a ``.pyc`` will be created for
@ -2116,7 +2122,7 @@ the ``compile()`` function in that module interactively::
This will write the ``.pyc`` to a ``__pycache__`` subdirectory in the same
location as ``foo.py`` (or you can override that with the optional parameter
``cfile``).
*cfile*).
You can also automatically compile all files in a directory or directories using
the :mod:`compileall` module. You can do it from the shell prompt by running
@ -2221,7 +2227,7 @@ changed module, do this::
importlib.reload(modname)
Warning: this technique is not 100% fool-proof. In particular, modules
containing statements like ::
containing statements like::
from modname import some_objects

9
Doc/glossary.rst
View file

@ -39,10 +39,11 @@ Glossary
ABCs with the :mod:`abc` module.
annotate function
A function that can be called to retrieve the :term:`annotations <annotation>`
of an object. This function is accessible as the :attr:`~object.__annotate__`
attribute of functions, classes, and modules. Annotate functions are a
subset of :term:`evaluate functions <evaluate function>`.
A callable that can be called to retrieve the :term:`annotations <annotation>` of
an object. Annotate functions are usually :term:`functions <function>`,
automatically generated as the :attr:`~object.__annotate__` attribute of functions,
classes, and modules. Annotate functions are a subset of
:term:`evaluate functions <evaluate function>`.
annotation
A label associated with a variable, a class

6
Doc/howto/descriptor.rst
View file

@ -594,7 +594,7 @@ a pure Python equivalent:
def object_getattribute(obj, name):
"Emulate PyObject_GenericGetAttr() in Objects/object.c"
null = object()
null = sentinel('null')
objtype = type(obj)
cls_var = find_name_in_mro(objtype, name, null)
descr_get = getattr(type(cls_var), '__get__', null)
@ -1635,12 +1635,12 @@ by member descriptors:
.. testcode::
null = object()
null = sentinel('null')
class Member:
def __init__(self, name, clsname, offset):
'Emulate PyMemberDef in Include/structmember.h'
'Emulate PyMemberDef in Include/descrobject.h'
# Also see descr_new() in Objects/descrobject.c
self.name = name
self.clsname = clsname

10
Doc/howto/enum.rst
View file

@ -105,8 +105,8 @@ The complete :class:`!Weekday` enum now looks like this::
Now we can find out what today is! Observe::
>>> from datetime import date
>>> Weekday.from_date(date.today()) # doctest: +SKIP
>>> import datetime as dt
>>> Weekday.from_date(dt.date.today()) # doctest: +SKIP
<Weekday.TUESDAY: 2>
Of course, if you're reading this on some other day, you'll see that day instead.
@ -371,7 +371,7 @@ Equality comparisons are defined though::
>>> Color.BLUE == Color.BLUE
True
Comparisons against non-enumeration values will always compare not equal
Equality comparisons against non-enumeration values will always return ``False``
(again, :class:`IntEnum` was explicitly designed to behave differently, see
below)::
@ -1480,8 +1480,8 @@ TimePeriod
An example to show the :attr:`~Enum._ignore_` attribute in use::
>>> from datetime import timedelta
>>> class Period(timedelta, Enum):
>>> import datetime as dt
>>> class Period(dt.timedelta, Enum):
... "different lengths of time"
... _ignore_ = 'Period i'
... Period = vars()

32
Doc/howto/free-threading-extensions.rst
View file

@ -384,6 +384,30 @@ Important Considerations
internal extension state, standard mutexes or other synchronization
primitives might be more appropriate.
.. _per-object-locks:
Per-Object Locks (``ob_mutex``)
...............................
In the free-threaded build, each Python object contains a :c:member:`~PyObject.ob_mutex`
field of type :c:type:`PyMutex`. This mutex is **reserved for use by the
critical section API** (:c:macro:`Py_BEGIN_CRITICAL_SECTION` /
:c:macro:`Py_END_CRITICAL_SECTION`).
.. warning::
Do **not** lock ``ob_mutex`` directly with ``PyMutex_Lock(&obj->ob_mutex)``.
Mixing direct ``PyMutex_Lock`` calls with the critical section API on the
same mutex can cause deadlocks.
Even if your own code never uses critical sections on a particular object type,
**CPython internals may use the critical section API on any Python object**.
If your extension type needs its own lock, add a separate :c:type:`PyMutex`
field (or another synchronization primitive) to your object struct.
:c:type:`PyMutex` is very lightweight, so there is negligible cost to having
an additional one.
Building Extensions for the Free-Threaded Build
===============================================
@ -392,11 +416,9 @@ C API extensions need to be built specifically for the free-threaded build.
The wheels, shared libraries, and binaries are indicated by a ``t`` suffix.
* `pypa/manylinux <https://github.com/pypa/manylinux>`_ supports the
free-threaded build, with the ``t`` suffix, such as ``python3.13t``.
* `pypa/cibuildwheel <https://github.com/pypa/cibuildwheel>`_ supports the
free-threaded build on Python 3.13 and 3.14. On Python 3.14, free-threaded
wheels will be built by default. On Python 3.13, you will need to set
`CIBW_ENABLE to cpython-freethreading <https://cibuildwheel.pypa.io/en/stable/options/#enable>`_.
free-threaded build, with the ``t`` suffix, such as ``python3.14t``.
* `pypa/cibuildwheel <https://github.com/pypa/cibuildwheel>`_ supports
building wheels for the free-threaded build of Python 3.14 and newer.
Limited C API and Stable ABI
............................

78
Doc/howto/instrumentation.rst
View file

@ -341,6 +341,84 @@ Available static markers
.. versionadded:: 3.8
C Entry Points
^^^^^^^^^^^^^^
To simplify triggering of DTrace markers, Python's C API comes with a number
of helper functions that mirror each static marker. On builds of Python without
DTrace enabled, these do nothing.
In general, it is not necessary to call these yourself, as Python will do
it for you.
.. list-table::
:widths: 50 25 25
:header-rows: 1
* * C API Function
* Static Marker
* Notes
* * .. c:function:: void PyDTrace_LINE(const char *arg0, const char *arg1, int arg2)
* :c:func:`!line`
*
* * .. c:function:: void PyDTrace_FUNCTION_ENTRY(const char *arg0, const char *arg1, int arg2)
* :c:func:`!function__entry`
*
* * .. c:function:: void PyDTrace_FUNCTION_RETURN(const char *arg0, const char *arg1, int arg2)
* :c:func:`!function__return`
*
* * .. c:function:: void PyDTrace_GC_START(int arg0)
* :c:func:`!gc__start`
*
* * .. c:function:: void PyDTrace_GC_DONE(Py_ssize_t arg0)
* :c:func:`!gc__done`
*
* * .. c:function:: void PyDTrace_INSTANCE_NEW_START(int arg0)
* :c:func:`!instance__new__start`
* Not used by Python
* * .. c:function:: void PyDTrace_INSTANCE_NEW_DONE(int arg0)
* :c:func:`!instance__new__done`
* Not used by Python
* * .. c:function:: void PyDTrace_INSTANCE_DELETE_START(int arg0)
* :c:func:`!instance__delete__start`
* Not used by Python
* * .. c:function:: void PyDTrace_INSTANCE_DELETE_DONE(int arg0)
* :c:func:`!instance__delete__done`
* Not used by Python
* * .. c:function:: void PyDTrace_IMPORT_FIND_LOAD_START(const char *arg0)
* :c:func:`!import__find__load__start`
*
* * .. c:function:: void PyDTrace_IMPORT_FIND_LOAD_DONE(const char *arg0, int arg1)
* :c:func:`!import__find__load__done`
*
* * .. c:function:: void PyDTrace_AUDIT(const char *arg0, void *arg1)
* :c:func:`!audit`
*
C Probing Checks
^^^^^^^^^^^^^^^^
.. c:function:: int PyDTrace_LINE_ENABLED(void)
.. c:function:: int PyDTrace_FUNCTION_ENTRY_ENABLED(void)
.. c:function:: int PyDTrace_FUNCTION_RETURN_ENABLED(void)
.. c:function:: int PyDTrace_GC_START_ENABLED(void)
.. c:function:: int PyDTrace_GC_DONE_ENABLED(void)
.. c:function:: int PyDTrace_INSTANCE_NEW_START_ENABLED(void)
.. c:function:: int PyDTrace_INSTANCE_NEW_DONE_ENABLED(void)
.. c:function:: int PyDTrace_INSTANCE_DELETE_START_ENABLED(void)
.. c:function:: int PyDTrace_INSTANCE_DELETE_DONE_ENABLED(void)
.. c:function:: int PyDTrace_IMPORT_FIND_LOAD_START_ENABLED(void)
.. c:function:: int PyDTrace_IMPORT_FIND_LOAD_DONE_ENABLED(void)
.. c:function:: int PyDTrace_AUDIT_ENABLED(void)
All calls to ``PyDTrace`` functions must be guarded by a call to one
of these functions. This allows Python to minimize performance impact
when probing is disabled.
On builds without DTrace enabled, these functions do nothing and return
``0``.
SystemTap Tapsets
-----------------

13
Doc/howto/logging-cookbook.rst
View file

@ -1549,10 +1549,10 @@ to this (remembering to first import :mod:`concurrent.futures`)::
for i in range(10):
executor.submit(worker_process, queue, worker_configurer)
Deploying Web applications using Gunicorn and uWSGI
Deploying web applications using Gunicorn and uWSGI
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
When deploying Web applications using `Gunicorn <https://gunicorn.org/>`_ or `uWSGI
When deploying web applications using `Gunicorn <https://gunicorn.org/>`_ or `uWSGI
<https://uwsgi-docs.readthedocs.io/en/latest/>`_ (or similar), multiple worker
processes are created to handle client requests. In such environments, avoid creating
file-based handlers directly in your web application. Instead, use a
@ -3616,7 +3616,6 @@ detailed information.
.. code-block:: python3
import datetime
import logging
import random
import sys
@ -3851,7 +3850,7 @@ Logging to syslog with RFC5424 support
Although :rfc:`5424` dates from 2009, most syslog servers are configured by default to
use the older :rfc:`3164`, which hails from 2001. When ``logging`` was added to Python
in 2003, it supported the earlier (and only existing) protocol at the time. Since
RFC5424 came out, as there has not been widespread deployment of it in syslog
RFC 5424 came out, as there has not been widespread deployment of it in syslog
servers, the :class:`~logging.handlers.SysLogHandler` functionality has not been
updated.
@ -3859,7 +3858,7 @@ RFC 5424 contains some useful features such as support for structured data, and
need to be able to log to a syslog server with support for it, you can do so with a
subclassed handler which looks something like this::
import datetime
import datetime as dt
import logging.handlers
import re
import socket
@ -3877,8 +3876,8 @@ subclassed handler which looks something like this::
def format(self, record):
version = 1
asctime = datetime.datetime.fromtimestamp(record.created).isoformat()
m = self.tz_offset.match(time.strftime('%z'))
asctime = dt.datetime.fromtimestamp(record.created).isoformat()
m = self.tz_offset.prefixmatch(time.strftime('%z'))
has_offset = False
if m and time.timezone:
hrs, mins = m.groups()

2
Doc/howto/logging.rst
View file

@ -28,7 +28,7 @@ When to use logging
^^^^^^^^^^^^^^^^^^^
You can access logging functionality by creating a logger via ``logger =
getLogger(__name__)``, and then calling the logger's :meth:`~Logger.debug`,
logging.getLogger(__name__)``, and then calling the logger's :meth:`~Logger.debug`,
:meth:`~Logger.info`, :meth:`~Logger.warning`, :meth:`~Logger.error` and
:meth:`~Logger.critical` methods. To determine when to use logging, and to see
which logger methods to use when, see the table below. It states, for each of a

5
Doc/howto/perf_profiling.rst
View file

@ -217,8 +217,9 @@ Example, using the :mod:`sys` APIs in file :file:`example.py`:
How to obtain the best results
------------------------------
For best results, Python should be compiled with
``CFLAGS="-fno-omit-frame-pointer -mno-omit-leaf-frame-pointer"`` as this allows
For best results, keep frame pointers enabled. On supported GCC-compatible
toolchains, CPython builds itself with ``-fno-omit-frame-pointer`` and, when
available, ``-mno-omit-leaf-frame-pointer`` by default. These flags allow
profilers to unwind using only the frame pointer and not on DWARF debug
information. This is because as the code that is interposed to allow ``perf``
support is dynamically generated it doesn't have any DWARF debugging information

159
Doc/howto/regex.rst
View file

@ -1,7 +1,7 @@
.. _regex-howto:
****************************
Regular Expression HOWTO
Regular expression HOWTO
****************************
:Author: A.M. Kuchling <amk@amk.ca>
@ -47,7 +47,7 @@ Python code to do the processing; while Python code will be slower than an
elaborate regular expression, it will also probably be more understandable.
Simple Patterns
Simple patterns
===============
We'll start by learning about the simplest possible regular expressions. Since
@ -59,7 +59,7 @@ expressions (deterministic and non-deterministic finite automata), you can refer
to almost any textbook on writing compilers.
Matching Characters
Matching characters
-------------------
Most letters and characters will simply match themselves. For example, the
@ -159,7 +159,7 @@ match even a newline. ``.`` is often used where you want to match "any
character".
Repeating Things
Repeating things
----------------
Being able to match varying sets of characters is the first thing regular
@ -210,7 +210,7 @@ this RE against the string ``'abcbd'``.
| | | ``[bcd]*`` is only matching |
| | | ``bc``. |
+------+-----------+---------------------------------+
| 6 | ``abcb`` | Try ``b`` again. This time |
| 7 | ``abcb`` | Try ``b`` again. This time |
| | | the character at the |
| | | current position is ``'b'``, so |
| | | it succeeds. |
@ -255,7 +255,7 @@ is equivalent to ``+``, and ``{0,1}`` is the same as ``?``. It's better to use
to read.
Using Regular Expressions
Using regular expressions
=========================
Now that we've looked at some simple regular expressions, how do we actually use
@ -264,7 +264,7 @@ expression engine, allowing you to compile REs into objects and then perform
matches with them.
Compiling Regular Expressions
Compiling regular expressions
-----------------------------
Regular expressions are compiled into pattern objects, which have
@ -295,7 +295,7 @@ disadvantage which is the topic of the next section.
.. _the-backslash-plague:
The Backslash Plague
The backslash plague
--------------------
As stated earlier, regular expressions use the backslash character (``'\'``) to
@ -335,7 +335,7 @@ expressions will often be written in Python code using this raw string notation.
In addition, special escape sequences that are valid in regular expressions,
but not valid as Python string literals, now result in a
:exc:`DeprecationWarning` and will eventually become a :exc:`SyntaxError`,
:exc:`SyntaxWarning` and will eventually become a :exc:`SyntaxError`,
which means the sequences will be invalid if raw string notation or escaping
the backslashes isn't used.
@ -351,7 +351,7 @@ the backslashes isn't used.
+-------------------+------------------+
Performing Matches
Performing matches
------------------
Once you have an object representing a compiled regular expression, what do you
@ -362,20 +362,21 @@ for a complete listing.
+------------------+-----------------------------------------------+
| Method/Attribute | Purpose |
+==================+===============================================+
| ``match()`` | Determine if the RE matches at the beginning |
| | of the string. |
+------------------+-----------------------------------------------+
| ``search()`` | Scan through a string, looking for any |
| | location where this RE matches. |
+------------------+-----------------------------------------------+
| ``prefixmatch()``| Determine if the RE matches at the beginning |
| | of the string. Previously named :ref:`match() |
| | <prefixmatch-vs-match>`. |
+------------------+-----------------------------------------------+
| ``findall()`` | Find all substrings where the RE matches, and |
| | returns them as a list. |
| | return them as a list. |
+------------------+-----------------------------------------------+
| ``finditer()`` | Find all substrings where the RE matches, and |
| | returns them as an :term:`iterator`. |
| | return them as an :term:`iterator`. |
+------------------+-----------------------------------------------+
:meth:`~re.Pattern.match` and :meth:`~re.Pattern.search` return ``None`` if no match can be found. If
:meth:`~re.Pattern.search` and :meth:`~re.Pattern.prefixmatch` return ``None`` if no match can be found. If
they're successful, a :ref:`match object <match-objects>` instance is returned,
containing information about the match: where it starts and ends, the substring
it matched, and more.
@ -393,19 +394,19 @@ Python interpreter, import the :mod:`re` module, and compile a RE::
Now, you can try matching various strings against the RE ``[a-z]+``. An empty
string shouldn't match at all, since ``+`` means 'one or more repetitions'.
:meth:`~re.Pattern.match` should return ``None`` in this case, which will cause the
:meth:`~re.Pattern.search` should return ``None`` in this case, which will cause the
interpreter to print no output. You can explicitly print the result of
:meth:`!match` to make this clear. ::
:meth:`!search` to make this clear. ::
>>> p.match("")
>>> print(p.match(""))
>>> p.search("")
>>> print(p.search(""))
None
Now, let's try it on a string that it should match, such as ``tempo``. In this
case, :meth:`~re.Pattern.match` will return a :ref:`match object <match-objects>`, so you
case, :meth:`~re.Pattern.search` will return a :ref:`match object <match-objects>`, so you
should store the result in a variable for later use. ::
>>> m = p.match('tempo')
>>> m = p.search('tempo')
>>> m
<re.Match object; span=(0, 5), match='tempo'>
@ -437,27 +438,28 @@ Trying these methods will soon clarify their meaning::
:meth:`~re.Match.group` returns the substring that was matched by the RE. :meth:`~re.Match.start`
and :meth:`~re.Match.end` return the starting and ending index of the match. :meth:`~re.Match.span`
returns both start and end indexes in a single tuple. Since the :meth:`~re.Pattern.match`
method only checks if the RE matches at the start of a string, :meth:`!start`
will always be zero. However, the :meth:`~re.Pattern.search` method of patterns
scans through the string, so the match may not start at zero in that
case. ::
returns both start and end indexes in a single tuple.
The :meth:`~re.Pattern.search` method of patterns
scans through the string, so the match may not start at zero.
However, the :meth:`~re.Pattern.prefixmatch`
method only checks if the RE matches at the start of a string, so :meth:`!start`
will always be zero in that case. ::
>>> print(p.match('::: message'))
None
>>> m = p.search('::: message'); print(m)
<re.Match object; span=(4, 11), match='message'>
>>> m.group()
'message'
>>> m.span()
(4, 11)
>>> print(p.prefixmatch('::: message'))
None
In actual programs, the most common style is to store the
:ref:`match object <match-objects>` in a variable, and then check if it was
``None``. This usually looks like::
p = re.compile( ... )
m = p.match( 'string goes here' )
m = p.search( 'string goes here' )
if m:
print('Match found: ', m.group())
else:
@ -473,7 +475,7 @@ Two pattern methods return all of the matches for a pattern.
The ``r`` prefix, making the literal a raw string literal, is needed in this
example because escape sequences in a normal "cooked" string literal that are
not recognized by Python, as opposed to regular expressions, now result in a
:exc:`DeprecationWarning` and will eventually become a :exc:`SyntaxError`. See
:exc:`SyntaxWarning` and will eventually become a :exc:`SyntaxError`. See
:ref:`the-backslash-plague`.
:meth:`~re.Pattern.findall` has to create the entire list before it can be returned as the
@ -491,19 +493,19 @@ result. The :meth:`~re.Pattern.finditer` method returns a sequence of
(29, 31)
Module-Level Functions
Module-level functions
----------------------
You don't have to create a pattern object and call its methods; the
:mod:`re` module also provides top-level functions called :func:`~re.match`,
:func:`~re.search`, :func:`~re.findall`, :func:`~re.sub`, and so forth. These functions
:mod:`re` module also provides top-level functions called :func:`~re.search`,
:func:`~re.prefixmatch`, :func:`~re.findall`, :func:`~re.sub`, and so forth. These functions
take the same arguments as the corresponding pattern method with
the RE string added as the first argument, and still return either ``None`` or a
:ref:`match object <match-objects>` instance. ::
>>> print(re.match(r'From\s+', 'Fromage amk'))
>>> print(re.prefixmatch(r'From\s+', 'Fromage amk'))
None
>>> re.match(r'From\s+', 'From amk Thu May 14 19:12:10 1998') #doctest: +ELLIPSIS
>>> re.prefixmatch(r'From\s+', 'From amk Thu May 14 19:12:10 1998') #doctest: +ELLIPSIS
<re.Match object; span=(0, 5), match='From '>
Under the hood, these functions simply create a pattern object for you
@ -518,7 +520,7 @@ Outside of loops, there's not much difference thanks to the internal
cache.
Compilation Flags
Compilation flags
-----------------
.. currentmodule:: re
@ -642,7 +644,7 @@ of each one.
whitespace is in a character class or preceded by an unescaped backslash; this
lets you organize and indent the RE more clearly. This flag also lets you put
comments within a RE that will be ignored by the engine; comments are marked by
a ``'#'`` that's neither in a character class or preceded by an unescaped
a ``'#'`` that's neither in a character class nor preceded by an unescaped
backslash.
For example, here's a RE that uses :const:`re.VERBOSE`; see how much easier it
@ -669,7 +671,7 @@ of each one.
to understand than the version using :const:`re.VERBOSE`.
More Pattern Power
More pattern power
==================
So far we've only covered a part of the features of regular expressions. In
@ -679,7 +681,7 @@ retrieve portions of the text that was matched.
.. _more-metacharacters:
More Metacharacters
More metacharacters
-------------------
There are some metacharacters that we haven't covered yet. Most of them will be
@ -812,7 +814,7 @@ of a group with a quantifier, such as ``*``, ``+``, ``?``, or
``ab``. ::
>>> p = re.compile('(ab)*')
>>> print(p.match('ababababab').span())
>>> print(p.search('ababababab').span())
(0, 10)
Groups indicated with ``'('``, ``')'`` also capture the starting and ending
@ -825,7 +827,7 @@ argument. Later we'll see how to express groups that don't capture the span
of text that they match. ::
>>> p = re.compile('(a)b')
>>> m = p.match('ab')
>>> m = p.search('ab')
>>> m.group()
'ab'
>>> m.group(0)
@ -836,7 +838,7 @@ to determine the number, just count the opening parenthesis characters, going
from left to right. ::
>>> p = re.compile('(a(b)c)d')
>>> m = p.match('abcd')
>>> m = p.search('abcd')
>>> m.group(0)
'abcd'
>>> m.group(1)
@ -875,7 +877,7 @@ Backreferences like this aren't often useful for just searching through a string
find out that they're *very* useful when performing string substitutions.
Non-capturing and Named Groups
Non-capturing and named groups
------------------------------
Elaborate REs may use many groups, both to capture substrings of interest, and
@ -912,10 +914,10 @@ but aren't interested in retrieving the group's contents. You can make this fact
explicit by using a non-capturing group: ``(?:...)``, where you can replace the
``...`` with any other regular expression. ::
>>> m = re.match("([abc])+", "abc")
>>> m = re.search("([abc])+", "abc")
>>> m.groups()
('c',)
>>> m = re.match("(?:[abc])+", "abc")
>>> m = re.search("(?:[abc])+", "abc")
>>> m.groups()
()
@ -949,7 +951,7 @@ given numbers, so you can retrieve information about a group in two ways::
Additionally, you can retrieve named groups as a dictionary with
:meth:`~re.Match.groupdict`::
>>> m = re.match(r'(?P<first>\w+) (?P<last>\w+)', 'Jane Doe')
>>> m = re.search(r'(?P<first>\w+) (?P<last>\w+)', 'Jane Doe')
>>> m.groupdict()
{'first': 'Jane', 'last': 'Doe'}
@ -979,7 +981,7 @@ current point. The regular expression for finding doubled words,
'the the'
Lookahead Assertions
Lookahead assertions
--------------------
Another zero-width assertion is the lookahead assertion. Lookahead assertions
@ -1061,7 +1063,7 @@ end in either ``bat`` or ``exe``:
``.*[.](?!bat$|exe$)[^.]*$``
Modifying Strings
Modifying strings
=================
Up to this point, we've simply performed searches against a static string.
@ -1083,7 +1085,7 @@ using the following pattern methods:
+------------------+-----------------------------------------------+
Splitting Strings
Splitting strings
-----------------
The :meth:`~re.Pattern.split` method of a pattern splits a string apart
@ -1137,7 +1139,7 @@ argument, but is otherwise the same. ::
['Words', 'words, words.']
Search and Replace
Search and replace
------------------
Another common task is to find all the matches for a pattern, and replace them
@ -1236,7 +1238,7 @@ pattern object as the first parameter, or use embedded modifiers in the
pattern string, e.g. ``sub("(?i)b+", "x", "bbbb BBBB")`` returns ``'x x'``.
Common Problems
Common problems
===============
Regular expressions are a powerful tool for some applications, but in some ways
@ -1244,7 +1246,7 @@ their behaviour isn't intuitive and at times they don't behave the way you may
expect them to. This section will point out some of the most common pitfalls.
Use String Methods
Use string methods
------------------
Sometimes using the :mod:`re` module is a mistake. If you're matching a fixed
@ -1274,21 +1276,26 @@ In short, before turning to the :mod:`re` module, consider whether your problem
can be solved with a faster and simpler string method.
match() versus search()
-----------------------
.. _match-versus-search:
The :func:`~re.match` function only checks if the RE matches at the beginning of the
string while :func:`~re.search` will scan forward through the string for a match.
It's important to keep this distinction in mind. Remember, :func:`!match` will
only report a successful match which will start at 0; if the match wouldn't
start at zero, :func:`!match` will *not* report it. ::
prefixmatch() (aka match) versus search()
-----------------------------------------
>>> print(re.match('super', 'superstition').span())
:func:`~re.prefixmatch` was added in Python 3.15 as the :ref:`preferred name
<prefixmatch-vs-match>` for :func:`~re.match`. Before this, it was only known
as :func:`!match` and the distinction with :func:`~re.search` was often
misunderstood.
:func:`!prefixmatch` aka :func:`!match` only checks if the RE matches at the
beginning of the string while :func:`!search` scans forward through the
string for a match. ::
>>> print(re.prefixmatch('super', 'superstition').span())
(0, 5)
>>> print(re.match('super', 'insuperable'))
>>> print(re.prefixmatch('super', 'insuperable'))
None
On the other hand, :func:`~re.search` will scan forward through the string,
On the other hand, :func:`~re.search` scans forward through the string,
reporting the first match it finds. ::
>>> print(re.search('super', 'superstition').span())
@ -1296,21 +1303,11 @@ reporting the first match it finds. ::
>>> print(re.search('super', 'insuperable').span())
(2, 7)
Sometimes you'll be tempted to keep using :func:`re.match`, and just add ``.*``
to the front of your RE. Resist this temptation and use :func:`re.search`
instead. The regular expression compiler does some analysis of REs in order to
speed up the process of looking for a match. One such analysis figures out what
the first character of a match must be; for example, a pattern starting with
``Crow`` must match starting with a ``'C'``. The analysis lets the engine
quickly scan through the string looking for the starting character, only trying
the full match if a ``'C'`` is found.
Adding ``.*`` defeats this optimization, requiring scanning to the end of the
string and then backtracking to find a match for the rest of the RE. Use
:func:`re.search` instead.
This distinction is important to remember when using the old :func:`~re.match`
name in code requiring compatibility with older Python versions.
Greedy versus Non-Greedy
Greedy versus non-greedy
------------------------
When repeating a regular expression, as in ``a*``, the resulting action is to
@ -1322,9 +1319,9 @@ doesn't work because of the greedy nature of ``.*``. ::
>>> s = '<html><head><title>Title</title>'
>>> len(s)
32
>>> print(re.match('<.*>', s).span())
>>> print(re.prefixmatch('<.*>', s).span())
(0, 32)
>>> print(re.match('<.*>', s).group())
>>> print(re.prefixmatch('<.*>', s).group())
<html><head><title>Title</title>
The RE matches the ``'<'`` in ``'<html>'``, and the ``.*`` consumes the rest of
@ -1340,7 +1337,7 @@ example, the ``'>'`` is tried immediately after the first ``'<'`` matches, and
when it fails, the engine advances a character at a time, retrying the ``'>'``
at every step. This produces just the right result::
>>> print(re.match('<.*?>', s).group())
>>> print(re.prefixmatch('<.*?>', s).group())
<html>
(Note that parsing HTML or XML with regular expressions is painful.
@ -1388,9 +1385,9 @@ Feedback
========
Regular expressions are a complicated topic. Did this document help you
understand them? Were there parts that were unclear, or Problems you
understand them? Were there parts that were unclear, or problems you
encountered that weren't covered here? If so, please send suggestions for
improvements to the author.
improvements to the :ref:`issue tracker <using-the-tracker>`.
The most complete book on regular expressions is almost certainly Jeffrey
Friedl's Mastering Regular Expressions, published by O'Reilly. Unfortunately,

55
Doc/howto/remote_debugging.rst
View file

@ -624,3 +624,58 @@ To inject and execute a Python script in a remote process:
6. Set ``_PY_EVAL_PLEASE_STOP_BIT`` in the ``eval_breaker`` field.
7. Resume the process (if suspended). The script will execute at the next safe
evaluation point.
.. _remote-debugging-threat-model:
Security and threat model
=========================
The remote debugging protocol relies on the same operating system primitives
used by native debuggers such as GDB and LLDB. Attaching to a process
requires the **same privileges** that those debuggers require, for example
``ptrace`` / Yama LSM on Linux, ``task_for_pid`` on macOS, and
``SeDebugPrivilege`` on Windows. Python does not introduce any new privilege
escalation path; if an attacker already possesses the permissions needed to
attach to a process, they could equally use GDB to read memory or inject
code.
The following principles define what is, and is not, considered a security
vulnerability in this feature:
Attaching requires OS-level privileges
On every supported platform the operating system gates cross-process
memory access behind privilege checks (``CAP_SYS_PTRACE``, root, or
administrator rights). A report that demonstrates an issue only after
these privileges have already been obtained is **not** a vulnerability in
CPython, since the OS security boundary was already crossed.
Crashes or memory errors when reading a compromised process are not vulnerabilities
A tool that reads internal interpreter state from a target process must
trust that memory to be well-formed. If the target process has been
corrupted or is controlled by an attacker, the debugger or profiler may
crash, produce garbage output, or behave unpredictably. This is the same
risk accepted by every ``ptrace``-based debugger. Bugs in this category
(buffer overflows, segmentation faults, or undefined behaviour triggered
by reading corrupted state) are **not** treated as security issues, though
fixes that improve robustness are welcome.
Vulnerabilities in the target process are not in scope
If the Python process being debugged has already been compromised, the
attacker already controls execution in that process. Demonstrating further
impact from that starting point does not constitute a vulnerability in the
remote debugging protocol.
When to use ``PYTHON_DISABLE_REMOTE_DEBUG``
-------------------------------------------
The environment variable :envvar:`PYTHON_DISABLE_REMOTE_DEBUG` (and the
equivalent :option:`-X disable_remote_debug` flag) allows operators to disable
the in-process side of the protocol as a **defence-in-depth** measure. This
may be useful in hardened or sandboxed deployment environments where no
debugging or profiling of the process is expected and reducing attack surface
is a priority, even though the OS-level privilege checks already prevent
unprivileged access.
Setting this variable does **not** affect other OS-level debugging interfaces
(``ptrace``, ``/proc``, ``task_for_pid``, etc.), which remain available
according to their own permission models.

3
Doc/includes/capi-extension/spammodule-01.c
View file

@ -35,7 +35,10 @@ static PyMethodDef spam_methods[] = {
/// Module slot table
PyABIInfo_VAR(abi_info);
static PyModuleDef_Slot spam_slots[] = {
{Py_mod_abi, &abi_info},
{Py_mod_name, "spam"},
{Py_mod_doc, "A wonderful module with an example function"},
{Py_mod_methods, spam_methods},

8
Doc/includes/diff.py
View file

@ -1,4 +1,4 @@
""" Command line interface to difflib.py providing diffs in four formats:
""" Command-line interface to difflib.py providing diffs in four formats:
* ndiff: lists every line and highlights interline changes.
* context: highlights clusters of changes in a before/after format.
@ -8,11 +8,11 @@
"""
import sys, os, difflib, argparse
from datetime import datetime, timezone
import datetime as dt
def file_mtime(path):
t = datetime.fromtimestamp(os.stat(path).st_mtime,
timezone.utc)
t = dt.datetime.fromtimestamp(os.stat(path).st_mtime,
dt.timezone.utc)
return t.astimezone().isoformat()
def main():

127
Doc/includes/tzinfo_examples.py
View file

@ -1,68 +1,70 @@
from datetime import tzinfo, timedelta, datetime
ZERO = timedelta(0)
HOUR = timedelta(hours=1)
SECOND = timedelta(seconds=1)
import datetime as dt
# A class capturing the platform's idea of local time.
# (May result in wrong values on historical times in
# timezones where UTC offset and/or the DST rules had
# changed in the past.)
import time as _time
import time
STDOFFSET = timedelta(seconds = -_time.timezone)
if _time.daylight:
DSTOFFSET = timedelta(seconds = -_time.altzone)
ZERO = dt.timedelta(0)
HOUR = dt.timedelta(hours=1)
SECOND = dt.timedelta(seconds=1)
STDOFFSET = dt.timedelta(seconds=-time.timezone)
if time.daylight:
DSTOFFSET = dt.timedelta(seconds=-time.altzone)
else:
DSTOFFSET = STDOFFSET
DSTDIFF = DSTOFFSET - STDOFFSET
class LocalTimezone(tzinfo):
def fromutc(self, dt):
assert dt.tzinfo is self
stamp = (dt - datetime(1970, 1, 1, tzinfo=self)) // SECOND
args = _time.localtime(stamp)[:6]
class LocalTimezone(dt.tzinfo):
def fromutc(self, when):
assert when.tzinfo is self
stamp = (when - dt.datetime(1970, 1, 1, tzinfo=self)) // SECOND
args = time.localtime(stamp)[:6]
dst_diff = DSTDIFF // SECOND
# Detect fold
fold = (args == _time.localtime(stamp - dst_diff))
return datetime(*args, microsecond=dt.microsecond,
tzinfo=self, fold=fold)
fold = (args == time.localtime(stamp - dst_diff))
return dt.datetime(*args, microsecond=when.microsecond,
tzinfo=self, fold=fold)
def utcoffset(self, dt):
if self._isdst(dt):
def utcoffset(self, when):
if self._isdst(when):
return DSTOFFSET
else:
return STDOFFSET
def dst(self, dt):
if self._isdst(dt):
def dst(self, when):
if self._isdst(when):
return DSTDIFF
else:
return ZERO
def tzname(self, dt):
return _time.tzname[self._isdst(dt)]
def tzname(self, when):
return time.tzname[self._isdst(when)]
def _isdst(self, dt):
tt = (dt.year, dt.month, dt.day,
dt.hour, dt.minute, dt.second,
dt.weekday(), 0, 0)
stamp = _time.mktime(tt)
tt = _time.localtime(stamp)
def _isdst(self, when):
tt = (when.year, when.month, when.day,
when.hour, when.minute, when.second,
when.weekday(), 0, 0)
stamp = time.mktime(tt)
tt = time.localtime(stamp)
return tt.tm_isdst > 0
Local = LocalTimezone()
# A complete implementation of current DST rules for major US time zones.
def first_sunday_on_or_after(dt):
days_to_go = 6 - dt.weekday()
def first_sunday_on_or_after(when):
days_to_go = 6 - when.weekday()
if days_to_go:
dt += timedelta(days_to_go)
return dt
when += dt.timedelta(days_to_go)
return when
# US DST Rules
@ -75,21 +77,22 @@ def first_sunday_on_or_after(dt):
#
# In the US, since 2007, DST starts at 2am (standard time) on the second
# Sunday in March, which is the first Sunday on or after Mar 8.
DSTSTART_2007 = datetime(1, 3, 8, 2)
DSTSTART_2007 = dt.datetime(1, 3, 8, 2)
# and ends at 2am (DST time) on the first Sunday of Nov.
DSTEND_2007 = datetime(1, 11, 1, 2)
DSTEND_2007 = dt.datetime(1, 11, 1, 2)
# From 1987 to 2006, DST used to start at 2am (standard time) on the first
# Sunday in April and to end at 2am (DST time) on the last
# Sunday of October, which is the first Sunday on or after Oct 25.
DSTSTART_1987_2006 = datetime(1, 4, 1, 2)
DSTEND_1987_2006 = datetime(1, 10, 25, 2)
DSTSTART_1987_2006 = dt.datetime(1, 4, 1, 2)
DSTEND_1987_2006 = dt.datetime(1, 10, 25, 2)
# From 1967 to 1986, DST used to start at 2am (standard time) on the last
# Sunday in April (the one on or after April 24) and to end at 2am (DST time)
# on the last Sunday of October, which is the first Sunday
# on or after Oct 25.
DSTSTART_1967_1986 = datetime(1, 4, 24, 2)
DSTSTART_1967_1986 = dt.datetime(1, 4, 24, 2)
DSTEND_1967_1986 = DSTEND_1987_2006
def us_dst_range(year):
# Find start and end times for US DST. For years before 1967, return
# start = end for no DST.
@ -100,17 +103,17 @@ def us_dst_range(year):
elif 1966 < year < 1987:
dststart, dstend = DSTSTART_1967_1986, DSTEND_1967_1986
else:
return (datetime(year, 1, 1), ) * 2
return (dt.datetime(year, 1, 1), ) * 2
start = first_sunday_on_or_after(dststart.replace(year=year))
end = first_sunday_on_or_after(dstend.replace(year=year))
return start, end
class USTimeZone(tzinfo):
class USTimeZone(dt.tzinfo):
def __init__(self, hours, reprname, stdname, dstname):
self.stdoffset = timedelta(hours=hours)
self.stdoffset = dt.timedelta(hours=hours)
self.reprname = reprname
self.stdname = stdname
self.dstname = dstname
@ -118,45 +121,45 @@ def __init__(self, hours, reprname, stdname, dstname):
def __repr__(self):
return self.reprname
def tzname(self, dt):
if self.dst(dt):
def tzname(self, when):
if self.dst(when):
return self.dstname
else:
return self.stdname
def utcoffset(self, dt):
return self.stdoffset + self.dst(dt)
def utcoffset(self, when):
return self.stdoffset + self.dst(when)
def dst(self, dt):
if dt is None or dt.tzinfo is None:
def dst(self, when):
if when is None or when.tzinfo is None:
# An exception may be sensible here, in one or both cases.
# It depends on how you want to treat them. The default
# fromutc() implementation (called by the default astimezone()
# implementation) passes a datetime with dt.tzinfo is self.
# implementation) passes a datetime with when.tzinfo is self.
return ZERO
assert dt.tzinfo is self
start, end = us_dst_range(dt.year)
assert when.tzinfo is self
start, end = us_dst_range(when.year)
# Can't compare naive to aware objects, so strip the timezone from
# dt first.
dt = dt.replace(tzinfo=None)
if start + HOUR <= dt < end - HOUR:
# when first.
when = when.replace(tzinfo=None)
if start + HOUR <= when < end - HOUR:
# DST is in effect.
return HOUR
if end - HOUR <= dt < end:
# Fold (an ambiguous hour): use dt.fold to disambiguate.
return ZERO if dt.fold else HOUR
if start <= dt < start + HOUR:
if end - HOUR <= when < end:
# Fold (an ambiguous hour): use when.fold to disambiguate.
return ZERO if when.fold else HOUR
if start <= when < start + HOUR:
# Gap (a non-existent hour): reverse the fold rule.
return HOUR if dt.fold else ZERO
return HOUR if when.fold else ZERO
# DST is off.
return ZERO
def fromutc(self, dt):
assert dt.tzinfo is self
start, end = us_dst_range(dt.year)
def fromutc(self, when):
assert when.tzinfo is self
start, end = us_dst_range(when.year)
start = start.replace(tzinfo=self)
end = end.replace(tzinfo=self)
std_time = dt + self.stdoffset
std_time = when + self.stdoffset
dst_time = std_time + HOUR
if end <= dst_time < end + HOUR:
# Repeated hour

92
Doc/installing/index.rst
View file

@ -1,16 +1,14 @@
.. highlight:: none
.. highlight:: shell
.. _installing-index:
*************************
Installing Python Modules
Installing Python modules
*************************
:Email: distutils-sig@python.org
As a popular open source development project, Python has an active
supporting community of contributors and users that also make their software
available for other Python developers to use under open source license terms.
available for other Python developers to use under open-source license terms.
This allows Python users to share and collaborate effectively, benefiting
from the solutions others have already created to common (and sometimes
@ -34,34 +32,24 @@ creating and sharing your own Python projects, refer to the
Key terms
=========
* ``pip`` is the preferred installer program. Starting with Python 3.4, it
* :program:`pip` is the preferred installer program. It
is included by default with the Python binary installers.
* A *virtual environment* is a semi-isolated Python environment that allows
packages to be installed for use by a particular application, rather than
being installed system wide.
* ``venv`` is the standard tool for creating virtual environments, and has
been part of Python since Python 3.3. Starting with Python 3.4, it
defaults to installing ``pip`` into all created virtual environments.
* ``virtualenv`` is a third party alternative (and predecessor) to
``venv``. It allows virtual environments to be used on versions of
Python prior to 3.4, which either don't provide ``venv`` at all, or
aren't able to automatically install ``pip`` into created environments.
* The `Python Package Index <https://pypi.org>`__ is a public
* ``venv`` is the standard tool for creating virtual environments.
It defaults to installing :program:`pip` into all created virtual environments.
* ``virtualenv`` is a third-party alternative (and predecessor) to
``venv``.
* The `Python Package Index (PyPI) <https://pypi.org>`__ is a public
repository of open source licensed packages made available for use by
other Python users.
* the `Python Packaging Authority
* The `Python Packaging Authority
<https://www.pypa.io/>`__ is the group of
developers and documentation authors responsible for the maintenance and
evolution of the standard packaging tools and the associated metadata and
file format standards. They maintain a variety of tools, documentation,
and issue trackers on `GitHub <https://github.com/pypa>`__.
* ``distutils`` is the original build and distribution system first added to
the Python standard library in 1998. While direct use of ``distutils`` is
being phased out, it still laid the foundation for the current packaging
and distribution infrastructure, and it not only remains part of the
standard library, but its name lives on in other ways (such as the name
of the mailing list used to coordinate Python packaging standards
development).
.. versionchanged:: 3.5
The use of ``venv`` is now recommended for creating virtual environments.
@ -79,7 +67,7 @@ The standard packaging tools are all designed to be used from the command
line.
The following command will install the latest version of a module and its
dependencies from the Python Package Index::
dependencies from PyPI::
python -m pip install SomePackage
@ -106,7 +94,7 @@ explicitly::
python -m pip install --upgrade SomePackage
More information and resources regarding ``pip`` and its capabilities can be
More information and resources regarding :program:`pip` and its capabilities can be
found in the `Python Packaging User Guide <https://packaging.python.org>`__.
Creation of virtual environments is done through the :mod:`venv` module.
@ -124,19 +112,6 @@ How do I ...?
These are quick answers or links for some common tasks.
... install ``pip`` in versions of Python prior to Python 3.4?
--------------------------------------------------------------
Python only started bundling ``pip`` with Python 3.4. For earlier versions,
``pip`` needs to be "bootstrapped" as described in the Python Packaging
User Guide.
.. seealso::
`Python Packaging User Guide: Requirements for Installing Packages
<https://packaging.python.org/installing/#requirements-for-installing-packages>`__
.. installing-per-user-installation:
... install packages just for the current user?
@ -150,10 +125,10 @@ package just for the current user, rather than for all users of the system.
---------------------------------------
A number of scientific Python packages have complex binary dependencies, and
aren't currently easy to install using ``pip`` directly. At this point in
time, it will often be easier for users to install these packages by
aren't currently easy to install using :program:`pip` directly.
It will often be easier for users to install these packages by
`other means <https://packaging.python.org/science/>`__
rather than attempting to install them with ``pip``.
rather than attempting to install them with :program:`pip`.
.. seealso::
@ -166,22 +141,18 @@ rather than attempting to install them with ``pip``.
On Linux, macOS, and other POSIX systems, use the versioned Python commands
in combination with the ``-m`` switch to run the appropriate copy of
``pip``::
:program:`pip`::
python2 -m pip install SomePackage # default Python 2
python2.7 -m pip install SomePackage # specifically Python 2.7
python3 -m pip install SomePackage # default Python 3
python3.4 -m pip install SomePackage # specifically Python 3.4
python3 -m pip install SomePackage # default Python 3
python3.14 -m pip install SomePackage # specifically Python 3.14
Appropriately versioned ``pip`` commands may also be available.
Appropriately versioned :program:`pip` commands may also be available.
On Windows, use the ``py`` Python launcher in combination with the ``-m``
On Windows, use the :program:`py` Python launcher in combination with the ``-m``
switch::
py -2 -m pip install SomePackage # default Python 2
py -2.7 -m pip install SomePackage # specifically Python 2.7
py -3 -m pip install SomePackage # default Python 3
py -3.4 -m pip install SomePackage # specifically Python 3.4
py -3 -m pip install SomePackage # default Python 3
py -3.14 -m pip install SomePackage # specifically Python 3.14
.. other questions:
@ -201,39 +172,38 @@ On Linux systems, a Python installation will typically be included as part
of the distribution. Installing into this Python installation requires
root access to the system, and may interfere with the operation of the
system package manager and other components of the system if a component
is unexpectedly upgraded using ``pip``.
is unexpectedly upgraded using :program:`pip`.
On such systems, it is often better to use a virtual environment or a
per-user installation when installing packages with ``pip``.
per-user installation when installing packages with :program:`pip`.
Pip not installed
-----------------
It is possible that ``pip`` does not get installed by default. One potential fix is::
It is possible that :program:`pip` does not get installed by default. One potential fix is::
python -m ensurepip --default-pip
There are also additional resources for `installing pip.
<https://packaging.python.org/en/latest/tutorials/installing-packages/#ensure-pip-setuptools-and-wheel-are-up-to-date>`__
There are also additional resources for `installing pip
<https://packaging.python.org/en/latest/tutorials/installing-packages/#ensure-pip-setuptools-and-wheel-are-up-to-date>`__.
Installing binary extensions
----------------------------
Python has typically relied heavily on source based distribution, with end
Python once relied heavily on source-based distribution, with end
users being expected to compile extension modules from source as part of
the installation process.
With the introduction of support for the binary ``wheel`` format, and the
ability to publish wheels for at least Windows and macOS through the
Python Package Index, this problem is expected to diminish over time,
With the introduction of the binary wheel format, and the
ability to publish wheels through PyPI, this problem is diminishing,
as users are more regularly able to install pre-built extensions rather
than needing to build them themselves.
Some of the solutions for installing `scientific software
<https://packaging.python.org/science/>`__
that are not yet available as pre-built ``wheel`` files may also help with
that are not yet available as pre-built wheel files may also help with
obtaining other binary extensions without needing to build them locally.
.. seealso::

75
Doc/library/annotationlib.rst
View file

@ -510,6 +510,81 @@ annotations from the class and puts them in a separate attribute:
return typ
Creating a custom callable annotate function
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Custom :term:`annotate functions <annotate function>` may be literal functions like those
automatically generated for functions, classes, and modules. Or, they may wish to utilise
the encapsulation provided by classes, in which case any :term:`callable` can be used as
an :term:`annotate function`.
To provide the :attr:`~Format.VALUE`, :attr:`~Format.STRING`, or
:attr:`~Format.FORWARDREF` formats directly, an :term:`annotate function` must provide
the following attribute:
* A callable ``__call__`` with signature ``__call__(format, /) -> dict``, that does not
raise a :exc:`NotImplementedError` when called with a supported format.
To provide the :attr:`~Format.VALUE_WITH_FAKE_GLOBALS` format, which is used to
automatically generate :attr:`~Format.STRING` or :attr:`~Format.FORWARDREF` if they are
not supported directly, :term:`annotate functions <annotate function>` must provide the
following attributes:
* A callable ``__call__`` with signature ``__call__(format, /) -> dict``, that does not
raise a :exc:`NotImplementedError` when called with
:attr:`~Format.VALUE_WITH_FAKE_GLOBALS`.
* A :ref:`code object <code-objects>` ``__code__`` containing the compiled code for the
annotate function.
* Optional: A tuple of the function's positional defaults ``__kwdefaults__``, if the
function represented by ``__code__`` uses any positional defaults.
* Optional: A dict of the function's keyword defaults ``__defaults__``, if the function
represented by ``__code__`` uses any keyword defaults.
* Optional: All other :ref:`function attributes <inspect-types>`.
.. code-block:: python
class Annotate:
called_formats = []
def __call__(self, format=None, /, *, _self=None):
# When called with fake globals, `_self` will be the
# actual self value, and `self` will be the format.
if _self is not None:
self, format = _self, self
self.called_formats.append(format)
if format <= 2: # VALUE or VALUE_WITH_FAKE_GLOBALS
return {"x": MyType}
raise NotImplementedError
__code__ = __call__.__code__
__defaults__ = (None,)
__kwdefaults__ = property(lambda self: dict(_self=self))
__globals__ = {}
__builtins__ = {}
__closure__ = None
This can then be called with:
.. code-block:: pycon
>>> from annotationlib import call_annotate_function, Format
>>> call_annotate_function(Annotate(), format=Format.STRING)
{'x': 'MyType'}
Or used as the annotate function for an object:
.. code-block:: pycon
>>> from annotationlib import get_annotations, Format
>>> class C:
... pass
>>> C.__annotate__ = Annotate()
>>> get_annotations(Annotate(), format=Format.STRING)
{'x': 'MyType'}
Limitations of the ``STRING`` format
------------------------------------

Some files were not shown because too many files have changed in this diff Show more