cpython/Doc
Barry Warsaw 27ebd9abce
gh-150228: Improve the PEP 829 batch processing APIs (#150542)
* gh-150228: Improve the PEP 829 batch processing APIs

As previously discussed with @ncoghlan and approved for 3.15b2 by @hugovk,
this implements the batch processing APIs for addsitedir() and friends.  We
remove the `defer_processing_start_files` flag which required some implicit
module global state, and promote StartupState to the public documented API.

This also moves the bulk of the module global functions into methods of the
`StartupState` class, so it removes the awkward APIs in 3.15b1.  Now, instances
of this class are an accumulator for startup state, using `StartupState.process()`
to process them.  Callers can now batch up startup state themselves by using
the methods on this class.  The module global functions are shims for this
which preserve the legacy APIs and semantics using the new state class.

This PR also fixes the interleaving regression identified by @ncoghlan in the
same issue.  Now, .pth file sys.path extensions are added to sys.path after
the sitedir that the .pth file is found in, restoring the legacy behavior.

Along the way, I've made a lot of improvements to function docstrings,
site.rst documentation, and comments in the code explaining what's going on.

* Add a note that if known_paths is provided to StartupState.__init__(), it
  will get mutated in place.
* Improve some conditional flows.
* Improve some comments.
* Improve the what's new entry.

* Make test_impl_exec_imports_suppressed_by_matching_start() more robust

Based on PR comment, we need to read both the .pth and .start files, and prove
that the .pth file's import line (which passes a bigger increment) is not
called, but the .start file's entry point (which uses the default increment)
is called.

* As per review, move some methods to the private API

_read_pth_file() and _read_start_file() are not intended to be part of the
public API surface outside of the site module, so even though they are used by
methods outside of the StartupState class, make them privately named.

* Resolve several review feedbacks

* Move a `versionadded`
* Better list comprehension formatting (use the output from
  `ruff format --line-length 78`)

* Add docs for site.makepath() and point the case-normalization requirement to
  this utility function.
* Note that StartupState.process() is not idempotent.

* Address another feedback comment

This time, we get rid of the legacy implementation `reset` local, which was
always difficult to understand, and just implement a return value based on the
processing mode selected.

* Changes based on gh-150228 review

The comment by @encukou that started this change:

```
I still see two red flags here though: an argument that doesn't combine with
other arguments, and (another instance of) changing the return type based on
an argument.

Did you consider adding a StartupState.addsitedir(sitedir) method, instead of
the startup_state argument?
```

As it turns out, this is an even cleaner design.  By moving the bulk of the
previous module global functions into `StartupState` methods, we can get rid
of all the awkward `startup_state` keyword-only arguments which conflict
with `known_path` (Petr's first point).  We can also get rid of the
return value dichotomy (Petr's second point) because now we can preserve
exactly the Python 3.14 API in the module global functions, and implement
the better APIs in the class methods.  We also generally don't have to
pass around `process_known_sitedirs`.

Now the following module global functions are essentially shims around
class methods:

* site.addsitedir() -> StartupState.addsitedir()
* site.addusersitepackages() -> StartupState.addusersitepackages()
* site.addsitepackages() -> StartupState.addsitepackages()
* Additional minor changes
* Remove a now unused parameter


Co-authored-by: Hugo van Kemenade <1324225+hugovk@users.noreply.github.com>
2026-06-01 18:43:18 -07:00
..
_static gh-138122: Add sampling profiler visualisation to docs (#142772) 2026-02-10 23:09:07 +00:00
c-api Fix docs for PyThreadState_GetDict() in threads.rst (#150401) 2026-05-28 12:13:58 +00:00
data gh-148829: Make sentinels' repr and module customizable (#149654) 2026-05-19 09:18:56 -07:00
deprecations gh-149609: Raise deprecation warnings for abc.{abstractclassmethod,abstractstaticmethod,abstractproperty} (#149636) 2026-05-31 07:26:52 +00:00
distributing GH-107987: Remove the Distributing Python Modules guide (#108016) 2023-08-17 13:01:14 +03:00
extending gh-149044: Implement PEP 820 – PySlot: Unified slot system for the C API (GH-149055) 2026-05-05 09:18:04 +02:00
faq gh-149083: Change several other docs examples to use sentinel() (#149213) 2026-05-01 22:53:28 +03:00
howto gh-135898: Add section to free-threading howto about memory usage (#143279) 2026-05-30 10:41:33 +00:00
includes gh-149044: Implement PEP 820 – PySlot: Unified slot system for the C API (GH-149055) 2026-05-05 09:18:04 +02:00
installing Docs: Update "Installing Python modules" (#146249) 2026-04-06 14:21:59 +03:00
library gh-150228: Improve the PEP 829 batch processing APIs (#150542) 2026-06-01 18:43:18 -07:00
reference gh-148672: Document namespace subpackages inside regular packages (#150056) 2026-06-01 12:22:42 -07:00
tools gh-141984: Remove accidental merge markers (GH-150517) 2026-05-27 14:20:50 +00:00
tutorial gh-149189: Revert "Modern defaults for pprint (#149190)" (#150249) 2026-05-22 23:22:03 +03:00
using gh-129851: Fix the documentation for -m command (GH-129862) 2026-05-30 15:47:14 +00:00
whatsnew gh-150228: Improve the PEP 829 batch processing APIs (#150542) 2026-06-01 18:43:18 -07:00
.ruff.toml GH-145000: Add a tool to record/check removed HTML IDs (#145001) 2026-02-25 13:37:59 +01:00
about.rst Remove `Misc/ACKS` check from patchcheck, documentation (#141960) 2025-11-26 00:00:00 +00:00
bugs.rst Fix typos and grammar errors across documentation (#144709) 2026-02-11 16:35:25 +00:00
conf.py Remove unneeded LaTeX code from latex_elements['preamble'] (GH-124630) 2026-05-30 15:19:56 +00:00
constraints.txt GH-125722: Increase minimum supported Sphinx to 8.1.3 (#128922) 2025-01-20 00:26:24 +00:00
contents.rst gh-122085: Create dedicated page for deprecations (#122352) 2024-07-28 10:53:21 +03:00
copyright.rst gh-126133: Only use start year in PSF copyright, remove end years (#126236) 2024-11-12 15:59:19 +02:00
glossary.rst gh-141388: Improve docs/tests for non-function callables as annotate functions (#142327) 2026-05-02 18:21:59 -07:00
improve-page-nojs.rst Docs: an "improve this page" feature (#144939) 2026-02-18 04:58:30 -05:00
improve-page.rst Docs: an "improve this page" feature (#144939) 2026-02-18 04:58:30 -05:00
license.rst Docs: Pull expat license from Modules/expat/ in license.rst (#144488) 2026-02-06 09:54:34 +01:00
make.bat Revert pylock.toml change to make MSI builds pass (#149175) 2026-04-30 10:30:34 +02:00
Makefile Build docs from pylock.toml (#149058) 2026-04-28 21:45:38 +03:00
pylock.toml Update Doc/pylock.toml (#149467) 2026-05-06 16:05:30 -04:00
README.rst Add lightweight comments to conf.py and update docs readme (GH-126100) 2024-10-29 17:45:03 +00:00
requirements.txt Docs: use a Sphinx extension to eliminate excessive links (#145130) 2026-03-03 15:48:43 -05:00

Python Documentation README
~~~~~~~~~~~~~~~~~~~~~~~~~~~

This directory contains the reStructuredText (reST) sources to the Python
documentation.  You don't need to build them yourself, `prebuilt versions are
available <https://docs.python.org/dev/download.html>`_.

Documentation on authoring Python documentation, including information about
both style and markup, is available in the "`Documenting Python
<https://devguide.python.org/documenting/>`_" chapter of the
developers guide.


Building the docs
=================

The documentation is built with several tools which are not included in this
tree but are maintained separately and are available from
`PyPI <https://pypi.org/>`_.

* `Sphinx <https://pypi.org/project/Sphinx/>`_
* `blurb <https://pypi.org/project/blurb/>`_
* `python-docs-theme <https://pypi.org/project/python-docs-theme/>`_

The easiest way to install these tools is to create a virtual environment and
install the tools into there.

Using make
----------

To get started on Unix, you can create a virtual environment and build
documentation with the commands::

  make venv
  make html

The virtual environment in the ``venv`` directory will contain all the tools
necessary to build the documentation downloaded and installed from PyPI.
If you'd like to create the virtual environment in a different location,
you can specify it using the ``VENVDIR`` variable.

You can also skip creating the virtual environment altogether, in which case
the ``Makefile`` will look for instances of ``sphinx-build`` and ``blurb``
installed on your process ``PATH`` (configurable with the ``SPHINXBUILD`` and
``BLURB`` variables).

On Windows, we try to emulate the ``Makefile`` as closely as possible with a
``make.bat`` file. If you need to specify the Python interpreter to use,
set the ``PYTHON`` environment variable.

Available make targets are:

* "clean", which removes all build files and the virtual environment.

* "clean-venv", which removes the virtual environment directory.

* "venv", which creates a virtual environment with all necessary tools
  installed.

* "html", which builds standalone HTML files for offline viewing.

* "htmlview", which re-uses the "html" builder, but then opens the main page
  in your default web browser.

* "htmllive", which re-uses the "html" builder, rebuilds the docs,
  starts a local server, and automatically reloads the page in your browser
  when you make changes to reST files (Unix only).

* "htmlhelp", which builds HTML files and a HTML Help project file usable to
  convert them into a single Compiled HTML (.chm) file -- these are popular
  under Microsoft Windows, but very handy on every platform.

  To create the CHM file, you need to run the Microsoft HTML Help Workshop
  over the generated project (.hhp) file.  The ``make.bat`` script does this for
  you on Windows.

* "latex", which builds LaTeX source files as input to ``pdflatex`` to produce
  PDF documents.

* "text", which builds a plain text file for each source file.

* "epub", which builds an EPUB document, suitable to be viewed on e-book
  readers.

* "linkcheck", which checks all external references to see whether they are
  broken, redirected or malformed, and outputs this information to stdout as
  well as a plain-text (.txt) file.

* "changes", which builds an overview over all versionadded/versionchanged/
  deprecated items in the current version. This is meant as a help for the
  writer of the "What's New" document.

* "coverage", which builds a coverage overview for standard library modules and
  C API.

* "pydoc-topics", which builds a Python module containing a dictionary with
  plain text documentation for the labels defined in
  ``tools/pyspecific.py`` -- pydoc needs these to show topic and keyword help.

* "check", which checks for frequent markup errors.

* "dist", (Unix only) which creates distributable archives of HTML, text,
  PDF, and EPUB builds.


Without make
------------

First, install the tool dependencies from PyPI.

Then, from the ``Doc`` directory, run ::

   sphinx-build -b<builder> . build/<builder>

where ``<builder>`` is one of html, text, latex, or htmlhelp (for explanations
see the make targets above).

Deprecation header
==================

You can define the ``outdated`` variable in ``html_context`` to show a
red banner on each page redirecting to the "latest" version.

The link points to the same page on ``/3/``, sadly for the moment the
language is lost during the process.


Contributing
============

Bugs in the content should be reported to the
`Python bug tracker <https://github.com/python/cpython/issues>`_.

Bugs in the toolset should be reported to the tools themselves.

To help with the documentation, or report any problems, please leave a message
on `discuss.python.org <https://discuss.python.org/c/documentation>`_.