mirror of
https://github.com/python/cpython.git
synced 2025-10-30 21:21:22 +00:00
8f88190af5
* bpo-40204: Allow pre-Sphinx 3 syntax in the doc (GH-21844) Enable Sphinx 3.2 "c_allow_pre_v3" option and disable the c_warn_on_allowed_pre_v3 option to make the documentation compatible with Sphinx 2 and Sphinx 3. (cherry picked from commit423e77d6de) * bpo-40204: Fix Sphinx sytanx in howto/instrumentation.rst (GH-21858) Use generic '.. object::' to declare markers, rather than abusing '.. c:function::' which fails on Sphinx 3. (cherry picked from commit43577c01a2) * bpo-40204: Fix duplicates in the documentation (GH-21857) Fix two Sphinx 3 issues: Doc/c-api/buffer.rst:304: WARNING: Duplicate C declaration, also defined in 'c-api/buffer'. Declaration is 'PyBUF_ND'. Doc/c-api/unicode.rst:1603: WARNING: Duplicate C declaration, also defined in 'c-api/unicode'. Declaration is 'PyObject* PyUnicode_Translate(PyObject *str, PyObject *table, const char *errors)'. (cherry picked from commit46d10b1237) * bpo-40204: Add :noindex: in the documentation (GH-21859) Add :noindex: to duplicated documentation to fix "duplicate object description" errors. For example, fix this Sphinx 3 issue: Doc/library/configparser.rst:1146: WARNING: duplicate object description of configparser.ConfigParser.optionxform, other instance in library/configparser, use :noindex: for one of them (cherry picked from commitd3ded08048) * bpo-40204, doc: Fix syntax of C variables (GH-21846) For example, fix the following Sphinx 3 errors: Doc/c-api/buffer.rst:102: WARNING: Error in declarator or parameters Invalid C declaration: Expected identifier in nested name. [error at 5] void \*obj -----^ Doc/c-api/arg.rst:130: WARNING: Unparseable C cross-reference: 'PyObject*' Invalid C declaration: Expected end of definition. [error at 8] PyObject* --------^ The modified documentation is compatible with Sphinx 2 and Sphinx 3. (cherry picked from commit474652fe93) * bpo-40204: Fix reference to terms in the doc (GH-21865) Sphinx 3 requires to refer to terms with the exact case. For example, fix the Sphinx 3 warning: Doc/library/pkgutil.rst:71: WARNING: term Loader not found in case sensitive match.made a reference to loader instead. (cherry picked from commitbb0b08540c) * bpo-40204: Fix duplicated productionlist names in the doc (GH-21900) Sphinx 3 disallows having more than one productionlist markup with the same name. Simply remove names in this case, since names are not shown anyway. For example, fix the Sphinx 3 warning: Doc/reference/introduction.rst:96: duplicate token description of *:name, other instance in reference/expressions (cherry picked from commit1abeda80f7)
94 lines
2.7 KiB
ReStructuredText
94 lines
2.7 KiB
ReStructuredText
:mod:`token` --- Constants used with Python parse trees
|
|
=======================================================
|
|
|
|
.. module:: token
|
|
:synopsis: Constants representing terminal nodes of the parse tree.
|
|
|
|
.. sectionauthor:: Fred L. Drake, Jr. <fdrake@acm.org>
|
|
|
|
**Source code:** :source:`Lib/token.py`
|
|
|
|
--------------
|
|
|
|
This module provides constants which represent the numeric values of leaf nodes
|
|
of the parse tree (terminal tokens). Refer to the file :file:`Grammar/Grammar`
|
|
in the Python distribution for the definitions of the names in the context of
|
|
the language grammar. The specific numeric values which the names map to may
|
|
change between Python versions.
|
|
|
|
The module also provides a mapping from numeric codes to names and some
|
|
functions. The functions mirror definitions in the Python C header files.
|
|
|
|
|
|
.. data:: tok_name
|
|
|
|
Dictionary mapping the numeric values of the constants defined in this module
|
|
back to name strings, allowing more human-readable representation of parse trees
|
|
to be generated.
|
|
|
|
|
|
.. function:: ISTERMINAL(x)
|
|
|
|
Return ``True`` for terminal token values.
|
|
|
|
|
|
.. function:: ISNONTERMINAL(x)
|
|
|
|
Return ``True`` for non-terminal token values.
|
|
|
|
|
|
.. function:: ISEOF(x)
|
|
|
|
Return ``True`` if *x* is the marker indicating the end of input.
|
|
|
|
|
|
The token constants are:
|
|
|
|
.. include:: token-list.inc
|
|
|
|
The following token type values aren't used by the C tokenizer but are needed for
|
|
the :mod:`tokenize` module.
|
|
|
|
.. data:: COMMENT
|
|
|
|
Token value used to indicate a comment.
|
|
|
|
|
|
.. data:: NL
|
|
|
|
Token value used to indicate a non-terminating newline. The
|
|
:data:`NEWLINE` token indicates the end of a logical line of Python code;
|
|
``NL`` tokens are generated when a logical line of code is continued over
|
|
multiple physical lines.
|
|
|
|
|
|
.. data:: ENCODING
|
|
|
|
Token value that indicates the encoding used to decode the source bytes
|
|
into text. The first token returned by :func:`tokenize.tokenize` will
|
|
always be an ``ENCODING`` token.
|
|
|
|
|
|
.. data:: TYPE_COMMENT
|
|
:noindex:
|
|
|
|
Token value indicating that a type comment was recognized. Such
|
|
tokens are only produced when :func:`ast.parse()` is invoked with
|
|
``type_comments=True``.
|
|
|
|
|
|
.. versionchanged:: 3.5
|
|
Added :data:`AWAIT` and :data:`ASYNC` tokens.
|
|
|
|
.. versionchanged:: 3.7
|
|
Added :data:`COMMENT`, :data:`NL` and :data:`ENCODING` tokens.
|
|
|
|
.. versionchanged:: 3.7
|
|
Removed :data:`AWAIT` and :data:`ASYNC` tokens. "async" and "await" are
|
|
now tokenized as :data:`NAME` tokens.
|
|
|
|
.. versionchanged:: 3.8
|
|
Added :data:`TYPE_COMMENT`, :data:`TYPE_IGNORE`, :data:`COLONEQUAL`.
|
|
Added :data:`AWAIT` and :data:`ASYNC` tokens back (they're needed
|
|
to support parsing older Python versions for :func:`ast.parse` with
|
|
``feature_version`` set to 6 or lower).
|