Stowage/cpython
2
0
Fork 0
mirror of https://github.com/python/cpython.git synced 2026-01-07 07:52:29 +00:00
Code Issues Projects Releases Packages Wiki Activity

[3.12] Clarify the supported cases in the tokenize module (GH-105569) (#105573)

Browse source 
Co-authored-by: Pablo Galindo Salgado <Pablogsal@gmail.com>
This commit is contained in:
Miss Islington (bot) 2023-06-09 09:00:42 -07:00 • committed by GitHub
parent 7c298d2dc5
commit 97d846dc2b
No known key found for this signature in database
GPG key ID: 4AEE18F83AFDEB23
1 changed files with 9 additions and 0 deletions
Download patch file Download diff file Expand all files Collapse all files

9
Doc/library/tokenize.rst
View file

@ -22,6 +22,15 @@ the generic :data:`~token.OP` token type. The exact
type can be determined by checking the ``exact_type`` property on the
:term:`named tuple` returned from :func:`tokenize.tokenize`.
.. warning::
Note that the functions in this module are only designed to parse
syntactically valid Python code (code that does not raise when parsed
using :func:`ast.parse`). The behavior of the functions in this module is
**undefined** when providing invalid Python code and it can change at any
point.
Tokenizing Input
----------------