Stowage/cpython
2
0
Fork 0
mirror of https://github.com/python/cpython.git synced 2026-04-02 10:00:45 +00:00
Code Issues Projects Releases Packages Wiki Activity
cpython/Lib
History
Ivan Levkivskyi 9932a22897
bpo-33416: Add end positions to Python AST (GH-11605) 
The majority of this PR is tediously passing `end_lineno` and `end_col_offset` everywhere. Here are non-trivial points:
* It is not possible to reconstruct end positions in AST "on the fly", some information is lost after an AST node is constructed, so we need two more attributes for every AST node `end_lineno` and `end_col_offset`.
* I add end position information to both CST and AST.  Although it may be technically possible to avoid adding end positions to CST, the code becomes more cumbersome and less efficient.
* Since the end position is not known for non-leaf CST nodes while the next token is added, this requires a bit of extra care (see `_PyNode_FinalizeEndPos`). Unless I made some mistake, the algorithm should be linear.
* For statements, I "trim" the end position of suites to not include the terminal newlines and dedent (this seems to be what people would expect), for example in
  ```python
  class C:
      pass

  pass
  ```
  the end line and end column for the class definition is (2, 8).
* For `end_col_offset` I use the common Python convention for indexing, for example for `pass` the `end_col_offset` is 4 (not 3), so that `[0:4]` gives one the source code that corresponds to the node.
* I added a helper function `ast.get_source_segment()`, to get source text segment corresponding to a given AST node. It is also useful for testing.

An (inevitable) downside of this PR is that AST now takes almost 25% more memory. I think however it is probably justified by the benefits.
2019-01-22 11:18:22 +00:00
..
asyncio bpo-23846: Fix ProactorEventLoop._write_to_self() (GH-11566) 2019-01-15 13:58:38 +01:00
collections bpo-32492: Tweak _collections._tuplegetter. (GH-11367) 2018-12-31 14:15:16 +02:00
concurrent bpo-35133: Fix mistakes when concatenate string literals on different lines. (GH-10284) 2018-11-05 16:20:25 +02:00
ctypes bpo-27643 - skip test_ctypes test case with XLC compiler. (GH-5164) 2018-12-26 13:54:22 +10:00
curses
dbm bpo-33106: change dbm key deletion error for readonly file from KeyError to dbm.error (#6295) 2018-12-12 20:46:55 +08:00
distutils bpo-35699: fix distuils cannot detect Build Tools 2017 anymore (GH-11495) 2019-01-20 13:47:42 -05:00
email bpo-35133: Fix mistakes when concatenate string literals on different lines. (GH-10284) 2018-11-05 16:20:25 +02:00
encodings Remove obsolete comment about latin-1 in normalize_encoding (GH-8739) 2018-09-10 17:54:37 -07:00
ensurepip Upgrade pip to 18.1 and setuptools to 40.6.2 (#10598) 2018-11-19 07:41:52 -05:00
html bpo-30629: Remove second call of str.lower() in html.parser.parse_endtag. (#2099) 2017-06-16 21:15:25 -04:00
http bpo-34711: Return HTTPStatus.NOT_FOUND if path.endswith('/') and not a directory (GH-9687) 2018-12-26 15:43:42 +10:00
idlelib bpo-35770: Fix off-by-1 error. (#11618) 2019-01-18 17:05:40 -05:00
importlib bpo-35133: Fix mistakes when concatenate string literals on different lines. (GH-10284) 2018-11-05 16:20:25 +02:00
json bpo-31553: add --json-lines option to json.tool (#10051) 2018-11-07 12:09:32 +02:00
lib2to3 bpo-35312: Make lib2to3.pgen2.parse.ParseError round-trip pickle-able. (GH-10710) 2018-11-27 20:39:49 +02:00
logging bpo-35046: do only one system call per line (logging.StreamHandler) (GH-10042) 2018-10-23 07:48:38 +01:00
msilib bpo-34251: Restore msilib.Win64 to preserve compatibility (GH-8510) 2018-07-28 16:02:56 -05:00
multiprocessing bpo-35424: emit ResourceWarning at multiprocessing.Pool destruction (GH-10974) 2018-12-20 20:33:51 +01:00
pydoc_data Update NEWS, docs, and patchlevel for 3.7.0b1 2018-01-31 17:14:30 -05:00
site-packages
sqlite3 bpo-35504: Fix segfaults and SystemErrors when deleting certain attrs. (GH-11175) 2018-12-17 16:52:45 +02:00
test bpo-33416: Add end positions to Python AST (GH-11605) 2019-01-22 11:18:22 +00:00
tkinter Fix yet one error in checking Tcl version. (GH-10189) 2018-10-28 20:58:04 +02:00
turtledemo turtledemo/penrose.py: remove unused clock() calls (GH-10033) 2018-10-25 00:43:39 +02:00
unittest bpo-20239: Allow repeated deletion of unittest.mock.Mock attributes (#11057) 2019-01-21 08:57:46 +00:00
urllib closes bpo-35309: cpath should be capath (GH-10699) 2018-11-25 12:32:50 -06:00
venv bpo-34977: Use venv redirector instead of original python.exe on Windows (GH-11029) 2018-12-10 08:11:21 -08:00
wsgiref bpo-35565: Add detail to assertion failure message in wsgiref (GH-11293) 2018-12-25 15:19:11 -08:00
xml bpo-35052: Fix handler on xml.dom.minidom.cloneNode() (GH-11061) 2018-12-10 11:12:53 +01:00
xmlrpc bpo-33911: Fixed deprecation warning in xmlrpc.server (GH-7847) 2018-07-16 10:46:04 +02:00
__future__.py bpo-35526: make __future__.barry_as_FLUFL mandatory for Python 4.0 (#11218) 2018-12-19 08:19:39 -08:00
__phello__.foo.py
_bootlocale.py bpo-29240: PEP 540: Add a new UTF-8 Mode (#855) 2017-12-13 12:29:09 +01:00
_collections_abc.py bpo-34427: Fix infinite loop when calling MutableSequence.extend() on self (GH-8813) 2018-08-30 09:56:14 -07:00
_compat_pickle.py
_compression.py
_dummy_thread.py Restore dummy_threading and _dummy_thread, but deprecate them (bpo-31370) (#3648) 2017-09-18 22:04:20 +02:00
_markupbase.py
_osx_support.py bpo-35257: Avoid leaking LTO linker flags into distutils (GH-10900) 2018-12-19 18:19:01 +01:00
_py_abc.py bpo-33018: Improve issubclass() error checking and message. (GH-5944) 2018-03-22 11:26:06 +00:00
_pydecimal.py bpo-35133: Fix mistakes when concatenate string literals on different lines. (GH-10284) 2018-11-05 16:20:25 +02:00
_pyio.py bpo-33138: Change standard error message for non-pickleable and non-copyable types. (GH-6239) 2018-10-31 02:28:07 +02:00
_sitebuiltins.py
_strptime.py bpo-33541: Remove unused __pad function (GH-4377) 2018-05-20 13:42:30 -04:00
_threading_local.py Delete a broken threading.local example (#5870) 2018-02-25 10:03:40 -05:00
_weakrefset.py bpo-30296 Remove unnecessary tuples, lists, sets, and dicts (#1489) 2017-05-18 07:35:54 -07:00
abc.py bpo-35609: Remove examples for deprecated decorators in the abc module. (GH-11355) 2018-12-31 09:56:21 +02:00
aifc.py bpo-32056: Improve exceptions in aifc, wave and sunau. (GH-5951) 2018-03-18 09:55:53 +02:00
antigravity.py Change the xkcd link in comment over https. (GH-5452) 2018-09-13 22:45:00 -07:00
argparse.py Remove superseded line from argparse.HelpFormatter() (GH-8839) 2018-08-22 23:14:14 +03:00
ast.py bpo-33416: Add end positions to Python AST (GH-11605) 2019-01-22 11:18:22 +00:00
asynchat.py
asyncore.py bpo-29639: change test.support.HOST to "localhost" 2017-09-09 00:30:15 -07:00
base64.py bpo-34164: Fix handling of incorrect padding in base64.b32decode(). (GH-8351) 2018-07-24 12:52:51 +03:00
bdb.py bpo-34906: Doc: Fix typos (GH-9712) 2018-10-05 16:17:18 +02:00
binhex.py
bisect.py
bz2.py bpo-35128: Fix spacing issues in warning.warn() messages. (GH-10268) 2018-11-01 12:33:35 +02:00
calendar.py Closes bpo-28281: Remove year (1-9999) limits on the weekday() function. (#4109) 2017-10-26 15:34:11 -04:00
cgi.py bpo-35028: cgi: Fix max_num_fields off by one error (GH-9973) 2018-10-23 01:14:35 -07:00
cgitb.py bpo-33311: Do not display parameters displayed in parentheses for module call. (GH-6677) 2018-05-09 12:39:32 +03:00
chunk.py bpo-29762: More use "raise from None". (#569) 2017-04-05 09:37:24 +03:00
cmd.py
code.py bpo-30166: Import command-line parsing modules only when needed. (#1293) 2017-05-04 08:17:47 +03:00
codecs.py bpo-32236: open() emits RuntimeWarning if buffering=1 for binary mode (GH-4842) 2018-10-20 02:22:31 +02:00
codeop.py
colorsys.py
compileall.py bpo-29877: compileall: import ProcessPoolExecutor only when needed (GH-4856) 2018-11-23 09:06:55 -08:00
configparser.py bpo-27351: Fix ConfigParser.read() documentation and docstring (GH-8123) 2018-09-29 10:15:55 -06:00
contextlib.py bpo-33786: Fix asynchronous generators to handle GeneratorExit in athrow() (GH-7467) 2018-06-07 20:31:26 -04:00
contextvars.py bpo-32436: Implement PEP 567 (#5027) 2018-01-22 19:11:18 -05:00
copy.py bpo-11572: Make minor improvements to copy module (GH-8208) 2018-07-09 23:14:54 +03:00
copyreg.py bpo-33138: Change standard error message for non-pickleable and non-copyable types. (GH-6239) 2018-10-31 02:28:07 +02:00
cProfile.py bpo-23420: Verify the value of '-s' when execute the CLI of cProfile (GH-9925) 2018-10-17 12:03:40 +02:00
crypt.py bpo-31702: Allow to specify rounds for SHA-2 hashing in crypt.mksalt(). (#4110) 2017-11-16 13:22:51 +02:00
csv.py bpo-30157: Fix csv.Sniffer.sniff() regex pattern. (GH-5601) 2018-02-09 20:00:49 +02:00
dataclasses.py bpo-33947: dataclasses no longer can raise RecursionError in repr (GF9916) 2018-10-19 12:54:50 -04:00
datetime.py bpo-22005: Fixed unpickling instances of datetime classes pickled by Python 2. (GH-11017) 2018-12-07 13:42:10 +02:00
decimal.py
difflib.py Revert "bpo-35603: Escape table header of make_table output that can cause potential XSS. (GH-11341)" (GH-11356) 2019-01-02 14:49:25 +02:00
dis.py bpo-32970: Improve disassembly of the MAKE_FUNCTION instruction. (GH-5937) 2018-03-11 11:07:06 +02:00
doctest.py bpo-24746: Avoid stripping trailing whitespace in doctest fancy diff (#10639) 2019-01-09 05:38:38 -08:00
dummy_threading.py Restore dummy_threading and _dummy_thread, but deprecate them (bpo-31370) (#3648) 2017-09-18 22:04:20 +02:00
enum.py Speed-up building enums by value, e.g. http.HTTPStatus(200) (#11318) 2018-12-26 10:45:33 -08:00
filecmp.py
fileinput.py bpo-9372: Deprecate several __getitem__ methods (GH-8609) 2018-08-11 09:05:04 +03:00
fnmatch.py bpo-32775: Fix regular expression warnings in fnmatch. (#5583) 2018-02-09 13:30:19 +02:00
formatter.py
fractions.py bpo-35588: Speed up mod, divmod and floordiv operations for Fraction type (#11322) 2019-01-02 14:22:06 +02:00
ftplib.py bpo-35128: Fix spacing issues in warning.warn() messages. (GH-10268) 2018-11-01 12:33:35 +02:00
functools.py bpo-34890: Make iscoroutinefunction, isgeneratorfunction and isasyncgenfunction work with functools.partial (GH-9903) 2018-10-26 12:19:14 +01:00
genericpath.py bpo-33721: Make some os.path functions and pathlib.Path methods be tolerant to invalid paths. (#7695) 2018-09-18 11:28:51 +03:00
getopt.py
getpass.py getpass: update docstrings (#49) 2017-02-12 23:10:48 +01:00
gettext.py bpo-2504: Add pgettext() and variants to gettext. (GH-7253) 2018-11-07 16:12:20 +02:00
glob.py
gzip.py bpo-34898: Add mtime parameter to gzip.compress(). (GH-9704) 2018-11-07 11:50:23 +02:00
hashlib.py bpo-33729: Fix issues with arguments parsing in hashlib. (GH-8346) 2018-07-31 09:50:16 +03:00
heapq.py bpo-34149: Behavior of the min/max with key=None (GH-8328) 2018-07-23 20:58:21 -07:00
hmac.py bpo-33604: Remove deprecated HMAC default value marked for removal in 3.8 (GH-7063) 2018-09-10 11:10:01 -07:00
imaplib.py bpo-35128: Fix spacing issues in warning.warn() messages. (GH-10268) 2018-11-01 12:33:35 +02:00
imghdr.py
imp.py closes bpo-34056: Always return bytes from _HackedGetData.get_data(). (GH-8130) 2018-07-06 20:41:06 -07:00
inspect.py bpo-34890: Make iscoroutinefunction, isgeneratorfunction and isasyncgenfunction work with functools.partial (GH-9903) 2018-10-26 12:19:14 +01:00
io.py
ipaddress.py bpo-27683: Fix a regression for host() of ipaddress network objects (GH-6016) 2018-03-21 08:25:13 +08:00
keyword.py bpo-30406: Make async and await proper keywords (#1669) 2017-10-05 23:24:46 -04:00
linecache.py
locale.py Remove mojibake in the locale aliases mapping. (GH-6716) 2018-05-06 10:52:38 +03:00
lzma.py
mailbox.py bpo-31522: mailbox.get_string: pass from_ parameter to get_bytes (#9857) 2018-10-18 20:21:47 -04:00
mailcap.py
mimetypes.py bpo-34926: Make mimetypes.guess_type accept os.PathLike objects (GH-9777) 2018-10-10 16:46:44 +02:00
modulefinder.py closes bpo-31650: PEP 552 (Deterministic pycs) implementation (#4575) 2017-12-09 10:26:52 -08:00
netrc.py bpo-28334: netrc() now uses expanduser() to find .netrc file (GH-4537) 2017-11-25 13:37:22 +03:00
nntplib.py bpo-29762: More use "raise from None". (#569) 2017-04-05 09:37:24 +03:00
ntpath.py bpo-31047: Fix ntpath.abspath to trim ending separator (GH-10082) 2018-10-25 11:26:37 -04:00
nturl2path.py bpo-29836: Remove nturl2path from test_sundry and amend the module docstring (GH-694) 2017-03-17 11:16:20 -07:00
numbers.py Fix miscellaneous typos (#4275) 2017-11-05 15:37:50 +02:00
opcode.py bpo-33041: Rework compiling an "async for" loop. (#6142) 2018-03-23 14:34:35 +02:00
operator.py
optparse.py bpo-34605: Avoid master/slave terms (GH-9101) 2018-09-07 17:30:33 +02:00
os.py Use generator instead of list in code examples (GH-11203) 2018-12-22 19:48:14 -08:00
pathlib.py bpo-33721: Make some os.path functions and pathlib.Path methods be tolerant to invalid paths. (#7695) 2018-09-18 11:28:51 +03:00
pdb.py bpo-32691: Use mod_spec.parent when running modules with pdb (GH-5474) 2018-02-03 16:40:11 +10:00
pickle.py bpo-11572: Make minor improvements to copy module (GH-8208) 2018-07-09 23:14:54 +03:00
pickletools.py Fix pickletools doc for NEWFALSE. (GH-9432) 2018-09-22 18:13:53 +03:00
pipes.py
pkgutil.py closes bpo-31650: PEP 552 (Deterministic pycs) implementation (#4575) 2017-12-09 10:26:52 -08:00
platform.py bpo-35516: platform.system_alias() don't replace Darwin (GH-11207) 2018-12-18 19:51:35 +01:00
plistlib.py bpo-33908: Remove two superfluous assignments (GH-7116) 2018-06-20 13:07:31 +08:00
poplib.py bpo-35128: Fix spacing issues in warning.warn() messages. (GH-10268) 2018-11-01 12:33:35 +02:00
posixpath.py bpo-35471: Remove the macpath module (GH-11129) 2018-12-14 13:37:26 +01:00
pprint.py bpo-35513, unittest: TextTestRunner uses time.perf_counter() (GH-11180) 2018-12-17 11:30:34 +01:00
profile.py bpo-32512: Add -m option to profile for profiling modules (#5132) 2018-11-05 22:03:46 +10:00
pstats.py Revert unneccessary changes made in bpo-30296 and apply other improvements. (GH-2624) 2018-02-26 16:50:11 +02:00
pty.py
py_compile.py bpo-34022: Stop forcing of hash-based invalidation with SOURCE_DATE_EPOCH (GH-9607) 2018-10-10 18:43:14 +02:00
pyclbr.py bpo-6691: Pyclbr now reports nested classes and functions. (#2503) 2017-07-03 21:31:25 -04:00
pydoc.py bpo-35619: Improve support of custom data descriptors in help() and pydoc. (GH-11366) 2019-01-15 10:53:18 +02:00
queue.py bpo-14976: Reentrant simple queue (#3346) 2018-01-16 00:27:16 +01:00
quopri.py
random.py bpo-35782: Fix error message in randrange (GH-11620) 2019-01-21 11:19:59 -08:00
re.py bpo-34681: Rename class Pattern in sre_parse to State. (GH-9310) 2018-09-18 09:16:26 +03:00
reprlib.py bpo-31370: Remove support for threads-less builds (#3385) 2017-09-07 18:56:24 +02:00
rlcompleter.py
runpy.py
sched.py bpo-31370: Remove support for threads-less builds (#3385) 2017-09-07 18:56:24 +02:00
secrets.py
selectors.py bpo-30624 remaining bare except (#2108) 2017-06-12 14:43:40 +02:00
shelve.py Fix misleading docsting of shelve.open(). (GH-6427) 2018-04-09 17:16:01 +03:00
shlex.py
shutil.py bpo-20849: add dirs_exist_ok arg to shutil.copytree (patch by Josh Bronson) 2018-12-28 19:03:40 +01:00
signal.py bpo-34282: Fix Enum._convert shadowing members named _convert (GH-8568) 2018-09-12 10:28:53 -07:00
site.py bpo-19891: Ignore error while writing history file (GH-8483) 2018-08-06 17:28:19 +09:00
smtpd.py
smtplib.py bpo-35128: Fix spacing issues in warning.warn() messages. (GH-10268) 2018-11-01 12:33:35 +02:00
sndhdr.py bpo-31985: Deprecate openfp in aifc, sunau, and wave (#4344) 2017-11-10 11:38:25 -05:00
socket.py bpo-33138: Change standard error message for non-pickleable and non-copyable types. (GH-6239) 2018-10-31 02:28:07 +02:00
socketserver.py Fix typo in socketserver docstring (GH-11252) 2018-12-21 14:22:09 -08:00
sre_compile.py Simplify flags checks in sre_compile.py. (GH-9718) 2018-10-05 20:53:45 +03:00
sre_constants.py bpo-31690: Allow the inline flags "a", "L", and "u" to be used as group flags for RE. (#3885) 2017-10-24 23:31:42 +03:00
sre_parse.py bpo-34681: Rename class Pattern in sre_parse to State. (GH-9310) 2018-09-18 09:16:26 +03:00
ssl.py bpo-33023: Fix NotImplemented to NotImplementedError. (GH-10934) 2018-12-06 22:36:55 +02:00
stat.py closes bpo-34353: Add sockets to stat.filemode fallback python implementation. (GH-8703) 2018-08-09 22:12:08 -07:00
statistics.py bpo-29919: Remove unused imports found by pyflakes (#137) 2017-03-27 16:05:26 +02:00
string.py bpo-31672: Restore the former behavior when override flags in Template. (#5099) 2018-01-04 19:20:11 +02:00
stringprep.py
struct.py
subprocess.py Revert "bpo-35537: subprocess can now use os.posix_spawnp (GH-11579)" (GH-11582) 2019-01-16 23:38:06 +01:00
sunau.py bpo-32056: Improve exceptions in aifc, wave and sunau. (GH-5951) 2018-03-18 09:55:53 +02:00
symbol.py bpo-30455: Generate all token related code and docs from Grammar/Tokens. (GH-10370) 2018-12-22 11:18:40 +02:00
symtable.py bpo-34983: Expose symtable.Symbol.is_nonlocal() in the symtable module (GH-9872) 2018-10-20 01:46:00 +01:00
sysconfig.py bpo-32430: Rename Modules/Setup.dist to Modules/Setup (GH-8229) 2018-07-16 19:03:03 +02:00
tabnanny.py bpo-30166: Import command-line parsing modules only when needed. (#1293) 2017-05-04 08:17:47 +03:00
tarfile.py bpo-34043: Optimize tarfile uncompress performance (GH-8089) 2018-07-06 14:06:00 +09:00
telnetlib.py bpo-30397: Add re.Pattern and re.Match. (#1646) 2017-10-04 20:09:49 +03:00
tempfile.py bpo-27300: Add the errors parameter to tempfile classes. (GH-6696) 2018-05-23 08:07:01 +03:00
textwrap.py bpo-30620: Remove dead lines from textwrap.dedent (GH-2064) 2017-06-15 19:19:43 -07:00
this.py
threading.py bpo-35283: Update the docstring of threading.Thread.join method (GH-11596) 2019-01-18 10:50:47 +01:00
timeit.py bpo-28240: timeit: Update repeat() doc (GH-7419) 2018-06-06 17:55:18 +02:00
token.py bpo-30455: Generate all token related code and docs from Grammar/Tokens. (GH-10370) 2018-12-22 11:18:40 +02:00
tokenize.py bpo-30455: Generate all token related code and docs from Grammar/Tokens. (GH-10370) 2018-12-22 11:18:40 +02:00
trace.py bpo-35202: Remove unused imports in Lib directory. (GH-10445) 2018-11-10 07:45:28 +02:00
traceback.py bpo-26502: Implement FrameSummary.__len__() (GH-8632) 2018-09-10 10:02:33 -07:00
tracemalloc.py bpo-32121: Add most_recent_first parameter to tracemalloc.Traceback.format (#4534) 2017-11-30 00:05:07 +01:00
tty.py
turtle.py bpo-35250: Correct argument name "num" -> "btn" in turtle docs. (GH-10565) 2018-11-16 15:28:51 +02:00
types.py bpo-32265: Classify class and static methods of builtin types. (#4776) 2017-12-15 14:13:41 +02:00
typing.py bpo-35341: Add generic version of OrderedDict to typing (GH-10850) 2018-12-02 15:53:14 +00:00
uu.py bpo-33687: Fix call to os.chmod() in uu.decode() (GH-7282) 2019-01-17 17:15:53 +03:00
uuid.py bpo-35701: Added __weakref__ slot to uuid.UUID (GH-11570) 2019-01-17 13:16:51 +01:00
warnings.py bpo-29564: warnings suggests to enable tracemalloc (GH-10486) 2018-11-13 02:41:00 +01:00
wave.py bpo-32056: Improve exceptions in aifc, wave and sunau. (GH-5951) 2018-03-18 09:55:53 +02:00
weakref.py bpo-30152: Reduce the number of imports for argparse. (#1269) 2017-09-26 00:55:55 +03:00
webbrowser.py bpo-35308: Fix regression where BROWSER env var is not respected. (GH-10693) 2018-11-26 23:29:45 +02:00
xdrlib.py
zipapp.py bpo-31638: Add compression support to zipapp (GH-3819) 2017-09-29 18:31:52 +01:00
zipfile.py bpo-32035: Fix words about strings and bytes in zipfile documentation. (GH-10592) 2018-11-25 09:51:14 +02:00
zipimport.py bpo-34726: Fix handling of hash-based pycs in zipimport. (GH-10327) 2018-11-07 20:34:59 +02:00