Commit graph

14964 commits

Author SHA1 Message Date
Irit Katriel
296cd128bf
Revert "gh-133395: add option for extension modules to specialize BINARY_OP/SUBSCR, apply to arrays (#133396)" (#133498) 2025-05-06 13:12:26 +03:00
Petr Viktorin
3c73cf51df
gh-132775: Revert "gh-132775: Add _PyCode_VerifyStateless() (gh-133221)" (#133497) 2025-05-06 13:09:41 +03:00
Emma Smith
c273f59fb3
gh-132983: Add the compression.zstd pacakge and tests (#133365)
Co-authored-by: Adam Turner <9087854+AA-Turner@users.noreply.github.com>
Co-authored-by: Gregory P. Smith <greg@krypto.org>
Co-authored-by: Tomas R. <tomas.roun8@gmail.com>
Co-authored-by: Rogdham <contact@rogdham.net>
2025-05-06 01:38:08 +01:00
Adam Turner
e6f8e0a035
GH-132983: Build `_zstd` on Windows (#133366) 2025-05-06 00:58:47 +01:00
Eric Snow
ea598730ef
gh-132775: Add _PyCode_GetXIData() (gh-133475) 2025-05-05 23:46:03 +00:00
Brandt Bucher
b1aa515bd6
GH-133231: Add JIT utilities in sys._jit (GH-133233) 2025-05-05 15:25:22 -07:00
Eric Snow
d270bb5792
gh-132775: Add _PyCode_VerifyStateless() (gh-133221)
"Stateless" code is a function or code object which does not rely on external state or internal state.
It may rely on arguments and builtins, but not globals or a closure. I've left a comment in
pycore_code.h that provides more detail.

We also add _PyFunction_VerifyStateless(). The new functions will be used in several later changes
that facilitate "sharing" functions and code objects between interpreters.
2025-05-05 21:48:58 +00:00
Eric Snow
24ebb9ccfd
gh-132775: Unrevert "Add _PyCode_GetVarCounts()" (gh-133265)
This reverts commit 811edcf (gh-133232), which itself reverted the original commit 811edcf (gh-133128).

We reverted the original change due to failing s390 builds (a big-endian architecture).
It ended up that I had not accommodated op caches.
2025-05-05 13:24:29 -06:00
Peter Bierma
b275b8f342
gh-133140: Add PyUnstable_Object_IsUniquelyReferenced for free-threading (#133144) 2025-05-05 21:01:20 +02:00
Irit Katriel
082dbf7788
gh-133395: add option for extension modules to specialize BINARY_OP/SUBSCR, apply to arrays (#133396) 2025-05-05 17:46:56 +01:00
Wulian233
b6c2ef0c7a
gh-131296: Avoid posixmodule.c warning (GH-133142)
Clang-cl detects that on 32-bit builds the variable is always smaller than the value. But since the same code is used for other architectures, we can't just _fix_ it. This cast avoids the tautological-constant-out-of-range-compare warning.
2025-05-05 17:45:15 +01:00
Stan Ulbrych
f5b784741d
gh-110067: Make max heap methods public and add missing ones (GH-130725)
Co-authored-by: Bénédikt Tran <10796600+picnixz@users.noreply.github.com>
Co-authored-by: Petr Viktorin <encukou@gmail.com>
2025-05-05 17:52:49 +02:00
Sergey B Kirpichev
1bc16504ef
gh-61103: drop unused Py_HAVE_C_COMPLEX define (GH-133435)
Py_HAVE_C_COMPLEX was added in 3.14 so the removal doesn't need a deprecation
period even under a strict reading of PEP 387.

The Py_FFI_SUPPORT_C_COMPLEX check configure check implies support for
complex types in ctypes.
2025-05-05 15:50:57 +02:00
Sergey B Kirpichev
8d0e07eb89
gh-61103: don't use C _Complex types to implement F/D/G in ctypes (GH-133237)
According to the C standard, the memory representation of _Complex types
is equivalent to 2-element arrays. Unlike _Complex, arrays are always available.

- drop _complex.h header
- use appropriate real arrays to replace complex types

Co-authored-by: Lisandro Dalcin <dalcinl@gmail.com>
Co-authored-by: Petr Viktorin <encukou@gmail.com>
Co-authored-by: Stan Ulbrych <89152624+StanFromIreland@users.noreply.github.com>
2025-05-05 11:23:30 +02:00
Adam Turner
3f80165a26
GH-91048: Minor fixes for `_remotedebugging & rename to _remote_debugging` (#133398) 2025-05-05 02:30:14 +02:00
Pablo Galindo Salgado
3109c47be8
gh-91048: Add filename and line number to external inspection routines (GH-133385)
Signed-off-by: Pablo Galindo <pablogsal@gmail.com>
2025-05-04 23:33:37 +02:00
Serhiy Storchaka
5a57248b22
gh-81793: Always call linkat() from os.link(), if available (GH-132517)
This fixes os.link() on platforms (like Linux and OpenIndiana) where the
system link() function does not follow symlinks.

* On Linux, it now follows symlinks by default and if
  follow_symlinks=True is specified.
* On Windows, it now raises error if follow_symlinks=True is passed.
* On macOS, it now raises error if follow_symlinks=False is passed and
  the system linkat() function is not available at runtime.
* On other platforms, it now raises error if follow_symlinks is passed
  with a value that does not match the system link() function behavior
  if if the behavior is not known.

Co-authored-by: Joachim Henke <37883863+jo-he@users.noreply.github.com>
Co-authored-by: Thomas Kluyver <takowl@gmail.com>
2025-05-04 17:24:10 +03:00
Serhiy Storchaka
e9253ebf74
gh-122559: Synchronize C and Python implementation of the io module about pickling (GH-122628)
In the C implementation, remove __reduce__ and __reduce_ex__ methods
that always raise TypeError and restore __getstate__ methods that always
raise TypeErrori.

This restores fine details of the pre-3.12 behavior and unifies
both implementations.
2025-05-04 14:06:49 +00:00
Michael Forney
0af61fe2f4
bpo-44172: Keep reference to original window in curses subwindow objects (GH-26226)
The X/Open curses specification[0] and ncurses documentation[1]
both state that subwindows must be deleted before the main window.

Deleting the windows in the wrong order causes a double-free with
NetBSD's curses implementation.

To fix this, keep track of the original window object in the subwindow
object, and keep a reference to the original for the lifetime of
the subwindow.

[0] https://pubs.opengroup.org/onlinepubs/7908799/xcurses/delwin.html
[1] https://invisible-island.net/ncurses/man/curs_window.3x.html

Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>
2025-05-04 13:29:44 +03:00
Emma Smith
3b4333583f
gh-132983: Introduce _zstd bindings module (GH-133027)
* Add _zstd module for https://peps.python.org/pep-0784/

This commit introduces the `_zstd` module, with bindings to libzstd from
the pyzstd project. It also includes the unix build system configuration.
Windows build system support will be integrated independently as it
depends on integration with cpython-source-deps.

* Add _zstd to modules

* Fix path for compression.zstd module

* Ignore _zstd module like _io

* Expand module state macros to improve code quality

Also removes module state references from the classes in the _zstd
module and instead uses PyType_GetModuleState()

* Remove backticks suggested in review

Co-authored-by: Stan Ulbrych <89152624+StanFromIreland@users.noreply.github.com>

* Use critical sections to lock object state

This should avoid races and deadlocks.

* Remove compress/decompress and mark module as not reliant on the GIL

The `compress`/`decompress` functions will be moved to Python code for simplicity.
C implementations can always be re-added in the future.

Also, mark _zstd as not requiring the GIL.

* Lift critical section to avoid clang warning

* Respond to comments by picnixz

* Call out pyzstd explicitly in license description

Co-authored-by: Adam Turner <9087854+AA-Turner@users.noreply.github.com>

* Use a much more robust implementation...

... for `get_zstd_state_from_type`

Co-authored-by: Bénédikt Tran <10796600+picnixz@users.noreply.github.com>

* Use PyList_GetItemRef for thread safety purposes

* Use a macro for the minimum supported version

* remove const from primivite types

* Use PyMem_New in another spot

* Simplify error handling in _get_frame_size

* Another simplification of error handling in get_frame_info

* Rename _module_state to mod_state

* Rewrite comment explaining the context of the code

* Add link to pyzstd

* Add TODO about refactoring dict training code

* Use PyModule_AddObjectRef over PyModule_AddObject

PyModule_AddObject is soft-deprecated, so we should use PyModule_AddObjectRef

* Check result of OutputBufferGrow

* Simplify return logic in `add_constant_to_type`

Co-authored-by: Bénédikt Tran <10796600+picnixz@users.noreply.github.com>

* Ignore return value of _zstd_clear()

Co-authored-by: Bénédikt Tran <10796600+picnixz@users.noreply.github.com>

* Remove redundant comments

* Remove __reduce__ from ZstdDict

We should instead document that to pickle a dictionary a user should use
the `.dict_content` attribute.

* Use PyUnicode_FromFormat instead of a buffer

* Don't use C constants/types in error messages

* Make error messages easier to understand for Python users

* Lower minimum required version 1.4.0

* Use casts and make slot function signatures correct

* Be consistent with CPython on const usage

* Make else clauses in line with PEP 7

* Fix over-indented blocks in argument clinic

* Add critical section around ZSTD_DCtx_setParameter

* Add a TODO about refactoring critical sections

* Use Py_UNREACHABLE

* Move bytes operations out of Py_BEGIN_ALLOW_THREADS

* Add TODO about ensuring a lock is held

* Remove asserts that may not be correct

* Add TODO to make ZstdDict and others GC objects

* Make objects GC tracked

* Remove unused include

* Fix some memory issues

* Fix refleaks on module and in ZstdDict

* Update configure to check for ZDICT_finalizeDictionary

* Properly check version in configure

* exit(1) if check fails

* Use AC_RUN_IFELSE

* Use a define() to re-use version check

* Actually properly set _zstd module status based on version

---------

Co-authored-by: Stan Ulbrych <89152624+StanFromIreland@users.noreply.github.com>
Co-authored-by: Adam Turner <9087854+AA-Turner@users.noreply.github.com>
Co-authored-by: Bénédikt Tran <10796600+picnixz@users.noreply.github.com>
2025-05-04 01:29:55 +00:00
Pablo Galindo Salgado
2bc8365231
GH-91048: Add utils for printing the call stack for asyncio tasks (#133284) 2025-05-04 00:51:57 +00:00
Serhiy Storchaka
7363e8d24d
gh-133139: Add curses.assume_default_colors() (GH-133145)
This is a refinement of the curses.use_default_colors() function which
allows to change the color pair 0.
2025-05-03 23:33:22 +03:00
Sergey Miryanov
bd2ed7c7ce
gh-91048: Chain some exceptions in _testexternalinspection.c (#132970) 2025-05-03 01:35:30 +02:00
Sergey Miryanov
a0bc0c462f
gh-100926: Move ctype's pointers cache from _pointer_type_cache to StgInfo (GH-131282)
Deprecate _pointer_type_cache and calling POINTER on a string.

Co-authored-by: neonene <53406459+neonene@users.noreply.github.com>
Co-authored-by: Jun Komoda <45822440+junkmd@users.noreply.github.com>
Co-authored-by: Petr Viktorin <encukou@gmail.com>
2025-05-02 19:06:37 +02:00
Sergey B Kirpichev
f425509349
gh-121249: unconditionally support complex types in struct (GH-132864)
Co-authored-by: Lisandro Dalcin <dalcinl@gmail.com>
Co-authored-by: Bénédikt Tran <10796600+picnixz@users.noreply.github.com>
2025-05-02 18:24:52 +02:00
Sam Gross
f2379535fe
gh-133164: Add PyUnstable_Object_IsUniqueReferencedTemporary C API (gh-133170)
After gh-130704, the interpreter replaces some uses of `LOAD_FAST` with
`LOAD_FAST_BORROW` which avoid incref/decrefs by "borrowing" references
on the interpreter stack when the bytecode compiler can determine that
it's safe.

This change broke some checks in C API extensions that relied on
`Py_REFCNT()` of `1` to determine if it's safe to modify an object
in-place. Objects may have a reference count of one, but still be
referenced further up the interpreter stack due to borrowing of
references.

This provides a replacement function for those checks.
`PyUnstable_Object_IsUniqueReferencedTemporary` is more conservative:
it checks that the object has a reference count of one and that it exists as a
unique strong reference in the interpreter's stack of temporary
variables in the top most frame.

See also:

* https://github.com/numpy/numpy/issues/28681

Co-authored-by: Pieter Eendebak <pieter.eendebak@gmail.com>
Co-authored-by: T. Wouters <thomas@python.org>
Co-authored-by: mpage <mpage@cs.stanford.edu>
Co-authored-by: Mark Shannon <mark@hotpy.org>
Co-authored-by: Victor Stinner <vstinner@python.org>
2025-05-02 13:24:57 +00:00
Petr Viktorin
2590774c9b
gh-133290: Use PyObject_SetAttr to set _type_ (GH-133292) 2025-05-02 14:47:07 +02:00
Victor Stinner
e26bafd107
gh-133256: Add _Py_NONSTRING macro (#133257)
Fix GCC 15 compiler warnings such as:

    Modules/fcntlmodule.c:27:36: warning: initializer-string for
    array of 'char' truncates NUL terminator but destination lacks
    'nonstring' attribute (9 chars into 8 available)
    [-Wunterminated-string-initialization]
    static const char guard[GUARDSZ] = "\x00\xfa\x69\xc4\x67\xa3\x6c\x58";
2025-05-01 17:55:49 +02:00
Sergey B Kirpichev
ad2f0884b1
gh-130317: Fix test_pack_unpack_roundtrip() and add docs (#133204)
* Skip sNaN's testing in 32-bit mode.
* Drop float_set_snan() helper.
* Use memcpy() workaround for sNaN's in PyFloat_Unpack4().
* Document, that sNaN's may not be preserved by PyFloat_Pack/Unpack API.
2025-05-01 16:20:36 +02:00
neonene
fa52f289a3
gh-133166: Fix missing error emission of PyType_GetModuleByDef (GH-133240) 2025-05-01 14:32:57 +02:00
Nybblista
b99d970bcd
Add missing whitespace in is_running_main comment (#133174) 2025-05-01 04:36:59 +01:00
Russell Keith-Magee
ecbc16a915
Bump the HACL* revision. (#133226)
Bump the HACL* revision to include iOS platform identification.
2025-05-01 10:57:49 +08:00
Eric Snow
811edcf9cd
Revert "gh-132775: Add _PyCode_GetVarCounts() (gh-133128)" (gh-133232)
The change broke the s390 builds, so I'm reverting it while I investigate.

This reverts commit 94b4fcd806.
2025-05-01 02:35:20 +00:00
Eric Snow
cb35c11d82
gh-132775: Add _PyPickle_GetXIData() (gh-133107)
There's some extra complexity due to making sure we we get things right when handling functions and classes defined in the __main__ module.  This is also reflected in the tests, including the addition of extra functions in test.support.import_helper.
2025-04-30 17:34:05 -06:00
Eric Snow
94b4fcd806
gh-132775: Add _PyCode_GetVarCounts() (gh-133128)
This helper is useful in a variety of ways, including in demonstrating how the different counts relate to one another.

It will be used in a later change to help identify if a function is "stateless", meaning it doesn't have any free vars or globals.

Note that a majority of this change is tests.
2025-04-30 18:19:20 +00:00
Mark Shannon
44e4c479fb
GH-124715: Move trashcan mechanism into Py_Dealloc (GH-132280) 2025-04-30 11:37:53 +01:00
Inada Naoki
4e294f6feb
gh-133036: Deprecate codecs.open (#133038)
Co-authored-by: Hugo van Kemenade <1324225+hugovk@users.noreply.github.com>
Co-authored-by: Victor Stinner <vstinner@python.org>
2025-04-30 10:11:09 +09:00
Serhiy Storchaka
4265854d96
gh-132987: Support __index__() in the socket module (GH-133093)
ntohl(), htonl(), if_indextoname(), getaddrinfo() now use __index__() if
available.

Also fix the Argument Clinic names for module-level functions (although
this does not affect the user).
2025-04-29 19:27:07 +03:00
Serhiy Storchaka
c33efa8735
gh-132987: Support __index__() in the stat module (GH-133097)
Use it for the mode arguments in filemode(), S_IMODE(), S_ISDIR(), etc.
2025-04-29 19:25:44 +03:00
Victor Stinner
02cd6d7097
gh-130317: Fix strict aliasing in PyFloat_Pack8() (#133150)
* Fix strict aliasing in PyFloat_Pack8() and PyFloat_Pack4().
* Fix _testcapi.float_set_snan() on x86 (32-bit).
2025-04-29 14:27:50 +00:00
Serhiy Storchaka
a64fdc7513
gh-132987: Support __index__() in the lzma module (GH-133099) 2025-04-29 14:14:33 +00:00
Serhiy Storchaka
d2d490064d
gh-132987: Support __index__() in the select.kqueue_event constructor (GH-133094) 2025-04-29 13:55:14 +00:00
Serhiy Storchaka
0fb4c38c27
gh-132987: Support __index__() in the ssl.SSLContext.options setter (GH-133098) 2025-04-29 16:28:47 +03:00
Serhiy Storchaka
d6da6803a1
gh-132987: Support __index__() in hashlib.scrypt() (GH-133100)
Even if such signature is not supported by PyArg_ParseTupleAndKeywords(),
Argument Clinic supports it with inlined converters.
2025-04-29 16:16:05 +03:00
Serhiy Storchaka
07edc0d2b2
gh-132987: Support __index__() in the posix module (GH-133096)
Support it for the dev_t values in mknod(), major(), minor() and makedev(),
CPU numbers in sched_setaffinity(), group numbers in setgroups(),
configuration name in fpathconf(), pathconf(), confstr(), and sysconf().
2025-04-29 16:14:47 +03:00
Eric Snow
219d8d24b5
gh-87859: Track Code Object Local Kinds For Arguments (gh-132980)
Doing this was always the intention. I was finally motivated to find the time to do it.

See #87859 (comment).
2025-04-29 02:21:47 +00:00
Eric Snow
96a7fb93a8
gh-132775: Add _PyCode_ReturnsOnlyNone() (gh-132981)
The function indicates whether or not the function has a return statement.

This is used by a later change related treating some functions like scripts.
2025-04-28 20:12:52 -06:00
Eric Snow
bdd23c0bb9
gh-132775: Add _PyMarshal_GetXIData() (gh-133108)
Note that the bulk of this change is tests.
2025-04-28 17:23:46 -06:00
Neil Schemenauer
22f0730d40
gh-122320: Limit dict key versions used by test_opcache. (gh-132961)
The `test_load_global_module()` test consumes a lot of dict key versions.
Skip the test if we have consumed half of the available versions that can be
used for the "load global" cache.
2025-04-28 12:54:55 -07:00
Eric Snow
6f04325992
gh-132775: Cleanup Related to crossinterp.c Before Further Changes (gh-132974)
This change consists of adding tests and moving code around, with some renaming thrown in.
2025-04-28 11:55:15 -06:00