Stowage/cpython
2
0
Fork 0
mirror of https://github.com/python/cpython.git synced 2026-03-02 19:10:46 +00:00
Code Issues Projects Releases Packages Wiki Activity
cpython/Modules
History
Emma Smith 3b4333583f
gh-132983: Introduce _zstd bindings module (GH-133027) 
* Add _zstd module for https://peps.python.org/pep-0784/

This commit introduces the `_zstd` module, with bindings to libzstd from
the pyzstd project. It also includes the unix build system configuration.
Windows build system support will be integrated independently as it
depends on integration with cpython-source-deps.

* Add _zstd to modules

* Fix path for compression.zstd module

* Ignore _zstd module like _io

* Expand module state macros to improve code quality

Also removes module state references from the classes in the _zstd
module and instead uses PyType_GetModuleState()

* Remove backticks suggested in review

Co-authored-by: Stan Ulbrych <89152624+StanFromIreland@users.noreply.github.com>

* Use critical sections to lock object state

This should avoid races and deadlocks.

* Remove compress/decompress and mark module as not reliant on the GIL

The `compress`/`decompress` functions will be moved to Python code for simplicity.
C implementations can always be re-added in the future.

Also, mark _zstd as not requiring the GIL.

* Lift critical section to avoid clang warning

* Respond to comments by picnixz

* Call out pyzstd explicitly in license description

Co-authored-by: Adam Turner <9087854+AA-Turner@users.noreply.github.com>

* Use a much more robust implementation...

... for `get_zstd_state_from_type`

Co-authored-by: Bénédikt Tran <10796600+picnixz@users.noreply.github.com>

* Use PyList_GetItemRef for thread safety purposes

* Use a macro for the minimum supported version

* remove const from primivite types

* Use PyMem_New in another spot

* Simplify error handling in _get_frame_size

* Another simplification of error handling in get_frame_info

* Rename _module_state to mod_state

* Rewrite comment explaining the context of the code

* Add link to pyzstd

* Add TODO about refactoring dict training code

* Use PyModule_AddObjectRef over PyModule_AddObject

PyModule_AddObject is soft-deprecated, so we should use PyModule_AddObjectRef

* Check result of OutputBufferGrow

* Simplify return logic in `add_constant_to_type`

Co-authored-by: Bénédikt Tran <10796600+picnixz@users.noreply.github.com>

* Ignore return value of _zstd_clear()

Co-authored-by: Bénédikt Tran <10796600+picnixz@users.noreply.github.com>

* Remove redundant comments

* Remove __reduce__ from ZstdDict

We should instead document that to pickle a dictionary a user should use
the `.dict_content` attribute.

* Use PyUnicode_FromFormat instead of a buffer

* Don't use C constants/types in error messages

* Make error messages easier to understand for Python users

* Lower minimum required version 1.4.0

* Use casts and make slot function signatures correct

* Be consistent with CPython on const usage

* Make else clauses in line with PEP 7

* Fix over-indented blocks in argument clinic

* Add critical section around ZSTD_DCtx_setParameter

* Add a TODO about refactoring critical sections

* Use Py_UNREACHABLE

* Move bytes operations out of Py_BEGIN_ALLOW_THREADS

* Add TODO about ensuring a lock is held

* Remove asserts that may not be correct

* Add TODO to make ZstdDict and others GC objects

* Make objects GC tracked

* Remove unused include

* Fix some memory issues

* Fix refleaks on module and in ZstdDict

* Update configure to check for ZDICT_finalizeDictionary

* Properly check version in configure

* exit(1) if check fails

* Use AC_RUN_IFELSE

* Use a define() to re-use version check

* Actually properly set _zstd module status based on version

---------

Co-authored-by: Stan Ulbrych <89152624+StanFromIreland@users.noreply.github.com>
Co-authored-by: Adam Turner <9087854+AA-Turner@users.noreply.github.com>
Co-authored-by: Bénédikt Tran <10796600+picnixz@users.noreply.github.com>
2025-05-04 01:29:55 +00:00
..
_ctypes gh-100926: Move ctype's pointers cache from _pointer_type_cache to StgInfo (GH-131282) 2025-05-02 19:06:37 +02:00
_decimal gh-53032: support IEEE 754 contexts in the decimal module (#122003) 2025-04-28 15:05:56 +02:00
_hacl Bump the HACL* revision. (#133226) 2025-05-01 10:57:49 +08:00
_io gh-133036: Deprecate codecs.open (#133038) 2025-04-30 10:11:09 +09:00
_multiprocessing gh-131525: Cache the result of tuple_hash (#131529) 2025-03-27 09:57:06 -04:00
_sqlite gh-131525: Cache the result of tuple_hash (#131529) 2025-03-27 09:57:06 -04:00
_sre GH-131296: Disable MSVC-specific optimization in sre.c when compiling with clang-cl (GH-131593) 2025-04-01 17:29:45 +01:00
_ssl gh-131423: Update to OpenSSL 3.0.16. (GH-131839) 2025-03-28 15:07:57 +00:00
_testcapi gh-133164: Add PyUnstable_Object_IsUniqueReferencedTemporary C API (gh-133170) 2025-05-02 13:24:57 +00:00
_testinternalcapi gh-128813: hide mixed-mode functions for complex arithmetic from C-API (#131703) 2025-04-22 14:18:18 +02:00
_testlimitedcapi gh-111178: Fix function signatures for test_capi (#131659) 2025-03-24 14:04:45 +00:00
_xxtestfuzz gh-111178: remove redundant casts for functions with correct signatures (#131673) 2025-04-01 17:18:11 +02:00
_zstd gh-132983: Introduce _zstd bindings module (GH-133027) 2025-05-04 01:29:55 +00:00
cjkcodecs gh-111178: remove redundant casts for functions with correct signatures (#131673) 2025-04-01 17:18:11 +02:00
clinic gh-133139: Add curses.assume_default_colors() (GH-133145) 2025-05-03 23:33:22 +03:00
expat gh-131809: Upgrade vendored expat to 2.7.1 (#132192) 2025-04-07 23:42:00 +03:00
_abc.c gh-111178: fix UBSan failures in Modules/_abc.c (GH-128253) 2025-01-07 12:45:29 +01:00
_asynciomodule.c gh-91048: Prevent optimizing away the asyncio debug offsets structure on Windows (#132963) 2025-04-25 17:43:32 +00:00
_bisectmodule.c gh-126035: add missing whitespace to *Py_EnterRecursiveCall() messages (#126036) 2024-10-27 22:55:48 +01:00
_bz2module.c gh-111178: fix UBSan failures in Modules/_bz2module.c (GH-128238) 2025-01-10 11:47:19 +01:00
_codecsmodule.c GH-131238: Core header refactor (GH-131250) 2025-03-17 09:19:04 +00:00
_collectionsmodule.c gh-111178: Change Argument Clinic signature for METH_O (#130682) 2025-03-11 16:33:36 +01:00
_complex.h gh-61103: Support float and long double complex types in ctypes module (#121248) 2024-07-03 11:08:11 +02:00
_csv.c gh-129409: Fix Integer overflow - SEGV while writing data more than 2GB in CSV file (#129413) 2025-01-29 11:15:47 +00:00
_curses_panel.c gh-111178: fix UBSan failures in Modules/curses*.c (GH-128244) 2025-01-03 15:12:40 +01:00
_cursesmodule.c gh-133139: Add curses.assume_default_colors() (GH-133145) 2025-05-03 23:33:22 +03:00
_datetimemodule.c gh-70647: Raise a more informative error for when date is out of range (GH-131335) 2025-03-20 20:47:09 -07:00
_dbmmodule.c gh-111178: fix UBSan failures in Modules/_dbmmodule.c (#129775) 2025-02-10 11:02:47 +01:00
_elementtree.c GH-124715: Move trashcan mechanism into Py_Dealloc (GH-132280) 2025-04-30 11:37:53 +01:00
_functoolsmodule.c gh-131757: allow lru_cache functions to execute concurrently (#131758) 2025-04-14 18:31:19 +02:00
_gdbmmodule.c gh-111178: fix UBSan failures for gdbmobject (GH-128178) 2025-01-10 11:52:36 +01:00
_hashopenssl.c gh-132987: Support __index__() in hashlib.scrypt() (GH-133100) 2025-04-29 16:16:05 +03:00
_heapqmodule.c Remove almost all unpaired backticks in docstrings (#119231) 2024-05-22 12:35:18 -04:00
_interpchannelsmodule.c gh-132775: Cleanup Related to crossinterp.c Before Further Changes (gh-132974) 2025-04-28 11:55:15 -06:00
_interpqueuesmodule.c gh-132775: Cleanup Related to crossinterp.c Before Further Changes (gh-132974) 2025-04-28 11:55:15 -06:00
_interpreters_common.h gh-132781: Cleanup Code Related to NotShareableError (gh-132782) 2025-04-25 14:43:38 -06:00
_interpretersmodule.c Add missing whitespace in is_running_main comment (#133174) 2025-05-01 04:36:59 +01:00
_json.c gh-112068: C API: Add support of nullable arguments in PyArg_Parse (GH-121303) 2025-04-08 22:08:00 +03:00
_localemodule.c Fix a typo in _localemodule.c (GH-130085) 2025-02-13 17:29:26 +00:00
_lsprof.c gh-132250: Clear error in lsprof callback when method descriptor raises an excep… (#132251) 2025-04-08 13:36:47 -04:00
_lzmamodule.c gh-132987: Support __index__() in the lzma module (GH-133099) 2025-04-29 14:14:33 +00:00
_math.h gh-122681: remove m_atan2()/c_atan2() helpers (#122715) 2024-08-17 13:48:16 +05:30
_opcode.c gh-100239: replace BINARY_SUBSCR & family by BINARY_OP with oparg NB_SUBSCR (#129700) 2025-02-07 22:39:54 +00:00
_operator.c gh-131238: Add explicit includes to pycore headers (#131257) 2025-03-17 12:32:43 +01:00
_pickle.c gh-131238: Remove more includes from pycore_interp.h (#131480) 2025-03-19 23:01:32 +01:00
_posixsubprocess.c GH-115322: Add missing audit hooks (GH-115624) 2025-04-13 21:46:20 +00:00
_queuemodule.c gh-111178: Fix function signatures for test_os (#131227) 2025-03-14 13:53:06 +00:00
_randommodule.c gh-111178: fix UBSan failures in Modules/_randommodule.c (GH-129791) 2025-02-21 14:59:04 +00:00
_remotedebuggingmodule.c GH-91048: Add utils for printing the call stack for asyncio tasks (#133284) 2025-05-04 00:51:57 +00:00
_scproxy.c gh-116322: Add Py_mod_gil module slot (#116882) 2024-05-03 11:30:55 -04:00
_ssl.c gh-132987: Support __index__() in the ssl.SSLContext.options setter (GH-133098) 2025-04-29 16:28:47 +03:00
_ssl.h GH-103092: isolate _ssl (#104725) 2023-05-22 06:14:48 +05:30
_ssl_data_34.h gh-131423: Update OpenSSL data to 3.4.1 on Linux (#131618) 2025-04-25 10:26:58 +02:00
_ssl_data_111.h gh-131423: Update OpenSSL data to 3.4.1 on Linux (#131618) 2025-04-25 10:26:58 +02:00
_ssl_data_300.h gh-127330: Update for OpenSSL 3.4 & document+improve the update process (GH-127331) 2024-11-28 13:29:27 +01:00
_ssl_data_340.h gh-131423: Update OpenSSL data to 3.4.1 on Linux (#131618) 2025-04-25 10:26:58 +02:00
_stat.c gh-132987: Support __index__() in the stat module (GH-133097) 2025-04-29 19:25:44 +03:00
_statisticsmodule.c gh-116322: Add Py_mod_gil module slot (#116882) 2024-05-03 11:30:55 -04:00
_struct.c gh-121249: unconditionally support complex types in struct (GH-132864) 2025-05-02 18:24:52 +02:00
_suggestions.c gh-131936: Strengthen check in _suggestions._generate_suggestions (#131945) 2025-03-31 19:30:29 +02:00
_sysconfig.c gh-127405: Add ABIFLAGS to sysconfig variables on Windows (GH-131799) 2025-04-11 16:19:03 +01:00
_testbuffer.c gh-111178: fix UBSan failures for Modules/_testbuffer.c (#131612) 2025-03-24 10:46:25 +01:00
_testcapi_feature_macros.inc gh-91325: Skip Stable ABI checks with Py_TRACE_REFS special build (GH-92046) 2024-01-29 16:45:31 +01:00
_testcapimodule.c gh-111178: fix UBSan failures for Modules/_testcapimodule.c (#131614) 2025-03-24 10:14:22 +00:00
_testclinic.c gh-132753: Argument Clinic: Fix support of c_default for the bool converter (GH-132754) 2025-04-20 23:21:29 +00:00
_testclinic_limited.c gh-116322: Rename PyModule_ExperimentalSetGIL to PyUnstable_Module_SetGIL (GH-118645) 2024-05-06 18:59:36 +02:00
_testimportmultiple.c gh-116322: Add Py_mod_gil module slot (#116882) 2024-05-03 11:30:55 -04:00
_testinternalcapi.c Revert "gh-132775: Add _PyCode_GetVarCounts() (gh-133128)" (gh-133232) 2025-05-01 02:35:20 +00:00
_testlimitedcapi.c gh-111495: Add PyFile tests (#129449) 2025-01-30 18:05:32 +01:00
_testmultiphase.c gh-111178: fix UBSan failures for Modules/_testmultiphase.c (#131615) 2025-03-24 10:59:30 +01:00
_testsinglephase.c gh-123880: Allow recursive import of single-phase-init modules (GH-123950) 2024-09-20 10:27:34 +02:00
_threadmodule.c gh-87135: Raise PythonFinalizationError when joining a blocked daemon thread (gh-130402) 2025-04-28 15:48:48 +02:00
_tkinter.c GH-131238: Core header refactor (GH-131250) 2025-03-17 09:19:04 +00:00
_tracemalloc.c gh-128679: Redesign tracemalloc locking (#128888) 2025-01-15 20:22:44 +00:00
_typesmodule.c GH-118761: Expose more core interpreter types in `_types` (#132103) 2025-04-05 18:11:07 +00:00
_typingmodule.c gh-105499: Merge typing.Union and types.UnionType (#105511) 2025-03-04 11:44:19 -08:00
_uuidmodule.c gh-116322: Add Py_mod_gil module slot (#116882) 2024-05-03 11:30:55 -04:00
_weakref.c gh-120974: Make _asyncio._leave_task atomic in the free-threaded build (#122139) 2024-07-23 17:06:03 +00:00
_winapi.c GH-131296: Fix incompatible pointer type warning in _winapi.c (GH-131600) 2025-03-31 17:33:55 +01:00
_zoneinfo.c gh-111178: remove redundant casts for functions with correct signatures (#131673) 2025-04-01 17:18:11 +02:00
addrinfo.h gh-114917: add support for AI_NUMERICSERV in getaddrinfo emulation (#114918) 2025-03-18 11:26:51 +01:00
arraymodule.c gh-132527: Added missing w typecode to array() error message (#132529) 2025-04-25 16:31:05 +03:00
atexitmodule.c gh-111178: remove redundant casts for functions with correct signatures (#131673) 2025-04-01 17:18:11 +02:00
binascii.c gh-118314: Fix padding edge case in binascii.a2b_base64 strict mode (GH-118320) 2024-05-07 11:18:45 +02:00
blake2module.c gh-132993: expose HASHLIB_GIL_MINSIZE to private extension modules (#132999) 2025-04-27 22:20:15 +00:00
cmathmodule.c gh-120026: soft deprecate Py_HUGE_VAL macro (#120027) 2024-11-01 22:04:31 +00:00
config.c.in gh-128384: Use a context variable for warnings.catch_warnings (gh-130010) 2025-04-09 16:18:54 -07:00
errnomodule.c gh-126585: Add EHWPOISON error code (#126586) 2024-12-03 12:45:50 +00:00
faulthandler.c gh-127604: Fix refleak in faulthandler_dump_c_stack_py() (#132840) 2025-04-23 14:29:31 +00:00
fcntlmodule.c gh-133256: Add _Py_NONSTRING macro (#133257) 2025-05-01 17:55:49 +02:00
gc_weakref.txt Fix links to old SF bugs (#95648) 2022-08-04 18:12:35 +02:00
gcmodule.c gh-122943: Rework support of var-positional parameter in Argument Clinic (GH-122945) 2024-11-07 23:40:03 +02:00
getaddrinfo.c gh-114917: Fix typo in getaddrinfo emulation (#131413) 2025-03-18 14:31:13 +01:00
getbuildinfo.c gh-100388: Change undefined __DATE__ to the Unix epoch (#100389) 2025-03-03 14:59:46 +01:00
getnameinfo.c Replace K&R function declarations with ANSI declarations (#114818) 2025-03-11 13:06:36 +01:00
getpath.c GH-131296: Avoid invalid signed char comparison in getpath.c assertion (GH-131594) 2025-04-22 14:06:30 +01:00
getpath.py GH-128469: Revert "warn when libpython was loaded from outside the build directory (#128645)" (#129506) 2025-01-31 11:17:37 +00:00
getpath_noop.c bpo-45582: Port getpath[p].c to Python (GH-29041) 2021-12-03 00:08:42 +00:00
grpmodule.c gh-126316: Make grp.getgrall() thread-safe: add a mutex (#127055) 2024-11-21 15:47:24 +01:00
hashlib.h gh-111863: Rename Py_NOGIL to Py_GIL_DISABLED (#111864) 2023-11-20 15:52:00 +02:00
hmacmodule.c gh-133061: do not mention UINT32_MAX in HMAC user-facing messages (#133062) 2025-04-28 01:14:12 +02:00
itertoolsmodule.c gh-123471: Make concurrent iteration over itertools.repeat safe under free-threading (#131247) 2025-04-13 13:26:58 +05:30
ld_so_aix.in Issue #10656: Fix out-of-tree building on AIX 2016-11-20 07:56:37 +00:00
main.c gh-103997: Automatically dedent the argument to "-c" (#103998) 2025-04-18 17:39:30 +09:00
makesetup gh-131298: eliminate HACL* static libraries for cryptographic modules (GH-132438) 2025-04-20 17:40:17 +00:00
makexp_aix bpo-42087: Remove support for AIX 5.3 and below (GH-22830) 2020-11-16 16:16:10 +01:00
mathmodule.c gh-101410: Customize error messages for 1-arg math functions (#129497) 2025-04-22 11:29:20 +02:00
md5module.c gh-132993: expose HASHLIB_GIL_MINSIZE to private extension modules (#132999) 2025-04-27 22:20:15 +00:00
mmapmodule.c gh-112068: C API: Add support of nullable arguments in PyArg_Parse (GH-121303) 2025-04-08 22:08:00 +03:00
overlapped.c GH-131296: Remove unused variable from overlapped.c (GH-131590) 2025-04-16 22:44:57 +01:00
posixmodule.c gh-132987: Support __index__() in the posix module (GH-133096) 2025-04-29 16:14:47 +03:00
posixmodule.h gh-85283: Convert grp extension to the limited C API (#116611) 2024-03-12 00:46:53 +00:00
pwdmodule.c gh-116322: Add Py_mod_gil module slot (#116882) 2024-05-03 11:30:55 -04:00
pyexpat.c gh-111178: Fix function signature in pyexpat.c (#131674) 2025-03-24 17:22:45 +00:00
readline.c GH-115322: Add missing audit hooks (GH-115624) 2025-04-13 21:46:20 +00:00
README
resource.c gh-116322: Add Py_mod_gil module slot (#116882) 2024-05-03 11:30:55 -04:00
rotatingtree.c gh-116181: Remove Py_BUILD_CORE_BUILTIN and Py_BUILD_CORE_MODULE in rotatingtree.c (#121260) 2024-07-03 13:05:05 +05:30
rotatingtree.h bpo-32150: Expand tabs to spaces in C files. (#4583) 2017-11-28 17:56:10 +02:00
selectmodule.c gh-132987: Support __index__() in the select.kqueue_event constructor (GH-133094) 2025-04-29 13:55:14 +00:00
Setup gh-132983: Introduce _zstd bindings module (GH-133027) 2025-05-04 01:29:55 +00:00
Setup.bootstrap.in gh-109599: Expose CapsuleType via the _types module (#131969) 2025-04-04 23:37:41 +01:00
Setup.stdlib.in gh-132983: Introduce _zstd bindings module (GH-133027) 2025-05-04 01:29:55 +00:00
sha1module.c gh-132993: expose HASHLIB_GIL_MINSIZE to private extension modules (#132999) 2025-04-27 22:20:15 +00:00
sha2module.c gh-132993: expose HASHLIB_GIL_MINSIZE to private extension modules (#132999) 2025-04-27 22:20:15 +00:00
sha3module.c gh-132993: expose HASHLIB_GIL_MINSIZE to private extension modules (#132999) 2025-04-27 22:20:15 +00:00
signalmodule.c gh-131591: Check for remote debug in PyErr_CheckSignals (#132853) 2025-04-23 20:59:41 +01:00
socketmodule.c gh-132987: Support __index__() in the socket module (GH-133093) 2025-04-29 19:27:07 +03:00
socketmodule.h gh-132429: Fix support of Bluetooth sockets on NetBSD and DragonFly BSD (GH-132431) 2025-04-12 23:57:34 +03:00
symtablemodule.c gh-119933: Improve `SyntaxError` message for invalid type parameters expressions (#119976) 2024-06-17 06:51:03 -07:00
syslogmodule.c gh-130163: Fix crashes related to PySys_GetObject() (GH-130503) 2025-02-25 23:04:27 +02:00
termios.c gh-119770: Make termios ioctl() constants positive (#119840) 2024-05-31 17:18:40 +02:00
timemodule.c gh-130617 : fix time_clockid_converter on DragonFlyBSD (#130634) 2025-02-28 08:56:00 +01:00
tkappinit.c gh-103538: Remove unused TK_AQUA code (GH-103539) 2023-05-10 18:53:13 +00:00
tkinter.h gh-103532: Remove TKINTER_PROTECT_LOADTK code (GH-103535) 2023-04-14 09:04:16 -05:00
unicodedata.c gh-130790: Remove references about unicode's readiness from comments (#130801) 2025-03-03 19:18:09 +00:00
unicodedata_db.h closes gh-124016: update Unicode to 16.0.0 (#124017) 2024-09-13 07:47:04 -07:00
unicodename_db.h closes gh-124016: update Unicode to 16.0.0 (#124017) 2024-09-13 07:47:04 -07:00
winreparse.h bpo-31512: Add non-elevated symlink support for Windows (GH-3652) 2019-04-09 11:19:46 -07:00
xxlimited.c gh-111178: fix UBSan failures in Modules/xx*.c (GH-129797) 2025-02-25 13:02:32 +01:00
xxlimited_35.c gh-111178: Fix function signatures for multiple tests (#131496) 2025-03-20 12:27:03 +01:00
xxmodule.c gh-122040: reword Modules/xxmodule.c module-level comment (#132201) 2025-04-07 08:03:03 +00:00
xxsubtype.c gh-111178: fix UBSan failures in Modules/xx*.c (GH-129797) 2025-02-25 13:02:32 +01:00
zlibmodule.c gh-111178: Fix function signatures for multiple tests (#131496) 2025-03-20 12:27:03 +01:00

README 

Source files for standard library extension modules,
and former extension modules that are now builtin modules.