cpython/Modules/Setup
Emma Smith 3b4333583f
gh-132983: Introduce _zstd bindings module (GH-133027)
* Add _zstd module for https://peps.python.org/pep-0784/

This commit introduces the `_zstd` module, with bindings to libzstd from
the pyzstd project. It also includes the unix build system configuration.
Windows build system support will be integrated independently as it
depends on integration with cpython-source-deps.

* Add _zstd to modules

* Fix path for compression.zstd module

* Ignore _zstd module like _io

* Expand module state macros to improve code quality

Also removes module state references from the classes in the _zstd
module and instead uses PyType_GetModuleState()

* Remove backticks suggested in review

Co-authored-by: Stan Ulbrych <89152624+StanFromIreland@users.noreply.github.com>

* Use critical sections to lock object state

This should avoid races and deadlocks.

* Remove compress/decompress and mark module as not reliant on the GIL

The `compress`/`decompress` functions will be moved to Python code for simplicity.
C implementations can always be re-added in the future.

Also, mark _zstd as not requiring the GIL.

* Lift critical section to avoid clang warning

* Respond to comments by picnixz

* Call out pyzstd explicitly in license description

Co-authored-by: Adam Turner <9087854+AA-Turner@users.noreply.github.com>

* Use a much more robust implementation...

... for `get_zstd_state_from_type`

Co-authored-by: Bénédikt Tran <10796600+picnixz@users.noreply.github.com>

* Use PyList_GetItemRef for thread safety purposes

* Use a macro for the minimum supported version

* remove const from primivite types

* Use PyMem_New in another spot

* Simplify error handling in _get_frame_size

* Another simplification of error handling in get_frame_info

* Rename _module_state to mod_state

* Rewrite comment explaining the context of the code

* Add link to pyzstd

* Add TODO about refactoring dict training code

* Use PyModule_AddObjectRef over PyModule_AddObject

PyModule_AddObject is soft-deprecated, so we should use PyModule_AddObjectRef

* Check result of OutputBufferGrow

* Simplify return logic in `add_constant_to_type`

Co-authored-by: Bénédikt Tran <10796600+picnixz@users.noreply.github.com>

* Ignore return value of _zstd_clear()

Co-authored-by: Bénédikt Tran <10796600+picnixz@users.noreply.github.com>

* Remove redundant comments

* Remove __reduce__ from ZstdDict

We should instead document that to pickle a dictionary a user should use
the `.dict_content` attribute.

* Use PyUnicode_FromFormat instead of a buffer

* Don't use C constants/types in error messages

* Make error messages easier to understand for Python users

* Lower minimum required version 1.4.0

* Use casts and make slot function signatures correct

* Be consistent with CPython on const usage

* Make else clauses in line with PEP 7

* Fix over-indented blocks in argument clinic

* Add critical section around ZSTD_DCtx_setParameter

* Add a TODO about refactoring critical sections

* Use Py_UNREACHABLE

* Move bytes operations out of Py_BEGIN_ALLOW_THREADS

* Add TODO about ensuring a lock is held

* Remove asserts that may not be correct

* Add TODO to make ZstdDict and others GC objects

* Make objects GC tracked

* Remove unused include

* Fix some memory issues

* Fix refleaks on module and in ZstdDict

* Update configure to check for ZDICT_finalizeDictionary

* Properly check version in configure

* exit(1) if check fails

* Use AC_RUN_IFELSE

* Use a define() to re-use version check

* Actually properly set _zstd module status based on version

---------

Co-authored-by: Stan Ulbrych <89152624+StanFromIreland@users.noreply.github.com>
Co-authored-by: Adam Turner <9087854+AA-Turner@users.noreply.github.com>
Co-authored-by: Bénédikt Tran <10796600+picnixz@users.noreply.github.com>
2025-05-04 01:29:55 +00:00

300 lines
11 KiB
Makefile

# -*- makefile -*-
# The file Setup is used by the makesetup script to construct the files
# Makefile and config.c, from Makefile.pre and config.c.in,
# respectively. Note that Makefile.pre is created from Makefile.pre.in
# by the toplevel configure script.
# (VPATH notes: Setup and Makefile.pre are in the build directory, as
# are Makefile and config.c; the *.in files are in the source directory.)
# Each line in this file describes one or more optional modules.
# Modules configured here will not be compiled by the setup.py script,
# so the file can be used to override setup.py's behavior.
# Tag lines containing just the word "*static*", "*shared*" or "*disabled*"
# (without the quotes but with the stars) are used to tag the following module
# descriptions. Tag lines may alternate throughout this file. Modules are
# built statically when they are preceded by a "*static*" tag line or when
# there is no tag line between the start of the file and the module
# description. Modules are built as a shared library when they are preceded by
# a "*shared*" tag line. Modules are not built at all, not by the Makefile,
# nor by the setup.py script, when they are preceded by a "*disabled*" tag
# line.
# Lines have the following structure:
#
# <module> ... [<sourcefile> ...] [<cpparg> ...] [<library> ...]
#
# <sourcefile> is anything ending in .c (.C, .cc, .c++ are C++ files)
# <cpparg> is anything starting with -I, -D, -U or -C
# <library> is anything ending in .a or beginning with -l or -L
# <module> is anything else but should be a valid Python
# identifier (letters, digits, underscores, beginning with non-digit)
#
# (As the makesetup script changes, it may recognize some other
# arguments as well, e.g. *.so and *.sl as libraries. See the big
# case statement in the makesetup script.)
#
# Lines can also have the form
#
# <name> = <value>
#
# which defines a Make variable definition inserted into Makefile.in.
# You can also use any Make variable that is detected by configure and
# defined in Makefile.pre.in, e.g. OpenSSL flags $(OPENSSL_INCLUDES).
#
# Rules generated by makesetup use additional variables:
#
# - All source file rules have a dependency on $(PYTHON_HEADERS) and on
# optional variable $(MODULES_{mod_upper}_DEPS).
# - If no <cpparg> and no <library> arguments are given, then makesetup
# defaults to $(MODULES_{mod_upper}_CFLAGS) cppargs and
# $(MODULES_{mod_upper}_LDFLAGS) libraries. The variables are typically
# defined by configure.
#
# The build process works like this:
#
# 1. Build all modules that are declared as static in Modules/Setup,
# combine them into libpythonxy.a, combine that into python.
# 2. Build all modules that are listed as shared in Modules/Setup.
# 3. Invoke setup.py. That builds all modules that
# a) are not builtin, and
# b) are not listed in Modules/Setup, and
# c) can be build on the target
#
# Therefore, modules declared to be shared will not be
# included in the config.c file, nor in the list of objects to be
# added to the library archive, and their linker options won't be
# added to the linker options. Rules to create their .o files and
# their shared libraries will still be added to the Makefile, and
# their names will be collected in the Make variable SHAREDMODS. This
# is used to build modules as shared libraries. (They can be
# installed using "make sharedinstall", which is implied by the
# toplevel "make install" target.) (For compatibility,
# *noconfig* has the same effect as *shared*.)
#
# NOTE: As a standard policy, as many modules as can be supported by a
# platform should be listed below. The distribution comes with all
# modules enabled that are supported by most platforms and don't
# require you to download sources from elsewhere.
#
# NOTE: Avoid editing this file directly. Local changes should go into
# Modules/Setup.local file. To enable all modules for testing, run
#
# sed -n -E 's/^#([a-z_\*].*)$/\1/p' Modules/Setup > Modules/Setup.local
# Some special rules to define PYTHONPATH.
# Edit the definitions below to indicate which options you are using.
# Don't add any whitespace or comments!
# Directories where library files get installed.
# DESTLIB is for Python modules; MACHDESTLIB for shared libraries.
DESTLIB=$(LIBDEST)
MACHDESTLIB=$(BINLIBDEST)
# NOTE: all the paths are now relative to the prefix that is computed
# at run time!
# Standard path -- don't edit.
# No leading colon since this is the first entry.
# Empty since this is now just the runtime prefix.
DESTPATH=
# Site specific path components -- should begin with : if non-empty
SITEPATH=
# Standard path components for test modules
TESTPATH=
COREPYTHONPATH=$(DESTPATH)$(SITEPATH)$(TESTPATH)
PYTHONPATH=$(COREPYTHONPATH)
# ---
# Built-in modules required to get a functioning interpreter are listed in
# Modules/Setup.bootstrap.
# ---
# The rest of the modules listed in this file are all commented out by
# default. Usually they can be detected and built as dynamically
# loaded modules by setup.py. If you're on a platform that doesn't
# support dynamic loading, want to compile modules statically into the
# Python binary, or need to specify some odd set of compiler switches,
# you can uncomment the appropriate lines below.
# Uncommenting the following line tells makesetup that all following
# modules are to be built as shared libraries (see above for more
# detail; also note that *static* or *disabled* cancels this effect):
#*shared*
# Modules that should always be present (POSIX and Windows):
#_asyncio _asynciomodule.c
#_bisect _bisectmodule.c
#_csv _csv.c
#_datetime _datetimemodule.c
#_decimal _decimal/_decimal.c
#_heapq _heapqmodule.c
#_interpchannels _interpchannelsmodule.c
#_interpqueues _interpqueuesmodule.c
#_interpreters _interpretersmodule.c
#_json _json.c
#_lsprof _lsprof.c rotatingtree.c
#_multiprocessing -I$(srcdir)/Modules/_multiprocessing _multiprocessing/multiprocessing.c _multiprocessing/semaphore.c
#_opcode _opcode.c
#_pickle _pickle.c
#_queue _queuemodule.c
#_random _randommodule.c
#_socket socketmodule.c
#_statistics _statisticsmodule.c
#_struct _struct.c
#_types _typesmodule.c
#_typing _typingmodule.c
#_zoneinfo _zoneinfo.c
#array arraymodule.c
#binascii binascii.c
#cmath cmathmodule.c
#math mathmodule.c
#mmap mmapmodule.c
#select selectmodule.c
#_sysconfig _sysconfig.c
# XML
#_elementtree _elementtree.c
#pyexpat pyexpat.c
# hashing builtins
#_blake2 blake2module.c
#_md5 md5module.c
#_sha1 sha1module.c
#_sha2 sha2module.c
#_sha3 sha3module.c
#_hmac hmacmodule.c
# text encodings and unicode
#_codecs_cn cjkcodecs/_codecs_cn.c
#_codecs_hk cjkcodecs/_codecs_hk.c
#_codecs_iso2022 cjkcodecs/_codecs_iso2022.c
#_codecs_jp cjkcodecs/_codecs_jp.c
#_codecs_kr cjkcodecs/_codecs_kr.c
#_codecs_tw cjkcodecs/_codecs_tw.c
#_multibytecodec cjkcodecs/multibytecodec.c
#unicodedata unicodedata.c
# Modules with some UNIX dependencies
#_posixsubprocess _posixsubprocess.c
#_posixshmem -I$(srcdir)/Modules/_multiprocessing _multiprocessing/posixshmem.c -lrt
#fcntl fcntlmodule.c
#grp grpmodule.c
#resource resource.c
#syslog syslogmodule.c
#termios termios.c
# Modules that require external libraries.
#_bz2 _bz2module.c -lbz2
#_ctypes _ctypes/_ctypes.c _ctypes/callbacks.c _ctypes/callproc.c _ctypes/stgdict.c _ctypes/cfield.c -ldl -lffi -DHAVE_FFI_PREP_CIF_VAR -DHAVE_FFI_PREP_CLOSURE_LOC -DHAVE_FFI_CLOSURE_ALLOC
# The _dbm module supports NDBM, GDBM with compat module, and Berkeley DB.
#_dbm _dbmmodule.c -lgdbm_compat -DUSE_GDBM_COMPAT
#_gdbm _gdbmmodule.c -lgdbm
#_lzma _lzmamodule.c -llzma
#_zstd _zstd/_zstdmodule.c -lzstd -I$(srcdir)/Modules/_zstd
#_uuid _uuidmodule.c -luuid
#zlib zlibmodule.c -lz
# The readline module also supports libeditline (-leditline).
# Some systems may require -ltermcap or -ltermlib.
#readline readline.c -lreadline -ltermcap
# OpenSSL bindings
#_ssl _ssl.c $(OPENSSL_INCLUDES) $(OPENSSL_LDFLAGS) $(OPENSSL_LIBS)
#_hashlib _hashopenssl.c $(OPENSSL_INCLUDES) $(OPENSSL_LDFLAGS) -lcrypto
# To statically link OpenSSL:
# _ssl _ssl.c $(OPENSSL_INCLUDES) $(OPENSSL_LDFLAGS) \
# -l:libssl.a -Wl,--exclude-libs,libssl.a \
# -l:libcrypto.a -Wl,--exclude-libs,libcrypto.a
# _hashlib _hashopenssl.c $(OPENSSL_INCLUDES) $(OPENSSL_LDFLAGS) \
# -l:libcrypto.a -Wl,--exclude-libs,libcrypto.a
# The _tkinter module.
#
# The command for _tkinter is long and site specific. Please
# uncomment and/or edit those parts as indicated. If you don't have a
# specific extension (e.g. Tix or BLT), leave the corresponding line
# commented out. (Leave the trailing backslashes in! If you
# experience strange errors, you may want to join all uncommented
# lines and remove the backslashes -- the backslash interpretation is
# done by the shell's "read" command and it may not be implemented on
# every system.
# *** Always uncomment this (leave the leading underscore in!):
#_tkinter _tkinter.c tkappinit.c -DWITH_APPINIT $(TCLTK_INCLUDES) $(TCLTK_LIBS) \
# *** Uncomment and edit to reflect where your Tcl/Tk libraries are:
# -L/usr/local/lib \
# *** Uncomment and edit to reflect where your Tcl/Tk headers are:
# -I/usr/local/include \
# *** Uncomment and edit to reflect where your X11 header files are:
# -I/usr/X11R6/include \
# *** Or uncomment this for Solaris:
# -I/usr/openwin/include \
# *** Uncomment and edit for Tix extension only:
# -DWITH_TIX -ltix8.1.8.2 \
# *** Uncomment and edit for BLT extension only:
# -DWITH_BLT -I/usr/local/blt/blt8.0-unoff/include -lBLT8.0 \
# *** Uncomment and edit for PIL (TkImaging) extension only:
# (See http://www.pythonware.com/products/pil/ for more info)
# -DWITH_PIL -I../Extensions/Imaging/libImaging tkImaging.c \
# *** Uncomment and edit for TOGL extension only:
# -DWITH_TOGL togl.c \
# *** Uncomment and edit to reflect where your X11 libraries are:
# -L/usr/X11R6/lib \
# *** Or uncomment this for Solaris:
# -L/usr/openwin/lib \
# *** Uncomment these for TOGL extension only:
# -lGL -lGLU -lXext -lXmu \
# *** Uncomment for AIX:
# -lld \
# *** Always uncomment this; X11 libraries to link with:
# -lX11
# Some system have -lcurses
#_curses -lncurses -lncursesw -ltermcap _cursesmodule.c
#_curses_panel -lpanel -lncurses _curses_panel.c
# macOS specific module, needs SystemConfiguration and CoreFoundation framework
# _scproxy _scproxy.c
# Examples
#xx xxmodule.c
#xxlimited xxlimited.c
#xxlimited_35 xxlimited_35.c
#xxsubtype xxsubtype.c
# Testing
#_xxtestfuzz _xxtestfuzz/_xxtestfuzz.c _xxtestfuzz/fuzzer.c
#_testbuffer _testbuffer.c
#_testinternalcapi _testinternalcapi.c
# Some testing modules MUST be built as shared libraries.
#*shared*
#_ctypes_test _ctypes/_ctypes_test.c
#_testcapi _testcapimodule.c
#_testimportmultiple _testimportmultiple.c
#_testmultiphase _testmultiphase.c
#_remotedebugging _remotedebuggingmodule.c
#_testsinglephase _testsinglephase.c
# ---
# Uncommenting the following line tells makesetup that all following modules
# are not built (see above for more detail).
#
#*disabled*
#
# _sqlite3 _tkinter _curses pyexpat
# _codecs_jp _codecs_kr _codecs_tw unicodedata