cpython/Include/internal
Eric Snow da151fdc7a
[3.12] gh-105699: Use a _Py_hashtable_t for the PyModuleDef Cache (gh-106974) (gh-107412)
gh-105699: Use a _Py_hashtable_t for the PyModuleDef Cache (gh-106974)

This fixes a crasher due to a race condition, triggered infrequently when two isolated (own GIL) subinterpreters simultaneously initialize their sys or builtins modules.  The crash happened due the combination of the "detached" thread state we were using and the "last holder" logic we use for the GIL.  It turns out it's tricky to use the same thread state for different threads.  Who could have guessed?

We solve the problem by eliminating the one object we were still sharing between interpreters.  We replace it with a low-level hashtable, using the "raw" allocator to avoid tying it to the main interpreter.

We also remove the accommodations for "detached" thread states, which were a dubious idea to start with.

(cherry picked from commit 8ba4df91ae)
2023-07-28 23:16:12 +00:00
..
pycore_abstract.h bpo-45636: Simplify BINARY_OP (GH-29565) 2021-11-16 05:53:57 -08:00
pycore_asdl.h gh-87347: Add parenthesis around macro arguments (#93915) 2022-06-20 16:04:52 +02:00
pycore_ast.h [3.12] gh-104799: Move location of type_params AST fields (GH-104828) (#104974) 2023-05-30 14:33:01 +00:00
pycore_ast_state.h gh-104656: Rename typeparams AST node to type_params (#104657) 2023-05-21 21:25:09 -07:00
pycore_atexit.h [3.12] gh-100227: Lock Around Use of the Global "atexit" State (gh-105514) (gh-105517) 2023-06-08 19:27:44 +00:00
pycore_atomic.h Fix typo in internal/pycore_atomic.h (GH-95939) 2022-08-13 11:40:41 +08:00
pycore_atomic_funcs.h bpo-39465: Add pycore_atomic_funcs.h header (GH-20766) 2020-12-23 03:41:08 +01:00
pycore_bitutils.h bpo-29882: Fix portability bug introduced in GH-30774 (#30794) 2022-01-23 09:59:34 +00:00
pycore_blocks_output_buffer.h bpo-44458: Ensure BUFFER_BLOCK_SIZE symbol is statically allocated. (GH-26808) 2021-06-21 23:36:36 -07:00
pycore_bytes_methods.h bpo-35081: Move bytes_methods.h to the internal C API (GH-18492) 2020-02-12 22:32:34 +01:00
pycore_bytesobject.h gh-94673: Ensure Builtin Static Types are Readied Properly (gh-103940) 2023-04-27 16:19:43 -06:00
pycore_call.h gh-98003: Inline call frames for CALL_FUNCTION_EX (GH-98004) 2023-04-30 21:08:26 +08:00
pycore_ceval.h [3.12] gh-105340: include hidden fast-locals in locals() (GH-105715) (#106470) 2023-07-05 23:31:37 +00:00
pycore_ceval_state.h [3.12] gh-104812: Run Pending Calls in any Thread (gh-104813) (gh-105752) 2023-06-14 00:50:08 +00:00
pycore_code.h gh-87729: improve hit rate of LOAD_SUPER_ATTR specialization (#104270) 2023-05-11 08:08:13 -06:00
pycore_compile.h gh-104615: don't make unsafe swaps in apply_static_swaps (#104620) 2023-05-18 21:22:03 +00:00
pycore_condvar.h bpo-46315: Add ifdef HAVE_ feature checks for WASI compatibility (GH-30507) 2022-01-13 09:46:04 +01:00
pycore_context.h gh-81057: Move contextvars-related Globals to _PyRuntimeState (gh-99400) 2022-11-16 09:54:28 -07:00
pycore_descrobject.h gh-93911: Specialize LOAD_ATTR_PROPERTY (GH-93912) 2022-06-17 23:13:17 +08:00
pycore_dict.h gh-100227: Move dict_state.global_version to PyInterpreterState (gh-102338) 2023-03-09 08:16:30 -07:00
pycore_dict_state.h gh-104252: Immortalize Py_EMPTY_KEYS (gh-104253) 2023-05-10 07:28:40 -06:00
pycore_dtoa.h gh-104263: Rely on Py_NAN and introduce Py_INFINITY (GH-104202) 2023-05-10 17:44:52 +01:00
pycore_emscripten_signal.h bpo-47176: Interrupt handling for wasm32-emscripten builds without pthreads (GH-32209) 2022-04-03 22:58:52 +02:00
pycore_exceptions.h bpo-46417: Factorize _PyExc_InitTypes() code (GH-30804) 2022-01-22 21:48:56 +01:00
pycore_faulthandler.h gh-81057: Move faulthandler Globals to _PyRuntimeState (gh-100152) 2022-12-12 09:58:46 -07:00
pycore_fileutils.h gh-99726: Improves correctness of stat results for Windows, and uses faster API when available (GH-102149) 2023-03-16 17:27:21 +00:00
pycore_fileutils_windows.h gh-102765: Update ntpath.isdir/isfile/islink/exists to use GetFileInformationByName when available (GH-103485) 2023-04-27 14:23:26 +00:00
pycore_floatobject.h gh-81057: Move More Globals in Core Code to _PyRuntimeState (gh-99516) 2022-11-16 09:37:14 -07:00
pycore_flowgraph.h gh-97933: (PEP 709) inline list/dict/set comprehensions (#101441) 2023-05-09 11:02:14 -06:00
pycore_format.h gh-104018: remove unused format "z" handling in string formatfloat() (#104107) 2023-05-07 10:11:42 +05:30
pycore_frame.h [3.12] gh-105340: include hidden fast-locals in locals() (GH-105715) (#106470) 2023-07-05 23:31:37 +00:00
pycore_function.h gh-103763: Implement PEP 695 (#103764) 2023-05-15 20:36:23 -07:00
pycore_gc.h gh-97922: Run the GC only on eval breaker (#97920) 2022-10-08 07:57:09 -07:00
pycore_genobject.h GH-99005: More intrinsics (GH-100774) 2023-01-06 14:47:57 +00:00
pycore_getopt.h bpo-36763: Cleanup precmdline in _PyCoreConfig_Read() (GH-13371) 2019-05-17 03:15:12 +02:00
pycore_gil.h bpo-38353: Cleanup includes in the internal C API (GH-16548) 2019-10-02 23:51:20 +02:00
pycore_global_objects.h gh-104549: Set __module__ on TypeAliasType (#104550) 2023-05-18 15:56:15 -07:00
pycore_global_objects_fini_generated.h [3.12] gh-105587: Remove assertion from _PyStaticObject_CheckRefcnt (GH-105638) (#105769) 2023-06-14 05:04:22 +00:00
pycore_global_strings.h gh-88745: Add _winapi.CopyFile2 and update shutil.copy2 to use it (GH-105055) 2023-05-30 20:33:17 +01:00
pycore_hamt.h gh-81057: Move contextvars-related Globals to _PyRuntimeState (gh-99400) 2022-11-16 09:54:28 -07:00
pycore_hashtable.h Use _Py_RVALUE() in macros (#99844) 2022-11-28 17:42:22 +01:00
pycore_import.h [3.12] gh-105699: Use a _Py_hashtable_t for the PyModuleDef Cache (gh-106974) (gh-107412) 2023-07-28 23:16:12 +00:00
pycore_initconfig.h gh-98608: Stop Treating All Errors from _Py_NewInterpreterFromConfig() as Fatal (gh-102657) 2023-03-21 10:49:12 -06:00
pycore_instruments.h [3.12] GH-106895: Raise a ValueError when attempting to disable events that cannot be disabled. (GH-107337) (GH-107351) 2023-07-28 10:53:33 +01:00
pycore_interp.h [3.12] GH-103082: Rename PY_MONITORING_EVENTS to _PY_MONITORING_EVENTS (#107069) (#107075) 2023-07-22 22:20:38 +00:00
pycore_intrinsics.h gh-103763: Implement PEP 695 (#103764) 2023-05-15 20:36:23 -07:00
pycore_list.h gh-100146: Steal references from stack when building a list (#100147) 2023-01-03 10:49:49 -08:00
pycore_long.h GH-101291: Add low level, unstable API for pylong (GH-101685) 2023-05-21 14:45:48 +01:00
pycore_memoryobject.h gh-104223: Fix issues with inheriting from buffer classes (#104227) 2023-05-08 09:52:41 -07:00
pycore_moduleobject.h gh-104066: Improve performance of hasattr for module objects (#104063) 2023-05-04 08:50:26 -06:00
pycore_namespace.h bpo-45482: Rename namespaceobject.h to pycore_namespace.h (GH-28975) 2021-10-15 15:21:21 +02:00
pycore_object.h [3.12] gh-105020: Share tp_bases and tp_mro Between Interpreters For All Static Builtin Types (gh-105115) (gh-105124) 2023-06-01 22:24:55 +00:00
pycore_object_state.h gh-102304: Move the Total Refcount to PyInterpreterState (gh-102545) 2023-03-21 11:46:09 -06:00
pycore_obmalloc.h gh-101659: Isolate "obmalloc" State to Each Interpreter (gh-101660) 2023-04-24 17:23:57 -06:00
pycore_obmalloc_init.h gh-101659: Isolate "obmalloc" State to Each Interpreter (gh-101660) 2023-04-24 17:23:57 -06:00
pycore_opcode.h gh-103865: add monitoring support to LOAD_SUPER_ATTR (#103866) 2023-05-16 10:29:00 -06:00
pycore_opcode_utils.h gh-91276: remove unused _PyOpcode_RelativeJump (#103156) 2023-04-11 11:20:39 +01:00
pycore_parser.h gh-81057: Move the Cached Parser Dummy Name to _PyRuntimeState (#100277) 2022-12-16 13:48:03 +00:00
pycore_pathconfig.h bpo-45582: Port getpath[p].c to Python (GH-29041) 2021-12-03 00:08:42 +00:00
pycore_pyarena.h bpo-43244: Remove the pyarena.h header (GH-25007) 2021-03-24 02:23:01 +01:00
pycore_pyerrors.h gh-77757: replace exception wrapping by PEP-678 notes in typeobject's __set_name__ (#103402) 2023-04-11 11:53:06 +01:00
pycore_pyhash.h gh-81057: Move More Globals in Core Code to _PyRuntimeState (gh-99516) 2022-11-16 09:37:14 -07:00
pycore_pylifecycle.h GH-101520: Move tracemalloc functionality into core, leaving interface in Modules. (#104508) 2023-05-17 14:17:16 +01:00
pycore_pymath.h Remove unused internal macros (#102415) 2023-03-04 12:20:14 +00:00
pycore_pymem.h [3.12] gh-101524: Only Use Public C-API in the _xxsubinterpreters Module (gh-105258) (gh-107303) 2023-07-27 13:15:47 -06:00
pycore_pymem_init.h gh-81057: Move the Allocators to _PyRuntimeState (gh-99217) 2022-11-11 16:30:46 -07:00
pycore_pystate.h [3.12] gh-105699: Use a _Py_hashtable_t for the PyModuleDef Cache (gh-106974) (gh-107412) 2023-07-28 23:16:12 +00:00
pycore_pythread.h gh-81057: Fix the wasm32-wasi Buildbot (gh-100139) 2022-12-09 10:17:54 -07:00
pycore_range.h gh-89189: More compact range iterator (GH-27986) 2022-11-30 23:04:30 +02:00
pycore_runtime.h [3.12] gh-106140: Reorder some fields to facilitate out-of-process inspection (GH-106143) (#106147) 2023-06-27 17:44:37 +00:00
pycore_runtime_init.h [3.12] gh-105699: Use a _Py_hashtable_t for the PyModuleDef Cache (gh-106974) (gh-107412) 2023-07-28 23:16:12 +00:00
pycore_runtime_init_generated.h gh-88745: Add _winapi.CopyFile2 and update shutil.copy2 to use it (GH-105055) 2023-05-30 20:33:17 +01:00
pycore_signal.h gh-81057: Move Signal-Related Globals to _PyRuntimeState (gh-100085) 2022-12-12 16:50:19 -07:00
pycore_sliceobject.h GH-94163: Add BINARY_SLICE and STORE_SLICE instructions. (GH-94168) 2022-06-27 12:24:23 +01:00
pycore_strhex.h gh-91768: C API no longer use "const PyObject*" type (#91769) 2022-04-21 22:07:19 +02:00
pycore_structseq.h gh-94673: Properly Initialize and Finalize Static Builtin Types for Each Interpreter (gh-104072) 2023-05-01 19:36:00 -06:00
pycore_symtable.h gh-104602: ensure all cellvars are known up front (#104603) 2023-05-19 00:07:35 +00:00
pycore_sysmodule.h gh-101758: Clean Up Uses of Import State (gh-101919) 2023-02-15 15:32:31 -07:00
pycore_time.h gh-81057: Move time Globals to _PyRuntimeState (gh-100122) 2022-12-08 16:46:09 -07:00
pycore_token.h gh-102856: Python tokenizer implementation for PEP 701 (#104323) 2023-05-21 01:03:02 +01:00
pycore_traceback.h bpo-45459: Use type names in the internal C API (GH-31669) 2022-03-03 23:08:07 +01:00
pycore_tracemalloc.h gh-77532: Minor tweaks to allow compiling with PlatformToolset=ClangCL on Windows (GH-101352) 2023-01-27 14:45:08 +00:00
pycore_tuple.h gh-94673: Ensure Builtin Static Types are Readied Properly (gh-103940) 2023-04-27 16:19:43 -06:00
pycore_typeobject.h [3.12] gh-105020: Share tp_bases and tp_mro Between Interpreters For All Static Builtin Types (gh-105115) (gh-105124) 2023-06-01 22:24:55 +00:00
pycore_typevarobject.h gh-104549: Set __module__ on TypeAliasType (#104550) 2023-05-18 15:56:15 -07:00
pycore_ucnhash.h bpo-42157: Rename unicodedata.ucnhash_CAPI (GH-22994) 2020-10-27 04:36:22 +01:00
pycore_unicodeobject.h gh-84436: Implement Immortal Objects (gh-19474) 2023-04-22 13:39:37 -06:00
pycore_unicodeobject_generated.h gh-88745: Add _winapi.CopyFile2 and update shutil.copy2 to use it (GH-105055) 2023-05-30 20:33:17 +01:00
pycore_unionobject.h gh-87347: Add parenthesis around PyXXX_Check() arguments (#92815) 2022-06-16 13:49:43 +02:00
pycore_warnings.h bpo-35134: Split warnings.h and weakrefobject.h (GH-29042) 2021-10-19 01:31:57 +02:00