Stowage/cpython
2
0
Fork 0
mirror of https://github.com/python/cpython.git synced 2026-01-18 13:19:59 +00:00
Code Issues Projects Releases Packages Wiki Activity
cpython/Python/executor_cases.c.h

18168 lines
728 KiB
C
Raw Normal View History

GH-111485: Factor out tier 2 code generation from the rest of the interpreter code generator (GH-112968)
2023-12-12 12:12:17 +00:00
// This file is generated by Tools/cases_generator/tier2_generator.py
gh-104584: Baby steps towards generating and executing traces (#105924) Added a new, experimental, tracing optimizer and interpreter (a.k.a. "tier 2"). This currently pessimizes, so don't use yet -- this is infrastructure so we can experiment with optimizing passes. To enable it, pass ``-Xuops`` or set ``PYTHONUOPS=1``. To get debug output, set ``PYTHONUOPSDEBUG=N`` where ``N`` is a debug level (0-4, where 0 is no debug output and 4 is excessively verbose). All of this code is likely to change dramatically before the 3.13 feature freeze. But this is a first step.
2023-06-26 19:02:57 -07:00
// from:
// Python/bytecodes.c
// Do not edit!
gh-111520: Integrate the Tier 2 interpreter in the Tier 1 interpreter (#111428) - There is no longer a separate Python/executor.c file. - Conventions in Python/bytecodes.c are slightly different -- don't use `goto error`, you must use `GOTO_ERROR(error)` (same for others like `unused_local_error`). - The `TIER_ONE` and `TIER_TWO` symbols are only valid in the generated (.c.h) files. - In Lib/test/support/__init__.py, `Py_C_RECURSION_LIMIT` is imported from `_testcapi`. - On Windows, in debug mode, stack allocation grows from 8MiB to 12MiB. - **Beware!** This changes the env vars to enable uops and their debugging to `PYTHON_UOPS` and `PYTHON_LLTRACE`.
2023-11-01 13:13:02 -07:00
#ifdef TIER_ONE
#error "This file is for Tier 2 only"
#endif
#define TIER_TWO 2
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
case _NOP_r00: {
CHECK_CURRENT_CACHED_VALUES(0);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(0);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-104584: Baby steps towards generating and executing traces (#105924) Added a new, experimental, tracing optimizer and interpreter (a.k.a. "tier 2"). This currently pessimizes, so don't use yet -- this is infrastructure so we can experiment with optimizing passes. To enable it, pass ``-Xuops`` or set ``PYTHONUOPS=1``. To get debug output, set ``PYTHONUOPSDEBUG=N`` where ``N`` is a debug level (0-4, where 0 is no debug output and 4 is excessively verbose). All of this code is likely to change dramatically before the 3.13 feature freeze. But this is a first step.
2023-06-26 19:02:57 -07:00
break;
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
case _NOP_r11: {
CHECK_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef _stack_item_0 = _tos_cache0;
_tos_cache0 = _stack_item_0;
SET_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _NOP_r22: {
CHECK_CURRENT_CACHED_VALUES(2);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef _stack_item_0 = _tos_cache0;
_PyStackRef _stack_item_1 = _tos_cache1;
_tos_cache1 = _stack_item_1;
_tos_cache0 = _stack_item_0;
SET_CURRENT_CACHED_VALUES(2);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _NOP_r33: {
CHECK_CURRENT_CACHED_VALUES(3);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef _stack_item_0 = _tos_cache0;
_PyStackRef _stack_item_1 = _tos_cache1;
_PyStackRef _stack_item_2 = _tos_cache2;
_tos_cache2 = _stack_item_2;
_tos_cache1 = _stack_item_1;
_tos_cache0 = _stack_item_0;
SET_CURRENT_CACHED_VALUES(3);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _CHECK_PERIODIC_r00: {
CHECK_CURRENT_CACHED_VALUES(0);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-132532: Add new DSL macros to better declare semantics of exits at ends of instructions/uops. (GH-137098)
2025-08-09 15:41:28 +01:00
_PyFrame_SetStackPointer(frame, stack_pointer);
int err = check_periodics(tstate);
stack_pointer = _PyFrame_GetStackPointer(frame);
if (err != 0) {
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(0);
GH-132532: Add new DSL macros to better declare semantics of exits at ends of instructions/uops. (GH-137098)
2025-08-09 15:41:28 +01:00
JUMP_TO_ERROR();
GH-120024: Remove `CHECK_EVAL_BREAKER` macro. (GH-122968) * Factor some instructions into micro-ops to isolate CHECK_EVAL_BREAKER for escape analysis * Eliminate CHECK_EVAL_BREAKER macro
2024-08-14 12:04:05 +01:00
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_tos_cache0 = PyStackRef_ZERO_BITS;
_tos_cache1 = PyStackRef_ZERO_BITS;
_tos_cache2 = PyStackRef_ZERO_BITS;
SET_CURRENT_CACHED_VALUES(0);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-120024: Remove `CHECK_EVAL_BREAKER` macro. (GH-122968) * Factor some instructions into micro-ops to isolate CHECK_EVAL_BREAKER for escape analysis * Eliminate CHECK_EVAL_BREAKER macro
2024-08-14 12:04:05 +01:00
break;
}
GH-132532: Add new DSL macros to better declare semantics of exits at ends of instructions/uops. (GH-137098)
2025-08-09 15:41:28 +01:00
/* _CHECK_PERIODIC_AT_END is not a viable micro-op for tier 2 because it is replaced */
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
case _CHECK_PERIODIC_IF_NOT_YIELD_FROM_r00: {
CHECK_CURRENT_CACHED_VALUES(0);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-120024: Remove `CHECK_EVAL_BREAKER` macro. (GH-122968) * Factor some instructions into micro-ops to isolate CHECK_EVAL_BREAKER for escape analysis * Eliminate CHECK_EVAL_BREAKER macro
2024-08-14 12:04:05 +01:00
oparg = CURRENT_OPARG();
if ((oparg & RESUME_OPARG_LOCATION_MASK) < RESUME_AFTER_YIELD_FROM) {
GH-132532: Add new DSL macros to better declare semantics of exits at ends of instructions/uops. (GH-137098)
2025-08-09 15:41:28 +01:00
_PyFrame_SetStackPointer(frame, stack_pointer);
int err = check_periodics(tstate);
stack_pointer = _PyFrame_GetStackPointer(frame);
if (err != 0) {
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(0);
GH-132532: Add new DSL macros to better declare semantics of exits at ends of instructions/uops. (GH-137098)
2025-08-09 15:41:28 +01:00
JUMP_TO_ERROR();
GH-120024: Remove `CHECK_EVAL_BREAKER` macro. (GH-122968) * Factor some instructions into micro-ops to isolate CHECK_EVAL_BREAKER for escape analysis * Eliminate CHECK_EVAL_BREAKER macro
2024-08-14 12:04:05 +01:00
}
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_tos_cache0 = PyStackRef_ZERO_BITS;
_tos_cache1 = PyStackRef_ZERO_BITS;
_tos_cache2 = PyStackRef_ZERO_BITS;
SET_CURRENT_CACHED_VALUES(0);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-120024: Remove `CHECK_EVAL_BREAKER` macro. (GH-122968) * Factor some instructions into micro-ops to isolate CHECK_EVAL_BREAKER for escape analysis * Eliminate CHECK_EVAL_BREAKER macro
2024-08-14 12:04:05 +01:00
break;
}
/* _QUICKEN_RESUME is not a viable micro-op for tier 2 because it uses the 'this_instr' variable */
gh-115999: Implement thread-local bytecode and enable specialization for `BINARY_OP` (#123926) Each thread specializes a thread-local copy of the bytecode, created on the first RESUME, in free-threaded builds. All copies of the bytecode for a code object are stored in the co_tlbc array on the code object. Threads reserve a globally unique index identifying its copy of the bytecode in all co_tlbc arrays at thread creation and release the index at thread destruction. The first entry in every co_tlbc array always points to the "main" copy of the bytecode that is stored at the end of the code object. This ensures that no bytecode is copied for programs that do not use threads. Thread-local bytecode can be disabled at runtime by providing either -X tlbc=0 or PYTHON_TLBC=0. Disabling thread-local bytecode also disables specialization. Concurrent modifications to the bytecode made by the specializing interpreter and instrumentation use atomics, with specialization taking care not to overwrite an instruction that was instrumented concurrently.
2024-11-04 11:13:32 -08:00
/* _LOAD_BYTECODE is not a viable micro-op for tier 2 because it uses the 'this_instr' variable */
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
case _RESUME_CHECK_r00: {
CHECK_CURRENT_CACHED_VALUES(0);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
#if defined(__EMSCRIPTEN__)
if (_Py_emscripten_signal_clock == 0) {
UOP_STAT_INC(uopcode, miss);
SET_CURRENT_CACHED_VALUES(0);
JUMP_TO_JUMP_TARGET();
}
_Py_emscripten_signal_clock -= Py_EMSCRIPTEN_SIGNAL_HANDLING;
#endif
uintptr_t eval_breaker = _Py_atomic_load_uintptr_relaxed(&tstate->eval_breaker);
uintptr_t version = FT_ATOMIC_LOAD_UINTPTR_ACQUIRE(_PyFrame_GetCode(frame)->_co_instrumentation_version);
assert((version & _PY_EVAL_EVENTS_MASK) == 0);
if (eval_breaker != version) {
UOP_STAT_INC(uopcode, miss);
SET_CURRENT_CACHED_VALUES(0);
JUMP_TO_JUMP_TARGET();
}
#ifdef Py_GIL_DISABLED
if (frame->tlbc_index !=
((_PyThreadStateImpl *)tstate)->tlbc_index) {
UOP_STAT_INC(uopcode, miss);
SET_CURRENT_CACHED_VALUES(0);
JUMP_TO_JUMP_TARGET();
}
#endif
SET_CURRENT_CACHED_VALUES(0);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _RESUME_CHECK_r11: {
CHECK_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef _stack_item_0 = _tos_cache0;
#if defined(__EMSCRIPTEN__)
if (_Py_emscripten_signal_clock == 0) {
UOP_STAT_INC(uopcode, miss);
GH-142718: JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. (GH-142762) JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. * Make sure that stack is in correct state at side exits with TOS cached values * Simplify choice of cached items for side exits
2025-12-15 19:01:29 +00:00
_tos_cache0 = _stack_item_0;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(1);
JUMP_TO_JUMP_TARGET();
}
_Py_emscripten_signal_clock -= Py_EMSCRIPTEN_SIGNAL_HANDLING;
#endif
uintptr_t eval_breaker = _Py_atomic_load_uintptr_relaxed(&tstate->eval_breaker);
uintptr_t version = FT_ATOMIC_LOAD_UINTPTR_ACQUIRE(_PyFrame_GetCode(frame)->_co_instrumentation_version);
assert((version & _PY_EVAL_EVENTS_MASK) == 0);
if (eval_breaker != version) {
UOP_STAT_INC(uopcode, miss);
GH-142718: JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. (GH-142762) JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. * Make sure that stack is in correct state at side exits with TOS cached values * Simplify choice of cached items for side exits
2025-12-15 19:01:29 +00:00
_tos_cache0 = _stack_item_0;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(1);
JUMP_TO_JUMP_TARGET();
}
#ifdef Py_GIL_DISABLED
if (frame->tlbc_index !=
((_PyThreadStateImpl *)tstate)->tlbc_index) {
UOP_STAT_INC(uopcode, miss);
GH-142718: JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. (GH-142762) JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. * Make sure that stack is in correct state at side exits with TOS cached values * Simplify choice of cached items for side exits
2025-12-15 19:01:29 +00:00
_tos_cache0 = _stack_item_0;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(1);
JUMP_TO_JUMP_TARGET();
}
#endif
_tos_cache0 = _stack_item_0;
SET_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _RESUME_CHECK_r22: {
CHECK_CURRENT_CACHED_VALUES(2);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef _stack_item_0 = _tos_cache0;
_PyStackRef _stack_item_1 = _tos_cache1;
#if defined(__EMSCRIPTEN__)
if (_Py_emscripten_signal_clock == 0) {
UOP_STAT_INC(uopcode, miss);
GH-142718: JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. (GH-142762) JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. * Make sure that stack is in correct state at side exits with TOS cached values * Simplify choice of cached items for side exits
2025-12-15 19:01:29 +00:00
_tos_cache1 = _stack_item_1;
_tos_cache0 = _stack_item_0;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(2);
JUMP_TO_JUMP_TARGET();
}
_Py_emscripten_signal_clock -= Py_EMSCRIPTEN_SIGNAL_HANDLING;
#endif
uintptr_t eval_breaker = _Py_atomic_load_uintptr_relaxed(&tstate->eval_breaker);
uintptr_t version = FT_ATOMIC_LOAD_UINTPTR_ACQUIRE(_PyFrame_GetCode(frame)->_co_instrumentation_version);
assert((version & _PY_EVAL_EVENTS_MASK) == 0);
if (eval_breaker != version) {
UOP_STAT_INC(uopcode, miss);
GH-142718: JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. (GH-142762) JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. * Make sure that stack is in correct state at side exits with TOS cached values * Simplify choice of cached items for side exits
2025-12-15 19:01:29 +00:00
_tos_cache1 = _stack_item_1;
_tos_cache0 = _stack_item_0;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(2);
JUMP_TO_JUMP_TARGET();
}
#ifdef Py_GIL_DISABLED
if (frame->tlbc_index !=
((_PyThreadStateImpl *)tstate)->tlbc_index) {
UOP_STAT_INC(uopcode, miss);
GH-142718: JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. (GH-142762) JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. * Make sure that stack is in correct state at side exits with TOS cached values * Simplify choice of cached items for side exits
2025-12-15 19:01:29 +00:00
_tos_cache1 = _stack_item_1;
_tos_cache0 = _stack_item_0;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(2);
JUMP_TO_JUMP_TARGET();
}
#endif
_tos_cache1 = _stack_item_1;
_tos_cache0 = _stack_item_0;
SET_CURRENT_CACHED_VALUES(2);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _RESUME_CHECK_r33: {
CHECK_CURRENT_CACHED_VALUES(3);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef _stack_item_0 = _tos_cache0;
_PyStackRef _stack_item_1 = _tos_cache1;
_PyStackRef _stack_item_2 = _tos_cache2;
GH-111485: Factor out tier 2 code generation from the rest of the interpreter code generator (GH-112968)
2023-12-12 12:12:17 +00:00
#if defined(__EMSCRIPTEN__)
GH-117457: Correct pystats uop "miss" counts (GH-117477)
2024-04-04 18:49:18 -04:00
if (_Py_emscripten_signal_clock == 0) {
UOP_STAT_INC(uopcode, miss);
GH-142718: JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. (GH-142762) JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. * Make sure that stack is in correct state at side exits with TOS cached values * Simplify choice of cached items for side exits
2025-12-15 19:01:29 +00:00
_tos_cache2 = _stack_item_2;
_tos_cache1 = _stack_item_1;
_tos_cache0 = _stack_item_0;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(3);
GH-117457: Correct pystats uop "miss" counts (GH-117477)
2024-04-04 18:49:18 -04:00
JUMP_TO_JUMP_TARGET();
}
GH-108614: Unbreak emscripten build (GH-109132)
2023-09-08 17:54:45 +01:00
_Py_emscripten_signal_clock -= Py_EMSCRIPTEN_SIGNAL_HANDLING;
GH-111485: Factor out tier 2 code generation from the rest of the interpreter code generator (GH-112968)
2023-12-12 12:12:17 +00:00
#endif
gh-112175: Add `eval_breaker` to `PyThreadState` (#115194) This change adds an `eval_breaker` field to `PyThreadState`. The primary motivation is for performance in free-threaded builds: with thread-local eval breakers, we can stop a specific thread (e.g., for an async exception) without interrupting other threads. The source of truth for the global instrumentation version is stored in the `instrumentation_version` field in PyInterpreterState. Threads usually read the version from their local `eval_breaker`, where it continues to be colocated with the eval breaker bits.
2024-02-20 06:57:48 -08:00
uintptr_t eval_breaker = _Py_atomic_load_uintptr_relaxed(&tstate->eval_breaker);
gh-116818: Make `sys.settrace`, `sys.setprofile`, and monitoring thread-safe (#116775) Makes sys.settrace, sys.setprofile, and monitoring generally thread-safe. Mostly uses a stop-the-world approach and synchronization around the code object's _co_instrumentation_version. There may be a little bit of extra synchronization around the monitoring data that's required to be TSAN clean.
2024-04-19 14:47:42 -07:00
uintptr_t version = FT_ATOMIC_LOAD_UINTPTR_ACQUIRE(_PyFrame_GetCode(frame)->_co_instrumentation_version);
GH-109369: Merge all eval-breaker flags and monitoring version into one word. (GH-109846)
2023-10-04 16:09:48 +01:00
assert((version & _PY_EVAL_EVENTS_MASK) == 0);
GH-117457: Correct pystats uop "miss" counts (GH-117477)
2024-04-04 18:49:18 -04:00
if (eval_breaker != version) {
UOP_STAT_INC(uopcode, miss);
GH-142718: JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. (GH-142762) JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. * Make sure that stack is in correct state at side exits with TOS cached values * Simplify choice of cached items for side exits
2025-12-15 19:01:29 +00:00
_tos_cache2 = _stack_item_2;
_tos_cache1 = _stack_item_1;
_tos_cache0 = _stack_item_0;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(3);
GH-117457: Correct pystats uop "miss" counts (GH-117477)
2024-04-04 18:49:18 -04:00
JUMP_TO_JUMP_TARGET();
}
gh-115999: Implement thread-local bytecode and enable specialization for `BINARY_OP` (#123926) Each thread specializes a thread-local copy of the bytecode, created on the first RESUME, in free-threaded builds. All copies of the bytecode for a code object are stored in the co_tlbc array on the code object. Threads reserve a globally unique index identifying its copy of the bytecode in all co_tlbc arrays at thread creation and release the index at thread destruction. The first entry in every co_tlbc array always points to the "main" copy of the bytecode that is stored at the end of the code object. This ensures that no bytecode is copied for programs that do not use threads. Thread-local bytecode can be disabled at runtime by providing either -X tlbc=0 or PYTHON_TLBC=0. Disabling thread-local bytecode also disables specialization. Concurrent modifications to the bytecode made by the specializing interpreter and instrumentation use atomics, with specialization taking care not to overwrite an instruction that was instrumented concurrently.
2024-11-04 11:13:32 -08:00
#ifdef Py_GIL_DISABLED
if (frame->tlbc_index !=
((_PyThreadStateImpl *)tstate)->tlbc_index) {
UOP_STAT_INC(uopcode, miss);
GH-142718: JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. (GH-142762) JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. * Make sure that stack is in correct state at side exits with TOS cached values * Simplify choice of cached items for side exits
2025-12-15 19:01:29 +00:00
_tos_cache2 = _stack_item_2;
_tos_cache1 = _stack_item_1;
_tos_cache0 = _stack_item_0;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(3);
gh-115999: Implement thread-local bytecode and enable specialization for `BINARY_OP` (#123926) Each thread specializes a thread-local copy of the bytecode, created on the first RESUME, in free-threaded builds. All copies of the bytecode for a code object are stored in the co_tlbc array on the code object. Threads reserve a globally unique index identifying its copy of the bytecode in all co_tlbc arrays at thread creation and release the index at thread destruction. The first entry in every co_tlbc array always points to the "main" copy of the bytecode that is stored at the end of the code object. This ensures that no bytecode is copied for programs that do not use threads. Thread-local bytecode can be disabled at runtime by providing either -X tlbc=0 or PYTHON_TLBC=0. Disabling thread-local bytecode also disables specialization. Concurrent modifications to the bytecode made by the specializing interpreter and instrumentation use atomics, with specialization taking care not to overwrite an instruction that was instrumented concurrently.
2024-11-04 11:13:32 -08:00
JUMP_TO_JUMP_TARGET();
}
#endif
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_tos_cache2 = _stack_item_2;
_tos_cache1 = _stack_item_1;
_tos_cache0 = _stack_item_0;
SET_CURRENT_CACHED_VALUES(3);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-106581: Project through calls (#108067) This finishes the work begun in gh-107760. When, while projecting a superblock, we encounter a call to a short, simple function, the superblock will now enter the function using `_PUSH_FRAME`, continue through it, and leave it using `_POP_FRAME`, and then continue through the original code. Multiple frame pushes and pops are even possible. It is also possible to stop appending to the superblock in the middle of a called function, when running out of space or encountering an unsupported bytecode.
2023-08-17 11:29:58 -07:00
break;
}
GH-120024: Remove `CHECK_EVAL_BREAKER` macro. (GH-122968) * Factor some instructions into micro-ops to isolate CHECK_EVAL_BREAKER for escape analysis * Eliminate CHECK_EVAL_BREAKER macro
2024-08-14 12:04:05 +01:00
/* _MONITOR_RESUME is not a viable micro-op for tier 2 because it uses the 'this_instr' variable */
GH-111485: Factor out tier 2 code generation from the rest of the interpreter code generator (GH-112968)
2023-12-12 12:12:17 +00:00
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
case _LOAD_FAST_CHECK_r01: {
CHECK_CURRENT_CACHED_VALUES(0);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
_PyStackRef value;
GH-111485: Factor out tier 2 code generation from the rest of the interpreter code generator (GH-112968)
2023-12-12 12:12:17 +00:00
oparg = CURRENT_OPARG();
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
_PyStackRef value_s = GETLOCAL(oparg);
if (PyStackRef_IsNull(value_s)) {
GH-119866: Spill the stack around escaping calls. (GH-124392) * Spill the evaluation around escaping calls in the generated interpreter and JIT. * The code generator tracks live, cached values so they can be saved to memory when needed. * Spills the stack pointer around escaping calls, so that the exact stack is visible to the cycle GC.
2024-10-07 14:56:39 +01:00
_PyFrame_SetStackPointer(frame, stack_pointer);
GH-116422: Tier2 hot/cold splitting (GH-116813) Splits the "cold" path, deopts and exits, from the "hot" path, reducing the size of most jitted instructions, at the cost of slower exits.
2024-03-26 09:35:11 +00:00
_PyEval_FormatExcCheckArg(tstate, PyExc_UnboundLocalError,
UNBOUNDLOCAL_ERROR_MSG,
PyTuple_GetItem(_PyFrame_GetCode(frame)->co_localsplusnames, oparg)
);
GH-119866: Spill the stack around escaping calls. (GH-124392) * Spill the evaluation around escaping calls in the generated interpreter and JIT. * The code generator tracks live, cached values so they can be saved to memory when needed. * Spills the stack pointer around escaping calls, so that the exact stack is visible to the cycle GC.
2024-10-07 14:56:39 +01:00
stack_pointer = _PyFrame_GetStackPointer(frame);
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(0);
GH-128682: Make `PyStackRef_CLOSE` escaping. (GH-129404)
2025-02-03 12:41:32 +00:00
JUMP_TO_ERROR();
GH-116422: Tier2 hot/cold splitting (GH-116813) Splits the "cold" path, deopts and exits, from the "hot" path, reducing the size of most jitted instructions, at the cost of slower exits.
2024-03-26 09:35:11 +00:00
}
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
value = PyStackRef_DUP(value_s);
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_tos_cache0 = value;
SET_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-106290: Fix edge cases around uops (#106319) - Tweak uops debugging output - Fix the bug from gh-106290 - Rename `SET_IP` to `SAVE_IP` (per https://github.com/faster-cpython/ideas/issues/558) - Add a `SAVE_IP` uop at the start of the trace (ditto) - Allow `unbound_local_error`; this gives us uops for `LOAD_FAST_CHECK`, `LOAD_CLOSURE`, and `DELETE_FAST` - Longer traces - Support `STORE_FAST_LOAD_FAST`, `STORE_FAST_STORE_FAST` - Add deps on pycore_uops.h to Makefile(.pre.in)
2023-07-03 13:05:11 -07:00
break;
}
GH-142718: JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. (GH-142762) JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. * Make sure that stack is in correct state at side exits with TOS cached values * Simplify choice of cached items for side exits
2025-12-15 19:01:29 +00:00
case _LOAD_FAST_CHECK_r12: {
CHECK_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-142718: JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. (GH-142762) JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. * Make sure that stack is in correct state at side exits with TOS cached values * Simplify choice of cached items for side exits
2025-12-15 19:01:29 +00:00
_PyStackRef value;
_PyStackRef _stack_item_0 = _tos_cache0;
oparg = CURRENT_OPARG();
_PyStackRef value_s = GETLOCAL(oparg);
if (PyStackRef_IsNull(value_s)) {
stack_pointer[0] = _stack_item_0;
stack_pointer += 1;
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
_PyFrame_SetStackPointer(frame, stack_pointer);
_PyEval_FormatExcCheckArg(tstate, PyExc_UnboundLocalError,
UNBOUNDLOCAL_ERROR_MSG,
PyTuple_GetItem(_PyFrame_GetCode(frame)->co_localsplusnames, oparg)
);
stack_pointer = _PyFrame_GetStackPointer(frame);
SET_CURRENT_CACHED_VALUES(0);
JUMP_TO_ERROR();
}
value = PyStackRef_DUP(value_s);
_tos_cache1 = value;
_tos_cache0 = _stack_item_0;
SET_CURRENT_CACHED_VALUES(2);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-142718: JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. (GH-142762) JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. * Make sure that stack is in correct state at side exits with TOS cached values * Simplify choice of cached items for side exits
2025-12-15 19:01:29 +00:00
break;
}
case _LOAD_FAST_CHECK_r23: {
CHECK_CURRENT_CACHED_VALUES(2);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-142718: JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. (GH-142762) JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. * Make sure that stack is in correct state at side exits with TOS cached values * Simplify choice of cached items for side exits
2025-12-15 19:01:29 +00:00
_PyStackRef value;
_PyStackRef _stack_item_0 = _tos_cache0;
_PyStackRef _stack_item_1 = _tos_cache1;
oparg = CURRENT_OPARG();
_PyStackRef value_s = GETLOCAL(oparg);
if (PyStackRef_IsNull(value_s)) {
stack_pointer[0] = _stack_item_0;
stack_pointer[1] = _stack_item_1;
stack_pointer += 2;
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
_PyFrame_SetStackPointer(frame, stack_pointer);
_PyEval_FormatExcCheckArg(tstate, PyExc_UnboundLocalError,
UNBOUNDLOCAL_ERROR_MSG,
PyTuple_GetItem(_PyFrame_GetCode(frame)->co_localsplusnames, oparg)
);
stack_pointer = _PyFrame_GetStackPointer(frame);
SET_CURRENT_CACHED_VALUES(0);
JUMP_TO_ERROR();
}
value = PyStackRef_DUP(value_s);
_tos_cache2 = value;
_tos_cache1 = _stack_item_1;
_tos_cache0 = _stack_item_0;
SET_CURRENT_CACHED_VALUES(3);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-142718: JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. (GH-142762) JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. * Make sure that stack is in correct state at side exits with TOS cached values * Simplify choice of cached items for side exits
2025-12-15 19:01:29 +00:00
break;
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
case _LOAD_FAST_0_r01: {
CHECK_CURRENT_CACHED_VALUES(0);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
_PyStackRef value;
GH-115457: Support splitting and replication of micro ops. (GH-115558)
2024-02-20 10:50:59 +00:00
oparg = 0;
assert(oparg == CURRENT_OPARG());
gh-117139: Fix a few wrong steals in bytecodes.c (GH-121127) Fix a few wrong steals in bytecodes.c
2024-06-29 02:14:48 +08:00
assert(!PyStackRef_IsNull(GETLOCAL(oparg)));
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
value = PyStackRef_DUP(GETLOCAL(oparg));
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_tos_cache0 = value;
SET_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-115457: Support splitting and replication of micro ops. (GH-115558)
2024-02-20 10:50:59 +00:00
break;
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
case _LOAD_FAST_0_r12: {
CHECK_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
_PyStackRef value;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef _stack_item_0 = _tos_cache0;
oparg = 0;
GH-115457: Support splitting and replication of micro ops. (GH-115558)
2024-02-20 10:50:59 +00:00
assert(oparg == CURRENT_OPARG());
gh-117139: Fix a few wrong steals in bytecodes.c (GH-121127) Fix a few wrong steals in bytecodes.c
2024-06-29 02:14:48 +08:00
assert(!PyStackRef_IsNull(GETLOCAL(oparg)));
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
value = PyStackRef_DUP(GETLOCAL(oparg));
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_tos_cache1 = value;
_tos_cache0 = _stack_item_0;
SET_CURRENT_CACHED_VALUES(2);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-115457: Support splitting and replication of micro ops. (GH-115558)
2024-02-20 10:50:59 +00:00
break;
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
case _LOAD_FAST_0_r23: {
CHECK_CURRENT_CACHED_VALUES(2);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
_PyStackRef value;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef _stack_item_0 = _tos_cache0;
_PyStackRef _stack_item_1 = _tos_cache1;
oparg = 0;
GH-115457: Support splitting and replication of micro ops. (GH-115558)
2024-02-20 10:50:59 +00:00
assert(oparg == CURRENT_OPARG());
gh-117139: Fix a few wrong steals in bytecodes.c (GH-121127) Fix a few wrong steals in bytecodes.c
2024-06-29 02:14:48 +08:00
assert(!PyStackRef_IsNull(GETLOCAL(oparg)));
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
value = PyStackRef_DUP(GETLOCAL(oparg));
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_tos_cache2 = value;
_tos_cache1 = _stack_item_1;
_tos_cache0 = _stack_item_0;
SET_CURRENT_CACHED_VALUES(3);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-115457: Support splitting and replication of micro ops. (GH-115558)
2024-02-20 10:50:59 +00:00
break;
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
case _LOAD_FAST_1_r01: {
CHECK_CURRENT_CACHED_VALUES(0);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
_PyStackRef value;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
oparg = 1;
GH-115457: Support splitting and replication of micro ops. (GH-115558)
2024-02-20 10:50:59 +00:00
assert(oparg == CURRENT_OPARG());
gh-117139: Fix a few wrong steals in bytecodes.c (GH-121127) Fix a few wrong steals in bytecodes.c
2024-06-29 02:14:48 +08:00
assert(!PyStackRef_IsNull(GETLOCAL(oparg)));
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
value = PyStackRef_DUP(GETLOCAL(oparg));
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_tos_cache0 = value;
SET_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-115457: Support splitting and replication of micro ops. (GH-115558)
2024-02-20 10:50:59 +00:00
break;
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
case _LOAD_FAST_1_r12: {
CHECK_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
_PyStackRef value;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef _stack_item_0 = _tos_cache0;
oparg = 1;
GH-115457: Support splitting and replication of micro ops. (GH-115558)
2024-02-20 10:50:59 +00:00
assert(oparg == CURRENT_OPARG());
gh-117139: Fix a few wrong steals in bytecodes.c (GH-121127) Fix a few wrong steals in bytecodes.c
2024-06-29 02:14:48 +08:00
assert(!PyStackRef_IsNull(GETLOCAL(oparg)));
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
value = PyStackRef_DUP(GETLOCAL(oparg));
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_tos_cache1 = value;
_tos_cache0 = _stack_item_0;
SET_CURRENT_CACHED_VALUES(2);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-115457: Support splitting and replication of micro ops. (GH-115558)
2024-02-20 10:50:59 +00:00
break;
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
case _LOAD_FAST_1_r23: {
CHECK_CURRENT_CACHED_VALUES(2);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
_PyStackRef value;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef _stack_item_0 = _tos_cache0;
_PyStackRef _stack_item_1 = _tos_cache1;
oparg = 1;
GH-115457: Support splitting and replication of micro ops. (GH-115558)
2024-02-20 10:50:59 +00:00
assert(oparg == CURRENT_OPARG());
gh-117139: Fix a few wrong steals in bytecodes.c (GH-121127) Fix a few wrong steals in bytecodes.c
2024-06-29 02:14:48 +08:00
assert(!PyStackRef_IsNull(GETLOCAL(oparg)));
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
value = PyStackRef_DUP(GETLOCAL(oparg));
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_tos_cache2 = value;
_tos_cache1 = _stack_item_1;
_tos_cache0 = _stack_item_0;
SET_CURRENT_CACHED_VALUES(3);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-115457: Support splitting and replication of micro ops. (GH-115558)
2024-02-20 10:50:59 +00:00
break;
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
case _LOAD_FAST_2_r01: {
CHECK_CURRENT_CACHED_VALUES(0);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
_PyStackRef value;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
oparg = 2;
GH-115457: Support splitting and replication of micro ops. (GH-115558)
2024-02-20 10:50:59 +00:00
assert(oparg == CURRENT_OPARG());
gh-117139: Fix a few wrong steals in bytecodes.c (GH-121127) Fix a few wrong steals in bytecodes.c
2024-06-29 02:14:48 +08:00
assert(!PyStackRef_IsNull(GETLOCAL(oparg)));
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
value = PyStackRef_DUP(GETLOCAL(oparg));
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_tos_cache0 = value;
SET_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-115457: Support splitting and replication of micro ops. (GH-115558)
2024-02-20 10:50:59 +00:00
break;
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
case _LOAD_FAST_2_r12: {
CHECK_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
_PyStackRef value;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef _stack_item_0 = _tos_cache0;
oparg = 2;
GH-115457: Support splitting and replication of micro ops. (GH-115558)
2024-02-20 10:50:59 +00:00
assert(oparg == CURRENT_OPARG());
gh-117139: Fix a few wrong steals in bytecodes.c (GH-121127) Fix a few wrong steals in bytecodes.c
2024-06-29 02:14:48 +08:00
assert(!PyStackRef_IsNull(GETLOCAL(oparg)));
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
value = PyStackRef_DUP(GETLOCAL(oparg));
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_tos_cache1 = value;
_tos_cache0 = _stack_item_0;
SET_CURRENT_CACHED_VALUES(2);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-115457: Support splitting and replication of micro ops. (GH-115558)
2024-02-20 10:50:59 +00:00
break;
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
case _LOAD_FAST_2_r23: {
CHECK_CURRENT_CACHED_VALUES(2);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
_PyStackRef value;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef _stack_item_0 = _tos_cache0;
_PyStackRef _stack_item_1 = _tos_cache1;
oparg = 2;
assert(oparg == CURRENT_OPARG());
gh-117139: Fix a few wrong steals in bytecodes.c (GH-121127) Fix a few wrong steals in bytecodes.c
2024-06-29 02:14:48 +08:00
assert(!PyStackRef_IsNull(GETLOCAL(oparg)));
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
value = PyStackRef_DUP(GETLOCAL(oparg));
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_tos_cache2 = value;
_tos_cache1 = _stack_item_1;
_tos_cache0 = _stack_item_0;
SET_CURRENT_CACHED_VALUES(3);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-104584: Baby steps towards generating and executing traces (#105924) Added a new, experimental, tracing optimizer and interpreter (a.k.a. "tier 2"). This currently pessimizes, so don't use yet -- this is infrastructure so we can experiment with optimizing passes. To enable it, pass ``-Xuops`` or set ``PYTHONUOPS=1``. To get debug output, set ``PYTHONUOPSDEBUG=N`` where ``N`` is a debug level (0-4, where 0 is no debug output and 4 is excessively verbose). All of this code is likely to change dramatically before the 3.13 feature freeze. But this is a first step.
2023-06-26 19:02:57 -07:00
break;
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
case _LOAD_FAST_3_r01: {
CHECK_CURRENT_CACHED_VALUES(0);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-130704: Strength reduce `LOAD_FAST{_LOAD_FAST}` (#130708) Optimize `LOAD_FAST` opcodes into faster versions that load borrowed references onto the operand stack when we can prove that the lifetime of the local outlives the lifetime of the temporary that is loaded onto the stack.
2025-04-01 10:18:42 -07:00
_PyStackRef value;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
oparg = 3;
gh-130704: Strength reduce `LOAD_FAST{_LOAD_FAST}` (#130708) Optimize `LOAD_FAST` opcodes into faster versions that load borrowed references onto the operand stack when we can prove that the lifetime of the local outlives the lifetime of the temporary that is loaded onto the stack.
2025-04-01 10:18:42 -07:00
assert(oparg == CURRENT_OPARG());
assert(!PyStackRef_IsNull(GETLOCAL(oparg)));
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
value = PyStackRef_DUP(GETLOCAL(oparg));
_tos_cache0 = value;
SET_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-130704: Strength reduce `LOAD_FAST{_LOAD_FAST}` (#130708) Optimize `LOAD_FAST` opcodes into faster versions that load borrowed references onto the operand stack when we can prove that the lifetime of the local outlives the lifetime of the temporary that is loaded onto the stack.
2025-04-01 10:18:42 -07:00
break;
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
case _LOAD_FAST_3_r12: {
CHECK_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-130704: Strength reduce `LOAD_FAST{_LOAD_FAST}` (#130708) Optimize `LOAD_FAST` opcodes into faster versions that load borrowed references onto the operand stack when we can prove that the lifetime of the local outlives the lifetime of the temporary that is loaded onto the stack.
2025-04-01 10:18:42 -07:00
_PyStackRef value;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef _stack_item_0 = _tos_cache0;
oparg = 3;
gh-130704: Strength reduce `LOAD_FAST{_LOAD_FAST}` (#130708) Optimize `LOAD_FAST` opcodes into faster versions that load borrowed references onto the operand stack when we can prove that the lifetime of the local outlives the lifetime of the temporary that is loaded onto the stack.
2025-04-01 10:18:42 -07:00
assert(oparg == CURRENT_OPARG());
assert(!PyStackRef_IsNull(GETLOCAL(oparg)));
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
value = PyStackRef_DUP(GETLOCAL(oparg));
_tos_cache1 = value;
_tos_cache0 = _stack_item_0;
SET_CURRENT_CACHED_VALUES(2);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-130704: Strength reduce `LOAD_FAST{_LOAD_FAST}` (#130708) Optimize `LOAD_FAST` opcodes into faster versions that load borrowed references onto the operand stack when we can prove that the lifetime of the local outlives the lifetime of the temporary that is loaded onto the stack.
2025-04-01 10:18:42 -07:00
break;
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
case _LOAD_FAST_3_r23: {
CHECK_CURRENT_CACHED_VALUES(2);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-130704: Strength reduce `LOAD_FAST{_LOAD_FAST}` (#130708) Optimize `LOAD_FAST` opcodes into faster versions that load borrowed references onto the operand stack when we can prove that the lifetime of the local outlives the lifetime of the temporary that is loaded onto the stack.
2025-04-01 10:18:42 -07:00
_PyStackRef value;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef _stack_item_0 = _tos_cache0;
_PyStackRef _stack_item_1 = _tos_cache1;
oparg = 3;
gh-130704: Strength reduce `LOAD_FAST{_LOAD_FAST}` (#130708) Optimize `LOAD_FAST` opcodes into faster versions that load borrowed references onto the operand stack when we can prove that the lifetime of the local outlives the lifetime of the temporary that is loaded onto the stack.
2025-04-01 10:18:42 -07:00
assert(oparg == CURRENT_OPARG());
assert(!PyStackRef_IsNull(GETLOCAL(oparg)));
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
value = PyStackRef_DUP(GETLOCAL(oparg));
_tos_cache2 = value;
_tos_cache1 = _stack_item_1;
_tos_cache0 = _stack_item_0;
SET_CURRENT_CACHED_VALUES(3);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-130704: Strength reduce `LOAD_FAST{_LOAD_FAST}` (#130708) Optimize `LOAD_FAST` opcodes into faster versions that load borrowed references onto the operand stack when we can prove that the lifetime of the local outlives the lifetime of the temporary that is loaded onto the stack.
2025-04-01 10:18:42 -07:00
break;
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
case _LOAD_FAST_4_r01: {
CHECK_CURRENT_CACHED_VALUES(0);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-130704: Strength reduce `LOAD_FAST{_LOAD_FAST}` (#130708) Optimize `LOAD_FAST` opcodes into faster versions that load borrowed references onto the operand stack when we can prove that the lifetime of the local outlives the lifetime of the temporary that is loaded onto the stack.
2025-04-01 10:18:42 -07:00
_PyStackRef value;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
oparg = 4;
gh-130704: Strength reduce `LOAD_FAST{_LOAD_FAST}` (#130708) Optimize `LOAD_FAST` opcodes into faster versions that load borrowed references onto the operand stack when we can prove that the lifetime of the local outlives the lifetime of the temporary that is loaded onto the stack.
2025-04-01 10:18:42 -07:00
assert(oparg == CURRENT_OPARG());
assert(!PyStackRef_IsNull(GETLOCAL(oparg)));
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
value = PyStackRef_DUP(GETLOCAL(oparg));
_tos_cache0 = value;
SET_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-130704: Strength reduce `LOAD_FAST{_LOAD_FAST}` (#130708) Optimize `LOAD_FAST` opcodes into faster versions that load borrowed references onto the operand stack when we can prove that the lifetime of the local outlives the lifetime of the temporary that is loaded onto the stack.
2025-04-01 10:18:42 -07:00
break;
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
case _LOAD_FAST_4_r12: {
CHECK_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-130704: Strength reduce `LOAD_FAST{_LOAD_FAST}` (#130708) Optimize `LOAD_FAST` opcodes into faster versions that load borrowed references onto the operand stack when we can prove that the lifetime of the local outlives the lifetime of the temporary that is loaded onto the stack.
2025-04-01 10:18:42 -07:00
_PyStackRef value;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef _stack_item_0 = _tos_cache0;
gh-130704: Strength reduce `LOAD_FAST{_LOAD_FAST}` (#130708) Optimize `LOAD_FAST` opcodes into faster versions that load borrowed references onto the operand stack when we can prove that the lifetime of the local outlives the lifetime of the temporary that is loaded onto the stack.
2025-04-01 10:18:42 -07:00
oparg = 4;
assert(oparg == CURRENT_OPARG());
assert(!PyStackRef_IsNull(GETLOCAL(oparg)));
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
value = PyStackRef_DUP(GETLOCAL(oparg));
_tos_cache1 = value;
_tos_cache0 = _stack_item_0;
SET_CURRENT_CACHED_VALUES(2);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-130704: Strength reduce `LOAD_FAST{_LOAD_FAST}` (#130708) Optimize `LOAD_FAST` opcodes into faster versions that load borrowed references onto the operand stack when we can prove that the lifetime of the local outlives the lifetime of the temporary that is loaded onto the stack.
2025-04-01 10:18:42 -07:00
break;
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
case _LOAD_FAST_4_r23: {
CHECK_CURRENT_CACHED_VALUES(2);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-130704: Strength reduce `LOAD_FAST{_LOAD_FAST}` (#130708) Optimize `LOAD_FAST` opcodes into faster versions that load borrowed references onto the operand stack when we can prove that the lifetime of the local outlives the lifetime of the temporary that is loaded onto the stack.
2025-04-01 10:18:42 -07:00
_PyStackRef value;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef _stack_item_0 = _tos_cache0;
_PyStackRef _stack_item_1 = _tos_cache1;
oparg = 4;
gh-130704: Strength reduce `LOAD_FAST{_LOAD_FAST}` (#130708) Optimize `LOAD_FAST` opcodes into faster versions that load borrowed references onto the operand stack when we can prove that the lifetime of the local outlives the lifetime of the temporary that is loaded onto the stack.
2025-04-01 10:18:42 -07:00
assert(oparg == CURRENT_OPARG());
assert(!PyStackRef_IsNull(GETLOCAL(oparg)));
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
value = PyStackRef_DUP(GETLOCAL(oparg));
_tos_cache2 = value;
_tos_cache1 = _stack_item_1;
_tos_cache0 = _stack_item_0;
SET_CURRENT_CACHED_VALUES(3);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-130704: Strength reduce `LOAD_FAST{_LOAD_FAST}` (#130708) Optimize `LOAD_FAST` opcodes into faster versions that load borrowed references onto the operand stack when we can prove that the lifetime of the local outlives the lifetime of the temporary that is loaded onto the stack.
2025-04-01 10:18:42 -07:00
break;
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
case _LOAD_FAST_5_r01: {
CHECK_CURRENT_CACHED_VALUES(0);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-130704: Strength reduce `LOAD_FAST{_LOAD_FAST}` (#130708) Optimize `LOAD_FAST` opcodes into faster versions that load borrowed references onto the operand stack when we can prove that the lifetime of the local outlives the lifetime of the temporary that is loaded onto the stack.
2025-04-01 10:18:42 -07:00
_PyStackRef value;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
oparg = 5;
gh-130704: Strength reduce `LOAD_FAST{_LOAD_FAST}` (#130708) Optimize `LOAD_FAST` opcodes into faster versions that load borrowed references onto the operand stack when we can prove that the lifetime of the local outlives the lifetime of the temporary that is loaded onto the stack.
2025-04-01 10:18:42 -07:00
assert(oparg == CURRENT_OPARG());
assert(!PyStackRef_IsNull(GETLOCAL(oparg)));
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
value = PyStackRef_DUP(GETLOCAL(oparg));
_tos_cache0 = value;
SET_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-130704: Strength reduce `LOAD_FAST{_LOAD_FAST}` (#130708) Optimize `LOAD_FAST` opcodes into faster versions that load borrowed references onto the operand stack when we can prove that the lifetime of the local outlives the lifetime of the temporary that is loaded onto the stack.
2025-04-01 10:18:42 -07:00
break;
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
case _LOAD_FAST_5_r12: {
CHECK_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-130704: Strength reduce `LOAD_FAST{_LOAD_FAST}` (#130708) Optimize `LOAD_FAST` opcodes into faster versions that load borrowed references onto the operand stack when we can prove that the lifetime of the local outlives the lifetime of the temporary that is loaded onto the stack.
2025-04-01 10:18:42 -07:00
_PyStackRef value;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef _stack_item_0 = _tos_cache0;
oparg = 5;
gh-130704: Strength reduce `LOAD_FAST{_LOAD_FAST}` (#130708) Optimize `LOAD_FAST` opcodes into faster versions that load borrowed references onto the operand stack when we can prove that the lifetime of the local outlives the lifetime of the temporary that is loaded onto the stack.
2025-04-01 10:18:42 -07:00
assert(oparg == CURRENT_OPARG());
assert(!PyStackRef_IsNull(GETLOCAL(oparg)));
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
value = PyStackRef_DUP(GETLOCAL(oparg));
_tos_cache1 = value;
_tos_cache0 = _stack_item_0;
SET_CURRENT_CACHED_VALUES(2);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-130704: Strength reduce `LOAD_FAST{_LOAD_FAST}` (#130708) Optimize `LOAD_FAST` opcodes into faster versions that load borrowed references onto the operand stack when we can prove that the lifetime of the local outlives the lifetime of the temporary that is loaded onto the stack.
2025-04-01 10:18:42 -07:00
break;
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
case _LOAD_FAST_5_r23: {
CHECK_CURRENT_CACHED_VALUES(2);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-130704: Strength reduce `LOAD_FAST{_LOAD_FAST}` (#130708) Optimize `LOAD_FAST` opcodes into faster versions that load borrowed references onto the operand stack when we can prove that the lifetime of the local outlives the lifetime of the temporary that is loaded onto the stack.
2025-04-01 10:18:42 -07:00
_PyStackRef value;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef _stack_item_0 = _tos_cache0;
_PyStackRef _stack_item_1 = _tos_cache1;
oparg = 5;
assert(oparg == CURRENT_OPARG());
gh-130704: Strength reduce `LOAD_FAST{_LOAD_FAST}` (#130708) Optimize `LOAD_FAST` opcodes into faster versions that load borrowed references onto the operand stack when we can prove that the lifetime of the local outlives the lifetime of the temporary that is loaded onto the stack.
2025-04-01 10:18:42 -07:00
assert(!PyStackRef_IsNull(GETLOCAL(oparg)));
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
value = PyStackRef_DUP(GETLOCAL(oparg));
_tos_cache2 = value;
_tos_cache1 = _stack_item_1;
_tos_cache0 = _stack_item_0;
SET_CURRENT_CACHED_VALUES(3);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-130704: Strength reduce `LOAD_FAST{_LOAD_FAST}` (#130708) Optimize `LOAD_FAST` opcodes into faster versions that load borrowed references onto the operand stack when we can prove that the lifetime of the local outlives the lifetime of the temporary that is loaded onto the stack.
2025-04-01 10:18:42 -07:00
break;
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
case _LOAD_FAST_6_r01: {
CHECK_CURRENT_CACHED_VALUES(0);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
_PyStackRef value;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
oparg = 6;
assert(oparg == CURRENT_OPARG());
assert(!PyStackRef_IsNull(GETLOCAL(oparg)));
value = PyStackRef_DUP(GETLOCAL(oparg));
_tos_cache0 = value;
SET_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-104584: Baby steps towards generating and executing traces (#105924) Added a new, experimental, tracing optimizer and interpreter (a.k.a. "tier 2"). This currently pessimizes, so don't use yet -- this is infrastructure so we can experiment with optimizing passes. To enable it, pass ``-Xuops`` or set ``PYTHONUOPS=1``. To get debug output, set ``PYTHONUOPSDEBUG=N`` where ``N`` is a debug level (0-4, where 0 is no debug output and 4 is excessively verbose). All of this code is likely to change dramatically before the 3.13 feature freeze. But this is a first step.
2023-06-26 19:02:57 -07:00
break;
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
case _LOAD_FAST_6_r12: {
CHECK_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-125837: Split `LOAD_CONST` into three. (GH-125972) * Add LOAD_CONST_IMMORTAL opcode * Add LOAD_SMALL_INT opcode * Remove RETURN_CONST opcode
2024-10-29 11:15:42 +00:00
_PyStackRef value;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef _stack_item_0 = _tos_cache0;
oparg = 6;
assert(oparg == CURRENT_OPARG());
assert(!PyStackRef_IsNull(GETLOCAL(oparg)));
value = PyStackRef_DUP(GETLOCAL(oparg));
_tos_cache1 = value;
_tos_cache0 = _stack_item_0;
SET_CURRENT_CACHED_VALUES(2);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-125837: Split `LOAD_CONST` into three. (GH-125972) * Add LOAD_CONST_IMMORTAL opcode * Add LOAD_SMALL_INT opcode * Remove RETURN_CONST opcode
2024-10-29 11:15:42 +00:00
break;
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
case _LOAD_FAST_6_r23: {
CHECK_CURRENT_CACHED_VALUES(2);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-125837: Split `LOAD_CONST` into three. (GH-125972) * Add LOAD_CONST_IMMORTAL opcode * Add LOAD_SMALL_INT opcode * Remove RETURN_CONST opcode
2024-10-29 11:15:42 +00:00
_PyStackRef value;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef _stack_item_0 = _tos_cache0;
_PyStackRef _stack_item_1 = _tos_cache1;
oparg = 6;
GH-125837: Split `LOAD_CONST` into three. (GH-125972) * Add LOAD_CONST_IMMORTAL opcode * Add LOAD_SMALL_INT opcode * Remove RETURN_CONST opcode
2024-10-29 11:15:42 +00:00
assert(oparg == CURRENT_OPARG());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
assert(!PyStackRef_IsNull(GETLOCAL(oparg)));
value = PyStackRef_DUP(GETLOCAL(oparg));
_tos_cache2 = value;
_tos_cache1 = _stack_item_1;
_tos_cache0 = _stack_item_0;
SET_CURRENT_CACHED_VALUES(3);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-125837: Split `LOAD_CONST` into three. (GH-125972) * Add LOAD_CONST_IMMORTAL opcode * Add LOAD_SMALL_INT opcode * Remove RETURN_CONST opcode
2024-10-29 11:15:42 +00:00
break;
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
case _LOAD_FAST_7_r01: {
CHECK_CURRENT_CACHED_VALUES(0);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-125837: Split `LOAD_CONST` into three. (GH-125972) * Add LOAD_CONST_IMMORTAL opcode * Add LOAD_SMALL_INT opcode * Remove RETURN_CONST opcode
2024-10-29 11:15:42 +00:00
_PyStackRef value;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
oparg = 7;
GH-125837: Split `LOAD_CONST` into three. (GH-125972) * Add LOAD_CONST_IMMORTAL opcode * Add LOAD_SMALL_INT opcode * Remove RETURN_CONST opcode
2024-10-29 11:15:42 +00:00
assert(oparg == CURRENT_OPARG());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
assert(!PyStackRef_IsNull(GETLOCAL(oparg)));
value = PyStackRef_DUP(GETLOCAL(oparg));
_tos_cache0 = value;
SET_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-125837: Split `LOAD_CONST` into three. (GH-125972) * Add LOAD_CONST_IMMORTAL opcode * Add LOAD_SMALL_INT opcode * Remove RETURN_CONST opcode
2024-10-29 11:15:42 +00:00
break;
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
case _LOAD_FAST_7_r12: {
CHECK_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-125837: Split `LOAD_CONST` into three. (GH-125972) * Add LOAD_CONST_IMMORTAL opcode * Add LOAD_SMALL_INT opcode * Remove RETURN_CONST opcode
2024-10-29 11:15:42 +00:00
_PyStackRef value;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef _stack_item_0 = _tos_cache0;
oparg = 7;
GH-125837: Split `LOAD_CONST` into three. (GH-125972) * Add LOAD_CONST_IMMORTAL opcode * Add LOAD_SMALL_INT opcode * Remove RETURN_CONST opcode
2024-10-29 11:15:42 +00:00
assert(oparg == CURRENT_OPARG());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
assert(!PyStackRef_IsNull(GETLOCAL(oparg)));
value = PyStackRef_DUP(GETLOCAL(oparg));
_tos_cache1 = value;
_tos_cache0 = _stack_item_0;
SET_CURRENT_CACHED_VALUES(2);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-125837: Split `LOAD_CONST` into three. (GH-125972) * Add LOAD_CONST_IMMORTAL opcode * Add LOAD_SMALL_INT opcode * Remove RETURN_CONST opcode
2024-10-29 11:15:42 +00:00
break;
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
case _LOAD_FAST_7_r23: {
CHECK_CURRENT_CACHED_VALUES(2);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-125837: Split `LOAD_CONST` into three. (GH-125972) * Add LOAD_CONST_IMMORTAL opcode * Add LOAD_SMALL_INT opcode * Remove RETURN_CONST opcode
2024-10-29 11:15:42 +00:00
_PyStackRef value;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef _stack_item_0 = _tos_cache0;
_PyStackRef _stack_item_1 = _tos_cache1;
oparg = 7;
GH-125837: Split `LOAD_CONST` into three. (GH-125972) * Add LOAD_CONST_IMMORTAL opcode * Add LOAD_SMALL_INT opcode * Remove RETURN_CONST opcode
2024-10-29 11:15:42 +00:00
assert(oparg == CURRENT_OPARG());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
assert(!PyStackRef_IsNull(GETLOCAL(oparg)));
value = PyStackRef_DUP(GETLOCAL(oparg));
_tos_cache2 = value;
_tos_cache1 = _stack_item_1;
_tos_cache0 = _stack_item_0;
SET_CURRENT_CACHED_VALUES(3);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-125837: Split `LOAD_CONST` into three. (GH-125972) * Add LOAD_CONST_IMMORTAL opcode * Add LOAD_SMALL_INT opcode * Remove RETURN_CONST opcode
2024-10-29 11:15:42 +00:00
break;
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
case _LOAD_FAST_r01: {
CHECK_CURRENT_CACHED_VALUES(0);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-125837: Split `LOAD_CONST` into three. (GH-125972) * Add LOAD_CONST_IMMORTAL opcode * Add LOAD_SMALL_INT opcode * Remove RETURN_CONST opcode
2024-10-29 11:15:42 +00:00
_PyStackRef value;
oparg = CURRENT_OPARG();
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
assert(!PyStackRef_IsNull(GETLOCAL(oparg)));
value = PyStackRef_DUP(GETLOCAL(oparg));
_tos_cache0 = value;
SET_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-104584: Baby steps towards generating and executing traces (#105924) Added a new, experimental, tracing optimizer and interpreter (a.k.a. "tier 2"). This currently pessimizes, so don't use yet -- this is infrastructure so we can experiment with optimizing passes. To enable it, pass ``-Xuops`` or set ``PYTHONUOPS=1``. To get debug output, set ``PYTHONUOPSDEBUG=N`` where ``N`` is a debug level (0-4, where 0 is no debug output and 4 is excessively verbose). All of this code is likely to change dramatically before the 3.13 feature freeze. But this is a first step.
2023-06-26 19:02:57 -07:00
break;
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
case _LOAD_FAST_r12: {
CHECK_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
_PyStackRef value;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef _stack_item_0 = _tos_cache0;
oparg = CURRENT_OPARG();
assert(!PyStackRef_IsNull(GETLOCAL(oparg)));
value = PyStackRef_DUP(GETLOCAL(oparg));
_tos_cache1 = value;
_tos_cache0 = _stack_item_0;
SET_CURRENT_CACHED_VALUES(2);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-115457: Support splitting and replication of micro ops. (GH-115558)
2024-02-20 10:50:59 +00:00
break;
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
case _LOAD_FAST_r23: {
CHECK_CURRENT_CACHED_VALUES(2);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
_PyStackRef value;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef _stack_item_0 = _tos_cache0;
_PyStackRef _stack_item_1 = _tos_cache1;
oparg = CURRENT_OPARG();
assert(!PyStackRef_IsNull(GETLOCAL(oparg)));
value = PyStackRef_DUP(GETLOCAL(oparg));
_tos_cache2 = value;
_tos_cache1 = _stack_item_1;
_tos_cache0 = _stack_item_0;
SET_CURRENT_CACHED_VALUES(3);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-115457: Support splitting and replication of micro ops. (GH-115558)
2024-02-20 10:50:59 +00:00
break;
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
case _LOAD_FAST_BORROW_0_r01: {
CHECK_CURRENT_CACHED_VALUES(0);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
_PyStackRef value;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
oparg = 0;
GH-115457: Support splitting and replication of micro ops. (GH-115558)
2024-02-20 10:50:59 +00:00
assert(oparg == CURRENT_OPARG());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
assert(!PyStackRef_IsNull(GETLOCAL(oparg)));
value = PyStackRef_Borrow(GETLOCAL(oparg));
_tos_cache0 = value;
SET_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-115457: Support splitting and replication of micro ops. (GH-115558)
2024-02-20 10:50:59 +00:00
break;
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
case _LOAD_FAST_BORROW_0_r12: {
CHECK_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
_PyStackRef value;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef _stack_item_0 = _tos_cache0;
oparg = 0;
GH-115457: Support splitting and replication of micro ops. (GH-115558)
2024-02-20 10:50:59 +00:00
assert(oparg == CURRENT_OPARG());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
assert(!PyStackRef_IsNull(GETLOCAL(oparg)));
value = PyStackRef_Borrow(GETLOCAL(oparg));
_tos_cache1 = value;
_tos_cache0 = _stack_item_0;
SET_CURRENT_CACHED_VALUES(2);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-115457: Support splitting and replication of micro ops. (GH-115558)
2024-02-20 10:50:59 +00:00
break;
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
case _LOAD_FAST_BORROW_0_r23: {
CHECK_CURRENT_CACHED_VALUES(2);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
_PyStackRef value;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef _stack_item_0 = _tos_cache0;
_PyStackRef _stack_item_1 = _tos_cache1;
oparg = 0;
GH-115457: Support splitting and replication of micro ops. (GH-115558)
2024-02-20 10:50:59 +00:00
assert(oparg == CURRENT_OPARG());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
assert(!PyStackRef_IsNull(GETLOCAL(oparg)));
value = PyStackRef_Borrow(GETLOCAL(oparg));
_tos_cache2 = value;
_tos_cache1 = _stack_item_1;
_tos_cache0 = _stack_item_0;
SET_CURRENT_CACHED_VALUES(3);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-115457: Support splitting and replication of micro ops. (GH-115558)
2024-02-20 10:50:59 +00:00
break;
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
case _LOAD_FAST_BORROW_1_r01: {
CHECK_CURRENT_CACHED_VALUES(0);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
_PyStackRef value;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
oparg = 1;
GH-115457: Support splitting and replication of micro ops. (GH-115558)
2024-02-20 10:50:59 +00:00
assert(oparg == CURRENT_OPARG());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
assert(!PyStackRef_IsNull(GETLOCAL(oparg)));
value = PyStackRef_Borrow(GETLOCAL(oparg));
_tos_cache0 = value;
SET_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-115457: Support splitting and replication of micro ops. (GH-115558)
2024-02-20 10:50:59 +00:00
break;
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
case _LOAD_FAST_BORROW_1_r12: {
CHECK_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
_PyStackRef value;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef _stack_item_0 = _tos_cache0;
oparg = 1;
GH-115457: Support splitting and replication of micro ops. (GH-115558)
2024-02-20 10:50:59 +00:00
assert(oparg == CURRENT_OPARG());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
assert(!PyStackRef_IsNull(GETLOCAL(oparg)));
value = PyStackRef_Borrow(GETLOCAL(oparg));
_tos_cache1 = value;
_tos_cache0 = _stack_item_0;
SET_CURRENT_CACHED_VALUES(2);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-115457: Support splitting and replication of micro ops. (GH-115558)
2024-02-20 10:50:59 +00:00
break;
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
case _LOAD_FAST_BORROW_1_r23: {
CHECK_CURRENT_CACHED_VALUES(2);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
_PyStackRef value;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef _stack_item_0 = _tos_cache0;
_PyStackRef _stack_item_1 = _tos_cache1;
oparg = 1;
GH-115457: Support splitting and replication of micro ops. (GH-115558)
2024-02-20 10:50:59 +00:00
assert(oparg == CURRENT_OPARG());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
assert(!PyStackRef_IsNull(GETLOCAL(oparg)));
value = PyStackRef_Borrow(GETLOCAL(oparg));
_tos_cache2 = value;
_tos_cache1 = _stack_item_1;
_tos_cache0 = _stack_item_0;
SET_CURRENT_CACHED_VALUES(3);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-115457: Support splitting and replication of micro ops. (GH-115558)
2024-02-20 10:50:59 +00:00
break;
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
case _LOAD_FAST_BORROW_2_r01: {
CHECK_CURRENT_CACHED_VALUES(0);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
_PyStackRef value;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
oparg = 2;
assert(oparg == CURRENT_OPARG());
assert(!PyStackRef_IsNull(GETLOCAL(oparg)));
value = PyStackRef_Borrow(GETLOCAL(oparg));
_tos_cache0 = value;
SET_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-104584: Baby steps towards generating and executing traces (#105924) Added a new, experimental, tracing optimizer and interpreter (a.k.a. "tier 2"). This currently pessimizes, so don't use yet -- this is infrastructure so we can experiment with optimizing passes. To enable it, pass ``-Xuops`` or set ``PYTHONUOPS=1``. To get debug output, set ``PYTHONUOPSDEBUG=N`` where ``N`` is a debug level (0-4, where 0 is no debug output and 4 is excessively verbose). All of this code is likely to change dramatically before the 3.13 feature freeze. But this is a first step.
2023-06-26 19:02:57 -07:00
break;
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
case _LOAD_FAST_BORROW_2_r12: {
CHECK_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
_PyStackRef value;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef _stack_item_0 = _tos_cache0;
oparg = 2;
assert(oparg == CURRENT_OPARG());
assert(!PyStackRef_IsNull(GETLOCAL(oparg)));
value = PyStackRef_Borrow(GETLOCAL(oparg));
_tos_cache1 = value;
_tos_cache0 = _stack_item_0;
SET_CURRENT_CACHED_VALUES(2);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-104584: Baby steps towards generating and executing traces (#105924) Added a new, experimental, tracing optimizer and interpreter (a.k.a. "tier 2"). This currently pessimizes, so don't use yet -- this is infrastructure so we can experiment with optimizing passes. To enable it, pass ``-Xuops`` or set ``PYTHONUOPS=1``. To get debug output, set ``PYTHONUOPSDEBUG=N`` where ``N`` is a debug level (0-4, where 0 is no debug output and 4 is excessively verbose). All of this code is likely to change dramatically before the 3.13 feature freeze. But this is a first step.
2023-06-26 19:02:57 -07:00
break;
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
case _LOAD_FAST_BORROW_2_r23: {
CHECK_CURRENT_CACHED_VALUES(2);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-134584: Specialize POP_TOP by reference and type in JIT (GH-135761)
2025-06-24 00:57:14 +08:00
_PyStackRef value;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef _stack_item_0 = _tos_cache0;
_PyStackRef _stack_item_1 = _tos_cache1;
oparg = 2;
assert(oparg == CURRENT_OPARG());
assert(!PyStackRef_IsNull(GETLOCAL(oparg)));
value = PyStackRef_Borrow(GETLOCAL(oparg));
_tos_cache2 = value;
_tos_cache1 = _stack_item_1;
_tos_cache0 = _stack_item_0;
SET_CURRENT_CACHED_VALUES(3);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-134584: Specialize POP_TOP by reference and type in JIT (GH-135761)
2025-06-24 00:57:14 +08:00
break;
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
case _LOAD_FAST_BORROW_3_r01: {
CHECK_CURRENT_CACHED_VALUES(0);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-134584: Specialize POP_TOP by reference and type in JIT (GH-135761)
2025-06-24 00:57:14 +08:00
_PyStackRef value;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
oparg = 3;
assert(oparg == CURRENT_OPARG());
assert(!PyStackRef_IsNull(GETLOCAL(oparg)));
value = PyStackRef_Borrow(GETLOCAL(oparg));
_tos_cache0 = value;
SET_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-134584: Specialize POP_TOP by reference and type in JIT (GH-135761)
2025-06-24 00:57:14 +08:00
break;
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
case _LOAD_FAST_BORROW_3_r12: {
CHECK_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-134584: Specialize POP_TOP by reference and type in JIT (GH-135761)
2025-06-24 00:57:14 +08:00
_PyStackRef value;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef _stack_item_0 = _tos_cache0;
oparg = 3;
assert(oparg == CURRENT_OPARG());
assert(!PyStackRef_IsNull(GETLOCAL(oparg)));
value = PyStackRef_Borrow(GETLOCAL(oparg));
_tos_cache1 = value;
_tos_cache0 = _stack_item_0;
SET_CURRENT_CACHED_VALUES(2);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-134584: Specialize POP_TOP by reference and type in JIT (GH-135761)
2025-06-24 00:57:14 +08:00
break;
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
case _LOAD_FAST_BORROW_3_r23: {
CHECK_CURRENT_CACHED_VALUES(2);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-134584: Specialize POP_TOP by reference and type in JIT (GH-135761)
2025-06-24 00:57:14 +08:00
_PyStackRef value;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef _stack_item_0 = _tos_cache0;
_PyStackRef _stack_item_1 = _tos_cache1;
oparg = 3;
assert(oparg == CURRENT_OPARG());
assert(!PyStackRef_IsNull(GETLOCAL(oparg)));
value = PyStackRef_Borrow(GETLOCAL(oparg));
_tos_cache2 = value;
_tos_cache1 = _stack_item_1;
_tos_cache0 = _stack_item_0;
SET_CURRENT_CACHED_VALUES(3);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-134584: Specialize POP_TOP by reference and type in JIT (GH-135761)
2025-06-24 00:57:14 +08:00
break;
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
case _LOAD_FAST_BORROW_4_r01: {
CHECK_CURRENT_CACHED_VALUES(0);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef value;
oparg = 4;
assert(oparg == CURRENT_OPARG());
assert(!PyStackRef_IsNull(GETLOCAL(oparg)));
value = PyStackRef_Borrow(GETLOCAL(oparg));
_tos_cache0 = value;
SET_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-131798: Narrow types more aggressively in the JIT (GH-134373)
2025-05-20 18:09:51 -04:00
break;
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
case _LOAD_FAST_BORROW_4_r12: {
CHECK_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef value;
_PyStackRef _stack_item_0 = _tos_cache0;
oparg = 4;
assert(oparg == CURRENT_OPARG());
assert(!PyStackRef_IsNull(GETLOCAL(oparg)));
value = PyStackRef_Borrow(GETLOCAL(oparg));
_tos_cache1 = value;
_tos_cache0 = _stack_item_0;
SET_CURRENT_CACHED_VALUES(2);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-104584: Baby steps towards generating and executing traces (#105924) Added a new, experimental, tracing optimizer and interpreter (a.k.a. "tier 2"). This currently pessimizes, so don't use yet -- this is infrastructure so we can experiment with optimizing passes. To enable it, pass ``-Xuops`` or set ``PYTHONUOPS=1``. To get debug output, set ``PYTHONUOPSDEBUG=N`` where ``N`` is a debug level (0-4, where 0 is no debug output and 4 is excessively verbose). All of this code is likely to change dramatically before the 3.13 feature freeze. But this is a first step.
2023-06-26 19:02:57 -07:00
break;
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
case _LOAD_FAST_BORROW_4_r23: {
CHECK_CURRENT_CACHED_VALUES(2);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-128375: Better instrument for `FOR_ITER` (GH-128445)
2025-01-06 17:54:47 +00:00
_PyStackRef value;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef _stack_item_0 = _tos_cache0;
_PyStackRef _stack_item_1 = _tos_cache1;
oparg = 4;
assert(oparg == CURRENT_OPARG());
assert(!PyStackRef_IsNull(GETLOCAL(oparg)));
value = PyStackRef_Borrow(GETLOCAL(oparg));
_tos_cache2 = value;
_tos_cache1 = _stack_item_1;
_tos_cache0 = _stack_item_0;
SET_CURRENT_CACHED_VALUES(3);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-128375: Better instrument for `FOR_ITER` (GH-128445)
2025-01-06 17:54:47 +00:00
break;
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
case _LOAD_FAST_BORROW_5_r01: {
CHECK_CURRENT_CACHED_VALUES(0);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef value;
oparg = 5;
assert(oparg == CURRENT_OPARG());
assert(!PyStackRef_IsNull(GETLOCAL(oparg)));
value = PyStackRef_Borrow(GETLOCAL(oparg));
_tos_cache0 = value;
SET_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-132554: "Virtual" iterators (GH-132555) * FOR_ITER now pushes either the iterator and NULL or leaves the iterable and pushes tagged zero * NEXT_ITER uses the tagged int as the index into the sequence or, if TOS is NULL, iterates as before.
2025-05-27 15:59:45 +01:00
break;
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
case _LOAD_FAST_BORROW_5_r12: {
CHECK_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
_PyStackRef value;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef _stack_item_0 = _tos_cache0;
oparg = 5;
assert(oparg == CURRENT_OPARG());
assert(!PyStackRef_IsNull(GETLOCAL(oparg)));
value = PyStackRef_Borrow(GETLOCAL(oparg));
_tos_cache1 = value;
_tos_cache0 = _stack_item_0;
SET_CURRENT_CACHED_VALUES(2);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-104584: Baby steps towards generating and executing traces (#105924) Added a new, experimental, tracing optimizer and interpreter (a.k.a. "tier 2"). This currently pessimizes, so don't use yet -- this is infrastructure so we can experiment with optimizing passes. To enable it, pass ``-Xuops`` or set ``PYTHONUOPS=1``. To get debug output, set ``PYTHONUOPSDEBUG=N`` where ``N`` is a debug level (0-4, where 0 is no debug output and 4 is excessively verbose). All of this code is likely to change dramatically before the 3.13 feature freeze. But this is a first step.
2023-06-26 19:02:57 -07:00
break;
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
case _LOAD_FAST_BORROW_5_r23: {
CHECK_CURRENT_CACHED_VALUES(2);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
_PyStackRef value;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef _stack_item_0 = _tos_cache0;
_PyStackRef _stack_item_1 = _tos_cache1;
oparg = 5;
assert(oparg == CURRENT_OPARG());
assert(!PyStackRef_IsNull(GETLOCAL(oparg)));
value = PyStackRef_Borrow(GETLOCAL(oparg));
_tos_cache2 = value;
_tos_cache1 = _stack_item_1;
_tos_cache0 = _stack_item_0;
SET_CURRENT_CACHED_VALUES(3);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-104584: Baby steps towards generating and executing traces (#105924) Added a new, experimental, tracing optimizer and interpreter (a.k.a. "tier 2"). This currently pessimizes, so don't use yet -- this is infrastructure so we can experiment with optimizing passes. To enable it, pass ``-Xuops`` or set ``PYTHONUOPS=1``. To get debug output, set ``PYTHONUOPSDEBUG=N`` where ``N`` is a debug level (0-4, where 0 is no debug output and 4 is excessively verbose). All of this code is likely to change dramatically before the 3.13 feature freeze. But this is a first step.
2023-06-26 19:02:57 -07:00
break;
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
case _LOAD_FAST_BORROW_6_r01: {
CHECK_CURRENT_CACHED_VALUES(0);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
_PyStackRef value;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
oparg = 6;
assert(oparg == CURRENT_OPARG());
assert(!PyStackRef_IsNull(GETLOCAL(oparg)));
value = PyStackRef_Borrow(GETLOCAL(oparg));
_tos_cache0 = value;
SET_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-106008: Make implicit boolean conversions explicit (GH-106003)
2023-06-29 13:49:54 -07:00
break;
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
case _LOAD_FAST_BORROW_6_r12: {
CHECK_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
_PyStackRef value;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef _stack_item_0 = _tos_cache0;
oparg = 6;
assert(oparg == CURRENT_OPARG());
assert(!PyStackRef_IsNull(GETLOCAL(oparg)));
value = PyStackRef_Borrow(GETLOCAL(oparg));
_tos_cache1 = value;
_tos_cache0 = _stack_item_0;
SET_CURRENT_CACHED_VALUES(2);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-104584: Allow unspecialized instructions in superblocks (#106497) This adds several of unspecialized opcodes to superblocks: TO_BOOL, BINARY_SUBSCR, STORE_SUBSCR, UNPACK_SEQUENCE, LOAD_GLOBAL, LOAD_ATTR, COMPARE_OP, BINARY_OP. While we may not want that eventually, for now this helps finding bugs. There is a rudimentary test checking for UNPACK_SEQUENCE. Once we're ready to undo this, that would be simple: just replace the call to variable_used_unspecialized with a call to variable_used (as shown in a comment). Or add individual opcdes to FORBIDDEN_NAMES_IN_UOPS.
2023-07-07 11:03:27 -07:00
break;
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
case _LOAD_FAST_BORROW_6_r23: {
CHECK_CURRENT_CACHED_VALUES(2);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
_PyStackRef value;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef _stack_item_0 = _tos_cache0;
_PyStackRef _stack_item_1 = _tos_cache1;
oparg = 6;
assert(oparg == CURRENT_OPARG());
assert(!PyStackRef_IsNull(GETLOCAL(oparg)));
value = PyStackRef_Borrow(GETLOCAL(oparg));
_tos_cache2 = value;
_tos_cache1 = _stack_item_1;
_tos_cache0 = _stack_item_0;
SET_CURRENT_CACHED_VALUES(3);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-106008: Make implicit boolean conversions explicit (GH-106003)
2023-06-29 13:49:54 -07:00
break;
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
case _LOAD_FAST_BORROW_7_r01: {
CHECK_CURRENT_CACHED_VALUES(0);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
_PyStackRef value;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
oparg = 7;
assert(oparg == CURRENT_OPARG());
assert(!PyStackRef_IsNull(GETLOCAL(oparg)));
value = PyStackRef_Borrow(GETLOCAL(oparg));
_tos_cache0 = value;
SET_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-106008: Make implicit boolean conversions explicit (GH-106003)
2023-06-29 13:49:54 -07:00
break;
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
case _LOAD_FAST_BORROW_7_r12: {
CHECK_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef value;
_PyStackRef _stack_item_0 = _tos_cache0;
oparg = 7;
assert(oparg == CURRENT_OPARG());
assert(!PyStackRef_IsNull(GETLOCAL(oparg)));
value = PyStackRef_Borrow(GETLOCAL(oparg));
_tos_cache1 = value;
_tos_cache0 = _stack_item_0;
SET_CURRENT_CACHED_VALUES(2);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-131798: Remove JIT guards for dict, frozenset, list, set, and tuple (GH-132289)
2025-04-09 14:32:21 -07:00
break;
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
case _LOAD_FAST_BORROW_7_r23: {
CHECK_CURRENT_CACHED_VALUES(2);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef value;
_PyStackRef _stack_item_0 = _tos_cache0;
_PyStackRef _stack_item_1 = _tos_cache1;
oparg = 7;
assert(oparg == CURRENT_OPARG());
assert(!PyStackRef_IsNull(GETLOCAL(oparg)));
value = PyStackRef_Borrow(GETLOCAL(oparg));
_tos_cache2 = value;
_tos_cache1 = _stack_item_1;
_tos_cache0 = _stack_item_0;
SET_CURRENT_CACHED_VALUES(3);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-131798: Remove JIT guards for dict, frozenset, list, set, and tuple (GH-132289)
2025-04-09 14:32:21 -07:00
break;
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
case _LOAD_FAST_BORROW_r01: {
CHECK_CURRENT_CACHED_VALUES(0);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef value;
oparg = CURRENT_OPARG();
assert(!PyStackRef_IsNull(GETLOCAL(oparg)));
value = PyStackRef_Borrow(GETLOCAL(oparg));
_tos_cache0 = value;
SET_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-100239: specialize BINARY_OP/SUBSCR for list-slice (#132626)
2025-05-01 11:28:52 +01:00
break;
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
case _LOAD_FAST_BORROW_r12: {
CHECK_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
_PyStackRef value;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef _stack_item_0 = _tos_cache0;
oparg = CURRENT_OPARG();
assert(!PyStackRef_IsNull(GETLOCAL(oparg)));
value = PyStackRef_Borrow(GETLOCAL(oparg));
_tos_cache1 = value;
_tos_cache0 = _stack_item_0;
SET_CURRENT_CACHED_VALUES(2);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-106008: Make implicit boolean conversions explicit (GH-106003)
2023-06-29 13:49:54 -07:00
break;
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
case _LOAD_FAST_BORROW_r23: {
CHECK_CURRENT_CACHED_VALUES(2);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
_PyStackRef value;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef _stack_item_0 = _tos_cache0;
_PyStackRef _stack_item_1 = _tos_cache1;
oparg = CURRENT_OPARG();
assert(!PyStackRef_IsNull(GETLOCAL(oparg)));
value = PyStackRef_Borrow(GETLOCAL(oparg));
_tos_cache2 = value;
_tos_cache1 = _stack_item_1;
_tos_cache0 = _stack_item_0;
SET_CURRENT_CACHED_VALUES(3);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-106008: Make implicit boolean conversions explicit (GH-106003)
2023-06-29 13:49:54 -07:00
break;
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
case _LOAD_FAST_AND_CLEAR_r01: {
CHECK_CURRENT_CACHED_VALUES(0);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef value;
oparg = CURRENT_OPARG();
value = GETLOCAL(oparg);
GETLOCAL(oparg) = PyStackRef_NULL;
_tos_cache0 = value;
SET_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-131798: Allow the JIT to remove more int/float/str guards (GH-131800)
2025-04-01 15:10:15 -07:00
break;
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
case _LOAD_FAST_AND_CLEAR_r12: {
CHECK_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
_PyStackRef value;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef _stack_item_0 = _tos_cache0;
oparg = CURRENT_OPARG();
value = GETLOCAL(oparg);
GETLOCAL(oparg) = PyStackRef_NULL;
_tos_cache1 = value;
_tos_cache0 = _stack_item_0;
SET_CURRENT_CACHED_VALUES(2);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-131798: Remove type checks for _TO_BOOL_STR (GH-131816)
2025-03-30 16:07:25 -07:00
break;
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
case _LOAD_FAST_AND_CLEAR_r23: {
CHECK_CURRENT_CACHED_VALUES(2);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-131798: Remove type checks for _TO_BOOL_STR (GH-131816)
2025-03-30 16:07:25 -07:00
_PyStackRef value;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef _stack_item_0 = _tos_cache0;
_PyStackRef _stack_item_1 = _tos_cache1;
oparg = CURRENT_OPARG();
value = GETLOCAL(oparg);
GETLOCAL(oparg) = PyStackRef_NULL;
_tos_cache2 = value;
_tos_cache1 = _stack_item_1;
_tos_cache0 = _stack_item_0;
SET_CURRENT_CACHED_VALUES(3);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-106008: Make implicit boolean conversions explicit (GH-106003)
2023-06-29 13:49:54 -07:00
break;
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
case _LOAD_CONST_r01: {
CHECK_CURRENT_CACHED_VALUES(0);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
_PyStackRef value;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
oparg = CURRENT_OPARG();
PyObject *obj = GETITEM(FRAME_CO_CONSTS, oparg);
value = PyStackRef_FromPyObjectBorrow(obj);
_tos_cache0 = value;
SET_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-104584: Baby steps towards generating and executing traces (#105924) Added a new, experimental, tracing optimizer and interpreter (a.k.a. "tier 2"). This currently pessimizes, so don't use yet -- this is infrastructure so we can experiment with optimizing passes. To enable it, pass ``-Xuops`` or set ``PYTHONUOPS=1``. To get debug output, set ``PYTHONUOPSDEBUG=N`` where ``N`` is a debug level (0-4, where 0 is no debug output and 4 is excessively verbose). All of this code is likely to change dramatically before the 3.13 feature freeze. But this is a first step.
2023-06-26 19:02:57 -07:00
break;
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
case _LOAD_CONST_r12: {
CHECK_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
_PyStackRef value;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef _stack_item_0 = _tos_cache0;
oparg = CURRENT_OPARG();
PyObject *obj = GETITEM(FRAME_CO_CONSTS, oparg);
value = PyStackRef_FromPyObjectBorrow(obj);
_tos_cache1 = value;
_tos_cache0 = _stack_item_0;
SET_CURRENT_CACHED_VALUES(2);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-104584: Baby steps towards generating and executing traces (#105924) Added a new, experimental, tracing optimizer and interpreter (a.k.a. "tier 2"). This currently pessimizes, so don't use yet -- this is infrastructure so we can experiment with optimizing passes. To enable it, pass ``-Xuops`` or set ``PYTHONUOPS=1``. To get debug output, set ``PYTHONUOPSDEBUG=N`` where ``N`` is a debug level (0-4, where 0 is no debug output and 4 is excessively verbose). All of this code is likely to change dramatically before the 3.13 feature freeze. But this is a first step.
2023-06-26 19:02:57 -07:00
break;
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
case _LOAD_CONST_r23: {
CHECK_CURRENT_CACHED_VALUES(2);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef value;
_PyStackRef _stack_item_0 = _tos_cache0;
_PyStackRef _stack_item_1 = _tos_cache1;
oparg = CURRENT_OPARG();
PyObject *obj = GETITEM(FRAME_CO_CONSTS, oparg);
value = PyStackRef_FromPyObjectBorrow(obj);
_tos_cache2 = value;
_tos_cache1 = _stack_item_1;
_tos_cache0 = _stack_item_0;
SET_CURRENT_CACHED_VALUES(3);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-115480: Reduce guard strength for binary ops when type of one operand is known already (GH-118050)
2024-04-22 13:34:06 +01:00
break;
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
case _LOAD_SMALL_INT_0_r01: {
CHECK_CURRENT_CACHED_VALUES(0);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
_PyStackRef value;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
oparg = 0;
assert(oparg == CURRENT_OPARG());
assert(oparg < _PY_NSMALLPOSINTS);
PyObject *obj = (PyObject *)&_PyLong_SMALL_INTS[_PY_NSMALLNEGINTS + oparg];
value = PyStackRef_FromPyObjectBorrow(obj);
_tos_cache0 = value;
SET_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Specialize int operations for compact ints only (GH-135668)
2025-06-19 11:10:29 +01:00
break;
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
case _LOAD_SMALL_INT_0_r12: {
CHECK_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef value;
_PyStackRef _stack_item_0 = _tos_cache0;
oparg = 0;
assert(oparg == CURRENT_OPARG());
assert(oparg < _PY_NSMALLPOSINTS);
PyObject *obj = (PyObject *)&_PyLong_SMALL_INTS[_PY_NSMALLNEGINTS + oparg];
value = PyStackRef_FromPyObjectBorrow(obj);
_tos_cache1 = value;
_tos_cache0 = _stack_item_0;
SET_CURRENT_CACHED_VALUES(2);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Specialize int operations for compact ints only (GH-135668)
2025-06-19 11:10:29 +01:00
break;
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
case _LOAD_SMALL_INT_0_r23: {
CHECK_CURRENT_CACHED_VALUES(2);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Specialize int operations for compact ints only (GH-135668)
2025-06-19 11:10:29 +01:00
_PyStackRef value;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef _stack_item_0 = _tos_cache0;
_PyStackRef _stack_item_1 = _tos_cache1;
oparg = 0;
assert(oparg == CURRENT_OPARG());
assert(oparg < _PY_NSMALLPOSINTS);
PyObject *obj = (PyObject *)&_PyLong_SMALL_INTS[_PY_NSMALLNEGINTS + oparg];
value = PyStackRef_FromPyObjectBorrow(obj);
_tos_cache2 = value;
_tos_cache1 = _stack_item_1;
_tos_cache0 = _stack_item_0;
SET_CURRENT_CACHED_VALUES(3);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _LOAD_SMALL_INT_1_r01: {
CHECK_CURRENT_CACHED_VALUES(0);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef value;
oparg = 1;
assert(oparg == CURRENT_OPARG());
assert(oparg < _PY_NSMALLPOSINTS);
PyObject *obj = (PyObject *)&_PyLong_SMALL_INTS[_PY_NSMALLNEGINTS + oparg];
value = PyStackRef_FromPyObjectBorrow(obj);
_tos_cache0 = value;
SET_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _LOAD_SMALL_INT_1_r12: {
CHECK_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef value;
_PyStackRef _stack_item_0 = _tos_cache0;
oparg = 1;
assert(oparg == CURRENT_OPARG());
assert(oparg < _PY_NSMALLPOSINTS);
PyObject *obj = (PyObject *)&_PyLong_SMALL_INTS[_PY_NSMALLNEGINTS + oparg];
value = PyStackRef_FromPyObjectBorrow(obj);
_tos_cache1 = value;
_tos_cache0 = _stack_item_0;
SET_CURRENT_CACHED_VALUES(2);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _LOAD_SMALL_INT_1_r23: {
CHECK_CURRENT_CACHED_VALUES(2);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef value;
_PyStackRef _stack_item_0 = _tos_cache0;
_PyStackRef _stack_item_1 = _tos_cache1;
oparg = 1;
assert(oparg == CURRENT_OPARG());
assert(oparg < _PY_NSMALLPOSINTS);
PyObject *obj = (PyObject *)&_PyLong_SMALL_INTS[_PY_NSMALLNEGINTS + oparg];
value = PyStackRef_FromPyObjectBorrow(obj);
_tos_cache2 = value;
_tos_cache1 = _stack_item_1;
_tos_cache0 = _stack_item_0;
SET_CURRENT_CACHED_VALUES(3);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _LOAD_SMALL_INT_2_r01: {
CHECK_CURRENT_CACHED_VALUES(0);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef value;
oparg = 2;
assert(oparg == CURRENT_OPARG());
assert(oparg < _PY_NSMALLPOSINTS);
PyObject *obj = (PyObject *)&_PyLong_SMALL_INTS[_PY_NSMALLNEGINTS + oparg];
value = PyStackRef_FromPyObjectBorrow(obj);
_tos_cache0 = value;
SET_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _LOAD_SMALL_INT_2_r12: {
CHECK_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef value;
_PyStackRef _stack_item_0 = _tos_cache0;
oparg = 2;
assert(oparg == CURRENT_OPARG());
assert(oparg < _PY_NSMALLPOSINTS);
PyObject *obj = (PyObject *)&_PyLong_SMALL_INTS[_PY_NSMALLNEGINTS + oparg];
value = PyStackRef_FromPyObjectBorrow(obj);
_tos_cache1 = value;
_tos_cache0 = _stack_item_0;
SET_CURRENT_CACHED_VALUES(2);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _LOAD_SMALL_INT_2_r23: {
CHECK_CURRENT_CACHED_VALUES(2);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef value;
_PyStackRef _stack_item_0 = _tos_cache0;
_PyStackRef _stack_item_1 = _tos_cache1;
oparg = 2;
assert(oparg == CURRENT_OPARG());
assert(oparg < _PY_NSMALLPOSINTS);
PyObject *obj = (PyObject *)&_PyLong_SMALL_INTS[_PY_NSMALLNEGINTS + oparg];
value = PyStackRef_FromPyObjectBorrow(obj);
_tos_cache2 = value;
_tos_cache1 = _stack_item_1;
_tos_cache0 = _stack_item_0;
SET_CURRENT_CACHED_VALUES(3);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _LOAD_SMALL_INT_3_r01: {
CHECK_CURRENT_CACHED_VALUES(0);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef value;
oparg = 3;
assert(oparg == CURRENT_OPARG());
assert(oparg < _PY_NSMALLPOSINTS);
PyObject *obj = (PyObject *)&_PyLong_SMALL_INTS[_PY_NSMALLNEGINTS + oparg];
value = PyStackRef_FromPyObjectBorrow(obj);
_tos_cache0 = value;
SET_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _LOAD_SMALL_INT_3_r12: {
CHECK_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef value;
_PyStackRef _stack_item_0 = _tos_cache0;
oparg = 3;
assert(oparg == CURRENT_OPARG());
assert(oparg < _PY_NSMALLPOSINTS);
PyObject *obj = (PyObject *)&_PyLong_SMALL_INTS[_PY_NSMALLNEGINTS + oparg];
value = PyStackRef_FromPyObjectBorrow(obj);
_tos_cache1 = value;
_tos_cache0 = _stack_item_0;
SET_CURRENT_CACHED_VALUES(2);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _LOAD_SMALL_INT_3_r23: {
CHECK_CURRENT_CACHED_VALUES(2);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef value;
_PyStackRef _stack_item_0 = _tos_cache0;
_PyStackRef _stack_item_1 = _tos_cache1;
oparg = 3;
assert(oparg == CURRENT_OPARG());
assert(oparg < _PY_NSMALLPOSINTS);
PyObject *obj = (PyObject *)&_PyLong_SMALL_INTS[_PY_NSMALLNEGINTS + oparg];
value = PyStackRef_FromPyObjectBorrow(obj);
_tos_cache2 = value;
_tos_cache1 = _stack_item_1;
_tos_cache0 = _stack_item_0;
SET_CURRENT_CACHED_VALUES(3);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _LOAD_SMALL_INT_r01: {
CHECK_CURRENT_CACHED_VALUES(0);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef value;
oparg = CURRENT_OPARG();
assert(oparg < _PY_NSMALLPOSINTS);
PyObject *obj = (PyObject *)&_PyLong_SMALL_INTS[_PY_NSMALLNEGINTS + oparg];
value = PyStackRef_FromPyObjectBorrow(obj);
_tos_cache0 = value;
SET_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _LOAD_SMALL_INT_r12: {
CHECK_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef value;
_PyStackRef _stack_item_0 = _tos_cache0;
oparg = CURRENT_OPARG();
assert(oparg < _PY_NSMALLPOSINTS);
PyObject *obj = (PyObject *)&_PyLong_SMALL_INTS[_PY_NSMALLNEGINTS + oparg];
value = PyStackRef_FromPyObjectBorrow(obj);
_tos_cache1 = value;
_tos_cache0 = _stack_item_0;
SET_CURRENT_CACHED_VALUES(2);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _LOAD_SMALL_INT_r23: {
CHECK_CURRENT_CACHED_VALUES(2);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef value;
_PyStackRef _stack_item_0 = _tos_cache0;
_PyStackRef _stack_item_1 = _tos_cache1;
oparg = CURRENT_OPARG();
assert(oparg < _PY_NSMALLPOSINTS);
PyObject *obj = (PyObject *)&_PyLong_SMALL_INTS[_PY_NSMALLNEGINTS + oparg];
value = PyStackRef_FromPyObjectBorrow(obj);
_tos_cache2 = value;
_tos_cache1 = _stack_item_1;
_tos_cache0 = _stack_item_0;
SET_CURRENT_CACHED_VALUES(3);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _STORE_FAST_0_r10: {
CHECK_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef value;
_PyStackRef _stack_item_0 = _tos_cache0;
oparg = 0;
assert(oparg == CURRENT_OPARG());
value = _stack_item_0;
_PyStackRef tmp = GETLOCAL(oparg);
GETLOCAL(oparg) = value;
_PyFrame_SetStackPointer(frame, stack_pointer);
PyStackRef_XCLOSE(tmp);
stack_pointer = _PyFrame_GetStackPointer(frame);
_tos_cache0 = PyStackRef_ZERO_BITS;
_tos_cache1 = PyStackRef_ZERO_BITS;
_tos_cache2 = PyStackRef_ZERO_BITS;
SET_CURRENT_CACHED_VALUES(0);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _STORE_FAST_1_r10: {
CHECK_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef value;
_PyStackRef _stack_item_0 = _tos_cache0;
oparg = 1;
assert(oparg == CURRENT_OPARG());
value = _stack_item_0;
_PyStackRef tmp = GETLOCAL(oparg);
GETLOCAL(oparg) = value;
_PyFrame_SetStackPointer(frame, stack_pointer);
PyStackRef_XCLOSE(tmp);
stack_pointer = _PyFrame_GetStackPointer(frame);
_tos_cache0 = PyStackRef_ZERO_BITS;
_tos_cache1 = PyStackRef_ZERO_BITS;
_tos_cache2 = PyStackRef_ZERO_BITS;
SET_CURRENT_CACHED_VALUES(0);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _STORE_FAST_2_r10: {
CHECK_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef value;
_PyStackRef _stack_item_0 = _tos_cache0;
oparg = 2;
assert(oparg == CURRENT_OPARG());
value = _stack_item_0;
_PyStackRef tmp = GETLOCAL(oparg);
GETLOCAL(oparg) = value;
_PyFrame_SetStackPointer(frame, stack_pointer);
PyStackRef_XCLOSE(tmp);
stack_pointer = _PyFrame_GetStackPointer(frame);
_tos_cache0 = PyStackRef_ZERO_BITS;
_tos_cache1 = PyStackRef_ZERO_BITS;
_tos_cache2 = PyStackRef_ZERO_BITS;
SET_CURRENT_CACHED_VALUES(0);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _STORE_FAST_3_r10: {
CHECK_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef value;
_PyStackRef _stack_item_0 = _tos_cache0;
oparg = 3;
assert(oparg == CURRENT_OPARG());
value = _stack_item_0;
_PyStackRef tmp = GETLOCAL(oparg);
GETLOCAL(oparg) = value;
_PyFrame_SetStackPointer(frame, stack_pointer);
PyStackRef_XCLOSE(tmp);
stack_pointer = _PyFrame_GetStackPointer(frame);
_tos_cache0 = PyStackRef_ZERO_BITS;
_tos_cache1 = PyStackRef_ZERO_BITS;
_tos_cache2 = PyStackRef_ZERO_BITS;
SET_CURRENT_CACHED_VALUES(0);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _STORE_FAST_4_r10: {
CHECK_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef value;
_PyStackRef _stack_item_0 = _tos_cache0;
oparg = 4;
assert(oparg == CURRENT_OPARG());
value = _stack_item_0;
_PyStackRef tmp = GETLOCAL(oparg);
GETLOCAL(oparg) = value;
_PyFrame_SetStackPointer(frame, stack_pointer);
PyStackRef_XCLOSE(tmp);
stack_pointer = _PyFrame_GetStackPointer(frame);
_tos_cache0 = PyStackRef_ZERO_BITS;
_tos_cache1 = PyStackRef_ZERO_BITS;
_tos_cache2 = PyStackRef_ZERO_BITS;
SET_CURRENT_CACHED_VALUES(0);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _STORE_FAST_5_r10: {
CHECK_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef value;
_PyStackRef _stack_item_0 = _tos_cache0;
oparg = 5;
assert(oparg == CURRENT_OPARG());
value = _stack_item_0;
_PyStackRef tmp = GETLOCAL(oparg);
GETLOCAL(oparg) = value;
_PyFrame_SetStackPointer(frame, stack_pointer);
PyStackRef_XCLOSE(tmp);
stack_pointer = _PyFrame_GetStackPointer(frame);
_tos_cache0 = PyStackRef_ZERO_BITS;
_tos_cache1 = PyStackRef_ZERO_BITS;
_tos_cache2 = PyStackRef_ZERO_BITS;
SET_CURRENT_CACHED_VALUES(0);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _STORE_FAST_6_r10: {
CHECK_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef value;
_PyStackRef _stack_item_0 = _tos_cache0;
oparg = 6;
assert(oparg == CURRENT_OPARG());
value = _stack_item_0;
_PyStackRef tmp = GETLOCAL(oparg);
GETLOCAL(oparg) = value;
_PyFrame_SetStackPointer(frame, stack_pointer);
PyStackRef_XCLOSE(tmp);
stack_pointer = _PyFrame_GetStackPointer(frame);
_tos_cache0 = PyStackRef_ZERO_BITS;
_tos_cache1 = PyStackRef_ZERO_BITS;
_tos_cache2 = PyStackRef_ZERO_BITS;
SET_CURRENT_CACHED_VALUES(0);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _STORE_FAST_7_r10: {
CHECK_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef value;
_PyStackRef _stack_item_0 = _tos_cache0;
oparg = 7;
assert(oparg == CURRENT_OPARG());
value = _stack_item_0;
_PyStackRef tmp = GETLOCAL(oparg);
GETLOCAL(oparg) = value;
_PyFrame_SetStackPointer(frame, stack_pointer);
PyStackRef_XCLOSE(tmp);
stack_pointer = _PyFrame_GetStackPointer(frame);
_tos_cache0 = PyStackRef_ZERO_BITS;
_tos_cache1 = PyStackRef_ZERO_BITS;
_tos_cache2 = PyStackRef_ZERO_BITS;
SET_CURRENT_CACHED_VALUES(0);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _STORE_FAST_r10: {
CHECK_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef value;
_PyStackRef _stack_item_0 = _tos_cache0;
oparg = CURRENT_OPARG();
value = _stack_item_0;
_PyStackRef tmp = GETLOCAL(oparg);
GETLOCAL(oparg) = value;
_PyFrame_SetStackPointer(frame, stack_pointer);
PyStackRef_XCLOSE(tmp);
stack_pointer = _PyFrame_GetStackPointer(frame);
_tos_cache0 = PyStackRef_ZERO_BITS;
_tos_cache1 = PyStackRef_ZERO_BITS;
_tos_cache2 = PyStackRef_ZERO_BITS;
SET_CURRENT_CACHED_VALUES(0);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _POP_TOP_r10: {
CHECK_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef value;
_PyStackRef _stack_item_0 = _tos_cache0;
value = _stack_item_0;
_PyFrame_SetStackPointer(frame, stack_pointer);
PyStackRef_XCLOSE(value);
stack_pointer = _PyFrame_GetStackPointer(frame);
_tos_cache0 = PyStackRef_ZERO_BITS;
_tos_cache1 = PyStackRef_ZERO_BITS;
_tos_cache2 = PyStackRef_ZERO_BITS;
SET_CURRENT_CACHED_VALUES(0);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _POP_TOP_NOP_r00: {
CHECK_CURRENT_CACHED_VALUES(0);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef value;
value = stack_pointer[-1];
assert(PyStackRef_IsNull(value) || (!PyStackRef_RefcountOnObject(value)) ||
_Py_IsImmortal((PyStackRef_AsPyObjectBorrow(value))));
SET_CURRENT_CACHED_VALUES(0);
stack_pointer += -1;
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _POP_TOP_NOP_r10: {
CHECK_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef value;
_PyStackRef _stack_item_0 = _tos_cache0;
value = _stack_item_0;
assert(PyStackRef_IsNull(value) || (!PyStackRef_RefcountOnObject(value)) ||
_Py_IsImmortal((PyStackRef_AsPyObjectBorrow(value))));
SET_CURRENT_CACHED_VALUES(0);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _POP_TOP_NOP_r21: {
CHECK_CURRENT_CACHED_VALUES(2);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef value;
_PyStackRef _stack_item_0 = _tos_cache0;
_PyStackRef _stack_item_1 = _tos_cache1;
value = _stack_item_1;
assert(PyStackRef_IsNull(value) || (!PyStackRef_RefcountOnObject(value)) ||
_Py_IsImmortal((PyStackRef_AsPyObjectBorrow(value))));
_tos_cache0 = _stack_item_0;
SET_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _POP_TOP_NOP_r32: {
CHECK_CURRENT_CACHED_VALUES(3);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef value;
_PyStackRef _stack_item_0 = _tos_cache0;
_PyStackRef _stack_item_1 = _tos_cache1;
_PyStackRef _stack_item_2 = _tos_cache2;
value = _stack_item_2;
assert(PyStackRef_IsNull(value) || (!PyStackRef_RefcountOnObject(value)) ||
_Py_IsImmortal((PyStackRef_AsPyObjectBorrow(value))));
_tos_cache1 = _stack_item_1;
_tos_cache0 = _stack_item_0;
SET_CURRENT_CACHED_VALUES(2);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _POP_TOP_INT_r00: {
CHECK_CURRENT_CACHED_VALUES(0);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef value;
value = stack_pointer[-1];
assert(PyLong_CheckExact(PyStackRef_AsPyObjectBorrow(value)));
PyStackRef_CLOSE_SPECIALIZED(value, _PyLong_ExactDealloc);
SET_CURRENT_CACHED_VALUES(0);
stack_pointer += -1;
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _POP_TOP_INT_r10: {
CHECK_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef value;
_PyStackRef _stack_item_0 = _tos_cache0;
value = _stack_item_0;
assert(PyLong_CheckExact(PyStackRef_AsPyObjectBorrow(value)));
PyStackRef_CLOSE_SPECIALIZED(value, _PyLong_ExactDealloc);
SET_CURRENT_CACHED_VALUES(0);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _POP_TOP_INT_r21: {
CHECK_CURRENT_CACHED_VALUES(2);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef value;
_PyStackRef _stack_item_0 = _tos_cache0;
_PyStackRef _stack_item_1 = _tos_cache1;
value = _stack_item_1;
assert(PyLong_CheckExact(PyStackRef_AsPyObjectBorrow(value)));
PyStackRef_CLOSE_SPECIALIZED(value, _PyLong_ExactDealloc);
_tos_cache0 = _stack_item_0;
SET_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _POP_TOP_INT_r32: {
CHECK_CURRENT_CACHED_VALUES(3);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef value;
_PyStackRef _stack_item_0 = _tos_cache0;
_PyStackRef _stack_item_1 = _tos_cache1;
_PyStackRef _stack_item_2 = _tos_cache2;
value = _stack_item_2;
assert(PyLong_CheckExact(PyStackRef_AsPyObjectBorrow(value)));
PyStackRef_CLOSE_SPECIALIZED(value, _PyLong_ExactDealloc);
_tos_cache1 = _stack_item_1;
_tos_cache0 = _stack_item_0;
SET_CURRENT_CACHED_VALUES(2);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _POP_TOP_FLOAT_r00: {
CHECK_CURRENT_CACHED_VALUES(0);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef value;
value = stack_pointer[-1];
assert(PyFloat_CheckExact(PyStackRef_AsPyObjectBorrow(value)));
PyStackRef_CLOSE_SPECIALIZED(value, _PyFloat_ExactDealloc);
SET_CURRENT_CACHED_VALUES(0);
stack_pointer += -1;
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _POP_TOP_FLOAT_r10: {
CHECK_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef value;
_PyStackRef _stack_item_0 = _tos_cache0;
value = _stack_item_0;
assert(PyFloat_CheckExact(PyStackRef_AsPyObjectBorrow(value)));
PyStackRef_CLOSE_SPECIALIZED(value, _PyFloat_ExactDealloc);
SET_CURRENT_CACHED_VALUES(0);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _POP_TOP_FLOAT_r21: {
CHECK_CURRENT_CACHED_VALUES(2);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef value;
_PyStackRef _stack_item_0 = _tos_cache0;
_PyStackRef _stack_item_1 = _tos_cache1;
value = _stack_item_1;
assert(PyFloat_CheckExact(PyStackRef_AsPyObjectBorrow(value)));
PyStackRef_CLOSE_SPECIALIZED(value, _PyFloat_ExactDealloc);
_tos_cache0 = _stack_item_0;
SET_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _POP_TOP_FLOAT_r32: {
CHECK_CURRENT_CACHED_VALUES(3);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef value;
_PyStackRef _stack_item_0 = _tos_cache0;
_PyStackRef _stack_item_1 = _tos_cache1;
_PyStackRef _stack_item_2 = _tos_cache2;
value = _stack_item_2;
assert(PyFloat_CheckExact(PyStackRef_AsPyObjectBorrow(value)));
PyStackRef_CLOSE_SPECIALIZED(value, _PyFloat_ExactDealloc);
_tos_cache1 = _stack_item_1;
_tos_cache0 = _stack_item_0;
SET_CURRENT_CACHED_VALUES(2);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _POP_TOP_UNICODE_r00: {
CHECK_CURRENT_CACHED_VALUES(0);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef value;
value = stack_pointer[-1];
assert(PyUnicode_CheckExact(PyStackRef_AsPyObjectBorrow(value)));
PyStackRef_CLOSE_SPECIALIZED(value, _PyUnicode_ExactDealloc);
SET_CURRENT_CACHED_VALUES(0);
stack_pointer += -1;
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _POP_TOP_UNICODE_r10: {
CHECK_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef value;
_PyStackRef _stack_item_0 = _tos_cache0;
value = _stack_item_0;
assert(PyUnicode_CheckExact(PyStackRef_AsPyObjectBorrow(value)));
PyStackRef_CLOSE_SPECIALIZED(value, _PyUnicode_ExactDealloc);
SET_CURRENT_CACHED_VALUES(0);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _POP_TOP_UNICODE_r21: {
CHECK_CURRENT_CACHED_VALUES(2);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef value;
_PyStackRef _stack_item_0 = _tos_cache0;
_PyStackRef _stack_item_1 = _tos_cache1;
value = _stack_item_1;
assert(PyUnicode_CheckExact(PyStackRef_AsPyObjectBorrow(value)));
PyStackRef_CLOSE_SPECIALIZED(value, _PyUnicode_ExactDealloc);
_tos_cache0 = _stack_item_0;
SET_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _POP_TOP_UNICODE_r32: {
CHECK_CURRENT_CACHED_VALUES(3);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef value;
_PyStackRef _stack_item_0 = _tos_cache0;
_PyStackRef _stack_item_1 = _tos_cache1;
_PyStackRef _stack_item_2 = _tos_cache2;
value = _stack_item_2;
assert(PyUnicode_CheckExact(PyStackRef_AsPyObjectBorrow(value)));
PyStackRef_CLOSE_SPECIALIZED(value, _PyUnicode_ExactDealloc);
_tos_cache1 = _stack_item_1;
_tos_cache0 = _stack_item_0;
SET_CURRENT_CACHED_VALUES(2);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _POP_TWO_r20: {
CHECK_CURRENT_CACHED_VALUES(2);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef tos;
_PyStackRef nos;
_PyStackRef _stack_item_0 = _tos_cache0;
_PyStackRef _stack_item_1 = _tos_cache1;
tos = _stack_item_1;
nos = _stack_item_0;
stack_pointer[0] = nos;
stack_pointer += 1;
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
_PyFrame_SetStackPointer(frame, stack_pointer);
PyStackRef_CLOSE(tos);
stack_pointer = _PyFrame_GetStackPointer(frame);
stack_pointer += -1;
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
_PyFrame_SetStackPointer(frame, stack_pointer);
PyStackRef_CLOSE(nos);
stack_pointer = _PyFrame_GetStackPointer(frame);
_tos_cache0 = PyStackRef_ZERO_BITS;
_tos_cache1 = PyStackRef_ZERO_BITS;
_tos_cache2 = PyStackRef_ZERO_BITS;
SET_CURRENT_CACHED_VALUES(0);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _PUSH_NULL_r01: {
CHECK_CURRENT_CACHED_VALUES(0);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef res;
res = PyStackRef_NULL;
_tos_cache0 = res;
SET_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _PUSH_NULL_r12: {
CHECK_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef res;
_PyStackRef _stack_item_0 = _tos_cache0;
res = PyStackRef_NULL;
_tos_cache1 = res;
_tos_cache0 = _stack_item_0;
SET_CURRENT_CACHED_VALUES(2);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _PUSH_NULL_r23: {
CHECK_CURRENT_CACHED_VALUES(2);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef res;
_PyStackRef _stack_item_0 = _tos_cache0;
_PyStackRef _stack_item_1 = _tos_cache1;
res = PyStackRef_NULL;
_tos_cache2 = res;
_tos_cache1 = _stack_item_1;
_tos_cache0 = _stack_item_0;
SET_CURRENT_CACHED_VALUES(3);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _END_FOR_r10: {
CHECK_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef value;
_PyStackRef _stack_item_0 = _tos_cache0;
value = _stack_item_0;
_PyFrame_SetStackPointer(frame, stack_pointer);
PyStackRef_CLOSE(value);
stack_pointer = _PyFrame_GetStackPointer(frame);
_tos_cache0 = PyStackRef_ZERO_BITS;
_tos_cache1 = PyStackRef_ZERO_BITS;
_tos_cache2 = PyStackRef_ZERO_BITS;
SET_CURRENT_CACHED_VALUES(0);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _POP_ITER_r20: {
CHECK_CURRENT_CACHED_VALUES(2);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef index_or_null;
_PyStackRef iter;
_PyStackRef _stack_item_0 = _tos_cache0;
_PyStackRef _stack_item_1 = _tos_cache1;
index_or_null = _stack_item_1;
iter = _stack_item_0;
(void)index_or_null;
_PyFrame_SetStackPointer(frame, stack_pointer);
PyStackRef_CLOSE(iter);
stack_pointer = _PyFrame_GetStackPointer(frame);
_tos_cache0 = PyStackRef_ZERO_BITS;
_tos_cache1 = PyStackRef_ZERO_BITS;
_tos_cache2 = PyStackRef_ZERO_BITS;
SET_CURRENT_CACHED_VALUES(0);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _END_SEND_r21: {
CHECK_CURRENT_CACHED_VALUES(2);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef value;
_PyStackRef receiver;
_PyStackRef val;
_PyStackRef _stack_item_0 = _tos_cache0;
_PyStackRef _stack_item_1 = _tos_cache1;
value = _stack_item_1;
receiver = _stack_item_0;
val = value;
stack_pointer[0] = val;
stack_pointer += 1;
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
_PyFrame_SetStackPointer(frame, stack_pointer);
PyStackRef_CLOSE(receiver);
stack_pointer = _PyFrame_GetStackPointer(frame);
_tos_cache0 = val;
_tos_cache1 = PyStackRef_ZERO_BITS;
_tos_cache2 = PyStackRef_ZERO_BITS;
SET_CURRENT_CACHED_VALUES(1);
stack_pointer += -1;
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _UNARY_NEGATIVE_r11: {
CHECK_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef value;
_PyStackRef res;
_PyStackRef _stack_item_0 = _tos_cache0;
value = _stack_item_0;
stack_pointer[0] = value;
stack_pointer += 1;
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
_PyFrame_SetStackPointer(frame, stack_pointer);
PyObject *res_o = PyNumber_Negative(PyStackRef_AsPyObjectBorrow(value));
stack_pointer = _PyFrame_GetStackPointer(frame);
stack_pointer += -1;
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
_PyFrame_SetStackPointer(frame, stack_pointer);
PyStackRef_CLOSE(value);
stack_pointer = _PyFrame_GetStackPointer(frame);
if (res_o == NULL) {
SET_CURRENT_CACHED_VALUES(0);
JUMP_TO_ERROR();
}
res = PyStackRef_FromPyObjectSteal(res_o);
_tos_cache0 = res;
_tos_cache1 = PyStackRef_ZERO_BITS;
_tos_cache2 = PyStackRef_ZERO_BITS;
SET_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _UNARY_NOT_r01: {
CHECK_CURRENT_CACHED_VALUES(0);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef value;
_PyStackRef res;
value = stack_pointer[-1];
assert(PyStackRef_BoolCheck(value));
res = PyStackRef_IsFalse(value)
? PyStackRef_True : PyStackRef_False;
_tos_cache0 = res;
SET_CURRENT_CACHED_VALUES(1);
stack_pointer += -1;
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _UNARY_NOT_r11: {
CHECK_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef value;
_PyStackRef res;
_PyStackRef _stack_item_0 = _tos_cache0;
value = _stack_item_0;
assert(PyStackRef_BoolCheck(value));
res = PyStackRef_IsFalse(value)
? PyStackRef_True : PyStackRef_False;
_tos_cache0 = res;
SET_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _UNARY_NOT_r22: {
CHECK_CURRENT_CACHED_VALUES(2);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef value;
_PyStackRef res;
_PyStackRef _stack_item_0 = _tos_cache0;
_PyStackRef _stack_item_1 = _tos_cache1;
value = _stack_item_1;
assert(PyStackRef_BoolCheck(value));
res = PyStackRef_IsFalse(value)
? PyStackRef_True : PyStackRef_False;
_tos_cache1 = res;
_tos_cache0 = _stack_item_0;
SET_CURRENT_CACHED_VALUES(2);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _UNARY_NOT_r33: {
CHECK_CURRENT_CACHED_VALUES(3);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef value;
_PyStackRef res;
_PyStackRef _stack_item_0 = _tos_cache0;
_PyStackRef _stack_item_1 = _tos_cache1;
_PyStackRef _stack_item_2 = _tos_cache2;
value = _stack_item_2;
assert(PyStackRef_BoolCheck(value));
res = PyStackRef_IsFalse(value)
? PyStackRef_True : PyStackRef_False;
_tos_cache2 = res;
_tos_cache1 = _stack_item_1;
_tos_cache0 = _stack_item_0;
SET_CURRENT_CACHED_VALUES(3);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _TO_BOOL_r11: {
CHECK_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef value;
_PyStackRef res;
_PyStackRef _stack_item_0 = _tos_cache0;
value = _stack_item_0;
stack_pointer[0] = value;
stack_pointer += 1;
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
_PyFrame_SetStackPointer(frame, stack_pointer);
int err = PyObject_IsTrue(PyStackRef_AsPyObjectBorrow(value));
stack_pointer = _PyFrame_GetStackPointer(frame);
stack_pointer += -1;
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
_PyFrame_SetStackPointer(frame, stack_pointer);
PyStackRef_CLOSE(value);
stack_pointer = _PyFrame_GetStackPointer(frame);
if (err < 0) {
SET_CURRENT_CACHED_VALUES(0);
JUMP_TO_ERROR();
}
res = err ? PyStackRef_True : PyStackRef_False;
_tos_cache0 = res;
_tos_cache1 = PyStackRef_ZERO_BITS;
_tos_cache2 = PyStackRef_ZERO_BITS;
SET_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _TO_BOOL_BOOL_r01: {
CHECK_CURRENT_CACHED_VALUES(0);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef value;
value = stack_pointer[-1];
if (!PyStackRef_BoolCheck(value)) {
UOP_STAT_INC(uopcode, miss);
SET_CURRENT_CACHED_VALUES(0);
JUMP_TO_JUMP_TARGET();
}
STAT_INC(TO_BOOL, hit);
_tos_cache0 = value;
SET_CURRENT_CACHED_VALUES(1);
stack_pointer += -1;
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _TO_BOOL_BOOL_r11: {
CHECK_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef value;
_PyStackRef _stack_item_0 = _tos_cache0;
value = _stack_item_0;
if (!PyStackRef_BoolCheck(value)) {
UOP_STAT_INC(uopcode, miss);
GH-142718: JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. (GH-142762) JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. * Make sure that stack is in correct state at side exits with TOS cached values * Simplify choice of cached items for side exits
2025-12-15 19:01:29 +00:00
_tos_cache0 = value;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(1);
JUMP_TO_JUMP_TARGET();
}
STAT_INC(TO_BOOL, hit);
_tos_cache0 = value;
SET_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _TO_BOOL_BOOL_r22: {
CHECK_CURRENT_CACHED_VALUES(2);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef value;
_PyStackRef _stack_item_0 = _tos_cache0;
_PyStackRef _stack_item_1 = _tos_cache1;
value = _stack_item_1;
if (!PyStackRef_BoolCheck(value)) {
UOP_STAT_INC(uopcode, miss);
GH-142718: JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. (GH-142762) JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. * Make sure that stack is in correct state at side exits with TOS cached values * Simplify choice of cached items for side exits
2025-12-15 19:01:29 +00:00
_tos_cache1 = value;
_tos_cache0 = _stack_item_0;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(2);
JUMP_TO_JUMP_TARGET();
}
STAT_INC(TO_BOOL, hit);
_tos_cache1 = value;
_tos_cache0 = _stack_item_0;
SET_CURRENT_CACHED_VALUES(2);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _TO_BOOL_BOOL_r33: {
CHECK_CURRENT_CACHED_VALUES(3);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef value;
_PyStackRef _stack_item_0 = _tos_cache0;
_PyStackRef _stack_item_1 = _tos_cache1;
_PyStackRef _stack_item_2 = _tos_cache2;
value = _stack_item_2;
if (!PyStackRef_BoolCheck(value)) {
UOP_STAT_INC(uopcode, miss);
GH-142718: JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. (GH-142762) JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. * Make sure that stack is in correct state at side exits with TOS cached values * Simplify choice of cached items for side exits
2025-12-15 19:01:29 +00:00
_tos_cache2 = value;
_tos_cache1 = _stack_item_1;
_tos_cache0 = _stack_item_0;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(3);
JUMP_TO_JUMP_TARGET();
}
STAT_INC(TO_BOOL, hit);
_tos_cache2 = value;
_tos_cache1 = _stack_item_1;
_tos_cache0 = _stack_item_0;
SET_CURRENT_CACHED_VALUES(3);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _TO_BOOL_INT_r11: {
CHECK_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef value;
_PyStackRef res;
_PyStackRef _stack_item_0 = _tos_cache0;
value = _stack_item_0;
PyObject *value_o = PyStackRef_AsPyObjectBorrow(value);
if (!PyLong_CheckExact(value_o)) {
UOP_STAT_INC(uopcode, miss);
GH-142718: JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. (GH-142762) JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. * Make sure that stack is in correct state at side exits with TOS cached values * Simplify choice of cached items for side exits
2025-12-15 19:01:29 +00:00
_tos_cache0 = value;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(1);
JUMP_TO_JUMP_TARGET();
}
STAT_INC(TO_BOOL, hit);
if (_PyLong_IsZero((PyLongObject *)value_o)) {
assert(_Py_IsImmortal(value_o));
res = PyStackRef_False;
}
else {
_PyFrame_SetStackPointer(frame, stack_pointer);
PyStackRef_CLOSE(value);
stack_pointer = _PyFrame_GetStackPointer(frame);
res = PyStackRef_True;
}
_tos_cache0 = res;
_tos_cache1 = PyStackRef_ZERO_BITS;
_tos_cache2 = PyStackRef_ZERO_BITS;
SET_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _GUARD_NOS_LIST_r02: {
CHECK_CURRENT_CACHED_VALUES(0);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef nos;
nos = stack_pointer[-2];
PyObject *o = PyStackRef_AsPyObjectBorrow(nos);
if (!PyList_CheckExact(o)) {
UOP_STAT_INC(uopcode, miss);
SET_CURRENT_CACHED_VALUES(0);
JUMP_TO_JUMP_TARGET();
}
_tos_cache1 = stack_pointer[-1];
_tos_cache0 = nos;
SET_CURRENT_CACHED_VALUES(2);
stack_pointer += -2;
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _GUARD_NOS_LIST_r12: {
CHECK_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef nos;
_PyStackRef _stack_item_0 = _tos_cache0;
nos = stack_pointer[-1];
PyObject *o = PyStackRef_AsPyObjectBorrow(nos);
if (!PyList_CheckExact(o)) {
UOP_STAT_INC(uopcode, miss);
GH-142718: JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. (GH-142762) JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. * Make sure that stack is in correct state at side exits with TOS cached values * Simplify choice of cached items for side exits
2025-12-15 19:01:29 +00:00
_tos_cache0 = _stack_item_0;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(1);
JUMP_TO_JUMP_TARGET();
}
_tos_cache1 = _stack_item_0;
_tos_cache0 = nos;
SET_CURRENT_CACHED_VALUES(2);
stack_pointer += -1;
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _GUARD_NOS_LIST_r22: {
CHECK_CURRENT_CACHED_VALUES(2);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef nos;
_PyStackRef _stack_item_0 = _tos_cache0;
_PyStackRef _stack_item_1 = _tos_cache1;
nos = _stack_item_0;
PyObject *o = PyStackRef_AsPyObjectBorrow(nos);
if (!PyList_CheckExact(o)) {
UOP_STAT_INC(uopcode, miss);
GH-142718: JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. (GH-142762) JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. * Make sure that stack is in correct state at side exits with TOS cached values * Simplify choice of cached items for side exits
2025-12-15 19:01:29 +00:00
_tos_cache1 = _stack_item_1;
_tos_cache0 = nos;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(2);
JUMP_TO_JUMP_TARGET();
}
_tos_cache1 = _stack_item_1;
_tos_cache0 = nos;
SET_CURRENT_CACHED_VALUES(2);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _GUARD_NOS_LIST_r33: {
CHECK_CURRENT_CACHED_VALUES(3);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef nos;
_PyStackRef _stack_item_0 = _tos_cache0;
_PyStackRef _stack_item_1 = _tos_cache1;
_PyStackRef _stack_item_2 = _tos_cache2;
nos = _stack_item_1;
PyObject *o = PyStackRef_AsPyObjectBorrow(nos);
if (!PyList_CheckExact(o)) {
UOP_STAT_INC(uopcode, miss);
GH-142718: JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. (GH-142762) JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. * Make sure that stack is in correct state at side exits with TOS cached values * Simplify choice of cached items for side exits
2025-12-15 19:01:29 +00:00
_tos_cache2 = _stack_item_2;
_tos_cache1 = nos;
_tos_cache0 = _stack_item_0;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(3);
JUMP_TO_JUMP_TARGET();
}
_tos_cache2 = _stack_item_2;
_tos_cache1 = nos;
_tos_cache0 = _stack_item_0;
SET_CURRENT_CACHED_VALUES(3);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _GUARD_TOS_LIST_r01: {
CHECK_CURRENT_CACHED_VALUES(0);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef tos;
tos = stack_pointer[-1];
PyObject *o = PyStackRef_AsPyObjectBorrow(tos);
if (!PyList_CheckExact(o)) {
UOP_STAT_INC(uopcode, miss);
SET_CURRENT_CACHED_VALUES(0);
JUMP_TO_JUMP_TARGET();
}
_tos_cache0 = tos;
SET_CURRENT_CACHED_VALUES(1);
stack_pointer += -1;
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _GUARD_TOS_LIST_r11: {
CHECK_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef tos;
_PyStackRef _stack_item_0 = _tos_cache0;
tos = _stack_item_0;
PyObject *o = PyStackRef_AsPyObjectBorrow(tos);
if (!PyList_CheckExact(o)) {
UOP_STAT_INC(uopcode, miss);
GH-142718: JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. (GH-142762) JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. * Make sure that stack is in correct state at side exits with TOS cached values * Simplify choice of cached items for side exits
2025-12-15 19:01:29 +00:00
_tos_cache0 = tos;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(1);
JUMP_TO_JUMP_TARGET();
}
_tos_cache0 = tos;
SET_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _GUARD_TOS_LIST_r22: {
CHECK_CURRENT_CACHED_VALUES(2);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef tos;
_PyStackRef _stack_item_0 = _tos_cache0;
_PyStackRef _stack_item_1 = _tos_cache1;
tos = _stack_item_1;
PyObject *o = PyStackRef_AsPyObjectBorrow(tos);
if (!PyList_CheckExact(o)) {
UOP_STAT_INC(uopcode, miss);
GH-142718: JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. (GH-142762) JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. * Make sure that stack is in correct state at side exits with TOS cached values * Simplify choice of cached items for side exits
2025-12-15 19:01:29 +00:00
_tos_cache1 = tos;
_tos_cache0 = _stack_item_0;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(2);
JUMP_TO_JUMP_TARGET();
}
_tos_cache1 = tos;
_tos_cache0 = _stack_item_0;
SET_CURRENT_CACHED_VALUES(2);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _GUARD_TOS_LIST_r33: {
CHECK_CURRENT_CACHED_VALUES(3);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef tos;
_PyStackRef _stack_item_0 = _tos_cache0;
_PyStackRef _stack_item_1 = _tos_cache1;
_PyStackRef _stack_item_2 = _tos_cache2;
tos = _stack_item_2;
PyObject *o = PyStackRef_AsPyObjectBorrow(tos);
if (!PyList_CheckExact(o)) {
UOP_STAT_INC(uopcode, miss);
GH-142718: JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. (GH-142762) JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. * Make sure that stack is in correct state at side exits with TOS cached values * Simplify choice of cached items for side exits
2025-12-15 19:01:29 +00:00
_tos_cache2 = tos;
_tos_cache1 = _stack_item_1;
_tos_cache0 = _stack_item_0;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(3);
JUMP_TO_JUMP_TARGET();
}
_tos_cache2 = tos;
_tos_cache1 = _stack_item_1;
_tos_cache0 = _stack_item_0;
SET_CURRENT_CACHED_VALUES(3);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _GUARD_TOS_SLICE_r01: {
CHECK_CURRENT_CACHED_VALUES(0);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef tos;
tos = stack_pointer[-1];
PyObject *o = PyStackRef_AsPyObjectBorrow(tos);
if (!PySlice_Check(o)) {
UOP_STAT_INC(uopcode, miss);
SET_CURRENT_CACHED_VALUES(0);
JUMP_TO_JUMP_TARGET();
}
_tos_cache0 = tos;
SET_CURRENT_CACHED_VALUES(1);
stack_pointer += -1;
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _GUARD_TOS_SLICE_r11: {
CHECK_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef tos;
_PyStackRef _stack_item_0 = _tos_cache0;
tos = _stack_item_0;
PyObject *o = PyStackRef_AsPyObjectBorrow(tos);
if (!PySlice_Check(o)) {
UOP_STAT_INC(uopcode, miss);
GH-142718: JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. (GH-142762) JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. * Make sure that stack is in correct state at side exits with TOS cached values * Simplify choice of cached items for side exits
2025-12-15 19:01:29 +00:00
_tos_cache0 = tos;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(1);
JUMP_TO_JUMP_TARGET();
}
_tos_cache0 = tos;
SET_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _GUARD_TOS_SLICE_r22: {
CHECK_CURRENT_CACHED_VALUES(2);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef tos;
_PyStackRef _stack_item_0 = _tos_cache0;
_PyStackRef _stack_item_1 = _tos_cache1;
tos = _stack_item_1;
PyObject *o = PyStackRef_AsPyObjectBorrow(tos);
if (!PySlice_Check(o)) {
UOP_STAT_INC(uopcode, miss);
GH-142718: JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. (GH-142762) JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. * Make sure that stack is in correct state at side exits with TOS cached values * Simplify choice of cached items for side exits
2025-12-15 19:01:29 +00:00
_tos_cache1 = tos;
_tos_cache0 = _stack_item_0;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(2);
JUMP_TO_JUMP_TARGET();
}
_tos_cache1 = tos;
_tos_cache0 = _stack_item_0;
SET_CURRENT_CACHED_VALUES(2);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _GUARD_TOS_SLICE_r33: {
CHECK_CURRENT_CACHED_VALUES(3);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef tos;
_PyStackRef _stack_item_0 = _tos_cache0;
_PyStackRef _stack_item_1 = _tos_cache1;
_PyStackRef _stack_item_2 = _tos_cache2;
tos = _stack_item_2;
PyObject *o = PyStackRef_AsPyObjectBorrow(tos);
if (!PySlice_Check(o)) {
UOP_STAT_INC(uopcode, miss);
GH-142718: JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. (GH-142762) JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. * Make sure that stack is in correct state at side exits with TOS cached values * Simplify choice of cached items for side exits
2025-12-15 19:01:29 +00:00
_tos_cache2 = tos;
_tos_cache1 = _stack_item_1;
_tos_cache0 = _stack_item_0;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(3);
JUMP_TO_JUMP_TARGET();
}
_tos_cache2 = tos;
_tos_cache1 = _stack_item_1;
_tos_cache0 = _stack_item_0;
SET_CURRENT_CACHED_VALUES(3);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _TO_BOOL_LIST_r11: {
CHECK_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef value;
_PyStackRef res;
_PyStackRef _stack_item_0 = _tos_cache0;
value = _stack_item_0;
PyObject *value_o = PyStackRef_AsPyObjectBorrow(value);
assert(PyList_CheckExact(value_o));
STAT_INC(TO_BOOL, hit);
res = PyList_GET_SIZE(value_o) ? PyStackRef_True : PyStackRef_False;
stack_pointer[0] = value;
stack_pointer += 1;
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
_PyFrame_SetStackPointer(frame, stack_pointer);
_PyStackRef tmp = value;
value = res;
stack_pointer[-1] = value;
PyStackRef_CLOSE(tmp);
stack_pointer = _PyFrame_GetStackPointer(frame);
_tos_cache0 = res;
_tos_cache1 = PyStackRef_ZERO_BITS;
_tos_cache2 = PyStackRef_ZERO_BITS;
SET_CURRENT_CACHED_VALUES(1);
stack_pointer += -1;
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _TO_BOOL_NONE_r01: {
CHECK_CURRENT_CACHED_VALUES(0);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef value;
_PyStackRef res;
value = stack_pointer[-1];
if (!PyStackRef_IsNone(value)) {
UOP_STAT_INC(uopcode, miss);
SET_CURRENT_CACHED_VALUES(0);
JUMP_TO_JUMP_TARGET();
}
STAT_INC(TO_BOOL, hit);
res = PyStackRef_False;
_tos_cache0 = res;
SET_CURRENT_CACHED_VALUES(1);
stack_pointer += -1;
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _TO_BOOL_NONE_r11: {
CHECK_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef value;
_PyStackRef res;
_PyStackRef _stack_item_0 = _tos_cache0;
value = _stack_item_0;
if (!PyStackRef_IsNone(value)) {
UOP_STAT_INC(uopcode, miss);
GH-142718: JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. (GH-142762) JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. * Make sure that stack is in correct state at side exits with TOS cached values * Simplify choice of cached items for side exits
2025-12-15 19:01:29 +00:00
_tos_cache0 = value;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(1);
JUMP_TO_JUMP_TARGET();
}
STAT_INC(TO_BOOL, hit);
res = PyStackRef_False;
_tos_cache0 = res;
SET_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _TO_BOOL_NONE_r22: {
CHECK_CURRENT_CACHED_VALUES(2);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef value;
_PyStackRef res;
_PyStackRef _stack_item_0 = _tos_cache0;
_PyStackRef _stack_item_1 = _tos_cache1;
value = _stack_item_1;
if (!PyStackRef_IsNone(value)) {
UOP_STAT_INC(uopcode, miss);
GH-142718: JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. (GH-142762) JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. * Make sure that stack is in correct state at side exits with TOS cached values * Simplify choice of cached items for side exits
2025-12-15 19:01:29 +00:00
_tos_cache1 = value;
_tos_cache0 = _stack_item_0;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(2);
JUMP_TO_JUMP_TARGET();
}
STAT_INC(TO_BOOL, hit);
res = PyStackRef_False;
_tos_cache1 = res;
_tos_cache0 = _stack_item_0;
SET_CURRENT_CACHED_VALUES(2);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _TO_BOOL_NONE_r33: {
CHECK_CURRENT_CACHED_VALUES(3);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef value;
_PyStackRef res;
_PyStackRef _stack_item_0 = _tos_cache0;
_PyStackRef _stack_item_1 = _tos_cache1;
_PyStackRef _stack_item_2 = _tos_cache2;
value = _stack_item_2;
if (!PyStackRef_IsNone(value)) {
UOP_STAT_INC(uopcode, miss);
GH-142718: JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. (GH-142762) JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. * Make sure that stack is in correct state at side exits with TOS cached values * Simplify choice of cached items for side exits
2025-12-15 19:01:29 +00:00
_tos_cache2 = value;
_tos_cache1 = _stack_item_1;
_tos_cache0 = _stack_item_0;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(3);
JUMP_TO_JUMP_TARGET();
}
STAT_INC(TO_BOOL, hit);
res = PyStackRef_False;
_tos_cache2 = res;
_tos_cache1 = _stack_item_1;
_tos_cache0 = _stack_item_0;
SET_CURRENT_CACHED_VALUES(3);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _GUARD_NOS_UNICODE_r02: {
CHECK_CURRENT_CACHED_VALUES(0);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef nos;
nos = stack_pointer[-2];
PyObject *o = PyStackRef_AsPyObjectBorrow(nos);
if (!PyUnicode_CheckExact(o)) {
UOP_STAT_INC(uopcode, miss);
SET_CURRENT_CACHED_VALUES(0);
JUMP_TO_JUMP_TARGET();
}
_tos_cache1 = stack_pointer[-1];
_tos_cache0 = nos;
SET_CURRENT_CACHED_VALUES(2);
stack_pointer += -2;
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _GUARD_NOS_UNICODE_r12: {
CHECK_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef nos;
_PyStackRef _stack_item_0 = _tos_cache0;
nos = stack_pointer[-1];
PyObject *o = PyStackRef_AsPyObjectBorrow(nos);
if (!PyUnicode_CheckExact(o)) {
UOP_STAT_INC(uopcode, miss);
GH-142718: JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. (GH-142762) JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. * Make sure that stack is in correct state at side exits with TOS cached values * Simplify choice of cached items for side exits
2025-12-15 19:01:29 +00:00
_tos_cache0 = _stack_item_0;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(1);
JUMP_TO_JUMP_TARGET();
}
_tos_cache1 = _stack_item_0;
_tos_cache0 = nos;
SET_CURRENT_CACHED_VALUES(2);
stack_pointer += -1;
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _GUARD_NOS_UNICODE_r22: {
CHECK_CURRENT_CACHED_VALUES(2);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef nos;
_PyStackRef _stack_item_0 = _tos_cache0;
_PyStackRef _stack_item_1 = _tos_cache1;
nos = _stack_item_0;
PyObject *o = PyStackRef_AsPyObjectBorrow(nos);
if (!PyUnicode_CheckExact(o)) {
UOP_STAT_INC(uopcode, miss);
GH-142718: JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. (GH-142762) JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. * Make sure that stack is in correct state at side exits with TOS cached values * Simplify choice of cached items for side exits
2025-12-15 19:01:29 +00:00
_tos_cache1 = _stack_item_1;
_tos_cache0 = nos;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(2);
JUMP_TO_JUMP_TARGET();
}
_tos_cache1 = _stack_item_1;
_tos_cache0 = nos;
SET_CURRENT_CACHED_VALUES(2);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _GUARD_NOS_UNICODE_r33: {
CHECK_CURRENT_CACHED_VALUES(3);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef nos;
_PyStackRef _stack_item_0 = _tos_cache0;
_PyStackRef _stack_item_1 = _tos_cache1;
_PyStackRef _stack_item_2 = _tos_cache2;
nos = _stack_item_1;
PyObject *o = PyStackRef_AsPyObjectBorrow(nos);
if (!PyUnicode_CheckExact(o)) {
UOP_STAT_INC(uopcode, miss);
GH-142718: JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. (GH-142762) JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. * Make sure that stack is in correct state at side exits with TOS cached values * Simplify choice of cached items for side exits
2025-12-15 19:01:29 +00:00
_tos_cache2 = _stack_item_2;
_tos_cache1 = nos;
_tos_cache0 = _stack_item_0;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(3);
JUMP_TO_JUMP_TARGET();
}
_tos_cache2 = _stack_item_2;
_tos_cache1 = nos;
_tos_cache0 = _stack_item_0;
SET_CURRENT_CACHED_VALUES(3);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _GUARD_TOS_UNICODE_r01: {
CHECK_CURRENT_CACHED_VALUES(0);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef value;
value = stack_pointer[-1];
PyObject *value_o = PyStackRef_AsPyObjectBorrow(value);
if (!PyUnicode_CheckExact(value_o)) {
UOP_STAT_INC(uopcode, miss);
SET_CURRENT_CACHED_VALUES(0);
JUMP_TO_JUMP_TARGET();
}
_tos_cache0 = value;
SET_CURRENT_CACHED_VALUES(1);
stack_pointer += -1;
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _GUARD_TOS_UNICODE_r11: {
CHECK_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef value;
_PyStackRef _stack_item_0 = _tos_cache0;
value = _stack_item_0;
PyObject *value_o = PyStackRef_AsPyObjectBorrow(value);
if (!PyUnicode_CheckExact(value_o)) {
UOP_STAT_INC(uopcode, miss);
GH-142718: JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. (GH-142762) JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. * Make sure that stack is in correct state at side exits with TOS cached values * Simplify choice of cached items for side exits
2025-12-15 19:01:29 +00:00
_tos_cache0 = value;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(1);
JUMP_TO_JUMP_TARGET();
}
_tos_cache0 = value;
SET_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _GUARD_TOS_UNICODE_r22: {
CHECK_CURRENT_CACHED_VALUES(2);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef value;
_PyStackRef _stack_item_0 = _tos_cache0;
_PyStackRef _stack_item_1 = _tos_cache1;
value = _stack_item_1;
PyObject *value_o = PyStackRef_AsPyObjectBorrow(value);
if (!PyUnicode_CheckExact(value_o)) {
UOP_STAT_INC(uopcode, miss);
GH-142718: JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. (GH-142762) JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. * Make sure that stack is in correct state at side exits with TOS cached values * Simplify choice of cached items for side exits
2025-12-15 19:01:29 +00:00
_tos_cache1 = value;
_tos_cache0 = _stack_item_0;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(2);
JUMP_TO_JUMP_TARGET();
}
_tos_cache1 = value;
_tos_cache0 = _stack_item_0;
SET_CURRENT_CACHED_VALUES(2);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _GUARD_TOS_UNICODE_r33: {
CHECK_CURRENT_CACHED_VALUES(3);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef value;
_PyStackRef _stack_item_0 = _tos_cache0;
_PyStackRef _stack_item_1 = _tos_cache1;
_PyStackRef _stack_item_2 = _tos_cache2;
value = _stack_item_2;
PyObject *value_o = PyStackRef_AsPyObjectBorrow(value);
if (!PyUnicode_CheckExact(value_o)) {
UOP_STAT_INC(uopcode, miss);
GH-142718: JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. (GH-142762) JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. * Make sure that stack is in correct state at side exits with TOS cached values * Simplify choice of cached items for side exits
2025-12-15 19:01:29 +00:00
_tos_cache2 = value;
_tos_cache1 = _stack_item_1;
_tos_cache0 = _stack_item_0;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(3);
JUMP_TO_JUMP_TARGET();
}
_tos_cache2 = value;
_tos_cache1 = _stack_item_1;
_tos_cache0 = _stack_item_0;
SET_CURRENT_CACHED_VALUES(3);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _TO_BOOL_STR_r11: {
CHECK_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef value;
_PyStackRef res;
_PyStackRef _stack_item_0 = _tos_cache0;
value = _stack_item_0;
STAT_INC(TO_BOOL, hit);
PyObject *value_o = PyStackRef_AsPyObjectBorrow(value);
if (value_o == &_Py_STR(empty)) {
assert(_Py_IsImmortal(value_o));
res = PyStackRef_False;
}
else {
assert(Py_SIZE(value_o));
_PyFrame_SetStackPointer(frame, stack_pointer);
PyStackRef_CLOSE(value);
stack_pointer = _PyFrame_GetStackPointer(frame);
res = PyStackRef_True;
}
_tos_cache0 = res;
_tos_cache1 = PyStackRef_ZERO_BITS;
_tos_cache2 = PyStackRef_ZERO_BITS;
SET_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _REPLACE_WITH_TRUE_r11: {
CHECK_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef value;
_PyStackRef res;
_PyStackRef _stack_item_0 = _tos_cache0;
value = _stack_item_0;
_PyFrame_SetStackPointer(frame, stack_pointer);
PyStackRef_CLOSE(value);
stack_pointer = _PyFrame_GetStackPointer(frame);
res = PyStackRef_True;
_tos_cache0 = res;
_tos_cache1 = PyStackRef_ZERO_BITS;
_tos_cache2 = PyStackRef_ZERO_BITS;
SET_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _UNARY_INVERT_r11: {
CHECK_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef value;
_PyStackRef res;
_PyStackRef _stack_item_0 = _tos_cache0;
value = _stack_item_0;
stack_pointer[0] = value;
stack_pointer += 1;
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
_PyFrame_SetStackPointer(frame, stack_pointer);
PyObject *res_o = PyNumber_Invert(PyStackRef_AsPyObjectBorrow(value));
stack_pointer = _PyFrame_GetStackPointer(frame);
stack_pointer += -1;
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
_PyFrame_SetStackPointer(frame, stack_pointer);
PyStackRef_CLOSE(value);
stack_pointer = _PyFrame_GetStackPointer(frame);
if (res_o == NULL) {
SET_CURRENT_CACHED_VALUES(0);
JUMP_TO_ERROR();
}
res = PyStackRef_FromPyObjectSteal(res_o);
_tos_cache0 = res;
_tos_cache1 = PyStackRef_ZERO_BITS;
_tos_cache2 = PyStackRef_ZERO_BITS;
SET_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _GUARD_NOS_INT_r02: {
CHECK_CURRENT_CACHED_VALUES(0);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef left;
left = stack_pointer[-2];
PyObject *left_o = PyStackRef_AsPyObjectBorrow(left);
if (!_PyLong_CheckExactAndCompact(left_o)) {
UOP_STAT_INC(uopcode, miss);
SET_CURRENT_CACHED_VALUES(0);
JUMP_TO_JUMP_TARGET();
}
_tos_cache1 = stack_pointer[-1];
_tos_cache0 = left;
SET_CURRENT_CACHED_VALUES(2);
stack_pointer += -2;
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _GUARD_NOS_INT_r12: {
CHECK_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef left;
_PyStackRef _stack_item_0 = _tos_cache0;
left = stack_pointer[-1];
PyObject *left_o = PyStackRef_AsPyObjectBorrow(left);
if (!_PyLong_CheckExactAndCompact(left_o)) {
UOP_STAT_INC(uopcode, miss);
GH-142718: JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. (GH-142762) JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. * Make sure that stack is in correct state at side exits with TOS cached values * Simplify choice of cached items for side exits
2025-12-15 19:01:29 +00:00
_tos_cache0 = _stack_item_0;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(1);
JUMP_TO_JUMP_TARGET();
}
_tos_cache1 = _stack_item_0;
_tos_cache0 = left;
SET_CURRENT_CACHED_VALUES(2);
stack_pointer += -1;
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _GUARD_NOS_INT_r22: {
CHECK_CURRENT_CACHED_VALUES(2);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef left;
_PyStackRef _stack_item_0 = _tos_cache0;
_PyStackRef _stack_item_1 = _tos_cache1;
left = _stack_item_0;
PyObject *left_o = PyStackRef_AsPyObjectBorrow(left);
if (!_PyLong_CheckExactAndCompact(left_o)) {
UOP_STAT_INC(uopcode, miss);
GH-142718: JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. (GH-142762) JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. * Make sure that stack is in correct state at side exits with TOS cached values * Simplify choice of cached items for side exits
2025-12-15 19:01:29 +00:00
_tos_cache1 = _stack_item_1;
_tos_cache0 = left;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(2);
JUMP_TO_JUMP_TARGET();
}
_tos_cache1 = _stack_item_1;
_tos_cache0 = left;
SET_CURRENT_CACHED_VALUES(2);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _GUARD_NOS_INT_r33: {
CHECK_CURRENT_CACHED_VALUES(3);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef left;
_PyStackRef _stack_item_0 = _tos_cache0;
_PyStackRef _stack_item_1 = _tos_cache1;
_PyStackRef _stack_item_2 = _tos_cache2;
left = _stack_item_1;
PyObject *left_o = PyStackRef_AsPyObjectBorrow(left);
if (!_PyLong_CheckExactAndCompact(left_o)) {
UOP_STAT_INC(uopcode, miss);
GH-142718: JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. (GH-142762) JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. * Make sure that stack is in correct state at side exits with TOS cached values * Simplify choice of cached items for side exits
2025-12-15 19:01:29 +00:00
_tos_cache2 = _stack_item_2;
_tos_cache1 = left;
_tos_cache0 = _stack_item_0;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(3);
JUMP_TO_JUMP_TARGET();
}
_tos_cache2 = _stack_item_2;
_tos_cache1 = left;
_tos_cache0 = _stack_item_0;
SET_CURRENT_CACHED_VALUES(3);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _GUARD_TOS_INT_r01: {
CHECK_CURRENT_CACHED_VALUES(0);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef value;
value = stack_pointer[-1];
PyObject *value_o = PyStackRef_AsPyObjectBorrow(value);
if (!_PyLong_CheckExactAndCompact(value_o)) {
UOP_STAT_INC(uopcode, miss);
SET_CURRENT_CACHED_VALUES(0);
JUMP_TO_JUMP_TARGET();
}
_tos_cache0 = value;
SET_CURRENT_CACHED_VALUES(1);
stack_pointer += -1;
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _GUARD_TOS_INT_r11: {
CHECK_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef value;
_PyStackRef _stack_item_0 = _tos_cache0;
value = _stack_item_0;
PyObject *value_o = PyStackRef_AsPyObjectBorrow(value);
if (!_PyLong_CheckExactAndCompact(value_o)) {
UOP_STAT_INC(uopcode, miss);
GH-142718: JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. (GH-142762) JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. * Make sure that stack is in correct state at side exits with TOS cached values * Simplify choice of cached items for side exits
2025-12-15 19:01:29 +00:00
_tos_cache0 = value;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(1);
JUMP_TO_JUMP_TARGET();
}
_tos_cache0 = value;
SET_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _GUARD_TOS_INT_r22: {
CHECK_CURRENT_CACHED_VALUES(2);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef value;
_PyStackRef _stack_item_0 = _tos_cache0;
_PyStackRef _stack_item_1 = _tos_cache1;
value = _stack_item_1;
PyObject *value_o = PyStackRef_AsPyObjectBorrow(value);
if (!_PyLong_CheckExactAndCompact(value_o)) {
UOP_STAT_INC(uopcode, miss);
GH-142718: JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. (GH-142762) JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. * Make sure that stack is in correct state at side exits with TOS cached values * Simplify choice of cached items for side exits
2025-12-15 19:01:29 +00:00
_tos_cache1 = value;
_tos_cache0 = _stack_item_0;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(2);
JUMP_TO_JUMP_TARGET();
}
_tos_cache1 = value;
_tos_cache0 = _stack_item_0;
SET_CURRENT_CACHED_VALUES(2);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _GUARD_TOS_INT_r33: {
CHECK_CURRENT_CACHED_VALUES(3);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef value;
_PyStackRef _stack_item_0 = _tos_cache0;
_PyStackRef _stack_item_1 = _tos_cache1;
_PyStackRef _stack_item_2 = _tos_cache2;
value = _stack_item_2;
PyObject *value_o = PyStackRef_AsPyObjectBorrow(value);
if (!_PyLong_CheckExactAndCompact(value_o)) {
UOP_STAT_INC(uopcode, miss);
GH-142718: JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. (GH-142762) JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. * Make sure that stack is in correct state at side exits with TOS cached values * Simplify choice of cached items for side exits
2025-12-15 19:01:29 +00:00
_tos_cache2 = value;
_tos_cache1 = _stack_item_1;
_tos_cache0 = _stack_item_0;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(3);
JUMP_TO_JUMP_TARGET();
}
_tos_cache2 = value;
_tos_cache1 = _stack_item_1;
_tos_cache0 = _stack_item_0;
SET_CURRENT_CACHED_VALUES(3);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _GUARD_NOS_OVERFLOWED_r02: {
CHECK_CURRENT_CACHED_VALUES(0);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef left;
left = stack_pointer[-2];
PyObject *left_o = PyStackRef_AsPyObjectBorrow(left);
assert(Py_TYPE(left_o) == &PyLong_Type);
if (!_PyLong_IsCompact((PyLongObject *)left_o)) {
UOP_STAT_INC(uopcode, miss);
SET_CURRENT_CACHED_VALUES(0);
JUMP_TO_JUMP_TARGET();
}
_tos_cache1 = stack_pointer[-1];
_tos_cache0 = left;
SET_CURRENT_CACHED_VALUES(2);
stack_pointer += -2;
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _GUARD_NOS_OVERFLOWED_r12: {
CHECK_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef left;
_PyStackRef _stack_item_0 = _tos_cache0;
left = stack_pointer[-1];
PyObject *left_o = PyStackRef_AsPyObjectBorrow(left);
assert(Py_TYPE(left_o) == &PyLong_Type);
if (!_PyLong_IsCompact((PyLongObject *)left_o)) {
UOP_STAT_INC(uopcode, miss);
GH-142718: JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. (GH-142762) JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. * Make sure that stack is in correct state at side exits with TOS cached values * Simplify choice of cached items for side exits
2025-12-15 19:01:29 +00:00
_tos_cache0 = _stack_item_0;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(1);
JUMP_TO_JUMP_TARGET();
}
_tos_cache1 = _stack_item_0;
_tos_cache0 = left;
SET_CURRENT_CACHED_VALUES(2);
stack_pointer += -1;
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _GUARD_NOS_OVERFLOWED_r22: {
CHECK_CURRENT_CACHED_VALUES(2);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef left;
_PyStackRef _stack_item_0 = _tos_cache0;
_PyStackRef _stack_item_1 = _tos_cache1;
left = _stack_item_0;
PyObject *left_o = PyStackRef_AsPyObjectBorrow(left);
assert(Py_TYPE(left_o) == &PyLong_Type);
if (!_PyLong_IsCompact((PyLongObject *)left_o)) {
UOP_STAT_INC(uopcode, miss);
GH-142718: JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. (GH-142762) JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. * Make sure that stack is in correct state at side exits with TOS cached values * Simplify choice of cached items for side exits
2025-12-15 19:01:29 +00:00
_tos_cache1 = _stack_item_1;
_tos_cache0 = left;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(2);
JUMP_TO_JUMP_TARGET();
}
_tos_cache1 = _stack_item_1;
_tos_cache0 = left;
SET_CURRENT_CACHED_VALUES(2);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _GUARD_NOS_OVERFLOWED_r33: {
CHECK_CURRENT_CACHED_VALUES(3);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef left;
_PyStackRef _stack_item_0 = _tos_cache0;
_PyStackRef _stack_item_1 = _tos_cache1;
_PyStackRef _stack_item_2 = _tos_cache2;
left = _stack_item_1;
PyObject *left_o = PyStackRef_AsPyObjectBorrow(left);
assert(Py_TYPE(left_o) == &PyLong_Type);
if (!_PyLong_IsCompact((PyLongObject *)left_o)) {
UOP_STAT_INC(uopcode, miss);
GH-142718: JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. (GH-142762) JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. * Make sure that stack is in correct state at side exits with TOS cached values * Simplify choice of cached items for side exits
2025-12-15 19:01:29 +00:00
_tos_cache2 = _stack_item_2;
_tos_cache1 = left;
_tos_cache0 = _stack_item_0;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(3);
JUMP_TO_JUMP_TARGET();
}
_tos_cache2 = _stack_item_2;
_tos_cache1 = left;
_tos_cache0 = _stack_item_0;
SET_CURRENT_CACHED_VALUES(3);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _GUARD_TOS_OVERFLOWED_r01: {
CHECK_CURRENT_CACHED_VALUES(0);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef value;
value = stack_pointer[-1];
PyObject *value_o = PyStackRef_AsPyObjectBorrow(value);
assert(Py_TYPE(value_o) == &PyLong_Type);
if (!_PyLong_IsCompact((PyLongObject *)value_o)) {
UOP_STAT_INC(uopcode, miss);
SET_CURRENT_CACHED_VALUES(0);
JUMP_TO_JUMP_TARGET();
}
_tos_cache0 = value;
SET_CURRENT_CACHED_VALUES(1);
stack_pointer += -1;
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _GUARD_TOS_OVERFLOWED_r11: {
CHECK_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef value;
_PyStackRef _stack_item_0 = _tos_cache0;
value = _stack_item_0;
PyObject *value_o = PyStackRef_AsPyObjectBorrow(value);
assert(Py_TYPE(value_o) == &PyLong_Type);
if (!_PyLong_IsCompact((PyLongObject *)value_o)) {
UOP_STAT_INC(uopcode, miss);
GH-142718: JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. (GH-142762) JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. * Make sure that stack is in correct state at side exits with TOS cached values * Simplify choice of cached items for side exits
2025-12-15 19:01:29 +00:00
_tos_cache0 = value;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(1);
JUMP_TO_JUMP_TARGET();
}
_tos_cache0 = value;
SET_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _GUARD_TOS_OVERFLOWED_r22: {
CHECK_CURRENT_CACHED_VALUES(2);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef value;
_PyStackRef _stack_item_0 = _tos_cache0;
_PyStackRef _stack_item_1 = _tos_cache1;
value = _stack_item_1;
PyObject *value_o = PyStackRef_AsPyObjectBorrow(value);
assert(Py_TYPE(value_o) == &PyLong_Type);
if (!_PyLong_IsCompact((PyLongObject *)value_o)) {
UOP_STAT_INC(uopcode, miss);
GH-142718: JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. (GH-142762) JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. * Make sure that stack is in correct state at side exits with TOS cached values * Simplify choice of cached items for side exits
2025-12-15 19:01:29 +00:00
_tos_cache1 = value;
_tos_cache0 = _stack_item_0;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(2);
JUMP_TO_JUMP_TARGET();
}
_tos_cache1 = value;
_tos_cache0 = _stack_item_0;
SET_CURRENT_CACHED_VALUES(2);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _GUARD_TOS_OVERFLOWED_r33: {
CHECK_CURRENT_CACHED_VALUES(3);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef value;
_PyStackRef _stack_item_0 = _tos_cache0;
_PyStackRef _stack_item_1 = _tos_cache1;
_PyStackRef _stack_item_2 = _tos_cache2;
value = _stack_item_2;
PyObject *value_o = PyStackRef_AsPyObjectBorrow(value);
assert(Py_TYPE(value_o) == &PyLong_Type);
if (!_PyLong_IsCompact((PyLongObject *)value_o)) {
UOP_STAT_INC(uopcode, miss);
GH-142718: JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. (GH-142762) JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. * Make sure that stack is in correct state at side exits with TOS cached values * Simplify choice of cached items for side exits
2025-12-15 19:01:29 +00:00
_tos_cache2 = value;
_tos_cache1 = _stack_item_1;
_tos_cache0 = _stack_item_0;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(3);
JUMP_TO_JUMP_TARGET();
}
_tos_cache2 = value;
_tos_cache1 = _stack_item_1;
_tos_cache0 = _stack_item_0;
SET_CURRENT_CACHED_VALUES(3);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
gh-134584: JIT: Eliminate redundant refcount ops for X_INT (GH-142765)
2025-12-16 06:22:18 +08:00
case _BINARY_OP_MULTIPLY_INT_r03: {
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
CHECK_CURRENT_CACHED_VALUES(0);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef right;
_PyStackRef left;
_PyStackRef res;
gh-134584: JIT: Eliminate redundant refcount ops for X_INT (GH-142765)
2025-12-16 06:22:18 +08:00
_PyStackRef l;
_PyStackRef r;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
right = stack_pointer[-1];
left = stack_pointer[-2];
PyObject *left_o = PyStackRef_AsPyObjectBorrow(left);
PyObject *right_o = PyStackRef_AsPyObjectBorrow(right);
assert(PyLong_CheckExact(left_o));
assert(PyLong_CheckExact(right_o));
assert(_PyLong_BothAreCompact((PyLongObject *)left_o, (PyLongObject *)right_o));
STAT_INC(BINARY_OP, hit);
res = _PyCompactLong_Multiply((PyLongObject *)left_o, (PyLongObject *)right_o);
if (PyStackRef_IsNull(res)) {
UOP_STAT_INC(uopcode, miss);
SET_CURRENT_CACHED_VALUES(0);
JUMP_TO_JUMP_TARGET();
}
gh-134584: JIT: Eliminate redundant refcount ops for X_INT (GH-142765)
2025-12-16 06:22:18 +08:00
l = left;
r = right;
_tos_cache2 = r;
_tos_cache1 = l;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_tos_cache0 = res;
gh-134584: JIT: Eliminate redundant refcount ops for X_INT (GH-142765)
2025-12-16 06:22:18 +08:00
SET_CURRENT_CACHED_VALUES(3);
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
stack_pointer += -2;
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
gh-134584: JIT: Eliminate redundant refcount ops for X_INT (GH-142765)
2025-12-16 06:22:18 +08:00
case _BINARY_OP_MULTIPLY_INT_r13: {
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
CHECK_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef right;
_PyStackRef left;
_PyStackRef res;
gh-134584: JIT: Eliminate redundant refcount ops for X_INT (GH-142765)
2025-12-16 06:22:18 +08:00
_PyStackRef l;
_PyStackRef r;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef _stack_item_0 = _tos_cache0;
right = _stack_item_0;
left = stack_pointer[-1];
PyObject *left_o = PyStackRef_AsPyObjectBorrow(left);
PyObject *right_o = PyStackRef_AsPyObjectBorrow(right);
assert(PyLong_CheckExact(left_o));
assert(PyLong_CheckExact(right_o));
assert(_PyLong_BothAreCompact((PyLongObject *)left_o, (PyLongObject *)right_o));
STAT_INC(BINARY_OP, hit);
res = _PyCompactLong_Multiply((PyLongObject *)left_o, (PyLongObject *)right_o);
if (PyStackRef_IsNull(res)) {
UOP_STAT_INC(uopcode, miss);
GH-142718: JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. (GH-142762) JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. * Make sure that stack is in correct state at side exits with TOS cached values * Simplify choice of cached items for side exits
2025-12-15 19:01:29 +00:00
_tos_cache0 = right;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(1);
JUMP_TO_JUMP_TARGET();
}
gh-134584: JIT: Eliminate redundant refcount ops for X_INT (GH-142765)
2025-12-16 06:22:18 +08:00
l = left;
r = right;
_tos_cache2 = r;
_tos_cache1 = l;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_tos_cache0 = res;
gh-134584: JIT: Eliminate redundant refcount ops for X_INT (GH-142765)
2025-12-16 06:22:18 +08:00
SET_CURRENT_CACHED_VALUES(3);
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
stack_pointer += -1;
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
gh-134584: JIT: Eliminate redundant refcount ops for X_INT (GH-142765)
2025-12-16 06:22:18 +08:00
case _BINARY_OP_MULTIPLY_INT_r23: {
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
CHECK_CURRENT_CACHED_VALUES(2);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef right;
_PyStackRef left;
_PyStackRef res;
gh-134584: JIT: Eliminate redundant refcount ops for X_INT (GH-142765)
2025-12-16 06:22:18 +08:00
_PyStackRef l;
_PyStackRef r;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef _stack_item_0 = _tos_cache0;
_PyStackRef _stack_item_1 = _tos_cache1;
right = _stack_item_1;
left = _stack_item_0;
PyObject *left_o = PyStackRef_AsPyObjectBorrow(left);
PyObject *right_o = PyStackRef_AsPyObjectBorrow(right);
assert(PyLong_CheckExact(left_o));
assert(PyLong_CheckExact(right_o));
assert(_PyLong_BothAreCompact((PyLongObject *)left_o, (PyLongObject *)right_o));
STAT_INC(BINARY_OP, hit);
res = _PyCompactLong_Multiply((PyLongObject *)left_o, (PyLongObject *)right_o);
if (PyStackRef_IsNull(res)) {
UOP_STAT_INC(uopcode, miss);
GH-142718: JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. (GH-142762) JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. * Make sure that stack is in correct state at side exits with TOS cached values * Simplify choice of cached items for side exits
2025-12-15 19:01:29 +00:00
_tos_cache1 = right;
_tos_cache0 = left;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(2);
JUMP_TO_JUMP_TARGET();
}
gh-134584: JIT: Eliminate redundant refcount ops for X_INT (GH-142765)
2025-12-16 06:22:18 +08:00
l = left;
r = right;
_tos_cache2 = r;
_tos_cache1 = l;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_tos_cache0 = res;
gh-134584: JIT: Eliminate redundant refcount ops for X_INT (GH-142765)
2025-12-16 06:22:18 +08:00
SET_CURRENT_CACHED_VALUES(3);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
gh-134584: JIT: Eliminate redundant refcount ops for X_INT (GH-142765)
2025-12-16 06:22:18 +08:00
case _BINARY_OP_ADD_INT_r03: {
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
CHECK_CURRENT_CACHED_VALUES(0);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef right;
_PyStackRef left;
_PyStackRef res;
gh-134584: JIT: Eliminate redundant refcount ops for X_INT (GH-142765)
2025-12-16 06:22:18 +08:00
_PyStackRef l;
_PyStackRef r;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
right = stack_pointer[-1];
left = stack_pointer[-2];
PyObject *left_o = PyStackRef_AsPyObjectBorrow(left);
PyObject *right_o = PyStackRef_AsPyObjectBorrow(right);
assert(PyLong_CheckExact(left_o));
assert(PyLong_CheckExact(right_o));
assert(_PyLong_BothAreCompact((PyLongObject *)left_o, (PyLongObject *)right_o));
STAT_INC(BINARY_OP, hit);
res = _PyCompactLong_Add((PyLongObject *)left_o, (PyLongObject *)right_o);
if (PyStackRef_IsNull(res)) {
UOP_STAT_INC(uopcode, miss);
SET_CURRENT_CACHED_VALUES(0);
JUMP_TO_JUMP_TARGET();
}
gh-134584: JIT: Eliminate redundant refcount ops for X_INT (GH-142765)
2025-12-16 06:22:18 +08:00
l = left;
r = right;
_tos_cache2 = r;
_tos_cache1 = l;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_tos_cache0 = res;
gh-134584: JIT: Eliminate redundant refcount ops for X_INT (GH-142765)
2025-12-16 06:22:18 +08:00
SET_CURRENT_CACHED_VALUES(3);
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
stack_pointer += -2;
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
gh-134584: JIT: Eliminate redundant refcount ops for X_INT (GH-142765)
2025-12-16 06:22:18 +08:00
case _BINARY_OP_ADD_INT_r13: {
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
CHECK_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef right;
_PyStackRef left;
_PyStackRef res;
gh-134584: JIT: Eliminate redundant refcount ops for X_INT (GH-142765)
2025-12-16 06:22:18 +08:00
_PyStackRef l;
_PyStackRef r;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef _stack_item_0 = _tos_cache0;
right = _stack_item_0;
left = stack_pointer[-1];
PyObject *left_o = PyStackRef_AsPyObjectBorrow(left);
PyObject *right_o = PyStackRef_AsPyObjectBorrow(right);
assert(PyLong_CheckExact(left_o));
assert(PyLong_CheckExact(right_o));
assert(_PyLong_BothAreCompact((PyLongObject *)left_o, (PyLongObject *)right_o));
STAT_INC(BINARY_OP, hit);
res = _PyCompactLong_Add((PyLongObject *)left_o, (PyLongObject *)right_o);
if (PyStackRef_IsNull(res)) {
UOP_STAT_INC(uopcode, miss);
GH-142718: JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. (GH-142762) JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. * Make sure that stack is in correct state at side exits with TOS cached values * Simplify choice of cached items for side exits
2025-12-15 19:01:29 +00:00
_tos_cache0 = right;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(1);
JUMP_TO_JUMP_TARGET();
}
gh-134584: JIT: Eliminate redundant refcount ops for X_INT (GH-142765)
2025-12-16 06:22:18 +08:00
l = left;
r = right;
_tos_cache2 = r;
_tos_cache1 = l;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_tos_cache0 = res;
gh-134584: JIT: Eliminate redundant refcount ops for X_INT (GH-142765)
2025-12-16 06:22:18 +08:00
SET_CURRENT_CACHED_VALUES(3);
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
stack_pointer += -1;
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
gh-134584: JIT: Eliminate redundant refcount ops for X_INT (GH-142765)
2025-12-16 06:22:18 +08:00
case _BINARY_OP_ADD_INT_r23: {
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
CHECK_CURRENT_CACHED_VALUES(2);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef right;
_PyStackRef left;
_PyStackRef res;
gh-134584: JIT: Eliminate redundant refcount ops for X_INT (GH-142765)
2025-12-16 06:22:18 +08:00
_PyStackRef l;
_PyStackRef r;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef _stack_item_0 = _tos_cache0;
_PyStackRef _stack_item_1 = _tos_cache1;
right = _stack_item_1;
left = _stack_item_0;
PyObject *left_o = PyStackRef_AsPyObjectBorrow(left);
PyObject *right_o = PyStackRef_AsPyObjectBorrow(right);
assert(PyLong_CheckExact(left_o));
assert(PyLong_CheckExact(right_o));
assert(_PyLong_BothAreCompact((PyLongObject *)left_o, (PyLongObject *)right_o));
STAT_INC(BINARY_OP, hit);
res = _PyCompactLong_Add((PyLongObject *)left_o, (PyLongObject *)right_o);
if (PyStackRef_IsNull(res)) {
UOP_STAT_INC(uopcode, miss);
GH-142718: JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. (GH-142762) JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. * Make sure that stack is in correct state at side exits with TOS cached values * Simplify choice of cached items for side exits
2025-12-15 19:01:29 +00:00
_tos_cache1 = right;
_tos_cache0 = left;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(2);
JUMP_TO_JUMP_TARGET();
}
gh-134584: JIT: Eliminate redundant refcount ops for X_INT (GH-142765)
2025-12-16 06:22:18 +08:00
l = left;
r = right;
_tos_cache2 = r;
_tos_cache1 = l;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_tos_cache0 = res;
gh-134584: JIT: Eliminate redundant refcount ops for X_INT (GH-142765)
2025-12-16 06:22:18 +08:00
SET_CURRENT_CACHED_VALUES(3);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
gh-134584: JIT: Eliminate redundant refcount ops for X_INT (GH-142765)
2025-12-16 06:22:18 +08:00
case _BINARY_OP_SUBTRACT_INT_r03: {
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
CHECK_CURRENT_CACHED_VALUES(0);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef right;
_PyStackRef left;
_PyStackRef res;
gh-134584: JIT: Eliminate redundant refcount ops for X_INT (GH-142765)
2025-12-16 06:22:18 +08:00
_PyStackRef l;
_PyStackRef r;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
right = stack_pointer[-1];
left = stack_pointer[-2];
PyObject *left_o = PyStackRef_AsPyObjectBorrow(left);
PyObject *right_o = PyStackRef_AsPyObjectBorrow(right);
assert(PyLong_CheckExact(left_o));
assert(PyLong_CheckExact(right_o));
assert(_PyLong_BothAreCompact((PyLongObject *)left_o, (PyLongObject *)right_o));
STAT_INC(BINARY_OP, hit);
res = _PyCompactLong_Subtract((PyLongObject *)left_o, (PyLongObject *)right_o);
if (PyStackRef_IsNull(res)) {
UOP_STAT_INC(uopcode, miss);
SET_CURRENT_CACHED_VALUES(0);
JUMP_TO_JUMP_TARGET();
}
gh-134584: JIT: Eliminate redundant refcount ops for X_INT (GH-142765)
2025-12-16 06:22:18 +08:00
l = left;
r = right;
_tos_cache2 = r;
_tos_cache1 = l;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_tos_cache0 = res;
gh-134584: JIT: Eliminate redundant refcount ops for X_INT (GH-142765)
2025-12-16 06:22:18 +08:00
SET_CURRENT_CACHED_VALUES(3);
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
stack_pointer += -2;
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
gh-134584: JIT: Eliminate redundant refcount ops for X_INT (GH-142765)
2025-12-16 06:22:18 +08:00
case _BINARY_OP_SUBTRACT_INT_r13: {
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
CHECK_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef right;
_PyStackRef left;
_PyStackRef res;
gh-134584: JIT: Eliminate redundant refcount ops for X_INT (GH-142765)
2025-12-16 06:22:18 +08:00
_PyStackRef l;
_PyStackRef r;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef _stack_item_0 = _tos_cache0;
right = _stack_item_0;
left = stack_pointer[-1];
PyObject *left_o = PyStackRef_AsPyObjectBorrow(left);
PyObject *right_o = PyStackRef_AsPyObjectBorrow(right);
assert(PyLong_CheckExact(left_o));
assert(PyLong_CheckExact(right_o));
assert(_PyLong_BothAreCompact((PyLongObject *)left_o, (PyLongObject *)right_o));
STAT_INC(BINARY_OP, hit);
res = _PyCompactLong_Subtract((PyLongObject *)left_o, (PyLongObject *)right_o);
if (PyStackRef_IsNull(res)) {
UOP_STAT_INC(uopcode, miss);
GH-142718: JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. (GH-142762) JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. * Make sure that stack is in correct state at side exits with TOS cached values * Simplify choice of cached items for side exits
2025-12-15 19:01:29 +00:00
_tos_cache0 = right;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(1);
JUMP_TO_JUMP_TARGET();
}
gh-134584: JIT: Eliminate redundant refcount ops for X_INT (GH-142765)
2025-12-16 06:22:18 +08:00
l = left;
r = right;
_tos_cache2 = r;
_tos_cache1 = l;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_tos_cache0 = res;
gh-134584: JIT: Eliminate redundant refcount ops for X_INT (GH-142765)
2025-12-16 06:22:18 +08:00
SET_CURRENT_CACHED_VALUES(3);
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
stack_pointer += -1;
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
gh-134584: JIT: Eliminate redundant refcount ops for X_INT (GH-142765)
2025-12-16 06:22:18 +08:00
case _BINARY_OP_SUBTRACT_INT_r23: {
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
CHECK_CURRENT_CACHED_VALUES(2);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef right;
_PyStackRef left;
_PyStackRef res;
gh-134584: JIT: Eliminate redundant refcount ops for X_INT (GH-142765)
2025-12-16 06:22:18 +08:00
_PyStackRef l;
_PyStackRef r;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef _stack_item_0 = _tos_cache0;
_PyStackRef _stack_item_1 = _tos_cache1;
right = _stack_item_1;
left = _stack_item_0;
PyObject *left_o = PyStackRef_AsPyObjectBorrow(left);
PyObject *right_o = PyStackRef_AsPyObjectBorrow(right);
assert(PyLong_CheckExact(left_o));
assert(PyLong_CheckExact(right_o));
assert(_PyLong_BothAreCompact((PyLongObject *)left_o, (PyLongObject *)right_o));
STAT_INC(BINARY_OP, hit);
res = _PyCompactLong_Subtract((PyLongObject *)left_o, (PyLongObject *)right_o);
if (PyStackRef_IsNull(res)) {
UOP_STAT_INC(uopcode, miss);
GH-142718: JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. (GH-142762) JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. * Make sure that stack is in correct state at side exits with TOS cached values * Simplify choice of cached items for side exits
2025-12-15 19:01:29 +00:00
_tos_cache1 = right;
_tos_cache0 = left;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(2);
JUMP_TO_JUMP_TARGET();
}
gh-134584: JIT: Eliminate redundant refcount ops for X_INT (GH-142765)
2025-12-16 06:22:18 +08:00
l = left;
r = right;
_tos_cache2 = r;
_tos_cache1 = l;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_tos_cache0 = res;
gh-134584: JIT: Eliminate redundant refcount ops for X_INT (GH-142765)
2025-12-16 06:22:18 +08:00
SET_CURRENT_CACHED_VALUES(3);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _GUARD_NOS_FLOAT_r02: {
CHECK_CURRENT_CACHED_VALUES(0);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef left;
left = stack_pointer[-2];
PyObject *left_o = PyStackRef_AsPyObjectBorrow(left);
if (!PyFloat_CheckExact(left_o)) {
UOP_STAT_INC(uopcode, miss);
SET_CURRENT_CACHED_VALUES(0);
JUMP_TO_JUMP_TARGET();
}
_tos_cache1 = stack_pointer[-1];
_tos_cache0 = left;
SET_CURRENT_CACHED_VALUES(2);
stack_pointer += -2;
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _GUARD_NOS_FLOAT_r12: {
CHECK_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef left;
_PyStackRef _stack_item_0 = _tos_cache0;
left = stack_pointer[-1];
PyObject *left_o = PyStackRef_AsPyObjectBorrow(left);
if (!PyFloat_CheckExact(left_o)) {
UOP_STAT_INC(uopcode, miss);
GH-142718: JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. (GH-142762) JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. * Make sure that stack is in correct state at side exits with TOS cached values * Simplify choice of cached items for side exits
2025-12-15 19:01:29 +00:00
_tos_cache0 = _stack_item_0;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(1);
JUMP_TO_JUMP_TARGET();
}
_tos_cache1 = _stack_item_0;
_tos_cache0 = left;
SET_CURRENT_CACHED_VALUES(2);
stack_pointer += -1;
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _GUARD_NOS_FLOAT_r22: {
CHECK_CURRENT_CACHED_VALUES(2);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef left;
_PyStackRef _stack_item_0 = _tos_cache0;
_PyStackRef _stack_item_1 = _tos_cache1;
left = _stack_item_0;
PyObject *left_o = PyStackRef_AsPyObjectBorrow(left);
if (!PyFloat_CheckExact(left_o)) {
UOP_STAT_INC(uopcode, miss);
GH-142718: JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. (GH-142762) JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. * Make sure that stack is in correct state at side exits with TOS cached values * Simplify choice of cached items for side exits
2025-12-15 19:01:29 +00:00
_tos_cache1 = _stack_item_1;
_tos_cache0 = left;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(2);
JUMP_TO_JUMP_TARGET();
}
_tos_cache1 = _stack_item_1;
_tos_cache0 = left;
SET_CURRENT_CACHED_VALUES(2);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _GUARD_NOS_FLOAT_r33: {
CHECK_CURRENT_CACHED_VALUES(3);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef left;
_PyStackRef _stack_item_0 = _tos_cache0;
_PyStackRef _stack_item_1 = _tos_cache1;
_PyStackRef _stack_item_2 = _tos_cache2;
left = _stack_item_1;
PyObject *left_o = PyStackRef_AsPyObjectBorrow(left);
if (!PyFloat_CheckExact(left_o)) {
UOP_STAT_INC(uopcode, miss);
GH-142718: JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. (GH-142762) JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. * Make sure that stack is in correct state at side exits with TOS cached values * Simplify choice of cached items for side exits
2025-12-15 19:01:29 +00:00
_tos_cache2 = _stack_item_2;
_tos_cache1 = left;
_tos_cache0 = _stack_item_0;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(3);
JUMP_TO_JUMP_TARGET();
}
_tos_cache2 = _stack_item_2;
_tos_cache1 = left;
_tos_cache0 = _stack_item_0;
SET_CURRENT_CACHED_VALUES(3);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _GUARD_TOS_FLOAT_r01: {
CHECK_CURRENT_CACHED_VALUES(0);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef value;
value = stack_pointer[-1];
PyObject *value_o = PyStackRef_AsPyObjectBorrow(value);
if (!PyFloat_CheckExact(value_o)) {
UOP_STAT_INC(uopcode, miss);
SET_CURRENT_CACHED_VALUES(0);
JUMP_TO_JUMP_TARGET();
}
_tos_cache0 = value;
SET_CURRENT_CACHED_VALUES(1);
stack_pointer += -1;
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _GUARD_TOS_FLOAT_r11: {
CHECK_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef value;
_PyStackRef _stack_item_0 = _tos_cache0;
value = _stack_item_0;
PyObject *value_o = PyStackRef_AsPyObjectBorrow(value);
if (!PyFloat_CheckExact(value_o)) {
UOP_STAT_INC(uopcode, miss);
GH-142718: JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. (GH-142762) JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. * Make sure that stack is in correct state at side exits with TOS cached values * Simplify choice of cached items for side exits
2025-12-15 19:01:29 +00:00
_tos_cache0 = value;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(1);
JUMP_TO_JUMP_TARGET();
}
_tos_cache0 = value;
SET_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _GUARD_TOS_FLOAT_r22: {
CHECK_CURRENT_CACHED_VALUES(2);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef value;
_PyStackRef _stack_item_0 = _tos_cache0;
_PyStackRef _stack_item_1 = _tos_cache1;
value = _stack_item_1;
PyObject *value_o = PyStackRef_AsPyObjectBorrow(value);
if (!PyFloat_CheckExact(value_o)) {
UOP_STAT_INC(uopcode, miss);
GH-142718: JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. (GH-142762) JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. * Make sure that stack is in correct state at side exits with TOS cached values * Simplify choice of cached items for side exits
2025-12-15 19:01:29 +00:00
_tos_cache1 = value;
_tos_cache0 = _stack_item_0;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(2);
JUMP_TO_JUMP_TARGET();
}
_tos_cache1 = value;
_tos_cache0 = _stack_item_0;
SET_CURRENT_CACHED_VALUES(2);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _GUARD_TOS_FLOAT_r33: {
CHECK_CURRENT_CACHED_VALUES(3);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef value;
_PyStackRef _stack_item_0 = _tos_cache0;
_PyStackRef _stack_item_1 = _tos_cache1;
_PyStackRef _stack_item_2 = _tos_cache2;
value = _stack_item_2;
PyObject *value_o = PyStackRef_AsPyObjectBorrow(value);
if (!PyFloat_CheckExact(value_o)) {
UOP_STAT_INC(uopcode, miss);
GH-142718: JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. (GH-142762) JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. * Make sure that stack is in correct state at side exits with TOS cached values * Simplify choice of cached items for side exits
2025-12-15 19:01:29 +00:00
_tos_cache2 = value;
_tos_cache1 = _stack_item_1;
_tos_cache0 = _stack_item_0;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(3);
JUMP_TO_JUMP_TARGET();
}
_tos_cache2 = value;
_tos_cache1 = _stack_item_1;
_tos_cache0 = _stack_item_0;
SET_CURRENT_CACHED_VALUES(3);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
case _BINARY_OP_MULTIPLY_FLOAT_r03: {
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
CHECK_CURRENT_CACHED_VALUES(0);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
_PyStackRef right;
_PyStackRef left;
_PyStackRef res;
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
_PyStackRef l;
_PyStackRef r;
gh-106812: Refactor cases_generator to allow uops with array stack effects (#107564) Introducing a new file, stacking.py, that takes over several responsibilities related to symbolic evaluation of push/pop operations, with more generality.
2023-08-04 09:35:56 -07:00
right = stack_pointer[-1];
left = stack_pointer[-2];
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
PyObject *left_o = PyStackRef_AsPyObjectBorrow(left);
PyObject *right_o = PyStackRef_AsPyObjectBorrow(right);
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
assert(PyFloat_CheckExact(left_o));
assert(PyFloat_CheckExact(right_o));
GH-135379: Specialize int operations for compact ints only (GH-135668)
2025-06-19 11:10:29 +01:00
STAT_INC(BINARY_OP, hit);
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
double dres =
((PyFloatObject *)left_o)->ob_fval *
((PyFloatObject *)right_o)->ob_fval;
res = PyStackRef_FromPyObjectSteal(PyFloat_FromDouble(dres));
GH-135379: Specialize int operations for compact ints only (GH-135668)
2025-06-19 11:10:29 +01:00
if (PyStackRef_IsNull(res)) {
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(0);
JUMP_TO_ERROR();
gh-135474: Specialize arithmetic only on compact ints (GH-135479) Specialize arithmetic only on compact ints. This also makes int operations non-escaping in the JIT and in tier 1.
2025-06-14 17:13:32 +08:00
}
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
l = left;
r = right;
_tos_cache2 = r;
_tos_cache1 = l;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_tos_cache0 = res;
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
SET_CURRENT_CACHED_VALUES(3);
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
stack_pointer += -2;
GH-141794: Limit size of generated machine code. (GH-142228) * Factor out bodies of the largest uops, to reduce jit code size. * Factor out common assert, also reducing jit code size. * Limit size of jitted code for a single executor to 1MB.
2025-12-03 17:43:35 +00:00
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-104584: Baby steps towards generating and executing traces (#105924) Added a new, experimental, tracing optimizer and interpreter (a.k.a. "tier 2"). This currently pessimizes, so don't use yet -- this is infrastructure so we can experiment with optimizing passes. To enable it, pass ``-Xuops`` or set ``PYTHONUOPS=1``. To get debug output, set ``PYTHONUOPSDEBUG=N`` where ``N`` is a debug level (0-4, where 0 is no debug output and 4 is excessively verbose). All of this code is likely to change dramatically before the 3.13 feature freeze. But this is a first step.
2023-06-26 19:02:57 -07:00
break;
}
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
case _BINARY_OP_MULTIPLY_FLOAT_r13: {
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
CHECK_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
_PyStackRef right;
_PyStackRef left;
_PyStackRef res;
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
_PyStackRef l;
_PyStackRef r;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef _stack_item_0 = _tos_cache0;
right = _stack_item_0;
left = stack_pointer[-1];
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
PyObject *left_o = PyStackRef_AsPyObjectBorrow(left);
PyObject *right_o = PyStackRef_AsPyObjectBorrow(right);
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
assert(PyFloat_CheckExact(left_o));
assert(PyFloat_CheckExact(right_o));
GH-135379: Specialize int operations for compact ints only (GH-135668)
2025-06-19 11:10:29 +01:00
STAT_INC(BINARY_OP, hit);
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
double dres =
((PyFloatObject *)left_o)->ob_fval *
((PyFloatObject *)right_o)->ob_fval;
res = PyStackRef_FromPyObjectSteal(PyFloat_FromDouble(dres));
GH-135379: Specialize int operations for compact ints only (GH-135668)
2025-06-19 11:10:29 +01:00
if (PyStackRef_IsNull(res)) {
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
stack_pointer[0] = right;
stack_pointer += 1;
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(0);
JUMP_TO_ERROR();
gh-135474: Specialize arithmetic only on compact ints (GH-135479) Specialize arithmetic only on compact ints. This also makes int operations non-escaping in the JIT and in tier 1.
2025-06-14 17:13:32 +08:00
}
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
l = left;
r = right;
_tos_cache2 = r;
_tos_cache1 = l;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_tos_cache0 = res;
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
SET_CURRENT_CACHED_VALUES(3);
GH-111485: Factor out tier 2 code generation from the rest of the interpreter code generator (GH-112968)
2023-12-12 12:12:17 +00:00
stack_pointer += -1;
GH-141794: Limit size of generated machine code. (GH-142228) * Factor out bodies of the largest uops, to reduce jit code size. * Factor out common assert, also reducing jit code size. * Limit size of jitted code for a single executor to 1MB.
2025-12-03 17:43:35 +00:00
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-104584: Baby steps towards generating and executing traces (#105924) Added a new, experimental, tracing optimizer and interpreter (a.k.a. "tier 2"). This currently pessimizes, so don't use yet -- this is infrastructure so we can experiment with optimizing passes. To enable it, pass ``-Xuops`` or set ``PYTHONUOPS=1``. To get debug output, set ``PYTHONUOPSDEBUG=N`` where ``N`` is a debug level (0-4, where 0 is no debug output and 4 is excessively verbose). All of this code is likely to change dramatically before the 3.13 feature freeze. But this is a first step.
2023-06-26 19:02:57 -07:00
break;
}
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
case _BINARY_OP_MULTIPLY_FLOAT_r23: {
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
CHECK_CURRENT_CACHED_VALUES(2);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
_PyStackRef right;
_PyStackRef left;
_PyStackRef res;
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
_PyStackRef l;
_PyStackRef r;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef _stack_item_0 = _tos_cache0;
_PyStackRef _stack_item_1 = _tos_cache1;
right = _stack_item_1;
left = _stack_item_0;
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
PyObject *left_o = PyStackRef_AsPyObjectBorrow(left);
PyObject *right_o = PyStackRef_AsPyObjectBorrow(right);
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
assert(PyFloat_CheckExact(left_o));
assert(PyFloat_CheckExact(right_o));
GH-135379: Specialize int operations for compact ints only (GH-135668)
2025-06-19 11:10:29 +01:00
STAT_INC(BINARY_OP, hit);
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
double dres =
((PyFloatObject *)left_o)->ob_fval *
((PyFloatObject *)right_o)->ob_fval;
res = PyStackRef_FromPyObjectSteal(PyFloat_FromDouble(dres));
GH-135379: Specialize int operations for compact ints only (GH-135668)
2025-06-19 11:10:29 +01:00
if (PyStackRef_IsNull(res)) {
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
stack_pointer[0] = left;
stack_pointer[1] = right;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
stack_pointer += 2;
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
SET_CURRENT_CACHED_VALUES(0);
JUMP_TO_ERROR();
GH-115480: Reduce guard strength for binary ops when type of one operand is known already (GH-118050)
2024-04-22 13:34:06 +01:00
}
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
l = left;
r = right;
_tos_cache2 = r;
_tos_cache1 = l;
_tos_cache0 = res;
SET_CURRENT_CACHED_VALUES(3);
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-115480: Reduce guard strength for binary ops when type of one operand is known already (GH-118050)
2024-04-22 13:34:06 +01:00
break;
}
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
case _BINARY_OP_ADD_FLOAT_r03: {
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
CHECK_CURRENT_CACHED_VALUES(0);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
_PyStackRef right;
_PyStackRef left;
_PyStackRef res;
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
_PyStackRef l;
_PyStackRef r;
gh-106812: Refactor cases_generator to allow uops with array stack effects (#107564) Introducing a new file, stacking.py, that takes over several responsibilities related to symbolic evaluation of push/pop operations, with more generality.
2023-08-04 09:35:56 -07:00
right = stack_pointer[-1];
left = stack_pointer[-2];
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
PyObject *left_o = PyStackRef_AsPyObjectBorrow(left);
PyObject *right_o = PyStackRef_AsPyObjectBorrow(right);
GH-127809: Fix the JIT's understanding of ** (GH-127844)
2025-01-07 17:25:48 -08:00
assert(PyFloat_CheckExact(left_o));
assert(PyFloat_CheckExact(right_o));
gh-104584: Baby steps towards generating and executing traces (#105924) Added a new, experimental, tracing optimizer and interpreter (a.k.a. "tier 2"). This currently pessimizes, so don't use yet -- this is infrastructure so we can experiment with optimizing passes. To enable it, pass ``-Xuops`` or set ``PYTHONUOPS=1``. To get debug output, set ``PYTHONUOPSDEBUG=N`` where ``N`` is a debug level (0-4, where 0 is no debug output and 4 is excessively verbose). All of this code is likely to change dramatically before the 3.13 feature freeze. But this is a first step.
2023-06-26 19:02:57 -07:00
STAT_INC(BINARY_OP, hit);
double dres =
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
((PyFloatObject *)left_o)->ob_fval +
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
((PyFloatObject *)right_o)->ob_fval;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
res = PyStackRef_FromPyObjectSteal(PyFloat_FromDouble(dres));
GH-127705: Use `_PyStackRef`s in the default build. (GH-127875)
2025-03-10 14:06:56 +00:00
if (PyStackRef_IsNull(res)) {
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(0);
GH-128682: Make `PyStackRef_CLOSE` escaping. (GH-129404)
2025-02-03 12:41:32 +00:00
JUMP_TO_ERROR();
}
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
l = left;
r = right;
_tos_cache2 = r;
_tos_cache1 = l;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_tos_cache0 = res;
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
SET_CURRENT_CACHED_VALUES(3);
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
stack_pointer += -2;
GH-141794: Limit size of generated machine code. (GH-142228) * Factor out bodies of the largest uops, to reduce jit code size. * Factor out common assert, also reducing jit code size. * Limit size of jitted code for a single executor to 1MB.
2025-12-03 17:43:35 +00:00
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-104584: Baby steps towards generating and executing traces (#105924) Added a new, experimental, tracing optimizer and interpreter (a.k.a. "tier 2"). This currently pessimizes, so don't use yet -- this is infrastructure so we can experiment with optimizing passes. To enable it, pass ``-Xuops`` or set ``PYTHONUOPS=1``. To get debug output, set ``PYTHONUOPSDEBUG=N`` where ``N`` is a debug level (0-4, where 0 is no debug output and 4 is excessively verbose). All of this code is likely to change dramatically before the 3.13 feature freeze. But this is a first step.
2023-06-26 19:02:57 -07:00
break;
}
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
case _BINARY_OP_ADD_FLOAT_r13: {
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
CHECK_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
_PyStackRef right;
_PyStackRef left;
_PyStackRef res;
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
_PyStackRef l;
_PyStackRef r;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef _stack_item_0 = _tos_cache0;
right = _stack_item_0;
left = stack_pointer[-1];
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
PyObject *left_o = PyStackRef_AsPyObjectBorrow(left);
PyObject *right_o = PyStackRef_AsPyObjectBorrow(right);
GH-127809: Fix the JIT's understanding of ** (GH-127844)
2025-01-07 17:25:48 -08:00
assert(PyFloat_CheckExact(left_o));
assert(PyFloat_CheckExact(right_o));
gh-104584: Baby steps towards generating and executing traces (#105924) Added a new, experimental, tracing optimizer and interpreter (a.k.a. "tier 2"). This currently pessimizes, so don't use yet -- this is infrastructure so we can experiment with optimizing passes. To enable it, pass ``-Xuops`` or set ``PYTHONUOPS=1``. To get debug output, set ``PYTHONUOPSDEBUG=N`` where ``N`` is a debug level (0-4, where 0 is no debug output and 4 is excessively verbose). All of this code is likely to change dramatically before the 3.13 feature freeze. But this is a first step.
2023-06-26 19:02:57 -07:00
STAT_INC(BINARY_OP, hit);
double dres =
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
((PyFloatObject *)left_o)->ob_fval +
((PyFloatObject *)right_o)->ob_fval;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
res = PyStackRef_FromPyObjectSteal(PyFloat_FromDouble(dres));
GH-127705: Use `_PyStackRef`s in the default build. (GH-127875)
2025-03-10 14:06:56 +00:00
if (PyStackRef_IsNull(res)) {
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
stack_pointer[0] = right;
stack_pointer += 1;
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(0);
GH-128682: Make `PyStackRef_CLOSE` escaping. (GH-129404)
2025-02-03 12:41:32 +00:00
JUMP_TO_ERROR();
}
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
l = left;
r = right;
_tos_cache2 = r;
_tos_cache1 = l;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_tos_cache0 = res;
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
SET_CURRENT_CACHED_VALUES(3);
GH-111485: Factor out tier 2 code generation from the rest of the interpreter code generator (GH-112968)
2023-12-12 12:12:17 +00:00
stack_pointer += -1;
GH-141794: Limit size of generated machine code. (GH-142228) * Factor out bodies of the largest uops, to reduce jit code size. * Factor out common assert, also reducing jit code size. * Limit size of jitted code for a single executor to 1MB.
2025-12-03 17:43:35 +00:00
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-104584: Baby steps towards generating and executing traces (#105924) Added a new, experimental, tracing optimizer and interpreter (a.k.a. "tier 2"). This currently pessimizes, so don't use yet -- this is infrastructure so we can experiment with optimizing passes. To enable it, pass ``-Xuops`` or set ``PYTHONUOPS=1``. To get debug output, set ``PYTHONUOPSDEBUG=N`` where ``N`` is a debug level (0-4, where 0 is no debug output and 4 is excessively verbose). All of this code is likely to change dramatically before the 3.13 feature freeze. But this is a first step.
2023-06-26 19:02:57 -07:00
break;
}
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
case _BINARY_OP_ADD_FLOAT_r23: {
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
CHECK_CURRENT_CACHED_VALUES(2);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
_PyStackRef right;
_PyStackRef left;
_PyStackRef res;
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
_PyStackRef l;
_PyStackRef r;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef _stack_item_0 = _tos_cache0;
_PyStackRef _stack_item_1 = _tos_cache1;
right = _stack_item_1;
left = _stack_item_0;
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
PyObject *left_o = PyStackRef_AsPyObjectBorrow(left);
PyObject *right_o = PyStackRef_AsPyObjectBorrow(right);
GH-127809: Fix the JIT's understanding of ** (GH-127844)
2025-01-07 17:25:48 -08:00
assert(PyFloat_CheckExact(left_o));
assert(PyFloat_CheckExact(right_o));
gh-104584: Baby steps towards generating and executing traces (#105924) Added a new, experimental, tracing optimizer and interpreter (a.k.a. "tier 2"). This currently pessimizes, so don't use yet -- this is infrastructure so we can experiment with optimizing passes. To enable it, pass ``-Xuops`` or set ``PYTHONUOPS=1``. To get debug output, set ``PYTHONUOPSDEBUG=N`` where ``N`` is a debug level (0-4, where 0 is no debug output and 4 is excessively verbose). All of this code is likely to change dramatically before the 3.13 feature freeze. But this is a first step.
2023-06-26 19:02:57 -07:00
STAT_INC(BINARY_OP, hit);
double dres =
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
((PyFloatObject *)left_o)->ob_fval +
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
((PyFloatObject *)right_o)->ob_fval;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
res = PyStackRef_FromPyObjectSteal(PyFloat_FromDouble(dres));
GH-127705: Use `_PyStackRef`s in the default build. (GH-127875)
2025-03-10 14:06:56 +00:00
if (PyStackRef_IsNull(res)) {
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
stack_pointer[0] = left;
stack_pointer[1] = right;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
stack_pointer += 2;
GH-141794: Limit size of generated machine code. (GH-142228) * Factor out bodies of the largest uops, to reduce jit code size. * Factor out common assert, also reducing jit code size. * Limit size of jitted code for a single executor to 1MB.
2025-12-03 17:43:35 +00:00
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(0);
GH-128682: Make `PyStackRef_CLOSE` escaping. (GH-129404)
2025-02-03 12:41:32 +00:00
JUMP_TO_ERROR();
}
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
l = left;
r = right;
_tos_cache2 = r;
_tos_cache1 = l;
_tos_cache0 = res;
SET_CURRENT_CACHED_VALUES(3);
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-104584: Baby steps towards generating and executing traces (#105924) Added a new, experimental, tracing optimizer and interpreter (a.k.a. "tier 2"). This currently pessimizes, so don't use yet -- this is infrastructure so we can experiment with optimizing passes. To enable it, pass ``-Xuops`` or set ``PYTHONUOPS=1``. To get debug output, set ``PYTHONUOPSDEBUG=N`` where ``N`` is a debug level (0-4, where 0 is no debug output and 4 is excessively verbose). All of this code is likely to change dramatically before the 3.13 feature freeze. But this is a first step.
2023-06-26 19:02:57 -07:00
break;
}
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
case _BINARY_OP_SUBTRACT_FLOAT_r03: {
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
CHECK_CURRENT_CACHED_VALUES(0);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-134584: Decref elimination for float ops in the JIT (GH-134588) This PR adds a PyJitRef API to the JIT's optimizer that mimics the _PyStackRef API. This allows it to track references and their stack lifetimes properly. Thus opening up the doorway to refcount elimination in the JIT.
2025-06-17 23:25:53 +08:00
_PyStackRef right;
_PyStackRef left;
_PyStackRef res;
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
_PyStackRef l;
_PyStackRef r;
gh-134584: Decref elimination for float ops in the JIT (GH-134588) This PR adds a PyJitRef API to the JIT's optimizer that mimics the _PyStackRef API. This allows it to track references and their stack lifetimes properly. Thus opening up the doorway to refcount elimination in the JIT.
2025-06-17 23:25:53 +08:00
right = stack_pointer[-1];
left = stack_pointer[-2];
PyObject *left_o = PyStackRef_AsPyObjectBorrow(left);
PyObject *right_o = PyStackRef_AsPyObjectBorrow(right);
assert(PyFloat_CheckExact(left_o));
assert(PyFloat_CheckExact(right_o));
STAT_INC(BINARY_OP, hit);
double dres =
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
((PyFloatObject *)left_o)->ob_fval -
gh-134584: Decref elimination for float ops in the JIT (GH-134588) This PR adds a PyJitRef API to the JIT's optimizer that mimics the _PyStackRef API. This allows it to track references and their stack lifetimes properly. Thus opening up the doorway to refcount elimination in the JIT.
2025-06-17 23:25:53 +08:00
((PyFloatObject *)right_o)->ob_fval;
res = PyStackRef_FromPyObjectSteal(PyFloat_FromDouble(dres));
if (PyStackRef_IsNull(res)) {
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(0);
gh-134584: Decref elimination for float ops in the JIT (GH-134588) This PR adds a PyJitRef API to the JIT's optimizer that mimics the _PyStackRef API. This allows it to track references and their stack lifetimes properly. Thus opening up the doorway to refcount elimination in the JIT.
2025-06-17 23:25:53 +08:00
JUMP_TO_ERROR();
}
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
l = left;
r = right;
_tos_cache2 = r;
_tos_cache1 = l;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_tos_cache0 = res;
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
SET_CURRENT_CACHED_VALUES(3);
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
stack_pointer += -2;
GH-141794: Limit size of generated machine code. (GH-142228) * Factor out bodies of the largest uops, to reduce jit code size. * Factor out common assert, also reducing jit code size. * Limit size of jitted code for a single executor to 1MB.
2025-12-03 17:43:35 +00:00
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-134584: Decref elimination for float ops in the JIT (GH-134588) This PR adds a PyJitRef API to the JIT's optimizer that mimics the _PyStackRef API. This allows it to track references and their stack lifetimes properly. Thus opening up the doorway to refcount elimination in the JIT.
2025-06-17 23:25:53 +08:00
break;
}
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
case _BINARY_OP_SUBTRACT_FLOAT_r13: {
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
CHECK_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-134584: Decref elimination for float ops in the JIT (GH-134588) This PR adds a PyJitRef API to the JIT's optimizer that mimics the _PyStackRef API. This allows it to track references and their stack lifetimes properly. Thus opening up the doorway to refcount elimination in the JIT.
2025-06-17 23:25:53 +08:00
_PyStackRef right;
_PyStackRef left;
_PyStackRef res;
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
_PyStackRef l;
_PyStackRef r;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef _stack_item_0 = _tos_cache0;
right = _stack_item_0;
left = stack_pointer[-1];
gh-134584: Decref elimination for float ops in the JIT (GH-134588) This PR adds a PyJitRef API to the JIT's optimizer that mimics the _PyStackRef API. This allows it to track references and their stack lifetimes properly. Thus opening up the doorway to refcount elimination in the JIT.
2025-06-17 23:25:53 +08:00
PyObject *left_o = PyStackRef_AsPyObjectBorrow(left);
PyObject *right_o = PyStackRef_AsPyObjectBorrow(right);
assert(PyFloat_CheckExact(left_o));
assert(PyFloat_CheckExact(right_o));
STAT_INC(BINARY_OP, hit);
double dres =
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
((PyFloatObject *)left_o)->ob_fval -
gh-134584: Decref elimination for float ops in the JIT (GH-134588) This PR adds a PyJitRef API to the JIT's optimizer that mimics the _PyStackRef API. This allows it to track references and their stack lifetimes properly. Thus opening up the doorway to refcount elimination in the JIT.
2025-06-17 23:25:53 +08:00
((PyFloatObject *)right_o)->ob_fval;
res = PyStackRef_FromPyObjectSteal(PyFloat_FromDouble(dres));
if (PyStackRef_IsNull(res)) {
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
stack_pointer[0] = right;
stack_pointer += 1;
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(0);
gh-134584: Decref elimination for float ops in the JIT (GH-134588) This PR adds a PyJitRef API to the JIT's optimizer that mimics the _PyStackRef API. This allows it to track references and their stack lifetimes properly. Thus opening up the doorway to refcount elimination in the JIT.
2025-06-17 23:25:53 +08:00
JUMP_TO_ERROR();
}
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
l = left;
r = right;
_tos_cache2 = r;
_tos_cache1 = l;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_tos_cache0 = res;
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
SET_CURRENT_CACHED_VALUES(3);
gh-134584: Decref elimination for float ops in the JIT (GH-134588) This PR adds a PyJitRef API to the JIT's optimizer that mimics the _PyStackRef API. This allows it to track references and their stack lifetimes properly. Thus opening up the doorway to refcount elimination in the JIT.
2025-06-17 23:25:53 +08:00
stack_pointer += -1;
GH-141794: Limit size of generated machine code. (GH-142228) * Factor out bodies of the largest uops, to reduce jit code size. * Factor out common assert, also reducing jit code size. * Limit size of jitted code for a single executor to 1MB.
2025-12-03 17:43:35 +00:00
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-134584: Decref elimination for float ops in the JIT (GH-134588) This PR adds a PyJitRef API to the JIT's optimizer that mimics the _PyStackRef API. This allows it to track references and their stack lifetimes properly. Thus opening up the doorway to refcount elimination in the JIT.
2025-06-17 23:25:53 +08:00
break;
}
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
case _BINARY_OP_SUBTRACT_FLOAT_r23: {
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
CHECK_CURRENT_CACHED_VALUES(2);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-134584: Decref elimination for float ops in the JIT (GH-134588) This PR adds a PyJitRef API to the JIT's optimizer that mimics the _PyStackRef API. This allows it to track references and their stack lifetimes properly. Thus opening up the doorway to refcount elimination in the JIT.
2025-06-17 23:25:53 +08:00
_PyStackRef right;
_PyStackRef left;
_PyStackRef res;
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
_PyStackRef l;
_PyStackRef r;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef _stack_item_0 = _tos_cache0;
_PyStackRef _stack_item_1 = _tos_cache1;
right = _stack_item_1;
left = _stack_item_0;
gh-134584: Decref elimination for float ops in the JIT (GH-134588) This PR adds a PyJitRef API to the JIT's optimizer that mimics the _PyStackRef API. This allows it to track references and their stack lifetimes properly. Thus opening up the doorway to refcount elimination in the JIT.
2025-06-17 23:25:53 +08:00
PyObject *left_o = PyStackRef_AsPyObjectBorrow(left);
PyObject *right_o = PyStackRef_AsPyObjectBorrow(right);
assert(PyFloat_CheckExact(left_o));
assert(PyFloat_CheckExact(right_o));
STAT_INC(BINARY_OP, hit);
double dres =
((PyFloatObject *)left_o)->ob_fval -
((PyFloatObject *)right_o)->ob_fval;
res = PyStackRef_FromPyObjectSteal(PyFloat_FromDouble(dres));
if (PyStackRef_IsNull(res)) {
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
stack_pointer[0] = left;
stack_pointer[1] = right;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
stack_pointer += 2;
GH-141794: Limit size of generated machine code. (GH-142228) * Factor out bodies of the largest uops, to reduce jit code size. * Factor out common assert, also reducing jit code size. * Limit size of jitted code for a single executor to 1MB.
2025-12-03 17:43:35 +00:00
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(0);
gh-134584: Decref elimination for float ops in the JIT (GH-134588) This PR adds a PyJitRef API to the JIT's optimizer that mimics the _PyStackRef API. This allows it to track references and their stack lifetimes properly. Thus opening up the doorway to refcount elimination in the JIT.
2025-06-17 23:25:53 +08:00
JUMP_TO_ERROR();
}
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
l = left;
r = right;
_tos_cache2 = r;
_tos_cache1 = l;
_tos_cache0 = res;
SET_CURRENT_CACHED_VALUES(3);
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-134584: Decref elimination for float ops in the JIT (GH-134588) This PR adds a PyJitRef API to the JIT's optimizer that mimics the _PyStackRef API. This allows it to track references and their stack lifetimes properly. Thus opening up the doorway to refcount elimination in the JIT.
2025-06-17 23:25:53 +08:00
break;
}
gh-134584: Remove redundant refcount from _BINARY_OP_ADD_UNICODE (gh-142825)
2025-12-18 21:33:18 +09:00
case _BINARY_OP_ADD_UNICODE_r03: {
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
CHECK_CURRENT_CACHED_VALUES(0);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
_PyStackRef right;
_PyStackRef left;
_PyStackRef res;
gh-134584: Remove redundant refcount from _BINARY_OP_ADD_UNICODE (gh-142825)
2025-12-18 21:33:18 +09:00
_PyStackRef l;
_PyStackRef r;
gh-106812: Refactor cases_generator to allow uops with array stack effects (#107564) Introducing a new file, stacking.py, that takes over several responsibilities related to symbolic evaluation of push/pop operations, with more generality.
2023-08-04 09:35:56 -07:00
right = stack_pointer[-1];
left = stack_pointer[-2];
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
PyObject *left_o = PyStackRef_AsPyObjectBorrow(left);
PyObject *right_o = PyStackRef_AsPyObjectBorrow(right);
GH-127809: Fix the JIT's understanding of ** (GH-127844)
2025-01-07 17:25:48 -08:00
assert(PyUnicode_CheckExact(left_o));
assert(PyUnicode_CheckExact(right_o));
gh-104584: Baby steps towards generating and executing traces (#105924) Added a new, experimental, tracing optimizer and interpreter (a.k.a. "tier 2"). This currently pessimizes, so don't use yet -- this is infrastructure so we can experiment with optimizing passes. To enable it, pass ``-Xuops`` or set ``PYTHONUOPS=1``. To get debug output, set ``PYTHONUOPSDEBUG=N`` where ``N`` is a debug level (0-4, where 0 is no debug output and 4 is excessively verbose). All of this code is likely to change dramatically before the 3.13 feature freeze. But this is a first step.
2023-06-26 19:02:57 -07:00
STAT_INC(BINARY_OP, hit);
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
PyObject *res_o = PyUnicode_Concat(left_o, right_o);
gh-134584: Remove redundant refcount from _BINARY_OP_ADD_UNICODE (gh-142825)
2025-12-18 21:33:18 +09:00
res = PyStackRef_FromPyObjectSteal(res_o);
if (PyStackRef_IsNull(res)) {
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(0);
JUMP_TO_ERROR();
}
gh-134584: Remove redundant refcount from _BINARY_OP_ADD_UNICODE (gh-142825)
2025-12-18 21:33:18 +09:00
l = left;
r = right;
_tos_cache2 = r;
_tos_cache1 = l;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_tos_cache0 = res;
gh-134584: Remove redundant refcount from _BINARY_OP_ADD_UNICODE (gh-142825)
2025-12-18 21:33:18 +09:00
SET_CURRENT_CACHED_VALUES(3);
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
stack_pointer += -2;
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
gh-134584: Remove redundant refcount from _BINARY_OP_ADD_UNICODE (gh-142825)
2025-12-18 21:33:18 +09:00
case _BINARY_OP_ADD_UNICODE_r13: {
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
CHECK_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef right;
_PyStackRef left;
_PyStackRef res;
gh-134584: Remove redundant refcount from _BINARY_OP_ADD_UNICODE (gh-142825)
2025-12-18 21:33:18 +09:00
_PyStackRef l;
_PyStackRef r;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef _stack_item_0 = _tos_cache0;
right = _stack_item_0;
left = stack_pointer[-1];
PyObject *left_o = PyStackRef_AsPyObjectBorrow(left);
PyObject *right_o = PyStackRef_AsPyObjectBorrow(right);
assert(PyUnicode_CheckExact(left_o));
assert(PyUnicode_CheckExact(right_o));
STAT_INC(BINARY_OP, hit);
PyObject *res_o = PyUnicode_Concat(left_o, right_o);
gh-134584: Remove redundant refcount from _BINARY_OP_ADD_UNICODE (gh-142825)
2025-12-18 21:33:18 +09:00
res = PyStackRef_FromPyObjectSteal(res_o);
if (PyStackRef_IsNull(res)) {
stack_pointer[0] = right;
stack_pointer += 1;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
SET_CURRENT_CACHED_VALUES(0);
GH-128682: Make `PyStackRef_CLOSE` escaping. (GH-129404)
2025-02-03 12:41:32 +00:00
JUMP_TO_ERROR();
}
gh-134584: Remove redundant refcount from _BINARY_OP_ADD_UNICODE (gh-142825)
2025-12-18 21:33:18 +09:00
l = left;
r = right;
_tos_cache2 = r;
_tos_cache1 = l;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_tos_cache0 = res;
gh-134584: Remove redundant refcount from _BINARY_OP_ADD_UNICODE (gh-142825)
2025-12-18 21:33:18 +09:00
SET_CURRENT_CACHED_VALUES(3);
GH-111485: Factor out tier 2 code generation from the rest of the interpreter code generator (GH-112968)
2023-12-12 12:12:17 +00:00
stack_pointer += -1;
GH-141794: Limit size of generated machine code. (GH-142228) * Factor out bodies of the largest uops, to reduce jit code size. * Factor out common assert, also reducing jit code size. * Limit size of jitted code for a single executor to 1MB.
2025-12-03 17:43:35 +00:00
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-111485: Factor out tier 2 code generation from the rest of the interpreter code generator (GH-112968)
2023-12-12 12:12:17 +00:00
break;
}
gh-134584: Remove redundant refcount from _BINARY_OP_ADD_UNICODE (gh-142825)
2025-12-18 21:33:18 +09:00
case _BINARY_OP_ADD_UNICODE_r23: {
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
CHECK_CURRENT_CACHED_VALUES(2);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-118093: Add tier two support for BINARY_OP_INPLACE_ADD_UNICODE (GH-122253)
2024-07-25 14:45:07 -07:00
_PyStackRef right;
_PyStackRef left;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef res;
gh-134584: Remove redundant refcount from _BINARY_OP_ADD_UNICODE (gh-142825)
2025-12-18 21:33:18 +09:00
_PyStackRef l;
_PyStackRef r;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef _stack_item_0 = _tos_cache0;
_PyStackRef _stack_item_1 = _tos_cache1;
right = _stack_item_1;
left = _stack_item_0;
PyObject *left_o = PyStackRef_AsPyObjectBorrow(left);
PyObject *right_o = PyStackRef_AsPyObjectBorrow(right);
assert(PyUnicode_CheckExact(left_o));
assert(PyUnicode_CheckExact(right_o));
STAT_INC(BINARY_OP, hit);
PyObject *res_o = PyUnicode_Concat(left_o, right_o);
res = PyStackRef_FromPyObjectSteal(res_o);
gh-134584: Remove redundant refcount from _BINARY_OP_ADD_UNICODE (gh-142825)
2025-12-18 21:33:18 +09:00
if (PyStackRef_IsNull(res)) {
stack_pointer[0] = left;
stack_pointer[1] = right;
stack_pointer += 2;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
SET_CURRENT_CACHED_VALUES(0);
JUMP_TO_ERROR();
}
gh-134584: Remove redundant refcount from _BINARY_OP_ADD_UNICODE (gh-142825)
2025-12-18 21:33:18 +09:00
l = left;
r = right;
_tos_cache2 = r;
_tos_cache1 = l;
_tos_cache0 = res;
SET_CURRENT_CACHED_VALUES(3);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
gh-143092: Make CALL_LIST_APPEND and BINARY_OP_INPLACE_ADD_UNICODE normal instructions (GH-143124) These super instructions need many special cases in the interpreter, specializer, and JIT. It's best we convert them to normal instructions.
2025-12-25 06:03:00 +08:00
case _BINARY_OP_INPLACE_ADD_UNICODE_r21: {
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
CHECK_CURRENT_CACHED_VALUES(2);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef right;
_PyStackRef left;
gh-143092: Make CALL_LIST_APPEND and BINARY_OP_INPLACE_ADD_UNICODE normal instructions (GH-143124) These super instructions need many special cases in the interpreter, specializer, and JIT. It's best we convert them to normal instructions.
2025-12-25 06:03:00 +08:00
_PyStackRef res;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef _stack_item_0 = _tos_cache0;
_PyStackRef _stack_item_1 = _tos_cache1;
right = _stack_item_1;
left = _stack_item_0;
GH-118093: Add tier two support for BINARY_OP_INPLACE_ADD_UNICODE (GH-122253)
2024-07-25 14:45:07 -07:00
PyObject *left_o = PyStackRef_AsPyObjectBorrow(left);
GH-127809: Fix the JIT's understanding of ** (GH-127844)
2025-01-07 17:25:48 -08:00
assert(PyUnicode_CheckExact(left_o));
gh-130704: Strength reduce `LOAD_FAST{_LOAD_FAST}` (#130708) Optimize `LOAD_FAST` opcodes into faster versions that load borrowed references onto the operand stack when we can prove that the lifetime of the local outlives the lifetime of the temporary that is loaded onto the stack.
2025-04-01 10:18:42 -07:00
assert(PyUnicode_CheckExact(PyStackRef_AsPyObjectBorrow(right)));
GH-118093: Add tier two support for BINARY_OP_INPLACE_ADD_UNICODE (GH-122253)
2024-07-25 14:45:07 -07:00
int next_oparg;
#if TIER_ONE
assert(next_instr->op.code == STORE_FAST);
next_oparg = next_instr->op.arg;
#else
GH-140683: JIT: Improve machine code for loading smaller constants on AArch64. (GH-142511) * Use movz and movk instructions for loading 16 and 32 bit operands and oparg. * Loading of 64 bit operands is unchanged.
2025-12-11 12:33:39 +00:00
next_oparg = (int)CURRENT_OPERAND0_16();
GH-118093: Add tier two support for BINARY_OP_INPLACE_ADD_UNICODE (GH-122253)
2024-07-25 14:45:07 -07:00
#endif
_PyStackRef *target_local = &GETLOCAL(next_oparg);
GH-127705: Use `_PyStackRef`s in the default build. (GH-127875)
2025-03-10 14:06:56 +00:00
assert(PyUnicode_CheckExact(left_o));
gh-127022: Simplify `PyStackRef_FromPyObjectSteal` (#127024) This gets rid of the immortal check in `PyStackRef_FromPyObjectSteal()`. Overall, this improves performance about 2% in the free threading build. This also renames `PyStackRef_Is()` to `PyStackRef_IsExactly()` because the macro requires that the tag bits of the arguments match, which is only true in certain special cases.
2024-11-22 17:55:33 +00:00
if (PyStackRef_AsPyObjectBorrow(*target_local) != left_o) {
GH-118093: Add tier two support for BINARY_OP_INPLACE_ADD_UNICODE (GH-122253)
2024-07-25 14:45:07 -07:00
UOP_STAT_INC(uopcode, miss);
GH-142718: JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. (GH-142762) JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. * Make sure that stack is in correct state at side exits with TOS cached values * Simplify choice of cached items for side exits
2025-12-15 19:01:29 +00:00
_tos_cache1 = right;
_tos_cache0 = left;
SET_CURRENT_CACHED_VALUES(2);
GH-118093: Add tier two support for BINARY_OP_INPLACE_ADD_UNICODE (GH-122253)
2024-07-25 14:45:07 -07:00
JUMP_TO_JUMP_TARGET();
}
STAT_INC(BINARY_OP, hit);
gh-130704: Strength reduce `LOAD_FAST{_LOAD_FAST}` (#130708) Optimize `LOAD_FAST` opcodes into faster versions that load borrowed references onto the operand stack when we can prove that the lifetime of the local outlives the lifetime of the temporary that is loaded onto the stack.
2025-04-01 10:18:42 -07:00
assert(Py_REFCNT(left_o) >= 2 || !PyStackRef_IsHeapSafe(left));
GH-127705: Add debug mode for `_PyStackRef`s inspired by HPy debug mode (GH-128121)
2024-12-20 16:52:20 +00:00
PyObject *temp = PyStackRef_AsPyObjectSteal(*target_local);
gh-143092: Make CALL_LIST_APPEND and BINARY_OP_INPLACE_ADD_UNICODE normal instructions (GH-143124) These super instructions need many special cases in the interpreter, specializer, and JIT. It's best we convert them to normal instructions.
2025-12-25 06:03:00 +08:00
PyObject *right_o = PyStackRef_AsPyObjectBorrow(right);
stack_pointer[0] = left;
stack_pointer[1] = right;
stack_pointer += 2;
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
GH-128682: Account for escapes in `DECREF_INPUTS` (GH-129953) * Handle escapes in DECREF_INPUTS * Mark a few more functions as escaping * Replace DECREF_INPUTS with PyStackRef_CLOSE where possible
2025-02-12 17:44:59 +00:00
_PyFrame_SetStackPointer(frame, stack_pointer);
GH-118093: Add tier two support for BINARY_OP_INPLACE_ADD_UNICODE (GH-122253)
2024-07-25 14:45:07 -07:00
PyUnicode_Append(&temp, right_o);
GH-128682: Account for escapes in `DECREF_INPUTS` (GH-129953) * Handle escapes in DECREF_INPUTS * Mark a few more functions as escaping * Replace DECREF_INPUTS with PyStackRef_CLOSE where possible
2025-02-12 17:44:59 +00:00
stack_pointer = _PyFrame_GetStackPointer(frame);
gh-143092: Make CALL_LIST_APPEND and BINARY_OP_INPLACE_ADD_UNICODE normal instructions (GH-143124) These super instructions need many special cases in the interpreter, specializer, and JIT. It's best we convert them to normal instructions.
2025-12-25 06:03:00 +08:00
PyStackRef_CLOSE_SPECIALIZED(right, _PyUnicode_ExactDealloc);
PyStackRef_CLOSE_SPECIALIZED(left, _PyUnicode_ExactDealloc);
if (temp == NULL) {
stack_pointer += -2;
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(0);
GH-128682: Make `PyStackRef_CLOSE` escaping. (GH-129404)
2025-02-03 12:41:32 +00:00
JUMP_TO_ERROR();
}
gh-143092: Make CALL_LIST_APPEND and BINARY_OP_INPLACE_ADD_UNICODE normal instructions (GH-143124) These super instructions need many special cases in the interpreter, specializer, and JIT. It's best we convert them to normal instructions.
2025-12-25 06:03:00 +08:00
res = PyStackRef_FromPyObjectSteal(temp);
*target_local = PyStackRef_NULL;
_tos_cache0 = res;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_tos_cache1 = PyStackRef_ZERO_BITS;
_tos_cache2 = PyStackRef_ZERO_BITS;
gh-143092: Make CALL_LIST_APPEND and BINARY_OP_INPLACE_ADD_UNICODE normal instructions (GH-143124) These super instructions need many special cases in the interpreter, specializer, and JIT. It's best we convert them to normal instructions.
2025-12-25 06:03:00 +08:00
SET_CURRENT_CACHED_VALUES(1);
stack_pointer += -2;
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-118093: Add tier two support for BINARY_OP_INPLACE_ADD_UNICODE (GH-122253)
2024-07-25 14:45:07 -07:00
break;
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
case _GUARD_BINARY_OP_EXTEND_r22: {
CHECK_CURRENT_CACHED_VALUES(2);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
Revert "gh-133395: add option for extension modules to specialize BINARY_OP/SUBSCR, apply to arrays (#133396)" (#133498)
2025-05-06 11:12:26 +01:00
_PyStackRef right;
_PyStackRef left;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef _stack_item_0 = _tos_cache0;
_PyStackRef _stack_item_1 = _tos_cache1;
right = _stack_item_1;
left = _stack_item_0;
GH-140683: JIT: Improve machine code for loading smaller constants on AArch64. (GH-142511) * Use movz and movk instructions for loading 16 and 32 bit operands and oparg. * Loading of 64 bit operands is unchanged.
2025-12-11 12:33:39 +00:00
PyObject *descr = (PyObject *)CURRENT_OPERAND0_64();
Revert "gh-133395: add option for extension modules to specialize BINARY_OP/SUBSCR, apply to arrays (#133396)" (#133498)
2025-05-06 11:12:26 +01:00
PyObject *left_o = PyStackRef_AsPyObjectBorrow(left);
PyObject *right_o = PyStackRef_AsPyObjectBorrow(right);
_PyBinaryOpSpecializationDescr *d = (_PyBinaryOpSpecializationDescr*)descr;
assert(INLINE_CACHE_ENTRIES_BINARY_OP == 5);
assert(d && d->guard);
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
stack_pointer[0] = left;
stack_pointer[1] = right;
stack_pointer += 2;
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
Revert "gh-133395: add option for extension modules to specialize BINARY_OP/SUBSCR, apply to arrays (#133396)" (#133498)
2025-05-06 11:12:26 +01:00
_PyFrame_SetStackPointer(frame, stack_pointer);
int res = d->guard(left_o, right_o);
stack_pointer = _PyFrame_GetStackPointer(frame);
if (!res) {
UOP_STAT_INC(uopcode, miss);
GH-142718: JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. (GH-142762) JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. * Make sure that stack is in correct state at side exits with TOS cached values * Simplify choice of cached items for side exits
2025-12-15 19:01:29 +00:00
_tos_cache1 = right;
_tos_cache0 = left;
SET_CURRENT_CACHED_VALUES(2);
stack_pointer += -2;
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
Revert "gh-133395: add option for extension modules to specialize BINARY_OP/SUBSCR, apply to arrays (#133396)" (#133498)
2025-05-06 11:12:26 +01:00
JUMP_TO_JUMP_TARGET();
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_tos_cache1 = right;
_tos_cache0 = left;
_tos_cache2 = PyStackRef_ZERO_BITS;
SET_CURRENT_CACHED_VALUES(2);
stack_pointer += -2;
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
Revert "gh-133395: add option for extension modules to specialize BINARY_OP/SUBSCR, apply to arrays (#133396)" (#133498)
2025-05-06 11:12:26 +01:00
break;
}
gh-100239: specialize long tail of binary operations (#128722)
2025-01-16 15:22:13 +00:00
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
case _BINARY_OP_EXTEND_r21: {
CHECK_CURRENT_CACHED_VALUES(2);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-100239: specialize long tail of binary operations (#128722)
2025-01-16 15:22:13 +00:00
_PyStackRef right;
_PyStackRef left;
_PyStackRef res;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef _stack_item_0 = _tos_cache0;
_PyStackRef _stack_item_1 = _tos_cache1;
right = _stack_item_1;
left = _stack_item_0;
GH-140683: JIT: Improve machine code for loading smaller constants on AArch64. (GH-142511) * Use movz and movk instructions for loading 16 and 32 bit operands and oparg. * Loading of 64 bit operands is unchanged.
2025-12-11 12:33:39 +00:00
PyObject *descr = (PyObject *)CURRENT_OPERAND0_64();
gh-100239: specialize long tail of binary operations (#128722)
2025-01-16 15:22:13 +00:00
PyObject *left_o = PyStackRef_AsPyObjectBorrow(left);
PyObject *right_o = PyStackRef_AsPyObjectBorrow(right);
assert(INLINE_CACHE_ENTRIES_BINARY_OP == 5);
_PyBinaryOpSpecializationDescr *d = (_PyBinaryOpSpecializationDescr*)descr;
STAT_INC(BINARY_OP, hit);
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
stack_pointer[0] = left;
stack_pointer[1] = right;
stack_pointer += 2;
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
gh-100239: specialize long tail of binary operations (#128722)
2025-01-16 15:22:13 +00:00
_PyFrame_SetStackPointer(frame, stack_pointer);
PyObject *res_o = d->action(left_o, right_o);
GH-128682: Account for escapes in `DECREF_INPUTS` (GH-129953) * Handle escapes in DECREF_INPUTS * Mark a few more functions as escaping * Replace DECREF_INPUTS with PyStackRef_CLOSE where possible
2025-02-12 17:44:59 +00:00
_PyStackRef tmp = right;
right = PyStackRef_NULL;
stack_pointer[-1] = right;
PyStackRef_CLOSE(tmp);
tmp = left;
left = PyStackRef_NULL;
stack_pointer[-2] = left;
PyStackRef_CLOSE(tmp);
gh-100239: specialize long tail of binary operations (#128722)
2025-01-16 15:22:13 +00:00
stack_pointer = _PyFrame_GetStackPointer(frame);
GH-128682: Account for escapes in `DECREF_INPUTS` (GH-129953) * Handle escapes in DECREF_INPUTS * Mark a few more functions as escaping * Replace DECREF_INPUTS with PyStackRef_CLOSE where possible
2025-02-12 17:44:59 +00:00
stack_pointer += -2;
GH-141794: Limit size of generated machine code. (GH-142228) * Factor out bodies of the largest uops, to reduce jit code size. * Factor out common assert, also reducing jit code size. * Limit size of jitted code for a single executor to 1MB.
2025-12-03 17:43:35 +00:00
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
gh-100239: specialize long tail of binary operations (#128722)
2025-01-16 15:22:13 +00:00
res = PyStackRef_FromPyObjectSteal(res_o);
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_tos_cache0 = res;
_tos_cache1 = PyStackRef_ZERO_BITS;
_tos_cache2 = PyStackRef_ZERO_BITS;
SET_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-100239: specialize long tail of binary operations (#128722)
2025-01-16 15:22:13 +00:00
break;
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
case _BINARY_SLICE_r31: {
CHECK_CURRENT_CACHED_VALUES(3);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
_PyStackRef stop;
_PyStackRef start;
_PyStackRef container;
_PyStackRef res;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef _stack_item_0 = _tos_cache0;
_PyStackRef _stack_item_1 = _tos_cache1;
_PyStackRef _stack_item_2 = _tos_cache2;
stop = _stack_item_2;
start = _stack_item_1;
container = _stack_item_0;
stack_pointer[0] = container;
stack_pointer[1] = start;
stack_pointer[2] = stop;
stack_pointer += 3;
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
GH-119866: Spill the stack around escaping calls. (GH-124392) * Spill the evaluation around escaping calls in the generated interpreter and JIT. * The code generator tracks live, cached values so they can be saved to memory when needed. * Spills the stack pointer around escaping calls, so that the exact stack is visible to the cycle GC.
2024-10-07 14:56:39 +01:00
_PyFrame_SetStackPointer(frame, stack_pointer);
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
PyObject *slice = _PyBuildSlice_ConsumeRefs(PyStackRef_AsPyObjectSteal(start),
PyStackRef_AsPyObjectSteal(stop));
GH-119866: Spill the stack around escaping calls. (GH-124392) * Spill the evaluation around escaping calls in the generated interpreter and JIT. * The code generator tracks live, cached values so they can be saved to memory when needed. * Spills the stack pointer around escaping calls, so that the exact stack is visible to the cycle GC.
2024-10-07 14:56:39 +01:00
stack_pointer = _PyFrame_GetStackPointer(frame);
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
PyObject *res_o;
gh-104584: Baby steps towards generating and executing traces (#105924) Added a new, experimental, tracing optimizer and interpreter (a.k.a. "tier 2"). This currently pessimizes, so don't use yet -- this is infrastructure so we can experiment with optimizing passes. To enable it, pass ``-Xuops`` or set ``PYTHONUOPS=1``. To get debug output, set ``PYTHONUOPSDEBUG=N`` where ``N`` is a debug level (0-4, where 0 is no debug output and 4 is excessively verbose). All of this code is likely to change dramatically before the 3.13 feature freeze. But this is a first step.
2023-06-26 19:02:57 -07:00
if (slice == NULL) {
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
res_o = NULL;
gh-104584: Baby steps towards generating and executing traces (#105924) Added a new, experimental, tracing optimizer and interpreter (a.k.a. "tier 2"). This currently pessimizes, so don't use yet -- this is infrastructure so we can experiment with optimizing passes. To enable it, pass ``-Xuops`` or set ``PYTHONUOPS=1``. To get debug output, set ``PYTHONUOPSDEBUG=N`` where ``N`` is a debug level (0-4, where 0 is no debug output and 4 is excessively verbose). All of this code is likely to change dramatically before the 3.13 feature freeze. But this is a first step.
2023-06-26 19:02:57 -07:00
}
else {
GH-119866: Spill the stack around escaping calls. (GH-124392) * Spill the evaluation around escaping calls in the generated interpreter and JIT. * The code generator tracks live, cached values so they can be saved to memory when needed. * Spills the stack pointer around escaping calls, so that the exact stack is visible to the cycle GC.
2024-10-07 14:56:39 +01:00
stack_pointer += -2;
GH-141794: Limit size of generated machine code. (GH-142228) * Factor out bodies of the largest uops, to reduce jit code size. * Factor out common assert, also reducing jit code size. * Limit size of jitted code for a single executor to 1MB.
2025-12-03 17:43:35 +00:00
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
GH-119866: Spill the stack around escaping calls. (GH-124392) * Spill the evaluation around escaping calls in the generated interpreter and JIT. * The code generator tracks live, cached values so they can be saved to memory when needed. * Spills the stack pointer around escaping calls, so that the exact stack is visible to the cycle GC.
2024-10-07 14:56:39 +01:00
_PyFrame_SetStackPointer(frame, stack_pointer);
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
res_o = PyObject_GetItem(PyStackRef_AsPyObjectBorrow(container), slice);
gh-104584: Baby steps towards generating and executing traces (#105924) Added a new, experimental, tracing optimizer and interpreter (a.k.a. "tier 2"). This currently pessimizes, so don't use yet -- this is infrastructure so we can experiment with optimizing passes. To enable it, pass ``-Xuops`` or set ``PYTHONUOPS=1``. To get debug output, set ``PYTHONUOPSDEBUG=N`` where ``N`` is a debug level (0-4, where 0 is no debug output and 4 is excessively verbose). All of this code is likely to change dramatically before the 3.13 feature freeze. But this is a first step.
2023-06-26 19:02:57 -07:00
Py_DECREF(slice);
GH-128682: Convert explicit loops closing arrays into `DECREF_INPUTS`. (GH-128822) * Mark Py_DECREF and Py_XDECREF as escaping * Remove explicit loops for clearing array inputs
2025-01-14 15:08:56 +00:00
stack_pointer = _PyFrame_GetStackPointer(frame);
GH-119866: Spill the stack around escaping calls. (GH-124392) * Spill the evaluation around escaping calls in the generated interpreter and JIT. * The code generator tracks live, cached values so they can be saved to memory when needed. * Spills the stack pointer around escaping calls, so that the exact stack is visible to the cycle GC.
2024-10-07 14:56:39 +01:00
stack_pointer += 2;
gh-104584: Baby steps towards generating and executing traces (#105924) Added a new, experimental, tracing optimizer and interpreter (a.k.a. "tier 2"). This currently pessimizes, so don't use yet -- this is infrastructure so we can experiment with optimizing passes. To enable it, pass ``-Xuops`` or set ``PYTHONUOPS=1``. To get debug output, set ``PYTHONUOPSDEBUG=N`` where ``N`` is a debug level (0-4, where 0 is no debug output and 4 is excessively verbose). All of this code is likely to change dramatically before the 3.13 feature freeze. But this is a first step.
2023-06-26 19:02:57 -07:00
}
GH-128682: Make `PyStackRef_CLOSE` escaping. (GH-129404)
2025-02-03 12:41:32 +00:00
stack_pointer += -3;
GH-141794: Limit size of generated machine code. (GH-142228) * Factor out bodies of the largest uops, to reduce jit code size. * Factor out common assert, also reducing jit code size. * Limit size of jitted code for a single executor to 1MB.
2025-12-03 17:43:35 +00:00
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
GH-128682: Make `PyStackRef_CLOSE` escaping. (GH-129404)
2025-02-03 12:41:32 +00:00
_PyFrame_SetStackPointer(frame, stack_pointer);
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
PyStackRef_CLOSE(container);
GH-128682: Make `PyStackRef_CLOSE` escaping. (GH-129404)
2025-02-03 12:41:32 +00:00
stack_pointer = _PyFrame_GetStackPointer(frame);
if (res_o == NULL) {
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(0);
GH-128682: Make `PyStackRef_CLOSE` escaping. (GH-129404)
2025-02-03 12:41:32 +00:00
JUMP_TO_ERROR();
}
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
res = PyStackRef_FromPyObjectSteal(res_o);
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_tos_cache0 = res;
_tos_cache1 = PyStackRef_ZERO_BITS;
_tos_cache2 = PyStackRef_ZERO_BITS;
SET_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-104584: Baby steps towards generating and executing traces (#105924) Added a new, experimental, tracing optimizer and interpreter (a.k.a. "tier 2"). This currently pessimizes, so don't use yet -- this is infrastructure so we can experiment with optimizing passes. To enable it, pass ``-Xuops`` or set ``PYTHONUOPS=1``. To get debug output, set ``PYTHONUOPSDEBUG=N`` where ``N`` is a debug level (0-4, where 0 is no debug output and 4 is excessively verbose). All of this code is likely to change dramatically before the 3.13 feature freeze. But this is a first step.
2023-06-26 19:02:57 -07:00
break;
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
case _STORE_SLICE_r30: {
CHECK_CURRENT_CACHED_VALUES(3);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
_PyStackRef stop;
_PyStackRef start;
_PyStackRef container;
_PyStackRef v;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef _stack_item_0 = _tos_cache0;
_PyStackRef _stack_item_1 = _tos_cache1;
_PyStackRef _stack_item_2 = _tos_cache2;
stop = _stack_item_2;
start = _stack_item_1;
container = _stack_item_0;
v = stack_pointer[-1];
stack_pointer[0] = container;
stack_pointer[1] = start;
stack_pointer[2] = stop;
stack_pointer += 3;
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
GH-119866: Spill the stack around escaping calls. (GH-124392) * Spill the evaluation around escaping calls in the generated interpreter and JIT. * The code generator tracks live, cached values so they can be saved to memory when needed. * Spills the stack pointer around escaping calls, so that the exact stack is visible to the cycle GC.
2024-10-07 14:56:39 +01:00
_PyFrame_SetStackPointer(frame, stack_pointer);
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
PyObject *slice = _PyBuildSlice_ConsumeRefs(PyStackRef_AsPyObjectSteal(start),
PyStackRef_AsPyObjectSteal(stop));
GH-119866: Spill the stack around escaping calls. (GH-124392) * Spill the evaluation around escaping calls in the generated interpreter and JIT. * The code generator tracks live, cached values so they can be saved to memory when needed. * Spills the stack pointer around escaping calls, so that the exact stack is visible to the cycle GC.
2024-10-07 14:56:39 +01:00
stack_pointer = _PyFrame_GetStackPointer(frame);
gh-104584: Baby steps towards generating and executing traces (#105924) Added a new, experimental, tracing optimizer and interpreter (a.k.a. "tier 2"). This currently pessimizes, so don't use yet -- this is infrastructure so we can experiment with optimizing passes. To enable it, pass ``-Xuops`` or set ``PYTHONUOPS=1``. To get debug output, set ``PYTHONUOPSDEBUG=N`` where ``N`` is a debug level (0-4, where 0 is no debug output and 4 is excessively verbose). All of this code is likely to change dramatically before the 3.13 feature freeze. But this is a first step.
2023-06-26 19:02:57 -07:00
int err;
if (slice == NULL) {
err = 1;
}
else {
GH-119866: Spill the stack around escaping calls. (GH-124392) * Spill the evaluation around escaping calls in the generated interpreter and JIT. * The code generator tracks live, cached values so they can be saved to memory when needed. * Spills the stack pointer around escaping calls, so that the exact stack is visible to the cycle GC.
2024-10-07 14:56:39 +01:00
stack_pointer += -2;
GH-141794: Limit size of generated machine code. (GH-142228) * Factor out bodies of the largest uops, to reduce jit code size. * Factor out common assert, also reducing jit code size. * Limit size of jitted code for a single executor to 1MB.
2025-12-03 17:43:35 +00:00
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
GH-119866: Spill the stack around escaping calls. (GH-124392) * Spill the evaluation around escaping calls in the generated interpreter and JIT. * The code generator tracks live, cached values so they can be saved to memory when needed. * Spills the stack pointer around escaping calls, so that the exact stack is visible to the cycle GC.
2024-10-07 14:56:39 +01:00
_PyFrame_SetStackPointer(frame, stack_pointer);
gh-117139: Fix a few wrong steals in bytecodes.c (GH-121127) Fix a few wrong steals in bytecodes.c
2024-06-29 02:14:48 +08:00
err = PyObject_SetItem(PyStackRef_AsPyObjectBorrow(container), slice, PyStackRef_AsPyObjectBorrow(v));
gh-104584: Baby steps towards generating and executing traces (#105924) Added a new, experimental, tracing optimizer and interpreter (a.k.a. "tier 2"). This currently pessimizes, so don't use yet -- this is infrastructure so we can experiment with optimizing passes. To enable it, pass ``-Xuops`` or set ``PYTHONUOPS=1``. To get debug output, set ``PYTHONUOPSDEBUG=N`` where ``N`` is a debug level (0-4, where 0 is no debug output and 4 is excessively verbose). All of this code is likely to change dramatically before the 3.13 feature freeze. But this is a first step.
2023-06-26 19:02:57 -07:00
Py_DECREF(slice);
GH-128682: Convert explicit loops closing arrays into `DECREF_INPUTS`. (GH-128822) * Mark Py_DECREF and Py_XDECREF as escaping * Remove explicit loops for clearing array inputs
2025-01-14 15:08:56 +00:00
stack_pointer = _PyFrame_GetStackPointer(frame);
GH-119866: Spill the stack around escaping calls. (GH-124392) * Spill the evaluation around escaping calls in the generated interpreter and JIT. * The code generator tracks live, cached values so they can be saved to memory when needed. * Spills the stack pointer around escaping calls, so that the exact stack is visible to the cycle GC.
2024-10-07 14:56:39 +01:00
stack_pointer += 2;
gh-104584: Baby steps towards generating and executing traces (#105924) Added a new, experimental, tracing optimizer and interpreter (a.k.a. "tier 2"). This currently pessimizes, so don't use yet -- this is infrastructure so we can experiment with optimizing passes. To enable it, pass ``-Xuops`` or set ``PYTHONUOPS=1``. To get debug output, set ``PYTHONUOPSDEBUG=N`` where ``N`` is a debug level (0-4, where 0 is no debug output and 4 is excessively verbose). All of this code is likely to change dramatically before the 3.13 feature freeze. But this is a first step.
2023-06-26 19:02:57 -07:00
}
GH-128682: Account for escapes in `DECREF_INPUTS` (GH-129953) * Handle escapes in DECREF_INPUTS * Mark a few more functions as escaping * Replace DECREF_INPUTS with PyStackRef_CLOSE where possible
2025-02-12 17:44:59 +00:00
_PyFrame_SetStackPointer(frame, stack_pointer);
_PyStackRef tmp = container;
container = PyStackRef_NULL;
GH-131498: Cases generator: manage stacks automatically (GH-132074)
2025-04-04 17:59:36 +01:00
stack_pointer[-3] = container;
GH-128682: Account for escapes in `DECREF_INPUTS` (GH-129953) * Handle escapes in DECREF_INPUTS * Mark a few more functions as escaping * Replace DECREF_INPUTS with PyStackRef_CLOSE where possible
2025-02-12 17:44:59 +00:00
PyStackRef_CLOSE(tmp);
tmp = v;
v = PyStackRef_NULL;
GH-131498: Cases generator: manage stacks automatically (GH-132074)
2025-04-04 17:59:36 +01:00
stack_pointer[-4] = v;
GH-128682: Account for escapes in `DECREF_INPUTS` (GH-129953) * Handle escapes in DECREF_INPUTS * Mark a few more functions as escaping * Replace DECREF_INPUTS with PyStackRef_CLOSE where possible
2025-02-12 17:44:59 +00:00
PyStackRef_CLOSE(tmp);
stack_pointer = _PyFrame_GetStackPointer(frame);
GH-131498: Cases generator: manage stacks automatically (GH-132074)
2025-04-04 17:59:36 +01:00
stack_pointer += -4;
GH-141794: Limit size of generated machine code. (GH-142228) * Factor out bodies of the largest uops, to reduce jit code size. * Factor out common assert, also reducing jit code size. * Limit size of jitted code for a single executor to 1MB.
2025-12-03 17:43:35 +00:00
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
GH-128682: Make `PyStackRef_CLOSE` escaping. (GH-129404)
2025-02-03 12:41:32 +00:00
if (err) {
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(0);
GH-128682: Make `PyStackRef_CLOSE` escaping. (GH-129404)
2025-02-03 12:41:32 +00:00
JUMP_TO_ERROR();
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_tos_cache0 = PyStackRef_ZERO_BITS;
_tos_cache1 = PyStackRef_ZERO_BITS;
_tos_cache2 = PyStackRef_ZERO_BITS;
SET_CURRENT_CACHED_VALUES(0);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-104584: Baby steps towards generating and executing traces (#105924) Added a new, experimental, tracing optimizer and interpreter (a.k.a. "tier 2"). This currently pessimizes, so don't use yet -- this is infrastructure so we can experiment with optimizing passes. To enable it, pass ``-Xuops`` or set ``PYTHONUOPS=1``. To get debug output, set ``PYTHONUOPSDEBUG=N`` where ``N`` is a debug level (0-4, where 0 is no debug output and 4 is excessively verbose). All of this code is likely to change dramatically before the 3.13 feature freeze. But this is a first step.
2023-06-26 19:02:57 -07:00
break;
}
gh-134584: Eliminate redundant refcounting from _BINARY_OP_SUBSCR_LIST_INT (GH-142926)
2025-12-19 02:25:36 +08:00
case _BINARY_OP_SUBSCR_LIST_INT_r23: {
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
CHECK_CURRENT_CACHED_VALUES(2);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
_PyStackRef sub_st;
_PyStackRef list_st;
_PyStackRef res;
gh-134584: Eliminate redundant refcounting from _BINARY_OP_SUBSCR_LIST_INT (GH-142926)
2025-12-19 02:25:36 +08:00
_PyStackRef ls;
_PyStackRef ss;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef _stack_item_0 = _tos_cache0;
_PyStackRef _stack_item_1 = _tos_cache1;
sub_st = _stack_item_1;
list_st = _stack_item_0;
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
PyObject *sub = PyStackRef_AsPyObjectBorrow(sub_st);
PyObject *list = PyStackRef_AsPyObjectBorrow(list_st);
GH-131798: Allow the JIT to remove more int/float/str guards (GH-131800)
2025-04-01 15:10:15 -07:00
assert(PyLong_CheckExact(sub));
GH-131798: Remove JIT guards for dict, frozenset, list, set, and tuple (GH-132289)
2025-04-09 14:32:21 -07:00
assert(PyList_CheckExact(list));
GH-117457: Correct pystats uop "miss" counts (GH-117477)
2024-04-04 18:49:18 -04:00
if (!_PyLong_IsNonNegativeCompact((PyLongObject *)sub)) {
UOP_STAT_INC(uopcode, miss);
GH-142718: JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. (GH-142762) JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. * Make sure that stack is in correct state at side exits with TOS cached values * Simplify choice of cached items for side exits
2025-12-15 19:01:29 +00:00
_tos_cache1 = sub_st;
_tos_cache0 = list_st;
SET_CURRENT_CACHED_VALUES(2);
GH-117457: Correct pystats uop "miss" counts (GH-117477)
2024-04-04 18:49:18 -04:00
JUMP_TO_JUMP_TARGET();
}
gh-104584: Baby steps towards generating and executing traces (#105924) Added a new, experimental, tracing optimizer and interpreter (a.k.a. "tier 2"). This currently pessimizes, so don't use yet -- this is infrastructure so we can experiment with optimizing passes. To enable it, pass ``-Xuops`` or set ``PYTHONUOPS=1``. To get debug output, set ``PYTHONUOPSDEBUG=N`` where ``N`` is a debug level (0-4, where 0 is no debug output and 4 is excessively verbose). All of this code is likely to change dramatically before the 3.13 feature freeze. But this is a first step.
2023-06-26 19:02:57 -07:00
Py_ssize_t index = ((PyLongObject*)sub)->long_value.ob_digit[0];
gh-115999: Add partial free-thread specialization for BINARY_SUBSCR (gh-127227)
2024-12-02 10:38:17 +09:00
#ifdef Py_GIL_DISABLED
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
stack_pointer[0] = list_st;
stack_pointer[1] = sub_st;
stack_pointer += 2;
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
gh-115999: Add partial free-thread specialization for BINARY_SUBSCR (gh-127227)
2024-12-02 10:38:17 +09:00
_PyFrame_SetStackPointer(frame, stack_pointer);
PyObject *res_o = _PyList_GetItemRef((PyListObject*)list, index);
stack_pointer = _PyFrame_GetStackPointer(frame);
if (res_o == NULL) {
UOP_STAT_INC(uopcode, miss);
GH-142718: JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. (GH-142762) JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. * Make sure that stack is in correct state at side exits with TOS cached values * Simplify choice of cached items for side exits
2025-12-15 19:01:29 +00:00
_tos_cache1 = sub_st;
_tos_cache0 = list_st;
SET_CURRENT_CACHED_VALUES(2);
stack_pointer += -2;
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
gh-115999: Add partial free-thread specialization for BINARY_SUBSCR (gh-127227)
2024-12-02 10:38:17 +09:00
JUMP_TO_JUMP_TARGET();
}
gh-100239: replace BINARY_SUBSCR & family by BINARY_OP with oparg NB_SUBSCR (#129700)
2025-02-07 22:39:54 +00:00
STAT_INC(BINARY_OP, hit);
GH-127705: Use `_PyStackRef`s in the default build. (GH-127875)
2025-03-10 14:06:56 +00:00
res = PyStackRef_FromPyObjectSteal(res_o);
gh-115999: Add partial free-thread specialization for BINARY_SUBSCR (gh-127227)
2024-12-02 10:38:17 +09:00
#else
GH-117457: Correct pystats uop "miss" counts (GH-117477)
2024-04-04 18:49:18 -04:00
if (index >= PyList_GET_SIZE(list)) {
UOP_STAT_INC(uopcode, miss);
GH-142718: JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. (GH-142762) JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. * Make sure that stack is in correct state at side exits with TOS cached values * Simplify choice of cached items for side exits
2025-12-15 19:01:29 +00:00
_tos_cache1 = sub_st;
_tos_cache0 = list_st;
SET_CURRENT_CACHED_VALUES(2);
GH-117457: Correct pystats uop "miss" counts (GH-117477)
2024-04-04 18:49:18 -04:00
JUMP_TO_JUMP_TARGET();
}
gh-100239: replace BINARY_SUBSCR & family by BINARY_OP with oparg NB_SUBSCR (#129700)
2025-02-07 22:39:54 +00:00
STAT_INC(BINARY_OP, hit);
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
PyObject *res_o = PyList_GET_ITEM(list, index);
assert(res_o != NULL);
GH-127705: Use `_PyStackRef`s in the default build. (GH-127875)
2025-03-10 14:06:56 +00:00
res = PyStackRef_FromPyObjectNew(res_o);
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
stack_pointer += 2;
gh-115999: Add partial free-thread specialization for BINARY_SUBSCR (gh-127227)
2024-12-02 10:38:17 +09:00
#endif
gh-131281: fix compile error due to `BINARY_SUBSCR` (GH-131283) * fix compile error due to `BINARY_SUBSCR` * replace stat_inc with `BINARY_OP`
2025-03-15 23:38:46 +08:00
STAT_INC(BINARY_OP, hit);
gh-134584: Eliminate redundant refcounting from _BINARY_OP_SUBSCR_LIST_INT (GH-142926)
2025-12-19 02:25:36 +08:00
ls = list_st;
ss = sub_st;
_tos_cache2 = ss;
_tos_cache1 = ls;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_tos_cache0 = res;
gh-134584: Eliminate redundant refcounting from _BINARY_OP_SUBSCR_LIST_INT (GH-142926)
2025-12-19 02:25:36 +08:00
SET_CURRENT_CACHED_VALUES(3);
stack_pointer += -2;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-104584: Baby steps towards generating and executing traces (#105924) Added a new, experimental, tracing optimizer and interpreter (a.k.a. "tier 2"). This currently pessimizes, so don't use yet -- this is infrastructure so we can experiment with optimizing passes. To enable it, pass ``-Xuops`` or set ``PYTHONUOPS=1``. To get debug output, set ``PYTHONUOPSDEBUG=N`` where ``N`` is a debug level (0-4, where 0 is no debug output and 4 is excessively verbose). All of this code is likely to change dramatically before the 3.13 feature freeze. But this is a first step.
2023-06-26 19:02:57 -07:00
break;
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
case _BINARY_OP_SUBSCR_LIST_SLICE_r21: {
CHECK_CURRENT_CACHED_VALUES(2);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-100239: specialize BINARY_OP/SUBSCR for list-slice (#132626)
2025-05-01 11:28:52 +01:00
_PyStackRef sub_st;
_PyStackRef list_st;
_PyStackRef res;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef _stack_item_0 = _tos_cache0;
_PyStackRef _stack_item_1 = _tos_cache1;
sub_st = _stack_item_1;
list_st = _stack_item_0;
gh-100239: specialize BINARY_OP/SUBSCR for list-slice (#132626)
2025-05-01 11:28:52 +01:00
PyObject *sub = PyStackRef_AsPyObjectBorrow(sub_st);
PyObject *list = PyStackRef_AsPyObjectBorrow(list_st);
assert(PySlice_Check(sub));
assert(PyList_CheckExact(list));
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
stack_pointer[0] = list_st;
stack_pointer[1] = sub_st;
stack_pointer += 2;
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
gh-100239: specialize BINARY_OP/SUBSCR for list-slice (#132626)
2025-05-01 11:28:52 +01:00
_PyFrame_SetStackPointer(frame, stack_pointer);
PyObject *res_o = _PyList_SliceSubscript(list, sub);
stack_pointer = _PyFrame_GetStackPointer(frame);
STAT_INC(BINARY_OP, hit);
_PyFrame_SetStackPointer(frame, stack_pointer);
gh-133258: Fix crash in test_index (GH-133262)
2025-05-01 18:15:53 +01:00
_PyStackRef tmp = sub_st;
gh-100239: specialize BINARY_OP/SUBSCR for list-slice (#132626)
2025-05-01 11:28:52 +01:00
sub_st = PyStackRef_NULL;
stack_pointer[-1] = sub_st;
PyStackRef_CLOSE(tmp);
gh-133258: Fix crash in test_index (GH-133262)
2025-05-01 18:15:53 +01:00
tmp = list_st;
list_st = PyStackRef_NULL;
stack_pointer[-2] = list_st;
PyStackRef_CLOSE(tmp);
gh-100239: specialize BINARY_OP/SUBSCR for list-slice (#132626)
2025-05-01 11:28:52 +01:00
stack_pointer = _PyFrame_GetStackPointer(frame);
gh-133258: Fix crash in test_index (GH-133262)
2025-05-01 18:15:53 +01:00
stack_pointer += -2;
GH-141794: Limit size of generated machine code. (GH-142228) * Factor out bodies of the largest uops, to reduce jit code size. * Factor out common assert, also reducing jit code size. * Limit size of jitted code for a single executor to 1MB.
2025-12-03 17:43:35 +00:00
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
gh-133258: Fix crash in test_index (GH-133262)
2025-05-01 18:15:53 +01:00
if (res_o == NULL) {
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(0);
gh-133258: Fix crash in test_index (GH-133262)
2025-05-01 18:15:53 +01:00
JUMP_TO_ERROR();
}
res = PyStackRef_FromPyObjectSteal(res_o);
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_tos_cache0 = res;
_tos_cache1 = PyStackRef_ZERO_BITS;
_tos_cache2 = PyStackRef_ZERO_BITS;
SET_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-100239: specialize BINARY_OP/SUBSCR for list-slice (#132626)
2025-05-01 11:28:52 +01:00
break;
}
GH-134584: Remove redundant refcount for `BINARY_OP_SUBSCR_STR_INT` (#142844)
2025-12-18 13:29:54 -08:00
case _BINARY_OP_SUBSCR_STR_INT_r23: {
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
CHECK_CURRENT_CACHED_VALUES(2);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
_PyStackRef sub_st;
_PyStackRef str_st;
_PyStackRef res;
GH-134584: Remove redundant refcount for `BINARY_OP_SUBSCR_STR_INT` (#142844)
2025-12-18 13:29:54 -08:00
_PyStackRef s;
_PyStackRef i;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef _stack_item_0 = _tos_cache0;
_PyStackRef _stack_item_1 = _tos_cache1;
sub_st = _stack_item_1;
str_st = _stack_item_0;
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
PyObject *sub = PyStackRef_AsPyObjectBorrow(sub_st);
PyObject *str = PyStackRef_AsPyObjectBorrow(str_st);
GH-131798: Allow the JIT to remove more int/float/str guards (GH-131800)
2025-04-01 15:10:15 -07:00
assert(PyLong_CheckExact(sub));
assert(PyUnicode_CheckExact(str));
GH-117457: Correct pystats uop "miss" counts (GH-117477)
2024-04-04 18:49:18 -04:00
if (!_PyLong_IsNonNegativeCompact((PyLongObject *)sub)) {
UOP_STAT_INC(uopcode, miss);
GH-142718: JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. (GH-142762) JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. * Make sure that stack is in correct state at side exits with TOS cached values * Simplify choice of cached items for side exits
2025-12-15 19:01:29 +00:00
_tos_cache1 = sub_st;
_tos_cache0 = str_st;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(2);
GH-117457: Correct pystats uop "miss" counts (GH-117477)
2024-04-04 18:49:18 -04:00
JUMP_TO_JUMP_TARGET();
}
GH-107596: Specialize str[int] (GH-107597)
2023-08-08 13:42:43 -07:00
Py_ssize_t index = ((PyLongObject*)sub)->long_value.ob_digit[0];
GH-117457: Correct pystats uop "miss" counts (GH-117477)
2024-04-04 18:49:18 -04:00
if (PyUnicode_GET_LENGTH(str) <= index) {
UOP_STAT_INC(uopcode, miss);
GH-142718: JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. (GH-142762) JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. * Make sure that stack is in correct state at side exits with TOS cached values * Simplify choice of cached items for side exits
2025-12-15 19:01:29 +00:00
_tos_cache1 = sub_st;
_tos_cache0 = str_st;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(2);
GH-117457: Correct pystats uop "miss" counts (GH-117477)
2024-04-04 18:49:18 -04:00
JUMP_TO_JUMP_TARGET();
}
GH-139757: JIT: Remove redundant branches to jumps in the assembly optimizer (GH-140800) JIT: Remove redundant branches to jump in the assembly optimizer * Refactor JIT assembly optimizer making instructions instances not just strings * Remove redundant jumps and branches where legal to do so * Modifies _BINARY_OP_SUBSCR_STR_INT to avoid excessive inlining depth
2025-12-08 17:57:11 +00:00
if (!PyUnicode_IS_COMPACT_ASCII(str)) {
GH-117457: Correct pystats uop "miss" counts (GH-117477)
2024-04-04 18:49:18 -04:00
UOP_STAT_INC(uopcode, miss);
GH-142718: JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. (GH-142762) JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. * Make sure that stack is in correct state at side exits with TOS cached values * Simplify choice of cached items for side exits
2025-12-15 19:01:29 +00:00
_tos_cache1 = sub_st;
_tos_cache0 = str_st;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(2);
GH-117457: Correct pystats uop "miss" counts (GH-117477)
2024-04-04 18:49:18 -04:00
JUMP_TO_JUMP_TARGET();
}
GH-139757: JIT: Remove redundant branches to jumps in the assembly optimizer (GH-140800) JIT: Remove redundant branches to jump in the assembly optimizer * Refactor JIT assembly optimizer making instructions instances not just strings * Remove redundant jumps and branches where legal to do so * Modifies _BINARY_OP_SUBSCR_STR_INT to avoid excessive inlining depth
2025-12-08 17:57:11 +00:00
uint8_t c = PyUnicode_1BYTE_DATA(str)[index];
assert(c < 128);
gh-100239: replace BINARY_SUBSCR & family by BINARY_OP with oparg NB_SUBSCR (#129700)
2025-02-07 22:39:54 +00:00
STAT_INC(BINARY_OP, hit);
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
PyObject *res_o = (PyObject*)&_Py_SINGLETON(strings).ascii[c];
GH-134584: Remove redundant refcount for `BINARY_OP_SUBSCR_STR_INT` (#142844)
2025-12-18 13:29:54 -08:00
s = str_st;
i = sub_st;
GH-134282: Always borrow references LOAD_CONST (GH-134284)
2025-05-20 11:24:11 -04:00
res = PyStackRef_FromPyObjectBorrow(res_o);
GH-134584: Remove redundant refcount for `BINARY_OP_SUBSCR_STR_INT` (#142844)
2025-12-18 13:29:54 -08:00
_tos_cache2 = i;
_tos_cache1 = s;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_tos_cache0 = res;
GH-134584: Remove redundant refcount for `BINARY_OP_SUBSCR_STR_INT` (#142844)
2025-12-18 13:29:54 -08:00
SET_CURRENT_CACHED_VALUES(3);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-107596: Specialize str[int] (GH-107597)
2023-08-08 13:42:43 -07:00
break;
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
case _GUARD_NOS_TUPLE_r02: {
CHECK_CURRENT_CACHED_VALUES(0);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-131798: Remove JIT guards for dict, frozenset, list, set, and tuple (GH-132289)
2025-04-09 14:32:21 -07:00
_PyStackRef nos;
nos = stack_pointer[-2];
PyObject *o = PyStackRef_AsPyObjectBorrow(nos);
if (!PyTuple_CheckExact(o)) {
UOP_STAT_INC(uopcode, miss);
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(0);
JUMP_TO_JUMP_TARGET();
}
_tos_cache1 = stack_pointer[-1];
_tos_cache0 = nos;
SET_CURRENT_CACHED_VALUES(2);
stack_pointer += -2;
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _GUARD_NOS_TUPLE_r12: {
CHECK_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef nos;
_PyStackRef _stack_item_0 = _tos_cache0;
nos = stack_pointer[-1];
PyObject *o = PyStackRef_AsPyObjectBorrow(nos);
if (!PyTuple_CheckExact(o)) {
UOP_STAT_INC(uopcode, miss);
GH-142718: JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. (GH-142762) JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. * Make sure that stack is in correct state at side exits with TOS cached values * Simplify choice of cached items for side exits
2025-12-15 19:01:29 +00:00
_tos_cache0 = _stack_item_0;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(1);
JUMP_TO_JUMP_TARGET();
}
_tos_cache1 = _stack_item_0;
_tos_cache0 = nos;
SET_CURRENT_CACHED_VALUES(2);
stack_pointer += -1;
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _GUARD_NOS_TUPLE_r22: {
CHECK_CURRENT_CACHED_VALUES(2);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef nos;
_PyStackRef _stack_item_0 = _tos_cache0;
_PyStackRef _stack_item_1 = _tos_cache1;
nos = _stack_item_0;
PyObject *o = PyStackRef_AsPyObjectBorrow(nos);
if (!PyTuple_CheckExact(o)) {
UOP_STAT_INC(uopcode, miss);
GH-142718: JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. (GH-142762) JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. * Make sure that stack is in correct state at side exits with TOS cached values * Simplify choice of cached items for side exits
2025-12-15 19:01:29 +00:00
_tos_cache1 = _stack_item_1;
_tos_cache0 = nos;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(2);
JUMP_TO_JUMP_TARGET();
}
_tos_cache1 = _stack_item_1;
_tos_cache0 = nos;
SET_CURRENT_CACHED_VALUES(2);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _GUARD_NOS_TUPLE_r33: {
CHECK_CURRENT_CACHED_VALUES(3);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef nos;
_PyStackRef _stack_item_0 = _tos_cache0;
_PyStackRef _stack_item_1 = _tos_cache1;
_PyStackRef _stack_item_2 = _tos_cache2;
nos = _stack_item_1;
PyObject *o = PyStackRef_AsPyObjectBorrow(nos);
if (!PyTuple_CheckExact(o)) {
UOP_STAT_INC(uopcode, miss);
GH-142718: JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. (GH-142762) JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. * Make sure that stack is in correct state at side exits with TOS cached values * Simplify choice of cached items for side exits
2025-12-15 19:01:29 +00:00
_tos_cache2 = _stack_item_2;
_tos_cache1 = nos;
_tos_cache0 = _stack_item_0;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(3);
GH-131798: Remove JIT guards for dict, frozenset, list, set, and tuple (GH-132289)
2025-04-09 14:32:21 -07:00
JUMP_TO_JUMP_TARGET();
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_tos_cache2 = _stack_item_2;
_tos_cache1 = nos;
_tos_cache0 = _stack_item_0;
SET_CURRENT_CACHED_VALUES(3);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-131798: Remove JIT guards for dict, frozenset, list, set, and tuple (GH-132289)
2025-04-09 14:32:21 -07:00
break;
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
case _GUARD_TOS_TUPLE_r01: {
CHECK_CURRENT_CACHED_VALUES(0);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-131798: Remove JIT guards for dict, frozenset, list, set, and tuple (GH-132289)
2025-04-09 14:32:21 -07:00
_PyStackRef tos;
tos = stack_pointer[-1];
PyObject *o = PyStackRef_AsPyObjectBorrow(tos);
if (!PyTuple_CheckExact(o)) {
UOP_STAT_INC(uopcode, miss);
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(0);
JUMP_TO_JUMP_TARGET();
}
_tos_cache0 = tos;
SET_CURRENT_CACHED_VALUES(1);
stack_pointer += -1;
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _GUARD_TOS_TUPLE_r11: {
CHECK_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef tos;
_PyStackRef _stack_item_0 = _tos_cache0;
tos = _stack_item_0;
PyObject *o = PyStackRef_AsPyObjectBorrow(tos);
if (!PyTuple_CheckExact(o)) {
UOP_STAT_INC(uopcode, miss);
GH-142718: JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. (GH-142762) JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. * Make sure that stack is in correct state at side exits with TOS cached values * Simplify choice of cached items for side exits
2025-12-15 19:01:29 +00:00
_tos_cache0 = tos;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(1);
JUMP_TO_JUMP_TARGET();
}
_tos_cache0 = tos;
SET_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _GUARD_TOS_TUPLE_r22: {
CHECK_CURRENT_CACHED_VALUES(2);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef tos;
_PyStackRef _stack_item_0 = _tos_cache0;
_PyStackRef _stack_item_1 = _tos_cache1;
tos = _stack_item_1;
PyObject *o = PyStackRef_AsPyObjectBorrow(tos);
if (!PyTuple_CheckExact(o)) {
UOP_STAT_INC(uopcode, miss);
GH-142718: JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. (GH-142762) JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. * Make sure that stack is in correct state at side exits with TOS cached values * Simplify choice of cached items for side exits
2025-12-15 19:01:29 +00:00
_tos_cache1 = tos;
_tos_cache0 = _stack_item_0;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(2);
JUMP_TO_JUMP_TARGET();
}
_tos_cache1 = tos;
_tos_cache0 = _stack_item_0;
SET_CURRENT_CACHED_VALUES(2);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _GUARD_TOS_TUPLE_r33: {
CHECK_CURRENT_CACHED_VALUES(3);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef tos;
_PyStackRef _stack_item_0 = _tos_cache0;
_PyStackRef _stack_item_1 = _tos_cache1;
_PyStackRef _stack_item_2 = _tos_cache2;
tos = _stack_item_2;
PyObject *o = PyStackRef_AsPyObjectBorrow(tos);
if (!PyTuple_CheckExact(o)) {
UOP_STAT_INC(uopcode, miss);
GH-142718: JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. (GH-142762) JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. * Make sure that stack is in correct state at side exits with TOS cached values * Simplify choice of cached items for side exits
2025-12-15 19:01:29 +00:00
_tos_cache2 = tos;
_tos_cache1 = _stack_item_1;
_tos_cache0 = _stack_item_0;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(3);
GH-131798: Remove JIT guards for dict, frozenset, list, set, and tuple (GH-132289)
2025-04-09 14:32:21 -07:00
JUMP_TO_JUMP_TARGET();
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_tos_cache2 = tos;
_tos_cache1 = _stack_item_1;
_tos_cache0 = _stack_item_0;
SET_CURRENT_CACHED_VALUES(3);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-131798: Remove JIT guards for dict, frozenset, list, set, and tuple (GH-132289)
2025-04-09 14:32:21 -07:00
break;
}
gh-131798: Remove bounds check when indexing into tuples with a constant index (#137607) * Remove bounds check when indexing into tuples with a constant index * Add news entry * fixup after rebase
2025-12-28 22:06:06 +01:00
case _GUARD_BINARY_OP_SUBSCR_TUPLE_INT_BOUNDS_r02: {
CHECK_CURRENT_CACHED_VALUES(0);
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
_PyStackRef sub_st;
_PyStackRef tuple_st;
sub_st = stack_pointer[-1];
tuple_st = stack_pointer[-2];
PyObject *sub = PyStackRef_AsPyObjectBorrow(sub_st);
PyObject *tuple = PyStackRef_AsPyObjectBorrow(tuple_st);
assert(PyLong_CheckExact(sub));
assert(PyTuple_CheckExact(tuple));
if (!_PyLong_IsNonNegativeCompact((PyLongObject *)sub)) {
UOP_STAT_INC(uopcode, miss);
SET_CURRENT_CACHED_VALUES(0);
JUMP_TO_JUMP_TARGET();
}
Py_ssize_t index = ((PyLongObject*)sub)->long_value.ob_digit[0];
if (index >= PyTuple_GET_SIZE(tuple)) {
UOP_STAT_INC(uopcode, miss);
SET_CURRENT_CACHED_VALUES(0);
JUMP_TO_JUMP_TARGET();
}
_tos_cache1 = sub_st;
_tos_cache0 = tuple_st;
SET_CURRENT_CACHED_VALUES(2);
stack_pointer += -2;
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
break;
}
case _GUARD_BINARY_OP_SUBSCR_TUPLE_INT_BOUNDS_r12: {
CHECK_CURRENT_CACHED_VALUES(1);
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
_PyStackRef sub_st;
_PyStackRef tuple_st;
_PyStackRef _stack_item_0 = _tos_cache0;
sub_st = _stack_item_0;
tuple_st = stack_pointer[-1];
PyObject *sub = PyStackRef_AsPyObjectBorrow(sub_st);
PyObject *tuple = PyStackRef_AsPyObjectBorrow(tuple_st);
assert(PyLong_CheckExact(sub));
assert(PyTuple_CheckExact(tuple));
if (!_PyLong_IsNonNegativeCompact((PyLongObject *)sub)) {
UOP_STAT_INC(uopcode, miss);
_tos_cache0 = sub_st;
SET_CURRENT_CACHED_VALUES(1);
JUMP_TO_JUMP_TARGET();
}
Py_ssize_t index = ((PyLongObject*)sub)->long_value.ob_digit[0];
if (index >= PyTuple_GET_SIZE(tuple)) {
UOP_STAT_INC(uopcode, miss);
_tos_cache0 = sub_st;
SET_CURRENT_CACHED_VALUES(1);
JUMP_TO_JUMP_TARGET();
}
_tos_cache1 = sub_st;
_tos_cache0 = tuple_st;
SET_CURRENT_CACHED_VALUES(2);
stack_pointer += -1;
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
break;
}
case _GUARD_BINARY_OP_SUBSCR_TUPLE_INT_BOUNDS_r22: {
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
CHECK_CURRENT_CACHED_VALUES(2);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
_PyStackRef sub_st;
_PyStackRef tuple_st;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef _stack_item_0 = _tos_cache0;
_PyStackRef _stack_item_1 = _tos_cache1;
sub_st = _stack_item_1;
tuple_st = _stack_item_0;
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
PyObject *sub = PyStackRef_AsPyObjectBorrow(sub_st);
PyObject *tuple = PyStackRef_AsPyObjectBorrow(tuple_st);
GH-131798: Allow the JIT to remove more int/float/str guards (GH-131800)
2025-04-01 15:10:15 -07:00
assert(PyLong_CheckExact(sub));
GH-131798: Remove JIT guards for dict, frozenset, list, set, and tuple (GH-132289)
2025-04-09 14:32:21 -07:00
assert(PyTuple_CheckExact(tuple));
GH-117457: Correct pystats uop "miss" counts (GH-117477)
2024-04-04 18:49:18 -04:00
if (!_PyLong_IsNonNegativeCompact((PyLongObject *)sub)) {
UOP_STAT_INC(uopcode, miss);
GH-142718: JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. (GH-142762) JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. * Make sure that stack is in correct state at side exits with TOS cached values * Simplify choice of cached items for side exits
2025-12-15 19:01:29 +00:00
_tos_cache1 = sub_st;
_tos_cache0 = tuple_st;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(2);
GH-117457: Correct pystats uop "miss" counts (GH-117477)
2024-04-04 18:49:18 -04:00
JUMP_TO_JUMP_TARGET();
}
gh-104584: Baby steps towards generating and executing traces (#105924) Added a new, experimental, tracing optimizer and interpreter (a.k.a. "tier 2"). This currently pessimizes, so don't use yet -- this is infrastructure so we can experiment with optimizing passes. To enable it, pass ``-Xuops`` or set ``PYTHONUOPS=1``. To get debug output, set ``PYTHONUOPSDEBUG=N`` where ``N`` is a debug level (0-4, where 0 is no debug output and 4 is excessively verbose). All of this code is likely to change dramatically before the 3.13 feature freeze. But this is a first step.
2023-06-26 19:02:57 -07:00
Py_ssize_t index = ((PyLongObject*)sub)->long_value.ob_digit[0];
GH-117457: Correct pystats uop "miss" counts (GH-117477)
2024-04-04 18:49:18 -04:00
if (index >= PyTuple_GET_SIZE(tuple)) {
UOP_STAT_INC(uopcode, miss);
GH-142718: JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. (GH-142762) JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. * Make sure that stack is in correct state at side exits with TOS cached values * Simplify choice of cached items for side exits
2025-12-15 19:01:29 +00:00
_tos_cache1 = sub_st;
_tos_cache0 = tuple_st;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(2);
GH-117457: Correct pystats uop "miss" counts (GH-117477)
2024-04-04 18:49:18 -04:00
JUMP_TO_JUMP_TARGET();
}
gh-131798: Remove bounds check when indexing into tuples with a constant index (#137607) * Remove bounds check when indexing into tuples with a constant index * Add news entry * fixup after rebase
2025-12-28 22:06:06 +01:00
_tos_cache1 = sub_st;
_tos_cache0 = tuple_st;
SET_CURRENT_CACHED_VALUES(2);
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
break;
}
case _GUARD_BINARY_OP_SUBSCR_TUPLE_INT_BOUNDS_r33: {
CHECK_CURRENT_CACHED_VALUES(3);
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
_PyStackRef sub_st;
_PyStackRef tuple_st;
_PyStackRef _stack_item_0 = _tos_cache0;
_PyStackRef _stack_item_1 = _tos_cache1;
_PyStackRef _stack_item_2 = _tos_cache2;
sub_st = _stack_item_2;
tuple_st = _stack_item_1;
PyObject *sub = PyStackRef_AsPyObjectBorrow(sub_st);
PyObject *tuple = PyStackRef_AsPyObjectBorrow(tuple_st);
assert(PyLong_CheckExact(sub));
assert(PyTuple_CheckExact(tuple));
if (!_PyLong_IsNonNegativeCompact((PyLongObject *)sub)) {
UOP_STAT_INC(uopcode, miss);
_tos_cache2 = sub_st;
_tos_cache1 = tuple_st;
_tos_cache0 = _stack_item_0;
SET_CURRENT_CACHED_VALUES(3);
JUMP_TO_JUMP_TARGET();
}
Py_ssize_t index = ((PyLongObject*)sub)->long_value.ob_digit[0];
if (index >= PyTuple_GET_SIZE(tuple)) {
UOP_STAT_INC(uopcode, miss);
_tos_cache2 = sub_st;
_tos_cache1 = tuple_st;
_tos_cache0 = _stack_item_0;
SET_CURRENT_CACHED_VALUES(3);
JUMP_TO_JUMP_TARGET();
}
_tos_cache2 = sub_st;
_tos_cache1 = tuple_st;
_tos_cache0 = _stack_item_0;
SET_CURRENT_CACHED_VALUES(3);
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
break;
}
case _BINARY_OP_SUBSCR_TUPLE_INT_r03: {
CHECK_CURRENT_CACHED_VALUES(0);
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
_PyStackRef sub_st;
_PyStackRef tuple_st;
_PyStackRef res;
_PyStackRef ts;
_PyStackRef ss;
sub_st = stack_pointer[-1];
tuple_st = stack_pointer[-2];
PyObject *sub = PyStackRef_AsPyObjectBorrow(sub_st);
PyObject *tuple = PyStackRef_AsPyObjectBorrow(tuple_st);
assert(PyLong_CheckExact(sub));
assert(PyTuple_CheckExact(tuple));
gh-100239: replace BINARY_SUBSCR & family by BINARY_OP with oparg NB_SUBSCR (#129700)
2025-02-07 22:39:54 +00:00
STAT_INC(BINARY_OP, hit);
gh-131798: Remove bounds check when indexing into tuples with a constant index (#137607) * Remove bounds check when indexing into tuples with a constant index * Add news entry * fixup after rebase
2025-12-28 22:06:06 +01:00
Py_ssize_t index = ((PyLongObject*)sub)->long_value.ob_digit[0];
PyObject *res_o = PyTuple_GET_ITEM(tuple, index);
assert(res_o != NULL);
res = PyStackRef_FromPyObjectNew(res_o);
ts = tuple_st;
ss = sub_st;
_tos_cache2 = ss;
_tos_cache1 = ts;
_tos_cache0 = res;
SET_CURRENT_CACHED_VALUES(3);
stack_pointer += -2;
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
break;
}
case _BINARY_OP_SUBSCR_TUPLE_INT_r13: {
CHECK_CURRENT_CACHED_VALUES(1);
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
_PyStackRef sub_st;
_PyStackRef tuple_st;
_PyStackRef res;
_PyStackRef ts;
_PyStackRef ss;
_PyStackRef _stack_item_0 = _tos_cache0;
sub_st = _stack_item_0;
tuple_st = stack_pointer[-1];
PyObject *sub = PyStackRef_AsPyObjectBorrow(sub_st);
PyObject *tuple = PyStackRef_AsPyObjectBorrow(tuple_st);
assert(PyLong_CheckExact(sub));
assert(PyTuple_CheckExact(tuple));
STAT_INC(BINARY_OP, hit);
Py_ssize_t index = ((PyLongObject*)sub)->long_value.ob_digit[0];
PyObject *res_o = PyTuple_GET_ITEM(tuple, index);
assert(res_o != NULL);
res = PyStackRef_FromPyObjectNew(res_o);
ts = tuple_st;
ss = sub_st;
_tos_cache2 = ss;
_tos_cache1 = ts;
_tos_cache0 = res;
SET_CURRENT_CACHED_VALUES(3);
stack_pointer += -1;
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
break;
}
case _BINARY_OP_SUBSCR_TUPLE_INT_r23: {
CHECK_CURRENT_CACHED_VALUES(2);
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
_PyStackRef sub_st;
_PyStackRef tuple_st;
_PyStackRef res;
_PyStackRef ts;
_PyStackRef ss;
_PyStackRef _stack_item_0 = _tos_cache0;
_PyStackRef _stack_item_1 = _tos_cache1;
sub_st = _stack_item_1;
tuple_st = _stack_item_0;
PyObject *sub = PyStackRef_AsPyObjectBorrow(sub_st);
PyObject *tuple = PyStackRef_AsPyObjectBorrow(tuple_st);
assert(PyLong_CheckExact(sub));
assert(PyTuple_CheckExact(tuple));
STAT_INC(BINARY_OP, hit);
Py_ssize_t index = ((PyLongObject*)sub)->long_value.ob_digit[0];
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
PyObject *res_o = PyTuple_GET_ITEM(tuple, index);
assert(res_o != NULL);
GH-127705: Use `_PyStackRef`s in the default build. (GH-127875)
2025-03-10 14:06:56 +00:00
res = PyStackRef_FromPyObjectNew(res_o);
gh-134584: Eliminate redundant refcounting from `_BINARY_OP_SUBSCR_TUPLE_INT` (GH-143094)
2025-12-24 05:47:12 +08:00
ts = tuple_st;
ss = sub_st;
_tos_cache2 = ss;
_tos_cache1 = ts;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_tos_cache0 = res;
gh-134584: Eliminate redundant refcounting from `_BINARY_OP_SUBSCR_TUPLE_INT` (GH-143094)
2025-12-24 05:47:12 +08:00
SET_CURRENT_CACHED_VALUES(3);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-104584: Baby steps towards generating and executing traces (#105924) Added a new, experimental, tracing optimizer and interpreter (a.k.a. "tier 2"). This currently pessimizes, so don't use yet -- this is infrastructure so we can experiment with optimizing passes. To enable it, pass ``-Xuops`` or set ``PYTHONUOPS=1``. To get debug output, set ``PYTHONUOPSDEBUG=N`` where ``N`` is a debug level (0-4, where 0 is no debug output and 4 is excessively verbose). All of this code is likely to change dramatically before the 3.13 feature freeze. But this is a first step.
2023-06-26 19:02:57 -07:00
break;
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
case _GUARD_NOS_DICT_r02: {
CHECK_CURRENT_CACHED_VALUES(0);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-131798: Remove JIT guards for dict, frozenset, list, set, and tuple (GH-132289)
2025-04-09 14:32:21 -07:00
_PyStackRef nos;
nos = stack_pointer[-2];
PyObject *o = PyStackRef_AsPyObjectBorrow(nos);
if (!PyDict_CheckExact(o)) {
UOP_STAT_INC(uopcode, miss);
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(0);
JUMP_TO_JUMP_TARGET();
}
_tos_cache1 = stack_pointer[-1];
_tos_cache0 = nos;
SET_CURRENT_CACHED_VALUES(2);
stack_pointer += -2;
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _GUARD_NOS_DICT_r12: {
CHECK_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef nos;
_PyStackRef _stack_item_0 = _tos_cache0;
nos = stack_pointer[-1];
PyObject *o = PyStackRef_AsPyObjectBorrow(nos);
if (!PyDict_CheckExact(o)) {
UOP_STAT_INC(uopcode, miss);
GH-142718: JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. (GH-142762) JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. * Make sure that stack is in correct state at side exits with TOS cached values * Simplify choice of cached items for side exits
2025-12-15 19:01:29 +00:00
_tos_cache0 = _stack_item_0;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(1);
JUMP_TO_JUMP_TARGET();
}
_tos_cache1 = _stack_item_0;
_tos_cache0 = nos;
SET_CURRENT_CACHED_VALUES(2);
stack_pointer += -1;
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _GUARD_NOS_DICT_r22: {
CHECK_CURRENT_CACHED_VALUES(2);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef nos;
_PyStackRef _stack_item_0 = _tos_cache0;
_PyStackRef _stack_item_1 = _tos_cache1;
nos = _stack_item_0;
PyObject *o = PyStackRef_AsPyObjectBorrow(nos);
if (!PyDict_CheckExact(o)) {
UOP_STAT_INC(uopcode, miss);
GH-142718: JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. (GH-142762) JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. * Make sure that stack is in correct state at side exits with TOS cached values * Simplify choice of cached items for side exits
2025-12-15 19:01:29 +00:00
_tos_cache1 = _stack_item_1;
_tos_cache0 = nos;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(2);
JUMP_TO_JUMP_TARGET();
}
_tos_cache1 = _stack_item_1;
_tos_cache0 = nos;
SET_CURRENT_CACHED_VALUES(2);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _GUARD_NOS_DICT_r33: {
CHECK_CURRENT_CACHED_VALUES(3);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef nos;
_PyStackRef _stack_item_0 = _tos_cache0;
_PyStackRef _stack_item_1 = _tos_cache1;
_PyStackRef _stack_item_2 = _tos_cache2;
nos = _stack_item_1;
PyObject *o = PyStackRef_AsPyObjectBorrow(nos);
if (!PyDict_CheckExact(o)) {
UOP_STAT_INC(uopcode, miss);
GH-142718: JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. (GH-142762) JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. * Make sure that stack is in correct state at side exits with TOS cached values * Simplify choice of cached items for side exits
2025-12-15 19:01:29 +00:00
_tos_cache2 = _stack_item_2;
_tos_cache1 = nos;
_tos_cache0 = _stack_item_0;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(3);
GH-131798: Remove JIT guards for dict, frozenset, list, set, and tuple (GH-132289)
2025-04-09 14:32:21 -07:00
JUMP_TO_JUMP_TARGET();
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_tos_cache2 = _stack_item_2;
_tos_cache1 = nos;
_tos_cache0 = _stack_item_0;
SET_CURRENT_CACHED_VALUES(3);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-131798: Remove JIT guards for dict, frozenset, list, set, and tuple (GH-132289)
2025-04-09 14:32:21 -07:00
break;
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
case _GUARD_TOS_DICT_r01: {
CHECK_CURRENT_CACHED_VALUES(0);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-131798: Remove JIT guards for dict, frozenset, list, set, and tuple (GH-132289)
2025-04-09 14:32:21 -07:00
_PyStackRef tos;
tos = stack_pointer[-1];
PyObject *o = PyStackRef_AsPyObjectBorrow(tos);
if (!PyDict_CheckExact(o)) {
UOP_STAT_INC(uopcode, miss);
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(0);
JUMP_TO_JUMP_TARGET();
}
_tos_cache0 = tos;
SET_CURRENT_CACHED_VALUES(1);
stack_pointer += -1;
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _GUARD_TOS_DICT_r11: {
CHECK_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef tos;
_PyStackRef _stack_item_0 = _tos_cache0;
tos = _stack_item_0;
PyObject *o = PyStackRef_AsPyObjectBorrow(tos);
if (!PyDict_CheckExact(o)) {
UOP_STAT_INC(uopcode, miss);
GH-142718: JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. (GH-142762) JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. * Make sure that stack is in correct state at side exits with TOS cached values * Simplify choice of cached items for side exits
2025-12-15 19:01:29 +00:00
_tos_cache0 = tos;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(1);
JUMP_TO_JUMP_TARGET();
}
_tos_cache0 = tos;
SET_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _GUARD_TOS_DICT_r22: {
CHECK_CURRENT_CACHED_VALUES(2);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef tos;
_PyStackRef _stack_item_0 = _tos_cache0;
_PyStackRef _stack_item_1 = _tos_cache1;
tos = _stack_item_1;
PyObject *o = PyStackRef_AsPyObjectBorrow(tos);
if (!PyDict_CheckExact(o)) {
UOP_STAT_INC(uopcode, miss);
GH-142718: JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. (GH-142762) JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. * Make sure that stack is in correct state at side exits with TOS cached values * Simplify choice of cached items for side exits
2025-12-15 19:01:29 +00:00
_tos_cache1 = tos;
_tos_cache0 = _stack_item_0;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(2);
JUMP_TO_JUMP_TARGET();
}
_tos_cache1 = tos;
_tos_cache0 = _stack_item_0;
SET_CURRENT_CACHED_VALUES(2);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _GUARD_TOS_DICT_r33: {
CHECK_CURRENT_CACHED_VALUES(3);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef tos;
_PyStackRef _stack_item_0 = _tos_cache0;
_PyStackRef _stack_item_1 = _tos_cache1;
_PyStackRef _stack_item_2 = _tos_cache2;
tos = _stack_item_2;
PyObject *o = PyStackRef_AsPyObjectBorrow(tos);
if (!PyDict_CheckExact(o)) {
UOP_STAT_INC(uopcode, miss);
GH-142718: JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. (GH-142762) JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. * Make sure that stack is in correct state at side exits with TOS cached values * Simplify choice of cached items for side exits
2025-12-15 19:01:29 +00:00
_tos_cache2 = tos;
_tos_cache1 = _stack_item_1;
_tos_cache0 = _stack_item_0;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(3);
GH-131798: Remove JIT guards for dict, frozenset, list, set, and tuple (GH-132289)
2025-04-09 14:32:21 -07:00
JUMP_TO_JUMP_TARGET();
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_tos_cache2 = tos;
_tos_cache1 = _stack_item_1;
_tos_cache0 = _stack_item_0;
SET_CURRENT_CACHED_VALUES(3);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-131798: Remove JIT guards for dict, frozenset, list, set, and tuple (GH-132289)
2025-04-09 14:32:21 -07:00
break;
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
case _BINARY_OP_SUBSCR_DICT_r21: {
CHECK_CURRENT_CACHED_VALUES(2);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
_PyStackRef sub_st;
_PyStackRef dict_st;
_PyStackRef res;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef _stack_item_0 = _tos_cache0;
_PyStackRef _stack_item_1 = _tos_cache1;
sub_st = _stack_item_1;
dict_st = _stack_item_0;
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
PyObject *sub = PyStackRef_AsPyObjectBorrow(sub_st);
PyObject *dict = PyStackRef_AsPyObjectBorrow(dict_st);
GH-131798: Remove JIT guards for dict, frozenset, list, set, and tuple (GH-132289)
2025-04-09 14:32:21 -07:00
assert(PyDict_CheckExact(dict));
gh-100239: replace BINARY_SUBSCR & family by BINARY_OP with oparg NB_SUBSCR (#129700)
2025-02-07 22:39:54 +00:00
STAT_INC(BINARY_OP, hit);
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
PyObject *res_o;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
stack_pointer[0] = dict_st;
stack_pointer[1] = sub_st;
stack_pointer += 2;
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
GH-119866: Spill the stack around escaping calls. (GH-124392) * Spill the evaluation around escaping calls in the generated interpreter and JIT. * The code generator tracks live, cached values so they can be saved to memory when needed. * Spills the stack pointer around escaping calls, so that the exact stack is visible to the cycle GC.
2024-10-07 14:56:39 +01:00
_PyFrame_SetStackPointer(frame, stack_pointer);
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
int rc = PyDict_GetItemRef(dict, sub, &res_o);
GH-119866: Spill the stack around escaping calls. (GH-124392) * Spill the evaluation around escaping calls in the generated interpreter and JIT. * The code generator tracks live, cached values so they can be saved to memory when needed. * Spills the stack pointer around escaping calls, so that the exact stack is visible to the cycle GC.
2024-10-07 14:56:39 +01:00
stack_pointer = _PyFrame_GetStackPointer(frame);
gh-111789: Simplify bytecodes.c by using PyDict_GetItemRef() (GH-111978)
2023-11-14 15:38:49 +02:00
if (rc == 0) {
GH-119866: Spill the stack around escaping calls. (GH-124392) * Spill the evaluation around escaping calls in the generated interpreter and JIT. * The code generator tracks live, cached values so they can be saved to memory when needed. * Spills the stack pointer around escaping calls, so that the exact stack is visible to the cycle GC.
2024-10-07 14:56:39 +01:00
_PyFrame_SetStackPointer(frame, stack_pointer);
gh-111789: Simplify bytecodes.c by using PyDict_GetItemRef() (GH-111978)
2023-11-14 15:38:49 +02:00
_PyErr_SetKeyError(sub);
GH-119866: Spill the stack around escaping calls. (GH-124392) * Spill the evaluation around escaping calls in the generated interpreter and JIT. * The code generator tracks live, cached values so they can be saved to memory when needed. * Spills the stack pointer around escaping calls, so that the exact stack is visible to the cycle GC.
2024-10-07 14:56:39 +01:00
stack_pointer = _PyFrame_GetStackPointer(frame);
gh-104584: Baby steps towards generating and executing traces (#105924) Added a new, experimental, tracing optimizer and interpreter (a.k.a. "tier 2"). This currently pessimizes, so don't use yet -- this is infrastructure so we can experiment with optimizing passes. To enable it, pass ``-Xuops`` or set ``PYTHONUOPS=1``. To get debug output, set ``PYTHONUOPSDEBUG=N`` where ``N`` is a debug level (0-4, where 0 is no debug output and 4 is excessively verbose). All of this code is likely to change dramatically before the 3.13 feature freeze. But this is a first step.
2023-06-26 19:02:57 -07:00
}
GH-128682: Account for escapes in `DECREF_INPUTS` (GH-129953) * Handle escapes in DECREF_INPUTS * Mark a few more functions as escaping * Replace DECREF_INPUTS with PyStackRef_CLOSE where possible
2025-02-12 17:44:59 +00:00
_PyFrame_SetStackPointer(frame, stack_pointer);
_PyStackRef tmp = sub_st;
sub_st = PyStackRef_NULL;
stack_pointer[-1] = sub_st;
PyStackRef_CLOSE(tmp);
tmp = dict_st;
dict_st = PyStackRef_NULL;
stack_pointer[-2] = dict_st;
PyStackRef_CLOSE(tmp);
stack_pointer = _PyFrame_GetStackPointer(frame);
stack_pointer += -2;
GH-141794: Limit size of generated machine code. (GH-142228) * Factor out bodies of the largest uops, to reduce jit code size. * Factor out common assert, also reducing jit code size. * Limit size of jitted code for a single executor to 1MB.
2025-12-03 17:43:35 +00:00
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
GH-128682: Make `PyStackRef_CLOSE` escaping. (GH-129404)
2025-02-03 12:41:32 +00:00
if (rc <= 0) {
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(0);
GH-128682: Make `PyStackRef_CLOSE` escaping. (GH-129404)
2025-02-03 12:41:32 +00:00
JUMP_TO_ERROR();
}
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
res = PyStackRef_FromPyObjectSteal(res_o);
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_tos_cache0 = res;
_tos_cache1 = PyStackRef_ZERO_BITS;
_tos_cache2 = PyStackRef_ZERO_BITS;
SET_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-104584: Baby steps towards generating and executing traces (#105924) Added a new, experimental, tracing optimizer and interpreter (a.k.a. "tier 2"). This currently pessimizes, so don't use yet -- this is infrastructure so we can experiment with optimizing passes. To enable it, pass ``-Xuops`` or set ``PYTHONUOPS=1``. To get debug output, set ``PYTHONUOPSDEBUG=N`` where ``N`` is a debug level (0-4, where 0 is no debug output and 4 is excessively verbose). All of this code is likely to change dramatically before the 3.13 feature freeze. But this is a first step.
2023-06-26 19:02:57 -07:00
break;
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
case _BINARY_OP_SUBSCR_CHECK_FUNC_r23: {
CHECK_CURRENT_CACHED_VALUES(2);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-118095: Add tier two support for BINARY_SUBSCR_GETITEM (GH-120793)
2024-08-02 00:19:05 +01:00
_PyStackRef container;
gh-115999: Enable BINARY_SUBSCR_GETITEM for free-threaded build (gh-127737)
2024-12-19 11:08:17 +09:00
_PyStackRef getitem;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef _stack_item_0 = _tos_cache0;
_PyStackRef _stack_item_1 = _tos_cache1;
container = _stack_item_0;
GH-118095: Add tier two support for BINARY_SUBSCR_GETITEM (GH-120793)
2024-08-02 00:19:05 +01:00
PyTypeObject *tp = Py_TYPE(PyStackRef_AsPyObjectBorrow(container));
if (!PyType_HasFeature(tp, Py_TPFLAGS_HEAPTYPE)) {
UOP_STAT_INC(uopcode, miss);
GH-142718: JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. (GH-142762) JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. * Make sure that stack is in correct state at side exits with TOS cached values * Simplify choice of cached items for side exits
2025-12-15 19:01:29 +00:00
_tos_cache1 = _stack_item_1;
_tos_cache0 = container;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(2);
GH-118095: Add tier two support for BINARY_SUBSCR_GETITEM (GH-120793)
2024-08-02 00:19:05 +01:00
JUMP_TO_JUMP_TARGET();
}
PyHeapTypeObject *ht = (PyHeapTypeObject *)tp;
gh-115999: Enable BINARY_SUBSCR_GETITEM for free-threaded build (gh-127737)
2024-12-19 11:08:17 +09:00
PyObject *getitem_o = FT_ATOMIC_LOAD_PTR_ACQUIRE(ht->_spec_cache.getitem);
if (getitem_o == NULL) {
GH-118095: Add tier two support for BINARY_SUBSCR_GETITEM (GH-120793)
2024-08-02 00:19:05 +01:00
UOP_STAT_INC(uopcode, miss);
GH-142718: JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. (GH-142762) JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. * Make sure that stack is in correct state at side exits with TOS cached values * Simplify choice of cached items for side exits
2025-12-15 19:01:29 +00:00
_tos_cache1 = _stack_item_1;
_tos_cache0 = container;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(2);
GH-118095: Add tier two support for BINARY_SUBSCR_GETITEM (GH-120793)
2024-08-02 00:19:05 +01:00
JUMP_TO_JUMP_TARGET();
}
gh-115999: Enable BINARY_SUBSCR_GETITEM for free-threaded build (gh-127737)
2024-12-19 11:08:17 +09:00
assert(PyFunction_Check(getitem_o));
uint32_t cached_version = FT_ATOMIC_LOAD_UINT32_RELAXED(ht->_spec_cache.getitem_version);
if (((PyFunctionObject *)getitem_o)->func_version != cached_version) {
GH-118095: Add tier two support for BINARY_SUBSCR_GETITEM (GH-120793)
2024-08-02 00:19:05 +01:00
UOP_STAT_INC(uopcode, miss);
GH-142718: JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. (GH-142762) JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. * Make sure that stack is in correct state at side exits with TOS cached values * Simplify choice of cached items for side exits
2025-12-15 19:01:29 +00:00
_tos_cache1 = _stack_item_1;
_tos_cache0 = container;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(2);
GH-118095: Add tier two support for BINARY_SUBSCR_GETITEM (GH-120793)
2024-08-02 00:19:05 +01:00
JUMP_TO_JUMP_TARGET();
}
gh-115999: Enable BINARY_SUBSCR_GETITEM for free-threaded build (gh-127737)
2024-12-19 11:08:17 +09:00
PyCodeObject *code = (PyCodeObject *)PyFunction_GET_CODE(getitem_o);
GH-118095: Add tier two support for BINARY_SUBSCR_GETITEM (GH-120793)
2024-08-02 00:19:05 +01:00
assert(code->co_argcount == 2);
if (!_PyThreadState_HasStackSpace(tstate, code->co_framesize)) {
UOP_STAT_INC(uopcode, miss);
GH-142718: JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. (GH-142762) JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. * Make sure that stack is in correct state at side exits with TOS cached values * Simplify choice of cached items for side exits
2025-12-15 19:01:29 +00:00
_tos_cache1 = _stack_item_1;
_tos_cache0 = container;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(2);
GH-118095: Add tier two support for BINARY_SUBSCR_GETITEM (GH-120793)
2024-08-02 00:19:05 +01:00
JUMP_TO_JUMP_TARGET();
}
gh-115999: Enable BINARY_SUBSCR_GETITEM for free-threaded build (gh-127737)
2024-12-19 11:08:17 +09:00
getitem = PyStackRef_FromPyObjectNew(getitem_o);
gh-100239: replace BINARY_SUBSCR & family by BINARY_OP with oparg NB_SUBSCR (#129700)
2025-02-07 22:39:54 +00:00
STAT_INC(BINARY_OP, hit);
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_tos_cache2 = getitem;
_tos_cache1 = _stack_item_1;
_tos_cache0 = container;
SET_CURRENT_CACHED_VALUES(3);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-118095: Add tier two support for BINARY_SUBSCR_GETITEM (GH-120793)
2024-08-02 00:19:05 +01:00
break;
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
case _BINARY_OP_SUBSCR_INIT_CALL_r01: {
CHECK_CURRENT_CACHED_VALUES(0);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-115999: Enable BINARY_SUBSCR_GETITEM for free-threaded build (gh-127737)
2024-12-19 11:08:17 +09:00
_PyStackRef getitem;
GH-118095: Add tier two support for BINARY_SUBSCR_GETITEM (GH-120793)
2024-08-02 00:19:05 +01:00
_PyStackRef sub;
_PyStackRef container;
GH-135379: Remove types from stack items in code generator. (GH-135384) * Make casts explicit in the instruction definitions
2025-06-11 15:52:25 +01:00
_PyStackRef new_frame;
gh-115999: Enable BINARY_SUBSCR_GETITEM for free-threaded build (gh-127737)
2024-12-19 11:08:17 +09:00
getitem = stack_pointer[-1];
sub = stack_pointer[-2];
container = stack_pointer[-3];
GH-135379: Remove types from stack items in code generator. (GH-135384) * Make casts explicit in the instruction definitions
2025-06-11 15:52:25 +01:00
_PyInterpreterFrame* pushed_frame = _PyFrame_PushUnchecked(tstate, getitem, 2, frame);
pushed_frame->localsplus[0] = container;
pushed_frame->localsplus[1] = sub;
gh-138264: Fix gcc 14 compiler warnings (GH-138265)
2025-09-03 11:26:56 +03:00
frame->return_offset = 6u ;
GH-135379: Remove types from stack items in code generator. (GH-135384) * Make casts explicit in the instruction definitions
2025-06-11 15:52:25 +01:00
new_frame = PyStackRef_Wrap(pushed_frame);
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_tos_cache0 = new_frame;
SET_CURRENT_CACHED_VALUES(1);
stack_pointer += -3;
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _BINARY_OP_SUBSCR_INIT_CALL_r11: {
CHECK_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef getitem;
_PyStackRef sub;
_PyStackRef container;
_PyStackRef new_frame;
_PyStackRef _stack_item_0 = _tos_cache0;
getitem = _stack_item_0;
sub = stack_pointer[-1];
container = stack_pointer[-2];
_PyInterpreterFrame* pushed_frame = _PyFrame_PushUnchecked(tstate, getitem, 2, frame);
pushed_frame->localsplus[0] = container;
pushed_frame->localsplus[1] = sub;
frame->return_offset = 6u ;
new_frame = PyStackRef_Wrap(pushed_frame);
_tos_cache0 = new_frame;
SET_CURRENT_CACHED_VALUES(1);
gh-115999: Enable BINARY_SUBSCR_GETITEM for free-threaded build (gh-127737)
2024-12-19 11:08:17 +09:00
stack_pointer += -2;
GH-141794: Limit size of generated machine code. (GH-142228) * Factor out bodies of the largest uops, to reduce jit code size. * Factor out common assert, also reducing jit code size. * Limit size of jitted code for a single executor to 1MB.
2025-12-03 17:43:35 +00:00
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _BINARY_OP_SUBSCR_INIT_CALL_r21: {
CHECK_CURRENT_CACHED_VALUES(2);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef getitem;
_PyStackRef sub;
_PyStackRef container;
_PyStackRef new_frame;
_PyStackRef _stack_item_0 = _tos_cache0;
_PyStackRef _stack_item_1 = _tos_cache1;
getitem = _stack_item_1;
sub = _stack_item_0;
container = stack_pointer[-1];
_PyInterpreterFrame* pushed_frame = _PyFrame_PushUnchecked(tstate, getitem, 2, frame);
pushed_frame->localsplus[0] = container;
pushed_frame->localsplus[1] = sub;
frame->return_offset = 6u ;
new_frame = PyStackRef_Wrap(pushed_frame);
_tos_cache0 = new_frame;
SET_CURRENT_CACHED_VALUES(1);
stack_pointer += -1;
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _BINARY_OP_SUBSCR_INIT_CALL_r31: {
CHECK_CURRENT_CACHED_VALUES(3);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef getitem;
_PyStackRef sub;
_PyStackRef container;
_PyStackRef new_frame;
_PyStackRef _stack_item_0 = _tos_cache0;
_PyStackRef _stack_item_1 = _tos_cache1;
_PyStackRef _stack_item_2 = _tos_cache2;
getitem = _stack_item_2;
sub = _stack_item_1;
container = _stack_item_0;
_PyInterpreterFrame* pushed_frame = _PyFrame_PushUnchecked(tstate, getitem, 2, frame);
pushed_frame->localsplus[0] = container;
pushed_frame->localsplus[1] = sub;
frame->return_offset = 6u ;
new_frame = PyStackRef_Wrap(pushed_frame);
_tos_cache0 = new_frame;
SET_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-118095: Add tier two support for BINARY_SUBSCR_GETITEM (GH-120793)
2024-08-02 00:19:05 +01:00
break;
}
GH-111485: Factor out tier 2 code generation from the rest of the interpreter code generator (GH-112968)
2023-12-12 12:12:17 +00:00
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
case _LIST_APPEND_r10: {
CHECK_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
_PyStackRef v;
_PyStackRef list;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef _stack_item_0 = _tos_cache0;
GH-111485: Factor out tier 2 code generation from the rest of the interpreter code generator (GH-112968)
2023-12-12 12:12:17 +00:00
oparg = CURRENT_OPARG();
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
v = _stack_item_0;
list = stack_pointer[-1 - (oparg-1)];
GH-119866: Spill the stack around escaping calls. (GH-124392) * Spill the evaluation around escaping calls in the generated interpreter and JIT. * The code generator tracks live, cached values so they can be saved to memory when needed. * Spills the stack pointer around escaping calls, so that the exact stack is visible to the cycle GC.
2024-10-07 14:56:39 +01:00
int err = _PyList_AppendTakeRef((PyListObject *)PyStackRef_AsPyObjectBorrow(list),
PyStackRef_AsPyObjectSteal(v));
GH-128682: Make `PyStackRef_CLOSE` escaping. (GH-129404)
2025-02-03 12:41:32 +00:00
if (err < 0) {
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(0);
GH-128682: Make `PyStackRef_CLOSE` escaping. (GH-129404)
2025-02-03 12:41:32 +00:00
JUMP_TO_ERROR();
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(0);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-104584: Baby steps towards generating and executing traces (#105924) Added a new, experimental, tracing optimizer and interpreter (a.k.a. "tier 2"). This currently pessimizes, so don't use yet -- this is infrastructure so we can experiment with optimizing passes. To enable it, pass ``-Xuops`` or set ``PYTHONUOPS=1``. To get debug output, set ``PYTHONUOPSDEBUG=N`` where ``N`` is a debug level (0-4, where 0 is no debug output and 4 is excessively verbose). All of this code is likely to change dramatically before the 3.13 feature freeze. But this is a first step.
2023-06-26 19:02:57 -07:00
break;
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
case _SET_ADD_r10: {
CHECK_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
_PyStackRef v;
_PyStackRef set;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef _stack_item_0 = _tos_cache0;
GH-111485: Factor out tier 2 code generation from the rest of the interpreter code generator (GH-112968)
2023-12-12 12:12:17 +00:00
oparg = CURRENT_OPARG();
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
v = _stack_item_0;
set = stack_pointer[-1 - (oparg-1)];
stack_pointer[0] = v;
stack_pointer += 1;
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
GH-119866: Spill the stack around escaping calls. (GH-124392) * Spill the evaluation around escaping calls in the generated interpreter and JIT. * The code generator tracks live, cached values so they can be saved to memory when needed. * Spills the stack pointer around escaping calls, so that the exact stack is visible to the cycle GC.
2024-10-07 14:56:39 +01:00
_PyFrame_SetStackPointer(frame, stack_pointer);
gh-130312: SET_ADD should not lock (#130136) SET_ADD should not lock
2025-03-21 15:58:32 -07:00
int err = _PySet_AddTakeRef((PySetObject *)PyStackRef_AsPyObjectBorrow(set),
PyStackRef_AsPyObjectSteal(v));
GH-128682: Account for escapes in `DECREF_INPUTS` (GH-129953) * Handle escapes in DECREF_INPUTS * Mark a few more functions as escaping * Replace DECREF_INPUTS with PyStackRef_CLOSE where possible
2025-02-12 17:44:59 +00:00
stack_pointer = _PyFrame_GetStackPointer(frame);
GH-128682: Make `PyStackRef_CLOSE` escaping. (GH-129404)
2025-02-03 12:41:32 +00:00
if (err) {
gh-130312: SET_ADD should not lock (#130136) SET_ADD should not lock
2025-03-21 15:58:32 -07:00
stack_pointer += -1;
GH-141794: Limit size of generated machine code. (GH-142228) * Factor out bodies of the largest uops, to reduce jit code size. * Factor out common assert, also reducing jit code size. * Limit size of jitted code for a single executor to 1MB.
2025-12-03 17:43:35 +00:00
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(0);
GH-128682: Make `PyStackRef_CLOSE` escaping. (GH-129404)
2025-02-03 12:41:32 +00:00
JUMP_TO_ERROR();
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_tos_cache0 = PyStackRef_ZERO_BITS;
_tos_cache1 = PyStackRef_ZERO_BITS;
_tos_cache2 = PyStackRef_ZERO_BITS;
SET_CURRENT_CACHED_VALUES(0);
gh-130312: SET_ADD should not lock (#130136) SET_ADD should not lock
2025-03-21 15:58:32 -07:00
stack_pointer += -1;
GH-141794: Limit size of generated machine code. (GH-142228) * Factor out bodies of the largest uops, to reduce jit code size. * Factor out common assert, also reducing jit code size. * Limit size of jitted code for a single executor to 1MB.
2025-12-03 17:43:35 +00:00
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-104584: Baby steps towards generating and executing traces (#105924) Added a new, experimental, tracing optimizer and interpreter (a.k.a. "tier 2"). This currently pessimizes, so don't use yet -- this is infrastructure so we can experiment with optimizing passes. To enable it, pass ``-Xuops`` or set ``PYTHONUOPS=1``. To get debug output, set ``PYTHONUOPSDEBUG=N`` where ``N`` is a debug level (0-4, where 0 is no debug output and 4 is excessively verbose). All of this code is likely to change dramatically before the 3.13 feature freeze. But this is a first step.
2023-06-26 19:02:57 -07:00
break;
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
case _STORE_SUBSCR_r30: {
CHECK_CURRENT_CACHED_VALUES(3);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
_PyStackRef sub;
_PyStackRef container;
_PyStackRef v;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef _stack_item_0 = _tos_cache0;
_PyStackRef _stack_item_1 = _tos_cache1;
_PyStackRef _stack_item_2 = _tos_cache2;
sub = _stack_item_2;
container = _stack_item_1;
v = _stack_item_0;
stack_pointer[0] = v;
stack_pointer[1] = container;
stack_pointer[2] = sub;
stack_pointer += 3;
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
GH-119866: Spill the stack around escaping calls. (GH-124392) * Spill the evaluation around escaping calls in the generated interpreter and JIT. * The code generator tracks live, cached values so they can be saved to memory when needed. * Spills the stack pointer around escaping calls, so that the exact stack is visible to the cycle GC.
2024-10-07 14:56:39 +01:00
_PyFrame_SetStackPointer(frame, stack_pointer);
gh-117139: Fix a few wrong steals in bytecodes.c (GH-121127) Fix a few wrong steals in bytecodes.c
2024-06-29 02:14:48 +08:00
int err = PyObject_SetItem(PyStackRef_AsPyObjectBorrow(container), PyStackRef_AsPyObjectBorrow(sub), PyStackRef_AsPyObjectBorrow(v));
GH-128682: Account for escapes in `DECREF_INPUTS` (GH-129953) * Handle escapes in DECREF_INPUTS * Mark a few more functions as escaping * Replace DECREF_INPUTS with PyStackRef_CLOSE where possible
2025-02-12 17:44:59 +00:00
_PyStackRef tmp = sub;
sub = PyStackRef_NULL;
stack_pointer[-1] = sub;
PyStackRef_CLOSE(tmp);
tmp = container;
container = PyStackRef_NULL;
stack_pointer[-2] = container;
PyStackRef_CLOSE(tmp);
tmp = v;
v = PyStackRef_NULL;
stack_pointer[-3] = v;
PyStackRef_CLOSE(tmp);
GH-119866: Spill the stack around escaping calls. (GH-124392) * Spill the evaluation around escaping calls in the generated interpreter and JIT. * The code generator tracks live, cached values so they can be saved to memory when needed. * Spills the stack pointer around escaping calls, so that the exact stack is visible to the cycle GC.
2024-10-07 14:56:39 +01:00
stack_pointer = _PyFrame_GetStackPointer(frame);
GH-128682: Account for escapes in `DECREF_INPUTS` (GH-129953) * Handle escapes in DECREF_INPUTS * Mark a few more functions as escaping * Replace DECREF_INPUTS with PyStackRef_CLOSE where possible
2025-02-12 17:44:59 +00:00
stack_pointer += -3;
GH-141794: Limit size of generated machine code. (GH-142228) * Factor out bodies of the largest uops, to reduce jit code size. * Factor out common assert, also reducing jit code size. * Limit size of jitted code for a single executor to 1MB.
2025-12-03 17:43:35 +00:00
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
GH-128682: Make `PyStackRef_CLOSE` escaping. (GH-129404)
2025-02-03 12:41:32 +00:00
if (err) {
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(0);
GH-128682: Make `PyStackRef_CLOSE` escaping. (GH-129404)
2025-02-03 12:41:32 +00:00
JUMP_TO_ERROR();
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_tos_cache0 = PyStackRef_ZERO_BITS;
_tos_cache1 = PyStackRef_ZERO_BITS;
_tos_cache2 = PyStackRef_ZERO_BITS;
SET_CURRENT_CACHED_VALUES(0);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-104584: Allow unspecialized instructions in superblocks (#106497) This adds several of unspecialized opcodes to superblocks: TO_BOOL, BINARY_SUBSCR, STORE_SUBSCR, UNPACK_SEQUENCE, LOAD_GLOBAL, LOAD_ATTR, COMPARE_OP, BINARY_OP. While we may not want that eventually, for now this helps finding bugs. There is a rudimentary test checking for UNPACK_SEQUENCE. Once we're ready to undo this, that would be simple: just replace the call to variable_used_unspecialized with a call to variable_used (as shown in a comment). Or add individual opcdes to FORBIDDEN_NAMES_IN_UOPS.
2023-07-07 11:03:27 -07:00
break;
}
gh-134584: Eliminate redundant refcounting from _STORE_SUBSCR_LIST_INT (gh-142703)
2025-12-14 21:26:00 +09:00
case _STORE_SUBSCR_LIST_INT_r32: {
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
CHECK_CURRENT_CACHED_VALUES(3);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
_PyStackRef sub_st;
_PyStackRef list_st;
_PyStackRef value;
gh-134584: Eliminate redundant refcounting from _STORE_SUBSCR_LIST_INT (gh-142703)
2025-12-14 21:26:00 +09:00
_PyStackRef ls;
_PyStackRef ss;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef _stack_item_0 = _tos_cache0;
_PyStackRef _stack_item_1 = _tos_cache1;
_PyStackRef _stack_item_2 = _tos_cache2;
sub_st = _stack_item_2;
list_st = _stack_item_1;
value = _stack_item_0;
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
PyObject *sub = PyStackRef_AsPyObjectBorrow(sub_st);
PyObject *list = PyStackRef_AsPyObjectBorrow(list_st);
GH-131798: Allow the JIT to remove more int/float/str guards (GH-131800)
2025-04-01 15:10:15 -07:00
assert(PyLong_CheckExact(sub));
GH-131798: Remove JIT guards for dict, frozenset, list, set, and tuple (GH-132289)
2025-04-09 14:32:21 -07:00
assert(PyList_CheckExact(list));
GH-117457: Correct pystats uop "miss" counts (GH-117477)
2024-04-04 18:49:18 -04:00
if (!_PyLong_IsNonNegativeCompact((PyLongObject *)sub)) {
UOP_STAT_INC(uopcode, miss);
GH-142718: JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. (GH-142762) JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. * Make sure that stack is in correct state at side exits with TOS cached values * Simplify choice of cached items for side exits
2025-12-15 19:01:29 +00:00
_tos_cache2 = sub_st;
_tos_cache1 = list_st;
_tos_cache0 = value;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(3);
GH-117457: Correct pystats uop "miss" counts (GH-117477)
2024-04-04 18:49:18 -04:00
JUMP_TO_JUMP_TARGET();
}
gh-104584: Baby steps towards generating and executing traces (#105924) Added a new, experimental, tracing optimizer and interpreter (a.k.a. "tier 2"). This currently pessimizes, so don't use yet -- this is infrastructure so we can experiment with optimizing passes. To enable it, pass ``-Xuops`` or set ``PYTHONUOPS=1``. To get debug output, set ``PYTHONUOPSDEBUG=N`` where ``N`` is a debug level (0-4, where 0 is no debug output and 4 is excessively verbose). All of this code is likely to change dramatically before the 3.13 feature freeze. But this is a first step.
2023-06-26 19:02:57 -07:00
Py_ssize_t index = ((PyLongObject*)sub)->long_value.ob_digit[0];
gh-115999: Add free-threaded specialization for `STORE_SUBSCR` (#127169) The specialization only depends on the type, so no special thread-safety considerations there. STORE_SUBSCR_LIST_INT needs to lock the list before modifying it. `_PyDict_SetItem_Take2` already internally locks the dictionary using a critical section.
2024-11-26 21:46:06 +00:00
if (!LOCK_OBJECT(list)) {
GH-117457: Correct pystats uop "miss" counts (GH-117477)
2024-04-04 18:49:18 -04:00
UOP_STAT_INC(uopcode, miss);
GH-142718: JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. (GH-142762) JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. * Make sure that stack is in correct state at side exits with TOS cached values * Simplify choice of cached items for side exits
2025-12-15 19:01:29 +00:00
_tos_cache2 = sub_st;
_tos_cache1 = list_st;
_tos_cache0 = value;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(3);
GH-117457: Correct pystats uop "miss" counts (GH-117477)
2024-04-04 18:49:18 -04:00
JUMP_TO_JUMP_TARGET();
}
gh-115999: Add free-threaded specialization for `STORE_SUBSCR` (#127169) The specialization only depends on the type, so no special thread-safety considerations there. STORE_SUBSCR_LIST_INT needs to lock the list before modifying it. `_PyDict_SetItem_Take2` already internally locks the dictionary using a critical section.
2024-11-26 21:46:06 +00:00
if (index >= PyList_GET_SIZE(list)) {
UNLOCK_OBJECT(list);
if (true) {
UOP_STAT_INC(uopcode, miss);
GH-142718: JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. (GH-142762) JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. * Make sure that stack is in correct state at side exits with TOS cached values * Simplify choice of cached items for side exits
2025-12-15 19:01:29 +00:00
_tos_cache2 = sub_st;
_tos_cache1 = list_st;
_tos_cache0 = value;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(3);
gh-115999: Add free-threaded specialization for `STORE_SUBSCR` (#127169) The specialization only depends on the type, so no special thread-safety considerations there. STORE_SUBSCR_LIST_INT needs to lock the list before modifying it. `_PyDict_SetItem_Take2` already internally locks the dictionary using a critical section.
2024-11-26 21:46:06 +00:00
JUMP_TO_JUMP_TARGET();
}
}
gh-104584: Baby steps towards generating and executing traces (#105924) Added a new, experimental, tracing optimizer and interpreter (a.k.a. "tier 2"). This currently pessimizes, so don't use yet -- this is infrastructure so we can experiment with optimizing passes. To enable it, pass ``-Xuops`` or set ``PYTHONUOPS=1``. To get debug output, set ``PYTHONUOPSDEBUG=N`` where ``N`` is a debug level (0-4, where 0 is no debug output and 4 is excessively verbose). All of this code is likely to change dramatically before the 3.13 feature freeze. But this is a first step.
2023-06-26 19:02:57 -07:00
STAT_INC(STORE_SUBSCR, hit);
PyObject *old_value = PyList_GET_ITEM(list, index);
gh-130920: Fix data race in STORE_SUBSCR_LIST_INT (#130923) The write of the item to the list needs to use an atomic operation in the free threading build. Co-authored-by: Tomasz Pytel <tompytel@gmail.com>
2025-03-06 15:59:48 -05:00
FT_ATOMIC_STORE_PTR_RELEASE(_PyList_ITEMS(list)[index],
PyStackRef_AsPyObjectSteal(value));
gh-104584: Baby steps towards generating and executing traces (#105924) Added a new, experimental, tracing optimizer and interpreter (a.k.a. "tier 2"). This currently pessimizes, so don't use yet -- this is infrastructure so we can experiment with optimizing passes. To enable it, pass ``-Xuops`` or set ``PYTHONUOPS=1``. To get debug output, set ``PYTHONUOPSDEBUG=N`` where ``N`` is a debug level (0-4, where 0 is no debug output and 4 is excessively verbose). All of this code is likely to change dramatically before the 3.13 feature freeze. But this is a first step.
2023-06-26 19:02:57 -07:00
assert(old_value != NULL);
GH-131498: Cases generator: Parse down to C statement level. (GH-131948) * Parse down to statement level in the cases generator * Add handling for #if macros, treating them much like normal ifs.
2025-04-02 16:31:59 +01:00
UNLOCK_OBJECT(list);
gh-134584: Eliminate redundant refcounting from _STORE_SUBSCR_LIST_INT (gh-142703)
2025-12-14 21:26:00 +09:00
ls = list_st;
ss = sub_st;
stack_pointer[0] = ls;
stack_pointer[1] = ss;
stack_pointer += 2;
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
GH-128682: Convert explicit loops closing arrays into `DECREF_INPUTS`. (GH-128822) * Mark Py_DECREF and Py_XDECREF as escaping * Remove explicit loops for clearing array inputs
2025-01-14 15:08:56 +00:00
_PyFrame_SetStackPointer(frame, stack_pointer);
Py_DECREF(old_value);
stack_pointer = _PyFrame_GetStackPointer(frame);
gh-134584: Eliminate redundant refcounting from _STORE_SUBSCR_LIST_INT (gh-142703)
2025-12-14 21:26:00 +09:00
_tos_cache1 = ss;
_tos_cache0 = ls;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_tos_cache2 = PyStackRef_ZERO_BITS;
gh-134584: Eliminate redundant refcounting from _STORE_SUBSCR_LIST_INT (gh-142703)
2025-12-14 21:26:00 +09:00
SET_CURRENT_CACHED_VALUES(2);
stack_pointer += -2;
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-104584: Baby steps towards generating and executing traces (#105924) Added a new, experimental, tracing optimizer and interpreter (a.k.a. "tier 2"). This currently pessimizes, so don't use yet -- this is infrastructure so we can experiment with optimizing passes. To enable it, pass ``-Xuops`` or set ``PYTHONUOPS=1``. To get debug output, set ``PYTHONUOPSDEBUG=N`` where ``N`` is a debug level (0-4, where 0 is no debug output and 4 is excessively verbose). All of this code is likely to change dramatically before the 3.13 feature freeze. But this is a first step.
2023-06-26 19:02:57 -07:00
break;
}
gh-134584: Eliminate redundant refcounting from _STORE_SUBSCR_DICT (GH-142712) Co-authored-by: Ken Jin <kenjin4096@gmail.com>
2025-12-16 01:42:32 +09:00
case _STORE_SUBSCR_DICT_r31: {
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
CHECK_CURRENT_CACHED_VALUES(3);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-117139: Fix an incorrect borrow in bytecodes.c (#122318) `_PyDict_SetItem_Take2` steals both the key (i.e., `sub`) and the value.
2024-08-07 09:36:19 -04:00
_PyStackRef sub;
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
_PyStackRef dict_st;
_PyStackRef value;
gh-134584: Eliminate redundant refcounting from _STORE_SUBSCR_DICT (GH-142712) Co-authored-by: Ken Jin <kenjin4096@gmail.com>
2025-12-16 01:42:32 +09:00
_PyStackRef st;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef _stack_item_0 = _tos_cache0;
_PyStackRef _stack_item_1 = _tos_cache1;
_PyStackRef _stack_item_2 = _tos_cache2;
sub = _stack_item_2;
dict_st = _stack_item_1;
value = _stack_item_0;
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
PyObject *dict = PyStackRef_AsPyObjectBorrow(dict_st);
GH-131798: Remove JIT guards for dict, frozenset, list, set, and tuple (GH-132289)
2025-04-09 14:32:21 -07:00
assert(PyDict_CheckExact(dict));
gh-104584: Baby steps towards generating and executing traces (#105924) Added a new, experimental, tracing optimizer and interpreter (a.k.a. "tier 2"). This currently pessimizes, so don't use yet -- this is infrastructure so we can experiment with optimizing passes. To enable it, pass ``-Xuops`` or set ``PYTHONUOPS=1``. To get debug output, set ``PYTHONUOPSDEBUG=N`` where ``N`` is a debug level (0-4, where 0 is no debug output and 4 is excessively verbose). All of this code is likely to change dramatically before the 3.13 feature freeze. But this is a first step.
2023-06-26 19:02:57 -07:00
STAT_INC(STORE_SUBSCR, hit);
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
stack_pointer[0] = value;
stack_pointer[1] = dict_st;
stack_pointer[2] = sub;
stack_pointer += 3;
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
GH-119866: Spill the stack around escaping calls. (GH-124392) * Spill the evaluation around escaping calls in the generated interpreter and JIT. * The code generator tracks live, cached values so they can be saved to memory when needed. * Spills the stack pointer around escaping calls, so that the exact stack is visible to the cycle GC.
2024-10-07 14:56:39 +01:00
_PyFrame_SetStackPointer(frame, stack_pointer);
gh-117139: Fix an incorrect borrow in bytecodes.c (#122318) `_PyDict_SetItem_Take2` steals both the key (i.e., `sub`) and the value.
2024-08-07 09:36:19 -04:00
int err = _PyDict_SetItem_Take2((PyDictObject *)dict,
PyStackRef_AsPyObjectSteal(sub),
PyStackRef_AsPyObjectSteal(value));
GH-119866: Spill the stack around escaping calls. (GH-124392) * Spill the evaluation around escaping calls in the generated interpreter and JIT. * The code generator tracks live, cached values so they can be saved to memory when needed. * Spills the stack pointer around escaping calls, so that the exact stack is visible to the cycle GC.
2024-10-07 14:56:39 +01:00
stack_pointer = _PyFrame_GetStackPointer(frame);
GH-128682: Make `PyStackRef_CLOSE` escaping. (GH-129404)
2025-02-03 12:41:32 +00:00
if (err) {
gh-134584: Eliminate redundant refcounting from _STORE_SUBSCR_DICT (GH-142712) Co-authored-by: Ken Jin <kenjin4096@gmail.com>
2025-12-16 01:42:32 +09:00
stack_pointer += -3;
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
_PyFrame_SetStackPointer(frame, stack_pointer);
PyStackRef_CLOSE(dict_st);
stack_pointer = _PyFrame_GetStackPointer(frame);
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(0);
GH-128682: Make `PyStackRef_CLOSE` escaping. (GH-129404)
2025-02-03 12:41:32 +00:00
JUMP_TO_ERROR();
}
gh-134584: Eliminate redundant refcounting from _STORE_SUBSCR_DICT (GH-142712) Co-authored-by: Ken Jin <kenjin4096@gmail.com>
2025-12-16 01:42:32 +09:00
st = dict_st;
_tos_cache0 = st;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_tos_cache1 = PyStackRef_ZERO_BITS;
_tos_cache2 = PyStackRef_ZERO_BITS;
gh-134584: Eliminate redundant refcounting from _STORE_SUBSCR_DICT (GH-142712) Co-authored-by: Ken Jin <kenjin4096@gmail.com>
2025-12-16 01:42:32 +09:00
SET_CURRENT_CACHED_VALUES(1);
stack_pointer += -3;
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-104584: Baby steps towards generating and executing traces (#105924) Added a new, experimental, tracing optimizer and interpreter (a.k.a. "tier 2"). This currently pessimizes, so don't use yet -- this is infrastructure so we can experiment with optimizing passes. To enable it, pass ``-Xuops`` or set ``PYTHONUOPS=1``. To get debug output, set ``PYTHONUOPSDEBUG=N`` where ``N`` is a debug level (0-4, where 0 is no debug output and 4 is excessively verbose). All of this code is likely to change dramatically before the 3.13 feature freeze. But this is a first step.
2023-06-26 19:02:57 -07:00
break;
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
case _DELETE_SUBSCR_r20: {
CHECK_CURRENT_CACHED_VALUES(2);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
_PyStackRef sub;
_PyStackRef container;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef _stack_item_0 = _tos_cache0;
_PyStackRef _stack_item_1 = _tos_cache1;
sub = _stack_item_1;
container = _stack_item_0;
stack_pointer[0] = container;
stack_pointer[1] = sub;
stack_pointer += 2;
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
GH-119866: Spill the stack around escaping calls. (GH-124392) * Spill the evaluation around escaping calls in the generated interpreter and JIT. * The code generator tracks live, cached values so they can be saved to memory when needed. * Spills the stack pointer around escaping calls, so that the exact stack is visible to the cycle GC.
2024-10-07 14:56:39 +01:00
_PyFrame_SetStackPointer(frame, stack_pointer);
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
int err = PyObject_DelItem(PyStackRef_AsPyObjectBorrow(container),
PyStackRef_AsPyObjectBorrow(sub));
GH-128682: Account for escapes in `DECREF_INPUTS` (GH-129953) * Handle escapes in DECREF_INPUTS * Mark a few more functions as escaping * Replace DECREF_INPUTS with PyStackRef_CLOSE where possible
2025-02-12 17:44:59 +00:00
_PyStackRef tmp = sub;
sub = PyStackRef_NULL;
stack_pointer[-1] = sub;
PyStackRef_CLOSE(tmp);
tmp = container;
container = PyStackRef_NULL;
stack_pointer[-2] = container;
PyStackRef_CLOSE(tmp);
GH-119866: Spill the stack around escaping calls. (GH-124392) * Spill the evaluation around escaping calls in the generated interpreter and JIT. * The code generator tracks live, cached values so they can be saved to memory when needed. * Spills the stack pointer around escaping calls, so that the exact stack is visible to the cycle GC.
2024-10-07 14:56:39 +01:00
stack_pointer = _PyFrame_GetStackPointer(frame);
GH-128682: Account for escapes in `DECREF_INPUTS` (GH-129953) * Handle escapes in DECREF_INPUTS * Mark a few more functions as escaping * Replace DECREF_INPUTS with PyStackRef_CLOSE where possible
2025-02-12 17:44:59 +00:00
stack_pointer += -2;
GH-141794: Limit size of generated machine code. (GH-142228) * Factor out bodies of the largest uops, to reduce jit code size. * Factor out common assert, also reducing jit code size. * Limit size of jitted code for a single executor to 1MB.
2025-12-03 17:43:35 +00:00
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
GH-128682: Make `PyStackRef_CLOSE` escaping. (GH-129404)
2025-02-03 12:41:32 +00:00
if (err) {
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(0);
GH-128682: Make `PyStackRef_CLOSE` escaping. (GH-129404)
2025-02-03 12:41:32 +00:00
JUMP_TO_ERROR();
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_tos_cache0 = PyStackRef_ZERO_BITS;
_tos_cache1 = PyStackRef_ZERO_BITS;
_tos_cache2 = PyStackRef_ZERO_BITS;
SET_CURRENT_CACHED_VALUES(0);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-104584: Baby steps towards generating and executing traces (#105924) Added a new, experimental, tracing optimizer and interpreter (a.k.a. "tier 2"). This currently pessimizes, so don't use yet -- this is infrastructure so we can experiment with optimizing passes. To enable it, pass ``-Xuops`` or set ``PYTHONUOPS=1``. To get debug output, set ``PYTHONUOPSDEBUG=N`` where ``N`` is a debug level (0-4, where 0 is no debug output and 4 is excessively verbose). All of this code is likely to change dramatically before the 3.13 feature freeze. But this is a first step.
2023-06-26 19:02:57 -07:00
break;
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
case _CALL_INTRINSIC_1_r11: {
CHECK_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
_PyStackRef value;
_PyStackRef res;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef _stack_item_0 = _tos_cache0;
GH-111485: Factor out tier 2 code generation from the rest of the interpreter code generator (GH-112968)
2023-12-12 12:12:17 +00:00
oparg = CURRENT_OPARG();
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
value = _stack_item_0;
gh-104584: Baby steps towards generating and executing traces (#105924) Added a new, experimental, tracing optimizer and interpreter (a.k.a. "tier 2"). This currently pessimizes, so don't use yet -- this is infrastructure so we can experiment with optimizing passes. To enable it, pass ``-Xuops`` or set ``PYTHONUOPS=1``. To get debug output, set ``PYTHONUOPSDEBUG=N`` where ``N`` is a debug level (0-4, where 0 is no debug output and 4 is excessively verbose). All of this code is likely to change dramatically before the 3.13 feature freeze. But this is a first step.
2023-06-26 19:02:57 -07:00
assert(oparg <= MAX_INTRINSIC_1);
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
stack_pointer[0] = value;
stack_pointer += 1;
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
GH-119866: Spill the stack around escaping calls. (GH-124392) * Spill the evaluation around escaping calls in the generated interpreter and JIT. * The code generator tracks live, cached values so they can be saved to memory when needed. * Spills the stack pointer around escaping calls, so that the exact stack is visible to the cycle GC.
2024-10-07 14:56:39 +01:00
_PyFrame_SetStackPointer(frame, stack_pointer);
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
PyObject *res_o = _PyIntrinsics_UnaryFunctions[oparg].func(tstate, PyStackRef_AsPyObjectBorrow(value));
GH-119866: Spill the stack around escaping calls. (GH-124392) * Spill the evaluation around escaping calls in the generated interpreter and JIT. * The code generator tracks live, cached values so they can be saved to memory when needed. * Spills the stack pointer around escaping calls, so that the exact stack is visible to the cycle GC.
2024-10-07 14:56:39 +01:00
stack_pointer = _PyFrame_GetStackPointer(frame);
GH-128682: Account for escapes in `DECREF_INPUTS` (GH-129953) * Handle escapes in DECREF_INPUTS * Mark a few more functions as escaping * Replace DECREF_INPUTS with PyStackRef_CLOSE where possible
2025-02-12 17:44:59 +00:00
stack_pointer += -1;
GH-141794: Limit size of generated machine code. (GH-142228) * Factor out bodies of the largest uops, to reduce jit code size. * Factor out common assert, also reducing jit code size. * Limit size of jitted code for a single executor to 1MB.
2025-12-03 17:43:35 +00:00
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
GH-128682: Account for escapes in `DECREF_INPUTS` (GH-129953) * Handle escapes in DECREF_INPUTS * Mark a few more functions as escaping * Replace DECREF_INPUTS with PyStackRef_CLOSE where possible
2025-02-12 17:44:59 +00:00
_PyFrame_SetStackPointer(frame, stack_pointer);
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
PyStackRef_CLOSE(value);
GH-128682: Account for escapes in `DECREF_INPUTS` (GH-129953) * Handle escapes in DECREF_INPUTS * Mark a few more functions as escaping * Replace DECREF_INPUTS with PyStackRef_CLOSE where possible
2025-02-12 17:44:59 +00:00
stack_pointer = _PyFrame_GetStackPointer(frame);
GH-128682: Make `PyStackRef_CLOSE` escaping. (GH-129404)
2025-02-03 12:41:32 +00:00
if (res_o == NULL) {
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(0);
GH-128682: Make `PyStackRef_CLOSE` escaping. (GH-129404)
2025-02-03 12:41:32 +00:00
JUMP_TO_ERROR();
}
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
res = PyStackRef_FromPyObjectSteal(res_o);
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_tos_cache0 = res;
_tos_cache1 = PyStackRef_ZERO_BITS;
_tos_cache2 = PyStackRef_ZERO_BITS;
SET_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-104584: Baby steps towards generating and executing traces (#105924) Added a new, experimental, tracing optimizer and interpreter (a.k.a. "tier 2"). This currently pessimizes, so don't use yet -- this is infrastructure so we can experiment with optimizing passes. To enable it, pass ``-Xuops`` or set ``PYTHONUOPS=1``. To get debug output, set ``PYTHONUOPSDEBUG=N`` where ``N`` is a debug level (0-4, where 0 is no debug output and 4 is excessively verbose). All of this code is likely to change dramatically before the 3.13 feature freeze. But this is a first step.
2023-06-26 19:02:57 -07:00
break;
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
case _CALL_INTRINSIC_2_r21: {
CHECK_CURRENT_CACHED_VALUES(2);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
_PyStackRef value1_st;
_PyStackRef value2_st;
_PyStackRef res;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef _stack_item_0 = _tos_cache0;
_PyStackRef _stack_item_1 = _tos_cache1;
GH-111485: Factor out tier 2 code generation from the rest of the interpreter code generator (GH-112968)
2023-12-12 12:12:17 +00:00
oparg = CURRENT_OPARG();
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
value1_st = _stack_item_1;
value2_st = _stack_item_0;
gh-104584: Baby steps towards generating and executing traces (#105924) Added a new, experimental, tracing optimizer and interpreter (a.k.a. "tier 2"). This currently pessimizes, so don't use yet -- this is infrastructure so we can experiment with optimizing passes. To enable it, pass ``-Xuops`` or set ``PYTHONUOPS=1``. To get debug output, set ``PYTHONUOPSDEBUG=N`` where ``N`` is a debug level (0-4, where 0 is no debug output and 4 is excessively verbose). All of this code is likely to change dramatically before the 3.13 feature freeze. But this is a first step.
2023-06-26 19:02:57 -07:00
assert(oparg <= MAX_INTRINSIC_2);
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
PyObject *value1 = PyStackRef_AsPyObjectBorrow(value1_st);
PyObject *value2 = PyStackRef_AsPyObjectBorrow(value2_st);
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
stack_pointer[0] = value2_st;
stack_pointer[1] = value1_st;
stack_pointer += 2;
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
GH-119866: Spill the stack around escaping calls. (GH-124392) * Spill the evaluation around escaping calls in the generated interpreter and JIT. * The code generator tracks live, cached values so they can be saved to memory when needed. * Spills the stack pointer around escaping calls, so that the exact stack is visible to the cycle GC.
2024-10-07 14:56:39 +01:00
_PyFrame_SetStackPointer(frame, stack_pointer);
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
PyObject *res_o = _PyIntrinsics_BinaryFunctions[oparg].func(tstate, value2, value1);
GH-128682: Account for escapes in `DECREF_INPUTS` (GH-129953) * Handle escapes in DECREF_INPUTS * Mark a few more functions as escaping * Replace DECREF_INPUTS with PyStackRef_CLOSE where possible
2025-02-12 17:44:59 +00:00
_PyStackRef tmp = value1_st;
value1_st = PyStackRef_NULL;
stack_pointer[-1] = value1_st;
PyStackRef_CLOSE(tmp);
tmp = value2_st;
value2_st = PyStackRef_NULL;
stack_pointer[-2] = value2_st;
PyStackRef_CLOSE(tmp);
GH-119866: Spill the stack around escaping calls. (GH-124392) * Spill the evaluation around escaping calls in the generated interpreter and JIT. * The code generator tracks live, cached values so they can be saved to memory when needed. * Spills the stack pointer around escaping calls, so that the exact stack is visible to the cycle GC.
2024-10-07 14:56:39 +01:00
stack_pointer = _PyFrame_GetStackPointer(frame);
GH-128682: Account for escapes in `DECREF_INPUTS` (GH-129953) * Handle escapes in DECREF_INPUTS * Mark a few more functions as escaping * Replace DECREF_INPUTS with PyStackRef_CLOSE where possible
2025-02-12 17:44:59 +00:00
stack_pointer += -2;
GH-141794: Limit size of generated machine code. (GH-142228) * Factor out bodies of the largest uops, to reduce jit code size. * Factor out common assert, also reducing jit code size. * Limit size of jitted code for a single executor to 1MB.
2025-12-03 17:43:35 +00:00
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
GH-128682: Make `PyStackRef_CLOSE` escaping. (GH-129404)
2025-02-03 12:41:32 +00:00
if (res_o == NULL) {
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(0);
GH-128682: Make `PyStackRef_CLOSE` escaping. (GH-129404)
2025-02-03 12:41:32 +00:00
JUMP_TO_ERROR();
}
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
res = PyStackRef_FromPyObjectSteal(res_o);
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_tos_cache0 = res;
_tos_cache1 = PyStackRef_ZERO_BITS;
_tos_cache2 = PyStackRef_ZERO_BITS;
SET_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-104584: Baby steps towards generating and executing traces (#105924) Added a new, experimental, tracing optimizer and interpreter (a.k.a. "tier 2"). This currently pessimizes, so don't use yet -- this is infrastructure so we can experiment with optimizing passes. To enable it, pass ``-Xuops`` or set ``PYTHONUOPS=1``. To get debug output, set ``PYTHONUOPSDEBUG=N`` where ``N`` is a debug level (0-4, where 0 is no debug output and 4 is excessively verbose). All of this code is likely to change dramatically before the 3.13 feature freeze. But this is a first step.
2023-06-26 19:02:57 -07:00
break;
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
case _RETURN_VALUE_r11: {
CHECK_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
_PyStackRef retval;
_PyStackRef res;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef _stack_item_0 = _tos_cache0;
retval = _stack_item_0;
GH-128563: Add new frame owner type for interpreter entry frames (GH-129078) Add new frame owner type for interpreter entry frames
2025-01-21 10:15:02 +00:00
assert(frame->owner != FRAME_OWNED_BY_INTERPRETER);
gh-130704: Strength reduce `LOAD_FAST{_LOAD_FAST}` (#130708) Optimize `LOAD_FAST` opcodes into faster versions that load borrowed references onto the operand stack when we can prove that the lifetime of the local outlives the lifetime of the temporary that is loaded onto the stack.
2025-04-01 10:18:42 -07:00
_PyStackRef temp = PyStackRef_MakeHeapSafe(retval);
GH-111485: Factor out tier 2 code generation from the rest of the interpreter code generator (GH-112968)
2023-12-12 12:12:17 +00:00
_PyFrame_SetStackPointer(frame, stack_pointer);
GH-131498: Cases generator: manage stacks automatically (GH-132074)
2025-04-04 17:59:36 +01:00
assert(STACK_LEVEL() == 0);
GH-108614: Remove non-debug uses of `#if TIER_ONE` and `#if TIER_TWO` from `_POP_FRAME` op. (GH-108685)
2023-08-31 11:34:52 +01:00
_Py_LeaveRecursiveCallPy(tstate);
_PyInterpreterFrame *dying = frame;
gh-106581: Project through calls (#108067) This finishes the work begun in gh-107760. When, while projecting a superblock, we encounter a call to a short, simple function, the superblock will now enter the function using `_PUSH_FRAME`, continue through it, and leave it using `_POP_FRAME`, and then continue through the original code. Multiple frame pushes and pops are even possible. It is also possible to stop appending to the superblock in the middle of a called function, when running out of space or encountering an unsupported bytecode.
2023-08-17 11:29:58 -07:00
frame = tstate->current_frame = dying->previous;
_PyEval_FrameClearAndPop(tstate, dying);
GH-119866: Spill the stack around escaping calls. (GH-124392) * Spill the evaluation around escaping calls in the generated interpreter and JIT. * The code generator tracks live, cached values so they can be saved to memory when needed. * Spills the stack pointer around escaping calls, so that the exact stack is visible to the cycle GC.
2024-10-07 14:56:39 +01:00
stack_pointer = _PyFrame_GetStackPointer(frame);
gh-109094: replace frame->prev_instr by frame->instr_ptr (#109095)
2023-10-26 14:43:10 +01:00
LOAD_IP(frame->return_offset);
GH-119866: Spill the stack around escaping calls. (GH-124392) * Spill the evaluation around escaping calls in the generated interpreter and JIT. * The code generator tracks live, cached values so they can be saved to memory when needed. * Spills the stack pointer around escaping calls, so that the exact stack is visible to the cycle GC.
2024-10-07 14:56:39 +01:00
res = temp;
GH-118095: Handle `RETURN_GENERATOR` in tier 2 (GH-118180)
2024-04-25 11:32:47 +01:00
LLTRACE_RESUME_FRAME();
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_tos_cache0 = res;
_tos_cache1 = PyStackRef_ZERO_BITS;
_tos_cache2 = PyStackRef_ZERO_BITS;
SET_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-106581: Project through calls (#108067) This finishes the work begun in gh-107760. When, while projecting a superblock, we encounter a call to a short, simple function, the superblock will now enter the function using `_PUSH_FRAME`, continue through it, and leave it using `_POP_FRAME`, and then continue through the original code. Multiple frame pushes and pops are even possible. It is also possible to stop appending to the superblock in the middle of a called function, when running out of space or encountering an unsupported bytecode.
2023-08-17 11:29:58 -07:00
break;
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
case _GET_AITER_r11: {
CHECK_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
_PyStackRef obj;
_PyStackRef iter;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef _stack_item_0 = _tos_cache0;
obj = _stack_item_0;
gh-104584: Baby steps towards generating and executing traces (#105924) Added a new, experimental, tracing optimizer and interpreter (a.k.a. "tier 2"). This currently pessimizes, so don't use yet -- this is infrastructure so we can experiment with optimizing passes. To enable it, pass ``-Xuops`` or set ``PYTHONUOPS=1``. To get debug output, set ``PYTHONUOPSDEBUG=N`` where ``N`` is a debug level (0-4, where 0 is no debug output and 4 is excessively verbose). All of this code is likely to change dramatically before the 3.13 feature freeze. But this is a first step.
2023-06-26 19:02:57 -07:00
unaryfunc getter = NULL;
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
PyObject *obj_o = PyStackRef_AsPyObjectBorrow(obj);
PyObject *iter_o;
PyTypeObject *type = Py_TYPE(obj_o);
gh-104584: Baby steps towards generating and executing traces (#105924) Added a new, experimental, tracing optimizer and interpreter (a.k.a. "tier 2"). This currently pessimizes, so don't use yet -- this is infrastructure so we can experiment with optimizing passes. To enable it, pass ``-Xuops`` or set ``PYTHONUOPS=1``. To get debug output, set ``PYTHONUOPSDEBUG=N`` where ``N`` is a debug level (0-4, where 0 is no debug output and 4 is excessively verbose). All of this code is likely to change dramatically before the 3.13 feature freeze. But this is a first step.
2023-06-26 19:02:57 -07:00
if (type->tp_as_async != NULL) {
getter = type->tp_as_async->am_aiter;
}
if (getter == NULL) {
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
stack_pointer[0] = obj;
stack_pointer += 1;
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
GH-119866: Spill the stack around escaping calls. (GH-124392) * Spill the evaluation around escaping calls in the generated interpreter and JIT. * The code generator tracks live, cached values so they can be saved to memory when needed. * Spills the stack pointer around escaping calls, so that the exact stack is visible to the cycle GC.
2024-10-07 14:56:39 +01:00
_PyFrame_SetStackPointer(frame, stack_pointer);
gh-104584: Baby steps towards generating and executing traces (#105924) Added a new, experimental, tracing optimizer and interpreter (a.k.a. "tier 2"). This currently pessimizes, so don't use yet -- this is infrastructure so we can experiment with optimizing passes. To enable it, pass ``-Xuops`` or set ``PYTHONUOPS=1``. To get debug output, set ``PYTHONUOPSDEBUG=N`` where ``N`` is a debug level (0-4, where 0 is no debug output and 4 is excessively verbose). All of this code is likely to change dramatically before the 3.13 feature freeze. But this is a first step.
2023-06-26 19:02:57 -07:00
_PyErr_Format(tstate, PyExc_TypeError,
"'async for' requires an object with "
"__aiter__ method, got %.100s",
type->tp_name);
GH-119866: Spill the stack around escaping calls. (GH-124392) * Spill the evaluation around escaping calls in the generated interpreter and JIT. * The code generator tracks live, cached values so they can be saved to memory when needed. * Spills the stack pointer around escaping calls, so that the exact stack is visible to the cycle GC.
2024-10-07 14:56:39 +01:00
stack_pointer = _PyFrame_GetStackPointer(frame);
GH-128682: Make `PyStackRef_CLOSE` escaping. (GH-129404)
2025-02-03 12:41:32 +00:00
stack_pointer += -1;
GH-141794: Limit size of generated machine code. (GH-142228) * Factor out bodies of the largest uops, to reduce jit code size. * Factor out common assert, also reducing jit code size. * Limit size of jitted code for a single executor to 1MB.
2025-12-03 17:43:35 +00:00
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
GH-128682: Account for escapes in `DECREF_INPUTS` (GH-129953) * Handle escapes in DECREF_INPUTS * Mark a few more functions as escaping * Replace DECREF_INPUTS with PyStackRef_CLOSE where possible
2025-02-12 17:44:59 +00:00
_PyFrame_SetStackPointer(frame, stack_pointer);
PyStackRef_CLOSE(obj);
stack_pointer = _PyFrame_GetStackPointer(frame);
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(0);
GH-128682: Make `PyStackRef_CLOSE` escaping. (GH-129404)
2025-02-03 12:41:32 +00:00
JUMP_TO_ERROR();
gh-104584: Baby steps towards generating and executing traces (#105924) Added a new, experimental, tracing optimizer and interpreter (a.k.a. "tier 2"). This currently pessimizes, so don't use yet -- this is infrastructure so we can experiment with optimizing passes. To enable it, pass ``-Xuops`` or set ``PYTHONUOPS=1``. To get debug output, set ``PYTHONUOPSDEBUG=N`` where ``N`` is a debug level (0-4, where 0 is no debug output and 4 is excessively verbose). All of this code is likely to change dramatically before the 3.13 feature freeze. But this is a first step.
2023-06-26 19:02:57 -07:00
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
stack_pointer[0] = obj;
stack_pointer += 1;
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
GH-119866: Spill the stack around escaping calls. (GH-124392) * Spill the evaluation around escaping calls in the generated interpreter and JIT. * The code generator tracks live, cached values so they can be saved to memory when needed. * Spills the stack pointer around escaping calls, so that the exact stack is visible to the cycle GC.
2024-10-07 14:56:39 +01:00
_PyFrame_SetStackPointer(frame, stack_pointer);
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
iter_o = (*getter)(obj_o);
GH-119866: Spill the stack around escaping calls. (GH-124392) * Spill the evaluation around escaping calls in the generated interpreter and JIT. * The code generator tracks live, cached values so they can be saved to memory when needed. * Spills the stack pointer around escaping calls, so that the exact stack is visible to the cycle GC.
2024-10-07 14:56:39 +01:00
stack_pointer = _PyFrame_GetStackPointer(frame);
GH-128682: Account for escapes in `DECREF_INPUTS` (GH-129953) * Handle escapes in DECREF_INPUTS * Mark a few more functions as escaping * Replace DECREF_INPUTS with PyStackRef_CLOSE where possible
2025-02-12 17:44:59 +00:00
stack_pointer += -1;
GH-141794: Limit size of generated machine code. (GH-142228) * Factor out bodies of the largest uops, to reduce jit code size. * Factor out common assert, also reducing jit code size. * Limit size of jitted code for a single executor to 1MB.
2025-12-03 17:43:35 +00:00
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
GH-128682: Account for escapes in `DECREF_INPUTS` (GH-129953) * Handle escapes in DECREF_INPUTS * Mark a few more functions as escaping * Replace DECREF_INPUTS with PyStackRef_CLOSE where possible
2025-02-12 17:44:59 +00:00
_PyFrame_SetStackPointer(frame, stack_pointer);
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
PyStackRef_CLOSE(obj);
GH-128682: Account for escapes in `DECREF_INPUTS` (GH-129953) * Handle escapes in DECREF_INPUTS * Mark a few more functions as escaping * Replace DECREF_INPUTS with PyStackRef_CLOSE where possible
2025-02-12 17:44:59 +00:00
stack_pointer = _PyFrame_GetStackPointer(frame);
GH-128682: Make `PyStackRef_CLOSE` escaping. (GH-129404)
2025-02-03 12:41:32 +00:00
if (iter_o == NULL) {
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(0);
GH-128682: Make `PyStackRef_CLOSE` escaping. (GH-129404)
2025-02-03 12:41:32 +00:00
JUMP_TO_ERROR();
}
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
if (Py_TYPE(iter_o)->tp_as_async == NULL ||
Py_TYPE(iter_o)->tp_as_async->am_anext == NULL) {
GH-119866: Spill the stack around escaping calls. (GH-124392) * Spill the evaluation around escaping calls in the generated interpreter and JIT. * The code generator tracks live, cached values so they can be saved to memory when needed. * Spills the stack pointer around escaping calls, so that the exact stack is visible to the cycle GC.
2024-10-07 14:56:39 +01:00
_PyFrame_SetStackPointer(frame, stack_pointer);
gh-104584: Baby steps towards generating and executing traces (#105924) Added a new, experimental, tracing optimizer and interpreter (a.k.a. "tier 2"). This currently pessimizes, so don't use yet -- this is infrastructure so we can experiment with optimizing passes. To enable it, pass ``-Xuops`` or set ``PYTHONUOPS=1``. To get debug output, set ``PYTHONUOPSDEBUG=N`` where ``N`` is a debug level (0-4, where 0 is no debug output and 4 is excessively verbose). All of this code is likely to change dramatically before the 3.13 feature freeze. But this is a first step.
2023-06-26 19:02:57 -07:00
_PyErr_Format(tstate, PyExc_TypeError,
"'async for' received an object from __aiter__ "
"that does not implement __anext__: %.100s",
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
Py_TYPE(iter_o)->tp_name);
Py_DECREF(iter_o);
GH-128682: Convert explicit loops closing arrays into `DECREF_INPUTS`. (GH-128822) * Mark Py_DECREF and Py_XDECREF as escaping * Remove explicit loops for clearing array inputs
2025-01-14 15:08:56 +00:00
stack_pointer = _PyFrame_GetStackPointer(frame);
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(0);
GH-128682: Make `PyStackRef_CLOSE` escaping. (GH-129404)
2025-02-03 12:41:32 +00:00
JUMP_TO_ERROR();
gh-104584: Baby steps towards generating and executing traces (#105924) Added a new, experimental, tracing optimizer and interpreter (a.k.a. "tier 2"). This currently pessimizes, so don't use yet -- this is infrastructure so we can experiment with optimizing passes. To enable it, pass ``-Xuops`` or set ``PYTHONUOPS=1``. To get debug output, set ``PYTHONUOPSDEBUG=N`` where ``N`` is a debug level (0-4, where 0 is no debug output and 4 is excessively verbose). All of this code is likely to change dramatically before the 3.13 feature freeze. But this is a first step.
2023-06-26 19:02:57 -07:00
}
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
iter = PyStackRef_FromPyObjectSteal(iter_o);
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_tos_cache0 = iter;
_tos_cache1 = PyStackRef_ZERO_BITS;
_tos_cache2 = PyStackRef_ZERO_BITS;
SET_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-104584: Baby steps towards generating and executing traces (#105924) Added a new, experimental, tracing optimizer and interpreter (a.k.a. "tier 2"). This currently pessimizes, so don't use yet -- this is infrastructure so we can experiment with optimizing passes. To enable it, pass ``-Xuops`` or set ``PYTHONUOPS=1``. To get debug output, set ``PYTHONUOPSDEBUG=N`` where ``N`` is a debug level (0-4, where 0 is no debug output and 4 is excessively verbose). All of this code is likely to change dramatically before the 3.13 feature freeze. But this is a first step.
2023-06-26 19:02:57 -07:00
break;
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
case _GET_ANEXT_r12: {
CHECK_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
_PyStackRef aiter;
_PyStackRef awaitable;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef _stack_item_0 = _tos_cache0;
aiter = _stack_item_0;
stack_pointer[0] = aiter;
stack_pointer += 1;
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
GH-119866: Spill the stack around escaping calls. (GH-124392) * Spill the evaluation around escaping calls in the generated interpreter and JIT. * The code generator tracks live, cached values so they can be saved to memory when needed. * Spills the stack pointer around escaping calls, so that the exact stack is visible to the cycle GC.
2024-10-07 14:56:39 +01:00
_PyFrame_SetStackPointer(frame, stack_pointer);
GH-117224: Move the body of a few large-ish micro-ops into helper functions (GH-122601)
2024-08-02 16:31:17 +01:00
PyObject *awaitable_o = _PyEval_GetANext(PyStackRef_AsPyObjectBorrow(aiter));
GH-119866: Spill the stack around escaping calls. (GH-124392) * Spill the evaluation around escaping calls in the generated interpreter and JIT. * The code generator tracks live, cached values so they can be saved to memory when needed. * Spills the stack pointer around escaping calls, so that the exact stack is visible to the cycle GC.
2024-10-07 14:56:39 +01:00
stack_pointer = _PyFrame_GetStackPointer(frame);
GH-117224: Move the body of a few large-ish micro-ops into helper functions (GH-122601)
2024-08-02 16:31:17 +01:00
if (awaitable_o == NULL) {
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(0);
GH-117224: Move the body of a few large-ish micro-ops into helper functions (GH-122601)
2024-08-02 16:31:17 +01:00
JUMP_TO_ERROR();
gh-104584: Baby steps towards generating and executing traces (#105924) Added a new, experimental, tracing optimizer and interpreter (a.k.a. "tier 2"). This currently pessimizes, so don't use yet -- this is infrastructure so we can experiment with optimizing passes. To enable it, pass ``-Xuops`` or set ``PYTHONUOPS=1``. To get debug output, set ``PYTHONUOPSDEBUG=N`` where ``N`` is a debug level (0-4, where 0 is no debug output and 4 is excessively verbose). All of this code is likely to change dramatically before the 3.13 feature freeze. But this is a first step.
2023-06-26 19:02:57 -07:00
}
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
awaitable = PyStackRef_FromPyObjectSteal(awaitable_o);
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_tos_cache1 = awaitable;
_tos_cache0 = aiter;
_tos_cache2 = PyStackRef_ZERO_BITS;
SET_CURRENT_CACHED_VALUES(2);
stack_pointer += -1;
GH-141794: Limit size of generated machine code. (GH-142228) * Factor out bodies of the largest uops, to reduce jit code size. * Factor out common assert, also reducing jit code size. * Limit size of jitted code for a single executor to 1MB.
2025-12-03 17:43:35 +00:00
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-104584: Baby steps towards generating and executing traces (#105924) Added a new, experimental, tracing optimizer and interpreter (a.k.a. "tier 2"). This currently pessimizes, so don't use yet -- this is infrastructure so we can experiment with optimizing passes. To enable it, pass ``-Xuops`` or set ``PYTHONUOPS=1``. To get debug output, set ``PYTHONUOPSDEBUG=N`` where ``N`` is a debug level (0-4, where 0 is no debug output and 4 is excessively verbose). All of this code is likely to change dramatically before the 3.13 feature freeze. But this is a first step.
2023-06-26 19:02:57 -07:00
break;
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
case _GET_AWAITABLE_r11: {
CHECK_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
_PyStackRef iterable;
_PyStackRef iter;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef _stack_item_0 = _tos_cache0;
GH-111485: Factor out tier 2 code generation from the rest of the interpreter code generator (GH-112968)
2023-12-12 12:12:17 +00:00
oparg = CURRENT_OPARG();
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
iterable = _stack_item_0;
stack_pointer[0] = iterable;
stack_pointer += 1;
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
GH-119866: Spill the stack around escaping calls. (GH-124392) * Spill the evaluation around escaping calls in the generated interpreter and JIT. * The code generator tracks live, cached values so they can be saved to memory when needed. * Spills the stack pointer around escaping calls, so that the exact stack is visible to the cycle GC.
2024-10-07 14:56:39 +01:00
_PyFrame_SetStackPointer(frame, stack_pointer);
GH-117224: Move the body of a few large-ish micro-ops into helper functions (GH-122601)
2024-08-02 16:31:17 +01:00
PyObject *iter_o = _PyEval_GetAwaitable(PyStackRef_AsPyObjectBorrow(iterable), oparg);
GH-119866: Spill the stack around escaping calls. (GH-124392) * Spill the evaluation around escaping calls in the generated interpreter and JIT. * The code generator tracks live, cached values so they can be saved to memory when needed. * Spills the stack pointer around escaping calls, so that the exact stack is visible to the cycle GC.
2024-10-07 14:56:39 +01:00
stack_pointer = _PyFrame_GetStackPointer(frame);
GH-128682: Account for escapes in `DECREF_INPUTS` (GH-129953) * Handle escapes in DECREF_INPUTS * Mark a few more functions as escaping * Replace DECREF_INPUTS with PyStackRef_CLOSE where possible
2025-02-12 17:44:59 +00:00
stack_pointer += -1;
GH-141794: Limit size of generated machine code. (GH-142228) * Factor out bodies of the largest uops, to reduce jit code size. * Factor out common assert, also reducing jit code size. * Limit size of jitted code for a single executor to 1MB.
2025-12-03 17:43:35 +00:00
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
GH-128682: Account for escapes in `DECREF_INPUTS` (GH-129953) * Handle escapes in DECREF_INPUTS * Mark a few more functions as escaping * Replace DECREF_INPUTS with PyStackRef_CLOSE where possible
2025-02-12 17:44:59 +00:00
_PyFrame_SetStackPointer(frame, stack_pointer);
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
PyStackRef_CLOSE(iterable);
GH-128682: Account for escapes in `DECREF_INPUTS` (GH-129953) * Handle escapes in DECREF_INPUTS * Mark a few more functions as escaping * Replace DECREF_INPUTS with PyStackRef_CLOSE where possible
2025-02-12 17:44:59 +00:00
stack_pointer = _PyFrame_GetStackPointer(frame);
GH-128682: Make `PyStackRef_CLOSE` escaping. (GH-129404)
2025-02-03 12:41:32 +00:00
if (iter_o == NULL) {
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(0);
GH-128682: Make `PyStackRef_CLOSE` escaping. (GH-129404)
2025-02-03 12:41:32 +00:00
JUMP_TO_ERROR();
}
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
iter = PyStackRef_FromPyObjectSteal(iter_o);
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_tos_cache0 = iter;
_tos_cache1 = PyStackRef_ZERO_BITS;
_tos_cache2 = PyStackRef_ZERO_BITS;
SET_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-104584: Baby steps towards generating and executing traces (#105924) Added a new, experimental, tracing optimizer and interpreter (a.k.a. "tier 2"). This currently pessimizes, so don't use yet -- this is infrastructure so we can experiment with optimizing passes. To enable it, pass ``-Xuops`` or set ``PYTHONUOPS=1``. To get debug output, set ``PYTHONUOPSDEBUG=N`` where ``N`` is a debug level (0-4, where 0 is no debug output and 4 is excessively verbose). All of this code is likely to change dramatically before the 3.13 feature freeze. But this is a first step.
2023-06-26 19:02:57 -07:00
break;
}
GH-116422: Tier2 hot/cold splitting (GH-116813) Splits the "cold" path, deopts and exits, from the "hot" path, reducing the size of most jitted instructions, at the cost of slower exits.
2024-03-26 09:35:11 +00:00
/* _SEND is not a viable micro-op for tier 2 because it uses the 'this_instr' variable */
GH-111485: Factor out tier 2 code generation from the rest of the interpreter code generator (GH-112968)
2023-12-12 12:12:17 +00:00
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
case _SEND_GEN_FRAME_r22: {
CHECK_CURRENT_CACHED_VALUES(2);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-118093: Add tier two support to several instructions (GH-121884)
2024-07-18 14:24:58 -07:00
_PyStackRef v;
_PyStackRef receiver;
GH-135379: Remove types from stack items in code generator. (GH-135384) * Make casts explicit in the instruction definitions
2025-06-11 15:52:25 +01:00
_PyStackRef gen_frame;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef _stack_item_0 = _tos_cache0;
_PyStackRef _stack_item_1 = _tos_cache1;
GH-118093: Add tier two support to several instructions (GH-121884)
2024-07-18 14:24:58 -07:00
oparg = CURRENT_OPARG();
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
v = _stack_item_1;
receiver = _stack_item_0;
GH-118093: Add tier two support to several instructions (GH-121884)
2024-07-18 14:24:58 -07:00
PyGenObject *gen = (PyGenObject *)PyStackRef_AsPyObjectBorrow(receiver);
if (Py_TYPE(gen) != &PyGen_Type && Py_TYPE(gen) != &PyCoro_Type) {
UOP_STAT_INC(uopcode, miss);
GH-142718: JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. (GH-142762) JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. * Make sure that stack is in correct state at side exits with TOS cached values * Simplify choice of cached items for side exits
2025-12-15 19:01:29 +00:00
_tos_cache1 = v;
_tos_cache0 = receiver;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(2);
GH-118093: Add tier two support to several instructions (GH-121884)
2024-07-18 14:24:58 -07:00
JUMP_TO_JUMP_TARGET();
}
gh-120321: Make gi_frame_state transitions atomic in FT build (gh-142599) This makes generator frame state transitions atomic in the free threading build, which avoids segfaults when trying to execute a generator from multiple threads concurrently. There are still a few operations that aren't thread-safe and may crash if performed concurrently on the same generator/coroutine: * Accessing gi_yieldfrom/cr_await/ag_await * Accessing gi_frame/cr_frame/ag_frame * Async generator operations
2025-12-19 14:10:37 -05:00
if (!gen_try_set_executing((PyGenObject *)gen)) {
GH-118093: Add tier two support to several instructions (GH-121884)
2024-07-18 14:24:58 -07:00
UOP_STAT_INC(uopcode, miss);
GH-142718: JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. (GH-142762) JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. * Make sure that stack is in correct state at side exits with TOS cached values * Simplify choice of cached items for side exits
2025-12-15 19:01:29 +00:00
_tos_cache1 = v;
_tos_cache0 = receiver;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(2);
GH-118093: Add tier two support to several instructions (GH-121884)
2024-07-18 14:24:58 -07:00
JUMP_TO_JUMP_TARGET();
}
STAT_INC(SEND, hit);
GH-135379: Remove types from stack items in code generator. (GH-135384) * Make casts explicit in the instruction definitions
2025-06-11 15:52:25 +01:00
_PyInterpreterFrame *pushed_frame = &gen->gi_iframe;
_PyFrame_StackPush(pushed_frame, PyStackRef_MakeHeapSafe(v));
GH-118093: Add tier two support to several instructions (GH-121884)
2024-07-18 14:24:58 -07:00
gen->gi_exc_state.previous_item = tstate->exc_info;
tstate->exc_info = &gen->gi_exc_state;
gh-138264: Fix gcc 14 compiler warnings (GH-138265)
2025-09-03 11:26:56 +03:00
assert( 2u + oparg <= UINT16_MAX);
frame->return_offset = (uint16_t)( 2u + oparg);
GH-135379: Remove types from stack items in code generator. (GH-135384) * Make casts explicit in the instruction definitions
2025-06-11 15:52:25 +01:00
pushed_frame->previous = frame;
gen_frame = PyStackRef_Wrap(pushed_frame);
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_tos_cache1 = gen_frame;
_tos_cache0 = receiver;
_tos_cache2 = PyStackRef_ZERO_BITS;
SET_CURRENT_CACHED_VALUES(2);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-118093: Add tier two support to several instructions (GH-121884)
2024-07-18 14:24:58 -07:00
break;
}
GH-111485: Factor out tier 2 code generation from the rest of the interpreter code generator (GH-112968)
2023-12-12 12:12:17 +00:00
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
case _YIELD_VALUE_r11: {
CHECK_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
_PyStackRef retval;
_PyStackRef value;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef _stack_item_0 = _tos_cache0;
GH-118095: Add tier 2 support for YIELD_VALUE (GH-118380)
2024-04-30 11:33:13 +01:00
oparg = CURRENT_OPARG();
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
retval = _stack_item_0;
GH-128563: Add new frame owner type for interpreter entry frames (GH-129078) Add new frame owner type for interpreter entry frames
2025-01-21 10:15:02 +00:00
assert(frame->owner != FRAME_OWNED_BY_INTERPRETER);
GH-118095: Add tier 2 support for YIELD_VALUE (GH-118380)
2024-04-30 11:33:13 +01:00
frame->instr_ptr++;
gh-120834: fix type of *_iframe field in _PyGenObject_HEAD declaration (#120835)
2024-06-24 10:23:38 +01:00
PyGenObject *gen = _PyGen_GetGeneratorFromFrame(frame);
GH-118095: Add tier 2 support for YIELD_VALUE (GH-118380)
2024-04-30 11:33:13 +01:00
assert(FRAME_SUSPENDED_YIELD_FROM == FRAME_SUSPENDED + 1);
assert(oparg == 0 || oparg == 1);
GH-119866: Spill the stack around escaping calls. (GH-124392) * Spill the evaluation around escaping calls in the generated interpreter and JIT. * The code generator tracks live, cached values so they can be saved to memory when needed. * Spills the stack pointer around escaping calls, so that the exact stack is visible to the cycle GC.
2024-10-07 14:56:39 +01:00
_PyStackRef temp = retval;
GH-118095: Add tier 2 support for YIELD_VALUE (GH-118380)
2024-04-30 11:33:13 +01:00
_PyFrame_SetStackPointer(frame, stack_pointer);
tstate->exc_info = gen->gi_exc_state.previous_item;
gen->gi_exc_state.previous_item = NULL;
_Py_LeaveRecursiveCallPy(tstate);
_PyInterpreterFrame *gen_frame = frame;
frame = tstate->current_frame = frame->previous;
gen_frame->previous = NULL;
gh-120321: Make gi_frame_state transitions atomic in FT build (gh-142599) This makes generator frame state transitions atomic in the free threading build, which avoids segfaults when trying to execute a generator from multiple threads concurrently. There are still a few operations that aren't thread-safe and may crash if performed concurrently on the same generator/coroutine: * Accessing gi_yieldfrom/cr_await/ag_await * Accessing gi_frame/cr_frame/ag_frame * Async generator operations
2025-12-19 14:10:37 -05:00
((_PyThreadStateImpl *)tstate)->generator_return_kind = GENERATOR_YIELD;
FT_ATOMIC_STORE_INT8_RELEASE(gen->gi_frame_state, FRAME_SUSPENDED + oparg);
GH-118095: Add tier 2 support for YIELD_VALUE (GH-118380)
2024-04-30 11:33:13 +01:00
assert(INLINE_CACHE_ENTRIES_SEND == INLINE_CACHE_ENTRIES_FOR_ITER);
#if TIER_ONE
gh-118414: Fix assertion in YIELD_VALUE when tracing lines or instrs (#118683)
2024-05-06 21:22:59 -07:00
assert(frame->instr_ptr->op.code == INSTRUMENTED_LINE ||
GH-119866: Spill the stack around escaping calls. (GH-124392) * Spill the evaluation around escaping calls in the generated interpreter and JIT. * The code generator tracks live, cached values so they can be saved to memory when needed. * Spills the stack pointer around escaping calls, so that the exact stack is visible to the cycle GC.
2024-10-07 14:56:39 +01:00
frame->instr_ptr->op.code == INSTRUMENTED_INSTRUCTION ||
_PyOpcode_Deopt[frame->instr_ptr->op.code] == SEND ||
_PyOpcode_Deopt[frame->instr_ptr->op.code] == FOR_ITER ||
_PyOpcode_Deopt[frame->instr_ptr->op.code] == INTERPRETER_EXIT ||
_PyOpcode_Deopt[frame->instr_ptr->op.code] == ENTER_EXECUTOR);
GH-118095: Add tier 2 support for YIELD_VALUE (GH-118380)
2024-04-30 11:33:13 +01:00
#endif
GH-119866: Spill the stack around escaping calls. (GH-124392) * Spill the evaluation around escaping calls in the generated interpreter and JIT. * The code generator tracks live, cached values so they can be saved to memory when needed. * Spills the stack pointer around escaping calls, so that the exact stack is visible to the cycle GC.
2024-10-07 14:56:39 +01:00
stack_pointer = _PyFrame_GetStackPointer(frame);
GH-118095: Add tier 2 support for YIELD_VALUE (GH-118380)
2024-04-30 11:33:13 +01:00
LOAD_IP(1 + INLINE_CACHE_ENTRIES_SEND);
gh-130704: Strength reduce `LOAD_FAST{_LOAD_FAST}` (#130708) Optimize `LOAD_FAST` opcodes into faster versions that load borrowed references onto the operand stack when we can prove that the lifetime of the local outlives the lifetime of the temporary that is loaded onto the stack.
2025-04-01 10:18:42 -07:00
value = PyStackRef_MakeHeapSafe(temp);
GH-118095: Add tier 2 support for YIELD_VALUE (GH-118380)
2024-04-30 11:33:13 +01:00
LLTRACE_RESUME_FRAME();
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_tos_cache0 = value;
_tos_cache1 = PyStackRef_ZERO_BITS;
_tos_cache2 = PyStackRef_ZERO_BITS;
SET_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-118095: Add tier 2 support for YIELD_VALUE (GH-118380)
2024-04-30 11:33:13 +01:00
break;
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
case _POP_EXCEPT_r10: {
CHECK_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
_PyStackRef exc_value;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef _stack_item_0 = _tos_cache0;
exc_value = _stack_item_0;
gh-104584: Baby steps towards generating and executing traces (#105924) Added a new, experimental, tracing optimizer and interpreter (a.k.a. "tier 2"). This currently pessimizes, so don't use yet -- this is infrastructure so we can experiment with optimizing passes. To enable it, pass ``-Xuops`` or set ``PYTHONUOPS=1``. To get debug output, set ``PYTHONUOPSDEBUG=N`` where ``N`` is a debug level (0-4, where 0 is no debug output and 4 is excessively verbose). All of this code is likely to change dramatically before the 3.13 feature freeze. But this is a first step.
2023-06-26 19:02:57 -07:00
_PyErr_StackItem *exc_info = tstate->exc_info;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
stack_pointer[0] = exc_value;
stack_pointer += 1;
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
GH-119866: Spill the stack around escaping calls. (GH-124392) * Spill the evaluation around escaping calls in the generated interpreter and JIT. * The code generator tracks live, cached values so they can be saved to memory when needed. * Spills the stack pointer around escaping calls, so that the exact stack is visible to the cycle GC.
2024-10-07 14:56:39 +01:00
_PyFrame_SetStackPointer(frame, stack_pointer);
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
Py_XSETREF(exc_info->exc_value,
gh-127022: Simplify `PyStackRef_FromPyObjectSteal` (#127024) This gets rid of the immortal check in `PyStackRef_FromPyObjectSteal()`. Overall, this improves performance about 2% in the free threading build. This also renames `PyStackRef_Is()` to `PyStackRef_IsExactly()` because the macro requires that the tag bits of the arguments match, which is only true in certain special cases.
2024-11-22 17:55:33 +00:00
PyStackRef_IsNone(exc_value)
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
? NULL : PyStackRef_AsPyObjectSteal(exc_value));
GH-119866: Spill the stack around escaping calls. (GH-124392) * Spill the evaluation around escaping calls in the generated interpreter and JIT. * The code generator tracks live, cached values so they can be saved to memory when needed. * Spills the stack pointer around escaping calls, so that the exact stack is visible to the cycle GC.
2024-10-07 14:56:39 +01:00
stack_pointer = _PyFrame_GetStackPointer(frame);
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_tos_cache0 = PyStackRef_ZERO_BITS;
_tos_cache1 = PyStackRef_ZERO_BITS;
_tos_cache2 = PyStackRef_ZERO_BITS;
SET_CURRENT_CACHED_VALUES(0);
GH-111485: Factor out tier 2 code generation from the rest of the interpreter code generator (GH-112968)
2023-12-12 12:12:17 +00:00
stack_pointer += -1;
GH-141794: Limit size of generated machine code. (GH-142228) * Factor out bodies of the largest uops, to reduce jit code size. * Factor out common assert, also reducing jit code size. * Limit size of jitted code for a single executor to 1MB.
2025-12-03 17:43:35 +00:00
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _LOAD_COMMON_CONSTANT_r01: {
CHECK_CURRENT_CACHED_VALUES(0);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef value;
oparg = CURRENT_OPARG();
assert(oparg < NUM_COMMON_CONSTANTS);
value = PyStackRef_FromPyObjectNew(tstate->interp->common_consts[oparg]);
_tos_cache0 = value;
SET_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-104584: Baby steps towards generating and executing traces (#105924) Added a new, experimental, tracing optimizer and interpreter (a.k.a. "tier 2"). This currently pessimizes, so don't use yet -- this is infrastructure so we can experiment with optimizing passes. To enable it, pass ``-Xuops`` or set ``PYTHONUOPS=1``. To get debug output, set ``PYTHONUOPSDEBUG=N`` where ``N`` is a debug level (0-4, where 0 is no debug output and 4 is excessively verbose). All of this code is likely to change dramatically before the 3.13 feature freeze. But this is a first step.
2023-06-26 19:02:57 -07:00
break;
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
case _LOAD_COMMON_CONSTANT_r12: {
CHECK_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
_PyStackRef value;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef _stack_item_0 = _tos_cache0;
gh-119180: Add LOAD_COMMON_CONSTANT opcode (#119321) The PEP 649 implementation will require a way to load NotImplementedError from the bytecode. @markshannon suggested implementing this by converting LOAD_ASSERTION_ERROR into a more general mechanism for loading constants. This PR adds this new opcode. I will work on the rest of the implementation of the PEP separately. Co-authored-by: Irit Katriel <1055913+iritkatriel@users.noreply.github.com>
2024-05-21 20:46:39 -04:00
oparg = CURRENT_OPARG();
gh-131738: optimize builtin any/all/tuple calls with a generator expression arg (#131737)
2025-03-28 10:35:20 +00:00
assert(oparg < NUM_COMMON_CONSTANTS);
value = PyStackRef_FromPyObjectNew(tstate->interp->common_consts[oparg]);
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_tos_cache1 = value;
_tos_cache0 = _stack_item_0;
SET_CURRENT_CACHED_VALUES(2);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _LOAD_COMMON_CONSTANT_r23: {
CHECK_CURRENT_CACHED_VALUES(2);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef value;
_PyStackRef _stack_item_0 = _tos_cache0;
_PyStackRef _stack_item_1 = _tos_cache1;
oparg = CURRENT_OPARG();
assert(oparg < NUM_COMMON_CONSTANTS);
value = PyStackRef_FromPyObjectNew(tstate->interp->common_consts[oparg]);
_tos_cache2 = value;
_tos_cache1 = _stack_item_1;
_tos_cache0 = _stack_item_0;
SET_CURRENT_CACHED_VALUES(3);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-104584: Baby steps towards generating and executing traces (#105924) Added a new, experimental, tracing optimizer and interpreter (a.k.a. "tier 2"). This currently pessimizes, so don't use yet -- this is infrastructure so we can experiment with optimizing passes. To enable it, pass ``-Xuops`` or set ``PYTHONUOPS=1``. To get debug output, set ``PYTHONUOPSDEBUG=N`` where ``N`` is a debug level (0-4, where 0 is no debug output and 4 is excessively verbose). All of this code is likely to change dramatically before the 3.13 feature freeze. But this is a first step.
2023-06-26 19:02:57 -07:00
break;
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
case _LOAD_BUILD_CLASS_r01: {
CHECK_CURRENT_CACHED_VALUES(0);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
_PyStackRef bc;
GH-139922: Tail calling for MSVC (VS 2026) (GH-143068) Co-authored-by: Ken Jin <28750310+Fidget-Spinner@users.noreply.github.com> Co-authored-by: Brandt Bucher <brandt@python.org> Co-authored-by: Hugo van Kemenade <1324225+hugovk@users.noreply.github.com>
2025-12-23 00:01:34 +01:00
int err;
GH-119866: Spill the stack around escaping calls. (GH-124392) * Spill the evaluation around escaping calls in the generated interpreter and JIT. * The code generator tracks live, cached values so they can be saved to memory when needed. * Spills the stack pointer around escaping calls, so that the exact stack is visible to the cycle GC.
2024-10-07 14:56:39 +01:00
_PyFrame_SetStackPointer(frame, stack_pointer);
GH-139922: Tail calling for MSVC (VS 2026) (GH-143068) Co-authored-by: Ken Jin <28750310+Fidget-Spinner@users.noreply.github.com> Co-authored-by: Brandt Bucher <brandt@python.org> Co-authored-by: Hugo van Kemenade <1324225+hugovk@users.noreply.github.com>
2025-12-23 00:01:34 +01:00
PyObject *bc_o = _PyMapping_GetOptionalItem2(BUILTINS(), &_Py_ID(__build_class__), &err);
GH-119866: Spill the stack around escaping calls. (GH-124392) * Spill the evaluation around escaping calls in the generated interpreter and JIT. * The code generator tracks live, cached values so they can be saved to memory when needed. * Spills the stack pointer around escaping calls, so that the exact stack is visible to the cycle GC.
2024-10-07 14:56:39 +01:00
stack_pointer = _PyFrame_GetStackPointer(frame);
GH-128682: Make `PyStackRef_CLOSE` escaping. (GH-129404)
2025-02-03 12:41:32 +00:00
if (err < 0) {
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(0);
GH-128682: Make `PyStackRef_CLOSE` escaping. (GH-129404)
2025-02-03 12:41:32 +00:00
JUMP_TO_ERROR();
}
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
if (bc_o == NULL) {
GH-119866: Spill the stack around escaping calls. (GH-124392) * Spill the evaluation around escaping calls in the generated interpreter and JIT. * The code generator tracks live, cached values so they can be saved to memory when needed. * Spills the stack pointer around escaping calls, so that the exact stack is visible to the cycle GC.
2024-10-07 14:56:39 +01:00
_PyFrame_SetStackPointer(frame, stack_pointer);
gh-106307: C API: Add PyMapping_GetOptionalItem() function (GH-106308) Also add PyMapping_GetOptionalItemString() function.
2023-07-11 23:04:12 +03:00
_PyErr_SetString(tstate, PyExc_NameError,
"__build_class__ not found");
GH-119866: Spill the stack around escaping calls. (GH-124392) * Spill the evaluation around escaping calls in the generated interpreter and JIT. * The code generator tracks live, cached values so they can be saved to memory when needed. * Spills the stack pointer around escaping calls, so that the exact stack is visible to the cycle GC.
2024-10-07 14:56:39 +01:00
stack_pointer = _PyFrame_GetStackPointer(frame);
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(0);
GH-128682: Make `PyStackRef_CLOSE` escaping. (GH-129404)
2025-02-03 12:41:32 +00:00
JUMP_TO_ERROR();
gh-104584: Baby steps towards generating and executing traces (#105924) Added a new, experimental, tracing optimizer and interpreter (a.k.a. "tier 2"). This currently pessimizes, so don't use yet -- this is infrastructure so we can experiment with optimizing passes. To enable it, pass ``-Xuops`` or set ``PYTHONUOPS=1``. To get debug output, set ``PYTHONUOPSDEBUG=N`` where ``N`` is a debug level (0-4, where 0 is no debug output and 4 is excessively verbose). All of this code is likely to change dramatically before the 3.13 feature freeze. But this is a first step.
2023-06-26 19:02:57 -07:00
}
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
bc = PyStackRef_FromPyObjectSteal(bc_o);
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_tos_cache0 = bc;
_tos_cache1 = PyStackRef_ZERO_BITS;
_tos_cache2 = PyStackRef_ZERO_BITS;
SET_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-104584: Baby steps towards generating and executing traces (#105924) Added a new, experimental, tracing optimizer and interpreter (a.k.a. "tier 2"). This currently pessimizes, so don't use yet -- this is infrastructure so we can experiment with optimizing passes. To enable it, pass ``-Xuops`` or set ``PYTHONUOPS=1``. To get debug output, set ``PYTHONUOPSDEBUG=N`` where ``N`` is a debug level (0-4, where 0 is no debug output and 4 is excessively verbose). All of this code is likely to change dramatically before the 3.13 feature freeze. But this is a first step.
2023-06-26 19:02:57 -07:00
break;
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
case _STORE_NAME_r10: {
CHECK_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
_PyStackRef v;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef _stack_item_0 = _tos_cache0;
GH-111485: Factor out tier 2 code generation from the rest of the interpreter code generator (GH-112968)
2023-12-12 12:12:17 +00:00
oparg = CURRENT_OPARG();
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
v = _stack_item_0;
gh-104584: Baby steps towards generating and executing traces (#105924) Added a new, experimental, tracing optimizer and interpreter (a.k.a. "tier 2"). This currently pessimizes, so don't use yet -- this is infrastructure so we can experiment with optimizing passes. To enable it, pass ``-Xuops`` or set ``PYTHONUOPS=1``. To get debug output, set ``PYTHONUOPSDEBUG=N`` where ``N`` is a debug level (0-4, where 0 is no debug output and 4 is excessively verbose). All of this code is likely to change dramatically before the 3.13 feature freeze. But this is a first step.
2023-06-26 19:02:57 -07:00
PyObject *name = GETITEM(FRAME_CO_NAMES, oparg);
PyObject *ns = LOCALS();
int err;
if (ns == NULL) {
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
stack_pointer[0] = v;
stack_pointer += 1;
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
GH-119866: Spill the stack around escaping calls. (GH-124392) * Spill the evaluation around escaping calls in the generated interpreter and JIT. * The code generator tracks live, cached values so they can be saved to memory when needed. * Spills the stack pointer around escaping calls, so that the exact stack is visible to the cycle GC.
2024-10-07 14:56:39 +01:00
_PyFrame_SetStackPointer(frame, stack_pointer);
gh-104584: Baby steps towards generating and executing traces (#105924) Added a new, experimental, tracing optimizer and interpreter (a.k.a. "tier 2"). This currently pessimizes, so don't use yet -- this is infrastructure so we can experiment with optimizing passes. To enable it, pass ``-Xuops`` or set ``PYTHONUOPS=1``. To get debug output, set ``PYTHONUOPSDEBUG=N`` where ``N`` is a debug level (0-4, where 0 is no debug output and 4 is excessively verbose). All of this code is likely to change dramatically before the 3.13 feature freeze. But this is a first step.
2023-06-26 19:02:57 -07:00
_PyErr_Format(tstate, PyExc_SystemError,
"no locals found when storing %R", name);
GH-119866: Spill the stack around escaping calls. (GH-124392) * Spill the evaluation around escaping calls in the generated interpreter and JIT. * The code generator tracks live, cached values so they can be saved to memory when needed. * Spills the stack pointer around escaping calls, so that the exact stack is visible to the cycle GC.
2024-10-07 14:56:39 +01:00
stack_pointer = _PyFrame_GetStackPointer(frame);
GH-128682: Make `PyStackRef_CLOSE` escaping. (GH-129404)
2025-02-03 12:41:32 +00:00
stack_pointer += -1;
GH-141794: Limit size of generated machine code. (GH-142228) * Factor out bodies of the largest uops, to reduce jit code size. * Factor out common assert, also reducing jit code size. * Limit size of jitted code for a single executor to 1MB.
2025-12-03 17:43:35 +00:00
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
GH-128682: Account for escapes in `DECREF_INPUTS` (GH-129953) * Handle escapes in DECREF_INPUTS * Mark a few more functions as escaping * Replace DECREF_INPUTS with PyStackRef_CLOSE where possible
2025-02-12 17:44:59 +00:00
_PyFrame_SetStackPointer(frame, stack_pointer);
PyStackRef_CLOSE(v);
stack_pointer = _PyFrame_GetStackPointer(frame);
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(0);
GH-128682: Make `PyStackRef_CLOSE` escaping. (GH-129404)
2025-02-03 12:41:32 +00:00
JUMP_TO_ERROR();
gh-104584: Baby steps towards generating and executing traces (#105924) Added a new, experimental, tracing optimizer and interpreter (a.k.a. "tier 2"). This currently pessimizes, so don't use yet -- this is infrastructure so we can experiment with optimizing passes. To enable it, pass ``-Xuops`` or set ``PYTHONUOPS=1``. To get debug output, set ``PYTHONUOPSDEBUG=N`` where ``N`` is a debug level (0-4, where 0 is no debug output and 4 is excessively verbose). All of this code is likely to change dramatically before the 3.13 feature freeze. But this is a first step.
2023-06-26 19:02:57 -07:00
}
GH-119866: Spill the stack around escaping calls. (GH-124392) * Spill the evaluation around escaping calls in the generated interpreter and JIT. * The code generator tracks live, cached values so they can be saved to memory when needed. * Spills the stack pointer around escaping calls, so that the exact stack is visible to the cycle GC.
2024-10-07 14:56:39 +01:00
if (PyDict_CheckExact(ns)) {
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
stack_pointer[0] = v;
stack_pointer += 1;
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
GH-119866: Spill the stack around escaping calls. (GH-124392) * Spill the evaluation around escaping calls in the generated interpreter and JIT. * The code generator tracks live, cached values so they can be saved to memory when needed. * Spills the stack pointer around escaping calls, so that the exact stack is visible to the cycle GC.
2024-10-07 14:56:39 +01:00
_PyFrame_SetStackPointer(frame, stack_pointer);
err = PyDict_SetItem(ns, name, PyStackRef_AsPyObjectBorrow(v));
stack_pointer = _PyFrame_GetStackPointer(frame);
}
else {
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
stack_pointer[0] = v;
stack_pointer += 1;
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
GH-119866: Spill the stack around escaping calls. (GH-124392) * Spill the evaluation around escaping calls in the generated interpreter and JIT. * The code generator tracks live, cached values so they can be saved to memory when needed. * Spills the stack pointer around escaping calls, so that the exact stack is visible to the cycle GC.
2024-10-07 14:56:39 +01:00
_PyFrame_SetStackPointer(frame, stack_pointer);
err = PyObject_SetItem(ns, name, PyStackRef_AsPyObjectBorrow(v));
stack_pointer = _PyFrame_GetStackPointer(frame);
}
GH-128682: Account for escapes in `DECREF_INPUTS` (GH-129953) * Handle escapes in DECREF_INPUTS * Mark a few more functions as escaping * Replace DECREF_INPUTS with PyStackRef_CLOSE where possible
2025-02-12 17:44:59 +00:00
stack_pointer += -1;
GH-141794: Limit size of generated machine code. (GH-142228) * Factor out bodies of the largest uops, to reduce jit code size. * Factor out common assert, also reducing jit code size. * Limit size of jitted code for a single executor to 1MB.
2025-12-03 17:43:35 +00:00
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
GH-128682: Account for escapes in `DECREF_INPUTS` (GH-129953) * Handle escapes in DECREF_INPUTS * Mark a few more functions as escaping * Replace DECREF_INPUTS with PyStackRef_CLOSE where possible
2025-02-12 17:44:59 +00:00
_PyFrame_SetStackPointer(frame, stack_pointer);
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
PyStackRef_CLOSE(v);
GH-128682: Account for escapes in `DECREF_INPUTS` (GH-129953) * Handle escapes in DECREF_INPUTS * Mark a few more functions as escaping * Replace DECREF_INPUTS with PyStackRef_CLOSE where possible
2025-02-12 17:44:59 +00:00
stack_pointer = _PyFrame_GetStackPointer(frame);
GH-128682: Make `PyStackRef_CLOSE` escaping. (GH-129404)
2025-02-03 12:41:32 +00:00
if (err) {
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(0);
GH-128682: Make `PyStackRef_CLOSE` escaping. (GH-129404)
2025-02-03 12:41:32 +00:00
JUMP_TO_ERROR();
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_tos_cache0 = PyStackRef_ZERO_BITS;
_tos_cache1 = PyStackRef_ZERO_BITS;
_tos_cache2 = PyStackRef_ZERO_BITS;
SET_CURRENT_CACHED_VALUES(0);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-104584: Baby steps towards generating and executing traces (#105924) Added a new, experimental, tracing optimizer and interpreter (a.k.a. "tier 2"). This currently pessimizes, so don't use yet -- this is infrastructure so we can experiment with optimizing passes. To enable it, pass ``-Xuops`` or set ``PYTHONUOPS=1``. To get debug output, set ``PYTHONUOPSDEBUG=N`` where ``N`` is a debug level (0-4, where 0 is no debug output and 4 is excessively verbose). All of this code is likely to change dramatically before the 3.13 feature freeze. But this is a first step.
2023-06-26 19:02:57 -07:00
break;
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
case _DELETE_NAME_r00: {
CHECK_CURRENT_CACHED_VALUES(0);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-112287: Speed up Tier 2 (uop) interpreter a little (#112286) This makes the Tier 2 interpreter a little faster. I calculated by about 3%, though I hesitate to claim an exact number. This starts by doubling the trace size limit (to 512), making it more likely that loops fit in a trace. The rest of the approach is to only load `oparg` and `operand` in cases that use them. The code generator know when these are used. For `oparg`, it will conditionally emit ``` oparg = CURRENT_OPARG(); ``` at the top of the case block. (The `oparg` variable may be referenced multiple times by the instructions code block, so it must be in a variable.) For `operand`, it will use `CURRENT_OPERAND()` directly instead of referencing the `operand` variable, which no longer exists. (There is only one place where this will be used.)
2023-11-20 11:25:32 -08:00
oparg = CURRENT_OPARG();
gh-104584: Baby steps towards generating and executing traces (#105924) Added a new, experimental, tracing optimizer and interpreter (a.k.a. "tier 2"). This currently pessimizes, so don't use yet -- this is infrastructure so we can experiment with optimizing passes. To enable it, pass ``-Xuops`` or set ``PYTHONUOPS=1``. To get debug output, set ``PYTHONUOPSDEBUG=N`` where ``N`` is a debug level (0-4, where 0 is no debug output and 4 is excessively verbose). All of this code is likely to change dramatically before the 3.13 feature freeze. But this is a first step.
2023-06-26 19:02:57 -07:00
PyObject *name = GETITEM(FRAME_CO_NAMES, oparg);
PyObject *ns = LOCALS();
int err;
if (ns == NULL) {
GH-119866: Spill the stack around escaping calls. (GH-124392) * Spill the evaluation around escaping calls in the generated interpreter and JIT. * The code generator tracks live, cached values so they can be saved to memory when needed. * Spills the stack pointer around escaping calls, so that the exact stack is visible to the cycle GC.
2024-10-07 14:56:39 +01:00
_PyFrame_SetStackPointer(frame, stack_pointer);
gh-104584: Baby steps towards generating and executing traces (#105924) Added a new, experimental, tracing optimizer and interpreter (a.k.a. "tier 2"). This currently pessimizes, so don't use yet -- this is infrastructure so we can experiment with optimizing passes. To enable it, pass ``-Xuops`` or set ``PYTHONUOPS=1``. To get debug output, set ``PYTHONUOPSDEBUG=N`` where ``N`` is a debug level (0-4, where 0 is no debug output and 4 is excessively verbose). All of this code is likely to change dramatically before the 3.13 feature freeze. But this is a first step.
2023-06-26 19:02:57 -07:00
_PyErr_Format(tstate, PyExc_SystemError,
"no locals when deleting %R", name);
GH-119866: Spill the stack around escaping calls. (GH-124392) * Spill the evaluation around escaping calls in the generated interpreter and JIT. * The code generator tracks live, cached values so they can be saved to memory when needed. * Spills the stack pointer around escaping calls, so that the exact stack is visible to the cycle GC.
2024-10-07 14:56:39 +01:00
stack_pointer = _PyFrame_GetStackPointer(frame);
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(0);
GH-116422: Tier2 hot/cold splitting (GH-116813) Splits the "cold" path, deopts and exits, from the "hot" path, reducing the size of most jitted instructions, at the cost of slower exits.
2024-03-26 09:35:11 +00:00
JUMP_TO_ERROR();
gh-104584: Baby steps towards generating and executing traces (#105924) Added a new, experimental, tracing optimizer and interpreter (a.k.a. "tier 2"). This currently pessimizes, so don't use yet -- this is infrastructure so we can experiment with optimizing passes. To enable it, pass ``-Xuops`` or set ``PYTHONUOPS=1``. To get debug output, set ``PYTHONUOPSDEBUG=N`` where ``N`` is a debug level (0-4, where 0 is no debug output and 4 is excessively verbose). All of this code is likely to change dramatically before the 3.13 feature freeze. But this is a first step.
2023-06-26 19:02:57 -07:00
}
GH-119866: Spill the stack around escaping calls. (GH-124392) * Spill the evaluation around escaping calls in the generated interpreter and JIT. * The code generator tracks live, cached values so they can be saved to memory when needed. * Spills the stack pointer around escaping calls, so that the exact stack is visible to the cycle GC.
2024-10-07 14:56:39 +01:00
_PyFrame_SetStackPointer(frame, stack_pointer);
gh-104584: Baby steps towards generating and executing traces (#105924) Added a new, experimental, tracing optimizer and interpreter (a.k.a. "tier 2"). This currently pessimizes, so don't use yet -- this is infrastructure so we can experiment with optimizing passes. To enable it, pass ``-Xuops`` or set ``PYTHONUOPS=1``. To get debug output, set ``PYTHONUOPSDEBUG=N`` where ``N`` is a debug level (0-4, where 0 is no debug output and 4 is excessively verbose). All of this code is likely to change dramatically before the 3.13 feature freeze. But this is a first step.
2023-06-26 19:02:57 -07:00
err = PyObject_DelItem(ns, name);
GH-119866: Spill the stack around escaping calls. (GH-124392) * Spill the evaluation around escaping calls in the generated interpreter and JIT. * The code generator tracks live, cached values so they can be saved to memory when needed. * Spills the stack pointer around escaping calls, so that the exact stack is visible to the cycle GC.
2024-10-07 14:56:39 +01:00
stack_pointer = _PyFrame_GetStackPointer(frame);
gh-104584: Baby steps towards generating and executing traces (#105924) Added a new, experimental, tracing optimizer and interpreter (a.k.a. "tier 2"). This currently pessimizes, so don't use yet -- this is infrastructure so we can experiment with optimizing passes. To enable it, pass ``-Xuops`` or set ``PYTHONUOPS=1``. To get debug output, set ``PYTHONUOPSDEBUG=N`` where ``N`` is a debug level (0-4, where 0 is no debug output and 4 is excessively verbose). All of this code is likely to change dramatically before the 3.13 feature freeze. But this is a first step.
2023-06-26 19:02:57 -07:00
if (err != 0) {
GH-119866: Spill the stack around escaping calls. (GH-124392) * Spill the evaluation around escaping calls in the generated interpreter and JIT. * The code generator tracks live, cached values so they can be saved to memory when needed. * Spills the stack pointer around escaping calls, so that the exact stack is visible to the cycle GC.
2024-10-07 14:56:39 +01:00
_PyFrame_SetStackPointer(frame, stack_pointer);
GH-106701: Move _PyUopExecute to Python/executor.c (GH-106924)
2023-07-20 13:37:19 -07:00
_PyEval_FormatExcCheckArg(tstate, PyExc_NameError,
GH-111485: Factor out tier 2 code generation from the rest of the interpreter code generator (GH-112968)
2023-12-12 12:12:17 +00:00
NAME_ERROR_MSG,
name);
GH-119866: Spill the stack around escaping calls. (GH-124392) * Spill the evaluation around escaping calls in the generated interpreter and JIT. * The code generator tracks live, cached values so they can be saved to memory when needed. * Spills the stack pointer around escaping calls, so that the exact stack is visible to the cycle GC.
2024-10-07 14:56:39 +01:00
stack_pointer = _PyFrame_GetStackPointer(frame);
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(0);
GH-116422: Tier2 hot/cold splitting (GH-116813) Splits the "cold" path, deopts and exits, from the "hot" path, reducing the size of most jitted instructions, at the cost of slower exits.
2024-03-26 09:35:11 +00:00
JUMP_TO_ERROR();
gh-104584: Baby steps towards generating and executing traces (#105924) Added a new, experimental, tracing optimizer and interpreter (a.k.a. "tier 2"). This currently pessimizes, so don't use yet -- this is infrastructure so we can experiment with optimizing passes. To enable it, pass ``-Xuops`` or set ``PYTHONUOPS=1``. To get debug output, set ``PYTHONUOPSDEBUG=N`` where ``N`` is a debug level (0-4, where 0 is no debug output and 4 is excessively verbose). All of this code is likely to change dramatically before the 3.13 feature freeze. But this is a first step.
2023-06-26 19:02:57 -07:00
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_tos_cache0 = PyStackRef_ZERO_BITS;
_tos_cache1 = PyStackRef_ZERO_BITS;
_tos_cache2 = PyStackRef_ZERO_BITS;
SET_CURRENT_CACHED_VALUES(0);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-104584: Allow unspecialized instructions in superblocks (#106497) This adds several of unspecialized opcodes to superblocks: TO_BOOL, BINARY_SUBSCR, STORE_SUBSCR, UNPACK_SEQUENCE, LOAD_GLOBAL, LOAD_ATTR, COMPARE_OP, BINARY_OP. While we may not want that eventually, for now this helps finding bugs. There is a rudimentary test checking for UNPACK_SEQUENCE. Once we're ready to undo this, that would be simple: just replace the call to variable_used_unspecialized with a call to variable_used (as shown in a comment). Or add individual opcdes to FORBIDDEN_NAMES_IN_UOPS.
2023-07-07 11:03:27 -07:00
break;
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
case _UNPACK_SEQUENCE_r10: {
CHECK_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
_PyStackRef seq;
GH-128682: Account for escapes in `DECREF_INPUTS` (GH-129953) * Handle escapes in DECREF_INPUTS * Mark a few more functions as escaping * Replace DECREF_INPUTS with PyStackRef_CLOSE where possible
2025-02-12 17:44:59 +00:00
_PyStackRef *top;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef _stack_item_0 = _tos_cache0;
GH-111485: Factor out tier 2 code generation from the rest of the interpreter code generator (GH-112968)
2023-12-12 12:12:17 +00:00
oparg = CURRENT_OPARG();
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
seq = _stack_item_0;
top = &stack_pointer[oparg];
GH-128682: Account for escapes in `DECREF_INPUTS` (GH-129953) * Handle escapes in DECREF_INPUTS * Mark a few more functions as escaping * Replace DECREF_INPUTS with PyStackRef_CLOSE where possible
2025-02-12 17:44:59 +00:00
PyObject *seq_o = PyStackRef_AsPyObjectSteal(seq);
GH-119866: Spill the stack around escaping calls. (GH-124392) * Spill the evaluation around escaping calls in the generated interpreter and JIT. * The code generator tracks live, cached values so they can be saved to memory when needed. * Spills the stack pointer around escaping calls, so that the exact stack is visible to the cycle GC.
2024-10-07 14:56:39 +01:00
_PyFrame_SetStackPointer(frame, stack_pointer);
GH-128682: Account for escapes in `DECREF_INPUTS` (GH-129953) * Handle escapes in DECREF_INPUTS * Mark a few more functions as escaping * Replace DECREF_INPUTS with PyStackRef_CLOSE where possible
2025-02-12 17:44:59 +00:00
int res = _PyEval_UnpackIterableStackRef(tstate, seq_o, oparg, -1, top);
Py_DECREF(seq_o);
GH-119866: Spill the stack around escaping calls. (GH-124392) * Spill the evaluation around escaping calls in the generated interpreter and JIT. * The code generator tracks live, cached values so they can be saved to memory when needed. * Spills the stack pointer around escaping calls, so that the exact stack is visible to the cycle GC.
2024-10-07 14:56:39 +01:00
stack_pointer = _PyFrame_GetStackPointer(frame);
GH-128682: Make `PyStackRef_CLOSE` escaping. (GH-129404)
2025-02-03 12:41:32 +00:00
if (res == 0) {
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(0);
GH-128682: Make `PyStackRef_CLOSE` escaping. (GH-129404)
2025-02-03 12:41:32 +00:00
JUMP_TO_ERROR();
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_tos_cache0 = PyStackRef_ZERO_BITS;
_tos_cache1 = PyStackRef_ZERO_BITS;
_tos_cache2 = PyStackRef_ZERO_BITS;
SET_CURRENT_CACHED_VALUES(0);
GH-128682: Account for escapes in `DECREF_INPUTS` (GH-129953) * Handle escapes in DECREF_INPUTS * Mark a few more functions as escaping * Replace DECREF_INPUTS with PyStackRef_CLOSE where possible
2025-02-12 17:44:59 +00:00
stack_pointer += oparg;
GH-141794: Limit size of generated machine code. (GH-142228) * Factor out bodies of the largest uops, to reduce jit code size. * Factor out common assert, also reducing jit code size. * Limit size of jitted code for a single executor to 1MB.
2025-12-03 17:43:35 +00:00
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-104584: Baby steps towards generating and executing traces (#105924) Added a new, experimental, tracing optimizer and interpreter (a.k.a. "tier 2"). This currently pessimizes, so don't use yet -- this is infrastructure so we can experiment with optimizing passes. To enable it, pass ``-Xuops`` or set ``PYTHONUOPS=1``. To get debug output, set ``PYTHONUOPSDEBUG=N`` where ``N`` is a debug level (0-4, where 0 is no debug output and 4 is excessively verbose). All of this code is likely to change dramatically before the 3.13 feature freeze. But this is a first step.
2023-06-26 19:02:57 -07:00
break;
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
case _UNPACK_SEQUENCE_TWO_TUPLE_r12: {
CHECK_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
_PyStackRef seq;
_PyStackRef val1;
_PyStackRef val0;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef _stack_item_0 = _tos_cache0;
GH-111485: Factor out tier 2 code generation from the rest of the interpreter code generator (GH-112968)
2023-12-12 12:12:17 +00:00
oparg = CURRENT_OPARG();
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
seq = _stack_item_0;
GH-116468: Use constants instead of `oparg` in stack effects when `oparg` is known to be a constant. (GH-116469)
2024-03-11 09:30:15 +00:00
assert(oparg == 2);
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
PyObject *seq_o = PyStackRef_AsPyObjectBorrow(seq);
GH-131798: Remove JIT guards for dict, frozenset, list, set, and tuple (GH-132289)
2025-04-09 14:32:21 -07:00
assert(PyTuple_CheckExact(seq_o));
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
if (PyTuple_GET_SIZE(seq_o) != 2) {
GH-117457: Correct pystats uop "miss" counts (GH-117477)
2024-04-04 18:49:18 -04:00
UOP_STAT_INC(uopcode, miss);
GH-142718: JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. (GH-142762) JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. * Make sure that stack is in correct state at side exits with TOS cached values * Simplify choice of cached items for side exits
2025-12-15 19:01:29 +00:00
_tos_cache0 = seq;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(1);
GH-117457: Correct pystats uop "miss" counts (GH-117477)
2024-04-04 18:49:18 -04:00
JUMP_TO_JUMP_TARGET();
}
gh-104584: Baby steps towards generating and executing traces (#105924) Added a new, experimental, tracing optimizer and interpreter (a.k.a. "tier 2"). This currently pessimizes, so don't use yet -- this is infrastructure so we can experiment with optimizing passes. To enable it, pass ``-Xuops`` or set ``PYTHONUOPS=1``. To get debug output, set ``PYTHONUOPSDEBUG=N`` where ``N`` is a debug level (0-4, where 0 is no debug output and 4 is excessively verbose). All of this code is likely to change dramatically before the 3.13 feature freeze. But this is a first step.
2023-06-26 19:02:57 -07:00
STAT_INC(UNPACK_SEQUENCE, hit);
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
val0 = PyStackRef_FromPyObjectNew(PyTuple_GET_ITEM(seq_o, 0));
val1 = PyStackRef_FromPyObjectNew(PyTuple_GET_ITEM(seq_o, 1));
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
stack_pointer[0] = val1;
stack_pointer[1] = val0;
stack_pointer += 2;
GH-141794: Limit size of generated machine code. (GH-142228) * Factor out bodies of the largest uops, to reduce jit code size. * Factor out common assert, also reducing jit code size. * Limit size of jitted code for a single executor to 1MB.
2025-12-03 17:43:35 +00:00
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
GH-128682: Account for escapes in `DECREF_INPUTS` (GH-129953) * Handle escapes in DECREF_INPUTS * Mark a few more functions as escaping * Replace DECREF_INPUTS with PyStackRef_CLOSE where possible
2025-02-12 17:44:59 +00:00
_PyFrame_SetStackPointer(frame, stack_pointer);
PyStackRef_CLOSE(seq);
stack_pointer = _PyFrame_GetStackPointer(frame);
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_tos_cache1 = val0;
_tos_cache0 = val1;
_tos_cache2 = PyStackRef_ZERO_BITS;
SET_CURRENT_CACHED_VALUES(2);
stack_pointer += -2;
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-104584: Baby steps towards generating and executing traces (#105924) Added a new, experimental, tracing optimizer and interpreter (a.k.a. "tier 2"). This currently pessimizes, so don't use yet -- this is infrastructure so we can experiment with optimizing passes. To enable it, pass ``-Xuops`` or set ``PYTHONUOPS=1``. To get debug output, set ``PYTHONUOPSDEBUG=N`` where ``N`` is a debug level (0-4, where 0 is no debug output and 4 is excessively verbose). All of this code is likely to change dramatically before the 3.13 feature freeze. But this is a first step.
2023-06-26 19:02:57 -07:00
break;
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
case _UNPACK_SEQUENCE_TUPLE_r10: {
CHECK_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
_PyStackRef seq;
_PyStackRef *values;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef _stack_item_0 = _tos_cache0;
GH-111485: Factor out tier 2 code generation from the rest of the interpreter code generator (GH-112968)
2023-12-12 12:12:17 +00:00
oparg = CURRENT_OPARG();
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
seq = _stack_item_0;
values = &stack_pointer[0];
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
PyObject *seq_o = PyStackRef_AsPyObjectBorrow(seq);
GH-131798: Remove JIT guards for dict, frozenset, list, set, and tuple (GH-132289)
2025-04-09 14:32:21 -07:00
assert(PyTuple_CheckExact(seq_o));
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
if (PyTuple_GET_SIZE(seq_o) != oparg) {
GH-117457: Correct pystats uop "miss" counts (GH-117477)
2024-04-04 18:49:18 -04:00
UOP_STAT_INC(uopcode, miss);
GH-142718: JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. (GH-142762) JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. * Make sure that stack is in correct state at side exits with TOS cached values * Simplify choice of cached items for side exits
2025-12-15 19:01:29 +00:00
_tos_cache0 = seq;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(1);
GH-117457: Correct pystats uop "miss" counts (GH-117477)
2024-04-04 18:49:18 -04:00
JUMP_TO_JUMP_TARGET();
}
gh-104584: Baby steps towards generating and executing traces (#105924) Added a new, experimental, tracing optimizer and interpreter (a.k.a. "tier 2"). This currently pessimizes, so don't use yet -- this is infrastructure so we can experiment with optimizing passes. To enable it, pass ``-Xuops`` or set ``PYTHONUOPS=1``. To get debug output, set ``PYTHONUOPSDEBUG=N`` where ``N`` is a debug level (0-4, where 0 is no debug output and 4 is excessively verbose). All of this code is likely to change dramatically before the 3.13 feature freeze. But this is a first step.
2023-06-26 19:02:57 -07:00
STAT_INC(UNPACK_SEQUENCE, hit);
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
PyObject **items = _PyTuple_ITEMS(seq_o);
gh-104584: Baby steps towards generating and executing traces (#105924) Added a new, experimental, tracing optimizer and interpreter (a.k.a. "tier 2"). This currently pessimizes, so don't use yet -- this is infrastructure so we can experiment with optimizing passes. To enable it, pass ``-Xuops`` or set ``PYTHONUOPS=1``. To get debug output, set ``PYTHONUOPSDEBUG=N`` where ``N`` is a debug level (0-4, where 0 is no debug output and 4 is excessively verbose). All of this code is likely to change dramatically before the 3.13 feature freeze. But this is a first step.
2023-06-26 19:02:57 -07:00
for (int i = oparg; --i >= 0; ) {
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
*values++ = PyStackRef_FromPyObjectNew(items[i]);
gh-104584: Baby steps towards generating and executing traces (#105924) Added a new, experimental, tracing optimizer and interpreter (a.k.a. "tier 2"). This currently pessimizes, so don't use yet -- this is infrastructure so we can experiment with optimizing passes. To enable it, pass ``-Xuops`` or set ``PYTHONUOPS=1``. To get debug output, set ``PYTHONUOPSDEBUG=N`` where ``N`` is a debug level (0-4, where 0 is no debug output and 4 is excessively verbose). All of this code is likely to change dramatically before the 3.13 feature freeze. But this is a first step.
2023-06-26 19:02:57 -07:00
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
stack_pointer += oparg;
GH-141794: Limit size of generated machine code. (GH-142228) * Factor out bodies of the largest uops, to reduce jit code size. * Factor out common assert, also reducing jit code size. * Limit size of jitted code for a single executor to 1MB.
2025-12-03 17:43:35 +00:00
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
GH-128682: Account for escapes in `DECREF_INPUTS` (GH-129953) * Handle escapes in DECREF_INPUTS * Mark a few more functions as escaping * Replace DECREF_INPUTS with PyStackRef_CLOSE where possible
2025-02-12 17:44:59 +00:00
_PyFrame_SetStackPointer(frame, stack_pointer);
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
PyStackRef_CLOSE(seq);
GH-128682: Account for escapes in `DECREF_INPUTS` (GH-129953) * Handle escapes in DECREF_INPUTS * Mark a few more functions as escaping * Replace DECREF_INPUTS with PyStackRef_CLOSE where possible
2025-02-12 17:44:59 +00:00
stack_pointer = _PyFrame_GetStackPointer(frame);
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_tos_cache0 = PyStackRef_ZERO_BITS;
_tos_cache1 = PyStackRef_ZERO_BITS;
_tos_cache2 = PyStackRef_ZERO_BITS;
SET_CURRENT_CACHED_VALUES(0);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-104584: Baby steps towards generating and executing traces (#105924) Added a new, experimental, tracing optimizer and interpreter (a.k.a. "tier 2"). This currently pessimizes, so don't use yet -- this is infrastructure so we can experiment with optimizing passes. To enable it, pass ``-Xuops`` or set ``PYTHONUOPS=1``. To get debug output, set ``PYTHONUOPSDEBUG=N`` where ``N`` is a debug level (0-4, where 0 is no debug output and 4 is excessively verbose). All of this code is likely to change dramatically before the 3.13 feature freeze. But this is a first step.
2023-06-26 19:02:57 -07:00
break;
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
case _UNPACK_SEQUENCE_LIST_r10: {
CHECK_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
_PyStackRef seq;
_PyStackRef *values;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef _stack_item_0 = _tos_cache0;
GH-111485: Factor out tier 2 code generation from the rest of the interpreter code generator (GH-112968)
2023-12-12 12:12:17 +00:00
oparg = CURRENT_OPARG();
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
seq = _stack_item_0;
values = &stack_pointer[0];
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
PyObject *seq_o = PyStackRef_AsPyObjectBorrow(seq);
GH-131798: Remove JIT guards for dict, frozenset, list, set, and tuple (GH-132289)
2025-04-09 14:32:21 -07:00
assert(PyList_CheckExact(seq_o));
gh-115999: Use light-weight lock for UNPACK_SEQUENCE_LIST (gh-127514)
2024-12-03 00:14:40 +09:00
if (!LOCK_OBJECT(seq_o)) {
UOP_STAT_INC(uopcode, miss);
GH-142718: JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. (GH-142762) JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. * Make sure that stack is in correct state at side exits with TOS cached values * Simplify choice of cached items for side exits
2025-12-15 19:01:29 +00:00
_tos_cache0 = seq;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(1);
gh-115999: Use light-weight lock for UNPACK_SEQUENCE_LIST (gh-127514)
2024-12-03 00:14:40 +09:00
JUMP_TO_JUMP_TARGET();
}
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
if (PyList_GET_SIZE(seq_o) != oparg) {
gh-115999: Use light-weight lock for UNPACK_SEQUENCE_LIST (gh-127514)
2024-12-03 00:14:40 +09:00
UNLOCK_OBJECT(seq_o);
gh-115999: Add free-threaded specialization for `UNPACK_SEQUENCE` (#126600) Add free-threaded specialization for `UNPACK_SEQUENCE` opcode. `UNPACK_SEQUENCE_TUPLE/UNPACK_SEQUENCE_TWO_TUPLE` are already thread safe since tuples are immutable. `UNPACK_SEQUENCE_LIST` is not thread safe because of nature of lists (there is nothing preventing another thread from adding items to or removing them the list while the instruction is executing). To achieve thread safety we add a critical section to the implementation of `UNPACK_SEQUENCE_LIST`, especially around the parts where we check the size of the list and push items onto the stack. --------- Co-authored-by: Matt Page <mpage@meta.com> Co-authored-by: mpage <mpage@cs.stanford.edu>
2024-11-22 19:00:35 +02:00
if (true) {
UOP_STAT_INC(uopcode, miss);
GH-142718: JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. (GH-142762) JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. * Make sure that stack is in correct state at side exits with TOS cached values * Simplify choice of cached items for side exits
2025-12-15 19:01:29 +00:00
_tos_cache0 = seq;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(1);
gh-115999: Add free-threaded specialization for `UNPACK_SEQUENCE` (#126600) Add free-threaded specialization for `UNPACK_SEQUENCE` opcode. `UNPACK_SEQUENCE_TUPLE/UNPACK_SEQUENCE_TWO_TUPLE` are already thread safe since tuples are immutable. `UNPACK_SEQUENCE_LIST` is not thread safe because of nature of lists (there is nothing preventing another thread from adding items to or removing them the list while the instruction is executing). To achieve thread safety we add a critical section to the implementation of `UNPACK_SEQUENCE_LIST`, especially around the parts where we check the size of the list and push items onto the stack. --------- Co-authored-by: Matt Page <mpage@meta.com> Co-authored-by: mpage <mpage@cs.stanford.edu>
2024-11-22 19:00:35 +02:00
JUMP_TO_JUMP_TARGET();
}
GH-117457: Correct pystats uop "miss" counts (GH-117477)
2024-04-04 18:49:18 -04:00
}
gh-104584: Baby steps towards generating and executing traces (#105924) Added a new, experimental, tracing optimizer and interpreter (a.k.a. "tier 2"). This currently pessimizes, so don't use yet -- this is infrastructure so we can experiment with optimizing passes. To enable it, pass ``-Xuops`` or set ``PYTHONUOPS=1``. To get debug output, set ``PYTHONUOPSDEBUG=N`` where ``N`` is a debug level (0-4, where 0 is no debug output and 4 is excessively verbose). All of this code is likely to change dramatically before the 3.13 feature freeze. But this is a first step.
2023-06-26 19:02:57 -07:00
STAT_INC(UNPACK_SEQUENCE, hit);
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
PyObject **items = _PyList_ITEMS(seq_o);
gh-104584: Baby steps towards generating and executing traces (#105924) Added a new, experimental, tracing optimizer and interpreter (a.k.a. "tier 2"). This currently pessimizes, so don't use yet -- this is infrastructure so we can experiment with optimizing passes. To enable it, pass ``-Xuops`` or set ``PYTHONUOPS=1``. To get debug output, set ``PYTHONUOPSDEBUG=N`` where ``N`` is a debug level (0-4, where 0 is no debug output and 4 is excessively verbose). All of this code is likely to change dramatically before the 3.13 feature freeze. But this is a first step.
2023-06-26 19:02:57 -07:00
for (int i = oparg; --i >= 0; ) {
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
*values++ = PyStackRef_FromPyObjectNew(items[i]);
gh-104584: Baby steps towards generating and executing traces (#105924) Added a new, experimental, tracing optimizer and interpreter (a.k.a. "tier 2"). This currently pessimizes, so don't use yet -- this is infrastructure so we can experiment with optimizing passes. To enable it, pass ``-Xuops`` or set ``PYTHONUOPS=1``. To get debug output, set ``PYTHONUOPSDEBUG=N`` where ``N`` is a debug level (0-4, where 0 is no debug output and 4 is excessively verbose). All of this code is likely to change dramatically before the 3.13 feature freeze. But this is a first step.
2023-06-26 19:02:57 -07:00
}
gh-115999: Use light-weight lock for UNPACK_SEQUENCE_LIST (gh-127514)
2024-12-03 00:14:40 +09:00
UNLOCK_OBJECT(seq_o);
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
stack_pointer += oparg;
GH-141794: Limit size of generated machine code. (GH-142228) * Factor out bodies of the largest uops, to reduce jit code size. * Factor out common assert, also reducing jit code size. * Limit size of jitted code for a single executor to 1MB.
2025-12-03 17:43:35 +00:00
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
GH-128682: Account for escapes in `DECREF_INPUTS` (GH-129953) * Handle escapes in DECREF_INPUTS * Mark a few more functions as escaping * Replace DECREF_INPUTS with PyStackRef_CLOSE where possible
2025-02-12 17:44:59 +00:00
_PyFrame_SetStackPointer(frame, stack_pointer);
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
PyStackRef_CLOSE(seq);
GH-128682: Account for escapes in `DECREF_INPUTS` (GH-129953) * Handle escapes in DECREF_INPUTS * Mark a few more functions as escaping * Replace DECREF_INPUTS with PyStackRef_CLOSE where possible
2025-02-12 17:44:59 +00:00
stack_pointer = _PyFrame_GetStackPointer(frame);
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_tos_cache0 = PyStackRef_ZERO_BITS;
_tos_cache1 = PyStackRef_ZERO_BITS;
_tos_cache2 = PyStackRef_ZERO_BITS;
SET_CURRENT_CACHED_VALUES(0);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-104584: Baby steps towards generating and executing traces (#105924) Added a new, experimental, tracing optimizer and interpreter (a.k.a. "tier 2"). This currently pessimizes, so don't use yet -- this is infrastructure so we can experiment with optimizing passes. To enable it, pass ``-Xuops`` or set ``PYTHONUOPS=1``. To get debug output, set ``PYTHONUOPSDEBUG=N`` where ``N`` is a debug level (0-4, where 0 is no debug output and 4 is excessively verbose). All of this code is likely to change dramatically before the 3.13 feature freeze. But this is a first step.
2023-06-26 19:02:57 -07:00
break;
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
case _UNPACK_EX_r10: {
CHECK_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
_PyStackRef seq;
GH-128682: Account for escapes in `DECREF_INPUTS` (GH-129953) * Handle escapes in DECREF_INPUTS * Mark a few more functions as escaping * Replace DECREF_INPUTS with PyStackRef_CLOSE where possible
2025-02-12 17:44:59 +00:00
_PyStackRef *top;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef _stack_item_0 = _tos_cache0;
GH-111485: Factor out tier 2 code generation from the rest of the interpreter code generator (GH-112968)
2023-12-12 12:12:17 +00:00
oparg = CURRENT_OPARG();
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
seq = _stack_item_0;
top = &stack_pointer[1 + (oparg & 0xFF) + (oparg >> 8)];
GH-128682: Account for escapes in `DECREF_INPUTS` (GH-129953) * Handle escapes in DECREF_INPUTS * Mark a few more functions as escaping * Replace DECREF_INPUTS with PyStackRef_CLOSE where possible
2025-02-12 17:44:59 +00:00
PyObject *seq_o = PyStackRef_AsPyObjectSteal(seq);
GH-119866: Spill the stack around escaping calls. (GH-124392) * Spill the evaluation around escaping calls in the generated interpreter and JIT. * The code generator tracks live, cached values so they can be saved to memory when needed. * Spills the stack pointer around escaping calls, so that the exact stack is visible to the cycle GC.
2024-10-07 14:56:39 +01:00
_PyFrame_SetStackPointer(frame, stack_pointer);
GH-128682: Account for escapes in `DECREF_INPUTS` (GH-129953) * Handle escapes in DECREF_INPUTS * Mark a few more functions as escaping * Replace DECREF_INPUTS with PyStackRef_CLOSE where possible
2025-02-12 17:44:59 +00:00
int res = _PyEval_UnpackIterableStackRef(tstate, seq_o, oparg & 0xFF, oparg >> 8, top);
Py_DECREF(seq_o);
GH-119866: Spill the stack around escaping calls. (GH-124392) * Spill the evaluation around escaping calls in the generated interpreter and JIT. * The code generator tracks live, cached values so they can be saved to memory when needed. * Spills the stack pointer around escaping calls, so that the exact stack is visible to the cycle GC.
2024-10-07 14:56:39 +01:00
stack_pointer = _PyFrame_GetStackPointer(frame);
GH-128682: Make `PyStackRef_CLOSE` escaping. (GH-129404)
2025-02-03 12:41:32 +00:00
if (res == 0) {
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(0);
GH-128682: Make `PyStackRef_CLOSE` escaping. (GH-129404)
2025-02-03 12:41:32 +00:00
JUMP_TO_ERROR();
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_tos_cache0 = PyStackRef_ZERO_BITS;
_tos_cache1 = PyStackRef_ZERO_BITS;
_tos_cache2 = PyStackRef_ZERO_BITS;
SET_CURRENT_CACHED_VALUES(0);
GH-128682: Account for escapes in `DECREF_INPUTS` (GH-129953) * Handle escapes in DECREF_INPUTS * Mark a few more functions as escaping * Replace DECREF_INPUTS with PyStackRef_CLOSE where possible
2025-02-12 17:44:59 +00:00
stack_pointer += 1 + (oparg & 0xFF) + (oparg >> 8);
GH-141794: Limit size of generated machine code. (GH-142228) * Factor out bodies of the largest uops, to reduce jit code size. * Factor out common assert, also reducing jit code size. * Limit size of jitted code for a single executor to 1MB.
2025-12-03 17:43:35 +00:00
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-104584: Baby steps towards generating and executing traces (#105924) Added a new, experimental, tracing optimizer and interpreter (a.k.a. "tier 2"). This currently pessimizes, so don't use yet -- this is infrastructure so we can experiment with optimizing passes. To enable it, pass ``-Xuops`` or set ``PYTHONUOPS=1``. To get debug output, set ``PYTHONUOPSDEBUG=N`` where ``N`` is a debug level (0-4, where 0 is no debug output and 4 is excessively verbose). All of this code is likely to change dramatically before the 3.13 feature freeze. But this is a first step.
2023-06-26 19:02:57 -07:00
break;
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
case _STORE_ATTR_r20: {
CHECK_CURRENT_CACHED_VALUES(2);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
_PyStackRef owner;
_PyStackRef v;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef _stack_item_0 = _tos_cache0;
_PyStackRef _stack_item_1 = _tos_cache1;
GH-111485: Factor out tier 2 code generation from the rest of the interpreter code generator (GH-112968)
2023-12-12 12:12:17 +00:00
oparg = CURRENT_OPARG();
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
owner = _stack_item_1;
v = _stack_item_0;
gh-106603: Make uop struct a triple (opcode, oparg, operand) (#106794)
2023-07-17 12:12:33 -07:00
PyObject *name = GETITEM(FRAME_CO_NAMES, oparg);
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
stack_pointer[0] = v;
stack_pointer[1] = owner;
stack_pointer += 2;
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
GH-119866: Spill the stack around escaping calls. (GH-124392) * Spill the evaluation around escaping calls in the generated interpreter and JIT. * The code generator tracks live, cached values so they can be saved to memory when needed. * Spills the stack pointer around escaping calls, so that the exact stack is visible to the cycle GC.
2024-10-07 14:56:39 +01:00
_PyFrame_SetStackPointer(frame, stack_pointer);
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
int err = PyObject_SetAttr(PyStackRef_AsPyObjectBorrow(owner),
gh-117139: Fix a few wrong steals in bytecodes.c (GH-121127) Fix a few wrong steals in bytecodes.c
2024-06-29 02:14:48 +08:00
name, PyStackRef_AsPyObjectBorrow(v));
GH-128682: Account for escapes in `DECREF_INPUTS` (GH-129953) * Handle escapes in DECREF_INPUTS * Mark a few more functions as escaping * Replace DECREF_INPUTS with PyStackRef_CLOSE where possible
2025-02-12 17:44:59 +00:00
_PyStackRef tmp = owner;
owner = PyStackRef_NULL;
stack_pointer[-1] = owner;
PyStackRef_CLOSE(tmp);
tmp = v;
v = PyStackRef_NULL;
stack_pointer[-2] = v;
PyStackRef_CLOSE(tmp);
GH-119866: Spill the stack around escaping calls. (GH-124392) * Spill the evaluation around escaping calls in the generated interpreter and JIT. * The code generator tracks live, cached values so they can be saved to memory when needed. * Spills the stack pointer around escaping calls, so that the exact stack is visible to the cycle GC.
2024-10-07 14:56:39 +01:00
stack_pointer = _PyFrame_GetStackPointer(frame);
GH-128682: Account for escapes in `DECREF_INPUTS` (GH-129953) * Handle escapes in DECREF_INPUTS * Mark a few more functions as escaping * Replace DECREF_INPUTS with PyStackRef_CLOSE where possible
2025-02-12 17:44:59 +00:00
stack_pointer += -2;
GH-141794: Limit size of generated machine code. (GH-142228) * Factor out bodies of the largest uops, to reduce jit code size. * Factor out common assert, also reducing jit code size. * Limit size of jitted code for a single executor to 1MB.
2025-12-03 17:43:35 +00:00
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
GH-128682: Make `PyStackRef_CLOSE` escaping. (GH-129404)
2025-02-03 12:41:32 +00:00
if (err) {
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(0);
GH-128682: Make `PyStackRef_CLOSE` escaping. (GH-129404)
2025-02-03 12:41:32 +00:00
JUMP_TO_ERROR();
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_tos_cache0 = PyStackRef_ZERO_BITS;
_tos_cache1 = PyStackRef_ZERO_BITS;
_tos_cache2 = PyStackRef_ZERO_BITS;
SET_CURRENT_CACHED_VALUES(0);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-106603: Make uop struct a triple (opcode, oparg, operand) (#106794)
2023-07-17 12:12:33 -07:00
break;
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
case _DELETE_ATTR_r10: {
CHECK_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
_PyStackRef owner;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef _stack_item_0 = _tos_cache0;
GH-111485: Factor out tier 2 code generation from the rest of the interpreter code generator (GH-112968)
2023-12-12 12:12:17 +00:00
oparg = CURRENT_OPARG();
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
owner = _stack_item_0;
gh-104584: Baby steps towards generating and executing traces (#105924) Added a new, experimental, tracing optimizer and interpreter (a.k.a. "tier 2"). This currently pessimizes, so don't use yet -- this is infrastructure so we can experiment with optimizing passes. To enable it, pass ``-Xuops`` or set ``PYTHONUOPS=1``. To get debug output, set ``PYTHONUOPSDEBUG=N`` where ``N`` is a debug level (0-4, where 0 is no debug output and 4 is excessively verbose). All of this code is likely to change dramatically before the 3.13 feature freeze. But this is a first step.
2023-06-26 19:02:57 -07:00
PyObject *name = GETITEM(FRAME_CO_NAMES, oparg);
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
stack_pointer[0] = owner;
stack_pointer += 1;
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
GH-119866: Spill the stack around escaping calls. (GH-124392) * Spill the evaluation around escaping calls in the generated interpreter and JIT. * The code generator tracks live, cached values so they can be saved to memory when needed. * Spills the stack pointer around escaping calls, so that the exact stack is visible to the cycle GC.
2024-10-07 14:56:39 +01:00
_PyFrame_SetStackPointer(frame, stack_pointer);
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
int err = PyObject_DelAttr(PyStackRef_AsPyObjectBorrow(owner), name);
GH-119866: Spill the stack around escaping calls. (GH-124392) * Spill the evaluation around escaping calls in the generated interpreter and JIT. * The code generator tracks live, cached values so they can be saved to memory when needed. * Spills the stack pointer around escaping calls, so that the exact stack is visible to the cycle GC.
2024-10-07 14:56:39 +01:00
stack_pointer = _PyFrame_GetStackPointer(frame);
GH-128682: Account for escapes in `DECREF_INPUTS` (GH-129953) * Handle escapes in DECREF_INPUTS * Mark a few more functions as escaping * Replace DECREF_INPUTS with PyStackRef_CLOSE where possible
2025-02-12 17:44:59 +00:00
stack_pointer += -1;
GH-141794: Limit size of generated machine code. (GH-142228) * Factor out bodies of the largest uops, to reduce jit code size. * Factor out common assert, also reducing jit code size. * Limit size of jitted code for a single executor to 1MB.
2025-12-03 17:43:35 +00:00
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
GH-128682: Account for escapes in `DECREF_INPUTS` (GH-129953) * Handle escapes in DECREF_INPUTS * Mark a few more functions as escaping * Replace DECREF_INPUTS with PyStackRef_CLOSE where possible
2025-02-12 17:44:59 +00:00
_PyFrame_SetStackPointer(frame, stack_pointer);
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
PyStackRef_CLOSE(owner);
GH-128682: Account for escapes in `DECREF_INPUTS` (GH-129953) * Handle escapes in DECREF_INPUTS * Mark a few more functions as escaping * Replace DECREF_INPUTS with PyStackRef_CLOSE where possible
2025-02-12 17:44:59 +00:00
stack_pointer = _PyFrame_GetStackPointer(frame);
GH-128682: Make `PyStackRef_CLOSE` escaping. (GH-129404)
2025-02-03 12:41:32 +00:00
if (err) {
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(0);
GH-128682: Make `PyStackRef_CLOSE` escaping. (GH-129404)
2025-02-03 12:41:32 +00:00
JUMP_TO_ERROR();
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_tos_cache0 = PyStackRef_ZERO_BITS;
_tos_cache1 = PyStackRef_ZERO_BITS;
_tos_cache2 = PyStackRef_ZERO_BITS;
SET_CURRENT_CACHED_VALUES(0);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-104584: Baby steps towards generating and executing traces (#105924) Added a new, experimental, tracing optimizer and interpreter (a.k.a. "tier 2"). This currently pessimizes, so don't use yet -- this is infrastructure so we can experiment with optimizing passes. To enable it, pass ``-Xuops`` or set ``PYTHONUOPS=1``. To get debug output, set ``PYTHONUOPSDEBUG=N`` where ``N`` is a debug level (0-4, where 0 is no debug output and 4 is excessively verbose). All of this code is likely to change dramatically before the 3.13 feature freeze. But this is a first step.
2023-06-26 19:02:57 -07:00
break;
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
case _STORE_GLOBAL_r10: {
CHECK_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
_PyStackRef v;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef _stack_item_0 = _tos_cache0;
GH-111485: Factor out tier 2 code generation from the rest of the interpreter code generator (GH-112968)
2023-12-12 12:12:17 +00:00
oparg = CURRENT_OPARG();
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
v = _stack_item_0;
gh-104584: Baby steps towards generating and executing traces (#105924) Added a new, experimental, tracing optimizer and interpreter (a.k.a. "tier 2"). This currently pessimizes, so don't use yet -- this is infrastructure so we can experiment with optimizing passes. To enable it, pass ``-Xuops`` or set ``PYTHONUOPS=1``. To get debug output, set ``PYTHONUOPSDEBUG=N`` where ``N`` is a debug level (0-4, where 0 is no debug output and 4 is excessively verbose). All of this code is likely to change dramatically before the 3.13 feature freeze. But this is a first step.
2023-06-26 19:02:57 -07:00
PyObject *name = GETITEM(FRAME_CO_NAMES, oparg);
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
stack_pointer[0] = v;
stack_pointer += 1;
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
GH-119866: Spill the stack around escaping calls. (GH-124392) * Spill the evaluation around escaping calls in the generated interpreter and JIT. * The code generator tracks live, cached values so they can be saved to memory when needed. * Spills the stack pointer around escaping calls, so that the exact stack is visible to the cycle GC.
2024-10-07 14:56:39 +01:00
_PyFrame_SetStackPointer(frame, stack_pointer);
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
int err = PyDict_SetItem(GLOBALS(), name, PyStackRef_AsPyObjectBorrow(v));
GH-119866: Spill the stack around escaping calls. (GH-124392) * Spill the evaluation around escaping calls in the generated interpreter and JIT. * The code generator tracks live, cached values so they can be saved to memory when needed. * Spills the stack pointer around escaping calls, so that the exact stack is visible to the cycle GC.
2024-10-07 14:56:39 +01:00
stack_pointer = _PyFrame_GetStackPointer(frame);
GH-128682: Account for escapes in `DECREF_INPUTS` (GH-129953) * Handle escapes in DECREF_INPUTS * Mark a few more functions as escaping * Replace DECREF_INPUTS with PyStackRef_CLOSE where possible
2025-02-12 17:44:59 +00:00
stack_pointer += -1;
GH-141794: Limit size of generated machine code. (GH-142228) * Factor out bodies of the largest uops, to reduce jit code size. * Factor out common assert, also reducing jit code size. * Limit size of jitted code for a single executor to 1MB.
2025-12-03 17:43:35 +00:00
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
GH-128682: Account for escapes in `DECREF_INPUTS` (GH-129953) * Handle escapes in DECREF_INPUTS * Mark a few more functions as escaping * Replace DECREF_INPUTS with PyStackRef_CLOSE where possible
2025-02-12 17:44:59 +00:00
_PyFrame_SetStackPointer(frame, stack_pointer);
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
PyStackRef_CLOSE(v);
GH-128682: Account for escapes in `DECREF_INPUTS` (GH-129953) * Handle escapes in DECREF_INPUTS * Mark a few more functions as escaping * Replace DECREF_INPUTS with PyStackRef_CLOSE where possible
2025-02-12 17:44:59 +00:00
stack_pointer = _PyFrame_GetStackPointer(frame);
GH-128682: Make `PyStackRef_CLOSE` escaping. (GH-129404)
2025-02-03 12:41:32 +00:00
if (err) {
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(0);
GH-128682: Make `PyStackRef_CLOSE` escaping. (GH-129404)
2025-02-03 12:41:32 +00:00
JUMP_TO_ERROR();
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_tos_cache0 = PyStackRef_ZERO_BITS;
_tos_cache1 = PyStackRef_ZERO_BITS;
_tos_cache2 = PyStackRef_ZERO_BITS;
SET_CURRENT_CACHED_VALUES(0);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-104584: Baby steps towards generating and executing traces (#105924) Added a new, experimental, tracing optimizer and interpreter (a.k.a. "tier 2"). This currently pessimizes, so don't use yet -- this is infrastructure so we can experiment with optimizing passes. To enable it, pass ``-Xuops`` or set ``PYTHONUOPS=1``. To get debug output, set ``PYTHONUOPSDEBUG=N`` where ``N`` is a debug level (0-4, where 0 is no debug output and 4 is excessively verbose). All of this code is likely to change dramatically before the 3.13 feature freeze. But this is a first step.
2023-06-26 19:02:57 -07:00
break;
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
case _DELETE_GLOBAL_r00: {
CHECK_CURRENT_CACHED_VALUES(0);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-112287: Speed up Tier 2 (uop) interpreter a little (#112286) This makes the Tier 2 interpreter a little faster. I calculated by about 3%, though I hesitate to claim an exact number. This starts by doubling the trace size limit (to 512), making it more likely that loops fit in a trace. The rest of the approach is to only load `oparg` and `operand` in cases that use them. The code generator know when these are used. For `oparg`, it will conditionally emit ``` oparg = CURRENT_OPARG(); ``` at the top of the case block. (The `oparg` variable may be referenced multiple times by the instructions code block, so it must be in a variable.) For `operand`, it will use `CURRENT_OPERAND()` directly instead of referencing the `operand` variable, which no longer exists. (There is only one place where this will be used.)
2023-11-20 11:25:32 -08:00
oparg = CURRENT_OPARG();
gh-104584: Baby steps towards generating and executing traces (#105924) Added a new, experimental, tracing optimizer and interpreter (a.k.a. "tier 2"). This currently pessimizes, so don't use yet -- this is infrastructure so we can experiment with optimizing passes. To enable it, pass ``-Xuops`` or set ``PYTHONUOPS=1``. To get debug output, set ``PYTHONUOPSDEBUG=N`` where ``N`` is a debug level (0-4, where 0 is no debug output and 4 is excessively verbose). All of this code is likely to change dramatically before the 3.13 feature freeze. But this is a first step.
2023-06-26 19:02:57 -07:00
PyObject *name = GETITEM(FRAME_CO_NAMES, oparg);
GH-119866: Spill the stack around escaping calls. (GH-124392) * Spill the evaluation around escaping calls in the generated interpreter and JIT. * The code generator tracks live, cached values so they can be saved to memory when needed. * Spills the stack pointer around escaping calls, so that the exact stack is visible to the cycle GC.
2024-10-07 14:56:39 +01:00
_PyFrame_SetStackPointer(frame, stack_pointer);
gh-116437: Use new C API PyDict_Pop() to simplify the code (GH-116438)
2024-03-07 11:21:08 +02:00
int err = PyDict_Pop(GLOBALS(), name, NULL);
GH-119866: Spill the stack around escaping calls. (GH-124392) * Spill the evaluation around escaping calls in the generated interpreter and JIT. * The code generator tracks live, cached values so they can be saved to memory when needed. * Spills the stack pointer around escaping calls, so that the exact stack is visible to the cycle GC.
2024-10-07 14:56:39 +01:00
stack_pointer = _PyFrame_GetStackPointer(frame);
gh-116437: Use new C API PyDict_Pop() to simplify the code (GH-116438)
2024-03-07 11:21:08 +02:00
if (err < 0) {
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(0);
GH-116422: Tier2 hot/cold splitting (GH-116813) Splits the "cold" path, deopts and exits, from the "hot" path, reducing the size of most jitted instructions, at the cost of slower exits.
2024-03-26 09:35:11 +00:00
JUMP_TO_ERROR();
gh-116437: Use new C API PyDict_Pop() to simplify the code (GH-116438)
2024-03-07 11:21:08 +02:00
}
if (err == 0) {
GH-119866: Spill the stack around escaping calls. (GH-124392) * Spill the evaluation around escaping calls in the generated interpreter and JIT. * The code generator tracks live, cached values so they can be saved to memory when needed. * Spills the stack pointer around escaping calls, so that the exact stack is visible to the cycle GC.
2024-10-07 14:56:39 +01:00
_PyFrame_SetStackPointer(frame, stack_pointer);
gh-116437: Use new C API PyDict_Pop() to simplify the code (GH-116438)
2024-03-07 11:21:08 +02:00
_PyEval_FormatExcCheckArg(tstate, PyExc_NameError,
NAME_ERROR_MSG, name);
GH-119866: Spill the stack around escaping calls. (GH-124392) * Spill the evaluation around escaping calls in the generated interpreter and JIT. * The code generator tracks live, cached values so they can be saved to memory when needed. * Spills the stack pointer around escaping calls, so that the exact stack is visible to the cycle GC.
2024-10-07 14:56:39 +01:00
stack_pointer = _PyFrame_GetStackPointer(frame);
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(0);
GH-116422: Tier2 hot/cold splitting (GH-116813) Splits the "cold" path, deopts and exits, from the "hot" path, reducing the size of most jitted instructions, at the cost of slower exits.
2024-03-26 09:35:11 +00:00
JUMP_TO_ERROR();
gh-104584: Baby steps towards generating and executing traces (#105924) Added a new, experimental, tracing optimizer and interpreter (a.k.a. "tier 2"). This currently pessimizes, so don't use yet -- this is infrastructure so we can experiment with optimizing passes. To enable it, pass ``-Xuops`` or set ``PYTHONUOPS=1``. To get debug output, set ``PYTHONUOPSDEBUG=N`` where ``N`` is a debug level (0-4, where 0 is no debug output and 4 is excessively verbose). All of this code is likely to change dramatically before the 3.13 feature freeze. But this is a first step.
2023-06-26 19:02:57 -07:00
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_tos_cache0 = PyStackRef_ZERO_BITS;
_tos_cache1 = PyStackRef_ZERO_BITS;
_tos_cache2 = PyStackRef_ZERO_BITS;
SET_CURRENT_CACHED_VALUES(0);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-104584: Baby steps towards generating and executing traces (#105924) Added a new, experimental, tracing optimizer and interpreter (a.k.a. "tier 2"). This currently pessimizes, so don't use yet -- this is infrastructure so we can experiment with optimizing passes. To enable it, pass ``-Xuops`` or set ``PYTHONUOPS=1``. To get debug output, set ``PYTHONUOPSDEBUG=N`` where ``N`` is a debug level (0-4, where 0 is no debug output and 4 is excessively verbose). All of this code is likely to change dramatically before the 3.13 feature freeze. But this is a first step.
2023-06-26 19:02:57 -07:00
break;
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
case _LOAD_LOCALS_r01: {
CHECK_CURRENT_CACHED_VALUES(0);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
_PyStackRef locals;
PyObject *l = LOCALS();
if (l == NULL) {
GH-119866: Spill the stack around escaping calls. (GH-124392) * Spill the evaluation around escaping calls in the generated interpreter and JIT. * The code generator tracks live, cached values so they can be saved to memory when needed. * Spills the stack pointer around escaping calls, so that the exact stack is visible to the cycle GC.
2024-10-07 14:56:39 +01:00
_PyFrame_SetStackPointer(frame, stack_pointer);
gh-104584: Baby steps towards generating and executing traces (#105924) Added a new, experimental, tracing optimizer and interpreter (a.k.a. "tier 2"). This currently pessimizes, so don't use yet -- this is infrastructure so we can experiment with optimizing passes. To enable it, pass ``-Xuops`` or set ``PYTHONUOPS=1``. To get debug output, set ``PYTHONUOPSDEBUG=N`` where ``N`` is a debug level (0-4, where 0 is no debug output and 4 is excessively verbose). All of this code is likely to change dramatically before the 3.13 feature freeze. But this is a first step.
2023-06-26 19:02:57 -07:00
_PyErr_SetString(tstate, PyExc_SystemError,
"no locals found");
GH-119866: Spill the stack around escaping calls. (GH-124392) * Spill the evaluation around escaping calls in the generated interpreter and JIT. * The code generator tracks live, cached values so they can be saved to memory when needed. * Spills the stack pointer around escaping calls, so that the exact stack is visible to the cycle GC.
2024-10-07 14:56:39 +01:00
stack_pointer = _PyFrame_GetStackPointer(frame);
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(0);
GH-128682: Make `PyStackRef_CLOSE` escaping. (GH-129404)
2025-02-03 12:41:32 +00:00
JUMP_TO_ERROR();
gh-104584: Baby steps towards generating and executing traces (#105924) Added a new, experimental, tracing optimizer and interpreter (a.k.a. "tier 2"). This currently pessimizes, so don't use yet -- this is infrastructure so we can experiment with optimizing passes. To enable it, pass ``-Xuops`` or set ``PYTHONUOPS=1``. To get debug output, set ``PYTHONUOPSDEBUG=N`` where ``N`` is a debug level (0-4, where 0 is no debug output and 4 is excessively verbose). All of this code is likely to change dramatically before the 3.13 feature freeze. But this is a first step.
2023-06-26 19:02:57 -07:00
}
gh-118926: Spill deferred references to stack in cases generator (#122748) This automatically spills the results from `_PyStackRef_FromPyObjectNew` to the in-memory stack so that the deferred references are visible to the GC before we make any possibly escaping call. Co-authored-by: Ken Jin <kenjin@python.org>
2024-08-07 13:23:53 -04:00
locals = PyStackRef_FromPyObjectNew(l);
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_tos_cache0 = locals;
SET_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-104584: Baby steps towards generating and executing traces (#105924) Added a new, experimental, tracing optimizer and interpreter (a.k.a. "tier 2"). This currently pessimizes, so don't use yet -- this is infrastructure so we can experiment with optimizing passes. To enable it, pass ``-Xuops`` or set ``PYTHONUOPS=1``. To get debug output, set ``PYTHONUOPSDEBUG=N`` where ``N`` is a debug level (0-4, where 0 is no debug output and 4 is excessively verbose). All of this code is likely to change dramatically before the 3.13 feature freeze. But this is a first step.
2023-06-26 19:02:57 -07:00
break;
}
GH-142718: JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. (GH-142762) JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. * Make sure that stack is in correct state at side exits with TOS cached values * Simplify choice of cached items for side exits
2025-12-15 19:01:29 +00:00
case _LOAD_LOCALS_r12: {
CHECK_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-142718: JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. (GH-142762) JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. * Make sure that stack is in correct state at side exits with TOS cached values * Simplify choice of cached items for side exits
2025-12-15 19:01:29 +00:00
_PyStackRef locals;
_PyStackRef _stack_item_0 = _tos_cache0;
PyObject *l = LOCALS();
if (l == NULL) {
stack_pointer[0] = _stack_item_0;
stack_pointer += 1;
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
_PyFrame_SetStackPointer(frame, stack_pointer);
_PyErr_SetString(tstate, PyExc_SystemError,
"no locals found");
stack_pointer = _PyFrame_GetStackPointer(frame);
SET_CURRENT_CACHED_VALUES(0);
JUMP_TO_ERROR();
}
locals = PyStackRef_FromPyObjectNew(l);
_tos_cache1 = locals;
_tos_cache0 = _stack_item_0;
SET_CURRENT_CACHED_VALUES(2);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-142718: JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. (GH-142762) JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. * Make sure that stack is in correct state at side exits with TOS cached values * Simplify choice of cached items for side exits
2025-12-15 19:01:29 +00:00
break;
}
case _LOAD_LOCALS_r23: {
CHECK_CURRENT_CACHED_VALUES(2);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-142718: JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. (GH-142762) JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. * Make sure that stack is in correct state at side exits with TOS cached values * Simplify choice of cached items for side exits
2025-12-15 19:01:29 +00:00
_PyStackRef locals;
_PyStackRef _stack_item_0 = _tos_cache0;
_PyStackRef _stack_item_1 = _tos_cache1;
PyObject *l = LOCALS();
if (l == NULL) {
stack_pointer[0] = _stack_item_0;
stack_pointer[1] = _stack_item_1;
stack_pointer += 2;
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
_PyFrame_SetStackPointer(frame, stack_pointer);
_PyErr_SetString(tstate, PyExc_SystemError,
"no locals found");
stack_pointer = _PyFrame_GetStackPointer(frame);
SET_CURRENT_CACHED_VALUES(0);
JUMP_TO_ERROR();
}
locals = PyStackRef_FromPyObjectNew(l);
_tos_cache2 = locals;
_tos_cache1 = _stack_item_1;
_tos_cache0 = _stack_item_0;
SET_CURRENT_CACHED_VALUES(3);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-142718: JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. (GH-142762) JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. * Make sure that stack is in correct state at side exits with TOS cached values * Simplify choice of cached items for side exits
2025-12-15 19:01:29 +00:00
break;
}
gh-119821: Support non-dict globals in LOAD_FROM_DICT_OR_GLOBALS (#119822) Support non-dict globals in LOAD_FROM_DICT_OR_GLOBALS The implementation basically copies LOAD_GLOBAL. Possibly it could be deduplicated, but that seems like it may get hairy since the two operations have different operands. This is important to fix in 3.14 for PEP 649, but it's a bug in earlier versions too, and we should backport to 3.13 and 3.12 if possible.
2024-05-31 14:05:24 -07:00
/* _LOAD_FROM_DICT_OR_GLOBALS is not a viable micro-op for tier 2 because it has both popping and not-popping errors */
gh-109118: Fix runtime crash when NameError happens in PEP 695 function (#109123)
2023-09-08 19:49:20 -07:00
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
case _LOAD_NAME_r01: {
CHECK_CURRENT_CACHED_VALUES(0);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-118093: Add tier two support to several instructions (GH-121884)
2024-07-18 14:24:58 -07:00
_PyStackRef v;
oparg = CURRENT_OPARG();
PyObject *name = GETITEM(FRAME_CO_NAMES, oparg);
GH-119866: Spill the stack around escaping calls. (GH-124392) * Spill the evaluation around escaping calls in the generated interpreter and JIT. * The code generator tracks live, cached values so they can be saved to memory when needed. * Spills the stack pointer around escaping calls, so that the exact stack is visible to the cycle GC.
2024-10-07 14:56:39 +01:00
_PyFrame_SetStackPointer(frame, stack_pointer);
GH-117224: Move the body of a few large-ish micro-ops into helper functions (GH-122601)
2024-08-02 16:31:17 +01:00
PyObject *v_o = _PyEval_LoadName(tstate, frame, name);
GH-119866: Spill the stack around escaping calls. (GH-124392) * Spill the evaluation around escaping calls in the generated interpreter and JIT. * The code generator tracks live, cached values so they can be saved to memory when needed. * Spills the stack pointer around escaping calls, so that the exact stack is visible to the cycle GC.
2024-10-07 14:56:39 +01:00
stack_pointer = _PyFrame_GetStackPointer(frame);
GH-128682: Make `PyStackRef_CLOSE` escaping. (GH-129404)
2025-02-03 12:41:32 +00:00
if (v_o == NULL) {
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(0);
GH-128682: Make `PyStackRef_CLOSE` escaping. (GH-129404)
2025-02-03 12:41:32 +00:00
JUMP_TO_ERROR();
}
GH-118093: Add tier two support to several instructions (GH-121884)
2024-07-18 14:24:58 -07:00
v = PyStackRef_FromPyObjectSteal(v_o);
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_tos_cache0 = v;
_tos_cache1 = PyStackRef_ZERO_BITS;
_tos_cache2 = PyStackRef_ZERO_BITS;
SET_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-118093: Add tier two support to several instructions (GH-121884)
2024-07-18 14:24:58 -07:00
break;
}
gh-104584: Baby steps towards generating and executing traces (#105924) Added a new, experimental, tracing optimizer and interpreter (a.k.a. "tier 2"). This currently pessimizes, so don't use yet -- this is infrastructure so we can experiment with optimizing passes. To enable it, pass ``-Xuops`` or set ``PYTHONUOPS=1``. To get debug output, set ``PYTHONUOPSDEBUG=N`` where ``N`` is a debug level (0-4, where 0 is no debug output and 4 is excessively verbose). All of this code is likely to change dramatically before the 3.13 feature freeze. But this is a first step.
2023-06-26 19:02:57 -07:00
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
case _LOAD_GLOBAL_r00: {
CHECK_CURRENT_CACHED_VALUES(0);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-121459: Deferred LOAD_GLOBAL (GH-123128) Co-authored-by: Bénédikt Tran <10796600+picnixz@users.noreply.github.com> Co-authored-by: Sam Gross <655866+colesbury@users.noreply.github.com>
2024-09-14 00:23:51 +08:00
_PyStackRef *res;
GH-111485: Factor out tier 2 code generation from the rest of the interpreter code generator (GH-112968)
2023-12-12 12:12:17 +00:00
oparg = CURRENT_OPARG();
gh-121459: Deferred LOAD_GLOBAL (GH-123128) Co-authored-by: Bénédikt Tran <10796600+picnixz@users.noreply.github.com> Co-authored-by: Sam Gross <655866+colesbury@users.noreply.github.com>
2024-09-14 00:23:51 +08:00
res = &stack_pointer[0];
Revert "GH-128914: Remove conditional stack effects from `bytecodes.c` and the code generators (GH-128918)" (GH-129202) The commit introduced a ~2.5-3% regression in the free threading build. This reverts commit ab61d3f4303d14a413bc9ae6557c730ffdf7579e.
2025-01-23 04:26:25 -05:00
PyObject *name = GETITEM(FRAME_CO_NAMES, oparg>>1);
GH-119866: Spill the stack around escaping calls. (GH-124392) * Spill the evaluation around escaping calls in the generated interpreter and JIT. * The code generator tracks live, cached values so they can be saved to memory when needed. * Spills the stack pointer around escaping calls, so that the exact stack is visible to the cycle GC.
2024-10-07 14:56:39 +01:00
_PyFrame_SetStackPointer(frame, stack_pointer);
gh-121459: Deferred LOAD_GLOBAL (GH-123128) Co-authored-by: Bénédikt Tran <10796600+picnixz@users.noreply.github.com> Co-authored-by: Sam Gross <655866+colesbury@users.noreply.github.com>
2024-09-14 00:23:51 +08:00
_PyEval_LoadGlobalStackRef(GLOBALS(), BUILTINS(), name, res);
GH-119866: Spill the stack around escaping calls. (GH-124392) * Spill the evaluation around escaping calls in the generated interpreter and JIT. * The code generator tracks live, cached values so they can be saved to memory when needed. * Spills the stack pointer around escaping calls, so that the exact stack is visible to the cycle GC.
2024-10-07 14:56:39 +01:00
stack_pointer = _PyFrame_GetStackPointer(frame);
GH-128682: Make `PyStackRef_CLOSE` escaping. (GH-129404)
2025-02-03 12:41:32 +00:00
if (PyStackRef_IsNull(*res)) {
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(0);
GH-128682: Make `PyStackRef_CLOSE` escaping. (GH-129404)
2025-02-03 12:41:32 +00:00
JUMP_TO_ERROR();
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_tos_cache0 = PyStackRef_ZERO_BITS;
_tos_cache1 = PyStackRef_ZERO_BITS;
_tos_cache2 = PyStackRef_ZERO_BITS;
SET_CURRENT_CACHED_VALUES(0);
GH-128914: Remove all but one conditional stack effects (GH-129226) * Remove all 'if (0)' and 'if (1)' conditional stack effects * Use array instead of conditional for BUILD_SLICE args * Refactor LOAD_GLOBAL to use a common conditional uop * Remove conditional stack effects from LOAD_ATTR specializations * Replace conditional stack effects in LOAD_ATTR with a 0 or 1 sized array. * Remove conditional stack effects from CALL_FUNCTION_EX
2025-01-27 16:24:48 +00:00
stack_pointer += 1;
GH-141794: Limit size of generated machine code. (GH-142228) * Factor out bodies of the largest uops, to reduce jit code size. * Factor out common assert, also reducing jit code size. * Limit size of jitted code for a single executor to 1MB.
2025-12-03 17:43:35 +00:00
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-128914: Remove all but one conditional stack effects (GH-129226) * Remove all 'if (0)' and 'if (1)' conditional stack effects * Use array instead of conditional for BUILD_SLICE args * Refactor LOAD_GLOBAL to use a common conditional uop * Remove conditional stack effects from LOAD_ATTR specializations * Replace conditional stack effects in LOAD_ATTR with a 0 or 1 sized array. * Remove conditional stack effects from CALL_FUNCTION_EX
2025-01-27 16:24:48 +00:00
break;
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
case _PUSH_NULL_CONDITIONAL_r00: {
CHECK_CURRENT_CACHED_VALUES(0);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-131498: Remove conditional stack effects (GH-131499) * Adds some missing #includes
2025-03-20 15:39:38 +00:00
_PyStackRef *null;
GH-128914: Remove all but one conditional stack effects (GH-129226) * Remove all 'if (0)' and 'if (1)' conditional stack effects * Use array instead of conditional for BUILD_SLICE args * Refactor LOAD_GLOBAL to use a common conditional uop * Remove conditional stack effects from LOAD_ATTR specializations * Replace conditional stack effects in LOAD_ATTR with a 0 or 1 sized array. * Remove conditional stack effects from CALL_FUNCTION_EX
2025-01-27 16:24:48 +00:00
oparg = CURRENT_OPARG();
GH-131498: Remove conditional stack effects (GH-131499) * Adds some missing #includes
2025-03-20 15:39:38 +00:00
null = &stack_pointer[0];
if (oparg & 1) {
null[0] = PyStackRef_NULL;
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(0);
GH-128914: Remove all but one conditional stack effects (GH-129226) * Remove all 'if (0)' and 'if (1)' conditional stack effects * Use array instead of conditional for BUILD_SLICE args * Refactor LOAD_GLOBAL to use a common conditional uop * Remove conditional stack effects from LOAD_ATTR specializations * Replace conditional stack effects in LOAD_ATTR with a 0 or 1 sized array. * Remove conditional stack effects from CALL_FUNCTION_EX
2025-01-27 16:24:48 +00:00
stack_pointer += (oparg & 1);
GH-141794: Limit size of generated machine code. (GH-142228) * Factor out bodies of the largest uops, to reduce jit code size. * Factor out common assert, also reducing jit code size. * Limit size of jitted code for a single executor to 1MB.
2025-12-03 17:43:35 +00:00
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _GUARD_GLOBALS_VERSION_r00: {
CHECK_CURRENT_CACHED_VALUES(0);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-140683: JIT: Improve machine code for loading smaller constants on AArch64. (GH-142511) * Use movz and movk instructions for loading 16 and 32 bit operands and oparg. * Loading of 64 bit operands is unchanged.
2025-12-11 12:33:39 +00:00
uint16_t version = (uint16_t)CURRENT_OPERAND0_16();
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
PyDictObject *dict = (PyDictObject *)GLOBALS();
if (!PyDict_CheckExact(dict)) {
UOP_STAT_INC(uopcode, miss);
SET_CURRENT_CACHED_VALUES(0);
JUMP_TO_JUMP_TARGET();
}
PyDictKeysObject *keys = FT_ATOMIC_LOAD_PTR_ACQUIRE(dict->ma_keys);
if (FT_ATOMIC_LOAD_UINT32_RELAXED(keys->dk_version) != version) {
UOP_STAT_INC(uopcode, miss);
SET_CURRENT_CACHED_VALUES(0);
JUMP_TO_JUMP_TARGET();
}
assert(DK_IS_UNICODE(keys));
SET_CURRENT_CACHED_VALUES(0);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _GUARD_GLOBALS_VERSION_r11: {
CHECK_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef _stack_item_0 = _tos_cache0;
GH-140683: JIT: Improve machine code for loading smaller constants on AArch64. (GH-142511) * Use movz and movk instructions for loading 16 and 32 bit operands and oparg. * Loading of 64 bit operands is unchanged.
2025-12-11 12:33:39 +00:00
uint16_t version = (uint16_t)CURRENT_OPERAND0_16();
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
PyDictObject *dict = (PyDictObject *)GLOBALS();
if (!PyDict_CheckExact(dict)) {
UOP_STAT_INC(uopcode, miss);
GH-142718: JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. (GH-142762) JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. * Make sure that stack is in correct state at side exits with TOS cached values * Simplify choice of cached items for side exits
2025-12-15 19:01:29 +00:00
_tos_cache0 = _stack_item_0;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(1);
JUMP_TO_JUMP_TARGET();
}
PyDictKeysObject *keys = FT_ATOMIC_LOAD_PTR_ACQUIRE(dict->ma_keys);
if (FT_ATOMIC_LOAD_UINT32_RELAXED(keys->dk_version) != version) {
UOP_STAT_INC(uopcode, miss);
GH-142718: JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. (GH-142762) JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. * Make sure that stack is in correct state at side exits with TOS cached values * Simplify choice of cached items for side exits
2025-12-15 19:01:29 +00:00
_tos_cache0 = _stack_item_0;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(1);
JUMP_TO_JUMP_TARGET();
}
assert(DK_IS_UNICODE(keys));
_tos_cache0 = _stack_item_0;
SET_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _GUARD_GLOBALS_VERSION_r22: {
CHECK_CURRENT_CACHED_VALUES(2);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef _stack_item_0 = _tos_cache0;
_PyStackRef _stack_item_1 = _tos_cache1;
GH-140683: JIT: Improve machine code for loading smaller constants on AArch64. (GH-142511) * Use movz and movk instructions for loading 16 and 32 bit operands and oparg. * Loading of 64 bit operands is unchanged.
2025-12-11 12:33:39 +00:00
uint16_t version = (uint16_t)CURRENT_OPERAND0_16();
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
PyDictObject *dict = (PyDictObject *)GLOBALS();
if (!PyDict_CheckExact(dict)) {
UOP_STAT_INC(uopcode, miss);
GH-142718: JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. (GH-142762) JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. * Make sure that stack is in correct state at side exits with TOS cached values * Simplify choice of cached items for side exits
2025-12-15 19:01:29 +00:00
_tos_cache1 = _stack_item_1;
_tos_cache0 = _stack_item_0;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(2);
JUMP_TO_JUMP_TARGET();
}
PyDictKeysObject *keys = FT_ATOMIC_LOAD_PTR_ACQUIRE(dict->ma_keys);
if (FT_ATOMIC_LOAD_UINT32_RELAXED(keys->dk_version) != version) {
UOP_STAT_INC(uopcode, miss);
GH-142718: JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. (GH-142762) JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. * Make sure that stack is in correct state at side exits with TOS cached values * Simplify choice of cached items for side exits
2025-12-15 19:01:29 +00:00
_tos_cache1 = _stack_item_1;
_tos_cache0 = _stack_item_0;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(2);
JUMP_TO_JUMP_TARGET();
}
assert(DK_IS_UNICODE(keys));
_tos_cache1 = _stack_item_1;
_tos_cache0 = _stack_item_0;
SET_CURRENT_CACHED_VALUES(2);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-104584: Allow unspecialized instructions in superblocks (#106497) This adds several of unspecialized opcodes to superblocks: TO_BOOL, BINARY_SUBSCR, STORE_SUBSCR, UNPACK_SEQUENCE, LOAD_GLOBAL, LOAD_ATTR, COMPARE_OP, BINARY_OP. While we may not want that eventually, for now this helps finding bugs. There is a rudimentary test checking for UNPACK_SEQUENCE. Once we're ready to undo this, that would be simple: just replace the call to variable_used_unspecialized with a call to variable_used (as shown in a comment). Or add individual opcdes to FORBIDDEN_NAMES_IN_UOPS.
2023-07-07 11:03:27 -07:00
break;
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
case _GUARD_GLOBALS_VERSION_r33: {
CHECK_CURRENT_CACHED_VALUES(3);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef _stack_item_0 = _tos_cache0;
_PyStackRef _stack_item_1 = _tos_cache1;
_PyStackRef _stack_item_2 = _tos_cache2;
GH-140683: JIT: Improve machine code for loading smaller constants on AArch64. (GH-142511) * Use movz and movk instructions for loading 16 and 32 bit operands and oparg. * Loading of 64 bit operands is unchanged.
2025-12-11 12:33:39 +00:00
uint16_t version = (uint16_t)CURRENT_OPERAND0_16();
GH-104909: Break LOAD_GLOBAL specializations in micro-ops. (GH-106677)
2023-07-12 14:34:14 +01:00
PyDictObject *dict = (PyDictObject *)GLOBALS();
GH-117457: Correct pystats uop "miss" counts (GH-117477)
2024-04-04 18:49:18 -04:00
if (!PyDict_CheckExact(dict)) {
UOP_STAT_INC(uopcode, miss);
GH-142718: JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. (GH-142762) JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. * Make sure that stack is in correct state at side exits with TOS cached values * Simplify choice of cached items for side exits
2025-12-15 19:01:29 +00:00
_tos_cache2 = _stack_item_2;
_tos_cache1 = _stack_item_1;
_tos_cache0 = _stack_item_0;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(3);
GH-117457: Correct pystats uop "miss" counts (GH-117477)
2024-04-04 18:49:18 -04:00
JUMP_TO_JUMP_TARGET();
}
gh-115999: Specialize `LOAD_GLOBAL` in free-threaded builds (#126607) Enable specialization of LOAD_GLOBAL in free-threaded builds. Thread-safety of specialization in free-threaded builds is provided by the following: A critical section is held on both the globals and builtins objects during specialization. This ensures we get an atomic view of both builtins and globals during specialization. Generation of new keys versions is made atomic in free-threaded builds. Existing helpers are used to atomically modify the opcode. Thread-safety of specialized instructions in free-threaded builds is provided by the following: Relaxed atomics are used when loading and storing dict keys versions. This avoids potential data races as the dict keys versions are read without holding the dictionary's per-object lock in version guards. Dicts keys objects are passed from keys version guards to the downstream uops. This ensures that we are loading from the correct offset in the keys object. Once a unicode key has been stored in a keys object for a combined dictionary in free-threaded builds, the offset that it is stored in will never be reused for a different key. Once the version guard passes, we know that we are reading from the correct offset. The dictionary read fast-path is used to read values from the dictionary once we know the correct offset.
2024-11-21 11:22:21 -08:00
PyDictKeysObject *keys = FT_ATOMIC_LOAD_PTR_ACQUIRE(dict->ma_keys);
if (FT_ATOMIC_LOAD_UINT32_RELAXED(keys->dk_version) != version) {
GH-117457: Correct pystats uop "miss" counts (GH-117477)
2024-04-04 18:49:18 -04:00
UOP_STAT_INC(uopcode, miss);
GH-142718: JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. (GH-142762) JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. * Make sure that stack is in correct state at side exits with TOS cached values * Simplify choice of cached items for side exits
2025-12-15 19:01:29 +00:00
_tos_cache2 = _stack_item_2;
_tos_cache1 = _stack_item_1;
_tos_cache0 = _stack_item_0;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(3);
GH-117457: Correct pystats uop "miss" counts (GH-117477)
2024-04-04 18:49:18 -04:00
JUMP_TO_JUMP_TARGET();
}
gh-115999: Specialize `LOAD_GLOBAL` in free-threaded builds (#126607) Enable specialization of LOAD_GLOBAL in free-threaded builds. Thread-safety of specialization in free-threaded builds is provided by the following: A critical section is held on both the globals and builtins objects during specialization. This ensures we get an atomic view of both builtins and globals during specialization. Generation of new keys versions is made atomic in free-threaded builds. Existing helpers are used to atomically modify the opcode. Thread-safety of specialized instructions in free-threaded builds is provided by the following: Relaxed atomics are used when loading and storing dict keys versions. This avoids potential data races as the dict keys versions are read without holding the dictionary's per-object lock in version guards. Dicts keys objects are passed from keys version guards to the downstream uops. This ensures that we are loading from the correct offset in the keys object. Once a unicode key has been stored in a keys object for a combined dictionary in free-threaded builds, the offset that it is stored in will never be reused for a different key. Once the version guard passes, we know that we are reading from the correct offset. The dictionary read fast-path is used to read values from the dictionary once we know the correct offset.
2024-11-21 11:22:21 -08:00
assert(DK_IS_UNICODE(keys));
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_tos_cache2 = _stack_item_2;
_tos_cache1 = _stack_item_1;
_tos_cache0 = _stack_item_0;
SET_CURRENT_CACHED_VALUES(3);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-104909: Break LOAD_GLOBAL specializations in micro-ops. (GH-106677)
2023-07-12 14:34:14 +01:00
break;
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
case _LOAD_GLOBAL_MODULE_r01: {
CHECK_CURRENT_CACHED_VALUES(0);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-130296: Avoid stack transients in four instructions. (GH-130310) * Combine _GUARD_GLOBALS_VERSION_PUSH_KEYS and _LOAD_GLOBAL_MODULE_FROM_KEYS into _LOAD_GLOBAL_MODULE * Combine _GUARD_BUILTINS_VERSION_PUSH_KEYS and _LOAD_GLOBAL_BUILTINS_FROM_KEYS into _LOAD_GLOBAL_BUILTINS * Combine _CHECK_ATTR_MODULE_PUSH_KEYS and _LOAD_ATTR_MODULE_FROM_KEYS into _LOAD_ATTR_MODULE * Remove stack transient in LOAD_ATTR_WITH_HINT
2025-02-28 18:00:38 +00:00
_PyStackRef res;
GH-140683: JIT: Improve machine code for loading smaller constants on AArch64. (GH-142511) * Use movz and movk instructions for loading 16 and 32 bit operands and oparg. * Loading of 64 bit operands is unchanged.
2025-12-11 12:33:39 +00:00
uint16_t version = (uint16_t)CURRENT_OPERAND0_16();
uint16_t index = (uint16_t)CURRENT_OPERAND1_16();
gh-115999: Refactor `LOAD_GLOBAL` specializations to avoid reloading {globals, builtins} keys (gh-124953) Each of the `LOAD_GLOBAL` specializations is implemented roughly as: 1. Load keys version. 2. Load cached keys version. 3. Deopt if (1) and (2) don't match. 4. Load keys. 5. Load cached index into keys. 6. Load object from (4) at offset from (5). This is not thread-safe in free-threaded builds; the keys object may be replaced in between steps (3) and (4). This change refactors the specializations to avoid reloading the keys object and instead pass the keys object from guards to be consumed by downstream uops.
2024-10-09 08:18:25 -07:00
PyDictObject *dict = (PyDictObject *)GLOBALS();
if (!PyDict_CheckExact(dict)) {
UOP_STAT_INC(uopcode, miss);
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(0);
gh-115999: Refactor `LOAD_GLOBAL` specializations to avoid reloading {globals, builtins} keys (gh-124953) Each of the `LOAD_GLOBAL` specializations is implemented roughly as: 1. Load keys version. 2. Load cached keys version. 3. Deopt if (1) and (2) don't match. 4. Load keys. 5. Load cached index into keys. 6. Load object from (4) at offset from (5). This is not thread-safe in free-threaded builds; the keys object may be replaced in between steps (3) and (4). This change refactors the specializations to avoid reloading the keys object and instead pass the keys object from guards to be consumed by downstream uops.
2024-10-09 08:18:25 -07:00
JUMP_TO_JUMP_TARGET();
}
gh-115999: Specialize `LOAD_GLOBAL` in free-threaded builds (#126607) Enable specialization of LOAD_GLOBAL in free-threaded builds. Thread-safety of specialization in free-threaded builds is provided by the following: A critical section is held on both the globals and builtins objects during specialization. This ensures we get an atomic view of both builtins and globals during specialization. Generation of new keys versions is made atomic in free-threaded builds. Existing helpers are used to atomically modify the opcode. Thread-safety of specialized instructions in free-threaded builds is provided by the following: Relaxed atomics are used when loading and storing dict keys versions. This avoids potential data races as the dict keys versions are read without holding the dictionary's per-object lock in version guards. Dicts keys objects are passed from keys version guards to the downstream uops. This ensures that we are loading from the correct offset in the keys object. Once a unicode key has been stored in a keys object for a combined dictionary in free-threaded builds, the offset that it is stored in will never be reused for a different key. Once the version guard passes, we know that we are reading from the correct offset. The dictionary read fast-path is used to read values from the dictionary once we know the correct offset.
2024-11-21 11:22:21 -08:00
PyDictKeysObject *keys = FT_ATOMIC_LOAD_PTR_ACQUIRE(dict->ma_keys);
if (FT_ATOMIC_LOAD_UINT32_RELAXED(keys->dk_version) != version) {
gh-115999: Refactor `LOAD_GLOBAL` specializations to avoid reloading {globals, builtins} keys (gh-124953) Each of the `LOAD_GLOBAL` specializations is implemented roughly as: 1. Load keys version. 2. Load cached keys version. 3. Deopt if (1) and (2) don't match. 4. Load keys. 5. Load cached index into keys. 6. Load object from (4) at offset from (5). This is not thread-safe in free-threaded builds; the keys object may be replaced in between steps (3) and (4). This change refactors the specializations to avoid reloading the keys object and instead pass the keys object from guards to be consumed by downstream uops.
2024-10-09 08:18:25 -07:00
UOP_STAT_INC(uopcode, miss);
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(0);
gh-115999: Refactor `LOAD_GLOBAL` specializations to avoid reloading {globals, builtins} keys (gh-124953) Each of the `LOAD_GLOBAL` specializations is implemented roughly as: 1. Load keys version. 2. Load cached keys version. 3. Deopt if (1) and (2) don't match. 4. Load keys. 5. Load cached index into keys. 6. Load object from (4) at offset from (5). This is not thread-safe in free-threaded builds; the keys object may be replaced in between steps (3) and (4). This change refactors the specializations to avoid reloading the keys object and instead pass the keys object from guards to be consumed by downstream uops.
2024-10-09 08:18:25 -07:00
JUMP_TO_JUMP_TARGET();
}
GH-130296: Avoid stack transients in four instructions. (GH-130310) * Combine _GUARD_GLOBALS_VERSION_PUSH_KEYS and _LOAD_GLOBAL_MODULE_FROM_KEYS into _LOAD_GLOBAL_MODULE * Combine _GUARD_BUILTINS_VERSION_PUSH_KEYS and _LOAD_GLOBAL_BUILTINS_FROM_KEYS into _LOAD_GLOBAL_BUILTINS * Combine _CHECK_ATTR_MODULE_PUSH_KEYS and _LOAD_ATTR_MODULE_FROM_KEYS into _LOAD_ATTR_MODULE * Remove stack transient in LOAD_ATTR_WITH_HINT
2025-02-28 18:00:38 +00:00
assert(DK_IS_UNICODE(keys));
PyDictUnicodeEntry *entries = DK_UNICODE_ENTRIES(keys);
assert(index < DK_SIZE(keys));
gh-115999: Specialize `LOAD_GLOBAL` in free-threaded builds (#126607) Enable specialization of LOAD_GLOBAL in free-threaded builds. Thread-safety of specialization in free-threaded builds is provided by the following: A critical section is held on both the globals and builtins objects during specialization. This ensures we get an atomic view of both builtins and globals during specialization. Generation of new keys versions is made atomic in free-threaded builds. Existing helpers are used to atomically modify the opcode. Thread-safety of specialized instructions in free-threaded builds is provided by the following: Relaxed atomics are used when loading and storing dict keys versions. This avoids potential data races as the dict keys versions are read without holding the dictionary's per-object lock in version guards. Dicts keys objects are passed from keys version guards to the downstream uops. This ensures that we are loading from the correct offset in the keys object. Once a unicode key has been stored in a keys object for a combined dictionary in free-threaded builds, the offset that it is stored in will never be reused for a different key. Once the version guard passes, we know that we are reading from the correct offset. The dictionary read fast-path is used to read values from the dictionary once we know the correct offset.
2024-11-21 11:22:21 -08:00
PyObject *res_o = FT_ATOMIC_LOAD_PTR_RELAXED(entries[index].me_value);
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
if (res_o == NULL) {
GH-117457: Correct pystats uop "miss" counts (GH-117477)
2024-04-04 18:49:18 -04:00
UOP_STAT_INC(uopcode, miss);
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(0);
GH-117457: Correct pystats uop "miss" counts (GH-117477)
2024-04-04 18:49:18 -04:00
JUMP_TO_JUMP_TARGET();
}
gh-115999: Specialize `LOAD_GLOBAL` in free-threaded builds (#126607) Enable specialization of LOAD_GLOBAL in free-threaded builds. Thread-safety of specialization in free-threaded builds is provided by the following: A critical section is held on both the globals and builtins objects during specialization. This ensures we get an atomic view of both builtins and globals during specialization. Generation of new keys versions is made atomic in free-threaded builds. Existing helpers are used to atomically modify the opcode. Thread-safety of specialized instructions in free-threaded builds is provided by the following: Relaxed atomics are used when loading and storing dict keys versions. This avoids potential data races as the dict keys versions are read without holding the dictionary's per-object lock in version guards. Dicts keys objects are passed from keys version guards to the downstream uops. This ensures that we are loading from the correct offset in the keys object. Once a unicode key has been stored in a keys object for a combined dictionary in free-threaded builds, the offset that it is stored in will never be reused for a different key. Once the version guard passes, we know that we are reading from the correct offset. The dictionary read fast-path is used to read values from the dictionary once we know the correct offset.
2024-11-21 11:22:21 -08:00
#if Py_GIL_DISABLED
int increfed = _Py_TryIncrefCompareStackRef(&entries[index].me_value, res_o, &res);
if (!increfed) {
UOP_STAT_INC(uopcode, miss);
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(0);
gh-115999: Specialize `LOAD_GLOBAL` in free-threaded builds (#126607) Enable specialization of LOAD_GLOBAL in free-threaded builds. Thread-safety of specialization in free-threaded builds is provided by the following: A critical section is held on both the globals and builtins objects during specialization. This ensures we get an atomic view of both builtins and globals during specialization. Generation of new keys versions is made atomic in free-threaded builds. Existing helpers are used to atomically modify the opcode. Thread-safety of specialized instructions in free-threaded builds is provided by the following: Relaxed atomics are used when loading and storing dict keys versions. This avoids potential data races as the dict keys versions are read without holding the dictionary's per-object lock in version guards. Dicts keys objects are passed from keys version guards to the downstream uops. This ensures that we are loading from the correct offset in the keys object. Once a unicode key has been stored in a keys object for a combined dictionary in free-threaded builds, the offset that it is stored in will never be reused for a different key. Once the version guard passes, we know that we are reading from the correct offset. The dictionary read fast-path is used to read values from the dictionary once we know the correct offset.
2024-11-21 11:22:21 -08:00
JUMP_TO_JUMP_TARGET();
}
#else
GH-130296: Avoid stack transients in four instructions. (GH-130310) * Combine _GUARD_GLOBALS_VERSION_PUSH_KEYS and _LOAD_GLOBAL_MODULE_FROM_KEYS into _LOAD_GLOBAL_MODULE * Combine _GUARD_BUILTINS_VERSION_PUSH_KEYS and _LOAD_GLOBAL_BUILTINS_FROM_KEYS into _LOAD_GLOBAL_BUILTINS * Combine _CHECK_ATTR_MODULE_PUSH_KEYS and _LOAD_ATTR_MODULE_FROM_KEYS into _LOAD_ATTR_MODULE * Remove stack transient in LOAD_ATTR_WITH_HINT
2025-02-28 18:00:38 +00:00
res = PyStackRef_FromPyObjectNew(res_o);
gh-115999: Specialize `LOAD_GLOBAL` in free-threaded builds (#126607) Enable specialization of LOAD_GLOBAL in free-threaded builds. Thread-safety of specialization in free-threaded builds is provided by the following: A critical section is held on both the globals and builtins objects during specialization. This ensures we get an atomic view of both builtins and globals during specialization. Generation of new keys versions is made atomic in free-threaded builds. Existing helpers are used to atomically modify the opcode. Thread-safety of specialized instructions in free-threaded builds is provided by the following: Relaxed atomics are used when loading and storing dict keys versions. This avoids potential data races as the dict keys versions are read without holding the dictionary's per-object lock in version guards. Dicts keys objects are passed from keys version guards to the downstream uops. This ensures that we are loading from the correct offset in the keys object. Once a unicode key has been stored in a keys object for a combined dictionary in free-threaded builds, the offset that it is stored in will never be reused for a different key. Once the version guard passes, we know that we are reading from the correct offset. The dictionary read fast-path is used to read values from the dictionary once we know the correct offset.
2024-11-21 11:22:21 -08:00
#endif
gh-106603: Make uop struct a triple (opcode, oparg, operand) (#106794)
2023-07-17 12:12:33 -07:00
STAT_INC(LOAD_GLOBAL, hit);
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_tos_cache0 = res;
_tos_cache1 = PyStackRef_ZERO_BITS;
_tos_cache2 = PyStackRef_ZERO_BITS;
SET_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-106603: Make uop struct a triple (opcode, oparg, operand) (#106794)
2023-07-17 12:12:33 -07:00
break;
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
case _LOAD_GLOBAL_BUILTINS_r01: {
CHECK_CURRENT_CACHED_VALUES(0);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
_PyStackRef res;
GH-140683: JIT: Improve machine code for loading smaller constants on AArch64. (GH-142511) * Use movz and movk instructions for loading 16 and 32 bit operands and oparg. * Loading of 64 bit operands is unchanged.
2025-12-11 12:33:39 +00:00
uint16_t version = (uint16_t)CURRENT_OPERAND0_16();
uint16_t index = (uint16_t)CURRENT_OPERAND1_16();
GH-130296: Avoid stack transients in four instructions. (GH-130310) * Combine _GUARD_GLOBALS_VERSION_PUSH_KEYS and _LOAD_GLOBAL_MODULE_FROM_KEYS into _LOAD_GLOBAL_MODULE * Combine _GUARD_BUILTINS_VERSION_PUSH_KEYS and _LOAD_GLOBAL_BUILTINS_FROM_KEYS into _LOAD_GLOBAL_BUILTINS * Combine _CHECK_ATTR_MODULE_PUSH_KEYS and _LOAD_ATTR_MODULE_FROM_KEYS into _LOAD_ATTR_MODULE * Remove stack transient in LOAD_ATTR_WITH_HINT
2025-02-28 18:00:38 +00:00
PyDictObject *dict = (PyDictObject *)BUILTINS();
if (!PyDict_CheckExact(dict)) {
UOP_STAT_INC(uopcode, miss);
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(0);
GH-130296: Avoid stack transients in four instructions. (GH-130310) * Combine _GUARD_GLOBALS_VERSION_PUSH_KEYS and _LOAD_GLOBAL_MODULE_FROM_KEYS into _LOAD_GLOBAL_MODULE * Combine _GUARD_BUILTINS_VERSION_PUSH_KEYS and _LOAD_GLOBAL_BUILTINS_FROM_KEYS into _LOAD_GLOBAL_BUILTINS * Combine _CHECK_ATTR_MODULE_PUSH_KEYS and _LOAD_ATTR_MODULE_FROM_KEYS into _LOAD_ATTR_MODULE * Remove stack transient in LOAD_ATTR_WITH_HINT
2025-02-28 18:00:38 +00:00
JUMP_TO_JUMP_TARGET();
}
PyDictKeysObject *keys = FT_ATOMIC_LOAD_PTR_ACQUIRE(dict->ma_keys);
if (FT_ATOMIC_LOAD_UINT32_RELAXED(keys->dk_version) != version) {
UOP_STAT_INC(uopcode, miss);
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(0);
GH-130296: Avoid stack transients in four instructions. (GH-130310) * Combine _GUARD_GLOBALS_VERSION_PUSH_KEYS and _LOAD_GLOBAL_MODULE_FROM_KEYS into _LOAD_GLOBAL_MODULE * Combine _GUARD_BUILTINS_VERSION_PUSH_KEYS and _LOAD_GLOBAL_BUILTINS_FROM_KEYS into _LOAD_GLOBAL_BUILTINS * Combine _CHECK_ATTR_MODULE_PUSH_KEYS and _LOAD_ATTR_MODULE_FROM_KEYS into _LOAD_ATTR_MODULE * Remove stack transient in LOAD_ATTR_WITH_HINT
2025-02-28 18:00:38 +00:00
JUMP_TO_JUMP_TARGET();
}
assert(DK_IS_UNICODE(keys));
PyDictUnicodeEntry *entries = DK_UNICODE_ENTRIES(keys);
gh-115999: Specialize `LOAD_GLOBAL` in free-threaded builds (#126607) Enable specialization of LOAD_GLOBAL in free-threaded builds. Thread-safety of specialization in free-threaded builds is provided by the following: A critical section is held on both the globals and builtins objects during specialization. This ensures we get an atomic view of both builtins and globals during specialization. Generation of new keys versions is made atomic in free-threaded builds. Existing helpers are used to atomically modify the opcode. Thread-safety of specialized instructions in free-threaded builds is provided by the following: Relaxed atomics are used when loading and storing dict keys versions. This avoids potential data races as the dict keys versions are read without holding the dictionary's per-object lock in version guards. Dicts keys objects are passed from keys version guards to the downstream uops. This ensures that we are loading from the correct offset in the keys object. Once a unicode key has been stored in a keys object for a combined dictionary in free-threaded builds, the offset that it is stored in will never be reused for a different key. Once the version guard passes, we know that we are reading from the correct offset. The dictionary read fast-path is used to read values from the dictionary once we know the correct offset.
2024-11-21 11:22:21 -08:00
PyObject *res_o = FT_ATOMIC_LOAD_PTR_RELAXED(entries[index].me_value);
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
if (res_o == NULL) {
GH-117457: Correct pystats uop "miss" counts (GH-117477)
2024-04-04 18:49:18 -04:00
UOP_STAT_INC(uopcode, miss);
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(0);
GH-117457: Correct pystats uop "miss" counts (GH-117477)
2024-04-04 18:49:18 -04:00
JUMP_TO_JUMP_TARGET();
}
gh-115999: Specialize `LOAD_GLOBAL` in free-threaded builds (#126607) Enable specialization of LOAD_GLOBAL in free-threaded builds. Thread-safety of specialization in free-threaded builds is provided by the following: A critical section is held on both the globals and builtins objects during specialization. This ensures we get an atomic view of both builtins and globals during specialization. Generation of new keys versions is made atomic in free-threaded builds. Existing helpers are used to atomically modify the opcode. Thread-safety of specialized instructions in free-threaded builds is provided by the following: Relaxed atomics are used when loading and storing dict keys versions. This avoids potential data races as the dict keys versions are read without holding the dictionary's per-object lock in version guards. Dicts keys objects are passed from keys version guards to the downstream uops. This ensures that we are loading from the correct offset in the keys object. Once a unicode key has been stored in a keys object for a combined dictionary in free-threaded builds, the offset that it is stored in will never be reused for a different key. Once the version guard passes, we know that we are reading from the correct offset. The dictionary read fast-path is used to read values from the dictionary once we know the correct offset.
2024-11-21 11:22:21 -08:00
#if Py_GIL_DISABLED
int increfed = _Py_TryIncrefCompareStackRef(&entries[index].me_value, res_o, &res);
if (!increfed) {
UOP_STAT_INC(uopcode, miss);
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(0);
gh-115999: Specialize `LOAD_GLOBAL` in free-threaded builds (#126607) Enable specialization of LOAD_GLOBAL in free-threaded builds. Thread-safety of specialization in free-threaded builds is provided by the following: A critical section is held on both the globals and builtins objects during specialization. This ensures we get an atomic view of both builtins and globals during specialization. Generation of new keys versions is made atomic in free-threaded builds. Existing helpers are used to atomically modify the opcode. Thread-safety of specialized instructions in free-threaded builds is provided by the following: Relaxed atomics are used when loading and storing dict keys versions. This avoids potential data races as the dict keys versions are read without holding the dictionary's per-object lock in version guards. Dicts keys objects are passed from keys version guards to the downstream uops. This ensures that we are loading from the correct offset in the keys object. Once a unicode key has been stored in a keys object for a combined dictionary in free-threaded builds, the offset that it is stored in will never be reused for a different key. Once the version guard passes, we know that we are reading from the correct offset. The dictionary read fast-path is used to read values from the dictionary once we know the correct offset.
2024-11-21 11:22:21 -08:00
JUMP_TO_JUMP_TARGET();
}
#else
GH-130296: Avoid stack transients in four instructions. (GH-130310) * Combine _GUARD_GLOBALS_VERSION_PUSH_KEYS and _LOAD_GLOBAL_MODULE_FROM_KEYS into _LOAD_GLOBAL_MODULE * Combine _GUARD_BUILTINS_VERSION_PUSH_KEYS and _LOAD_GLOBAL_BUILTINS_FROM_KEYS into _LOAD_GLOBAL_BUILTINS * Combine _CHECK_ATTR_MODULE_PUSH_KEYS and _LOAD_ATTR_MODULE_FROM_KEYS into _LOAD_ATTR_MODULE * Remove stack transient in LOAD_ATTR_WITH_HINT
2025-02-28 18:00:38 +00:00
res = PyStackRef_FromPyObjectNew(res_o);
gh-115999: Specialize `LOAD_GLOBAL` in free-threaded builds (#126607) Enable specialization of LOAD_GLOBAL in free-threaded builds. Thread-safety of specialization in free-threaded builds is provided by the following: A critical section is held on both the globals and builtins objects during specialization. This ensures we get an atomic view of both builtins and globals during specialization. Generation of new keys versions is made atomic in free-threaded builds. Existing helpers are used to atomically modify the opcode. Thread-safety of specialized instructions in free-threaded builds is provided by the following: Relaxed atomics are used when loading and storing dict keys versions. This avoids potential data races as the dict keys versions are read without holding the dictionary's per-object lock in version guards. Dicts keys objects are passed from keys version guards to the downstream uops. This ensures that we are loading from the correct offset in the keys object. Once a unicode key has been stored in a keys object for a combined dictionary in free-threaded builds, the offset that it is stored in will never be reused for a different key. Once the version guard passes, we know that we are reading from the correct offset. The dictionary read fast-path is used to read values from the dictionary once we know the correct offset.
2024-11-21 11:22:21 -08:00
#endif
gh-106603: Make uop struct a triple (opcode, oparg, operand) (#106794)
2023-07-17 12:12:33 -07:00
STAT_INC(LOAD_GLOBAL, hit);
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_tos_cache0 = res;
_tos_cache1 = PyStackRef_ZERO_BITS;
_tos_cache2 = PyStackRef_ZERO_BITS;
SET_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-106603: Make uop struct a triple (opcode, oparg, operand) (#106794)
2023-07-17 12:12:33 -07:00
break;
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
case _DELETE_FAST_r00: {
CHECK_CURRENT_CACHED_VALUES(0);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-112287: Speed up Tier 2 (uop) interpreter a little (#112286) This makes the Tier 2 interpreter a little faster. I calculated by about 3%, though I hesitate to claim an exact number. This starts by doubling the trace size limit (to 512), making it more likely that loops fit in a trace. The rest of the approach is to only load `oparg` and `operand` in cases that use them. The code generator know when these are used. For `oparg`, it will conditionally emit ``` oparg = CURRENT_OPARG(); ``` at the top of the case block. (The `oparg` variable may be referenced multiple times by the instructions code block, so it must be in a variable.) For `operand`, it will use `CURRENT_OPERAND()` directly instead of referencing the `operand` variable, which no longer exists. (There is only one place where this will be used.)
2023-11-20 11:25:32 -08:00
oparg = CURRENT_OPARG();
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
_PyStackRef v = GETLOCAL(oparg);
if (PyStackRef_IsNull(v)) {
GH-119866: Spill the stack around escaping calls. (GH-124392) * Spill the evaluation around escaping calls in the generated interpreter and JIT. * The code generator tracks live, cached values so they can be saved to memory when needed. * Spills the stack pointer around escaping calls, so that the exact stack is visible to the cycle GC.
2024-10-07 14:56:39 +01:00
_PyFrame_SetStackPointer(frame, stack_pointer);
GH-116422: Tier2 hot/cold splitting (GH-116813) Splits the "cold" path, deopts and exits, from the "hot" path, reducing the size of most jitted instructions, at the cost of slower exits.
2024-03-26 09:35:11 +00:00
_PyEval_FormatExcCheckArg(tstate, PyExc_UnboundLocalError,
UNBOUNDLOCAL_ERROR_MSG,
PyTuple_GetItem(_PyFrame_GetCode(frame)->co_localsplusnames, oparg)
);
GH-119866: Spill the stack around escaping calls. (GH-124392) * Spill the evaluation around escaping calls in the generated interpreter and JIT. * The code generator tracks live, cached values so they can be saved to memory when needed. * Spills the stack pointer around escaping calls, so that the exact stack is visible to the cycle GC.
2024-10-07 14:56:39 +01:00
stack_pointer = _PyFrame_GetStackPointer(frame);
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(0);
GH-128682: Make `PyStackRef_CLOSE` escaping. (GH-129404)
2025-02-03 12:41:32 +00:00
JUMP_TO_ERROR();
GH-116422: Tier2 hot/cold splitting (GH-116813) Splits the "cold" path, deopts and exits, from the "hot" path, reducing the size of most jitted instructions, at the cost of slower exits.
2024-03-26 09:35:11 +00:00
}
GH-128682: Mark two more macros as escaping. (GH-129645) Expand out SETLOCAL so that code generator can see the decref. Mark Py_CLEAR as escaping
2025-02-04 14:00:51 +00:00
_PyStackRef tmp = GETLOCAL(oparg);
GETLOCAL(oparg) = PyStackRef_NULL;
_PyFrame_SetStackPointer(frame, stack_pointer);
PyStackRef_XCLOSE(tmp);
stack_pointer = _PyFrame_GetStackPointer(frame);
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_tos_cache0 = PyStackRef_ZERO_BITS;
_tos_cache1 = PyStackRef_ZERO_BITS;
_tos_cache2 = PyStackRef_ZERO_BITS;
SET_CURRENT_CACHED_VALUES(0);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-106290: Fix edge cases around uops (#106319) - Tweak uops debugging output - Fix the bug from gh-106290 - Rename `SET_IP` to `SAVE_IP` (per https://github.com/faster-cpython/ideas/issues/558) - Add a `SAVE_IP` uop at the start of the trace (ditto) - Allow `unbound_local_error`; this gives us uops for `LOAD_FAST_CHECK`, `LOAD_CLOSURE`, and `DELETE_FAST` - Longer traces - Support `STORE_FAST_LOAD_FAST`, `STORE_FAST_STORE_FAST` - Add deps on pycore_uops.h to Makefile(.pre.in)
2023-07-03 13:05:11 -07:00
break;
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
case _MAKE_CELL_r00: {
CHECK_CURRENT_CACHED_VALUES(0);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-112287: Speed up Tier 2 (uop) interpreter a little (#112286) This makes the Tier 2 interpreter a little faster. I calculated by about 3%, though I hesitate to claim an exact number. This starts by doubling the trace size limit (to 512), making it more likely that loops fit in a trace. The rest of the approach is to only load `oparg` and `operand` in cases that use them. The code generator know when these are used. For `oparg`, it will conditionally emit ``` oparg = CURRENT_OPARG(); ``` at the top of the case block. (The `oparg` variable may be referenced multiple times by the instructions code block, so it must be in a variable.) For `operand`, it will use `CURRENT_OPERAND()` directly instead of referencing the `operand` variable, which no longer exists. (There is only one place where this will be used.)
2023-11-20 11:25:32 -08:00
oparg = CURRENT_OPARG();
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
PyObject *initial = PyStackRef_AsPyObjectBorrow(GETLOCAL(oparg));
GH-111485: Remove some special cases from the code generator and bytecodes.c (GH-111540)
2023-10-31 13:21:07 +00:00
PyObject *cell = PyCell_New(initial);
if (cell == NULL) {
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(0);
GH-116422: Tier2 hot/cold splitting (GH-116813) Splits the "cold" path, deopts and exits, from the "hot" path, reducing the size of most jitted instructions, at the cost of slower exits.
2024-03-26 09:35:11 +00:00
JUMP_TO_ERROR();
GH-111485: Remove some special cases from the code generator and bytecodes.c (GH-111540)
2023-10-31 13:21:07 +00:00
}
GH-128682: Mark two more macros as escaping. (GH-129645) Expand out SETLOCAL so that code generator can see the decref. Mark Py_CLEAR as escaping
2025-02-04 14:00:51 +00:00
_PyStackRef tmp = GETLOCAL(oparg);
GETLOCAL(oparg) = PyStackRef_FromPyObjectSteal(cell);
_PyFrame_SetStackPointer(frame, stack_pointer);
PyStackRef_XCLOSE(tmp);
stack_pointer = _PyFrame_GetStackPointer(frame);
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_tos_cache0 = PyStackRef_ZERO_BITS;
_tos_cache1 = PyStackRef_ZERO_BITS;
_tos_cache2 = PyStackRef_ZERO_BITS;
SET_CURRENT_CACHED_VALUES(0);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-111485: Remove some special cases from the code generator and bytecodes.c (GH-111540)
2023-10-31 13:21:07 +00:00
break;
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
case _DELETE_DEREF_r00: {
CHECK_CURRENT_CACHED_VALUES(0);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-112287: Speed up Tier 2 (uop) interpreter a little (#112286) This makes the Tier 2 interpreter a little faster. I calculated by about 3%, though I hesitate to claim an exact number. This starts by doubling the trace size limit (to 512), making it more likely that loops fit in a trace. The rest of the approach is to only load `oparg` and `operand` in cases that use them. The code generator know when these are used. For `oparg`, it will conditionally emit ``` oparg = CURRENT_OPARG(); ``` at the top of the case block. (The `oparg` variable may be referenced multiple times by the instructions code block, so it must be in a variable.) For `operand`, it will use `CURRENT_OPERAND()` directly instead of referencing the `operand` variable, which no longer exists. (There is only one place where this will be used.)
2023-11-20 11:25:32 -08:00
oparg = CURRENT_OPARG();
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
PyObject *cell = PyStackRef_AsPyObjectBorrow(GETLOCAL(oparg));
gh-117323: Make `cell` thread-safe in free-threaded builds (#117330) Use critical sections to lock around accesses to cell contents. The critical sections are no-ops in the default (with GIL) build.
2024-03-29 13:35:43 -04:00
PyObject *oldobj = PyCell_SwapTakeRef((PyCellObject *)cell, NULL);
gh-104584: Baby steps towards generating and executing traces (#105924) Added a new, experimental, tracing optimizer and interpreter (a.k.a. "tier 2"). This currently pessimizes, so don't use yet -- this is infrastructure so we can experiment with optimizing passes. To enable it, pass ``-Xuops`` or set ``PYTHONUOPS=1``. To get debug output, set ``PYTHONUOPSDEBUG=N`` where ``N`` is a debug level (0-4, where 0 is no debug output and 4 is excessively verbose). All of this code is likely to change dramatically before the 3.13 feature freeze. But this is a first step.
2023-06-26 19:02:57 -07:00
if (oldobj == NULL) {
GH-119866: Spill the stack around escaping calls. (GH-124392) * Spill the evaluation around escaping calls in the generated interpreter and JIT. * The code generator tracks live, cached values so they can be saved to memory when needed. * Spills the stack pointer around escaping calls, so that the exact stack is visible to the cycle GC.
2024-10-07 14:56:39 +01:00
_PyFrame_SetStackPointer(frame, stack_pointer);
GH-106701: Move _PyUopExecute to Python/executor.c (GH-106924)
2023-07-20 13:37:19 -07:00
_PyEval_FormatExcUnbound(tstate, _PyFrame_GetCode(frame), oparg);
GH-119866: Spill the stack around escaping calls. (GH-124392) * Spill the evaluation around escaping calls in the generated interpreter and JIT. * The code generator tracks live, cached values so they can be saved to memory when needed. * Spills the stack pointer around escaping calls, so that the exact stack is visible to the cycle GC.
2024-10-07 14:56:39 +01:00
stack_pointer = _PyFrame_GetStackPointer(frame);
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(0);
GH-116422: Tier2 hot/cold splitting (GH-116813) Splits the "cold" path, deopts and exits, from the "hot" path, reducing the size of most jitted instructions, at the cost of slower exits.
2024-03-26 09:35:11 +00:00
JUMP_TO_ERROR();
gh-104584: Baby steps towards generating and executing traces (#105924) Added a new, experimental, tracing optimizer and interpreter (a.k.a. "tier 2"). This currently pessimizes, so don't use yet -- this is infrastructure so we can experiment with optimizing passes. To enable it, pass ``-Xuops`` or set ``PYTHONUOPS=1``. To get debug output, set ``PYTHONUOPSDEBUG=N`` where ``N`` is a debug level (0-4, where 0 is no debug output and 4 is excessively verbose). All of this code is likely to change dramatically before the 3.13 feature freeze. But this is a first step.
2023-06-26 19:02:57 -07:00
}
GH-128682: Convert explicit loops closing arrays into `DECREF_INPUTS`. (GH-128822) * Mark Py_DECREF and Py_XDECREF as escaping * Remove explicit loops for clearing array inputs
2025-01-14 15:08:56 +00:00
_PyFrame_SetStackPointer(frame, stack_pointer);
gh-104584: Baby steps towards generating and executing traces (#105924) Added a new, experimental, tracing optimizer and interpreter (a.k.a. "tier 2"). This currently pessimizes, so don't use yet -- this is infrastructure so we can experiment with optimizing passes. To enable it, pass ``-Xuops`` or set ``PYTHONUOPS=1``. To get debug output, set ``PYTHONUOPSDEBUG=N`` where ``N`` is a debug level (0-4, where 0 is no debug output and 4 is excessively verbose). All of this code is likely to change dramatically before the 3.13 feature freeze. But this is a first step.
2023-06-26 19:02:57 -07:00
Py_DECREF(oldobj);
GH-128682: Convert explicit loops closing arrays into `DECREF_INPUTS`. (GH-128822) * Mark Py_DECREF and Py_XDECREF as escaping * Remove explicit loops for clearing array inputs
2025-01-14 15:08:56 +00:00
stack_pointer = _PyFrame_GetStackPointer(frame);
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_tos_cache0 = PyStackRef_ZERO_BITS;
_tos_cache1 = PyStackRef_ZERO_BITS;
_tos_cache2 = PyStackRef_ZERO_BITS;
SET_CURRENT_CACHED_VALUES(0);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-104584: Baby steps towards generating and executing traces (#105924) Added a new, experimental, tracing optimizer and interpreter (a.k.a. "tier 2"). This currently pessimizes, so don't use yet -- this is infrastructure so we can experiment with optimizing passes. To enable it, pass ``-Xuops`` or set ``PYTHONUOPS=1``. To get debug output, set ``PYTHONUOPSDEBUG=N`` where ``N`` is a debug level (0-4, where 0 is no debug output and 4 is excessively verbose). All of this code is likely to change dramatically before the 3.13 feature freeze. But this is a first step.
2023-06-26 19:02:57 -07:00
break;
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
case _LOAD_FROM_DICT_OR_DEREF_r11: {
CHECK_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
_PyStackRef class_dict_st;
_PyStackRef value;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef _stack_item_0 = _tos_cache0;
GH-111485: Factor out tier 2 code generation from the rest of the interpreter code generator (GH-112968)
2023-12-12 12:12:17 +00:00
oparg = CURRENT_OPARG();
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
class_dict_st = _stack_item_0;
gh-104584: Baby steps towards generating and executing traces (#105924) Added a new, experimental, tracing optimizer and interpreter (a.k.a. "tier 2"). This currently pessimizes, so don't use yet -- this is infrastructure so we can experiment with optimizing passes. To enable it, pass ``-Xuops`` or set ``PYTHONUOPS=1``. To get debug output, set ``PYTHONUOPSDEBUG=N`` where ``N`` is a debug level (0-4, where 0 is no debug output and 4 is excessively verbose). All of this code is likely to change dramatically before the 3.13 feature freeze. But this is a first step.
2023-06-26 19:02:57 -07:00
PyObject *name;
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
PyObject *class_dict = PyStackRef_AsPyObjectBorrow(class_dict_st);
gh-104584: Baby steps towards generating and executing traces (#105924) Added a new, experimental, tracing optimizer and interpreter (a.k.a. "tier 2"). This currently pessimizes, so don't use yet -- this is infrastructure so we can experiment with optimizing passes. To enable it, pass ``-Xuops`` or set ``PYTHONUOPS=1``. To get debug output, set ``PYTHONUOPSDEBUG=N`` where ``N`` is a debug level (0-4, where 0 is no debug output and 4 is excessively verbose). All of this code is likely to change dramatically before the 3.13 feature freeze. But this is a first step.
2023-06-26 19:02:57 -07:00
assert(class_dict);
assert(oparg >= 0 && oparg < _PyFrame_GetCode(frame)->co_nlocalsplus);
name = PyTuple_GET_ITEM(_PyFrame_GetCode(frame)->co_localsplusnames, oparg);
GH-139922: Tail calling for MSVC (VS 2026) (GH-143068) Co-authored-by: Ken Jin <28750310+Fidget-Spinner@users.noreply.github.com> Co-authored-by: Brandt Bucher <brandt@python.org> Co-authored-by: Hugo van Kemenade <1324225+hugovk@users.noreply.github.com>
2025-12-23 00:01:34 +01:00
int err;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
stack_pointer[0] = class_dict_st;
stack_pointer += 1;
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
GH-119866: Spill the stack around escaping calls. (GH-124392) * Spill the evaluation around escaping calls in the generated interpreter and JIT. * The code generator tracks live, cached values so they can be saved to memory when needed. * Spills the stack pointer around escaping calls, so that the exact stack is visible to the cycle GC.
2024-10-07 14:56:39 +01:00
_PyFrame_SetStackPointer(frame, stack_pointer);
GH-139922: Tail calling for MSVC (VS 2026) (GH-143068) Co-authored-by: Ken Jin <28750310+Fidget-Spinner@users.noreply.github.com> Co-authored-by: Brandt Bucher <brandt@python.org> Co-authored-by: Hugo van Kemenade <1324225+hugovk@users.noreply.github.com>
2025-12-23 00:01:34 +01:00
PyObject* value_o = _PyMapping_GetOptionalItem2(class_dict, name, &err);
GH-119866: Spill the stack around escaping calls. (GH-124392) * Spill the evaluation around escaping calls in the generated interpreter and JIT. * The code generator tracks live, cached values so they can be saved to memory when needed. * Spills the stack pointer around escaping calls, so that the exact stack is visible to the cycle GC.
2024-10-07 14:56:39 +01:00
stack_pointer = _PyFrame_GetStackPointer(frame);
GH-120024: Refactor code a bit so that escaping calls can be wrapped in spill code in code generator (GH-122693)
2024-08-06 08:40:39 +01:00
if (err < 0) {
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(0);
GH-116422: Tier2 hot/cold splitting (GH-116813) Splits the "cold" path, deopts and exits, from the "hot" path, reducing the size of most jitted instructions, at the cost of slower exits.
2024-03-26 09:35:11 +00:00
JUMP_TO_ERROR();
gh-104584: Baby steps towards generating and executing traces (#105924) Added a new, experimental, tracing optimizer and interpreter (a.k.a. "tier 2"). This currently pessimizes, so don't use yet -- this is infrastructure so we can experiment with optimizing passes. To enable it, pass ``-Xuops`` or set ``PYTHONUOPS=1``. To get debug output, set ``PYTHONUOPSDEBUG=N`` where ``N`` is a debug level (0-4, where 0 is no debug output and 4 is excessively verbose). All of this code is likely to change dramatically before the 3.13 feature freeze. But this is a first step.
2023-06-26 19:02:57 -07:00
}
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
if (!value_o) {
PyCellObject *cell = (PyCellObject *)PyStackRef_AsPyObjectBorrow(GETLOCAL(oparg));
value_o = PyCell_GetRef(cell);
if (value_o == NULL) {
GH-119866: Spill the stack around escaping calls. (GH-124392) * Spill the evaluation around escaping calls in the generated interpreter and JIT. * The code generator tracks live, cached values so they can be saved to memory when needed. * Spills the stack pointer around escaping calls, so that the exact stack is visible to the cycle GC.
2024-10-07 14:56:39 +01:00
_PyFrame_SetStackPointer(frame, stack_pointer);
GH-106701: Move _PyUopExecute to Python/executor.c (GH-106924)
2023-07-20 13:37:19 -07:00
_PyEval_FormatExcUnbound(tstate, _PyFrame_GetCode(frame), oparg);
GH-119866: Spill the stack around escaping calls. (GH-124392) * Spill the evaluation around escaping calls in the generated interpreter and JIT. * The code generator tracks live, cached values so they can be saved to memory when needed. * Spills the stack pointer around escaping calls, so that the exact stack is visible to the cycle GC.
2024-10-07 14:56:39 +01:00
stack_pointer = _PyFrame_GetStackPointer(frame);
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(0);
GH-116422: Tier2 hot/cold splitting (GH-116813) Splits the "cold" path, deopts and exits, from the "hot" path, reducing the size of most jitted instructions, at the cost of slower exits.
2024-03-26 09:35:11 +00:00
JUMP_TO_ERROR();
gh-104584: Baby steps towards generating and executing traces (#105924) Added a new, experimental, tracing optimizer and interpreter (a.k.a. "tier 2"). This currently pessimizes, so don't use yet -- this is infrastructure so we can experiment with optimizing passes. To enable it, pass ``-Xuops`` or set ``PYTHONUOPS=1``. To get debug output, set ``PYTHONUOPSDEBUG=N`` where ``N`` is a debug level (0-4, where 0 is no debug output and 4 is excessively verbose). All of this code is likely to change dramatically before the 3.13 feature freeze. But this is a first step.
2023-06-26 19:02:57 -07:00
}
}
GH-128682: Make `PyStackRef_CLOSE` escaping. (GH-129404)
2025-02-03 12:41:32 +00:00
stack_pointer += -1;
GH-141794: Limit size of generated machine code. (GH-142228) * Factor out bodies of the largest uops, to reduce jit code size. * Factor out common assert, also reducing jit code size. * Limit size of jitted code for a single executor to 1MB.
2025-12-03 17:43:35 +00:00
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
GH-128682: Make `PyStackRef_CLOSE` escaping. (GH-129404)
2025-02-03 12:41:32 +00:00
_PyFrame_SetStackPointer(frame, stack_pointer);
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
PyStackRef_CLOSE(class_dict_st);
GH-128682: Make `PyStackRef_CLOSE` escaping. (GH-129404)
2025-02-03 12:41:32 +00:00
stack_pointer = _PyFrame_GetStackPointer(frame);
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
value = PyStackRef_FromPyObjectSteal(value_o);
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_tos_cache0 = value;
_tos_cache1 = PyStackRef_ZERO_BITS;
_tos_cache2 = PyStackRef_ZERO_BITS;
SET_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-104584: Baby steps towards generating and executing traces (#105924) Added a new, experimental, tracing optimizer and interpreter (a.k.a. "tier 2"). This currently pessimizes, so don't use yet -- this is infrastructure so we can experiment with optimizing passes. To enable it, pass ``-Xuops`` or set ``PYTHONUOPS=1``. To get debug output, set ``PYTHONUOPSDEBUG=N`` where ``N`` is a debug level (0-4, where 0 is no debug output and 4 is excessively verbose). All of this code is likely to change dramatically before the 3.13 feature freeze. But this is a first step.
2023-06-26 19:02:57 -07:00
break;
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
case _LOAD_DEREF_r01: {
CHECK_CURRENT_CACHED_VALUES(0);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
_PyStackRef value;
GH-111485: Factor out tier 2 code generation from the rest of the interpreter code generator (GH-112968)
2023-12-12 12:12:17 +00:00
oparg = CURRENT_OPARG();
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
PyCellObject *cell = (PyCellObject *)PyStackRef_AsPyObjectBorrow(GETLOCAL(oparg));
gh-123358: Use `_PyStackRef` in `LOAD_DEREF` (gh-130064) Concurrent accesses from multiple threads to the same `cell` object did not scale well in the free-threaded build. Use `_PyStackRef` and optimistically avoid locking to improve scaling. With the locks around cell reads gone, some of the free threading tests were prone to starvation: the readers were able to run in a tight loop and the writer threads weren't scheduled frequently enough to make timely progress. Adjust the tests to avoid this. Co-authored-by: Donghee Na <donghee.na@python.org>
2025-03-26 12:08:20 -04:00
_PyFrame_SetStackPointer(frame, stack_pointer);
value = _PyCell_GetStackRef(cell);
stack_pointer = _PyFrame_GetStackPointer(frame);
if (PyStackRef_IsNull(value)) {
stack_pointer[0] = value;
stack_pointer += 1;
GH-141794: Limit size of generated machine code. (GH-142228) * Factor out bodies of the largest uops, to reduce jit code size. * Factor out common assert, also reducing jit code size. * Limit size of jitted code for a single executor to 1MB.
2025-12-03 17:43:35 +00:00
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
GH-119866: Spill the stack around escaping calls. (GH-124392) * Spill the evaluation around escaping calls in the generated interpreter and JIT. * The code generator tracks live, cached values so they can be saved to memory when needed. * Spills the stack pointer around escaping calls, so that the exact stack is visible to the cycle GC.
2024-10-07 14:56:39 +01:00
_PyFrame_SetStackPointer(frame, stack_pointer);
GH-106701: Move _PyUopExecute to Python/executor.c (GH-106924)
2023-07-20 13:37:19 -07:00
_PyEval_FormatExcUnbound(tstate, _PyFrame_GetCode(frame), oparg);
GH-119866: Spill the stack around escaping calls. (GH-124392) * Spill the evaluation around escaping calls in the generated interpreter and JIT. * The code generator tracks live, cached values so they can be saved to memory when needed. * Spills the stack pointer around escaping calls, so that the exact stack is visible to the cycle GC.
2024-10-07 14:56:39 +01:00
stack_pointer = _PyFrame_GetStackPointer(frame);
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(0);
GH-128682: Make `PyStackRef_CLOSE` escaping. (GH-129404)
2025-02-03 12:41:32 +00:00
JUMP_TO_ERROR();
gh-104584: Baby steps towards generating and executing traces (#105924) Added a new, experimental, tracing optimizer and interpreter (a.k.a. "tier 2"). This currently pessimizes, so don't use yet -- this is infrastructure so we can experiment with optimizing passes. To enable it, pass ``-Xuops`` or set ``PYTHONUOPS=1``. To get debug output, set ``PYTHONUOPSDEBUG=N`` where ``N`` is a debug level (0-4, where 0 is no debug output and 4 is excessively verbose). All of this code is likely to change dramatically before the 3.13 feature freeze. But this is a first step.
2023-06-26 19:02:57 -07:00
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_tos_cache0 = value;
_tos_cache1 = PyStackRef_ZERO_BITS;
_tos_cache2 = PyStackRef_ZERO_BITS;
SET_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-104584: Baby steps towards generating and executing traces (#105924) Added a new, experimental, tracing optimizer and interpreter (a.k.a. "tier 2"). This currently pessimizes, so don't use yet -- this is infrastructure so we can experiment with optimizing passes. To enable it, pass ``-Xuops`` or set ``PYTHONUOPS=1``. To get debug output, set ``PYTHONUOPSDEBUG=N`` where ``N`` is a debug level (0-4, where 0 is no debug output and 4 is excessively verbose). All of this code is likely to change dramatically before the 3.13 feature freeze. But this is a first step.
2023-06-26 19:02:57 -07:00
break;
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
case _STORE_DEREF_r10: {
CHECK_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
_PyStackRef v;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef _stack_item_0 = _tos_cache0;
GH-111485: Factor out tier 2 code generation from the rest of the interpreter code generator (GH-112968)
2023-12-12 12:12:17 +00:00
oparg = CURRENT_OPARG();
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
v = _stack_item_0;
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
PyCellObject *cell = (PyCellObject *)PyStackRef_AsPyObjectBorrow(GETLOCAL(oparg));
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
stack_pointer[0] = v;
stack_pointer += 1;
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
GH-119866: Spill the stack around escaping calls. (GH-124392) * Spill the evaluation around escaping calls in the generated interpreter and JIT. * The code generator tracks live, cached values so they can be saved to memory when needed. * Spills the stack pointer around escaping calls, so that the exact stack is visible to the cycle GC.
2024-10-07 14:56:39 +01:00
_PyFrame_SetStackPointer(frame, stack_pointer);
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
PyCell_SetTakeRef(cell, PyStackRef_AsPyObjectSteal(v));
GH-119866: Spill the stack around escaping calls. (GH-124392) * Spill the evaluation around escaping calls in the generated interpreter and JIT. * The code generator tracks live, cached values so they can be saved to memory when needed. * Spills the stack pointer around escaping calls, so that the exact stack is visible to the cycle GC.
2024-10-07 14:56:39 +01:00
stack_pointer = _PyFrame_GetStackPointer(frame);
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_tos_cache0 = PyStackRef_ZERO_BITS;
_tos_cache1 = PyStackRef_ZERO_BITS;
_tos_cache2 = PyStackRef_ZERO_BITS;
SET_CURRENT_CACHED_VALUES(0);
GH-111485: Factor out tier 2 code generation from the rest of the interpreter code generator (GH-112968)
2023-12-12 12:12:17 +00:00
stack_pointer += -1;
GH-141794: Limit size of generated machine code. (GH-142228) * Factor out bodies of the largest uops, to reduce jit code size. * Factor out common assert, also reducing jit code size. * Limit size of jitted code for a single executor to 1MB.
2025-12-03 17:43:35 +00:00
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _COPY_FREE_VARS_r00: {
CHECK_CURRENT_CACHED_VALUES(0);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
oparg = CURRENT_OPARG();
PyCodeObject *co = _PyFrame_GetCode(frame);
assert(PyStackRef_FunctionCheck(frame->f_funcobj));
PyFunctionObject *func = (PyFunctionObject *)PyStackRef_AsPyObjectBorrow(frame->f_funcobj);
PyObject *closure = func->func_closure;
assert(oparg == co->co_nfreevars);
int offset = co->co_nlocalsplus - oparg;
for (int i = 0; i < oparg; ++i) {
PyObject *o = PyTuple_GET_ITEM(closure, i);
frame->localsplus[offset + i] = PyStackRef_FromPyObjectNew(o);
}
SET_CURRENT_CACHED_VALUES(0);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _COPY_FREE_VARS_r11: {
CHECK_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef _stack_item_0 = _tos_cache0;
oparg = CURRENT_OPARG();
PyCodeObject *co = _PyFrame_GetCode(frame);
assert(PyStackRef_FunctionCheck(frame->f_funcobj));
PyFunctionObject *func = (PyFunctionObject *)PyStackRef_AsPyObjectBorrow(frame->f_funcobj);
PyObject *closure = func->func_closure;
assert(oparg == co->co_nfreevars);
int offset = co->co_nlocalsplus - oparg;
for (int i = 0; i < oparg; ++i) {
PyObject *o = PyTuple_GET_ITEM(closure, i);
frame->localsplus[offset + i] = PyStackRef_FromPyObjectNew(o);
}
_tos_cache0 = _stack_item_0;
SET_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _COPY_FREE_VARS_r22: {
CHECK_CURRENT_CACHED_VALUES(2);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef _stack_item_0 = _tos_cache0;
_PyStackRef _stack_item_1 = _tos_cache1;
oparg = CURRENT_OPARG();
PyCodeObject *co = _PyFrame_GetCode(frame);
assert(PyStackRef_FunctionCheck(frame->f_funcobj));
PyFunctionObject *func = (PyFunctionObject *)PyStackRef_AsPyObjectBorrow(frame->f_funcobj);
PyObject *closure = func->func_closure;
assert(oparg == co->co_nfreevars);
int offset = co->co_nlocalsplus - oparg;
for (int i = 0; i < oparg; ++i) {
PyObject *o = PyTuple_GET_ITEM(closure, i);
frame->localsplus[offset + i] = PyStackRef_FromPyObjectNew(o);
}
_tos_cache1 = _stack_item_1;
_tos_cache0 = _stack_item_0;
SET_CURRENT_CACHED_VALUES(2);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-104584: Baby steps towards generating and executing traces (#105924) Added a new, experimental, tracing optimizer and interpreter (a.k.a. "tier 2"). This currently pessimizes, so don't use yet -- this is infrastructure so we can experiment with optimizing passes. To enable it, pass ``-Xuops`` or set ``PYTHONUOPS=1``. To get debug output, set ``PYTHONUOPSDEBUG=N`` where ``N`` is a debug level (0-4, where 0 is no debug output and 4 is excessively verbose). All of this code is likely to change dramatically before the 3.13 feature freeze. But this is a first step.
2023-06-26 19:02:57 -07:00
break;
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
case _COPY_FREE_VARS_r33: {
CHECK_CURRENT_CACHED_VALUES(3);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef _stack_item_0 = _tos_cache0;
_PyStackRef _stack_item_1 = _tos_cache1;
_PyStackRef _stack_item_2 = _tos_cache2;
gh-112287: Speed up Tier 2 (uop) interpreter a little (#112286) This makes the Tier 2 interpreter a little faster. I calculated by about 3%, though I hesitate to claim an exact number. This starts by doubling the trace size limit (to 512), making it more likely that loops fit in a trace. The rest of the approach is to only load `oparg` and `operand` in cases that use them. The code generator know when these are used. For `oparg`, it will conditionally emit ``` oparg = CURRENT_OPARG(); ``` at the top of the case block. (The `oparg` variable may be referenced multiple times by the instructions code block, so it must be in a variable.) For `operand`, it will use `CURRENT_OPERAND()` directly instead of referencing the `operand` variable, which no longer exists. (There is only one place where this will be used.)
2023-11-20 11:25:32 -08:00
oparg = CURRENT_OPARG();
gh-104584: Baby steps towards generating and executing traces (#105924) Added a new, experimental, tracing optimizer and interpreter (a.k.a. "tier 2"). This currently pessimizes, so don't use yet -- this is infrastructure so we can experiment with optimizing passes. To enable it, pass ``-Xuops`` or set ``PYTHONUOPS=1``. To get debug output, set ``PYTHONUOPSDEBUG=N`` where ``N`` is a debug level (0-4, where 0 is no debug output and 4 is excessively verbose). All of this code is likely to change dramatically before the 3.13 feature freeze. But this is a first step.
2023-06-26 19:02:57 -07:00
PyCodeObject *co = _PyFrame_GetCode(frame);
gh-123923: Defer refcounting for `f_funcobj` in `_PyInterpreterFrame` (#124026) Use a `_PyStackRef` and defer the reference to `f_funcobj` when possible. This avoids some reference count contention in the common case of executing the same code object from multiple threads concurrently in the free-threaded build.
2024-09-24 13:08:18 -07:00
assert(PyStackRef_FunctionCheck(frame->f_funcobj));
PyFunctionObject *func = (PyFunctionObject *)PyStackRef_AsPyObjectBorrow(frame->f_funcobj);
PyObject *closure = func->func_closure;
gh-104584: Baby steps towards generating and executing traces (#105924) Added a new, experimental, tracing optimizer and interpreter (a.k.a. "tier 2"). This currently pessimizes, so don't use yet -- this is infrastructure so we can experiment with optimizing passes. To enable it, pass ``-Xuops`` or set ``PYTHONUOPS=1``. To get debug output, set ``PYTHONUOPSDEBUG=N`` where ``N`` is a debug level (0-4, where 0 is no debug output and 4 is excessively verbose). All of this code is likely to change dramatically before the 3.13 feature freeze. But this is a first step.
2023-06-26 19:02:57 -07:00
assert(oparg == co->co_nfreevars);
int offset = co->co_nlocalsplus - oparg;
for (int i = 0; i < oparg; ++i) {
PyObject *o = PyTuple_GET_ITEM(closure, i);
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
frame->localsplus[offset + i] = PyStackRef_FromPyObjectNew(o);
gh-104584: Baby steps towards generating and executing traces (#105924) Added a new, experimental, tracing optimizer and interpreter (a.k.a. "tier 2"). This currently pessimizes, so don't use yet -- this is infrastructure so we can experiment with optimizing passes. To enable it, pass ``-Xuops`` or set ``PYTHONUOPS=1``. To get debug output, set ``PYTHONUOPSDEBUG=N`` where ``N`` is a debug level (0-4, where 0 is no debug output and 4 is excessively verbose). All of this code is likely to change dramatically before the 3.13 feature freeze. But this is a first step.
2023-06-26 19:02:57 -07:00
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_tos_cache2 = _stack_item_2;
_tos_cache1 = _stack_item_1;
_tos_cache0 = _stack_item_0;
SET_CURRENT_CACHED_VALUES(3);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-104584: Baby steps towards generating and executing traces (#105924) Added a new, experimental, tracing optimizer and interpreter (a.k.a. "tier 2"). This currently pessimizes, so don't use yet -- this is infrastructure so we can experiment with optimizing passes. To enable it, pass ``-Xuops`` or set ``PYTHONUOPS=1``. To get debug output, set ``PYTHONUOPSDEBUG=N`` where ``N`` is a debug level (0-4, where 0 is no debug output and 4 is excessively verbose). All of this code is likely to change dramatically before the 3.13 feature freeze. But this is a first step.
2023-06-26 19:02:57 -07:00
break;
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
case _BUILD_STRING_r01: {
CHECK_CURRENT_CACHED_VALUES(0);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
_PyStackRef *pieces;
_PyStackRef str;
GH-111485: Factor out tier 2 code generation from the rest of the interpreter code generator (GH-112968)
2023-12-12 12:12:17 +00:00
oparg = CURRENT_OPARG();
pieces = &stack_pointer[-oparg];
GH-128682: Account for escapes in `DECREF_INPUTS` (GH-129953) * Handle escapes in DECREF_INPUTS * Mark a few more functions as escaping * Replace DECREF_INPUTS with PyStackRef_CLOSE where possible
2025-02-12 17:44:59 +00:00
_PyFrame_SetStackPointer(frame, stack_pointer);
GH-141794: Limit size of generated machine code. (GH-142228) * Factor out bodies of the largest uops, to reduce jit code size. * Factor out common assert, also reducing jit code size. * Limit size of jitted code for a single executor to 1MB.
2025-12-03 17:43:35 +00:00
PyObject *str_o = _Py_BuildString_StackRefSteal(pieces, oparg);
GH-128682: Account for escapes in `DECREF_INPUTS` (GH-129953) * Handle escapes in DECREF_INPUTS * Mark a few more functions as escaping * Replace DECREF_INPUTS with PyStackRef_CLOSE where possible
2025-02-12 17:44:59 +00:00
stack_pointer = _PyFrame_GetStackPointer(frame);
GH-128682: Make `PyStackRef_CLOSE` escaping. (GH-129404)
2025-02-03 12:41:32 +00:00
if (str_o == NULL) {
GH-141794: Limit size of generated machine code. (GH-142228) * Factor out bodies of the largest uops, to reduce jit code size. * Factor out common assert, also reducing jit code size. * Limit size of jitted code for a single executor to 1MB.
2025-12-03 17:43:35 +00:00
stack_pointer += -oparg;
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(0);
GH-128682: Make `PyStackRef_CLOSE` escaping. (GH-129404)
2025-02-03 12:41:32 +00:00
JUMP_TO_ERROR();
}
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
str = PyStackRef_FromPyObjectSteal(str_o);
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_tos_cache0 = str;
_tos_cache1 = PyStackRef_ZERO_BITS;
_tos_cache2 = PyStackRef_ZERO_BITS;
SET_CURRENT_CACHED_VALUES(1);
stack_pointer += -oparg;
GH-141794: Limit size of generated machine code. (GH-142228) * Factor out bodies of the largest uops, to reduce jit code size. * Factor out common assert, also reducing jit code size. * Limit size of jitted code for a single executor to 1MB.
2025-12-03 17:43:35 +00:00
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-104584: Baby steps towards generating and executing traces (#105924) Added a new, experimental, tracing optimizer and interpreter (a.k.a. "tier 2"). This currently pessimizes, so don't use yet -- this is infrastructure so we can experiment with optimizing passes. To enable it, pass ``-Xuops`` or set ``PYTHONUOPS=1``. To get debug output, set ``PYTHONUOPSDEBUG=N`` where ``N`` is a debug level (0-4, where 0 is no debug output and 4 is excessively verbose). All of this code is likely to change dramatically before the 3.13 feature freeze. But this is a first step.
2023-06-26 19:02:57 -07:00
break;
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
case _BUILD_INTERPOLATION_r01: {
CHECK_CURRENT_CACHED_VALUES(0);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-132661: Implement PEP 750 (#132662) Co-authored-by: Lysandros Nikolaou <lisandrosnik@gmail.com> Co-authored-by: Bénédikt Tran <10796600+picnixz@users.noreply.github.com> Co-authored-by: Adam Turner <9087854+AA-Turner@users.noreply.github.com> Co-authored-by: Hugo van Kemenade <1324225+hugovk@users.noreply.github.com> Co-authored-by: Wingy <git@wingysam.xyz> Co-authored-by: Koudai Aono <koxudaxi@gmail.com> Co-authored-by: Dave Peck <davepeck@gmail.com> Co-authored-by: Terry Jan Reedy <tjreedy@udel.edu> Co-authored-by: Paul Everitt <pauleveritt@me.com> Co-authored-by: sobolevn <mail@sobolevn.me>
2025-04-30 11:46:41 +02:00
_PyStackRef *format;
_PyStackRef str;
_PyStackRef value;
_PyStackRef interpolation;
oparg = CURRENT_OPARG();
format = &stack_pointer[-(oparg & 1)];
str = stack_pointer[-1 - (oparg & 1)];
value = stack_pointer[-2 - (oparg & 1)];
PyObject *value_o = PyStackRef_AsPyObjectBorrow(value);
PyObject *str_o = PyStackRef_AsPyObjectBorrow(str);
int conversion = oparg >> 2;
PyObject *format_o;
if (oparg & 1) {
format_o = PyStackRef_AsPyObjectBorrow(format[0]);
}
else {
format_o = &_Py_STR(empty);
}
_PyFrame_SetStackPointer(frame, stack_pointer);
PyObject *interpolation_o = _PyInterpolation_Build(value_o, str_o, conversion, format_o);
stack_pointer = _PyFrame_GetStackPointer(frame);
if (oparg & 1) {
stack_pointer += -(oparg & 1);
GH-141794: Limit size of generated machine code. (GH-142228) * Factor out bodies of the largest uops, to reduce jit code size. * Factor out common assert, also reducing jit code size. * Limit size of jitted code for a single executor to 1MB.
2025-12-03 17:43:35 +00:00
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
gh-132661: Implement PEP 750 (#132662) Co-authored-by: Lysandros Nikolaou <lisandrosnik@gmail.com> Co-authored-by: Bénédikt Tran <10796600+picnixz@users.noreply.github.com> Co-authored-by: Adam Turner <9087854+AA-Turner@users.noreply.github.com> Co-authored-by: Hugo van Kemenade <1324225+hugovk@users.noreply.github.com> Co-authored-by: Wingy <git@wingysam.xyz> Co-authored-by: Koudai Aono <koxudaxi@gmail.com> Co-authored-by: Dave Peck <davepeck@gmail.com> Co-authored-by: Terry Jan Reedy <tjreedy@udel.edu> Co-authored-by: Paul Everitt <pauleveritt@me.com> Co-authored-by: sobolevn <mail@sobolevn.me>
2025-04-30 11:46:41 +02:00
_PyFrame_SetStackPointer(frame, stack_pointer);
PyStackRef_CLOSE(format[0]);
stack_pointer = _PyFrame_GetStackPointer(frame);
}
else {
stack_pointer += -(oparg & 1);
}
stack_pointer += -1;
GH-141794: Limit size of generated machine code. (GH-142228) * Factor out bodies of the largest uops, to reduce jit code size. * Factor out common assert, also reducing jit code size. * Limit size of jitted code for a single executor to 1MB.
2025-12-03 17:43:35 +00:00
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
gh-132661: Implement PEP 750 (#132662) Co-authored-by: Lysandros Nikolaou <lisandrosnik@gmail.com> Co-authored-by: Bénédikt Tran <10796600+picnixz@users.noreply.github.com> Co-authored-by: Adam Turner <9087854+AA-Turner@users.noreply.github.com> Co-authored-by: Hugo van Kemenade <1324225+hugovk@users.noreply.github.com> Co-authored-by: Wingy <git@wingysam.xyz> Co-authored-by: Koudai Aono <koxudaxi@gmail.com> Co-authored-by: Dave Peck <davepeck@gmail.com> Co-authored-by: Terry Jan Reedy <tjreedy@udel.edu> Co-authored-by: Paul Everitt <pauleveritt@me.com> Co-authored-by: sobolevn <mail@sobolevn.me>
2025-04-30 11:46:41 +02:00
_PyFrame_SetStackPointer(frame, stack_pointer);
PyStackRef_CLOSE(str);
stack_pointer = _PyFrame_GetStackPointer(frame);
stack_pointer += -1;
GH-141794: Limit size of generated machine code. (GH-142228) * Factor out bodies of the largest uops, to reduce jit code size. * Factor out common assert, also reducing jit code size. * Limit size of jitted code for a single executor to 1MB.
2025-12-03 17:43:35 +00:00
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
gh-132661: Implement PEP 750 (#132662) Co-authored-by: Lysandros Nikolaou <lisandrosnik@gmail.com> Co-authored-by: Bénédikt Tran <10796600+picnixz@users.noreply.github.com> Co-authored-by: Adam Turner <9087854+AA-Turner@users.noreply.github.com> Co-authored-by: Hugo van Kemenade <1324225+hugovk@users.noreply.github.com> Co-authored-by: Wingy <git@wingysam.xyz> Co-authored-by: Koudai Aono <koxudaxi@gmail.com> Co-authored-by: Dave Peck <davepeck@gmail.com> Co-authored-by: Terry Jan Reedy <tjreedy@udel.edu> Co-authored-by: Paul Everitt <pauleveritt@me.com> Co-authored-by: sobolevn <mail@sobolevn.me>
2025-04-30 11:46:41 +02:00
_PyFrame_SetStackPointer(frame, stack_pointer);
PyStackRef_CLOSE(value);
stack_pointer = _PyFrame_GetStackPointer(frame);
if (interpolation_o == NULL) {
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(0);
gh-132661: Implement PEP 750 (#132662) Co-authored-by: Lysandros Nikolaou <lisandrosnik@gmail.com> Co-authored-by: Bénédikt Tran <10796600+picnixz@users.noreply.github.com> Co-authored-by: Adam Turner <9087854+AA-Turner@users.noreply.github.com> Co-authored-by: Hugo van Kemenade <1324225+hugovk@users.noreply.github.com> Co-authored-by: Wingy <git@wingysam.xyz> Co-authored-by: Koudai Aono <koxudaxi@gmail.com> Co-authored-by: Dave Peck <davepeck@gmail.com> Co-authored-by: Terry Jan Reedy <tjreedy@udel.edu> Co-authored-by: Paul Everitt <pauleveritt@me.com> Co-authored-by: sobolevn <mail@sobolevn.me>
2025-04-30 11:46:41 +02:00
JUMP_TO_ERROR();
}
interpolation = PyStackRef_FromPyObjectSteal(interpolation_o);
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_tos_cache0 = interpolation;
_tos_cache1 = PyStackRef_ZERO_BITS;
_tos_cache2 = PyStackRef_ZERO_BITS;
SET_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-132661: Implement PEP 750 (#132662) Co-authored-by: Lysandros Nikolaou <lisandrosnik@gmail.com> Co-authored-by: Bénédikt Tran <10796600+picnixz@users.noreply.github.com> Co-authored-by: Adam Turner <9087854+AA-Turner@users.noreply.github.com> Co-authored-by: Hugo van Kemenade <1324225+hugovk@users.noreply.github.com> Co-authored-by: Wingy <git@wingysam.xyz> Co-authored-by: Koudai Aono <koxudaxi@gmail.com> Co-authored-by: Dave Peck <davepeck@gmail.com> Co-authored-by: Terry Jan Reedy <tjreedy@udel.edu> Co-authored-by: Paul Everitt <pauleveritt@me.com> Co-authored-by: sobolevn <mail@sobolevn.me>
2025-04-30 11:46:41 +02:00
break;
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
case _BUILD_TEMPLATE_r21: {
CHECK_CURRENT_CACHED_VALUES(2);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-132661: Implement PEP 750 (#132662) Co-authored-by: Lysandros Nikolaou <lisandrosnik@gmail.com> Co-authored-by: Bénédikt Tran <10796600+picnixz@users.noreply.github.com> Co-authored-by: Adam Turner <9087854+AA-Turner@users.noreply.github.com> Co-authored-by: Hugo van Kemenade <1324225+hugovk@users.noreply.github.com> Co-authored-by: Wingy <git@wingysam.xyz> Co-authored-by: Koudai Aono <koxudaxi@gmail.com> Co-authored-by: Dave Peck <davepeck@gmail.com> Co-authored-by: Terry Jan Reedy <tjreedy@udel.edu> Co-authored-by: Paul Everitt <pauleveritt@me.com> Co-authored-by: sobolevn <mail@sobolevn.me>
2025-04-30 11:46:41 +02:00
_PyStackRef interpolations;
_PyStackRef strings;
_PyStackRef template;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef _stack_item_0 = _tos_cache0;
_PyStackRef _stack_item_1 = _tos_cache1;
interpolations = _stack_item_1;
strings = _stack_item_0;
gh-132661: Implement PEP 750 (#132662) Co-authored-by: Lysandros Nikolaou <lisandrosnik@gmail.com> Co-authored-by: Bénédikt Tran <10796600+picnixz@users.noreply.github.com> Co-authored-by: Adam Turner <9087854+AA-Turner@users.noreply.github.com> Co-authored-by: Hugo van Kemenade <1324225+hugovk@users.noreply.github.com> Co-authored-by: Wingy <git@wingysam.xyz> Co-authored-by: Koudai Aono <koxudaxi@gmail.com> Co-authored-by: Dave Peck <davepeck@gmail.com> Co-authored-by: Terry Jan Reedy <tjreedy@udel.edu> Co-authored-by: Paul Everitt <pauleveritt@me.com> Co-authored-by: sobolevn <mail@sobolevn.me>
2025-04-30 11:46:41 +02:00
PyObject *strings_o = PyStackRef_AsPyObjectBorrow(strings);
PyObject *interpolations_o = PyStackRef_AsPyObjectBorrow(interpolations);
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
stack_pointer[0] = strings;
stack_pointer[1] = interpolations;
stack_pointer += 2;
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
gh-132661: Implement PEP 750 (#132662) Co-authored-by: Lysandros Nikolaou <lisandrosnik@gmail.com> Co-authored-by: Bénédikt Tran <10796600+picnixz@users.noreply.github.com> Co-authored-by: Adam Turner <9087854+AA-Turner@users.noreply.github.com> Co-authored-by: Hugo van Kemenade <1324225+hugovk@users.noreply.github.com> Co-authored-by: Wingy <git@wingysam.xyz> Co-authored-by: Koudai Aono <koxudaxi@gmail.com> Co-authored-by: Dave Peck <davepeck@gmail.com> Co-authored-by: Terry Jan Reedy <tjreedy@udel.edu> Co-authored-by: Paul Everitt <pauleveritt@me.com> Co-authored-by: sobolevn <mail@sobolevn.me>
2025-04-30 11:46:41 +02:00
_PyFrame_SetStackPointer(frame, stack_pointer);
PyObject *template_o = _PyTemplate_Build(strings_o, interpolations_o);
stack_pointer = _PyFrame_GetStackPointer(frame);
stack_pointer += -1;
GH-141794: Limit size of generated machine code. (GH-142228) * Factor out bodies of the largest uops, to reduce jit code size. * Factor out common assert, also reducing jit code size. * Limit size of jitted code for a single executor to 1MB.
2025-12-03 17:43:35 +00:00
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
gh-132661: Implement PEP 750 (#132662) Co-authored-by: Lysandros Nikolaou <lisandrosnik@gmail.com> Co-authored-by: Bénédikt Tran <10796600+picnixz@users.noreply.github.com> Co-authored-by: Adam Turner <9087854+AA-Turner@users.noreply.github.com> Co-authored-by: Hugo van Kemenade <1324225+hugovk@users.noreply.github.com> Co-authored-by: Wingy <git@wingysam.xyz> Co-authored-by: Koudai Aono <koxudaxi@gmail.com> Co-authored-by: Dave Peck <davepeck@gmail.com> Co-authored-by: Terry Jan Reedy <tjreedy@udel.edu> Co-authored-by: Paul Everitt <pauleveritt@me.com> Co-authored-by: sobolevn <mail@sobolevn.me>
2025-04-30 11:46:41 +02:00
_PyFrame_SetStackPointer(frame, stack_pointer);
PyStackRef_CLOSE(interpolations);
stack_pointer = _PyFrame_GetStackPointer(frame);
stack_pointer += -1;
GH-141794: Limit size of generated machine code. (GH-142228) * Factor out bodies of the largest uops, to reduce jit code size. * Factor out common assert, also reducing jit code size. * Limit size of jitted code for a single executor to 1MB.
2025-12-03 17:43:35 +00:00
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
gh-132661: Implement PEP 750 (#132662) Co-authored-by: Lysandros Nikolaou <lisandrosnik@gmail.com> Co-authored-by: Bénédikt Tran <10796600+picnixz@users.noreply.github.com> Co-authored-by: Adam Turner <9087854+AA-Turner@users.noreply.github.com> Co-authored-by: Hugo van Kemenade <1324225+hugovk@users.noreply.github.com> Co-authored-by: Wingy <git@wingysam.xyz> Co-authored-by: Koudai Aono <koxudaxi@gmail.com> Co-authored-by: Dave Peck <davepeck@gmail.com> Co-authored-by: Terry Jan Reedy <tjreedy@udel.edu> Co-authored-by: Paul Everitt <pauleveritt@me.com> Co-authored-by: sobolevn <mail@sobolevn.me>
2025-04-30 11:46:41 +02:00
_PyFrame_SetStackPointer(frame, stack_pointer);
PyStackRef_CLOSE(strings);
stack_pointer = _PyFrame_GetStackPointer(frame);
if (template_o == NULL) {
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(0);
gh-132661: Implement PEP 750 (#132662) Co-authored-by: Lysandros Nikolaou <lisandrosnik@gmail.com> Co-authored-by: Bénédikt Tran <10796600+picnixz@users.noreply.github.com> Co-authored-by: Adam Turner <9087854+AA-Turner@users.noreply.github.com> Co-authored-by: Hugo van Kemenade <1324225+hugovk@users.noreply.github.com> Co-authored-by: Wingy <git@wingysam.xyz> Co-authored-by: Koudai Aono <koxudaxi@gmail.com> Co-authored-by: Dave Peck <davepeck@gmail.com> Co-authored-by: Terry Jan Reedy <tjreedy@udel.edu> Co-authored-by: Paul Everitt <pauleveritt@me.com> Co-authored-by: sobolevn <mail@sobolevn.me>
2025-04-30 11:46:41 +02:00
JUMP_TO_ERROR();
}
template = PyStackRef_FromPyObjectSteal(template_o);
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_tos_cache0 = template;
_tos_cache1 = PyStackRef_ZERO_BITS;
_tos_cache2 = PyStackRef_ZERO_BITS;
SET_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-132661: Implement PEP 750 (#132662) Co-authored-by: Lysandros Nikolaou <lisandrosnik@gmail.com> Co-authored-by: Bénédikt Tran <10796600+picnixz@users.noreply.github.com> Co-authored-by: Adam Turner <9087854+AA-Turner@users.noreply.github.com> Co-authored-by: Hugo van Kemenade <1324225+hugovk@users.noreply.github.com> Co-authored-by: Wingy <git@wingysam.xyz> Co-authored-by: Koudai Aono <koxudaxi@gmail.com> Co-authored-by: Dave Peck <davepeck@gmail.com> Co-authored-by: Terry Jan Reedy <tjreedy@udel.edu> Co-authored-by: Paul Everitt <pauleveritt@me.com> Co-authored-by: sobolevn <mail@sobolevn.me>
2025-04-30 11:46:41 +02:00
break;
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
case _BUILD_TUPLE_r01: {
CHECK_CURRENT_CACHED_VALUES(0);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
_PyStackRef *values;
_PyStackRef tup;
GH-111485: Factor out tier 2 code generation from the rest of the interpreter code generator (GH-112968)
2023-12-12 12:12:17 +00:00
oparg = CURRENT_OPARG();
values = &stack_pointer[-oparg];
GH-128682: Change a couple of functions to only steal references on success. (GH-129132) Change PyTuple_FromStackRefSteal and PyList_FromStackRefSteal to only steal on success to avoid escaping
2025-01-22 10:51:37 +00:00
PyObject *tup_o = _PyTuple_FromStackRefStealOnSuccess(values, oparg);
if (tup_o == NULL) {
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(0);
GH-128682: Change a couple of functions to only steal references on success. (GH-129132) Change PyTuple_FromStackRefSteal and PyList_FromStackRefSteal to only steal on success to avoid escaping
2025-01-22 10:51:37 +00:00
JUMP_TO_ERROR();
}
GH-127705: Use `_PyStackRef`s in the default build. (GH-127875)
2025-03-10 14:06:56 +00:00
tup = PyStackRef_FromPyObjectStealMortal(tup_o);
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_tos_cache0 = tup;
SET_CURRENT_CACHED_VALUES(1);
stack_pointer += -oparg;
GH-141794: Limit size of generated machine code. (GH-142228) * Factor out bodies of the largest uops, to reduce jit code size. * Factor out common assert, also reducing jit code size. * Limit size of jitted code for a single executor to 1MB.
2025-12-03 17:43:35 +00:00
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-104584: Baby steps towards generating and executing traces (#105924) Added a new, experimental, tracing optimizer and interpreter (a.k.a. "tier 2"). This currently pessimizes, so don't use yet -- this is infrastructure so we can experiment with optimizing passes. To enable it, pass ``-Xuops`` or set ``PYTHONUOPS=1``. To get debug output, set ``PYTHONUOPSDEBUG=N`` where ``N`` is a debug level (0-4, where 0 is no debug output and 4 is excessively verbose). All of this code is likely to change dramatically before the 3.13 feature freeze. But this is a first step.
2023-06-26 19:02:57 -07:00
break;
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
case _BUILD_LIST_r01: {
CHECK_CURRENT_CACHED_VALUES(0);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
_PyStackRef *values;
_PyStackRef list;
GH-111485: Factor out tier 2 code generation from the rest of the interpreter code generator (GH-112968)
2023-12-12 12:12:17 +00:00
oparg = CURRENT_OPARG();
values = &stack_pointer[-oparg];
gh-118331: Fix a couple of issues when list allocation fails (#130811) * Fix use after free in list objects Set the items pointer in the list object to NULL after the items array is freed during list deallocation. Otherwise, we can end up with a list object added to the free list that contains a pointer to an already-freed items array. * Mark `_PyList_FromStackRefStealOnSuccess` as escaping I think technically it's not escaping, because the only object that can be decrefed if allocation fails is an exact list, which cannot execute arbitrary code when it is destroyed. However, this seems less intrusive than trying to special cases objects in the assert in `_Py_Dealloc` that checks for non-null stackpointers and shouldn't matter for performance.
2025-03-05 10:42:09 -08:00
_PyFrame_SetStackPointer(frame, stack_pointer);
GH-128682: Change a couple of functions to only steal references on success. (GH-129132) Change PyTuple_FromStackRefSteal and PyList_FromStackRefSteal to only steal on success to avoid escaping
2025-01-22 10:51:37 +00:00
PyObject *list_o = _PyList_FromStackRefStealOnSuccess(values, oparg);
gh-118331: Fix a couple of issues when list allocation fails (#130811) * Fix use after free in list objects Set the items pointer in the list object to NULL after the items array is freed during list deallocation. Otherwise, we can end up with a list object added to the free list that contains a pointer to an already-freed items array. * Mark `_PyList_FromStackRefStealOnSuccess` as escaping I think technically it's not escaping, because the only object that can be decrefed if allocation fails is an exact list, which cannot execute arbitrary code when it is destroyed. However, this seems less intrusive than trying to special cases objects in the assert in `_Py_Dealloc` that checks for non-null stackpointers and shouldn't matter for performance.
2025-03-05 10:42:09 -08:00
stack_pointer = _PyFrame_GetStackPointer(frame);
GH-128682: Change a couple of functions to only steal references on success. (GH-129132) Change PyTuple_FromStackRefSteal and PyList_FromStackRefSteal to only steal on success to avoid escaping
2025-01-22 10:51:37 +00:00
if (list_o == NULL) {
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(0);
GH-128682: Change a couple of functions to only steal references on success. (GH-129132) Change PyTuple_FromStackRefSteal and PyList_FromStackRefSteal to only steal on success to avoid escaping
2025-01-22 10:51:37 +00:00
JUMP_TO_ERROR();
}
GH-127705: Use `_PyStackRef`s in the default build. (GH-127875)
2025-03-10 14:06:56 +00:00
list = PyStackRef_FromPyObjectStealMortal(list_o);
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_tos_cache0 = list;
_tos_cache1 = PyStackRef_ZERO_BITS;
_tos_cache2 = PyStackRef_ZERO_BITS;
SET_CURRENT_CACHED_VALUES(1);
stack_pointer += -oparg;
GH-141794: Limit size of generated machine code. (GH-142228) * Factor out bodies of the largest uops, to reduce jit code size. * Factor out common assert, also reducing jit code size. * Limit size of jitted code for a single executor to 1MB.
2025-12-03 17:43:35 +00:00
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-104584: Baby steps towards generating and executing traces (#105924) Added a new, experimental, tracing optimizer and interpreter (a.k.a. "tier 2"). This currently pessimizes, so don't use yet -- this is infrastructure so we can experiment with optimizing passes. To enable it, pass ``-Xuops`` or set ``PYTHONUOPS=1``. To get debug output, set ``PYTHONUOPSDEBUG=N`` where ``N`` is a debug level (0-4, where 0 is no debug output and 4 is excessively verbose). All of this code is likely to change dramatically before the 3.13 feature freeze. But this is a first step.
2023-06-26 19:02:57 -07:00
break;
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
case _LIST_EXTEND_r10: {
CHECK_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
_PyStackRef iterable_st;
_PyStackRef list_st;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef _stack_item_0 = _tos_cache0;
GH-111485: Factor out tier 2 code generation from the rest of the interpreter code generator (GH-112968)
2023-12-12 12:12:17 +00:00
oparg = CURRENT_OPARG();
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
iterable_st = _stack_item_0;
list_st = stack_pointer[-1 - (oparg-1)];
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
PyObject *list = PyStackRef_AsPyObjectBorrow(list_st);
PyObject *iterable = PyStackRef_AsPyObjectBorrow(iterable_st);
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
stack_pointer[0] = iterable_st;
stack_pointer += 1;
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
GH-119866: Spill the stack around escaping calls. (GH-124392) * Spill the evaluation around escaping calls in the generated interpreter and JIT. * The code generator tracks live, cached values so they can be saved to memory when needed. * Spills the stack pointer around escaping calls, so that the exact stack is visible to the cycle GC.
2024-10-07 14:56:39 +01:00
_PyFrame_SetStackPointer(frame, stack_pointer);
gh-104584: Baby steps towards generating and executing traces (#105924) Added a new, experimental, tracing optimizer and interpreter (a.k.a. "tier 2"). This currently pessimizes, so don't use yet -- this is infrastructure so we can experiment with optimizing passes. To enable it, pass ``-Xuops`` or set ``PYTHONUOPS=1``. To get debug output, set ``PYTHONUOPSDEBUG=N`` where ``N`` is a debug level (0-4, where 0 is no debug output and 4 is excessively verbose). All of this code is likely to change dramatically before the 3.13 feature freeze. But this is a first step.
2023-06-26 19:02:57 -07:00
PyObject *none_val = _PyList_Extend((PyListObject *)list, iterable);
GH-119866: Spill the stack around escaping calls. (GH-124392) * Spill the evaluation around escaping calls in the generated interpreter and JIT. * The code generator tracks live, cached values so they can be saved to memory when needed. * Spills the stack pointer around escaping calls, so that the exact stack is visible to the cycle GC.
2024-10-07 14:56:39 +01:00
stack_pointer = _PyFrame_GetStackPointer(frame);
gh-104584: Baby steps towards generating and executing traces (#105924) Added a new, experimental, tracing optimizer and interpreter (a.k.a. "tier 2"). This currently pessimizes, so don't use yet -- this is infrastructure so we can experiment with optimizing passes. To enable it, pass ``-Xuops`` or set ``PYTHONUOPS=1``. To get debug output, set ``PYTHONUOPSDEBUG=N`` where ``N`` is a debug level (0-4, where 0 is no debug output and 4 is excessively verbose). All of this code is likely to change dramatically before the 3.13 feature freeze. But this is a first step.
2023-06-26 19:02:57 -07:00
if (none_val == NULL) {
GH-119866: Spill the stack around escaping calls. (GH-124392) * Spill the evaluation around escaping calls in the generated interpreter and JIT. * The code generator tracks live, cached values so they can be saved to memory when needed. * Spills the stack pointer around escaping calls, so that the exact stack is visible to the cycle GC.
2024-10-07 14:56:39 +01:00
_PyFrame_SetStackPointer(frame, stack_pointer);
GH-120024: Refactor code a bit so that escaping calls can be wrapped in spill code in code generator (GH-122693)
2024-08-06 08:40:39 +01:00
int matches = _PyErr_ExceptionMatches(tstate, PyExc_TypeError);
GH-119866: Spill the stack around escaping calls. (GH-124392) * Spill the evaluation around escaping calls in the generated interpreter and JIT. * The code generator tracks live, cached values so they can be saved to memory when needed. * Spills the stack pointer around escaping calls, so that the exact stack is visible to the cycle GC.
2024-10-07 14:56:39 +01:00
stack_pointer = _PyFrame_GetStackPointer(frame);
GH-120024: Refactor code a bit so that escaping calls can be wrapped in spill code in code generator (GH-122693)
2024-08-06 08:40:39 +01:00
if (matches &&
GH-111485: Factor out tier 2 code generation from the rest of the interpreter code generator (GH-112968)
2023-12-12 12:12:17 +00:00
(Py_TYPE(iterable)->tp_iter == NULL && !PySequence_Check(iterable)))
gh-104584: Baby steps towards generating and executing traces (#105924) Added a new, experimental, tracing optimizer and interpreter (a.k.a. "tier 2"). This currently pessimizes, so don't use yet -- this is infrastructure so we can experiment with optimizing passes. To enable it, pass ``-Xuops`` or set ``PYTHONUOPS=1``. To get debug output, set ``PYTHONUOPSDEBUG=N`` where ``N`` is a debug level (0-4, where 0 is no debug output and 4 is excessively verbose). All of this code is likely to change dramatically before the 3.13 feature freeze. But this is a first step.
2023-06-26 19:02:57 -07:00
{
GH-119866: Spill the stack around escaping calls. (GH-124392) * Spill the evaluation around escaping calls in the generated interpreter and JIT. * The code generator tracks live, cached values so they can be saved to memory when needed. * Spills the stack pointer around escaping calls, so that the exact stack is visible to the cycle GC.
2024-10-07 14:56:39 +01:00
_PyFrame_SetStackPointer(frame, stack_pointer);
gh-104584: Baby steps towards generating and executing traces (#105924) Added a new, experimental, tracing optimizer and interpreter (a.k.a. "tier 2"). This currently pessimizes, so don't use yet -- this is infrastructure so we can experiment with optimizing passes. To enable it, pass ``-Xuops`` or set ``PYTHONUOPS=1``. To get debug output, set ``PYTHONUOPSDEBUG=N`` where ``N`` is a debug level (0-4, where 0 is no debug output and 4 is excessively verbose). All of this code is likely to change dramatically before the 3.13 feature freeze. But this is a first step.
2023-06-26 19:02:57 -07:00
_PyErr_Clear(tstate);
_PyErr_Format(tstate, PyExc_TypeError,
GH-111485: Factor out tier 2 code generation from the rest of the interpreter code generator (GH-112968)
2023-12-12 12:12:17 +00:00
"Value after * must be an iterable, not %.200s",
Py_TYPE(iterable)->tp_name);
GH-119866: Spill the stack around escaping calls. (GH-124392) * Spill the evaluation around escaping calls in the generated interpreter and JIT. * The code generator tracks live, cached values so they can be saved to memory when needed. * Spills the stack pointer around escaping calls, so that the exact stack is visible to the cycle GC.
2024-10-07 14:56:39 +01:00
stack_pointer = _PyFrame_GetStackPointer(frame);
gh-104584: Baby steps towards generating and executing traces (#105924) Added a new, experimental, tracing optimizer and interpreter (a.k.a. "tier 2"). This currently pessimizes, so don't use yet -- this is infrastructure so we can experiment with optimizing passes. To enable it, pass ``-Xuops`` or set ``PYTHONUOPS=1``. To get debug output, set ``PYTHONUOPSDEBUG=N`` where ``N`` is a debug level (0-4, where 0 is no debug output and 4 is excessively verbose). All of this code is likely to change dramatically before the 3.13 feature freeze. But this is a first step.
2023-06-26 19:02:57 -07:00
}
GH-128682: Make `PyStackRef_CLOSE` escaping. (GH-129404)
2025-02-03 12:41:32 +00:00
stack_pointer += -1;
GH-141794: Limit size of generated machine code. (GH-142228) * Factor out bodies of the largest uops, to reduce jit code size. * Factor out common assert, also reducing jit code size. * Limit size of jitted code for a single executor to 1MB.
2025-12-03 17:43:35 +00:00
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
GH-128682: Account for escapes in `DECREF_INPUTS` (GH-129953) * Handle escapes in DECREF_INPUTS * Mark a few more functions as escaping * Replace DECREF_INPUTS with PyStackRef_CLOSE where possible
2025-02-12 17:44:59 +00:00
_PyFrame_SetStackPointer(frame, stack_pointer);
PyStackRef_CLOSE(iterable_st);
stack_pointer = _PyFrame_GetStackPointer(frame);
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(0);
GH-128682: Make `PyStackRef_CLOSE` escaping. (GH-129404)
2025-02-03 12:41:32 +00:00
JUMP_TO_ERROR();
gh-104584: Baby steps towards generating and executing traces (#105924) Added a new, experimental, tracing optimizer and interpreter (a.k.a. "tier 2"). This currently pessimizes, so don't use yet -- this is infrastructure so we can experiment with optimizing passes. To enable it, pass ``-Xuops`` or set ``PYTHONUOPS=1``. To get debug output, set ``PYTHONUOPSDEBUG=N`` where ``N`` is a debug level (0-4, where 0 is no debug output and 4 is excessively verbose). All of this code is likely to change dramatically before the 3.13 feature freeze. But this is a first step.
2023-06-26 19:02:57 -07:00
}
assert(Py_IsNone(none_val));
GH-111485: Factor out tier 2 code generation from the rest of the interpreter code generator (GH-112968)
2023-12-12 12:12:17 +00:00
stack_pointer += -1;
GH-141794: Limit size of generated machine code. (GH-142228) * Factor out bodies of the largest uops, to reduce jit code size. * Factor out common assert, also reducing jit code size. * Limit size of jitted code for a single executor to 1MB.
2025-12-03 17:43:35 +00:00
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
GH-128682: Account for escapes in `DECREF_INPUTS` (GH-129953) * Handle escapes in DECREF_INPUTS * Mark a few more functions as escaping * Replace DECREF_INPUTS with PyStackRef_CLOSE where possible
2025-02-12 17:44:59 +00:00
_PyFrame_SetStackPointer(frame, stack_pointer);
PyStackRef_CLOSE(iterable_st);
stack_pointer = _PyFrame_GetStackPointer(frame);
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_tos_cache0 = PyStackRef_ZERO_BITS;
_tos_cache1 = PyStackRef_ZERO_BITS;
_tos_cache2 = PyStackRef_ZERO_BITS;
SET_CURRENT_CACHED_VALUES(0);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-104584: Baby steps towards generating and executing traces (#105924) Added a new, experimental, tracing optimizer and interpreter (a.k.a. "tier 2"). This currently pessimizes, so don't use yet -- this is infrastructure so we can experiment with optimizing passes. To enable it, pass ``-Xuops`` or set ``PYTHONUOPS=1``. To get debug output, set ``PYTHONUOPSDEBUG=N`` where ``N`` is a debug level (0-4, where 0 is no debug output and 4 is excessively verbose). All of this code is likely to change dramatically before the 3.13 feature freeze. But this is a first step.
2023-06-26 19:02:57 -07:00
break;
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
case _SET_UPDATE_r10: {
CHECK_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
_PyStackRef iterable;
_PyStackRef set;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef _stack_item_0 = _tos_cache0;
GH-111485: Factor out tier 2 code generation from the rest of the interpreter code generator (GH-112968)
2023-12-12 12:12:17 +00:00
oparg = CURRENT_OPARG();
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
iterable = _stack_item_0;
set = stack_pointer[-1 - (oparg-1)];
stack_pointer[0] = iterable;
stack_pointer += 1;
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
GH-119866: Spill the stack around escaping calls. (GH-124392) * Spill the evaluation around escaping calls in the generated interpreter and JIT. * The code generator tracks live, cached values so they can be saved to memory when needed. * Spills the stack pointer around escaping calls, so that the exact stack is visible to the cycle GC.
2024-10-07 14:56:39 +01:00
_PyFrame_SetStackPointer(frame, stack_pointer);
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
int err = _PySet_Update(PyStackRef_AsPyObjectBorrow(set),
PyStackRef_AsPyObjectBorrow(iterable));
GH-119866: Spill the stack around escaping calls. (GH-124392) * Spill the evaluation around escaping calls in the generated interpreter and JIT. * The code generator tracks live, cached values so they can be saved to memory when needed. * Spills the stack pointer around escaping calls, so that the exact stack is visible to the cycle GC.
2024-10-07 14:56:39 +01:00
stack_pointer = _PyFrame_GetStackPointer(frame);
GH-128682: Account for escapes in `DECREF_INPUTS` (GH-129953) * Handle escapes in DECREF_INPUTS * Mark a few more functions as escaping * Replace DECREF_INPUTS with PyStackRef_CLOSE where possible
2025-02-12 17:44:59 +00:00
stack_pointer += -1;
GH-141794: Limit size of generated machine code. (GH-142228) * Factor out bodies of the largest uops, to reduce jit code size. * Factor out common assert, also reducing jit code size. * Limit size of jitted code for a single executor to 1MB.
2025-12-03 17:43:35 +00:00
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
GH-128682: Account for escapes in `DECREF_INPUTS` (GH-129953) * Handle escapes in DECREF_INPUTS * Mark a few more functions as escaping * Replace DECREF_INPUTS with PyStackRef_CLOSE where possible
2025-02-12 17:44:59 +00:00
_PyFrame_SetStackPointer(frame, stack_pointer);
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
PyStackRef_CLOSE(iterable);
GH-128682: Account for escapes in `DECREF_INPUTS` (GH-129953) * Handle escapes in DECREF_INPUTS * Mark a few more functions as escaping * Replace DECREF_INPUTS with PyStackRef_CLOSE where possible
2025-02-12 17:44:59 +00:00
stack_pointer = _PyFrame_GetStackPointer(frame);
GH-128682: Make `PyStackRef_CLOSE` escaping. (GH-129404)
2025-02-03 12:41:32 +00:00
if (err < 0) {
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(0);
GH-128682: Make `PyStackRef_CLOSE` escaping. (GH-129404)
2025-02-03 12:41:32 +00:00
JUMP_TO_ERROR();
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_tos_cache0 = PyStackRef_ZERO_BITS;
_tos_cache1 = PyStackRef_ZERO_BITS;
_tos_cache2 = PyStackRef_ZERO_BITS;
SET_CURRENT_CACHED_VALUES(0);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-104584: Baby steps towards generating and executing traces (#105924) Added a new, experimental, tracing optimizer and interpreter (a.k.a. "tier 2"). This currently pessimizes, so don't use yet -- this is infrastructure so we can experiment with optimizing passes. To enable it, pass ``-Xuops`` or set ``PYTHONUOPS=1``. To get debug output, set ``PYTHONUOPSDEBUG=N`` where ``N`` is a debug level (0-4, where 0 is no debug output and 4 is excessively verbose). All of this code is likely to change dramatically before the 3.13 feature freeze. But this is a first step.
2023-06-26 19:02:57 -07:00
break;
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
case _BUILD_SET_r01: {
CHECK_CURRENT_CACHED_VALUES(0);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-118093: Add tier two support to several instructions (GH-121884)
2024-07-18 14:24:58 -07:00
_PyStackRef *values;
_PyStackRef set;
oparg = CURRENT_OPARG();
values = &stack_pointer[-oparg];
GH-119866: Spill the stack around escaping calls. (GH-124392) * Spill the evaluation around escaping calls in the generated interpreter and JIT. * The code generator tracks live, cached values so they can be saved to memory when needed. * Spills the stack pointer around escaping calls, so that the exact stack is visible to the cycle GC.
2024-10-07 14:56:39 +01:00
_PyFrame_SetStackPointer(frame, stack_pointer);
GH-118093: Add tier two support to several instructions (GH-121884)
2024-07-18 14:24:58 -07:00
PyObject *set_o = PySet_New(NULL);
GH-119866: Spill the stack around escaping calls. (GH-124392) * Spill the evaluation around escaping calls in the generated interpreter and JIT. * The code generator tracks live, cached values so they can be saved to memory when needed. * Spills the stack pointer around escaping calls, so that the exact stack is visible to the cycle GC.
2024-10-07 14:56:39 +01:00
stack_pointer = _PyFrame_GetStackPointer(frame);
GH-118093: Add tier two support to several instructions (GH-121884)
2024-07-18 14:24:58 -07:00
if (set_o == NULL) {
GH-128682: Account for escapes in `DECREF_INPUTS` (GH-129953) * Handle escapes in DECREF_INPUTS * Mark a few more functions as escaping * Replace DECREF_INPUTS with PyStackRef_CLOSE where possible
2025-02-12 17:44:59 +00:00
_PyFrame_SetStackPointer(frame, stack_pointer);
_PyStackRef tmp;
GH-118093: Add tier two support to several instructions (GH-121884)
2024-07-18 14:24:58 -07:00
for (int _i = oparg; --_i >= 0;) {
GH-128682: Account for escapes in `DECREF_INPUTS` (GH-129953) * Handle escapes in DECREF_INPUTS * Mark a few more functions as escaping * Replace DECREF_INPUTS with PyStackRef_CLOSE where possible
2025-02-12 17:44:59 +00:00
tmp = values[_i];
values[_i] = PyStackRef_NULL;
PyStackRef_CLOSE(tmp);
GH-118093: Add tier two support to several instructions (GH-121884)
2024-07-18 14:24:58 -07:00
}
GH-128682: Account for escapes in `DECREF_INPUTS` (GH-129953) * Handle escapes in DECREF_INPUTS * Mark a few more functions as escaping * Replace DECREF_INPUTS with PyStackRef_CLOSE where possible
2025-02-12 17:44:59 +00:00
stack_pointer = _PyFrame_GetStackPointer(frame);
GH-128682: Make `PyStackRef_CLOSE` escaping. (GH-129404)
2025-02-03 12:41:32 +00:00
stack_pointer += -oparg;
GH-141794: Limit size of generated machine code. (GH-142228) * Factor out bodies of the largest uops, to reduce jit code size. * Factor out common assert, also reducing jit code size. * Limit size of jitted code for a single executor to 1MB.
2025-12-03 17:43:35 +00:00
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(0);
GH-128682: Make `PyStackRef_CLOSE` escaping. (GH-129404)
2025-02-03 12:41:32 +00:00
JUMP_TO_ERROR();
GH-118093: Add tier two support to several instructions (GH-121884)
2024-07-18 14:24:58 -07:00
}
int err = 0;
gh-130312: SET_ADD should not lock (#130136) SET_ADD should not lock
2025-03-21 15:58:32 -07:00
for (Py_ssize_t i = 0; i < oparg; i++) {
_PyStackRef value = values[i];
values[i] = PyStackRef_NULL;
GH-118093: Add tier two support to several instructions (GH-121884)
2024-07-18 14:24:58 -07:00
if (err == 0) {
GH-119866: Spill the stack around escaping calls. (GH-124392) * Spill the evaluation around escaping calls in the generated interpreter and JIT. * The code generator tracks live, cached values so they can be saved to memory when needed. * Spills the stack pointer around escaping calls, so that the exact stack is visible to the cycle GC.
2024-10-07 14:56:39 +01:00
_PyFrame_SetStackPointer(frame, stack_pointer);
gh-130312: SET_ADD should not lock (#130136) SET_ADD should not lock
2025-03-21 15:58:32 -07:00
err = _PySet_AddTakeRef((PySetObject *)set_o, PyStackRef_AsPyObjectSteal(value));
stack_pointer = _PyFrame_GetStackPointer(frame);
}
else {
_PyFrame_SetStackPointer(frame, stack_pointer);
PyStackRef_CLOSE(value);
GH-119866: Spill the stack around escaping calls. (GH-124392) * Spill the evaluation around escaping calls in the generated interpreter and JIT. * The code generator tracks live, cached values so they can be saved to memory when needed. * Spills the stack pointer around escaping calls, so that the exact stack is visible to the cycle GC.
2024-10-07 14:56:39 +01:00
stack_pointer = _PyFrame_GetStackPointer(frame);
GH-118093: Add tier two support to several instructions (GH-121884)
2024-07-18 14:24:58 -07:00
}
GH-128682: Convert explicit loops closing arrays into `DECREF_INPUTS`. (GH-128822) * Mark Py_DECREF and Py_XDECREF as escaping * Remove explicit loops for clearing array inputs
2025-01-14 15:08:56 +00:00
}
gh-130312: SET_ADD should not lock (#130136) SET_ADD should not lock
2025-03-21 15:58:32 -07:00
if (err) {
GH-131729: Code-gen better liveness analysis (GH-131732) * Rename 'defined' attribute to 'in_local' to more accurately reflect how it is used * Make death of variables explicit even for array variables. * Convert in_memory from boolean to stack offset * Don't apply liveness analysis to optimizer generated code * Fix RETURN_VALUE in optimizer
2025-03-26 15:21:35 +00:00
stack_pointer += -oparg;
GH-141794: Limit size of generated machine code. (GH-142228) * Factor out bodies of the largest uops, to reduce jit code size. * Factor out common assert, also reducing jit code size. * Limit size of jitted code for a single executor to 1MB.
2025-12-03 17:43:35 +00:00
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
GH-128682: Convert explicit loops closing arrays into `DECREF_INPUTS`. (GH-128822) * Mark Py_DECREF and Py_XDECREF as escaping * Remove explicit loops for clearing array inputs
2025-01-14 15:08:56 +00:00
_PyFrame_SetStackPointer(frame, stack_pointer);
GH-118093: Add tier two support to several instructions (GH-121884)
2024-07-18 14:24:58 -07:00
Py_DECREF(set_o);
GH-128682: Convert explicit loops closing arrays into `DECREF_INPUTS`. (GH-128822) * Mark Py_DECREF and Py_XDECREF as escaping * Remove explicit loops for clearing array inputs
2025-01-14 15:08:56 +00:00
stack_pointer = _PyFrame_GetStackPointer(frame);
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(0);
GH-128682: Make `PyStackRef_CLOSE` escaping. (GH-129404)
2025-02-03 12:41:32 +00:00
JUMP_TO_ERROR();
GH-118093: Add tier two support to several instructions (GH-121884)
2024-07-18 14:24:58 -07:00
}
GH-127705: Use `_PyStackRef`s in the default build. (GH-127875)
2025-03-10 14:06:56 +00:00
set = PyStackRef_FromPyObjectStealMortal(set_o);
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_tos_cache0 = set;
_tos_cache1 = PyStackRef_ZERO_BITS;
_tos_cache2 = PyStackRef_ZERO_BITS;
SET_CURRENT_CACHED_VALUES(1);
stack_pointer += -oparg;
GH-141794: Limit size of generated machine code. (GH-142228) * Factor out bodies of the largest uops, to reduce jit code size. * Factor out common assert, also reducing jit code size. * Limit size of jitted code for a single executor to 1MB.
2025-12-03 17:43:35 +00:00
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-118093: Add tier two support to several instructions (GH-121884)
2024-07-18 14:24:58 -07:00
break;
}
gh-104584: Baby steps towards generating and executing traces (#105924) Added a new, experimental, tracing optimizer and interpreter (a.k.a. "tier 2"). This currently pessimizes, so don't use yet -- this is infrastructure so we can experiment with optimizing passes. To enable it, pass ``-Xuops`` or set ``PYTHONUOPS=1``. To get debug output, set ``PYTHONUOPSDEBUG=N`` where ``N`` is a debug level (0-4, where 0 is no debug output and 4 is excessively verbose). All of this code is likely to change dramatically before the 3.13 feature freeze. But this is a first step.
2023-06-26 19:02:57 -07:00
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
case _BUILD_MAP_r01: {
CHECK_CURRENT_CACHED_VALUES(0);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
_PyStackRef *values;
_PyStackRef map;
GH-111485: Factor out tier 2 code generation from the rest of the interpreter code generator (GH-112968)
2023-12-12 12:12:17 +00:00
oparg = CURRENT_OPARG();
values = &stack_pointer[-oparg*2];
GH-119866: Spill the stack around escaping calls. (GH-124392) * Spill the evaluation around escaping calls in the generated interpreter and JIT. * The code generator tracks live, cached values so they can be saved to memory when needed. * Spills the stack pointer around escaping calls, so that the exact stack is visible to the cycle GC.
2024-10-07 14:56:39 +01:00
_PyFrame_SetStackPointer(frame, stack_pointer);
GH-141794: Limit size of generated machine code. (GH-142228) * Factor out bodies of the largest uops, to reduce jit code size. * Factor out common assert, also reducing jit code size. * Limit size of jitted code for a single executor to 1MB.
2025-12-03 17:43:35 +00:00
PyObject *map_o = _Py_BuildMap_StackRefSteal(values, oparg);
GH-119866: Spill the stack around escaping calls. (GH-124392) * Spill the evaluation around escaping calls in the generated interpreter and JIT. * The code generator tracks live, cached values so they can be saved to memory when needed. * Spills the stack pointer around escaping calls, so that the exact stack is visible to the cycle GC.
2024-10-07 14:56:39 +01:00
stack_pointer = _PyFrame_GetStackPointer(frame);
GH-128682: Make `PyStackRef_CLOSE` escaping. (GH-129404)
2025-02-03 12:41:32 +00:00
if (map_o == NULL) {
GH-141794: Limit size of generated machine code. (GH-142228) * Factor out bodies of the largest uops, to reduce jit code size. * Factor out common assert, also reducing jit code size. * Limit size of jitted code for a single executor to 1MB.
2025-12-03 17:43:35 +00:00
stack_pointer += -oparg*2;
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(0);
GH-128682: Make `PyStackRef_CLOSE` escaping. (GH-129404)
2025-02-03 12:41:32 +00:00
JUMP_TO_ERROR();
}
GH-127705: Use `_PyStackRef`s in the default build. (GH-127875)
2025-03-10 14:06:56 +00:00
map = PyStackRef_FromPyObjectStealMortal(map_o);
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_tos_cache0 = map;
_tos_cache1 = PyStackRef_ZERO_BITS;
_tos_cache2 = PyStackRef_ZERO_BITS;
SET_CURRENT_CACHED_VALUES(1);
stack_pointer += -oparg*2;
GH-141794: Limit size of generated machine code. (GH-142228) * Factor out bodies of the largest uops, to reduce jit code size. * Factor out common assert, also reducing jit code size. * Limit size of jitted code for a single executor to 1MB.
2025-12-03 17:43:35 +00:00
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-104584: Baby steps towards generating and executing traces (#105924) Added a new, experimental, tracing optimizer and interpreter (a.k.a. "tier 2"). This currently pessimizes, so don't use yet -- this is infrastructure so we can experiment with optimizing passes. To enable it, pass ``-Xuops`` or set ``PYTHONUOPS=1``. To get debug output, set ``PYTHONUOPSDEBUG=N`` where ``N`` is a debug level (0-4, where 0 is no debug output and 4 is excessively verbose). All of this code is likely to change dramatically before the 3.13 feature freeze. But this is a first step.
2023-06-26 19:02:57 -07:00
break;
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
case _SETUP_ANNOTATIONS_r00: {
CHECK_CURRENT_CACHED_VALUES(0);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-104584: Baby steps towards generating and executing traces (#105924) Added a new, experimental, tracing optimizer and interpreter (a.k.a. "tier 2"). This currently pessimizes, so don't use yet -- this is infrastructure so we can experiment with optimizing passes. To enable it, pass ``-Xuops`` or set ``PYTHONUOPS=1``. To get debug output, set ``PYTHONUOPSDEBUG=N`` where ``N`` is a debug level (0-4, where 0 is no debug output and 4 is excessively verbose). All of this code is likely to change dramatically before the 3.13 feature freeze. But this is a first step.
2023-06-26 19:02:57 -07:00
if (LOCALS() == NULL) {
GH-119866: Spill the stack around escaping calls. (GH-124392) * Spill the evaluation around escaping calls in the generated interpreter and JIT. * The code generator tracks live, cached values so they can be saved to memory when needed. * Spills the stack pointer around escaping calls, so that the exact stack is visible to the cycle GC.
2024-10-07 14:56:39 +01:00
_PyFrame_SetStackPointer(frame, stack_pointer);
gh-104584: Baby steps towards generating and executing traces (#105924) Added a new, experimental, tracing optimizer and interpreter (a.k.a. "tier 2"). This currently pessimizes, so don't use yet -- this is infrastructure so we can experiment with optimizing passes. To enable it, pass ``-Xuops`` or set ``PYTHONUOPS=1``. To get debug output, set ``PYTHONUOPSDEBUG=N`` where ``N`` is a debug level (0-4, where 0 is no debug output and 4 is excessively verbose). All of this code is likely to change dramatically before the 3.13 feature freeze. But this is a first step.
2023-06-26 19:02:57 -07:00
_PyErr_Format(tstate, PyExc_SystemError,
"no locals found when setting up annotations");
GH-119866: Spill the stack around escaping calls. (GH-124392) * Spill the evaluation around escaping calls in the generated interpreter and JIT. * The code generator tracks live, cached values so they can be saved to memory when needed. * Spills the stack pointer around escaping calls, so that the exact stack is visible to the cycle GC.
2024-10-07 14:56:39 +01:00
stack_pointer = _PyFrame_GetStackPointer(frame);
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(0);
GH-128682: Make `PyStackRef_CLOSE` escaping. (GH-129404)
2025-02-03 12:41:32 +00:00
JUMP_TO_ERROR();
gh-104584: Baby steps towards generating and executing traces (#105924) Added a new, experimental, tracing optimizer and interpreter (a.k.a. "tier 2"). This currently pessimizes, so don't use yet -- this is infrastructure so we can experiment with optimizing passes. To enable it, pass ``-Xuops`` or set ``PYTHONUOPS=1``. To get debug output, set ``PYTHONUOPSDEBUG=N`` where ``N`` is a debug level (0-4, where 0 is no debug output and 4 is excessively verbose). All of this code is likely to change dramatically before the 3.13 feature freeze. But this is a first step.
2023-06-26 19:02:57 -07:00
}
GH-139922: Tail calling for MSVC (VS 2026) (GH-143068) Co-authored-by: Ken Jin <28750310+Fidget-Spinner@users.noreply.github.com> Co-authored-by: Brandt Bucher <brandt@python.org> Co-authored-by: Hugo van Kemenade <1324225+hugovk@users.noreply.github.com>
2025-12-23 00:01:34 +01:00
int err;
GH-119866: Spill the stack around escaping calls. (GH-124392) * Spill the evaluation around escaping calls in the generated interpreter and JIT. * The code generator tracks live, cached values so they can be saved to memory when needed. * Spills the stack pointer around escaping calls, so that the exact stack is visible to the cycle GC.
2024-10-07 14:56:39 +01:00
_PyFrame_SetStackPointer(frame, stack_pointer);
GH-139922: Tail calling for MSVC (VS 2026) (GH-143068) Co-authored-by: Ken Jin <28750310+Fidget-Spinner@users.noreply.github.com> Co-authored-by: Brandt Bucher <brandt@python.org> Co-authored-by: Hugo van Kemenade <1324225+hugovk@users.noreply.github.com>
2025-12-23 00:01:34 +01:00
PyObject* ann_dict = _PyMapping_GetOptionalItem2(LOCALS(), &_Py_ID(__annotations__), &err);
GH-119866: Spill the stack around escaping calls. (GH-124392) * Spill the evaluation around escaping calls in the generated interpreter and JIT. * The code generator tracks live, cached values so they can be saved to memory when needed. * Spills the stack pointer around escaping calls, so that the exact stack is visible to the cycle GC.
2024-10-07 14:56:39 +01:00
stack_pointer = _PyFrame_GetStackPointer(frame);
GH-128682: Make `PyStackRef_CLOSE` escaping. (GH-129404)
2025-02-03 12:41:32 +00:00
if (err < 0) {
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(0);
GH-128682: Make `PyStackRef_CLOSE` escaping. (GH-129404)
2025-02-03 12:41:32 +00:00
JUMP_TO_ERROR();
}
gh-111789: Simplify bytecodes.c by using PyDict_GetItemRef() (GH-111978)
2023-11-14 15:38:49 +02:00
if (ann_dict == NULL) {
GH-119866: Spill the stack around escaping calls. (GH-124392) * Spill the evaluation around escaping calls in the generated interpreter and JIT. * The code generator tracks live, cached values so they can be saved to memory when needed. * Spills the stack pointer around escaping calls, so that the exact stack is visible to the cycle GC.
2024-10-07 14:56:39 +01:00
_PyFrame_SetStackPointer(frame, stack_pointer);
gh-111789: Simplify bytecodes.c by using PyDict_GetItemRef() (GH-111978)
2023-11-14 15:38:49 +02:00
ann_dict = PyDict_New();
GH-119866: Spill the stack around escaping calls. (GH-124392) * Spill the evaluation around escaping calls in the generated interpreter and JIT. * The code generator tracks live, cached values so they can be saved to memory when needed. * Spills the stack pointer around escaping calls, so that the exact stack is visible to the cycle GC.
2024-10-07 14:56:39 +01:00
stack_pointer = _PyFrame_GetStackPointer(frame);
GH-128682: Make `PyStackRef_CLOSE` escaping. (GH-129404)
2025-02-03 12:41:32 +00:00
if (ann_dict == NULL) {
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(0);
GH-128682: Make `PyStackRef_CLOSE` escaping. (GH-129404)
2025-02-03 12:41:32 +00:00
JUMP_TO_ERROR();
}
GH-119866: Spill the stack around escaping calls. (GH-124392) * Spill the evaluation around escaping calls in the generated interpreter and JIT. * The code generator tracks live, cached values so they can be saved to memory when needed. * Spills the stack pointer around escaping calls, so that the exact stack is visible to the cycle GC.
2024-10-07 14:56:39 +01:00
_PyFrame_SetStackPointer(frame, stack_pointer);
gh-111789: Simplify bytecodes.c by using PyDict_GetItemRef() (GH-111978)
2023-11-14 15:38:49 +02:00
err = PyObject_SetItem(LOCALS(), &_Py_ID(__annotations__),
ann_dict);
Py_DECREF(ann_dict);
GH-128682: Convert explicit loops closing arrays into `DECREF_INPUTS`. (GH-128822) * Mark Py_DECREF and Py_XDECREF as escaping * Remove explicit loops for clearing array inputs
2025-01-14 15:08:56 +00:00
stack_pointer = _PyFrame_GetStackPointer(frame);
GH-128682: Make `PyStackRef_CLOSE` escaping. (GH-129404)
2025-02-03 12:41:32 +00:00
if (err) {
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(0);
GH-128682: Make `PyStackRef_CLOSE` escaping. (GH-129404)
2025-02-03 12:41:32 +00:00
JUMP_TO_ERROR();
}
gh-104584: Baby steps towards generating and executing traces (#105924) Added a new, experimental, tracing optimizer and interpreter (a.k.a. "tier 2"). This currently pessimizes, so don't use yet -- this is infrastructure so we can experiment with optimizing passes. To enable it, pass ``-Xuops`` or set ``PYTHONUOPS=1``. To get debug output, set ``PYTHONUOPSDEBUG=N`` where ``N`` is a debug level (0-4, where 0 is no debug output and 4 is excessively verbose). All of this code is likely to change dramatically before the 3.13 feature freeze. But this is a first step.
2023-06-26 19:02:57 -07:00
}
else {
GH-128682: Convert explicit loops closing arrays into `DECREF_INPUTS`. (GH-128822) * Mark Py_DECREF and Py_XDECREF as escaping * Remove explicit loops for clearing array inputs
2025-01-14 15:08:56 +00:00
_PyFrame_SetStackPointer(frame, stack_pointer);
gh-111789: Simplify bytecodes.c by using PyDict_GetItemRef() (GH-111978)
2023-11-14 15:38:49 +02:00
Py_DECREF(ann_dict);
GH-128682: Convert explicit loops closing arrays into `DECREF_INPUTS`. (GH-128822) * Mark Py_DECREF and Py_XDECREF as escaping * Remove explicit loops for clearing array inputs
2025-01-14 15:08:56 +00:00
stack_pointer = _PyFrame_GetStackPointer(frame);
gh-104584: Baby steps towards generating and executing traces (#105924) Added a new, experimental, tracing optimizer and interpreter (a.k.a. "tier 2"). This currently pessimizes, so don't use yet -- this is infrastructure so we can experiment with optimizing passes. To enable it, pass ``-Xuops`` or set ``PYTHONUOPS=1``. To get debug output, set ``PYTHONUOPSDEBUG=N`` where ``N`` is a debug level (0-4, where 0 is no debug output and 4 is excessively verbose). All of this code is likely to change dramatically before the 3.13 feature freeze. But this is a first step.
2023-06-26 19:02:57 -07:00
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_tos_cache0 = PyStackRef_ZERO_BITS;
_tos_cache1 = PyStackRef_ZERO_BITS;
_tos_cache2 = PyStackRef_ZERO_BITS;
SET_CURRENT_CACHED_VALUES(0);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-104584: Baby steps towards generating and executing traces (#105924) Added a new, experimental, tracing optimizer and interpreter (a.k.a. "tier 2"). This currently pessimizes, so don't use yet -- this is infrastructure so we can experiment with optimizing passes. To enable it, pass ``-Xuops`` or set ``PYTHONUOPS=1``. To get debug output, set ``PYTHONUOPSDEBUG=N`` where ``N`` is a debug level (0-4, where 0 is no debug output and 4 is excessively verbose). All of this code is likely to change dramatically before the 3.13 feature freeze. But this is a first step.
2023-06-26 19:02:57 -07:00
break;
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
case _DICT_UPDATE_r10: {
CHECK_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
_PyStackRef update;
_PyStackRef dict;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef _stack_item_0 = _tos_cache0;
GH-111485: Factor out tier 2 code generation from the rest of the interpreter code generator (GH-112968)
2023-12-12 12:12:17 +00:00
oparg = CURRENT_OPARG();
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
update = _stack_item_0;
dict = stack_pointer[-1 - (oparg - 1)];
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
PyObject *dict_o = PyStackRef_AsPyObjectBorrow(dict);
PyObject *update_o = PyStackRef_AsPyObjectBorrow(update);
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
stack_pointer[0] = update;
stack_pointer += 1;
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
GH-119866: Spill the stack around escaping calls. (GH-124392) * Spill the evaluation around escaping calls in the generated interpreter and JIT. * The code generator tracks live, cached values so they can be saved to memory when needed. * Spills the stack pointer around escaping calls, so that the exact stack is visible to the cycle GC.
2024-10-07 14:56:39 +01:00
_PyFrame_SetStackPointer(frame, stack_pointer);
GH-120024: Refactor code a bit so that escaping calls can be wrapped in spill code in code generator (GH-122693)
2024-08-06 08:40:39 +01:00
int err = PyDict_Update(dict_o, update_o);
GH-119866: Spill the stack around escaping calls. (GH-124392) * Spill the evaluation around escaping calls in the generated interpreter and JIT. * The code generator tracks live, cached values so they can be saved to memory when needed. * Spills the stack pointer around escaping calls, so that the exact stack is visible to the cycle GC.
2024-10-07 14:56:39 +01:00
stack_pointer = _PyFrame_GetStackPointer(frame);
GH-120024: Refactor code a bit so that escaping calls can be wrapped in spill code in code generator (GH-122693)
2024-08-06 08:40:39 +01:00
if (err < 0) {
GH-119866: Spill the stack around escaping calls. (GH-124392) * Spill the evaluation around escaping calls in the generated interpreter and JIT. * The code generator tracks live, cached values so they can be saved to memory when needed. * Spills the stack pointer around escaping calls, so that the exact stack is visible to the cycle GC.
2024-10-07 14:56:39 +01:00
_PyFrame_SetStackPointer(frame, stack_pointer);
GH-120024: Refactor code a bit so that escaping calls can be wrapped in spill code in code generator (GH-122693)
2024-08-06 08:40:39 +01:00
int matches = _PyErr_ExceptionMatches(tstate, PyExc_AttributeError);
GH-119866: Spill the stack around escaping calls. (GH-124392) * Spill the evaluation around escaping calls in the generated interpreter and JIT. * The code generator tracks live, cached values so they can be saved to memory when needed. * Spills the stack pointer around escaping calls, so that the exact stack is visible to the cycle GC.
2024-10-07 14:56:39 +01:00
stack_pointer = _PyFrame_GetStackPointer(frame);
GH-120024: Refactor code a bit so that escaping calls can be wrapped in spill code in code generator (GH-122693)
2024-08-06 08:40:39 +01:00
if (matches) {
GH-119866: Spill the stack around escaping calls. (GH-124392) * Spill the evaluation around escaping calls in the generated interpreter and JIT. * The code generator tracks live, cached values so they can be saved to memory when needed. * Spills the stack pointer around escaping calls, so that the exact stack is visible to the cycle GC.
2024-10-07 14:56:39 +01:00
_PyFrame_SetStackPointer(frame, stack_pointer);
gh-104584: Baby steps towards generating and executing traces (#105924) Added a new, experimental, tracing optimizer and interpreter (a.k.a. "tier 2"). This currently pessimizes, so don't use yet -- this is infrastructure so we can experiment with optimizing passes. To enable it, pass ``-Xuops`` or set ``PYTHONUOPS=1``. To get debug output, set ``PYTHONUOPSDEBUG=N`` where ``N`` is a debug level (0-4, where 0 is no debug output and 4 is excessively verbose). All of this code is likely to change dramatically before the 3.13 feature freeze. But this is a first step.
2023-06-26 19:02:57 -07:00
_PyErr_Format(tstate, PyExc_TypeError,
GH-111485: Factor out tier 2 code generation from the rest of the interpreter code generator (GH-112968)
2023-12-12 12:12:17 +00:00
"'%.200s' object is not a mapping",
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
Py_TYPE(update_o)->tp_name);
GH-119866: Spill the stack around escaping calls. (GH-124392) * Spill the evaluation around escaping calls in the generated interpreter and JIT. * The code generator tracks live, cached values so they can be saved to memory when needed. * Spills the stack pointer around escaping calls, so that the exact stack is visible to the cycle GC.
2024-10-07 14:56:39 +01:00
stack_pointer = _PyFrame_GetStackPointer(frame);
gh-104584: Baby steps towards generating and executing traces (#105924) Added a new, experimental, tracing optimizer and interpreter (a.k.a. "tier 2"). This currently pessimizes, so don't use yet -- this is infrastructure so we can experiment with optimizing passes. To enable it, pass ``-Xuops`` or set ``PYTHONUOPS=1``. To get debug output, set ``PYTHONUOPSDEBUG=N`` where ``N`` is a debug level (0-4, where 0 is no debug output and 4 is excessively verbose). All of this code is likely to change dramatically before the 3.13 feature freeze. But this is a first step.
2023-06-26 19:02:57 -07:00
}
GH-128682: Make `PyStackRef_CLOSE` escaping. (GH-129404)
2025-02-03 12:41:32 +00:00
stack_pointer += -1;
GH-141794: Limit size of generated machine code. (GH-142228) * Factor out bodies of the largest uops, to reduce jit code size. * Factor out common assert, also reducing jit code size. * Limit size of jitted code for a single executor to 1MB.
2025-12-03 17:43:35 +00:00
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
GH-128682: Account for escapes in `DECREF_INPUTS` (GH-129953) * Handle escapes in DECREF_INPUTS * Mark a few more functions as escaping * Replace DECREF_INPUTS with PyStackRef_CLOSE where possible
2025-02-12 17:44:59 +00:00
_PyFrame_SetStackPointer(frame, stack_pointer);
PyStackRef_CLOSE(update);
stack_pointer = _PyFrame_GetStackPointer(frame);
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(0);
GH-128682: Make `PyStackRef_CLOSE` escaping. (GH-129404)
2025-02-03 12:41:32 +00:00
JUMP_TO_ERROR();
gh-104584: Baby steps towards generating and executing traces (#105924) Added a new, experimental, tracing optimizer and interpreter (a.k.a. "tier 2"). This currently pessimizes, so don't use yet -- this is infrastructure so we can experiment with optimizing passes. To enable it, pass ``-Xuops`` or set ``PYTHONUOPS=1``. To get debug output, set ``PYTHONUOPSDEBUG=N`` where ``N`` is a debug level (0-4, where 0 is no debug output and 4 is excessively verbose). All of this code is likely to change dramatically before the 3.13 feature freeze. But this is a first step.
2023-06-26 19:02:57 -07:00
}
GH-111485: Factor out tier 2 code generation from the rest of the interpreter code generator (GH-112968)
2023-12-12 12:12:17 +00:00
stack_pointer += -1;
GH-141794: Limit size of generated machine code. (GH-142228) * Factor out bodies of the largest uops, to reduce jit code size. * Factor out common assert, also reducing jit code size. * Limit size of jitted code for a single executor to 1MB.
2025-12-03 17:43:35 +00:00
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
GH-128682: Account for escapes in `DECREF_INPUTS` (GH-129953) * Handle escapes in DECREF_INPUTS * Mark a few more functions as escaping * Replace DECREF_INPUTS with PyStackRef_CLOSE where possible
2025-02-12 17:44:59 +00:00
_PyFrame_SetStackPointer(frame, stack_pointer);
PyStackRef_CLOSE(update);
stack_pointer = _PyFrame_GetStackPointer(frame);
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_tos_cache0 = PyStackRef_ZERO_BITS;
_tos_cache1 = PyStackRef_ZERO_BITS;
_tos_cache2 = PyStackRef_ZERO_BITS;
SET_CURRENT_CACHED_VALUES(0);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-104584: Baby steps towards generating and executing traces (#105924) Added a new, experimental, tracing optimizer and interpreter (a.k.a. "tier 2"). This currently pessimizes, so don't use yet -- this is infrastructure so we can experiment with optimizing passes. To enable it, pass ``-Xuops`` or set ``PYTHONUOPS=1``. To get debug output, set ``PYTHONUOPSDEBUG=N`` where ``N`` is a debug level (0-4, where 0 is no debug output and 4 is excessively verbose). All of this code is likely to change dramatically before the 3.13 feature freeze. But this is a first step.
2023-06-26 19:02:57 -07:00
break;
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
case _DICT_MERGE_r10: {
CHECK_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
_PyStackRef update;
_PyStackRef dict;
_PyStackRef callable;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef _stack_item_0 = _tos_cache0;
GH-111485: Factor out tier 2 code generation from the rest of the interpreter code generator (GH-112968)
2023-12-12 12:12:17 +00:00
oparg = CURRENT_OPARG();
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
update = _stack_item_0;
dict = stack_pointer[-1 - (oparg - 1)];
callable = stack_pointer[-4 - (oparg - 1)];
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
PyObject *callable_o = PyStackRef_AsPyObjectBorrow(callable);
PyObject *dict_o = PyStackRef_AsPyObjectBorrow(dict);
PyObject *update_o = PyStackRef_AsPyObjectBorrow(update);
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
stack_pointer[0] = update;
stack_pointer += 1;
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
GH-119866: Spill the stack around escaping calls. (GH-124392) * Spill the evaluation around escaping calls in the generated interpreter and JIT. * The code generator tracks live, cached values so they can be saved to memory when needed. * Spills the stack pointer around escaping calls, so that the exact stack is visible to the cycle GC.
2024-10-07 14:56:39 +01:00
_PyFrame_SetStackPointer(frame, stack_pointer);
GH-120024: Refactor code a bit so that escaping calls can be wrapped in spill code in code generator (GH-122693)
2024-08-06 08:40:39 +01:00
int err = _PyDict_MergeEx(dict_o, update_o, 2);
GH-119866: Spill the stack around escaping calls. (GH-124392) * Spill the evaluation around escaping calls in the generated interpreter and JIT. * The code generator tracks live, cached values so they can be saved to memory when needed. * Spills the stack pointer around escaping calls, so that the exact stack is visible to the cycle GC.
2024-10-07 14:56:39 +01:00
stack_pointer = _PyFrame_GetStackPointer(frame);
GH-120024: Refactor code a bit so that escaping calls can be wrapped in spill code in code generator (GH-122693)
2024-08-06 08:40:39 +01:00
if (err < 0) {
GH-119866: Spill the stack around escaping calls. (GH-124392) * Spill the evaluation around escaping calls in the generated interpreter and JIT. * The code generator tracks live, cached values so they can be saved to memory when needed. * Spills the stack pointer around escaping calls, so that the exact stack is visible to the cycle GC.
2024-10-07 14:56:39 +01:00
_PyFrame_SetStackPointer(frame, stack_pointer);
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
_PyEval_FormatKwargsError(tstate, callable_o, update_o);
GH-119866: Spill the stack around escaping calls. (GH-124392) * Spill the evaluation around escaping calls in the generated interpreter and JIT. * The code generator tracks live, cached values so they can be saved to memory when needed. * Spills the stack pointer around escaping calls, so that the exact stack is visible to the cycle GC.
2024-10-07 14:56:39 +01:00
stack_pointer = _PyFrame_GetStackPointer(frame);
GH-128682: Make `PyStackRef_CLOSE` escaping. (GH-129404)
2025-02-03 12:41:32 +00:00
stack_pointer += -1;
GH-141794: Limit size of generated machine code. (GH-142228) * Factor out bodies of the largest uops, to reduce jit code size. * Factor out common assert, also reducing jit code size. * Limit size of jitted code for a single executor to 1MB.
2025-12-03 17:43:35 +00:00
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
GH-128682: Account for escapes in `DECREF_INPUTS` (GH-129953) * Handle escapes in DECREF_INPUTS * Mark a few more functions as escaping * Replace DECREF_INPUTS with PyStackRef_CLOSE where possible
2025-02-12 17:44:59 +00:00
_PyFrame_SetStackPointer(frame, stack_pointer);
PyStackRef_CLOSE(update);
stack_pointer = _PyFrame_GetStackPointer(frame);
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(0);
GH-128682: Make `PyStackRef_CLOSE` escaping. (GH-129404)
2025-02-03 12:41:32 +00:00
JUMP_TO_ERROR();
gh-104584: Baby steps towards generating and executing traces (#105924) Added a new, experimental, tracing optimizer and interpreter (a.k.a. "tier 2"). This currently pessimizes, so don't use yet -- this is infrastructure so we can experiment with optimizing passes. To enable it, pass ``-Xuops`` or set ``PYTHONUOPS=1``. To get debug output, set ``PYTHONUOPSDEBUG=N`` where ``N`` is a debug level (0-4, where 0 is no debug output and 4 is excessively verbose). All of this code is likely to change dramatically before the 3.13 feature freeze. But this is a first step.
2023-06-26 19:02:57 -07:00
}
GH-111485: Factor out tier 2 code generation from the rest of the interpreter code generator (GH-112968)
2023-12-12 12:12:17 +00:00
stack_pointer += -1;
GH-141794: Limit size of generated machine code. (GH-142228) * Factor out bodies of the largest uops, to reduce jit code size. * Factor out common assert, also reducing jit code size. * Limit size of jitted code for a single executor to 1MB.
2025-12-03 17:43:35 +00:00
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
GH-128682: Account for escapes in `DECREF_INPUTS` (GH-129953) * Handle escapes in DECREF_INPUTS * Mark a few more functions as escaping * Replace DECREF_INPUTS with PyStackRef_CLOSE where possible
2025-02-12 17:44:59 +00:00
_PyFrame_SetStackPointer(frame, stack_pointer);
PyStackRef_CLOSE(update);
stack_pointer = _PyFrame_GetStackPointer(frame);
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_tos_cache0 = PyStackRef_ZERO_BITS;
_tos_cache1 = PyStackRef_ZERO_BITS;
_tos_cache2 = PyStackRef_ZERO_BITS;
SET_CURRENT_CACHED_VALUES(0);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-104584: Baby steps towards generating and executing traces (#105924) Added a new, experimental, tracing optimizer and interpreter (a.k.a. "tier 2"). This currently pessimizes, so don't use yet -- this is infrastructure so we can experiment with optimizing passes. To enable it, pass ``-Xuops`` or set ``PYTHONUOPS=1``. To get debug output, set ``PYTHONUOPSDEBUG=N`` where ``N`` is a debug level (0-4, where 0 is no debug output and 4 is excessively verbose). All of this code is likely to change dramatically before the 3.13 feature freeze. But this is a first step.
2023-06-26 19:02:57 -07:00
break;
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
case _MAP_ADD_r20: {
CHECK_CURRENT_CACHED_VALUES(2);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
_PyStackRef value;
_PyStackRef key;
_PyStackRef dict_st;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef _stack_item_0 = _tos_cache0;
_PyStackRef _stack_item_1 = _tos_cache1;
GH-111485: Factor out tier 2 code generation from the rest of the interpreter code generator (GH-112968)
2023-12-12 12:12:17 +00:00
oparg = CURRENT_OPARG();
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
value = _stack_item_1;
key = _stack_item_0;
dict_st = stack_pointer[-1 - (oparg - 1)];
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
PyObject *dict = PyStackRef_AsPyObjectBorrow(dict_st);
gh-104584: Baby steps towards generating and executing traces (#105924) Added a new, experimental, tracing optimizer and interpreter (a.k.a. "tier 2"). This currently pessimizes, so don't use yet -- this is infrastructure so we can experiment with optimizing passes. To enable it, pass ``-Xuops`` or set ``PYTHONUOPS=1``. To get debug output, set ``PYTHONUOPSDEBUG=N`` where ``N`` is a debug level (0-4, where 0 is no debug output and 4 is excessively verbose). All of this code is likely to change dramatically before the 3.13 feature freeze. But this is a first step.
2023-06-26 19:02:57 -07:00
assert(PyDict_CheckExact(dict));
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
stack_pointer[0] = key;
stack_pointer[1] = value;
stack_pointer += 2;
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
GH-119866: Spill the stack around escaping calls. (GH-124392) * Spill the evaluation around escaping calls in the generated interpreter and JIT. * The code generator tracks live, cached values so they can be saved to memory when needed. * Spills the stack pointer around escaping calls, so that the exact stack is visible to the cycle GC.
2024-10-07 14:56:39 +01:00
_PyFrame_SetStackPointer(frame, stack_pointer);
GH-120024: Move three more escaping calls out of conditional statements (GH-122734)
2024-08-06 14:14:52 +01:00
int err = _PyDict_SetItem_Take2(
(PyDictObject *)dict,
PyStackRef_AsPyObjectSteal(key),
PyStackRef_AsPyObjectSteal(value)
);
GH-119866: Spill the stack around escaping calls. (GH-124392) * Spill the evaluation around escaping calls in the generated interpreter and JIT. * The code generator tracks live, cached values so they can be saved to memory when needed. * Spills the stack pointer around escaping calls, so that the exact stack is visible to the cycle GC.
2024-10-07 14:56:39 +01:00
stack_pointer = _PyFrame_GetStackPointer(frame);
GH-128682: Make `PyStackRef_CLOSE` escaping. (GH-129404)
2025-02-03 12:41:32 +00:00
if (err != 0) {
stack_pointer += -2;
GH-141794: Limit size of generated machine code. (GH-142228) * Factor out bodies of the largest uops, to reduce jit code size. * Factor out common assert, also reducing jit code size. * Limit size of jitted code for a single executor to 1MB.
2025-12-03 17:43:35 +00:00
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(0);
GH-128682: Make `PyStackRef_CLOSE` escaping. (GH-129404)
2025-02-03 12:41:32 +00:00
JUMP_TO_ERROR();
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_tos_cache0 = PyStackRef_ZERO_BITS;
_tos_cache1 = PyStackRef_ZERO_BITS;
_tos_cache2 = PyStackRef_ZERO_BITS;
SET_CURRENT_CACHED_VALUES(0);
GH-111485: Factor out tier 2 code generation from the rest of the interpreter code generator (GH-112968)
2023-12-12 12:12:17 +00:00
stack_pointer += -2;
GH-141794: Limit size of generated machine code. (GH-142228) * Factor out bodies of the largest uops, to reduce jit code size. * Factor out common assert, also reducing jit code size. * Limit size of jitted code for a single executor to 1MB.
2025-12-03 17:43:35 +00:00
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-104584: Baby steps towards generating and executing traces (#105924) Added a new, experimental, tracing optimizer and interpreter (a.k.a. "tier 2"). This currently pessimizes, so don't use yet -- this is infrastructure so we can experiment with optimizing passes. To enable it, pass ``-Xuops`` or set ``PYTHONUOPS=1``. To get debug output, set ``PYTHONUOPSDEBUG=N`` where ``N`` is a debug level (0-4, where 0 is no debug output and 4 is excessively verbose). All of this code is likely to change dramatically before the 3.13 feature freeze. But this is a first step.
2023-06-26 19:02:57 -07:00
break;
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
case _LOAD_SUPER_ATTR_ATTR_r31: {
CHECK_CURRENT_CACHED_VALUES(3);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
_PyStackRef self_st;
_PyStackRef class_st;
_PyStackRef global_super_st;
_PyStackRef attr_st;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef _stack_item_0 = _tos_cache0;
_PyStackRef _stack_item_1 = _tos_cache1;
_PyStackRef _stack_item_2 = _tos_cache2;
GH-111485: Factor out tier 2 code generation from the rest of the interpreter code generator (GH-112968)
2023-12-12 12:12:17 +00:00
oparg = CURRENT_OPARG();
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
self_st = _stack_item_2;
class_st = _stack_item_1;
global_super_st = _stack_item_0;
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
PyObject *global_super = PyStackRef_AsPyObjectBorrow(global_super_st);
PyObject *class = PyStackRef_AsPyObjectBorrow(class_st);
PyObject *self = PyStackRef_AsPyObjectBorrow(self_st);
gh-104584: Baby steps towards generating and executing traces (#105924) Added a new, experimental, tracing optimizer and interpreter (a.k.a. "tier 2"). This currently pessimizes, so don't use yet -- this is infrastructure so we can experiment with optimizing passes. To enable it, pass ``-Xuops`` or set ``PYTHONUOPS=1``. To get debug output, set ``PYTHONUOPSDEBUG=N`` where ``N`` is a debug level (0-4, where 0 is no debug output and 4 is excessively verbose). All of this code is likely to change dramatically before the 3.13 feature freeze. But this is a first step.
2023-06-26 19:02:57 -07:00
assert(!(oparg & 1));
GH-117457: Correct pystats uop "miss" counts (GH-117477)
2024-04-04 18:49:18 -04:00
if (global_super != (PyObject *)&PySuper_Type) {
UOP_STAT_INC(uopcode, miss);
GH-142718: JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. (GH-142762) JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. * Make sure that stack is in correct state at side exits with TOS cached values * Simplify choice of cached items for side exits
2025-12-15 19:01:29 +00:00
_tos_cache2 = self_st;
_tos_cache1 = class_st;
_tos_cache0 = global_super_st;
SET_CURRENT_CACHED_VALUES(3);
GH-117457: Correct pystats uop "miss" counts (GH-117477)
2024-04-04 18:49:18 -04:00
JUMP_TO_JUMP_TARGET();
}
if (!PyType_Check(class)) {
UOP_STAT_INC(uopcode, miss);
GH-142718: JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. (GH-142762) JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. * Make sure that stack is in correct state at side exits with TOS cached values * Simplify choice of cached items for side exits
2025-12-15 19:01:29 +00:00
_tos_cache2 = self_st;
_tos_cache1 = class_st;
_tos_cache0 = global_super_st;
SET_CURRENT_CACHED_VALUES(3);
GH-117457: Correct pystats uop "miss" counts (GH-117477)
2024-04-04 18:49:18 -04:00
JUMP_TO_JUMP_TARGET();
}
gh-104584: Baby steps towards generating and executing traces (#105924) Added a new, experimental, tracing optimizer and interpreter (a.k.a. "tier 2"). This currently pessimizes, so don't use yet -- this is infrastructure so we can experiment with optimizing passes. To enable it, pass ``-Xuops`` or set ``PYTHONUOPS=1``. To get debug output, set ``PYTHONUOPSDEBUG=N`` where ``N`` is a debug level (0-4, where 0 is no debug output and 4 is excessively verbose). All of this code is likely to change dramatically before the 3.13 feature freeze. But this is a first step.
2023-06-26 19:02:57 -07:00
STAT_INC(LOAD_SUPER_ATTR, hit);
PyObject *name = GETITEM(FRAME_CO_NAMES, oparg >> 2);
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
stack_pointer[0] = global_super_st;
stack_pointer[1] = class_st;
stack_pointer[2] = self_st;
stack_pointer += 3;
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
GH-119866: Spill the stack around escaping calls. (GH-124392) * Spill the evaluation around escaping calls in the generated interpreter and JIT. * The code generator tracks live, cached values so they can be saved to memory when needed. * Spills the stack pointer around escaping calls, so that the exact stack is visible to the cycle GC.
2024-10-07 14:56:39 +01:00
_PyFrame_SetStackPointer(frame, stack_pointer);
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
PyObject *attr = _PySuper_Lookup((PyTypeObject *)class, self, name, NULL);
GH-128682: Account for escapes in `DECREF_INPUTS` (GH-129953) * Handle escapes in DECREF_INPUTS * Mark a few more functions as escaping * Replace DECREF_INPUTS with PyStackRef_CLOSE where possible
2025-02-12 17:44:59 +00:00
_PyStackRef tmp = self_st;
self_st = PyStackRef_NULL;
stack_pointer[-1] = self_st;
PyStackRef_CLOSE(tmp);
tmp = class_st;
class_st = PyStackRef_NULL;
stack_pointer[-2] = class_st;
PyStackRef_CLOSE(tmp);
tmp = global_super_st;
global_super_st = PyStackRef_NULL;
stack_pointer[-3] = global_super_st;
PyStackRef_CLOSE(tmp);
GH-119866: Spill the stack around escaping calls. (GH-124392) * Spill the evaluation around escaping calls in the generated interpreter and JIT. * The code generator tracks live, cached values so they can be saved to memory when needed. * Spills the stack pointer around escaping calls, so that the exact stack is visible to the cycle GC.
2024-10-07 14:56:39 +01:00
stack_pointer = _PyFrame_GetStackPointer(frame);
GH-128682: Account for escapes in `DECREF_INPUTS` (GH-129953) * Handle escapes in DECREF_INPUTS * Mark a few more functions as escaping * Replace DECREF_INPUTS with PyStackRef_CLOSE where possible
2025-02-12 17:44:59 +00:00
stack_pointer += -3;
GH-141794: Limit size of generated machine code. (GH-142228) * Factor out bodies of the largest uops, to reduce jit code size. * Factor out common assert, also reducing jit code size. * Limit size of jitted code for a single executor to 1MB.
2025-12-03 17:43:35 +00:00
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
GH-128682: Make `PyStackRef_CLOSE` escaping. (GH-129404)
2025-02-03 12:41:32 +00:00
if (attr == NULL) {
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(0);
GH-128682: Make `PyStackRef_CLOSE` escaping. (GH-129404)
2025-02-03 12:41:32 +00:00
JUMP_TO_ERROR();
}
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
attr_st = PyStackRef_FromPyObjectSteal(attr);
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_tos_cache0 = attr_st;
_tos_cache1 = PyStackRef_ZERO_BITS;
_tos_cache2 = PyStackRef_ZERO_BITS;
SET_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-104584: Baby steps towards generating and executing traces (#105924) Added a new, experimental, tracing optimizer and interpreter (a.k.a. "tier 2"). This currently pessimizes, so don't use yet -- this is infrastructure so we can experiment with optimizing passes. To enable it, pass ``-Xuops`` or set ``PYTHONUOPS=1``. To get debug output, set ``PYTHONUOPSDEBUG=N`` where ``N`` is a debug level (0-4, where 0 is no debug output and 4 is excessively verbose). All of this code is likely to change dramatically before the 3.13 feature freeze. But this is a first step.
2023-06-26 19:02:57 -07:00
break;
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
case _LOAD_SUPER_ATTR_METHOD_r32: {
CHECK_CURRENT_CACHED_VALUES(3);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
_PyStackRef self_st;
_PyStackRef class_st;
_PyStackRef global_super_st;
_PyStackRef attr;
_PyStackRef self_or_null;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef _stack_item_0 = _tos_cache0;
_PyStackRef _stack_item_1 = _tos_cache1;
_PyStackRef _stack_item_2 = _tos_cache2;
GH-111485: Factor out tier 2 code generation from the rest of the interpreter code generator (GH-112968)
2023-12-12 12:12:17 +00:00
oparg = CURRENT_OPARG();
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
self_st = _stack_item_2;
class_st = _stack_item_1;
global_super_st = _stack_item_0;
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
PyObject *global_super = PyStackRef_AsPyObjectBorrow(global_super_st);
PyObject *class = PyStackRef_AsPyObjectBorrow(class_st);
PyObject *self = PyStackRef_AsPyObjectBorrow(self_st);
gh-104584: Baby steps towards generating and executing traces (#105924) Added a new, experimental, tracing optimizer and interpreter (a.k.a. "tier 2"). This currently pessimizes, so don't use yet -- this is infrastructure so we can experiment with optimizing passes. To enable it, pass ``-Xuops`` or set ``PYTHONUOPS=1``. To get debug output, set ``PYTHONUOPSDEBUG=N`` where ``N`` is a debug level (0-4, where 0 is no debug output and 4 is excessively verbose). All of this code is likely to change dramatically before the 3.13 feature freeze. But this is a first step.
2023-06-26 19:02:57 -07:00
assert(oparg & 1);
GH-117457: Correct pystats uop "miss" counts (GH-117477)
2024-04-04 18:49:18 -04:00
if (global_super != (PyObject *)&PySuper_Type) {
UOP_STAT_INC(uopcode, miss);
GH-142718: JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. (GH-142762) JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. * Make sure that stack is in correct state at side exits with TOS cached values * Simplify choice of cached items for side exits
2025-12-15 19:01:29 +00:00
_tos_cache2 = self_st;
_tos_cache1 = class_st;
_tos_cache0 = global_super_st;
SET_CURRENT_CACHED_VALUES(3);
GH-117457: Correct pystats uop "miss" counts (GH-117477)
2024-04-04 18:49:18 -04:00
JUMP_TO_JUMP_TARGET();
}
if (!PyType_Check(class)) {
UOP_STAT_INC(uopcode, miss);
GH-142718: JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. (GH-142762) JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. * Make sure that stack is in correct state at side exits with TOS cached values * Simplify choice of cached items for side exits
2025-12-15 19:01:29 +00:00
_tos_cache2 = self_st;
_tos_cache1 = class_st;
_tos_cache0 = global_super_st;
SET_CURRENT_CACHED_VALUES(3);
GH-117457: Correct pystats uop "miss" counts (GH-117477)
2024-04-04 18:49:18 -04:00
JUMP_TO_JUMP_TARGET();
}
gh-104584: Baby steps towards generating and executing traces (#105924) Added a new, experimental, tracing optimizer and interpreter (a.k.a. "tier 2"). This currently pessimizes, so don't use yet -- this is infrastructure so we can experiment with optimizing passes. To enable it, pass ``-Xuops`` or set ``PYTHONUOPS=1``. To get debug output, set ``PYTHONUOPSDEBUG=N`` where ``N`` is a debug level (0-4, where 0 is no debug output and 4 is excessively verbose). All of this code is likely to change dramatically before the 3.13 feature freeze. But this is a first step.
2023-06-26 19:02:57 -07:00
STAT_INC(LOAD_SUPER_ATTR, hit);
PyObject *name = GETITEM(FRAME_CO_NAMES, oparg >> 2);
PyTypeObject *cls = (PyTypeObject *)class;
int method_found = 0;
GH-139922: Tail calling for MSVC (VS 2026) (GH-143068) Co-authored-by: Ken Jin <28750310+Fidget-Spinner@users.noreply.github.com> Co-authored-by: Brandt Bucher <brandt@python.org> Co-authored-by: Hugo van Kemenade <1324225+hugovk@users.noreply.github.com>
2025-12-23 00:01:34 +01:00
PyObject *attr_o;
{
int *method_found_ptr = &method_found;
stack_pointer[0] = global_super_st;
stack_pointer[1] = class_st;
stack_pointer[2] = self_st;
stack_pointer += 3;
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
_PyFrame_SetStackPointer(frame, stack_pointer);
attr_o = _PySuper_Lookup(cls, self, name,
Py_TYPE(self)->tp_getattro == PyObject_GenericGetAttr ? method_found_ptr : NULL);
stack_pointer = _PyFrame_GetStackPointer(frame);
}
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
if (attr_o == NULL) {
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(0);
GH-128682: Stronger checking of `PyStackRef_CLOSE` and `DEAD`. (GH-128683)
2025-01-13 12:37:48 +00:00
JUMP_TO_ERROR();
gh-104584: Baby steps towards generating and executing traces (#105924) Added a new, experimental, tracing optimizer and interpreter (a.k.a. "tier 2"). This currently pessimizes, so don't use yet -- this is infrastructure so we can experiment with optimizing passes. To enable it, pass ``-Xuops`` or set ``PYTHONUOPS=1``. To get debug output, set ``PYTHONUOPSDEBUG=N`` where ``N`` is a debug level (0-4, where 0 is no debug output and 4 is excessively verbose). All of this code is likely to change dramatically before the 3.13 feature freeze. But this is a first step.
2023-06-26 19:02:57 -07:00
}
if (method_found) {
GH-131498: Cases generator: Parse down to C statement level. (GH-131948) * Parse down to statement level in the cases generator * Add handling for #if macros, treating them much like normal ifs.
2025-04-02 16:31:59 +01:00
self_or_null = self_st;
gh-104584: Baby steps towards generating and executing traces (#105924) Added a new, experimental, tracing optimizer and interpreter (a.k.a. "tier 2"). This currently pessimizes, so don't use yet -- this is infrastructure so we can experiment with optimizing passes. To enable it, pass ``-Xuops`` or set ``PYTHONUOPS=1``. To get debug output, set ``PYTHONUOPSDEBUG=N`` where ``N`` is a debug level (0-4, where 0 is no debug output and 4 is excessively verbose). All of this code is likely to change dramatically before the 3.13 feature freeze. But this is a first step.
2023-06-26 19:02:57 -07:00
} else {
GH-128682: Make `PyStackRef_CLOSE` escaping. (GH-129404)
2025-02-03 12:41:32 +00:00
stack_pointer += -1;
GH-141794: Limit size of generated machine code. (GH-142228) * Factor out bodies of the largest uops, to reduce jit code size. * Factor out common assert, also reducing jit code size. * Limit size of jitted code for a single executor to 1MB.
2025-12-03 17:43:35 +00:00
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
GH-128682: Make `PyStackRef_CLOSE` escaping. (GH-129404)
2025-02-03 12:41:32 +00:00
_PyFrame_SetStackPointer(frame, stack_pointer);
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
PyStackRef_CLOSE(self_st);
GH-128682: Make `PyStackRef_CLOSE` escaping. (GH-129404)
2025-02-03 12:41:32 +00:00
stack_pointer = _PyFrame_GetStackPointer(frame);
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
self_or_null = PyStackRef_NULL;
GH-128682: Make `PyStackRef_CLOSE` escaping. (GH-129404)
2025-02-03 12:41:32 +00:00
stack_pointer += 1;
gh-104584: Baby steps towards generating and executing traces (#105924) Added a new, experimental, tracing optimizer and interpreter (a.k.a. "tier 2"). This currently pessimizes, so don't use yet -- this is infrastructure so we can experiment with optimizing passes. To enable it, pass ``-Xuops`` or set ``PYTHONUOPS=1``. To get debug output, set ``PYTHONUOPSDEBUG=N`` where ``N`` is a debug level (0-4, where 0 is no debug output and 4 is excessively verbose). All of this code is likely to change dramatically before the 3.13 feature freeze. But this is a first step.
2023-06-26 19:02:57 -07:00
}
GH-128682: Account for escapes in `DECREF_INPUTS` (GH-129953) * Handle escapes in DECREF_INPUTS * Mark a few more functions as escaping * Replace DECREF_INPUTS with PyStackRef_CLOSE where possible
2025-02-12 17:44:59 +00:00
stack_pointer += -1;
GH-141794: Limit size of generated machine code. (GH-142228) * Factor out bodies of the largest uops, to reduce jit code size. * Factor out common assert, also reducing jit code size. * Limit size of jitted code for a single executor to 1MB.
2025-12-03 17:43:35 +00:00
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
GH-128682: Account for escapes in `DECREF_INPUTS` (GH-129953) * Handle escapes in DECREF_INPUTS * Mark a few more functions as escaping * Replace DECREF_INPUTS with PyStackRef_CLOSE where possible
2025-02-12 17:44:59 +00:00
_PyFrame_SetStackPointer(frame, stack_pointer);
_PyStackRef tmp = global_super_st;
global_super_st = self_or_null;
stack_pointer[-2] = global_super_st;
PyStackRef_CLOSE(tmp);
GH-131729: Code-gen better liveness analysis (GH-131732) * Rename 'defined' attribute to 'in_local' to more accurately reflect how it is used * Make death of variables explicit even for array variables. * Convert in_memory from boolean to stack offset * Don't apply liveness analysis to optimizer generated code * Fix RETURN_VALUE in optimizer
2025-03-26 15:21:35 +00:00
tmp = class_st;
class_st = PyStackRef_NULL;
stack_pointer[-1] = class_st;
PyStackRef_CLOSE(tmp);
GH-128682: Account for escapes in `DECREF_INPUTS` (GH-129953) * Handle escapes in DECREF_INPUTS * Mark a few more functions as escaping * Replace DECREF_INPUTS with PyStackRef_CLOSE where possible
2025-02-12 17:44:59 +00:00
stack_pointer = _PyFrame_GetStackPointer(frame);
GH-131729: Code-gen better liveness analysis (GH-131732) * Rename 'defined' attribute to 'in_local' to more accurately reflect how it is used * Make death of variables explicit even for array variables. * Convert in_memory from boolean to stack offset * Don't apply liveness analysis to optimizer generated code * Fix RETURN_VALUE in optimizer
2025-03-26 15:21:35 +00:00
stack_pointer += -2;
GH-141794: Limit size of generated machine code. (GH-142228) * Factor out bodies of the largest uops, to reduce jit code size. * Factor out common assert, also reducing jit code size. * Limit size of jitted code for a single executor to 1MB.
2025-12-03 17:43:35 +00:00
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
GH-128682: Account for escapes in `DECREF_INPUTS` (GH-129953) * Handle escapes in DECREF_INPUTS * Mark a few more functions as escaping * Replace DECREF_INPUTS with PyStackRef_CLOSE where possible
2025-02-12 17:44:59 +00:00
attr = PyStackRef_FromPyObjectSteal(attr_o);
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_tos_cache1 = self_or_null;
_tos_cache0 = attr;
_tos_cache2 = PyStackRef_ZERO_BITS;
SET_CURRENT_CACHED_VALUES(2);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-104584: Baby steps towards generating and executing traces (#105924) Added a new, experimental, tracing optimizer and interpreter (a.k.a. "tier 2"). This currently pessimizes, so don't use yet -- this is infrastructure so we can experiment with optimizing passes. To enable it, pass ``-Xuops`` or set ``PYTHONUOPS=1``. To get debug output, set ``PYTHONUOPSDEBUG=N`` where ``N`` is a debug level (0-4, where 0 is no debug output and 4 is excessively verbose). All of this code is likely to change dramatically before the 3.13 feature freeze. But this is a first step.
2023-06-26 19:02:57 -07:00
break;
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
case _LOAD_ATTR_r10: {
CHECK_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
_PyStackRef owner;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef attr;
GH-128914: Remove all but one conditional stack effects (GH-129226) * Remove all 'if (0)' and 'if (1)' conditional stack effects * Use array instead of conditional for BUILD_SLICE args * Refactor LOAD_GLOBAL to use a common conditional uop * Remove conditional stack effects from LOAD_ATTR specializations * Replace conditional stack effects in LOAD_ATTR with a 0 or 1 sized array. * Remove conditional stack effects from CALL_FUNCTION_EX
2025-01-27 16:24:48 +00:00
_PyStackRef *self_or_null;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef _stack_item_0 = _tos_cache0;
GH-111485: Factor out tier 2 code generation from the rest of the interpreter code generator (GH-112968)
2023-12-12 12:12:17 +00:00
oparg = CURRENT_OPARG();
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
owner = _stack_item_0;
self_or_null = &stack_pointer[1];
Revert "GH-128914: Remove conditional stack effects from `bytecodes.c` and the code generators (GH-128918)" (GH-129202) The commit introduced a ~2.5-3% regression in the free threading build. This reverts commit ab61d3f4303d14a413bc9ae6557c730ffdf7579e.
2025-01-23 04:26:25 -05:00
PyObject *name = GETITEM(FRAME_CO_NAMES, oparg >> 1);
if (oparg & 1) {
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyCStackRef method;
stack_pointer[0] = owner;
stack_pointer += 1;
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
Revert "GH-128914: Remove conditional stack effects from `bytecodes.c` and the code generators (GH-128918)" (GH-129202) The commit introduced a ~2.5-3% regression in the free threading build. This reverts commit ab61d3f4303d14a413bc9ae6557c730ffdf7579e.
2025-01-23 04:26:25 -05:00
_PyFrame_SetStackPointer(frame, stack_pointer);
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyThreadState_PushCStackRef(tstate, &method);
int is_meth = _PyObject_GetMethodStackRef(tstate, PyStackRef_AsPyObjectBorrow(owner), name, &method.ref);
Revert "GH-128914: Remove conditional stack effects from `bytecodes.c` and the code generators (GH-128918)" (GH-129202) The commit introduced a ~2.5-3% regression in the free threading build. This reverts commit ab61d3f4303d14a413bc9ae6557c730ffdf7579e.
2025-01-23 04:26:25 -05:00
stack_pointer = _PyFrame_GetStackPointer(frame);
if (is_meth) {
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
assert(!PyStackRef_IsNull(method.ref));
GH-131498: Cases generator: Parse down to C statement level. (GH-131948) * Parse down to statement level in the cases generator * Add handling for #if macros, treating them much like normal ifs.
2025-04-02 16:31:59 +01:00
self_or_null[0] = owner;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
attr = _PyThreadState_PopCStackRefSteal(tstate, &method);
Revert "GH-128914: Remove conditional stack effects from `bytecodes.c` and the code generators (GH-128918)" (GH-129202) The commit introduced a ~2.5-3% regression in the free threading build. This reverts commit ab61d3f4303d14a413bc9ae6557c730ffdf7579e.
2025-01-23 04:26:25 -05:00
}
else {
GH-128682: Account for escapes in `DECREF_INPUTS` (GH-129953) * Handle escapes in DECREF_INPUTS * Mark a few more functions as escaping * Replace DECREF_INPUTS with PyStackRef_CLOSE where possible
2025-02-12 17:44:59 +00:00
stack_pointer += -1;
GH-141794: Limit size of generated machine code. (GH-142228) * Factor out bodies of the largest uops, to reduce jit code size. * Factor out common assert, also reducing jit code size. * Limit size of jitted code for a single executor to 1MB.
2025-12-03 17:43:35 +00:00
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
GH-128682: Account for escapes in `DECREF_INPUTS` (GH-129953) * Handle escapes in DECREF_INPUTS * Mark a few more functions as escaping * Replace DECREF_INPUTS with PyStackRef_CLOSE where possible
2025-02-12 17:44:59 +00:00
_PyFrame_SetStackPointer(frame, stack_pointer);
Revert "GH-128914: Remove conditional stack effects from `bytecodes.c` and the code generators (GH-128918)" (GH-129202) The commit introduced a ~2.5-3% regression in the free threading build. This reverts commit ab61d3f4303d14a413bc9ae6557c730ffdf7579e.
2025-01-23 04:26:25 -05:00
PyStackRef_CLOSE(owner);
GH-128682: Account for escapes in `DECREF_INPUTS` (GH-129953) * Handle escapes in DECREF_INPUTS * Mark a few more functions as escaping * Replace DECREF_INPUTS with PyStackRef_CLOSE where possible
2025-02-12 17:44:59 +00:00
stack_pointer = _PyFrame_GetStackPointer(frame);
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
self_or_null[0] = PyStackRef_NULL;
attr = _PyThreadState_PopCStackRefSteal(tstate, &method);
if (PyStackRef_IsNull(attr)) {
SET_CURRENT_CACHED_VALUES(0);
GH-128682: Make `PyStackRef_CLOSE` escaping. (GH-129404)
2025-02-03 12:41:32 +00:00
JUMP_TO_ERROR();
}
GH-128682: Account for escapes in `DECREF_INPUTS` (GH-129953) * Handle escapes in DECREF_INPUTS * Mark a few more functions as escaping * Replace DECREF_INPUTS with PyStackRef_CLOSE where possible
2025-02-12 17:44:59 +00:00
stack_pointer += 1;
Revert "GH-128914: Remove conditional stack effects from `bytecodes.c` and the code generators (GH-128918)" (GH-129202) The commit introduced a ~2.5-3% regression in the free threading build. This reverts commit ab61d3f4303d14a413bc9ae6557c730ffdf7579e.
2025-01-23 04:26:25 -05:00
}
gh-104584: Allow unspecialized instructions in superblocks (#106497) This adds several of unspecialized opcodes to superblocks: TO_BOOL, BINARY_SUBSCR, STORE_SUBSCR, UNPACK_SEQUENCE, LOAD_GLOBAL, LOAD_ATTR, COMPARE_OP, BINARY_OP. While we may not want that eventually, for now this helps finding bugs. There is a rudimentary test checking for UNPACK_SEQUENCE. Once we're ready to undo this, that would be simple: just replace the call to variable_used_unspecialized with a call to variable_used (as shown in a comment). Or add individual opcdes to FORBIDDEN_NAMES_IN_UOPS.
2023-07-07 11:03:27 -07:00
}
else {
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
stack_pointer[0] = owner;
stack_pointer += 1;
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
Revert "GH-128914: Remove conditional stack effects from `bytecodes.c` and the code generators (GH-128918)" (GH-129202) The commit introduced a ~2.5-3% regression in the free threading build. This reverts commit ab61d3f4303d14a413bc9ae6557c730ffdf7579e.
2025-01-23 04:26:25 -05:00
_PyFrame_SetStackPointer(frame, stack_pointer);
gh-136125: Use `_PyObject_GetMethodStackRef` for `LOAD_ATTR` (GH-136127)
2025-07-02 01:20:46 +08:00
PyObject *attr_o = PyObject_GetAttr(PyStackRef_AsPyObjectBorrow(owner), name);
Revert "GH-128914: Remove conditional stack effects from `bytecodes.c` and the code generators (GH-128918)" (GH-129202) The commit introduced a ~2.5-3% regression in the free threading build. This reverts commit ab61d3f4303d14a413bc9ae6557c730ffdf7579e.
2025-01-23 04:26:25 -05:00
stack_pointer = _PyFrame_GetStackPointer(frame);
GH-128682: Account for escapes in `DECREF_INPUTS` (GH-129953) * Handle escapes in DECREF_INPUTS * Mark a few more functions as escaping * Replace DECREF_INPUTS with PyStackRef_CLOSE where possible
2025-02-12 17:44:59 +00:00
stack_pointer += -1;
GH-141794: Limit size of generated machine code. (GH-142228) * Factor out bodies of the largest uops, to reduce jit code size. * Factor out common assert, also reducing jit code size. * Limit size of jitted code for a single executor to 1MB.
2025-12-03 17:43:35 +00:00
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
GH-128682: Account for escapes in `DECREF_INPUTS` (GH-129953) * Handle escapes in DECREF_INPUTS * Mark a few more functions as escaping * Replace DECREF_INPUTS with PyStackRef_CLOSE where possible
2025-02-12 17:44:59 +00:00
_PyFrame_SetStackPointer(frame, stack_pointer);
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
PyStackRef_CLOSE(owner);
GH-128682: Account for escapes in `DECREF_INPUTS` (GH-129953) * Handle escapes in DECREF_INPUTS * Mark a few more functions as escaping * Replace DECREF_INPUTS with PyStackRef_CLOSE where possible
2025-02-12 17:44:59 +00:00
stack_pointer = _PyFrame_GetStackPointer(frame);
GH-128682: Make `PyStackRef_CLOSE` escaping. (GH-129404)
2025-02-03 12:41:32 +00:00
if (attr_o == NULL) {
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(0);
GH-128682: Make `PyStackRef_CLOSE` escaping. (GH-129404)
2025-02-03 12:41:32 +00:00
JUMP_TO_ERROR();
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
attr = PyStackRef_FromPyObjectSteal(attr_o);
stack_pointer += 1;
}
_tos_cache0 = PyStackRef_ZERO_BITS;
_tos_cache1 = PyStackRef_ZERO_BITS;
_tos_cache2 = PyStackRef_ZERO_BITS;
SET_CURRENT_CACHED_VALUES(0);
stack_pointer[-1] = attr;
stack_pointer += (oparg&1);
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _GUARD_TYPE_VERSION_r01: {
CHECK_CURRENT_CACHED_VALUES(0);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef owner;
owner = stack_pointer[-1];
GH-140683: JIT: Improve machine code for loading smaller constants on AArch64. (GH-142511) * Use movz and movk instructions for loading 16 and 32 bit operands and oparg. * Loading of 64 bit operands is unchanged.
2025-12-11 12:33:39 +00:00
uint32_t type_version = (uint32_t)CURRENT_OPERAND0_32();
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
PyTypeObject *tp = Py_TYPE(PyStackRef_AsPyObjectBorrow(owner));
assert(type_version != 0);
if (FT_ATOMIC_LOAD_UINT_RELAXED(tp->tp_version_tag) != type_version) {
UOP_STAT_INC(uopcode, miss);
SET_CURRENT_CACHED_VALUES(0);
JUMP_TO_JUMP_TARGET();
}
_tos_cache0 = owner;
SET_CURRENT_CACHED_VALUES(1);
stack_pointer += -1;
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _GUARD_TYPE_VERSION_r11: {
CHECK_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef owner;
_PyStackRef _stack_item_0 = _tos_cache0;
owner = _stack_item_0;
GH-140683: JIT: Improve machine code for loading smaller constants on AArch64. (GH-142511) * Use movz and movk instructions for loading 16 and 32 bit operands and oparg. * Loading of 64 bit operands is unchanged.
2025-12-11 12:33:39 +00:00
uint32_t type_version = (uint32_t)CURRENT_OPERAND0_32();
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
PyTypeObject *tp = Py_TYPE(PyStackRef_AsPyObjectBorrow(owner));
assert(type_version != 0);
if (FT_ATOMIC_LOAD_UINT_RELAXED(tp->tp_version_tag) != type_version) {
UOP_STAT_INC(uopcode, miss);
GH-142718: JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. (GH-142762) JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. * Make sure that stack is in correct state at side exits with TOS cached values * Simplify choice of cached items for side exits
2025-12-15 19:01:29 +00:00
_tos_cache0 = owner;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(1);
JUMP_TO_JUMP_TARGET();
}
_tos_cache0 = owner;
SET_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _GUARD_TYPE_VERSION_r22: {
CHECK_CURRENT_CACHED_VALUES(2);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef owner;
_PyStackRef _stack_item_0 = _tos_cache0;
_PyStackRef _stack_item_1 = _tos_cache1;
owner = _stack_item_1;
GH-140683: JIT: Improve machine code for loading smaller constants on AArch64. (GH-142511) * Use movz and movk instructions for loading 16 and 32 bit operands and oparg. * Loading of 64 bit operands is unchanged.
2025-12-11 12:33:39 +00:00
uint32_t type_version = (uint32_t)CURRENT_OPERAND0_32();
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
PyTypeObject *tp = Py_TYPE(PyStackRef_AsPyObjectBorrow(owner));
assert(type_version != 0);
if (FT_ATOMIC_LOAD_UINT_RELAXED(tp->tp_version_tag) != type_version) {
UOP_STAT_INC(uopcode, miss);
GH-142718: JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. (GH-142762) JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. * Make sure that stack is in correct state at side exits with TOS cached values * Simplify choice of cached items for side exits
2025-12-15 19:01:29 +00:00
_tos_cache1 = owner;
_tos_cache0 = _stack_item_0;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(2);
JUMP_TO_JUMP_TARGET();
}
_tos_cache1 = owner;
_tos_cache0 = _stack_item_0;
SET_CURRENT_CACHED_VALUES(2);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _GUARD_TYPE_VERSION_r33: {
CHECK_CURRENT_CACHED_VALUES(3);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef owner;
_PyStackRef _stack_item_0 = _tos_cache0;
_PyStackRef _stack_item_1 = _tos_cache1;
_PyStackRef _stack_item_2 = _tos_cache2;
owner = _stack_item_2;
GH-140683: JIT: Improve machine code for loading smaller constants on AArch64. (GH-142511) * Use movz and movk instructions for loading 16 and 32 bit operands and oparg. * Loading of 64 bit operands is unchanged.
2025-12-11 12:33:39 +00:00
uint32_t type_version = (uint32_t)CURRENT_OPERAND0_32();
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
PyTypeObject *tp = Py_TYPE(PyStackRef_AsPyObjectBorrow(owner));
assert(type_version != 0);
if (FT_ATOMIC_LOAD_UINT_RELAXED(tp->tp_version_tag) != type_version) {
UOP_STAT_INC(uopcode, miss);
GH-142718: JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. (GH-142762) JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. * Make sure that stack is in correct state at side exits with TOS cached values * Simplify choice of cached items for side exits
2025-12-15 19:01:29 +00:00
_tos_cache2 = owner;
_tos_cache1 = _stack_item_1;
_tos_cache0 = _stack_item_0;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(3);
JUMP_TO_JUMP_TARGET();
}
_tos_cache2 = owner;
_tos_cache1 = _stack_item_1;
_tos_cache0 = _stack_item_0;
SET_CURRENT_CACHED_VALUES(3);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _GUARD_TYPE_VERSION_AND_LOCK_r01: {
CHECK_CURRENT_CACHED_VALUES(0);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef owner;
owner = stack_pointer[-1];
GH-140683: JIT: Improve machine code for loading smaller constants on AArch64. (GH-142511) * Use movz and movk instructions for loading 16 and 32 bit operands and oparg. * Loading of 64 bit operands is unchanged.
2025-12-11 12:33:39 +00:00
uint32_t type_version = (uint32_t)CURRENT_OPERAND0_32();
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
PyObject *owner_o = PyStackRef_AsPyObjectBorrow(owner);
assert(type_version != 0);
if (!LOCK_OBJECT(owner_o)) {
UOP_STAT_INC(uopcode, miss);
SET_CURRENT_CACHED_VALUES(0);
JUMP_TO_JUMP_TARGET();
}
PyTypeObject *tp = Py_TYPE(owner_o);
if (FT_ATOMIC_LOAD_UINT_RELAXED(tp->tp_version_tag) != type_version) {
UNLOCK_OBJECT(owner_o);
if (true) {
UOP_STAT_INC(uopcode, miss);
SET_CURRENT_CACHED_VALUES(0);
JUMP_TO_JUMP_TARGET();
}
gh-104584: Allow unspecialized instructions in superblocks (#106497) This adds several of unspecialized opcodes to superblocks: TO_BOOL, BINARY_SUBSCR, STORE_SUBSCR, UNPACK_SEQUENCE, LOAD_GLOBAL, LOAD_ATTR, COMPARE_OP, BINARY_OP. While we may not want that eventually, for now this helps finding bugs. There is a rudimentary test checking for UNPACK_SEQUENCE. Once we're ready to undo this, that would be simple: just replace the call to variable_used_unspecialized with a call to variable_used (as shown in a comment). Or add individual opcdes to FORBIDDEN_NAMES_IN_UOPS.
2023-07-07 11:03:27 -07:00
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_tos_cache0 = owner;
SET_CURRENT_CACHED_VALUES(1);
stack_pointer += -1;
GH-141794: Limit size of generated machine code. (GH-142228) * Factor out bodies of the largest uops, to reduce jit code size. * Factor out common assert, also reducing jit code size. * Limit size of jitted code for a single executor to 1MB.
2025-12-03 17:43:35 +00:00
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-104584: Allow unspecialized instructions in superblocks (#106497) This adds several of unspecialized opcodes to superblocks: TO_BOOL, BINARY_SUBSCR, STORE_SUBSCR, UNPACK_SEQUENCE, LOAD_GLOBAL, LOAD_ATTR, COMPARE_OP, BINARY_OP. While we may not want that eventually, for now this helps finding bugs. There is a rudimentary test checking for UNPACK_SEQUENCE. Once we're ready to undo this, that would be simple: just replace the call to variable_used_unspecialized with a call to variable_used (as shown in a comment). Or add individual opcdes to FORBIDDEN_NAMES_IN_UOPS.
2023-07-07 11:03:27 -07:00
break;
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
case _GUARD_TYPE_VERSION_AND_LOCK_r11: {
CHECK_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
_PyStackRef owner;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef _stack_item_0 = _tos_cache0;
owner = _stack_item_0;
GH-140683: JIT: Improve machine code for loading smaller constants on AArch64. (GH-142511) * Use movz and movk instructions for loading 16 and 32 bit operands and oparg. * Loading of 64 bit operands is unchanged.
2025-12-11 12:33:39 +00:00
uint32_t type_version = (uint32_t)CURRENT_OPERAND0_32();
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
PyObject *owner_o = PyStackRef_AsPyObjectBorrow(owner);
GH-104909: Split `LOAD_ATTR_INSTANCE_VALUE` into micro-ops (GH-106678)
2023-07-13 16:36:19 +01:00
assert(type_version != 0);
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
if (!LOCK_OBJECT(owner_o)) {
UOP_STAT_INC(uopcode, miss);
GH-142718: JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. (GH-142762) JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. * Make sure that stack is in correct state at side exits with TOS cached values * Simplify choice of cached items for side exits
2025-12-15 19:01:29 +00:00
_tos_cache0 = owner;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(1);
JUMP_TO_JUMP_TARGET();
}
PyTypeObject *tp = Py_TYPE(owner_o);
gh-115999: Specialize `STORE_ATTR` in free-threaded builds. (gh-127838) * Add `_PyDictKeys_StringLookupSplit` which does locking on dict keys and use in place of `_PyDictKeys_StringLookup`. * Change `_PyObject_TryGetInstanceAttribute` to use that function in the case of split keys. * Add `unicodekeys_lookup_split` helper which allows code sharing between `_Py_dict_lookup` and `_PyDictKeys_StringLookupSplit`. * Fix locking for `STORE_ATTR_INSTANCE_VALUE`. Create `_GUARD_TYPE_VERSION_AND_LOCK` uop so that object stays locked and `tp_version_tag` cannot change. * Pass `tp_version_tag` to `specialize_dict_access()`, ensuring the version we store on the cache is the correct one (in case of it changing during the specalize analysis). * Split `analyze_descriptor` into `analyze_descriptor_load` and `analyze_descriptor_store` since those don't share much logic. Add `descriptor_is_class` helper function. * In `specialize_dict_access`, double check `_PyObject_GetManagedDict()` in case we race and dict was materialized before the lock. * Avoid borrowed references in `_Py_Specialize_StoreAttr()`. * Use `specialize()` and `unspecialize()` helpers. * Add unit tests to ensure specializing happens as expected in FT builds. * Add unit tests to attempt to trigger data races (useful for running under TSAN). * Add `has_split_table` function to `_testinternalcapi`.
2024-12-19 10:21:17 -08:00
if (FT_ATOMIC_LOAD_UINT_RELAXED(tp->tp_version_tag) != type_version) {
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
UNLOCK_OBJECT(owner_o);
if (true) {
UOP_STAT_INC(uopcode, miss);
GH-142718: JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. (GH-142762) JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. * Make sure that stack is in correct state at side exits with TOS cached values * Simplify choice of cached items for side exits
2025-12-15 19:01:29 +00:00
_tos_cache0 = owner;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(1);
JUMP_TO_JUMP_TARGET();
}
}
_tos_cache0 = owner;
SET_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _GUARD_TYPE_VERSION_AND_LOCK_r22: {
CHECK_CURRENT_CACHED_VALUES(2);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef owner;
_PyStackRef _stack_item_0 = _tos_cache0;
_PyStackRef _stack_item_1 = _tos_cache1;
owner = _stack_item_1;
GH-140683: JIT: Improve machine code for loading smaller constants on AArch64. (GH-142511) * Use movz and movk instructions for loading 16 and 32 bit operands and oparg. * Loading of 64 bit operands is unchanged.
2025-12-11 12:33:39 +00:00
uint32_t type_version = (uint32_t)CURRENT_OPERAND0_32();
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
PyObject *owner_o = PyStackRef_AsPyObjectBorrow(owner);
assert(type_version != 0);
if (!LOCK_OBJECT(owner_o)) {
GH-117457: Correct pystats uop "miss" counts (GH-117477)
2024-04-04 18:49:18 -04:00
UOP_STAT_INC(uopcode, miss);
GH-142718: JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. (GH-142762) JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. * Make sure that stack is in correct state at side exits with TOS cached values * Simplify choice of cached items for side exits
2025-12-15 19:01:29 +00:00
_tos_cache1 = owner;
_tos_cache0 = _stack_item_0;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(2);
GH-117457: Correct pystats uop "miss" counts (GH-117477)
2024-04-04 18:49:18 -04:00
JUMP_TO_JUMP_TARGET();
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
PyTypeObject *tp = Py_TYPE(owner_o);
if (FT_ATOMIC_LOAD_UINT_RELAXED(tp->tp_version_tag) != type_version) {
UNLOCK_OBJECT(owner_o);
if (true) {
UOP_STAT_INC(uopcode, miss);
GH-142718: JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. (GH-142762) JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. * Make sure that stack is in correct state at side exits with TOS cached values * Simplify choice of cached items for side exits
2025-12-15 19:01:29 +00:00
_tos_cache1 = owner;
_tos_cache0 = _stack_item_0;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(2);
JUMP_TO_JUMP_TARGET();
}
}
_tos_cache1 = owner;
_tos_cache0 = _stack_item_0;
SET_CURRENT_CACHED_VALUES(2);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-104909: Split `LOAD_ATTR_INSTANCE_VALUE` into micro-ops (GH-106678)
2023-07-13 16:36:19 +01:00
break;
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
case _GUARD_TYPE_VERSION_AND_LOCK_r33: {
CHECK_CURRENT_CACHED_VALUES(3);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-115999: Specialize `STORE_ATTR` in free-threaded builds. (gh-127838) * Add `_PyDictKeys_StringLookupSplit` which does locking on dict keys and use in place of `_PyDictKeys_StringLookup`. * Change `_PyObject_TryGetInstanceAttribute` to use that function in the case of split keys. * Add `unicodekeys_lookup_split` helper which allows code sharing between `_Py_dict_lookup` and `_PyDictKeys_StringLookupSplit`. * Fix locking for `STORE_ATTR_INSTANCE_VALUE`. Create `_GUARD_TYPE_VERSION_AND_LOCK` uop so that object stays locked and `tp_version_tag` cannot change. * Pass `tp_version_tag` to `specialize_dict_access()`, ensuring the version we store on the cache is the correct one (in case of it changing during the specalize analysis). * Split `analyze_descriptor` into `analyze_descriptor_load` and `analyze_descriptor_store` since those don't share much logic. Add `descriptor_is_class` helper function. * In `specialize_dict_access`, double check `_PyObject_GetManagedDict()` in case we race and dict was materialized before the lock. * Avoid borrowed references in `_Py_Specialize_StoreAttr()`. * Use `specialize()` and `unspecialize()` helpers. * Add unit tests to ensure specializing happens as expected in FT builds. * Add unit tests to attempt to trigger data races (useful for running under TSAN). * Add `has_split_table` function to `_testinternalcapi`.
2024-12-19 10:21:17 -08:00
_PyStackRef owner;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef _stack_item_0 = _tos_cache0;
_PyStackRef _stack_item_1 = _tos_cache1;
_PyStackRef _stack_item_2 = _tos_cache2;
owner = _stack_item_2;
GH-140683: JIT: Improve machine code for loading smaller constants on AArch64. (GH-142511) * Use movz and movk instructions for loading 16 and 32 bit operands and oparg. * Loading of 64 bit operands is unchanged.
2025-12-11 12:33:39 +00:00
uint32_t type_version = (uint32_t)CURRENT_OPERAND0_32();
gh-115999: Specialize `STORE_ATTR` in free-threaded builds. (gh-127838) * Add `_PyDictKeys_StringLookupSplit` which does locking on dict keys and use in place of `_PyDictKeys_StringLookup`. * Change `_PyObject_TryGetInstanceAttribute` to use that function in the case of split keys. * Add `unicodekeys_lookup_split` helper which allows code sharing between `_Py_dict_lookup` and `_PyDictKeys_StringLookupSplit`. * Fix locking for `STORE_ATTR_INSTANCE_VALUE`. Create `_GUARD_TYPE_VERSION_AND_LOCK` uop so that object stays locked and `tp_version_tag` cannot change. * Pass `tp_version_tag` to `specialize_dict_access()`, ensuring the version we store on the cache is the correct one (in case of it changing during the specalize analysis). * Split `analyze_descriptor` into `analyze_descriptor_load` and `analyze_descriptor_store` since those don't share much logic. Add `descriptor_is_class` helper function. * In `specialize_dict_access`, double check `_PyObject_GetManagedDict()` in case we race and dict was materialized before the lock. * Avoid borrowed references in `_Py_Specialize_StoreAttr()`. * Use `specialize()` and `unspecialize()` helpers. * Add unit tests to ensure specializing happens as expected in FT builds. * Add unit tests to attempt to trigger data races (useful for running under TSAN). * Add `has_split_table` function to `_testinternalcapi`.
2024-12-19 10:21:17 -08:00
PyObject *owner_o = PyStackRef_AsPyObjectBorrow(owner);
assert(type_version != 0);
if (!LOCK_OBJECT(owner_o)) {
UOP_STAT_INC(uopcode, miss);
GH-142718: JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. (GH-142762) JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. * Make sure that stack is in correct state at side exits with TOS cached values * Simplify choice of cached items for side exits
2025-12-15 19:01:29 +00:00
_tos_cache2 = owner;
_tos_cache1 = _stack_item_1;
_tos_cache0 = _stack_item_0;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(3);
gh-115999: Specialize `STORE_ATTR` in free-threaded builds. (gh-127838) * Add `_PyDictKeys_StringLookupSplit` which does locking on dict keys and use in place of `_PyDictKeys_StringLookup`. * Change `_PyObject_TryGetInstanceAttribute` to use that function in the case of split keys. * Add `unicodekeys_lookup_split` helper which allows code sharing between `_Py_dict_lookup` and `_PyDictKeys_StringLookupSplit`. * Fix locking for `STORE_ATTR_INSTANCE_VALUE`. Create `_GUARD_TYPE_VERSION_AND_LOCK` uop so that object stays locked and `tp_version_tag` cannot change. * Pass `tp_version_tag` to `specialize_dict_access()`, ensuring the version we store on the cache is the correct one (in case of it changing during the specalize analysis). * Split `analyze_descriptor` into `analyze_descriptor_load` and `analyze_descriptor_store` since those don't share much logic. Add `descriptor_is_class` helper function. * In `specialize_dict_access`, double check `_PyObject_GetManagedDict()` in case we race and dict was materialized before the lock. * Avoid borrowed references in `_Py_Specialize_StoreAttr()`. * Use `specialize()` and `unspecialize()` helpers. * Add unit tests to ensure specializing happens as expected in FT builds. * Add unit tests to attempt to trigger data races (useful for running under TSAN). * Add `has_split_table` function to `_testinternalcapi`.
2024-12-19 10:21:17 -08:00
JUMP_TO_JUMP_TARGET();
}
PyTypeObject *tp = Py_TYPE(owner_o);
if (FT_ATOMIC_LOAD_UINT_RELAXED(tp->tp_version_tag) != type_version) {
UNLOCK_OBJECT(owner_o);
if (true) {
UOP_STAT_INC(uopcode, miss);
GH-142718: JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. (GH-142762) JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. * Make sure that stack is in correct state at side exits with TOS cached values * Simplify choice of cached items for side exits
2025-12-15 19:01:29 +00:00
_tos_cache2 = owner;
_tos_cache1 = _stack_item_1;
_tos_cache0 = _stack_item_0;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(3);
gh-115999: Specialize `STORE_ATTR` in free-threaded builds. (gh-127838) * Add `_PyDictKeys_StringLookupSplit` which does locking on dict keys and use in place of `_PyDictKeys_StringLookup`. * Change `_PyObject_TryGetInstanceAttribute` to use that function in the case of split keys. * Add `unicodekeys_lookup_split` helper which allows code sharing between `_Py_dict_lookup` and `_PyDictKeys_StringLookupSplit`. * Fix locking for `STORE_ATTR_INSTANCE_VALUE`. Create `_GUARD_TYPE_VERSION_AND_LOCK` uop so that object stays locked and `tp_version_tag` cannot change. * Pass `tp_version_tag` to `specialize_dict_access()`, ensuring the version we store on the cache is the correct one (in case of it changing during the specalize analysis). * Split `analyze_descriptor` into `analyze_descriptor_load` and `analyze_descriptor_store` since those don't share much logic. Add `descriptor_is_class` helper function. * In `specialize_dict_access`, double check `_PyObject_GetManagedDict()` in case we race and dict was materialized before the lock. * Avoid borrowed references in `_Py_Specialize_StoreAttr()`. * Use `specialize()` and `unspecialize()` helpers. * Add unit tests to ensure specializing happens as expected in FT builds. * Add unit tests to attempt to trigger data races (useful for running under TSAN). * Add `has_split_table` function to `_testinternalcapi`.
2024-12-19 10:21:17 -08:00
JUMP_TO_JUMP_TARGET();
}
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_tos_cache2 = owner;
_tos_cache1 = _stack_item_1;
_tos_cache0 = _stack_item_0;
SET_CURRENT_CACHED_VALUES(3);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-115999: Specialize `STORE_ATTR` in free-threaded builds. (gh-127838) * Add `_PyDictKeys_StringLookupSplit` which does locking on dict keys and use in place of `_PyDictKeys_StringLookup`. * Change `_PyObject_TryGetInstanceAttribute` to use that function in the case of split keys. * Add `unicodekeys_lookup_split` helper which allows code sharing between `_Py_dict_lookup` and `_PyDictKeys_StringLookupSplit`. * Fix locking for `STORE_ATTR_INSTANCE_VALUE`. Create `_GUARD_TYPE_VERSION_AND_LOCK` uop so that object stays locked and `tp_version_tag` cannot change. * Pass `tp_version_tag` to `specialize_dict_access()`, ensuring the version we store on the cache is the correct one (in case of it changing during the specalize analysis). * Split `analyze_descriptor` into `analyze_descriptor_load` and `analyze_descriptor_store` since those don't share much logic. Add `descriptor_is_class` helper function. * In `specialize_dict_access`, double check `_PyObject_GetManagedDict()` in case we race and dict was materialized before the lock. * Avoid borrowed references in `_Py_Specialize_StoreAttr()`. * Use `specialize()` and `unspecialize()` helpers. * Add unit tests to ensure specializing happens as expected in FT builds. * Add unit tests to attempt to trigger data races (useful for running under TSAN). * Add `has_split_table` function to `_testinternalcapi`.
2024-12-19 10:21:17 -08:00
break;
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
case _CHECK_MANAGED_OBJECT_HAS_VALUES_r01: {
CHECK_CURRENT_CACHED_VALUES(0);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
_PyStackRef owner;
gh-106812: Refactor cases_generator to allow uops with array stack effects (#107564) Introducing a new file, stacking.py, that takes over several responsibilities related to symbolic evaluation of push/pop operations, with more generality.
2023-08-04 09:35:56 -07:00
owner = stack_pointer[-1];
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
PyObject *owner_o = PyStackRef_AsPyObjectBorrow(owner);
assert(Py_TYPE(owner_o)->tp_dictoffset < 0);
assert(Py_TYPE(owner_o)->tp_flags & Py_TPFLAGS_INLINE_VALUES);
gh-115999: Specialize `LOAD_ATTR` for instance and class receivers in free-threaded builds (#128164) Finish specialization for LOAD_ATTR in the free-threaded build by adding support for class and instance receivers.
2025-01-14 11:56:11 -08:00
if (!FT_ATOMIC_LOAD_UINT8(_PyObject_InlineValues(owner_o)->valid)) {
GH-117457: Correct pystats uop "miss" counts (GH-117477)
2024-04-04 18:49:18 -04:00
UOP_STAT_INC(uopcode, miss);
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(0);
JUMP_TO_JUMP_TARGET();
}
_tos_cache0 = owner;
SET_CURRENT_CACHED_VALUES(1);
stack_pointer += -1;
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _CHECK_MANAGED_OBJECT_HAS_VALUES_r11: {
CHECK_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef owner;
_PyStackRef _stack_item_0 = _tos_cache0;
owner = _stack_item_0;
PyObject *owner_o = PyStackRef_AsPyObjectBorrow(owner);
assert(Py_TYPE(owner_o)->tp_dictoffset < 0);
assert(Py_TYPE(owner_o)->tp_flags & Py_TPFLAGS_INLINE_VALUES);
if (!FT_ATOMIC_LOAD_UINT8(_PyObject_InlineValues(owner_o)->valid)) {
UOP_STAT_INC(uopcode, miss);
GH-142718: JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. (GH-142762) JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. * Make sure that stack is in correct state at side exits with TOS cached values * Simplify choice of cached items for side exits
2025-12-15 19:01:29 +00:00
_tos_cache0 = owner;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(1);
JUMP_TO_JUMP_TARGET();
}
_tos_cache0 = owner;
SET_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _CHECK_MANAGED_OBJECT_HAS_VALUES_r22: {
CHECK_CURRENT_CACHED_VALUES(2);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef owner;
_PyStackRef _stack_item_0 = _tos_cache0;
_PyStackRef _stack_item_1 = _tos_cache1;
owner = _stack_item_1;
PyObject *owner_o = PyStackRef_AsPyObjectBorrow(owner);
assert(Py_TYPE(owner_o)->tp_dictoffset < 0);
assert(Py_TYPE(owner_o)->tp_flags & Py_TPFLAGS_INLINE_VALUES);
if (!FT_ATOMIC_LOAD_UINT8(_PyObject_InlineValues(owner_o)->valid)) {
UOP_STAT_INC(uopcode, miss);
GH-142718: JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. (GH-142762) JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. * Make sure that stack is in correct state at side exits with TOS cached values * Simplify choice of cached items for side exits
2025-12-15 19:01:29 +00:00
_tos_cache1 = owner;
_tos_cache0 = _stack_item_0;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(2);
JUMP_TO_JUMP_TARGET();
}
_tos_cache1 = owner;
_tos_cache0 = _stack_item_0;
SET_CURRENT_CACHED_VALUES(2);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _CHECK_MANAGED_OBJECT_HAS_VALUES_r33: {
CHECK_CURRENT_CACHED_VALUES(3);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef owner;
_PyStackRef _stack_item_0 = _tos_cache0;
_PyStackRef _stack_item_1 = _tos_cache1;
_PyStackRef _stack_item_2 = _tos_cache2;
owner = _stack_item_2;
PyObject *owner_o = PyStackRef_AsPyObjectBorrow(owner);
assert(Py_TYPE(owner_o)->tp_dictoffset < 0);
assert(Py_TYPE(owner_o)->tp_flags & Py_TPFLAGS_INLINE_VALUES);
if (!FT_ATOMIC_LOAD_UINT8(_PyObject_InlineValues(owner_o)->valid)) {
UOP_STAT_INC(uopcode, miss);
GH-142718: JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. (GH-142762) JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. * Make sure that stack is in correct state at side exits with TOS cached values * Simplify choice of cached items for side exits
2025-12-15 19:01:29 +00:00
_tos_cache2 = owner;
_tos_cache1 = _stack_item_1;
_tos_cache0 = _stack_item_0;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(3);
GH-117457: Correct pystats uop "miss" counts (GH-117477)
2024-04-04 18:49:18 -04:00
JUMP_TO_JUMP_TARGET();
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_tos_cache2 = owner;
_tos_cache1 = _stack_item_1;
_tos_cache0 = _stack_item_0;
SET_CURRENT_CACHED_VALUES(3);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-104909: Split `LOAD_ATTR_INSTANCE_VALUE` into micro-ops (GH-106678)
2023-07-13 16:36:19 +01:00
break;
}
gh-134584: Eliminate redundant refcounting from `_LOAD_ATTR_INSTANCE_VALUE` (GH-142769) Signed-off-by: Manjusaka <me@manjusaka.me>
2025-12-17 05:39:20 +09:00
case _LOAD_ATTR_INSTANCE_VALUE_r02: {
CHECK_CURRENT_CACHED_VALUES(0);
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
_PyStackRef owner;
_PyStackRef attr;
_PyStackRef o;
owner = stack_pointer[-1];
uint16_t offset = (uint16_t)CURRENT_OPERAND0_16();
PyObject *owner_o = PyStackRef_AsPyObjectBorrow(owner);
PyObject **value_ptr = (PyObject**)(((char *)owner_o) + offset);
PyObject *attr_o = FT_ATOMIC_LOAD_PTR_ACQUIRE(*value_ptr);
if (attr_o == NULL) {
UOP_STAT_INC(uopcode, miss);
SET_CURRENT_CACHED_VALUES(0);
JUMP_TO_JUMP_TARGET();
}
#ifdef Py_GIL_DISABLED
int increfed = _Py_TryIncrefCompareStackRef(value_ptr, attr_o, &attr);
if (!increfed) {
if (true) {
UOP_STAT_INC(uopcode, miss);
SET_CURRENT_CACHED_VALUES(0);
JUMP_TO_JUMP_TARGET();
}
}
#else
attr = PyStackRef_FromPyObjectNew(attr_o);
#endif
STAT_INC(LOAD_ATTR, hit);
o = owner;
_tos_cache1 = o;
_tos_cache0 = attr;
SET_CURRENT_CACHED_VALUES(2);
stack_pointer += -1;
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
break;
}
case _LOAD_ATTR_INSTANCE_VALUE_r12: {
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
CHECK_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
_PyStackRef owner;
_PyStackRef attr;
gh-134584: Eliminate redundant refcounting from `_LOAD_ATTR_INSTANCE_VALUE` (GH-142769) Signed-off-by: Manjusaka <me@manjusaka.me>
2025-12-17 05:39:20 +09:00
_PyStackRef o;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef _stack_item_0 = _tos_cache0;
owner = _stack_item_0;
GH-140683: JIT: Improve machine code for loading smaller constants on AArch64. (GH-142511) * Use movz and movk instructions for loading 16 and 32 bit operands and oparg. * Loading of 64 bit operands is unchanged.
2025-12-11 12:33:39 +00:00
uint16_t offset = (uint16_t)CURRENT_OPERAND0_16();
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
PyObject *owner_o = PyStackRef_AsPyObjectBorrow(owner);
GH-115776: Allow any fixed sized object to have inline values (GH-123192)
2024-08-21 15:52:04 +01:00
PyObject **value_ptr = (PyObject**)(((char *)owner_o) + offset);
gh-115999: Specialize `LOAD_ATTR` for instance and class receivers in free-threaded builds (#128164) Finish specialization for LOAD_ATTR in the free-threaded build by adding support for class and instance receivers.
2025-01-14 11:56:11 -08:00
PyObject *attr_o = FT_ATOMIC_LOAD_PTR_ACQUIRE(*value_ptr);
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
if (attr_o == NULL) {
GH-117457: Correct pystats uop "miss" counts (GH-117477)
2024-04-04 18:49:18 -04:00
UOP_STAT_INC(uopcode, miss);
GH-142718: JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. (GH-142762) JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. * Make sure that stack is in correct state at side exits with TOS cached values * Simplify choice of cached items for side exits
2025-12-15 19:01:29 +00:00
_tos_cache0 = owner;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(1);
GH-117457: Correct pystats uop "miss" counts (GH-117477)
2024-04-04 18:49:18 -04:00
JUMP_TO_JUMP_TARGET();
}
gh-115999: Specialize `LOAD_ATTR` for instance and class receivers in free-threaded builds (#128164) Finish specialization for LOAD_ATTR in the free-threaded build by adding support for class and instance receivers.
2025-01-14 11:56:11 -08:00
#ifdef Py_GIL_DISABLED
GH-131498: Cases generator: Parse down to C statement level. (GH-131948) * Parse down to statement level in the cases generator * Add handling for #if macros, treating them much like normal ifs.
2025-04-02 16:31:59 +01:00
int increfed = _Py_TryIncrefCompareStackRef(value_ptr, attr_o, &attr);
if (!increfed) {
gh-115999: Specialize `LOAD_ATTR` for instance and class receivers in free-threaded builds (#128164) Finish specialization for LOAD_ATTR in the free-threaded build by adding support for class and instance receivers.
2025-01-14 11:56:11 -08:00
if (true) {
UOP_STAT_INC(uopcode, miss);
GH-142718: JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. (GH-142762) JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. * Make sure that stack is in correct state at side exits with TOS cached values * Simplify choice of cached items for side exits
2025-12-15 19:01:29 +00:00
_tos_cache0 = owner;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(1);
gh-115999: Specialize `LOAD_ATTR` for instance and class receivers in free-threaded builds (#128164) Finish specialization for LOAD_ATTR in the free-threaded build by adding support for class and instance receivers.
2025-01-14 11:56:11 -08:00
JUMP_TO_JUMP_TARGET();
}
}
#else
attr = PyStackRef_FromPyObjectNew(attr_o);
#endif
GH-115457: Support splitting and replication of micro ops. (GH-115558)
2024-02-20 10:50:59 +00:00
STAT_INC(LOAD_ATTR, hit);
gh-134584: Eliminate redundant refcounting from `_LOAD_ATTR_INSTANCE_VALUE` (GH-142769) Signed-off-by: Manjusaka <me@manjusaka.me>
2025-12-17 05:39:20 +09:00
o = owner;
_tos_cache1 = o;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_tos_cache0 = attr;
gh-134584: Eliminate redundant refcounting from `_LOAD_ATTR_INSTANCE_VALUE` (GH-142769) Signed-off-by: Manjusaka <me@manjusaka.me>
2025-12-17 05:39:20 +09:00
SET_CURRENT_CACHED_VALUES(2);
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
break;
}
case _LOAD_ATTR_INSTANCE_VALUE_r23: {
CHECK_CURRENT_CACHED_VALUES(2);
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
_PyStackRef owner;
_PyStackRef attr;
_PyStackRef o;
_PyStackRef _stack_item_0 = _tos_cache0;
_PyStackRef _stack_item_1 = _tos_cache1;
owner = _stack_item_1;
uint16_t offset = (uint16_t)CURRENT_OPERAND0_16();
PyObject *owner_o = PyStackRef_AsPyObjectBorrow(owner);
PyObject **value_ptr = (PyObject**)(((char *)owner_o) + offset);
PyObject *attr_o = FT_ATOMIC_LOAD_PTR_ACQUIRE(*value_ptr);
if (attr_o == NULL) {
UOP_STAT_INC(uopcode, miss);
_tos_cache1 = owner;
_tos_cache0 = _stack_item_0;
SET_CURRENT_CACHED_VALUES(2);
JUMP_TO_JUMP_TARGET();
}
#ifdef Py_GIL_DISABLED
int increfed = _Py_TryIncrefCompareStackRef(value_ptr, attr_o, &attr);
if (!increfed) {
if (true) {
UOP_STAT_INC(uopcode, miss);
_tos_cache1 = owner;
_tos_cache0 = _stack_item_0;
SET_CURRENT_CACHED_VALUES(2);
JUMP_TO_JUMP_TARGET();
}
}
#else
attr = PyStackRef_FromPyObjectNew(attr_o);
#endif
STAT_INC(LOAD_ATTR, hit);
o = owner;
_tos_cache2 = o;
_tos_cache1 = attr;
_tos_cache0 = _stack_item_0;
SET_CURRENT_CACHED_VALUES(3);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-115457: Support splitting and replication of micro ops. (GH-115558)
2024-02-20 10:50:59 +00:00
break;
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
case _LOAD_ATTR_MODULE_r11: {
CHECK_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
_PyStackRef owner;
GH-130296: Avoid stack transients in four instructions. (GH-130310) * Combine _GUARD_GLOBALS_VERSION_PUSH_KEYS and _LOAD_GLOBAL_MODULE_FROM_KEYS into _LOAD_GLOBAL_MODULE * Combine _GUARD_BUILTINS_VERSION_PUSH_KEYS and _LOAD_GLOBAL_BUILTINS_FROM_KEYS into _LOAD_GLOBAL_BUILTINS * Combine _CHECK_ATTR_MODULE_PUSH_KEYS and _LOAD_ATTR_MODULE_FROM_KEYS into _LOAD_ATTR_MODULE * Remove stack transient in LOAD_ATTR_WITH_HINT
2025-02-28 18:00:38 +00:00
_PyStackRef attr;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef _stack_item_0 = _tos_cache0;
owner = _stack_item_0;
GH-140683: JIT: Improve machine code for loading smaller constants on AArch64. (GH-142511) * Use movz and movk instructions for loading 16 and 32 bit operands and oparg. * Loading of 64 bit operands is unchanged.
2025-12-11 12:33:39 +00:00
uint32_t dict_version = (uint32_t)CURRENT_OPERAND0_32();
uint16_t index = (uint16_t)CURRENT_OPERAND1_16();
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
PyObject *owner_o = PyStackRef_AsPyObjectBorrow(owner);
gh-103951: enable optimization for fast attribute access on module subclasses (GH-126264) Co-authored-by: Nicolas Tessore <n.tessore@ucl.ac.uk>
2024-11-15 11:03:38 +03:00
if (Py_TYPE(owner_o)->tp_getattro != PyModule_Type.tp_getattro) {
GH-117457: Correct pystats uop "miss" counts (GH-117477)
2024-04-04 18:49:18 -04:00
UOP_STAT_INC(uopcode, miss);
GH-142718: JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. (GH-142762) JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. * Make sure that stack is in correct state at side exits with TOS cached values * Simplify choice of cached items for side exits
2025-12-15 19:01:29 +00:00
_tos_cache0 = owner;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(1);
GH-117457: Correct pystats uop "miss" counts (GH-117477)
2024-04-04 18:49:18 -04:00
JUMP_TO_JUMP_TARGET();
}
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
PyDictObject *dict = (PyDictObject *)((PyModuleObject *)owner_o)->md_dict;
gh-104909: Split more LOAD_ATTR specializations (GH-110317) * Split LOAD_ATTR_MODULE * Split LOAD_ATTR_WITH_HINT * Split _GUARD_TYPE_VERSION out of the latter * Split LOAD_ATTR_CLASS * Split LOAD_ATTR_NONDESCRIPTOR_WITH_VALUES * Fix indent of DEOPT_IF in macros * Split LOAD_ATTR_METHOD_LAZY_DICT * Split LOAD_ATTR_NONDESCRIPTOR_NO_DICT * Fix omission of _CHECK_ATTR_METHOD_LAZY_DICT
2023-10-04 08:08:02 -07:00
assert(dict != NULL);
gh-115999: Specialize loading attributes from modules in free-threaded builds (#127711) We use the same approach that was used for specialization of LOAD_GLOBAL in free-threaded builds: _CHECK_ATTR_MODULE is renamed to _CHECK_ATTR_MODULE_PUSH_KEYS; it pushes the keys object for the following _LOAD_ATTR_MODULE_FROM_KEYS (nee _LOAD_ATTR_MODULE). This arrangement avoids having to recheck the keys version. _LOAD_ATTR_MODULE is renamed to _LOAD_ATTR_MODULE_FROM_KEYS; it loads the value from the keys object pushed by the preceding _CHECK_ATTR_MODULE_PUSH_KEYS at the cached index.
2024-12-13 10:17:16 -08:00
PyDictKeysObject *keys = FT_ATOMIC_LOAD_PTR_ACQUIRE(dict->ma_keys);
if (FT_ATOMIC_LOAD_UINT32_RELAXED(keys->dk_version) != dict_version) {
GH-117457: Correct pystats uop "miss" counts (GH-117477)
2024-04-04 18:49:18 -04:00
UOP_STAT_INC(uopcode, miss);
GH-142718: JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. (GH-142762) JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. * Make sure that stack is in correct state at side exits with TOS cached values * Simplify choice of cached items for side exits
2025-12-15 19:01:29 +00:00
_tos_cache0 = owner;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(1);
GH-117457: Correct pystats uop "miss" counts (GH-117477)
2024-04-04 18:49:18 -04:00
JUMP_TO_JUMP_TARGET();
}
GH-130296: Avoid stack transients in four instructions. (GH-130310) * Combine _GUARD_GLOBALS_VERSION_PUSH_KEYS and _LOAD_GLOBAL_MODULE_FROM_KEYS into _LOAD_GLOBAL_MODULE * Combine _GUARD_BUILTINS_VERSION_PUSH_KEYS and _LOAD_GLOBAL_BUILTINS_FROM_KEYS into _LOAD_GLOBAL_BUILTINS * Combine _CHECK_ATTR_MODULE_PUSH_KEYS and _LOAD_ATTR_MODULE_FROM_KEYS into _LOAD_ATTR_MODULE * Remove stack transient in LOAD_ATTR_WITH_HINT
2025-02-28 18:00:38 +00:00
assert(keys->dk_kind == DICT_KEYS_UNICODE);
assert(index < FT_ATOMIC_LOAD_SSIZE_RELAXED(keys->dk_nentries));
PyDictUnicodeEntry *ep = DK_UNICODE_ENTRIES(keys) + index;
gh-115999: Specialize loading attributes from modules in free-threaded builds (#127711) We use the same approach that was used for specialization of LOAD_GLOBAL in free-threaded builds: _CHECK_ATTR_MODULE is renamed to _CHECK_ATTR_MODULE_PUSH_KEYS; it pushes the keys object for the following _LOAD_ATTR_MODULE_FROM_KEYS (nee _LOAD_ATTR_MODULE). This arrangement avoids having to recheck the keys version. _LOAD_ATTR_MODULE is renamed to _LOAD_ATTR_MODULE_FROM_KEYS; it loads the value from the keys object pushed by the preceding _CHECK_ATTR_MODULE_PUSH_KEYS at the cached index.
2024-12-13 10:17:16 -08:00
PyObject *attr_o = FT_ATOMIC_LOAD_PTR_RELAXED(ep->me_value);
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
if (attr_o == NULL) {
GH-117457: Correct pystats uop "miss" counts (GH-117477)
2024-04-04 18:49:18 -04:00
UOP_STAT_INC(uopcode, miss);
GH-142718: JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. (GH-142762) JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. * Make sure that stack is in correct state at side exits with TOS cached values * Simplify choice of cached items for side exits
2025-12-15 19:01:29 +00:00
_tos_cache0 = owner;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(1);
GH-117457: Correct pystats uop "miss" counts (GH-117477)
2024-04-04 18:49:18 -04:00
JUMP_TO_JUMP_TARGET();
}
gh-115999: Specialize loading attributes from modules in free-threaded builds (#127711) We use the same approach that was used for specialization of LOAD_GLOBAL in free-threaded builds: _CHECK_ATTR_MODULE is renamed to _CHECK_ATTR_MODULE_PUSH_KEYS; it pushes the keys object for the following _LOAD_ATTR_MODULE_FROM_KEYS (nee _LOAD_ATTR_MODULE). This arrangement avoids having to recheck the keys version. _LOAD_ATTR_MODULE is renamed to _LOAD_ATTR_MODULE_FROM_KEYS; it loads the value from the keys object pushed by the preceding _CHECK_ATTR_MODULE_PUSH_KEYS at the cached index.
2024-12-13 10:17:16 -08:00
#ifdef Py_GIL_DISABLED
int increfed = _Py_TryIncrefCompareStackRef(&ep->me_value, attr_o, &attr);
if (!increfed) {
if (true) {
UOP_STAT_INC(uopcode, miss);
GH-142718: JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. (GH-142762) JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. * Make sure that stack is in correct state at side exits with TOS cached values * Simplify choice of cached items for side exits
2025-12-15 19:01:29 +00:00
_tos_cache0 = owner;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(1);
gh-115999: Specialize loading attributes from modules in free-threaded builds (#127711) We use the same approach that was used for specialization of LOAD_GLOBAL in free-threaded builds: _CHECK_ATTR_MODULE is renamed to _CHECK_ATTR_MODULE_PUSH_KEYS; it pushes the keys object for the following _LOAD_ATTR_MODULE_FROM_KEYS (nee _LOAD_ATTR_MODULE). This arrangement avoids having to recheck the keys version. _LOAD_ATTR_MODULE is renamed to _LOAD_ATTR_MODULE_FROM_KEYS; it loads the value from the keys object pushed by the preceding _CHECK_ATTR_MODULE_PUSH_KEYS at the cached index.
2024-12-13 10:17:16 -08:00
JUMP_TO_JUMP_TARGET();
}
}
#else
GH-130296: Avoid stack transients in four instructions. (GH-130310) * Combine _GUARD_GLOBALS_VERSION_PUSH_KEYS and _LOAD_GLOBAL_MODULE_FROM_KEYS into _LOAD_GLOBAL_MODULE * Combine _GUARD_BUILTINS_VERSION_PUSH_KEYS and _LOAD_GLOBAL_BUILTINS_FROM_KEYS into _LOAD_GLOBAL_BUILTINS * Combine _CHECK_ATTR_MODULE_PUSH_KEYS and _LOAD_ATTR_MODULE_FROM_KEYS into _LOAD_ATTR_MODULE * Remove stack transient in LOAD_ATTR_WITH_HINT
2025-02-28 18:00:38 +00:00
attr = PyStackRef_FromPyObjectNew(attr_o);
gh-115999: Specialize loading attributes from modules in free-threaded builds (#127711) We use the same approach that was used for specialization of LOAD_GLOBAL in free-threaded builds: _CHECK_ATTR_MODULE is renamed to _CHECK_ATTR_MODULE_PUSH_KEYS; it pushes the keys object for the following _LOAD_ATTR_MODULE_FROM_KEYS (nee _LOAD_ATTR_MODULE). This arrangement avoids having to recheck the keys version. _LOAD_ATTR_MODULE is renamed to _LOAD_ATTR_MODULE_FROM_KEYS; it loads the value from the keys object pushed by the preceding _CHECK_ATTR_MODULE_PUSH_KEYS at the cached index.
2024-12-13 10:17:16 -08:00
#endif
STAT_INC(LOAD_ATTR, hit);
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
stack_pointer[0] = attr;
stack_pointer += 1;
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
GH-128682: Make `PyStackRef_CLOSE` escaping. (GH-129404)
2025-02-03 12:41:32 +00:00
_PyFrame_SetStackPointer(frame, stack_pointer);
PyStackRef_CLOSE(owner);
stack_pointer = _PyFrame_GetStackPointer(frame);
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_tos_cache0 = attr;
_tos_cache1 = PyStackRef_ZERO_BITS;
_tos_cache2 = PyStackRef_ZERO_BITS;
SET_CURRENT_CACHED_VALUES(1);
stack_pointer += -1;
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-104909: Split more LOAD_ATTR specializations (GH-110317) * Split LOAD_ATTR_MODULE * Split LOAD_ATTR_WITH_HINT * Split _GUARD_TYPE_VERSION out of the latter * Split LOAD_ATTR_CLASS * Split LOAD_ATTR_NONDESCRIPTOR_WITH_VALUES * Fix indent of DEOPT_IF in macros * Split LOAD_ATTR_METHOD_LAZY_DICT * Split LOAD_ATTR_NONDESCRIPTOR_NO_DICT * Fix omission of _CHECK_ATTR_METHOD_LAZY_DICT
2023-10-04 08:08:02 -07:00
break;
}
gh-134584: Eliminate redundant refcounting from `_LOAD_ATTR_WITH_HINT` (GH-143062) Eliminate redundant refcounting from _LOAD_ATTR_WITH_HINT
2025-12-23 08:28:08 +08:00
case _LOAD_ATTR_WITH_HINT_r12: {
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
CHECK_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
_PyStackRef owner;
GH-130296: Avoid stack transients in four instructions. (GH-130310) * Combine _GUARD_GLOBALS_VERSION_PUSH_KEYS and _LOAD_GLOBAL_MODULE_FROM_KEYS into _LOAD_GLOBAL_MODULE * Combine _GUARD_BUILTINS_VERSION_PUSH_KEYS and _LOAD_GLOBAL_BUILTINS_FROM_KEYS into _LOAD_GLOBAL_BUILTINS * Combine _CHECK_ATTR_MODULE_PUSH_KEYS and _LOAD_ATTR_MODULE_FROM_KEYS into _LOAD_ATTR_MODULE * Remove stack transient in LOAD_ATTR_WITH_HINT
2025-02-28 18:00:38 +00:00
_PyStackRef attr;
gh-134584: Eliminate redundant refcounting from `_LOAD_ATTR_WITH_HINT` (GH-143062) Eliminate redundant refcounting from _LOAD_ATTR_WITH_HINT
2025-12-23 08:28:08 +08:00
_PyStackRef o;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef _stack_item_0 = _tos_cache0;
GH-130296: Avoid stack transients in four instructions. (GH-130310) * Combine _GUARD_GLOBALS_VERSION_PUSH_KEYS and _LOAD_GLOBAL_MODULE_FROM_KEYS into _LOAD_GLOBAL_MODULE * Combine _GUARD_BUILTINS_VERSION_PUSH_KEYS and _LOAD_GLOBAL_BUILTINS_FROM_KEYS into _LOAD_GLOBAL_BUILTINS * Combine _CHECK_ATTR_MODULE_PUSH_KEYS and _LOAD_ATTR_MODULE_FROM_KEYS into _LOAD_ATTR_MODULE * Remove stack transient in LOAD_ATTR_WITH_HINT
2025-02-28 18:00:38 +00:00
oparg = CURRENT_OPARG();
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
owner = _stack_item_0;
GH-140683: JIT: Improve machine code for loading smaller constants on AArch64. (GH-142511) * Use movz and movk instructions for loading 16 and 32 bit operands and oparg. * Loading of 64 bit operands is unchanged.
2025-12-11 12:33:39 +00:00
uint16_t hint = (uint16_t)CURRENT_OPERAND0_16();
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
PyObject *owner_o = PyStackRef_AsPyObjectBorrow(owner);
assert(Py_TYPE(owner_o)->tp_flags & Py_TPFLAGS_MANAGED_DICT);
GH-130296: Avoid stack transients in four instructions. (GH-130310) * Combine _GUARD_GLOBALS_VERSION_PUSH_KEYS and _LOAD_GLOBAL_MODULE_FROM_KEYS into _LOAD_GLOBAL_MODULE * Combine _GUARD_BUILTINS_VERSION_PUSH_KEYS and _LOAD_GLOBAL_BUILTINS_FROM_KEYS into _LOAD_GLOBAL_BUILTINS * Combine _CHECK_ATTR_MODULE_PUSH_KEYS and _LOAD_ATTR_MODULE_FROM_KEYS into _LOAD_ATTR_MODULE * Remove stack transient in LOAD_ATTR_WITH_HINT
2025-02-28 18:00:38 +00:00
PyDictObject *dict = _PyObject_GetManagedDict(owner_o);
if (dict == NULL) {
GH-117457: Correct pystats uop "miss" counts (GH-117477)
2024-04-04 18:49:18 -04:00
UOP_STAT_INC(uopcode, miss);
GH-142718: JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. (GH-142762) JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. * Make sure that stack is in correct state at side exits with TOS cached values * Simplify choice of cached items for side exits
2025-12-15 19:01:29 +00:00
_tos_cache0 = owner;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(1);
GH-117457: Correct pystats uop "miss" counts (GH-117477)
2024-04-04 18:49:18 -04:00
JUMP_TO_JUMP_TARGET();
}
gh-130373: Avoid locking in _LOAD_ATTR_WITH_HINT (#130372) Avoid locking in _LOAD_ATTR_WITH_HINT
2025-03-28 15:16:41 -07:00
PyDictKeysObject *dk = FT_ATOMIC_LOAD_PTR(dict->ma_keys);
GH-130296: Avoid stack transients in four instructions. (GH-130310) * Combine _GUARD_GLOBALS_VERSION_PUSH_KEYS and _LOAD_GLOBAL_MODULE_FROM_KEYS into _LOAD_GLOBAL_MODULE * Combine _GUARD_BUILTINS_VERSION_PUSH_KEYS and _LOAD_GLOBAL_BUILTINS_FROM_KEYS into _LOAD_GLOBAL_BUILTINS * Combine _CHECK_ATTR_MODULE_PUSH_KEYS and _LOAD_ATTR_MODULE_FROM_KEYS into _LOAD_ATTR_MODULE * Remove stack transient in LOAD_ATTR_WITH_HINT
2025-02-28 18:00:38 +00:00
assert(PyDict_CheckExact((PyObject *)dict));
gh-130373: Avoid locking in _LOAD_ATTR_WITH_HINT (#130372) Avoid locking in _LOAD_ATTR_WITH_HINT
2025-03-28 15:16:41 -07:00
#ifdef Py_GIL_DISABLED
if (!_Py_IsOwnedByCurrentThread((PyObject *)dict) && !_PyObject_GC_IS_SHARED(dict)) {
UOP_STAT_INC(uopcode, miss);
GH-142718: JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. (GH-142762) JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. * Make sure that stack is in correct state at side exits with TOS cached values * Simplify choice of cached items for side exits
2025-12-15 19:01:29 +00:00
_tos_cache0 = owner;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(1);
gh-130373: Avoid locking in _LOAD_ATTR_WITH_HINT (#130372) Avoid locking in _LOAD_ATTR_WITH_HINT
2025-03-28 15:16:41 -07:00
JUMP_TO_JUMP_TARGET();
}
#endif
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
PyObject *attr_o;
gh-130373: Avoid locking in _LOAD_ATTR_WITH_HINT (#130372) Avoid locking in _LOAD_ATTR_WITH_HINT
2025-03-28 15:16:41 -07:00
if (hint >= (size_t)FT_ATOMIC_LOAD_SSIZE_RELAXED(dk->dk_nentries)) {
gh-115999: Specialize `LOAD_ATTR` for instance and class receivers in free-threaded builds (#128164) Finish specialization for LOAD_ATTR in the free-threaded build by adding support for class and instance receivers.
2025-01-14 11:56:11 -08:00
if (true) {
UOP_STAT_INC(uopcode, miss);
GH-142718: JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. (GH-142762) JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. * Make sure that stack is in correct state at side exits with TOS cached values * Simplify choice of cached items for side exits
2025-12-15 19:01:29 +00:00
_tos_cache0 = owner;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(1);
gh-115999: Specialize `LOAD_ATTR` for instance and class receivers in free-threaded builds (#128164) Finish specialization for LOAD_ATTR in the free-threaded build by adding support for class and instance receivers.
2025-01-14 11:56:11 -08:00
JUMP_TO_JUMP_TARGET();
}
}
gh-130373: Avoid locking in _LOAD_ATTR_WITH_HINT (#130372) Avoid locking in _LOAD_ATTR_WITH_HINT
2025-03-28 15:16:41 -07:00
PyObject *name = GETITEM(FRAME_CO_NAMES, oparg>>1);
if (dk->dk_kind != DICT_KEYS_UNICODE) {
gh-115999: Specialize `LOAD_ATTR` for instance and class receivers in free-threaded builds (#128164) Finish specialization for LOAD_ATTR in the free-threaded build by adding support for class and instance receivers.
2025-01-14 11:56:11 -08:00
if (true) {
UOP_STAT_INC(uopcode, miss);
GH-142718: JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. (GH-142762) JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. * Make sure that stack is in correct state at side exits with TOS cached values * Simplify choice of cached items for side exits
2025-12-15 19:01:29 +00:00
_tos_cache0 = owner;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(1);
gh-115999: Specialize `LOAD_ATTR` for instance and class receivers in free-threaded builds (#128164) Finish specialization for LOAD_ATTR in the free-threaded build by adding support for class and instance receivers.
2025-01-14 11:56:11 -08:00
JUMP_TO_JUMP_TARGET();
}
GH-117457: Correct pystats uop "miss" counts (GH-117477)
2024-04-04 18:49:18 -04:00
}
gh-130373: Avoid locking in _LOAD_ATTR_WITH_HINT (#130372) Avoid locking in _LOAD_ATTR_WITH_HINT
2025-03-28 15:16:41 -07:00
PyDictUnicodeEntry *ep = DK_UNICODE_ENTRIES(dk) + hint;
if (FT_ATOMIC_LOAD_PTR_RELAXED(ep->me_key) != name) {
gh-115999: Specialize `LOAD_ATTR` for instance and class receivers in free-threaded builds (#128164) Finish specialization for LOAD_ATTR in the free-threaded build by adding support for class and instance receivers.
2025-01-14 11:56:11 -08:00
if (true) {
UOP_STAT_INC(uopcode, miss);
GH-142718: JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. (GH-142762) JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. * Make sure that stack is in correct state at side exits with TOS cached values * Simplify choice of cached items for side exits
2025-12-15 19:01:29 +00:00
_tos_cache0 = owner;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(1);
gh-115999: Specialize `LOAD_ATTR` for instance and class receivers in free-threaded builds (#128164) Finish specialization for LOAD_ATTR in the free-threaded build by adding support for class and instance receivers.
2025-01-14 11:56:11 -08:00
JUMP_TO_JUMP_TARGET();
}
gh-104909: Split more LOAD_ATTR specializations (GH-110317) * Split LOAD_ATTR_MODULE * Split LOAD_ATTR_WITH_HINT * Split _GUARD_TYPE_VERSION out of the latter * Split LOAD_ATTR_CLASS * Split LOAD_ATTR_NONDESCRIPTOR_WITH_VALUES * Fix indent of DEOPT_IF in macros * Split LOAD_ATTR_METHOD_LAZY_DICT * Split LOAD_ATTR_NONDESCRIPTOR_NO_DICT * Fix omission of _CHECK_ATTR_METHOD_LAZY_DICT
2023-10-04 08:08:02 -07:00
}
gh-130373: Avoid locking in _LOAD_ATTR_WITH_HINT (#130372) Avoid locking in _LOAD_ATTR_WITH_HINT
2025-03-28 15:16:41 -07:00
attr_o = FT_ATOMIC_LOAD_PTR(ep->me_value);
if (attr_o == NULL) {
gh-115999: Specialize `LOAD_ATTR` for instance and class receivers in free-threaded builds (#128164) Finish specialization for LOAD_ATTR in the free-threaded build by adding support for class and instance receivers.
2025-01-14 11:56:11 -08:00
if (true) {
UOP_STAT_INC(uopcode, miss);
GH-142718: JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. (GH-142762) JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. * Make sure that stack is in correct state at side exits with TOS cached values * Simplify choice of cached items for side exits
2025-12-15 19:01:29 +00:00
_tos_cache0 = owner;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(1);
gh-115999: Specialize `LOAD_ATTR` for instance and class receivers in free-threaded builds (#128164) Finish specialization for LOAD_ATTR in the free-threaded build by adding support for class and instance receivers.
2025-01-14 11:56:11 -08:00
JUMP_TO_JUMP_TARGET();
}
gh-104909: Split more LOAD_ATTR specializations (GH-110317) * Split LOAD_ATTR_MODULE * Split LOAD_ATTR_WITH_HINT * Split _GUARD_TYPE_VERSION out of the latter * Split LOAD_ATTR_CLASS * Split LOAD_ATTR_NONDESCRIPTOR_WITH_VALUES * Fix indent of DEOPT_IF in macros * Split LOAD_ATTR_METHOD_LAZY_DICT * Split LOAD_ATTR_NONDESCRIPTOR_NO_DICT * Fix omission of _CHECK_ATTR_METHOD_LAZY_DICT
2023-10-04 08:08:02 -07:00
}
gh-130373: Avoid locking in _LOAD_ATTR_WITH_HINT (#130372) Avoid locking in _LOAD_ATTR_WITH_HINT
2025-03-28 15:16:41 -07:00
STAT_INC(LOAD_ATTR, hit);
#ifdef Py_GIL_DISABLED
GH-131498: Cases generator: Parse down to C statement level. (GH-131948) * Parse down to statement level in the cases generator * Add handling for #if macros, treating them much like normal ifs.
2025-04-02 16:31:59 +01:00
int increfed = _Py_TryIncrefCompareStackRef(&ep->me_value, attr_o, &attr);
if (!increfed) {
gh-115999: Specialize `LOAD_ATTR` for instance and class receivers in free-threaded builds (#128164) Finish specialization for LOAD_ATTR in the free-threaded build by adding support for class and instance receivers.
2025-01-14 11:56:11 -08:00
if (true) {
UOP_STAT_INC(uopcode, miss);
GH-142718: JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. (GH-142762) JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. * Make sure that stack is in correct state at side exits with TOS cached values * Simplify choice of cached items for side exits
2025-12-15 19:01:29 +00:00
_tos_cache0 = owner;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(1);
gh-115999: Specialize `LOAD_ATTR` for instance and class receivers in free-threaded builds (#128164) Finish specialization for LOAD_ATTR in the free-threaded build by adding support for class and instance receivers.
2025-01-14 11:56:11 -08:00
JUMP_TO_JUMP_TARGET();
}
GH-117457: Correct pystats uop "miss" counts (GH-117477)
2024-04-04 18:49:18 -04:00
}
gh-130373: Avoid locking in _LOAD_ATTR_WITH_HINT (#130372) Avoid locking in _LOAD_ATTR_WITH_HINT
2025-03-28 15:16:41 -07:00
#else
gh-115999: Specialize `LOAD_ATTR` for instance and class receivers in free-threaded builds (#128164) Finish specialization for LOAD_ATTR in the free-threaded build by adding support for class and instance receivers.
2025-01-14 11:56:11 -08:00
attr = PyStackRef_FromPyObjectNew(attr_o);
gh-130373: Avoid locking in _LOAD_ATTR_WITH_HINT (#130372) Avoid locking in _LOAD_ATTR_WITH_HINT
2025-03-28 15:16:41 -07:00
#endif
gh-134584: Eliminate redundant refcounting from `_LOAD_ATTR_WITH_HINT` (GH-143062) Eliminate redundant refcounting from _LOAD_ATTR_WITH_HINT
2025-12-23 08:28:08 +08:00
o = owner;
_tos_cache1 = o;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_tos_cache0 = attr;
_tos_cache2 = PyStackRef_ZERO_BITS;
gh-134584: Eliminate redundant refcounting from `_LOAD_ATTR_WITH_HINT` (GH-143062) Eliminate redundant refcounting from _LOAD_ATTR_WITH_HINT
2025-12-23 08:28:08 +08:00
SET_CURRENT_CACHED_VALUES(2);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-104909: Split more LOAD_ATTR specializations (GH-110317) * Split LOAD_ATTR_MODULE * Split LOAD_ATTR_WITH_HINT * Split _GUARD_TYPE_VERSION out of the latter * Split LOAD_ATTR_CLASS * Split LOAD_ATTR_NONDESCRIPTOR_WITH_VALUES * Fix indent of DEOPT_IF in macros * Split LOAD_ATTR_METHOD_LAZY_DICT * Split LOAD_ATTR_NONDESCRIPTOR_NO_DICT * Fix omission of _CHECK_ATTR_METHOD_LAZY_DICT
2023-10-04 08:08:02 -07:00
break;
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
case _LOAD_ATTR_SLOT_r11: {
CHECK_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
Revert "GH-128914: Remove conditional stack effects from `bytecodes.c` and the code generators (GH-128918)" (GH-129202) The commit introduced a ~2.5-3% regression in the free threading build. This reverts commit ab61d3f4303d14a413bc9ae6557c730ffdf7579e.
2025-01-23 04:26:25 -05:00
_PyStackRef owner;
_PyStackRef attr;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef _stack_item_0 = _tos_cache0;
owner = _stack_item_0;
GH-140683: JIT: Improve machine code for loading smaller constants on AArch64. (GH-142511) * Use movz and movk instructions for loading 16 and 32 bit operands and oparg. * Loading of 64 bit operands is unchanged.
2025-12-11 12:33:39 +00:00
uint16_t index = (uint16_t)CURRENT_OPERAND0_16();
Revert "GH-128914: Remove conditional stack effects from `bytecodes.c` and the code generators (GH-128918)" (GH-129202) The commit introduced a ~2.5-3% regression in the free threading build. This reverts commit ab61d3f4303d14a413bc9ae6557c730ffdf7579e.
2025-01-23 04:26:25 -05:00
PyObject *owner_o = PyStackRef_AsPyObjectBorrow(owner);
PyObject **addr = (PyObject **)((char *)owner_o + index);
PyObject *attr_o = FT_ATOMIC_LOAD_PTR(*addr);
if (attr_o == NULL) {
UOP_STAT_INC(uopcode, miss);
GH-142718: JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. (GH-142762) JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. * Make sure that stack is in correct state at side exits with TOS cached values * Simplify choice of cached items for side exits
2025-12-15 19:01:29 +00:00
_tos_cache0 = owner;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(1);
Revert "GH-128914: Remove conditional stack effects from `bytecodes.c` and the code generators (GH-128918)" (GH-129202) The commit introduced a ~2.5-3% regression in the free threading build. This reverts commit ab61d3f4303d14a413bc9ae6557c730ffdf7579e.
2025-01-23 04:26:25 -05:00
JUMP_TO_JUMP_TARGET();
}
#ifdef Py_GIL_DISABLED
int increfed = _Py_TryIncrefCompareStackRef(addr, attr_o, &attr);
if (!increfed) {
UOP_STAT_INC(uopcode, miss);
GH-142718: JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. (GH-142762) JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. * Make sure that stack is in correct state at side exits with TOS cached values * Simplify choice of cached items for side exits
2025-12-15 19:01:29 +00:00
_tos_cache0 = owner;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(1);
Revert "GH-128914: Remove conditional stack effects from `bytecodes.c` and the code generators (GH-128918)" (GH-129202) The commit introduced a ~2.5-3% regression in the free threading build. This reverts commit ab61d3f4303d14a413bc9ae6557c730ffdf7579e.
2025-01-23 04:26:25 -05:00
JUMP_TO_JUMP_TARGET();
}
#else
attr = PyStackRef_FromPyObjectNew(attr_o);
#endif
STAT_INC(LOAD_ATTR, hit);
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
stack_pointer[0] = owner;
stack_pointer += 1;
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
GH-128682: Account for escapes in `DECREF_INPUTS` (GH-129953) * Handle escapes in DECREF_INPUTS * Mark a few more functions as escaping * Replace DECREF_INPUTS with PyStackRef_CLOSE where possible
2025-02-12 17:44:59 +00:00
_PyFrame_SetStackPointer(frame, stack_pointer);
_PyStackRef tmp = owner;
owner = attr;
stack_pointer[-1] = owner;
PyStackRef_CLOSE(tmp);
stack_pointer = _PyFrame_GetStackPointer(frame);
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_tos_cache0 = attr;
_tos_cache1 = PyStackRef_ZERO_BITS;
_tos_cache2 = PyStackRef_ZERO_BITS;
SET_CURRENT_CACHED_VALUES(1);
stack_pointer += -1;
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
Revert "GH-128914: Remove conditional stack effects from `bytecodes.c` and the code generators (GH-128918)" (GH-129202) The commit introduced a ~2.5-3% regression in the free threading build. This reverts commit ab61d3f4303d14a413bc9ae6557c730ffdf7579e.
2025-01-23 04:26:25 -05:00
break;
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
case _CHECK_ATTR_CLASS_r01: {
CHECK_CURRENT_CACHED_VALUES(0);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
_PyStackRef owner;
gh-104909: Split more LOAD_ATTR specializations (GH-110317) * Split LOAD_ATTR_MODULE * Split LOAD_ATTR_WITH_HINT * Split _GUARD_TYPE_VERSION out of the latter * Split LOAD_ATTR_CLASS * Split LOAD_ATTR_NONDESCRIPTOR_WITH_VALUES * Fix indent of DEOPT_IF in macros * Split LOAD_ATTR_METHOD_LAZY_DICT * Split LOAD_ATTR_NONDESCRIPTOR_NO_DICT * Fix omission of _CHECK_ATTR_METHOD_LAZY_DICT
2023-10-04 08:08:02 -07:00
owner = stack_pointer[-1];
GH-140683: JIT: Improve machine code for loading smaller constants on AArch64. (GH-142511) * Use movz and movk instructions for loading 16 and 32 bit operands and oparg. * Loading of 64 bit operands is unchanged.
2025-12-11 12:33:39 +00:00
uint32_t type_version = (uint32_t)CURRENT_OPERAND0_32();
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
PyObject *owner_o = PyStackRef_AsPyObjectBorrow(owner);
if (!PyType_Check(owner_o)) {
GH-117457: Correct pystats uop "miss" counts (GH-117477)
2024-04-04 18:49:18 -04:00
UOP_STAT_INC(uopcode, miss);
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(0);
JUMP_TO_JUMP_TARGET();
}
assert(type_version != 0);
if (FT_ATOMIC_LOAD_UINT_RELAXED(((PyTypeObject *)owner_o)->tp_version_tag) != type_version) {
UOP_STAT_INC(uopcode, miss);
SET_CURRENT_CACHED_VALUES(0);
JUMP_TO_JUMP_TARGET();
}
_tos_cache0 = owner;
SET_CURRENT_CACHED_VALUES(1);
stack_pointer += -1;
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _CHECK_ATTR_CLASS_r11: {
CHECK_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef owner;
_PyStackRef _stack_item_0 = _tos_cache0;
owner = _stack_item_0;
GH-140683: JIT: Improve machine code for loading smaller constants on AArch64. (GH-142511) * Use movz and movk instructions for loading 16 and 32 bit operands and oparg. * Loading of 64 bit operands is unchanged.
2025-12-11 12:33:39 +00:00
uint32_t type_version = (uint32_t)CURRENT_OPERAND0_32();
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
PyObject *owner_o = PyStackRef_AsPyObjectBorrow(owner);
if (!PyType_Check(owner_o)) {
UOP_STAT_INC(uopcode, miss);
GH-142718: JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. (GH-142762) JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. * Make sure that stack is in correct state at side exits with TOS cached values * Simplify choice of cached items for side exits
2025-12-15 19:01:29 +00:00
_tos_cache0 = owner;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(1);
JUMP_TO_JUMP_TARGET();
}
assert(type_version != 0);
if (FT_ATOMIC_LOAD_UINT_RELAXED(((PyTypeObject *)owner_o)->tp_version_tag) != type_version) {
UOP_STAT_INC(uopcode, miss);
GH-142718: JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. (GH-142762) JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. * Make sure that stack is in correct state at side exits with TOS cached values * Simplify choice of cached items for side exits
2025-12-15 19:01:29 +00:00
_tos_cache0 = owner;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(1);
JUMP_TO_JUMP_TARGET();
}
_tos_cache0 = owner;
SET_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _CHECK_ATTR_CLASS_r22: {
CHECK_CURRENT_CACHED_VALUES(2);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef owner;
_PyStackRef _stack_item_0 = _tos_cache0;
_PyStackRef _stack_item_1 = _tos_cache1;
owner = _stack_item_1;
GH-140683: JIT: Improve machine code for loading smaller constants on AArch64. (GH-142511) * Use movz and movk instructions for loading 16 and 32 bit operands and oparg. * Loading of 64 bit operands is unchanged.
2025-12-11 12:33:39 +00:00
uint32_t type_version = (uint32_t)CURRENT_OPERAND0_32();
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
PyObject *owner_o = PyStackRef_AsPyObjectBorrow(owner);
if (!PyType_Check(owner_o)) {
UOP_STAT_INC(uopcode, miss);
GH-142718: JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. (GH-142762) JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. * Make sure that stack is in correct state at side exits with TOS cached values * Simplify choice of cached items for side exits
2025-12-15 19:01:29 +00:00
_tos_cache1 = owner;
_tos_cache0 = _stack_item_0;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(2);
JUMP_TO_JUMP_TARGET();
}
assert(type_version != 0);
if (FT_ATOMIC_LOAD_UINT_RELAXED(((PyTypeObject *)owner_o)->tp_version_tag) != type_version) {
UOP_STAT_INC(uopcode, miss);
GH-142718: JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. (GH-142762) JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. * Make sure that stack is in correct state at side exits with TOS cached values * Simplify choice of cached items for side exits
2025-12-15 19:01:29 +00:00
_tos_cache1 = owner;
_tos_cache0 = _stack_item_0;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(2);
JUMP_TO_JUMP_TARGET();
}
_tos_cache1 = owner;
_tos_cache0 = _stack_item_0;
SET_CURRENT_CACHED_VALUES(2);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _CHECK_ATTR_CLASS_r33: {
CHECK_CURRENT_CACHED_VALUES(3);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef owner;
_PyStackRef _stack_item_0 = _tos_cache0;
_PyStackRef _stack_item_1 = _tos_cache1;
_PyStackRef _stack_item_2 = _tos_cache2;
owner = _stack_item_2;
GH-140683: JIT: Improve machine code for loading smaller constants on AArch64. (GH-142511) * Use movz and movk instructions for loading 16 and 32 bit operands and oparg. * Loading of 64 bit operands is unchanged.
2025-12-11 12:33:39 +00:00
uint32_t type_version = (uint32_t)CURRENT_OPERAND0_32();
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
PyObject *owner_o = PyStackRef_AsPyObjectBorrow(owner);
if (!PyType_Check(owner_o)) {
UOP_STAT_INC(uopcode, miss);
GH-142718: JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. (GH-142762) JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. * Make sure that stack is in correct state at side exits with TOS cached values * Simplify choice of cached items for side exits
2025-12-15 19:01:29 +00:00
_tos_cache2 = owner;
_tos_cache1 = _stack_item_1;
_tos_cache0 = _stack_item_0;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(3);
GH-117457: Correct pystats uop "miss" counts (GH-117477)
2024-04-04 18:49:18 -04:00
JUMP_TO_JUMP_TARGET();
}
gh-104909: Split more LOAD_ATTR specializations (GH-110317) * Split LOAD_ATTR_MODULE * Split LOAD_ATTR_WITH_HINT * Split _GUARD_TYPE_VERSION out of the latter * Split LOAD_ATTR_CLASS * Split LOAD_ATTR_NONDESCRIPTOR_WITH_VALUES * Fix indent of DEOPT_IF in macros * Split LOAD_ATTR_METHOD_LAZY_DICT * Split LOAD_ATTR_NONDESCRIPTOR_NO_DICT * Fix omission of _CHECK_ATTR_METHOD_LAZY_DICT
2023-10-04 08:08:02 -07:00
assert(type_version != 0);
gh-115999: Specialize `LOAD_ATTR` for instance and class receivers in free-threaded builds (#128164) Finish specialization for LOAD_ATTR in the free-threaded build by adding support for class and instance receivers.
2025-01-14 11:56:11 -08:00
if (FT_ATOMIC_LOAD_UINT_RELAXED(((PyTypeObject *)owner_o)->tp_version_tag) != type_version) {
GH-117457: Correct pystats uop "miss" counts (GH-117477)
2024-04-04 18:49:18 -04:00
UOP_STAT_INC(uopcode, miss);
GH-142718: JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. (GH-142762) JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. * Make sure that stack is in correct state at side exits with TOS cached values * Simplify choice of cached items for side exits
2025-12-15 19:01:29 +00:00
_tos_cache2 = owner;
_tos_cache1 = _stack_item_1;
_tos_cache0 = _stack_item_0;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(3);
GH-117457: Correct pystats uop "miss" counts (GH-117477)
2024-04-04 18:49:18 -04:00
JUMP_TO_JUMP_TARGET();
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_tos_cache2 = owner;
_tos_cache1 = _stack_item_1;
_tos_cache0 = _stack_item_0;
SET_CURRENT_CACHED_VALUES(3);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-104909: Split more LOAD_ATTR specializations (GH-110317) * Split LOAD_ATTR_MODULE * Split LOAD_ATTR_WITH_HINT * Split _GUARD_TYPE_VERSION out of the latter * Split LOAD_ATTR_CLASS * Split LOAD_ATTR_NONDESCRIPTOR_WITH_VALUES * Fix indent of DEOPT_IF in macros * Split LOAD_ATTR_METHOD_LAZY_DICT * Split LOAD_ATTR_NONDESCRIPTOR_NO_DICT * Fix omission of _CHECK_ATTR_METHOD_LAZY_DICT
2023-10-04 08:08:02 -07:00
break;
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
case _LOAD_ATTR_CLASS_r11: {
CHECK_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
_PyStackRef owner;
_PyStackRef attr;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef _stack_item_0 = _tos_cache0;
owner = _stack_item_0;
GH-140683: JIT: Improve machine code for loading smaller constants on AArch64. (GH-142511) * Use movz and movk instructions for loading 16 and 32 bit operands and oparg. * Loading of 64 bit operands is unchanged.
2025-12-11 12:33:39 +00:00
PyObject *descr = (PyObject *)CURRENT_OPERAND0_64();
gh-104909: Split more LOAD_ATTR specializations (GH-110317) * Split LOAD_ATTR_MODULE * Split LOAD_ATTR_WITH_HINT * Split _GUARD_TYPE_VERSION out of the latter * Split LOAD_ATTR_CLASS * Split LOAD_ATTR_NONDESCRIPTOR_WITH_VALUES * Fix indent of DEOPT_IF in macros * Split LOAD_ATTR_METHOD_LAZY_DICT * Split LOAD_ATTR_NONDESCRIPTOR_NO_DICT * Fix omission of _CHECK_ATTR_METHOD_LAZY_DICT
2023-10-04 08:08:02 -07:00
STAT_INC(LOAD_ATTR, hit);
assert(descr != NULL);
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
attr = PyStackRef_FromPyObjectNew(descr);
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
stack_pointer[0] = owner;
stack_pointer += 1;
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
GH-128682: Account for escapes in `DECREF_INPUTS` (GH-129953) * Handle escapes in DECREF_INPUTS * Mark a few more functions as escaping * Replace DECREF_INPUTS with PyStackRef_CLOSE where possible
2025-02-12 17:44:59 +00:00
_PyFrame_SetStackPointer(frame, stack_pointer);
_PyStackRef tmp = owner;
owner = attr;
stack_pointer[-1] = owner;
PyStackRef_CLOSE(tmp);
stack_pointer = _PyFrame_GetStackPointer(frame);
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_tos_cache0 = attr;
_tos_cache1 = PyStackRef_ZERO_BITS;
_tos_cache2 = PyStackRef_ZERO_BITS;
SET_CURRENT_CACHED_VALUES(1);
stack_pointer += -1;
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-104909: Split more LOAD_ATTR specializations (GH-110317) * Split LOAD_ATTR_MODULE * Split LOAD_ATTR_WITH_HINT * Split _GUARD_TYPE_VERSION out of the latter * Split LOAD_ATTR_CLASS * Split LOAD_ATTR_NONDESCRIPTOR_WITH_VALUES * Fix indent of DEOPT_IF in macros * Split LOAD_ATTR_METHOD_LAZY_DICT * Split LOAD_ATTR_NONDESCRIPTOR_NO_DICT * Fix omission of _CHECK_ATTR_METHOD_LAZY_DICT
2023-10-04 08:08:02 -07:00
break;
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
case _LOAD_ATTR_PROPERTY_FRAME_r11: {
CHECK_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-118093: Add tier two support for LOAD_ATTR_PROPERTY (GH-122283)
2024-07-25 10:45:28 -07:00
_PyStackRef owner;
GH-135379: Remove types from stack items in code generator. (GH-135384) * Make casts explicit in the instruction definitions
2025-06-11 15:52:25 +01:00
_PyStackRef new_frame;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef _stack_item_0 = _tos_cache0;
Revert "GH-128914: Remove conditional stack effects from `bytecodes.c` and the code generators (GH-128918)" (GH-129202) The commit introduced a ~2.5-3% regression in the free threading build. This reverts commit ab61d3f4303d14a413bc9ae6557c730ffdf7579e.
2025-01-23 04:26:25 -05:00
oparg = CURRENT_OPARG();
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
owner = _stack_item_0;
GH-140683: JIT: Improve machine code for loading smaller constants on AArch64. (GH-142511) * Use movz and movk instructions for loading 16 and 32 bit operands and oparg. * Loading of 64 bit operands is unchanged.
2025-12-11 12:33:39 +00:00
PyObject *fget = (PyObject *)CURRENT_OPERAND0_64();
Revert "GH-128914: Remove conditional stack effects from `bytecodes.c` and the code generators (GH-128918)" (GH-129202) The commit introduced a ~2.5-3% regression in the free threading build. This reverts commit ab61d3f4303d14a413bc9ae6557c730ffdf7579e.
2025-01-23 04:26:25 -05:00
assert((oparg & 1) == 0);
GH-118093: Add tier two support for LOAD_ATTR_PROPERTY (GH-122283)
2024-07-25 10:45:28 -07:00
assert(Py_IS_TYPE(fget, &PyFunction_Type));
PyFunctionObject *f = (PyFunctionObject *)fget;
PyCodeObject *code = (PyCodeObject *)f->func_code;
if ((code->co_flags & (CO_VARKEYWORDS | CO_VARARGS | CO_OPTIMIZED)) != CO_OPTIMIZED) {
UOP_STAT_INC(uopcode, miss);
GH-142718: JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. (GH-142762) JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. * Make sure that stack is in correct state at side exits with TOS cached values * Simplify choice of cached items for side exits
2025-12-15 19:01:29 +00:00
_tos_cache0 = owner;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(1);
GH-118093: Add tier two support for LOAD_ATTR_PROPERTY (GH-122283)
2024-07-25 10:45:28 -07:00
JUMP_TO_JUMP_TARGET();
}
if (code->co_kwonlyargcount) {
UOP_STAT_INC(uopcode, miss);
GH-142718: JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. (GH-142762) JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. * Make sure that stack is in correct state at side exits with TOS cached values * Simplify choice of cached items for side exits
2025-12-15 19:01:29 +00:00
_tos_cache0 = owner;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(1);
GH-118093: Add tier two support for LOAD_ATTR_PROPERTY (GH-122283)
2024-07-25 10:45:28 -07:00
JUMP_TO_JUMP_TARGET();
}
if (code->co_argcount != 1) {
UOP_STAT_INC(uopcode, miss);
GH-142718: JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. (GH-142762) JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. * Make sure that stack is in correct state at side exits with TOS cached values * Simplify choice of cached items for side exits
2025-12-15 19:01:29 +00:00
_tos_cache0 = owner;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(1);
GH-118093: Add tier two support for LOAD_ATTR_PROPERTY (GH-122283)
2024-07-25 10:45:28 -07:00
JUMP_TO_JUMP_TARGET();
}
if (!_PyThreadState_HasStackSpace(tstate, code->co_framesize)) {
UOP_STAT_INC(uopcode, miss);
GH-142718: JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. (GH-142762) JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. * Make sure that stack is in correct state at side exits with TOS cached values * Simplify choice of cached items for side exits
2025-12-15 19:01:29 +00:00
_tos_cache0 = owner;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(1);
GH-118093: Add tier two support for LOAD_ATTR_PROPERTY (GH-122283)
2024-07-25 10:45:28 -07:00
JUMP_TO_JUMP_TARGET();
}
STAT_INC(LOAD_ATTR, hit);
GH-135379: Remove types from stack items in code generator. (GH-135384) * Make casts explicit in the instruction definitions
2025-06-11 15:52:25 +01:00
_PyInterpreterFrame *pushed_frame = _PyFrame_PushUnchecked(tstate, PyStackRef_FromPyObjectNew(fget), 1, frame);
pushed_frame->localsplus[0] = owner;
new_frame = PyStackRef_Wrap(pushed_frame);
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_tos_cache0 = new_frame;
_tos_cache1 = PyStackRef_ZERO_BITS;
_tos_cache2 = PyStackRef_ZERO_BITS;
SET_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-118093: Add tier two support for LOAD_ATTR_PROPERTY (GH-122283)
2024-07-25 10:45:28 -07:00
break;
}
GH-111485: Factor out tier 2 code generation from the rest of the interpreter code generator (GH-112968)
2023-12-12 12:12:17 +00:00
GH-130296: Avoid stack transients in four instructions. (GH-130310) * Combine _GUARD_GLOBALS_VERSION_PUSH_KEYS and _LOAD_GLOBAL_MODULE_FROM_KEYS into _LOAD_GLOBAL_MODULE * Combine _GUARD_BUILTINS_VERSION_PUSH_KEYS and _LOAD_GLOBAL_BUILTINS_FROM_KEYS into _LOAD_GLOBAL_BUILTINS * Combine _CHECK_ATTR_MODULE_PUSH_KEYS and _LOAD_ATTR_MODULE_FROM_KEYS into _LOAD_ATTR_MODULE * Remove stack transient in LOAD_ATTR_WITH_HINT
2025-02-28 18:00:38 +00:00
/* _LOAD_ATTR_GETATTRIBUTE_OVERRIDDEN is not a viable micro-op for tier 2 because it has too many cache entries */
GH-111485: Factor out tier 2 code generation from the rest of the interpreter code generator (GH-112968)
2023-12-12 12:12:17 +00:00
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
case _GUARD_DORV_NO_DICT_r01: {
CHECK_CURRENT_CACHED_VALUES(0);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
_PyStackRef owner;
gh-104909: Split some more insts into ops (#109943) These are the most popular specializations of `LOAD_ATTR` and `STORE_ATTR` that weren't already viable uops: * Split LOAD_ATTR_METHOD_WITH_VALUES * Split LOAD_ATTR_METHOD_NO_DICT * Split LOAD_ATTR_SLOT * Split STORE_ATTR_SLOT * Split STORE_ATTR_INSTANCE_VALUE Also: * Add `-v` flag to code generator which prints a list of non-viable uops (easter-egg: it can print execution counts -- see source) * Double _Py_UOP_MAX_TRACE_LENGTH to 128 I had dropped one of the DEOPT_IF() calls! :-(
2023-09-27 15:27:44 -07:00
owner = stack_pointer[-1];
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
PyObject *owner_o = PyStackRef_AsPyObjectBorrow(owner);
assert(Py_TYPE(owner_o)->tp_dictoffset < 0);
assert(Py_TYPE(owner_o)->tp_flags & Py_TPFLAGS_INLINE_VALUES);
gh-115999: Specialize `STORE_ATTR` in free-threaded builds. (gh-127838) * Add `_PyDictKeys_StringLookupSplit` which does locking on dict keys and use in place of `_PyDictKeys_StringLookup`. * Change `_PyObject_TryGetInstanceAttribute` to use that function in the case of split keys. * Add `unicodekeys_lookup_split` helper which allows code sharing between `_Py_dict_lookup` and `_PyDictKeys_StringLookupSplit`. * Fix locking for `STORE_ATTR_INSTANCE_VALUE`. Create `_GUARD_TYPE_VERSION_AND_LOCK` uop so that object stays locked and `tp_version_tag` cannot change. * Pass `tp_version_tag` to `specialize_dict_access()`, ensuring the version we store on the cache is the correct one (in case of it changing during the specalize analysis). * Split `analyze_descriptor` into `analyze_descriptor_load` and `analyze_descriptor_store` since those don't share much logic. Add `descriptor_is_class` helper function. * In `specialize_dict_access`, double check `_PyObject_GetManagedDict()` in case we race and dict was materialized before the lock. * Avoid borrowed references in `_Py_Specialize_StoreAttr()`. * Use `specialize()` and `unspecialize()` helpers. * Add unit tests to ensure specializing happens as expected in FT builds. * Add unit tests to attempt to trigger data races (useful for running under TSAN). * Add `has_split_table` function to `_testinternalcapi`.
2024-12-19 10:21:17 -08:00
if (_PyObject_GetManagedDict(owner_o) ||
!FT_ATOMIC_LOAD_UINT8(_PyObject_InlineValues(owner_o)->valid)) {
UNLOCK_OBJECT(owner_o);
if (true) {
UOP_STAT_INC(uopcode, miss);
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(0);
JUMP_TO_JUMP_TARGET();
}
}
_tos_cache0 = owner;
SET_CURRENT_CACHED_VALUES(1);
stack_pointer += -1;
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _GUARD_DORV_NO_DICT_r11: {
CHECK_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef owner;
_PyStackRef _stack_item_0 = _tos_cache0;
owner = _stack_item_0;
PyObject *owner_o = PyStackRef_AsPyObjectBorrow(owner);
assert(Py_TYPE(owner_o)->tp_dictoffset < 0);
assert(Py_TYPE(owner_o)->tp_flags & Py_TPFLAGS_INLINE_VALUES);
if (_PyObject_GetManagedDict(owner_o) ||
!FT_ATOMIC_LOAD_UINT8(_PyObject_InlineValues(owner_o)->valid)) {
UNLOCK_OBJECT(owner_o);
if (true) {
UOP_STAT_INC(uopcode, miss);
GH-142718: JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. (GH-142762) JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. * Make sure that stack is in correct state at side exits with TOS cached values * Simplify choice of cached items for side exits
2025-12-15 19:01:29 +00:00
_tos_cache0 = owner;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(1);
JUMP_TO_JUMP_TARGET();
}
}
_tos_cache0 = owner;
SET_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _GUARD_DORV_NO_DICT_r22: {
CHECK_CURRENT_CACHED_VALUES(2);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef owner;
_PyStackRef _stack_item_0 = _tos_cache0;
_PyStackRef _stack_item_1 = _tos_cache1;
owner = _stack_item_1;
PyObject *owner_o = PyStackRef_AsPyObjectBorrow(owner);
assert(Py_TYPE(owner_o)->tp_dictoffset < 0);
assert(Py_TYPE(owner_o)->tp_flags & Py_TPFLAGS_INLINE_VALUES);
if (_PyObject_GetManagedDict(owner_o) ||
!FT_ATOMIC_LOAD_UINT8(_PyObject_InlineValues(owner_o)->valid)) {
UNLOCK_OBJECT(owner_o);
if (true) {
UOP_STAT_INC(uopcode, miss);
GH-142718: JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. (GH-142762) JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. * Make sure that stack is in correct state at side exits with TOS cached values * Simplify choice of cached items for side exits
2025-12-15 19:01:29 +00:00
_tos_cache1 = owner;
_tos_cache0 = _stack_item_0;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(2);
JUMP_TO_JUMP_TARGET();
}
}
_tos_cache1 = owner;
_tos_cache0 = _stack_item_0;
SET_CURRENT_CACHED_VALUES(2);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _GUARD_DORV_NO_DICT_r33: {
CHECK_CURRENT_CACHED_VALUES(3);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef owner;
_PyStackRef _stack_item_0 = _tos_cache0;
_PyStackRef _stack_item_1 = _tos_cache1;
_PyStackRef _stack_item_2 = _tos_cache2;
owner = _stack_item_2;
PyObject *owner_o = PyStackRef_AsPyObjectBorrow(owner);
assert(Py_TYPE(owner_o)->tp_dictoffset < 0);
assert(Py_TYPE(owner_o)->tp_flags & Py_TPFLAGS_INLINE_VALUES);
if (_PyObject_GetManagedDict(owner_o) ||
!FT_ATOMIC_LOAD_UINT8(_PyObject_InlineValues(owner_o)->valid)) {
UNLOCK_OBJECT(owner_o);
if (true) {
UOP_STAT_INC(uopcode, miss);
GH-142718: JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. (GH-142762) JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. * Make sure that stack is in correct state at side exits with TOS cached values * Simplify choice of cached items for side exits
2025-12-15 19:01:29 +00:00
_tos_cache2 = owner;
_tos_cache1 = _stack_item_1;
_tos_cache0 = _stack_item_0;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(3);
gh-115999: Specialize `STORE_ATTR` in free-threaded builds. (gh-127838) * Add `_PyDictKeys_StringLookupSplit` which does locking on dict keys and use in place of `_PyDictKeys_StringLookup`. * Change `_PyObject_TryGetInstanceAttribute` to use that function in the case of split keys. * Add `unicodekeys_lookup_split` helper which allows code sharing between `_Py_dict_lookup` and `_PyDictKeys_StringLookupSplit`. * Fix locking for `STORE_ATTR_INSTANCE_VALUE`. Create `_GUARD_TYPE_VERSION_AND_LOCK` uop so that object stays locked and `tp_version_tag` cannot change. * Pass `tp_version_tag` to `specialize_dict_access()`, ensuring the version we store on the cache is the correct one (in case of it changing during the specalize analysis). * Split `analyze_descriptor` into `analyze_descriptor_load` and `analyze_descriptor_store` since those don't share much logic. Add `descriptor_is_class` helper function. * In `specialize_dict_access`, double check `_PyObject_GetManagedDict()` in case we race and dict was materialized before the lock. * Avoid borrowed references in `_Py_Specialize_StoreAttr()`. * Use `specialize()` and `unspecialize()` helpers. * Add unit tests to ensure specializing happens as expected in FT builds. * Add unit tests to attempt to trigger data races (useful for running under TSAN). * Add `has_split_table` function to `_testinternalcapi`.
2024-12-19 10:21:17 -08:00
JUMP_TO_JUMP_TARGET();
}
GH-117457: Correct pystats uop "miss" counts (GH-117477)
2024-04-04 18:49:18 -04:00
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_tos_cache2 = owner;
_tos_cache1 = _stack_item_1;
_tos_cache0 = _stack_item_0;
SET_CURRENT_CACHED_VALUES(3);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-104909: Split some more insts into ops (#109943) These are the most popular specializations of `LOAD_ATTR` and `STORE_ATTR` that weren't already viable uops: * Split LOAD_ATTR_METHOD_WITH_VALUES * Split LOAD_ATTR_METHOD_NO_DICT * Split LOAD_ATTR_SLOT * Split STORE_ATTR_SLOT * Split STORE_ATTR_INSTANCE_VALUE Also: * Add `-v` flag to code generator which prints a list of non-viable uops (easter-egg: it can print execution counts -- see source) * Double _Py_UOP_MAX_TRACE_LENGTH to 128 I had dropped one of the DEOPT_IF() calls! :-(
2023-09-27 15:27:44 -07:00
break;
}
gh-134584: Eliminate redundant refcounting from `_STORE_ATTR_INSTANCE_VALUE` (GH-142759) Signed-off-by: Manjusaka <me@manjusaka.me>
2025-12-16 04:17:12 +08:00
case _STORE_ATTR_INSTANCE_VALUE_r21: {
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
CHECK_CURRENT_CACHED_VALUES(2);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
_PyStackRef owner;
_PyStackRef value;
gh-134584: Eliminate redundant refcounting from `_STORE_ATTR_INSTANCE_VALUE` (GH-142759) Signed-off-by: Manjusaka <me@manjusaka.me>
2025-12-16 04:17:12 +08:00
_PyStackRef o;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef _stack_item_0 = _tos_cache0;
_PyStackRef _stack_item_1 = _tos_cache1;
owner = _stack_item_1;
value = _stack_item_0;
GH-140683: JIT: Improve machine code for loading smaller constants on AArch64. (GH-142511) * Use movz and movk instructions for loading 16 and 32 bit operands and oparg. * Loading of 64 bit operands is unchanged.
2025-12-11 12:33:39 +00:00
uint16_t offset = (uint16_t)CURRENT_OPERAND0_16();
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
PyObject *owner_o = PyStackRef_AsPyObjectBorrow(owner);
gh-104909: Split some more insts into ops (#109943) These are the most popular specializations of `LOAD_ATTR` and `STORE_ATTR` that weren't already viable uops: * Split LOAD_ATTR_METHOD_WITH_VALUES * Split LOAD_ATTR_METHOD_NO_DICT * Split LOAD_ATTR_SLOT * Split STORE_ATTR_SLOT * Split STORE_ATTR_INSTANCE_VALUE Also: * Add `-v` flag to code generator which prints a list of non-viable uops (easter-egg: it can print execution counts -- see source) * Double _Py_UOP_MAX_TRACE_LENGTH to 128 I had dropped one of the DEOPT_IF() calls! :-(
2023-09-27 15:27:44 -07:00
STAT_INC(STORE_ATTR, hit);
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
assert(_PyObject_GetManagedDict(owner_o) == NULL);
GH-115776: Allow any fixed sized object to have inline values (GH-123192)
2024-08-21 15:52:04 +01:00
PyObject **value_ptr = (PyObject**)(((char *)owner_o) + offset);
PyObject *old_value = *value_ptr;
gh-115999: Specialize `STORE_ATTR` in free-threaded builds. (gh-127838) * Add `_PyDictKeys_StringLookupSplit` which does locking on dict keys and use in place of `_PyDictKeys_StringLookup`. * Change `_PyObject_TryGetInstanceAttribute` to use that function in the case of split keys. * Add `unicodekeys_lookup_split` helper which allows code sharing between `_Py_dict_lookup` and `_PyDictKeys_StringLookupSplit`. * Fix locking for `STORE_ATTR_INSTANCE_VALUE`. Create `_GUARD_TYPE_VERSION_AND_LOCK` uop so that object stays locked and `tp_version_tag` cannot change. * Pass `tp_version_tag` to `specialize_dict_access()`, ensuring the version we store on the cache is the correct one (in case of it changing during the specalize analysis). * Split `analyze_descriptor` into `analyze_descriptor_load` and `analyze_descriptor_store` since those don't share much logic. Add `descriptor_is_class` helper function. * In `specialize_dict_access`, double check `_PyObject_GetManagedDict()` in case we race and dict was materialized before the lock. * Avoid borrowed references in `_Py_Specialize_StoreAttr()`. * Use `specialize()` and `unspecialize()` helpers. * Add unit tests to ensure specializing happens as expected in FT builds. * Add unit tests to attempt to trigger data races (useful for running under TSAN). * Add `has_split_table` function to `_testinternalcapi`.
2024-12-19 10:21:17 -08:00
FT_ATOMIC_STORE_PTR_RELEASE(*value_ptr, PyStackRef_AsPyObjectSteal(value));
gh-104909: Split some more insts into ops (#109943) These are the most popular specializations of `LOAD_ATTR` and `STORE_ATTR` that weren't already viable uops: * Split LOAD_ATTR_METHOD_WITH_VALUES * Split LOAD_ATTR_METHOD_NO_DICT * Split LOAD_ATTR_SLOT * Split STORE_ATTR_SLOT * Split STORE_ATTR_INSTANCE_VALUE Also: * Add `-v` flag to code generator which prints a list of non-viable uops (easter-egg: it can print execution counts -- see source) * Double _Py_UOP_MAX_TRACE_LENGTH to 128 I had dropped one of the DEOPT_IF() calls! :-(
2023-09-27 15:27:44 -07:00
if (old_value == NULL) {
GH-115776: Allow any fixed sized object to have inline values (GH-123192)
2024-08-21 15:52:04 +01:00
PyDictValues *values = _PyObject_InlineValues(owner_o);
gh-123205: `Python/bytecodes.c`: Fix compiler warning (#123206) Fix MSVC warning "conversion from '__int64' to 'int'"
2024-08-23 22:35:25 +03:00
Py_ssize_t index = value_ptr - values->values;
gh-104909: Split some more insts into ops (#109943) These are the most popular specializations of `LOAD_ATTR` and `STORE_ATTR` that weren't already viable uops: * Split LOAD_ATTR_METHOD_WITH_VALUES * Split LOAD_ATTR_METHOD_NO_DICT * Split LOAD_ATTR_SLOT * Split STORE_ATTR_SLOT * Split STORE_ATTR_INSTANCE_VALUE Also: * Add `-v` flag to code generator which prints a list of non-viable uops (easter-egg: it can print execution counts -- see source) * Double _Py_UOP_MAX_TRACE_LENGTH to 128 I had dropped one of the DEOPT_IF() calls! :-(
2023-09-27 15:27:44 -07:00
_PyDictValues_AddToInsertionOrder(values, index);
}
gh-115999: Specialize `STORE_ATTR` in free-threaded builds. (gh-127838) * Add `_PyDictKeys_StringLookupSplit` which does locking on dict keys and use in place of `_PyDictKeys_StringLookup`. * Change `_PyObject_TryGetInstanceAttribute` to use that function in the case of split keys. * Add `unicodekeys_lookup_split` helper which allows code sharing between `_Py_dict_lookup` and `_PyDictKeys_StringLookupSplit`. * Fix locking for `STORE_ATTR_INSTANCE_VALUE`. Create `_GUARD_TYPE_VERSION_AND_LOCK` uop so that object stays locked and `tp_version_tag` cannot change. * Pass `tp_version_tag` to `specialize_dict_access()`, ensuring the version we store on the cache is the correct one (in case of it changing during the specalize analysis). * Split `analyze_descriptor` into `analyze_descriptor_load` and `analyze_descriptor_store` since those don't share much logic. Add `descriptor_is_class` helper function. * In `specialize_dict_access`, double check `_PyObject_GetManagedDict()` in case we race and dict was materialized before the lock. * Avoid borrowed references in `_Py_Specialize_StoreAttr()`. * Use `specialize()` and `unspecialize()` helpers. * Add unit tests to ensure specializing happens as expected in FT builds. * Add unit tests to attempt to trigger data races (useful for running under TSAN). * Add `has_split_table` function to `_testinternalcapi`.
2024-12-19 10:21:17 -08:00
UNLOCK_OBJECT(owner_o);
gh-134584: Eliminate redundant refcounting from `_STORE_ATTR_INSTANCE_VALUE` (GH-142759) Signed-off-by: Manjusaka <me@manjusaka.me>
2025-12-16 04:17:12 +08:00
o = owner;
stack_pointer[0] = o;
stack_pointer += 1;
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
GH-128682: Convert explicit loops closing arrays into `DECREF_INPUTS`. (GH-128822) * Mark Py_DECREF and Py_XDECREF as escaping * Remove explicit loops for clearing array inputs
2025-01-14 15:08:56 +00:00
_PyFrame_SetStackPointer(frame, stack_pointer);
Py_XDECREF(old_value);
stack_pointer = _PyFrame_GetStackPointer(frame);
gh-134584: Eliminate redundant refcounting from `_STORE_ATTR_INSTANCE_VALUE` (GH-142759) Signed-off-by: Manjusaka <me@manjusaka.me>
2025-12-16 04:17:12 +08:00
_tos_cache0 = o;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_tos_cache1 = PyStackRef_ZERO_BITS;
_tos_cache2 = PyStackRef_ZERO_BITS;
gh-134584: Eliminate redundant refcounting from `_STORE_ATTR_INSTANCE_VALUE` (GH-142759) Signed-off-by: Manjusaka <me@manjusaka.me>
2025-12-16 04:17:12 +08:00
SET_CURRENT_CACHED_VALUES(1);
stack_pointer += -1;
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-104909: Split some more insts into ops (#109943) These are the most popular specializations of `LOAD_ATTR` and `STORE_ATTR` that weren't already viable uops: * Split LOAD_ATTR_METHOD_WITH_VALUES * Split LOAD_ATTR_METHOD_NO_DICT * Split LOAD_ATTR_SLOT * Split STORE_ATTR_SLOT * Split STORE_ATTR_INSTANCE_VALUE Also: * Add `-v` flag to code generator which prints a list of non-viable uops (easter-egg: it can print execution counts -- see source) * Double _Py_UOP_MAX_TRACE_LENGTH to 128 I had dropped one of the DEOPT_IF() calls! :-(
2023-09-27 15:27:44 -07:00
break;
}
gh-134584: Eliminate redundant refcounting from `_STORE_ATTR_WITH_HINT` (GH-142767) Signed-off-by: Manjusaka <me@manjusaka.me>
2025-12-17 07:21:04 +09:00
case _STORE_ATTR_WITH_HINT_r21: {
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
CHECK_CURRENT_CACHED_VALUES(2);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
_PyStackRef owner;
_PyStackRef value;
gh-134584: Eliminate redundant refcounting from `_STORE_ATTR_WITH_HINT` (GH-142767) Signed-off-by: Manjusaka <me@manjusaka.me>
2025-12-17 07:21:04 +09:00
_PyStackRef o;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef _stack_item_0 = _tos_cache0;
_PyStackRef _stack_item_1 = _tos_cache1;
GH-119258: Handle STORE_ATTR_WITH_HINT in tier two (GH-119481)
2024-05-28 12:47:54 -07:00
oparg = CURRENT_OPARG();
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
owner = _stack_item_1;
value = _stack_item_0;
GH-140683: JIT: Improve machine code for loading smaller constants on AArch64. (GH-142511) * Use movz and movk instructions for loading 16 and 32 bit operands and oparg. * Loading of 64 bit operands is unchanged.
2025-12-11 12:33:39 +00:00
uint16_t hint = (uint16_t)CURRENT_OPERAND0_16();
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
PyObject *owner_o = PyStackRef_AsPyObjectBorrow(owner);
assert(Py_TYPE(owner_o)->tp_flags & Py_TPFLAGS_MANAGED_DICT);
PyDictObject *dict = _PyObject_GetManagedDict(owner_o);
GH-119258: Handle STORE_ATTR_WITH_HINT in tier two (GH-119481)
2024-05-28 12:47:54 -07:00
if (dict == NULL) {
UOP_STAT_INC(uopcode, miss);
GH-142718: JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. (GH-142762) JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. * Make sure that stack is in correct state at side exits with TOS cached values * Simplify choice of cached items for side exits
2025-12-15 19:01:29 +00:00
_tos_cache1 = owner;
_tos_cache0 = value;
SET_CURRENT_CACHED_VALUES(2);
GH-119258: Handle STORE_ATTR_WITH_HINT in tier two (GH-119481)
2024-05-28 12:47:54 -07:00
JUMP_TO_JUMP_TARGET();
}
gh-115999: Specialize `STORE_ATTR` in free-threaded builds. (gh-127838) * Add `_PyDictKeys_StringLookupSplit` which does locking on dict keys and use in place of `_PyDictKeys_StringLookup`. * Change `_PyObject_TryGetInstanceAttribute` to use that function in the case of split keys. * Add `unicodekeys_lookup_split` helper which allows code sharing between `_Py_dict_lookup` and `_PyDictKeys_StringLookupSplit`. * Fix locking for `STORE_ATTR_INSTANCE_VALUE`. Create `_GUARD_TYPE_VERSION_AND_LOCK` uop so that object stays locked and `tp_version_tag` cannot change. * Pass `tp_version_tag` to `specialize_dict_access()`, ensuring the version we store on the cache is the correct one (in case of it changing during the specalize analysis). * Split `analyze_descriptor` into `analyze_descriptor_load` and `analyze_descriptor_store` since those don't share much logic. Add `descriptor_is_class` helper function. * In `specialize_dict_access`, double check `_PyObject_GetManagedDict()` in case we race and dict was materialized before the lock. * Avoid borrowed references in `_Py_Specialize_StoreAttr()`. * Use `specialize()` and `unspecialize()` helpers. * Add unit tests to ensure specializing happens as expected in FT builds. * Add unit tests to attempt to trigger data races (useful for running under TSAN). * Add `has_split_table` function to `_testinternalcapi`.
2024-12-19 10:21:17 -08:00
if (!LOCK_OBJECT(dict)) {
GH-119258: Handle STORE_ATTR_WITH_HINT in tier two (GH-119481)
2024-05-28 12:47:54 -07:00
UOP_STAT_INC(uopcode, miss);
GH-142718: JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. (GH-142762) JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. * Make sure that stack is in correct state at side exits with TOS cached values * Simplify choice of cached items for side exits
2025-12-15 19:01:29 +00:00
_tos_cache1 = owner;
_tos_cache0 = value;
SET_CURRENT_CACHED_VALUES(2);
GH-119258: Handle STORE_ATTR_WITH_HINT in tier two (GH-119481)
2024-05-28 12:47:54 -07:00
JUMP_TO_JUMP_TARGET();
}
gh-115999: Specialize `STORE_ATTR` in free-threaded builds. (gh-127838) * Add `_PyDictKeys_StringLookupSplit` which does locking on dict keys and use in place of `_PyDictKeys_StringLookup`. * Change `_PyObject_TryGetInstanceAttribute` to use that function in the case of split keys. * Add `unicodekeys_lookup_split` helper which allows code sharing between `_Py_dict_lookup` and `_PyDictKeys_StringLookupSplit`. * Fix locking for `STORE_ATTR_INSTANCE_VALUE`. Create `_GUARD_TYPE_VERSION_AND_LOCK` uop so that object stays locked and `tp_version_tag` cannot change. * Pass `tp_version_tag` to `specialize_dict_access()`, ensuring the version we store on the cache is the correct one (in case of it changing during the specalize analysis). * Split `analyze_descriptor` into `analyze_descriptor_load` and `analyze_descriptor_store` since those don't share much logic. Add `descriptor_is_class` helper function. * In `specialize_dict_access`, double check `_PyObject_GetManagedDict()` in case we race and dict was materialized before the lock. * Avoid borrowed references in `_Py_Specialize_StoreAttr()`. * Use `specialize()` and `unspecialize()` helpers. * Add unit tests to ensure specializing happens as expected in FT builds. * Add unit tests to attempt to trigger data races (useful for running under TSAN). * Add `has_split_table` function to `_testinternalcapi`.
2024-12-19 10:21:17 -08:00
assert(PyDict_CheckExact((PyObject *)dict));
PyObject *name = GETITEM(FRAME_CO_NAMES, oparg);
if (hint >= (size_t)dict->ma_keys->dk_nentries ||
!DK_IS_UNICODE(dict->ma_keys)) {
UNLOCK_OBJECT(dict);
if (true) {
UOP_STAT_INC(uopcode, miss);
GH-142718: JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. (GH-142762) JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. * Make sure that stack is in correct state at side exits with TOS cached values * Simplify choice of cached items for side exits
2025-12-15 19:01:29 +00:00
_tos_cache1 = owner;
_tos_cache0 = value;
SET_CURRENT_CACHED_VALUES(2);
gh-115999: Specialize `STORE_ATTR` in free-threaded builds. (gh-127838) * Add `_PyDictKeys_StringLookupSplit` which does locking on dict keys and use in place of `_PyDictKeys_StringLookup`. * Change `_PyObject_TryGetInstanceAttribute` to use that function in the case of split keys. * Add `unicodekeys_lookup_split` helper which allows code sharing between `_Py_dict_lookup` and `_PyDictKeys_StringLookupSplit`. * Fix locking for `STORE_ATTR_INSTANCE_VALUE`. Create `_GUARD_TYPE_VERSION_AND_LOCK` uop so that object stays locked and `tp_version_tag` cannot change. * Pass `tp_version_tag` to `specialize_dict_access()`, ensuring the version we store on the cache is the correct one (in case of it changing during the specalize analysis). * Split `analyze_descriptor` into `analyze_descriptor_load` and `analyze_descriptor_store` since those don't share much logic. Add `descriptor_is_class` helper function. * In `specialize_dict_access`, double check `_PyObject_GetManagedDict()` in case we race and dict was materialized before the lock. * Avoid borrowed references in `_Py_Specialize_StoreAttr()`. * Use `specialize()` and `unspecialize()` helpers. * Add unit tests to ensure specializing happens as expected in FT builds. * Add unit tests to attempt to trigger data races (useful for running under TSAN). * Add `has_split_table` function to `_testinternalcapi`.
2024-12-19 10:21:17 -08:00
JUMP_TO_JUMP_TARGET();
}
GH-119258: Handle STORE_ATTR_WITH_HINT in tier two (GH-119481)
2024-05-28 12:47:54 -07:00
}
GH-122616: Simplify LOAD_ATTR_WITH_HINT and STORE_ATTR_WITH_HINT (GH-122620)
2024-08-05 16:27:48 +01:00
PyDictUnicodeEntry *ep = DK_UNICODE_ENTRIES(dict->ma_keys) + hint;
if (ep->me_key != name) {
gh-115999: Specialize `STORE_ATTR` in free-threaded builds. (gh-127838) * Add `_PyDictKeys_StringLookupSplit` which does locking on dict keys and use in place of `_PyDictKeys_StringLookup`. * Change `_PyObject_TryGetInstanceAttribute` to use that function in the case of split keys. * Add `unicodekeys_lookup_split` helper which allows code sharing between `_Py_dict_lookup` and `_PyDictKeys_StringLookupSplit`. * Fix locking for `STORE_ATTR_INSTANCE_VALUE`. Create `_GUARD_TYPE_VERSION_AND_LOCK` uop so that object stays locked and `tp_version_tag` cannot change. * Pass `tp_version_tag` to `specialize_dict_access()`, ensuring the version we store on the cache is the correct one (in case of it changing during the specalize analysis). * Split `analyze_descriptor` into `analyze_descriptor_load` and `analyze_descriptor_store` since those don't share much logic. Add `descriptor_is_class` helper function. * In `specialize_dict_access`, double check `_PyObject_GetManagedDict()` in case we race and dict was materialized before the lock. * Avoid borrowed references in `_Py_Specialize_StoreAttr()`. * Use `specialize()` and `unspecialize()` helpers. * Add unit tests to ensure specializing happens as expected in FT builds. * Add unit tests to attempt to trigger data races (useful for running under TSAN). * Add `has_split_table` function to `_testinternalcapi`.
2024-12-19 10:21:17 -08:00
UNLOCK_OBJECT(dict);
if (true) {
UOP_STAT_INC(uopcode, miss);
GH-142718: JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. (GH-142762) JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. * Make sure that stack is in correct state at side exits with TOS cached values * Simplify choice of cached items for side exits
2025-12-15 19:01:29 +00:00
_tos_cache1 = owner;
_tos_cache0 = value;
SET_CURRENT_CACHED_VALUES(2);
gh-115999: Specialize `STORE_ATTR` in free-threaded builds. (gh-127838) * Add `_PyDictKeys_StringLookupSplit` which does locking on dict keys and use in place of `_PyDictKeys_StringLookup`. * Change `_PyObject_TryGetInstanceAttribute` to use that function in the case of split keys. * Add `unicodekeys_lookup_split` helper which allows code sharing between `_Py_dict_lookup` and `_PyDictKeys_StringLookupSplit`. * Fix locking for `STORE_ATTR_INSTANCE_VALUE`. Create `_GUARD_TYPE_VERSION_AND_LOCK` uop so that object stays locked and `tp_version_tag` cannot change. * Pass `tp_version_tag` to `specialize_dict_access()`, ensuring the version we store on the cache is the correct one (in case of it changing during the specalize analysis). * Split `analyze_descriptor` into `analyze_descriptor_load` and `analyze_descriptor_store` since those don't share much logic. Add `descriptor_is_class` helper function. * In `specialize_dict_access`, double check `_PyObject_GetManagedDict()` in case we race and dict was materialized before the lock. * Avoid borrowed references in `_Py_Specialize_StoreAttr()`. * Use `specialize()` and `unspecialize()` helpers. * Add unit tests to ensure specializing happens as expected in FT builds. * Add unit tests to attempt to trigger data races (useful for running under TSAN). * Add `has_split_table` function to `_testinternalcapi`.
2024-12-19 10:21:17 -08:00
JUMP_TO_JUMP_TARGET();
}
GH-119258: Handle STORE_ATTR_WITH_HINT in tier two (GH-119481)
2024-05-28 12:47:54 -07:00
}
GH-125868: Fix STORE_ATTR_WITH_HINT specialization (GH-125876)
2024-10-24 11:57:02 +01:00
PyObject *old_value = ep->me_value;
if (old_value == NULL) {
gh-115999: Specialize `STORE_ATTR` in free-threaded builds. (gh-127838) * Add `_PyDictKeys_StringLookupSplit` which does locking on dict keys and use in place of `_PyDictKeys_StringLookup`. * Change `_PyObject_TryGetInstanceAttribute` to use that function in the case of split keys. * Add `unicodekeys_lookup_split` helper which allows code sharing between `_Py_dict_lookup` and `_PyDictKeys_StringLookupSplit`. * Fix locking for `STORE_ATTR_INSTANCE_VALUE`. Create `_GUARD_TYPE_VERSION_AND_LOCK` uop so that object stays locked and `tp_version_tag` cannot change. * Pass `tp_version_tag` to `specialize_dict_access()`, ensuring the version we store on the cache is the correct one (in case of it changing during the specalize analysis). * Split `analyze_descriptor` into `analyze_descriptor_load` and `analyze_descriptor_store` since those don't share much logic. Add `descriptor_is_class` helper function. * In `specialize_dict_access`, double check `_PyObject_GetManagedDict()` in case we race and dict was materialized before the lock. * Avoid borrowed references in `_Py_Specialize_StoreAttr()`. * Use `specialize()` and `unspecialize()` helpers. * Add unit tests to ensure specializing happens as expected in FT builds. * Add unit tests to attempt to trigger data races (useful for running under TSAN). * Add `has_split_table` function to `_testinternalcapi`.
2024-12-19 10:21:17 -08:00
UNLOCK_OBJECT(dict);
if (true) {
UOP_STAT_INC(uopcode, miss);
GH-142718: JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. (GH-142762) JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. * Make sure that stack is in correct state at side exits with TOS cached values * Simplify choice of cached items for side exits
2025-12-15 19:01:29 +00:00
_tos_cache1 = owner;
_tos_cache0 = value;
SET_CURRENT_CACHED_VALUES(2);
gh-115999: Specialize `STORE_ATTR` in free-threaded builds. (gh-127838) * Add `_PyDictKeys_StringLookupSplit` which does locking on dict keys and use in place of `_PyDictKeys_StringLookup`. * Change `_PyObject_TryGetInstanceAttribute` to use that function in the case of split keys. * Add `unicodekeys_lookup_split` helper which allows code sharing between `_Py_dict_lookup` and `_PyDictKeys_StringLookupSplit`. * Fix locking for `STORE_ATTR_INSTANCE_VALUE`. Create `_GUARD_TYPE_VERSION_AND_LOCK` uop so that object stays locked and `tp_version_tag` cannot change. * Pass `tp_version_tag` to `specialize_dict_access()`, ensuring the version we store on the cache is the correct one (in case of it changing during the specalize analysis). * Split `analyze_descriptor` into `analyze_descriptor_load` and `analyze_descriptor_store` since those don't share much logic. Add `descriptor_is_class` helper function. * In `specialize_dict_access`, double check `_PyObject_GetManagedDict()` in case we race and dict was materialized before the lock. * Avoid borrowed references in `_Py_Specialize_StoreAttr()`. * Use `specialize()` and `unspecialize()` helpers. * Add unit tests to ensure specializing happens as expected in FT builds. * Add unit tests to attempt to trigger data races (useful for running under TSAN). * Add `has_split_table` function to `_testinternalcapi`.
2024-12-19 10:21:17 -08:00
JUMP_TO_JUMP_TARGET();
}
GH-125868: Fix STORE_ATTR_WITH_HINT specialization (GH-125876)
2024-10-24 11:57:02 +01:00
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
stack_pointer[0] = value;
stack_pointer[1] = owner;
stack_pointer += 2;
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
GH-119866: Spill the stack around escaping calls. (GH-124392) * Spill the evaluation around escaping calls in the generated interpreter and JIT. * The code generator tracks live, cached values so they can be saved to memory when needed. * Spills the stack pointer around escaping calls, so that the exact stack is visible to the cycle GC.
2024-10-07 14:56:39 +01:00
_PyFrame_SetStackPointer(frame, stack_pointer);
gh-142890: remove unnecessary interp parameter from dict functions and `_PyDict_NotifyEvent` (#142923)
2025-12-18 22:48:56 +05:30
_PyDict_NotifyEvent(PyDict_EVENT_MODIFIED, dict, name, PyStackRef_AsPyObjectBorrow(value));
GH-119866: Spill the stack around escaping calls. (GH-124392) * Spill the evaluation around escaping calls in the generated interpreter and JIT. * The code generator tracks live, cached values so they can be saved to memory when needed. * Spills the stack pointer around escaping calls, so that the exact stack is visible to the cycle GC.
2024-10-07 14:56:39 +01:00
stack_pointer = _PyFrame_GetStackPointer(frame);
gh-115999: Specialize `STORE_ATTR` in free-threaded builds. (gh-127838) * Add `_PyDictKeys_StringLookupSplit` which does locking on dict keys and use in place of `_PyDictKeys_StringLookup`. * Change `_PyObject_TryGetInstanceAttribute` to use that function in the case of split keys. * Add `unicodekeys_lookup_split` helper which allows code sharing between `_Py_dict_lookup` and `_PyDictKeys_StringLookupSplit`. * Fix locking for `STORE_ATTR_INSTANCE_VALUE`. Create `_GUARD_TYPE_VERSION_AND_LOCK` uop so that object stays locked and `tp_version_tag` cannot change. * Pass `tp_version_tag` to `specialize_dict_access()`, ensuring the version we store on the cache is the correct one (in case of it changing during the specalize analysis). * Split `analyze_descriptor` into `analyze_descriptor_load` and `analyze_descriptor_store` since those don't share much logic. Add `descriptor_is_class` helper function. * In `specialize_dict_access`, double check `_PyObject_GetManagedDict()` in case we race and dict was materialized before the lock. * Avoid borrowed references in `_Py_Specialize_StoreAttr()`. * Use `specialize()` and `unspecialize()` helpers. * Add unit tests to ensure specializing happens as expected in FT builds. * Add unit tests to attempt to trigger data races (useful for running under TSAN). * Add `has_split_table` function to `_testinternalcapi`.
2024-12-19 10:21:17 -08:00
FT_ATOMIC_STORE_PTR_RELEASE(ep->me_value, PyStackRef_AsPyObjectSteal(value));
UNLOCK_OBJECT(dict);
GH-119258: Handle STORE_ATTR_WITH_HINT in tier two (GH-119481)
2024-05-28 12:47:54 -07:00
STAT_INC(STORE_ATTR, hit);
gh-134584: Eliminate redundant refcounting from `_STORE_ATTR_WITH_HINT` (GH-142767) Signed-off-by: Manjusaka <me@manjusaka.me>
2025-12-17 07:21:04 +09:00
o = owner;
stack_pointer[-2] = o;
stack_pointer += -1;
GH-141794: Limit size of generated machine code. (GH-142228) * Factor out bodies of the largest uops, to reduce jit code size. * Factor out common assert, also reducing jit code size. * Limit size of jitted code for a single executor to 1MB.
2025-12-03 17:43:35 +00:00
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
GH-128682: Convert explicit loops closing arrays into `DECREF_INPUTS`. (GH-128822) * Mark Py_DECREF and Py_XDECREF as escaping * Remove explicit loops for clearing array inputs
2025-01-14 15:08:56 +00:00
_PyFrame_SetStackPointer(frame, stack_pointer);
Py_XDECREF(old_value);
stack_pointer = _PyFrame_GetStackPointer(frame);
gh-134584: Eliminate redundant refcounting from `_STORE_ATTR_WITH_HINT` (GH-142767) Signed-off-by: Manjusaka <me@manjusaka.me>
2025-12-17 07:21:04 +09:00
_tos_cache0 = o;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_tos_cache1 = PyStackRef_ZERO_BITS;
_tos_cache2 = PyStackRef_ZERO_BITS;
gh-134584: Eliminate redundant refcounting from `_STORE_ATTR_WITH_HINT` (GH-142767) Signed-off-by: Manjusaka <me@manjusaka.me>
2025-12-17 07:21:04 +09:00
SET_CURRENT_CACHED_VALUES(1);
stack_pointer += -1;
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-119258: Handle STORE_ATTR_WITH_HINT in tier two (GH-119481)
2024-05-28 12:47:54 -07:00
break;
}
GH-111485: Factor out tier 2 code generation from the rest of the interpreter code generator (GH-112968)
2023-12-12 12:12:17 +00:00
GH-134584: Remove redundant refcount from `_STORE_ATTR_SLOT` (#142729)
2025-12-15 15:18:44 -08:00
case _STORE_ATTR_SLOT_r21: {
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
CHECK_CURRENT_CACHED_VALUES(2);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
_PyStackRef owner;
_PyStackRef value;
GH-134584: Remove redundant refcount from `_STORE_ATTR_SLOT` (#142729)
2025-12-15 15:18:44 -08:00
_PyStackRef o;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef _stack_item_0 = _tos_cache0;
_PyStackRef _stack_item_1 = _tos_cache1;
owner = _stack_item_1;
value = _stack_item_0;
GH-140683: JIT: Improve machine code for loading smaller constants on AArch64. (GH-142511) * Use movz and movk instructions for loading 16 and 32 bit operands and oparg. * Loading of 64 bit operands is unchanged.
2025-12-11 12:33:39 +00:00
uint16_t index = (uint16_t)CURRENT_OPERAND0_16();
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
PyObject *owner_o = PyStackRef_AsPyObjectBorrow(owner);
gh-115999: Specialize `STORE_ATTR` in free-threaded builds. (gh-127838) * Add `_PyDictKeys_StringLookupSplit` which does locking on dict keys and use in place of `_PyDictKeys_StringLookup`. * Change `_PyObject_TryGetInstanceAttribute` to use that function in the case of split keys. * Add `unicodekeys_lookup_split` helper which allows code sharing between `_Py_dict_lookup` and `_PyDictKeys_StringLookupSplit`. * Fix locking for `STORE_ATTR_INSTANCE_VALUE`. Create `_GUARD_TYPE_VERSION_AND_LOCK` uop so that object stays locked and `tp_version_tag` cannot change. * Pass `tp_version_tag` to `specialize_dict_access()`, ensuring the version we store on the cache is the correct one (in case of it changing during the specalize analysis). * Split `analyze_descriptor` into `analyze_descriptor_load` and `analyze_descriptor_store` since those don't share much logic. Add `descriptor_is_class` helper function. * In `specialize_dict_access`, double check `_PyObject_GetManagedDict()` in case we race and dict was materialized before the lock. * Avoid borrowed references in `_Py_Specialize_StoreAttr()`. * Use `specialize()` and `unspecialize()` helpers. * Add unit tests to ensure specializing happens as expected in FT builds. * Add unit tests to attempt to trigger data races (useful for running under TSAN). * Add `has_split_table` function to `_testinternalcapi`.
2024-12-19 10:21:17 -08:00
if (!LOCK_OBJECT(owner_o)) {
UOP_STAT_INC(uopcode, miss);
GH-142718: JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. (GH-142762) JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. * Make sure that stack is in correct state at side exits with TOS cached values * Simplify choice of cached items for side exits
2025-12-15 19:01:29 +00:00
_tos_cache1 = owner;
_tos_cache0 = value;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(2);
gh-115999: Specialize `STORE_ATTR` in free-threaded builds. (gh-127838) * Add `_PyDictKeys_StringLookupSplit` which does locking on dict keys and use in place of `_PyDictKeys_StringLookup`. * Change `_PyObject_TryGetInstanceAttribute` to use that function in the case of split keys. * Add `unicodekeys_lookup_split` helper which allows code sharing between `_Py_dict_lookup` and `_PyDictKeys_StringLookupSplit`. * Fix locking for `STORE_ATTR_INSTANCE_VALUE`. Create `_GUARD_TYPE_VERSION_AND_LOCK` uop so that object stays locked and `tp_version_tag` cannot change. * Pass `tp_version_tag` to `specialize_dict_access()`, ensuring the version we store on the cache is the correct one (in case of it changing during the specalize analysis). * Split `analyze_descriptor` into `analyze_descriptor_load` and `analyze_descriptor_store` since those don't share much logic. Add `descriptor_is_class` helper function. * In `specialize_dict_access`, double check `_PyObject_GetManagedDict()` in case we race and dict was materialized before the lock. * Avoid borrowed references in `_Py_Specialize_StoreAttr()`. * Use `specialize()` and `unspecialize()` helpers. * Add unit tests to ensure specializing happens as expected in FT builds. * Add unit tests to attempt to trigger data races (useful for running under TSAN). * Add `has_split_table` function to `_testinternalcapi`.
2024-12-19 10:21:17 -08:00
JUMP_TO_JUMP_TARGET();
}
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
char *addr = (char *)owner_o + index;
gh-104909: Split some more insts into ops (#109943) These are the most popular specializations of `LOAD_ATTR` and `STORE_ATTR` that weren't already viable uops: * Split LOAD_ATTR_METHOD_WITH_VALUES * Split LOAD_ATTR_METHOD_NO_DICT * Split LOAD_ATTR_SLOT * Split STORE_ATTR_SLOT * Split STORE_ATTR_INSTANCE_VALUE Also: * Add `-v` flag to code generator which prints a list of non-viable uops (easter-egg: it can print execution counts -- see source) * Double _Py_UOP_MAX_TRACE_LENGTH to 128 I had dropped one of the DEOPT_IF() calls! :-(
2023-09-27 15:27:44 -07:00
STAT_INC(STORE_ATTR, hit);
PyObject *old_value = *(PyObject **)addr;
gh-115999: Specialize `STORE_ATTR` in free-threaded builds. (gh-127838) * Add `_PyDictKeys_StringLookupSplit` which does locking on dict keys and use in place of `_PyDictKeys_StringLookup`. * Change `_PyObject_TryGetInstanceAttribute` to use that function in the case of split keys. * Add `unicodekeys_lookup_split` helper which allows code sharing between `_Py_dict_lookup` and `_PyDictKeys_StringLookupSplit`. * Fix locking for `STORE_ATTR_INSTANCE_VALUE`. Create `_GUARD_TYPE_VERSION_AND_LOCK` uop so that object stays locked and `tp_version_tag` cannot change. * Pass `tp_version_tag` to `specialize_dict_access()`, ensuring the version we store on the cache is the correct one (in case of it changing during the specalize analysis). * Split `analyze_descriptor` into `analyze_descriptor_load` and `analyze_descriptor_store` since those don't share much logic. Add `descriptor_is_class` helper function. * In `specialize_dict_access`, double check `_PyObject_GetManagedDict()` in case we race and dict was materialized before the lock. * Avoid borrowed references in `_Py_Specialize_StoreAttr()`. * Use `specialize()` and `unspecialize()` helpers. * Add unit tests to ensure specializing happens as expected in FT builds. * Add unit tests to attempt to trigger data races (useful for running under TSAN). * Add `has_split_table` function to `_testinternalcapi`.
2024-12-19 10:21:17 -08:00
FT_ATOMIC_STORE_PTR_RELEASE(*(PyObject **)addr, PyStackRef_AsPyObjectSteal(value));
UNLOCK_OBJECT(owner_o);
GH-134584: Remove redundant refcount from `_STORE_ATTR_SLOT` (#142729)
2025-12-15 15:18:44 -08:00
o = owner;
stack_pointer[0] = o;
stack_pointer += 1;
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
GH-128682: Convert explicit loops closing arrays into `DECREF_INPUTS`. (GH-128822) * Mark Py_DECREF and Py_XDECREF as escaping * Remove explicit loops for clearing array inputs
2025-01-14 15:08:56 +00:00
_PyFrame_SetStackPointer(frame, stack_pointer);
Py_XDECREF(old_value);
stack_pointer = _PyFrame_GetStackPointer(frame);
GH-134584: Remove redundant refcount from `_STORE_ATTR_SLOT` (#142729)
2025-12-15 15:18:44 -08:00
_tos_cache0 = o;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_tos_cache1 = PyStackRef_ZERO_BITS;
_tos_cache2 = PyStackRef_ZERO_BITS;
GH-134584: Remove redundant refcount from `_STORE_ATTR_SLOT` (#142729)
2025-12-15 15:18:44 -08:00
SET_CURRENT_CACHED_VALUES(1);
stack_pointer += -1;
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-104909: Split some more insts into ops (#109943) These are the most popular specializations of `LOAD_ATTR` and `STORE_ATTR` that weren't already viable uops: * Split LOAD_ATTR_METHOD_WITH_VALUES * Split LOAD_ATTR_METHOD_NO_DICT * Split LOAD_ATTR_SLOT * Split STORE_ATTR_SLOT * Split STORE_ATTR_INSTANCE_VALUE Also: * Add `-v` flag to code generator which prints a list of non-viable uops (easter-egg: it can print execution counts -- see source) * Double _Py_UOP_MAX_TRACE_LENGTH to 128 I had dropped one of the DEOPT_IF() calls! :-(
2023-09-27 15:27:44 -07:00
break;
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
case _COMPARE_OP_r21: {
CHECK_CURRENT_CACHED_VALUES(2);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
_PyStackRef right;
_PyStackRef left;
_PyStackRef res;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef _stack_item_0 = _tos_cache0;
_PyStackRef _stack_item_1 = _tos_cache1;
GH-111485: Factor out tier 2 code generation from the rest of the interpreter code generator (GH-112968)
2023-12-12 12:12:17 +00:00
oparg = CURRENT_OPARG();
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
right = _stack_item_1;
left = _stack_item_0;
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
PyObject *left_o = PyStackRef_AsPyObjectBorrow(left);
PyObject *right_o = PyStackRef_AsPyObjectBorrow(right);
gh-104584: Allow unspecialized instructions in superblocks (#106497) This adds several of unspecialized opcodes to superblocks: TO_BOOL, BINARY_SUBSCR, STORE_SUBSCR, UNPACK_SEQUENCE, LOAD_GLOBAL, LOAD_ATTR, COMPARE_OP, BINARY_OP. While we may not want that eventually, for now this helps finding bugs. There is a rudimentary test checking for UNPACK_SEQUENCE. Once we're ready to undo this, that would be simple: just replace the call to variable_used_unspecialized with a call to variable_used (as shown in a comment). Or add individual opcdes to FORBIDDEN_NAMES_IN_UOPS.
2023-07-07 11:03:27 -07:00
assert((oparg >> 5) <= Py_GE);
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
stack_pointer[0] = left;
stack_pointer[1] = right;
stack_pointer += 2;
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
GH-119866: Spill the stack around escaping calls. (GH-124392) * Spill the evaluation around escaping calls in the generated interpreter and JIT. * The code generator tracks live, cached values so they can be saved to memory when needed. * Spills the stack pointer around escaping calls, so that the exact stack is visible to the cycle GC.
2024-10-07 14:56:39 +01:00
_PyFrame_SetStackPointer(frame, stack_pointer);
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
PyObject *res_o = PyObject_RichCompare(left_o, right_o, oparg >> 5);
GH-128682: Account for escapes in `DECREF_INPUTS` (GH-129953) * Handle escapes in DECREF_INPUTS * Mark a few more functions as escaping * Replace DECREF_INPUTS with PyStackRef_CLOSE where possible
2025-02-12 17:44:59 +00:00
_PyStackRef tmp = right;
right = PyStackRef_NULL;
stack_pointer[-1] = right;
PyStackRef_CLOSE(tmp);
tmp = left;
left = PyStackRef_NULL;
stack_pointer[-2] = left;
PyStackRef_CLOSE(tmp);
GH-119866: Spill the stack around escaping calls. (GH-124392) * Spill the evaluation around escaping calls in the generated interpreter and JIT. * The code generator tracks live, cached values so they can be saved to memory when needed. * Spills the stack pointer around escaping calls, so that the exact stack is visible to the cycle GC.
2024-10-07 14:56:39 +01:00
stack_pointer = _PyFrame_GetStackPointer(frame);
GH-128682: Account for escapes in `DECREF_INPUTS` (GH-129953) * Handle escapes in DECREF_INPUTS * Mark a few more functions as escaping * Replace DECREF_INPUTS with PyStackRef_CLOSE where possible
2025-02-12 17:44:59 +00:00
stack_pointer += -2;
GH-141794: Limit size of generated machine code. (GH-142228) * Factor out bodies of the largest uops, to reduce jit code size. * Factor out common assert, also reducing jit code size. * Limit size of jitted code for a single executor to 1MB.
2025-12-03 17:43:35 +00:00
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
GH-128682: Make `PyStackRef_CLOSE` escaping. (GH-129404)
2025-02-03 12:41:32 +00:00
if (res_o == NULL) {
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(0);
GH-128682: Make `PyStackRef_CLOSE` escaping. (GH-129404)
2025-02-03 12:41:32 +00:00
JUMP_TO_ERROR();
}
gh-104584: Allow unspecialized instructions in superblocks (#106497) This adds several of unspecialized opcodes to superblocks: TO_BOOL, BINARY_SUBSCR, STORE_SUBSCR, UNPACK_SEQUENCE, LOAD_GLOBAL, LOAD_ATTR, COMPARE_OP, BINARY_OP. While we may not want that eventually, for now this helps finding bugs. There is a rudimentary test checking for UNPACK_SEQUENCE. Once we're ready to undo this, that would be simple: just replace the call to variable_used_unspecialized with a call to variable_used (as shown in a comment). Or add individual opcdes to FORBIDDEN_NAMES_IN_UOPS.
2023-07-07 11:03:27 -07:00
if (oparg & 16) {
GH-119866: Spill the stack around escaping calls. (GH-124392) * Spill the evaluation around escaping calls in the generated interpreter and JIT. * The code generator tracks live, cached values so they can be saved to memory when needed. * Spills the stack pointer around escaping calls, so that the exact stack is visible to the cycle GC.
2024-10-07 14:56:39 +01:00
_PyFrame_SetStackPointer(frame, stack_pointer);
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
int res_bool = PyObject_IsTrue(res_o);
Py_DECREF(res_o);
GH-128682: Convert explicit loops closing arrays into `DECREF_INPUTS`. (GH-128822) * Mark Py_DECREF and Py_XDECREF as escaping * Remove explicit loops for clearing array inputs
2025-01-14 15:08:56 +00:00
stack_pointer = _PyFrame_GetStackPointer(frame);
GH-128682: Make `PyStackRef_CLOSE` escaping. (GH-129404)
2025-02-03 12:41:32 +00:00
if (res_bool < 0) {
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(0);
GH-128682: Make `PyStackRef_CLOSE` escaping. (GH-129404)
2025-02-03 12:41:32 +00:00
JUMP_TO_ERROR();
}
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
res = res_bool ? PyStackRef_True : PyStackRef_False;
}
else {
res = PyStackRef_FromPyObjectSteal(res_o);
gh-104584: Allow unspecialized instructions in superblocks (#106497) This adds several of unspecialized opcodes to superblocks: TO_BOOL, BINARY_SUBSCR, STORE_SUBSCR, UNPACK_SEQUENCE, LOAD_GLOBAL, LOAD_ATTR, COMPARE_OP, BINARY_OP. While we may not want that eventually, for now this helps finding bugs. There is a rudimentary test checking for UNPACK_SEQUENCE. Once we're ready to undo this, that would be simple: just replace the call to variable_used_unspecialized with a call to variable_used (as shown in a comment). Or add individual opcdes to FORBIDDEN_NAMES_IN_UOPS.
2023-07-07 11:03:27 -07:00
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_tos_cache0 = res;
_tos_cache1 = PyStackRef_ZERO_BITS;
_tos_cache2 = PyStackRef_ZERO_BITS;
SET_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
gh-134584: Eliminate redundant refcounting from `_COMPARE_OP_X` (GH-143186)
2025-12-27 00:12:28 +08:00
case _COMPARE_OP_FLOAT_r03: {
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
CHECK_CURRENT_CACHED_VALUES(0);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef right;
_PyStackRef left;
_PyStackRef res;
gh-134584: Eliminate redundant refcounting from `_COMPARE_OP_X` (GH-143186)
2025-12-27 00:12:28 +08:00
_PyStackRef l;
_PyStackRef r;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
oparg = CURRENT_OPARG();
right = stack_pointer[-1];
left = stack_pointer[-2];
PyObject *left_o = PyStackRef_AsPyObjectBorrow(left);
PyObject *right_o = PyStackRef_AsPyObjectBorrow(right);
STAT_INC(COMPARE_OP, hit);
double dleft = PyFloat_AS_DOUBLE(left_o);
double dright = PyFloat_AS_DOUBLE(right_o);
int sign_ish = COMPARISON_BIT(dleft, dright);
gh-134584: Eliminate redundant refcounting from `_COMPARE_OP_X` (GH-143186)
2025-12-27 00:12:28 +08:00
l = left;
r = right;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
res = (sign_ish & oparg) ? PyStackRef_True : PyStackRef_False;
gh-134584: Eliminate redundant refcounting from `_COMPARE_OP_X` (GH-143186)
2025-12-27 00:12:28 +08:00
_tos_cache2 = r;
_tos_cache1 = l;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_tos_cache0 = res;
gh-134584: Eliminate redundant refcounting from `_COMPARE_OP_X` (GH-143186)
2025-12-27 00:12:28 +08:00
SET_CURRENT_CACHED_VALUES(3);
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
stack_pointer += -2;
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
gh-134584: Eliminate redundant refcounting from `_COMPARE_OP_X` (GH-143186)
2025-12-27 00:12:28 +08:00
case _COMPARE_OP_FLOAT_r13: {
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
CHECK_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef right;
_PyStackRef left;
_PyStackRef res;
gh-134584: Eliminate redundant refcounting from `_COMPARE_OP_X` (GH-143186)
2025-12-27 00:12:28 +08:00
_PyStackRef l;
_PyStackRef r;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef _stack_item_0 = _tos_cache0;
oparg = CURRENT_OPARG();
right = _stack_item_0;
left = stack_pointer[-1];
PyObject *left_o = PyStackRef_AsPyObjectBorrow(left);
PyObject *right_o = PyStackRef_AsPyObjectBorrow(right);
STAT_INC(COMPARE_OP, hit);
double dleft = PyFloat_AS_DOUBLE(left_o);
double dright = PyFloat_AS_DOUBLE(right_o);
int sign_ish = COMPARISON_BIT(dleft, dright);
gh-134584: Eliminate redundant refcounting from `_COMPARE_OP_X` (GH-143186)
2025-12-27 00:12:28 +08:00
l = left;
r = right;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
res = (sign_ish & oparg) ? PyStackRef_True : PyStackRef_False;
gh-134584: Eliminate redundant refcounting from `_COMPARE_OP_X` (GH-143186)
2025-12-27 00:12:28 +08:00
_tos_cache2 = r;
_tos_cache1 = l;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_tos_cache0 = res;
gh-134584: Eliminate redundant refcounting from `_COMPARE_OP_X` (GH-143186)
2025-12-27 00:12:28 +08:00
SET_CURRENT_CACHED_VALUES(3);
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
stack_pointer += -1;
GH-141794: Limit size of generated machine code. (GH-142228) * Factor out bodies of the largest uops, to reduce jit code size. * Factor out common assert, also reducing jit code size. * Limit size of jitted code for a single executor to 1MB.
2025-12-03 17:43:35 +00:00
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
gh-134584: Eliminate redundant refcounting from `_COMPARE_OP_X` (GH-143186)
2025-12-27 00:12:28 +08:00
case _COMPARE_OP_FLOAT_r23: {
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
CHECK_CURRENT_CACHED_VALUES(2);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef right;
_PyStackRef left;
_PyStackRef res;
gh-134584: Eliminate redundant refcounting from `_COMPARE_OP_X` (GH-143186)
2025-12-27 00:12:28 +08:00
_PyStackRef l;
_PyStackRef r;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef _stack_item_0 = _tos_cache0;
_PyStackRef _stack_item_1 = _tos_cache1;
oparg = CURRENT_OPARG();
right = _stack_item_1;
left = _stack_item_0;
PyObject *left_o = PyStackRef_AsPyObjectBorrow(left);
PyObject *right_o = PyStackRef_AsPyObjectBorrow(right);
STAT_INC(COMPARE_OP, hit);
double dleft = PyFloat_AS_DOUBLE(left_o);
double dright = PyFloat_AS_DOUBLE(right_o);
int sign_ish = COMPARISON_BIT(dleft, dright);
gh-134584: Eliminate redundant refcounting from `_COMPARE_OP_X` (GH-143186)
2025-12-27 00:12:28 +08:00
l = left;
r = right;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
res = (sign_ish & oparg) ? PyStackRef_True : PyStackRef_False;
gh-134584: Eliminate redundant refcounting from `_COMPARE_OP_X` (GH-143186)
2025-12-27 00:12:28 +08:00
_tos_cache2 = r;
_tos_cache1 = l;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_tos_cache0 = res;
gh-134584: Eliminate redundant refcounting from `_COMPARE_OP_X` (GH-143186)
2025-12-27 00:12:28 +08:00
SET_CURRENT_CACHED_VALUES(3);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-104584: Baby steps towards generating and executing traces (#105924) Added a new, experimental, tracing optimizer and interpreter (a.k.a. "tier 2"). This currently pessimizes, so don't use yet -- this is infrastructure so we can experiment with optimizing passes. To enable it, pass ``-Xuops`` or set ``PYTHONUOPS=1``. To get debug output, set ``PYTHONUOPSDEBUG=N`` where ``N`` is a debug level (0-4, where 0 is no debug output and 4 is excessively verbose). All of this code is likely to change dramatically before the 3.13 feature freeze. But this is a first step.
2023-06-26 19:02:57 -07:00
break;
}
gh-134584: JIT: Eliminate redundant refcount ops for X_INT (GH-142765)
2025-12-16 06:22:18 +08:00
case _COMPARE_OP_INT_r23: {
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
CHECK_CURRENT_CACHED_VALUES(2);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
_PyStackRef right;
_PyStackRef left;
_PyStackRef res;
gh-134584: JIT: Eliminate redundant refcount ops for X_INT (GH-142765)
2025-12-16 06:22:18 +08:00
_PyStackRef l;
_PyStackRef r;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef _stack_item_0 = _tos_cache0;
_PyStackRef _stack_item_1 = _tos_cache1;
GH-111485: Factor out tier 2 code generation from the rest of the interpreter code generator (GH-112968)
2023-12-12 12:12:17 +00:00
oparg = CURRENT_OPARG();
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
right = _stack_item_1;
left = _stack_item_0;
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
PyObject *left_o = PyStackRef_AsPyObjectBorrow(left);
PyObject *right_o = PyStackRef_AsPyObjectBorrow(right);
GH-135379: Specialize int operations for compact ints only (GH-135668)
2025-06-19 11:10:29 +01:00
assert(_PyLong_IsCompact((PyLongObject *)left_o));
assert(_PyLong_IsCompact((PyLongObject *)right_o));
gh-104584: Baby steps towards generating and executing traces (#105924) Added a new, experimental, tracing optimizer and interpreter (a.k.a. "tier 2"). This currently pessimizes, so don't use yet -- this is infrastructure so we can experiment with optimizing passes. To enable it, pass ``-Xuops`` or set ``PYTHONUOPS=1``. To get debug output, set ``PYTHONUOPSDEBUG=N`` where ``N`` is a debug level (0-4, where 0 is no debug output and 4 is excessively verbose). All of this code is likely to change dramatically before the 3.13 feature freeze. But this is a first step.
2023-06-26 19:02:57 -07:00
STAT_INC(COMPARE_OP, hit);
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
assert(_PyLong_DigitCount((PyLongObject *)left_o) <= 1 &&
_PyLong_DigitCount((PyLongObject *)right_o) <= 1);
Py_ssize_t ileft = _PyLong_CompactValue((PyLongObject *)left_o);
Py_ssize_t iright = _PyLong_CompactValue((PyLongObject *)right_o);
gh-104584: Baby steps towards generating and executing traces (#105924) Added a new, experimental, tracing optimizer and interpreter (a.k.a. "tier 2"). This currently pessimizes, so don't use yet -- this is infrastructure so we can experiment with optimizing passes. To enable it, pass ``-Xuops`` or set ``PYTHONUOPS=1``. To get debug output, set ``PYTHONUOPSDEBUG=N`` where ``N`` is a debug level (0-4, where 0 is no debug output and 4 is excessively verbose). All of this code is likely to change dramatically before the 3.13 feature freeze. But this is a first step.
2023-06-26 19:02:57 -07:00
int sign_ish = COMPARISON_BIT(ileft, iright);
gh-134584: JIT: Eliminate redundant refcount ops for X_INT (GH-142765)
2025-12-16 06:22:18 +08:00
l = left;
r = right;
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
res = (sign_ish & oparg) ? PyStackRef_True : PyStackRef_False;
gh-134584: JIT: Eliminate redundant refcount ops for X_INT (GH-142765)
2025-12-16 06:22:18 +08:00
_tos_cache2 = r;
_tos_cache1 = l;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_tos_cache0 = res;
gh-134584: JIT: Eliminate redundant refcount ops for X_INT (GH-142765)
2025-12-16 06:22:18 +08:00
SET_CURRENT_CACHED_VALUES(3);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-104584: Baby steps towards generating and executing traces (#105924) Added a new, experimental, tracing optimizer and interpreter (a.k.a. "tier 2"). This currently pessimizes, so don't use yet -- this is infrastructure so we can experiment with optimizing passes. To enable it, pass ``-Xuops`` or set ``PYTHONUOPS=1``. To get debug output, set ``PYTHONUOPSDEBUG=N`` where ``N`` is a debug level (0-4, where 0 is no debug output and 4 is excessively verbose). All of this code is likely to change dramatically before the 3.13 feature freeze. But this is a first step.
2023-06-26 19:02:57 -07:00
break;
}
gh-134584: Eliminate redundant refcounting from `_COMPARE_OP_X` (GH-143186)
2025-12-27 00:12:28 +08:00
case _COMPARE_OP_STR_r23: {
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
CHECK_CURRENT_CACHED_VALUES(2);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
_PyStackRef right;
_PyStackRef left;
_PyStackRef res;
gh-134584: Eliminate redundant refcounting from `_COMPARE_OP_X` (GH-143186)
2025-12-27 00:12:28 +08:00
_PyStackRef l;
_PyStackRef r;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef _stack_item_0 = _tos_cache0;
_PyStackRef _stack_item_1 = _tos_cache1;
GH-111485: Factor out tier 2 code generation from the rest of the interpreter code generator (GH-112968)
2023-12-12 12:12:17 +00:00
oparg = CURRENT_OPARG();
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
right = _stack_item_1;
left = _stack_item_0;
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
PyObject *left_o = PyStackRef_AsPyObjectBorrow(left);
PyObject *right_o = PyStackRef_AsPyObjectBorrow(right);
gh-104584: Baby steps towards generating and executing traces (#105924) Added a new, experimental, tracing optimizer and interpreter (a.k.a. "tier 2"). This currently pessimizes, so don't use yet -- this is infrastructure so we can experiment with optimizing passes. To enable it, pass ``-Xuops`` or set ``PYTHONUOPS=1``. To get debug output, set ``PYTHONUOPSDEBUG=N`` where ``N`` is a debug level (0-4, where 0 is no debug output and 4 is excessively verbose). All of this code is likely to change dramatically before the 3.13 feature freeze. But this is a first step.
2023-06-26 19:02:57 -07:00
STAT_INC(COMPARE_OP, hit);
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
int eq = _PyUnicode_Equal(left_o, right_o);
GH-106008: Make implicit boolean conversions explicit (GH-106003)
2023-06-29 13:49:54 -07:00
assert((oparg >> 5) == Py_EQ || (oparg >> 5) == Py_NE);
gh-134584: Eliminate redundant refcounting from `_COMPARE_OP_X` (GH-143186)
2025-12-27 00:12:28 +08:00
l = left;
r = right;
gh-104584: Baby steps towards generating and executing traces (#105924) Added a new, experimental, tracing optimizer and interpreter (a.k.a. "tier 2"). This currently pessimizes, so don't use yet -- this is infrastructure so we can experiment with optimizing passes. To enable it, pass ``-Xuops`` or set ``PYTHONUOPS=1``. To get debug output, set ``PYTHONUOPSDEBUG=N`` where ``N`` is a debug level (0-4, where 0 is no debug output and 4 is excessively verbose). All of this code is likely to change dramatically before the 3.13 feature freeze. But this is a first step.
2023-06-26 19:02:57 -07:00
assert(eq == 0 || eq == 1);
assert((oparg & 0xf) == COMPARISON_NOT_EQUALS || (oparg & 0xf) == COMPARISON_EQUALS);
assert(COMPARISON_NOT_EQUALS + 1 == COMPARISON_EQUALS);
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
res = ((COMPARISON_NOT_EQUALS + eq) & oparg) ? PyStackRef_True : PyStackRef_False;
gh-134584: Eliminate redundant refcounting from `_COMPARE_OP_X` (GH-143186)
2025-12-27 00:12:28 +08:00
_tos_cache2 = r;
_tos_cache1 = l;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_tos_cache0 = res;
gh-134584: Eliminate redundant refcounting from `_COMPARE_OP_X` (GH-143186)
2025-12-27 00:12:28 +08:00
SET_CURRENT_CACHED_VALUES(3);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-104584: Baby steps towards generating and executing traces (#105924) Added a new, experimental, tracing optimizer and interpreter (a.k.a. "tier 2"). This currently pessimizes, so don't use yet -- this is infrastructure so we can experiment with optimizing passes. To enable it, pass ``-Xuops`` or set ``PYTHONUOPS=1``. To get debug output, set ``PYTHONUOPSDEBUG=N`` where ``N`` is a debug level (0-4, where 0 is no debug output and 4 is excessively verbose). All of this code is likely to change dramatically before the 3.13 feature freeze. But this is a first step.
2023-06-26 19:02:57 -07:00
break;
}
gh-134584: Eliminate redundant refcounting from `IS_OP` (GH-143171) Eliminate redundant refcounting from IS_OP
2025-12-27 04:30:02 +08:00
case _IS_OP_r03: {
CHECK_CURRENT_CACHED_VALUES(0);
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
_PyStackRef right;
_PyStackRef left;
_PyStackRef b;
_PyStackRef l;
_PyStackRef r;
oparg = CURRENT_OPARG();
right = stack_pointer[-1];
left = stack_pointer[-2];
int res = Py_Is(PyStackRef_AsPyObjectBorrow(left), PyStackRef_AsPyObjectBorrow(right)) ^ oparg;
b = res ? PyStackRef_True : PyStackRef_False;
l = left;
r = right;
_tos_cache2 = r;
_tos_cache1 = l;
_tos_cache0 = b;
SET_CURRENT_CACHED_VALUES(3);
stack_pointer += -2;
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
break;
}
case _IS_OP_r13: {
CHECK_CURRENT_CACHED_VALUES(1);
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
_PyStackRef right;
_PyStackRef left;
_PyStackRef b;
_PyStackRef l;
_PyStackRef r;
_PyStackRef _stack_item_0 = _tos_cache0;
oparg = CURRENT_OPARG();
right = _stack_item_0;
left = stack_pointer[-1];
int res = Py_Is(PyStackRef_AsPyObjectBorrow(left), PyStackRef_AsPyObjectBorrow(right)) ^ oparg;
b = res ? PyStackRef_True : PyStackRef_False;
l = left;
r = right;
_tos_cache2 = r;
_tos_cache1 = l;
_tos_cache0 = b;
SET_CURRENT_CACHED_VALUES(3);
stack_pointer += -1;
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
break;
}
case _IS_OP_r23: {
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
CHECK_CURRENT_CACHED_VALUES(2);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
_PyStackRef right;
_PyStackRef left;
_PyStackRef b;
gh-134584: Eliminate redundant refcounting from `IS_OP` (GH-143171) Eliminate redundant refcounting from IS_OP
2025-12-27 04:30:02 +08:00
_PyStackRef l;
_PyStackRef r;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef _stack_item_0 = _tos_cache0;
_PyStackRef _stack_item_1 = _tos_cache1;
GH-111485: Factor out tier 2 code generation from the rest of the interpreter code generator (GH-112968)
2023-12-12 12:12:17 +00:00
oparg = CURRENT_OPARG();
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
right = _stack_item_1;
left = _stack_item_0;
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
int res = Py_Is(PyStackRef_AsPyObjectBorrow(left), PyStackRef_AsPyObjectBorrow(right)) ^ oparg;
b = res ? PyStackRef_True : PyStackRef_False;
gh-134584: Eliminate redundant refcounting from `IS_OP` (GH-143171) Eliminate redundant refcounting from IS_OP
2025-12-27 04:30:02 +08:00
l = left;
r = right;
_tos_cache2 = r;
_tos_cache1 = l;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_tos_cache0 = b;
gh-134584: Eliminate redundant refcounting from `IS_OP` (GH-143171) Eliminate redundant refcounting from IS_OP
2025-12-27 04:30:02 +08:00
SET_CURRENT_CACHED_VALUES(3);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-104584: Baby steps towards generating and executing traces (#105924) Added a new, experimental, tracing optimizer and interpreter (a.k.a. "tier 2"). This currently pessimizes, so don't use yet -- this is infrastructure so we can experiment with optimizing passes. To enable it, pass ``-Xuops`` or set ``PYTHONUOPS=1``. To get debug output, set ``PYTHONUOPSDEBUG=N`` where ``N`` is a debug level (0-4, where 0 is no debug output and 4 is excessively verbose). All of this code is likely to change dramatically before the 3.13 feature freeze. But this is a first step.
2023-06-26 19:02:57 -07:00
break;
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
case _CONTAINS_OP_r21: {
CHECK_CURRENT_CACHED_VALUES(2);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
_PyStackRef right;
_PyStackRef left;
_PyStackRef b;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef _stack_item_0 = _tos_cache0;
_PyStackRef _stack_item_1 = _tos_cache1;
GH-111485: Factor out tier 2 code generation from the rest of the interpreter code generator (GH-112968)
2023-12-12 12:12:17 +00:00
oparg = CURRENT_OPARG();
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
right = _stack_item_1;
left = _stack_item_0;
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
PyObject *left_o = PyStackRef_AsPyObjectBorrow(left);
PyObject *right_o = PyStackRef_AsPyObjectBorrow(right);
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
stack_pointer[0] = left;
stack_pointer[1] = right;
stack_pointer += 2;
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
GH-119866: Spill the stack around escaping calls. (GH-124392) * Spill the evaluation around escaping calls in the generated interpreter and JIT. * The code generator tracks live, cached values so they can be saved to memory when needed. * Spills the stack pointer around escaping calls, so that the exact stack is visible to the cycle GC.
2024-10-07 14:56:39 +01:00
_PyFrame_SetStackPointer(frame, stack_pointer);
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
int res = PySequence_Contains(right_o, left_o);
GH-128682: Account for escapes in `DECREF_INPUTS` (GH-129953) * Handle escapes in DECREF_INPUTS * Mark a few more functions as escaping * Replace DECREF_INPUTS with PyStackRef_CLOSE where possible
2025-02-12 17:44:59 +00:00
_PyStackRef tmp = right;
right = PyStackRef_NULL;
stack_pointer[-1] = right;
PyStackRef_CLOSE(tmp);
tmp = left;
left = PyStackRef_NULL;
stack_pointer[-2] = left;
PyStackRef_CLOSE(tmp);
GH-119866: Spill the stack around escaping calls. (GH-124392) * Spill the evaluation around escaping calls in the generated interpreter and JIT. * The code generator tracks live, cached values so they can be saved to memory when needed. * Spills the stack pointer around escaping calls, so that the exact stack is visible to the cycle GC.
2024-10-07 14:56:39 +01:00
stack_pointer = _PyFrame_GetStackPointer(frame);
GH-128682: Account for escapes in `DECREF_INPUTS` (GH-129953) * Handle escapes in DECREF_INPUTS * Mark a few more functions as escaping * Replace DECREF_INPUTS with PyStackRef_CLOSE where possible
2025-02-12 17:44:59 +00:00
stack_pointer += -2;
GH-141794: Limit size of generated machine code. (GH-142228) * Factor out bodies of the largest uops, to reduce jit code size. * Factor out common assert, also reducing jit code size. * Limit size of jitted code for a single executor to 1MB.
2025-12-03 17:43:35 +00:00
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
GH-128682: Make `PyStackRef_CLOSE` escaping. (GH-129404)
2025-02-03 12:41:32 +00:00
if (res < 0) {
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(0);
GH-128682: Make `PyStackRef_CLOSE` escaping. (GH-129404)
2025-02-03 12:41:32 +00:00
JUMP_TO_ERROR();
}
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
b = (res ^ oparg) ? PyStackRef_True : PyStackRef_False;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_tos_cache0 = b;
_tos_cache1 = PyStackRef_ZERO_BITS;
_tos_cache2 = PyStackRef_ZERO_BITS;
SET_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-104584: Baby steps towards generating and executing traces (#105924) Added a new, experimental, tracing optimizer and interpreter (a.k.a. "tier 2"). This currently pessimizes, so don't use yet -- this is infrastructure so we can experiment with optimizing passes. To enable it, pass ``-Xuops`` or set ``PYTHONUOPS=1``. To get debug output, set ``PYTHONUOPSDEBUG=N`` where ``N`` is a debug level (0-4, where 0 is no debug output and 4 is excessively verbose). All of this code is likely to change dramatically before the 3.13 feature freeze. But this is a first step.
2023-06-26 19:02:57 -07:00
break;
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
case _GUARD_TOS_ANY_SET_r01: {
CHECK_CURRENT_CACHED_VALUES(0);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-131798: Remove JIT guards for dict, frozenset, list, set, and tuple (GH-132289)
2025-04-09 14:32:21 -07:00
_PyStackRef tos;
tos = stack_pointer[-1];
PyObject *o = PyStackRef_AsPyObjectBorrow(tos);
if (!PyAnySet_CheckExact(o)) {
UOP_STAT_INC(uopcode, miss);
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(0);
JUMP_TO_JUMP_TARGET();
}
_tos_cache0 = tos;
SET_CURRENT_CACHED_VALUES(1);
stack_pointer += -1;
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _GUARD_TOS_ANY_SET_r11: {
CHECK_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef tos;
_PyStackRef _stack_item_0 = _tos_cache0;
tos = _stack_item_0;
PyObject *o = PyStackRef_AsPyObjectBorrow(tos);
if (!PyAnySet_CheckExact(o)) {
UOP_STAT_INC(uopcode, miss);
GH-142718: JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. (GH-142762) JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. * Make sure that stack is in correct state at side exits with TOS cached values * Simplify choice of cached items for side exits
2025-12-15 19:01:29 +00:00
_tos_cache0 = tos;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(1);
JUMP_TO_JUMP_TARGET();
}
_tos_cache0 = tos;
SET_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _GUARD_TOS_ANY_SET_r22: {
CHECK_CURRENT_CACHED_VALUES(2);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef tos;
_PyStackRef _stack_item_0 = _tos_cache0;
_PyStackRef _stack_item_1 = _tos_cache1;
tos = _stack_item_1;
PyObject *o = PyStackRef_AsPyObjectBorrow(tos);
if (!PyAnySet_CheckExact(o)) {
UOP_STAT_INC(uopcode, miss);
GH-142718: JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. (GH-142762) JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. * Make sure that stack is in correct state at side exits with TOS cached values * Simplify choice of cached items for side exits
2025-12-15 19:01:29 +00:00
_tos_cache1 = tos;
_tos_cache0 = _stack_item_0;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(2);
JUMP_TO_JUMP_TARGET();
}
_tos_cache1 = tos;
_tos_cache0 = _stack_item_0;
SET_CURRENT_CACHED_VALUES(2);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _GUARD_TOS_ANY_SET_r33: {
CHECK_CURRENT_CACHED_VALUES(3);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef tos;
_PyStackRef _stack_item_0 = _tos_cache0;
_PyStackRef _stack_item_1 = _tos_cache1;
_PyStackRef _stack_item_2 = _tos_cache2;
tos = _stack_item_2;
PyObject *o = PyStackRef_AsPyObjectBorrow(tos);
if (!PyAnySet_CheckExact(o)) {
UOP_STAT_INC(uopcode, miss);
GH-142718: JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. (GH-142762) JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. * Make sure that stack is in correct state at side exits with TOS cached values * Simplify choice of cached items for side exits
2025-12-15 19:01:29 +00:00
_tos_cache2 = tos;
_tos_cache1 = _stack_item_1;
_tos_cache0 = _stack_item_0;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(3);
GH-131798: Remove JIT guards for dict, frozenset, list, set, and tuple (GH-132289)
2025-04-09 14:32:21 -07:00
JUMP_TO_JUMP_TARGET();
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_tos_cache2 = tos;
_tos_cache1 = _stack_item_1;
_tos_cache0 = _stack_item_0;
SET_CURRENT_CACHED_VALUES(3);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-131798: Remove JIT guards for dict, frozenset, list, set, and tuple (GH-132289)
2025-04-09 14:32:21 -07:00
break;
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
case _CONTAINS_OP_SET_r21: {
CHECK_CURRENT_CACHED_VALUES(2);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
_PyStackRef right;
_PyStackRef left;
_PyStackRef b;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef _stack_item_0 = _tos_cache0;
_PyStackRef _stack_item_1 = _tos_cache1;
gh-116381: Specialize CONTAINS_OP (GH-116385) * Specialize CONTAINS_OP * 📜🤖 Added by blurb_it. * Add PyAPI_FUNC for JIT --------- Co-authored-by: blurb-it[bot] <43283697+blurb-it[bot]@users.noreply.github.com>
2024-03-07 03:30:11 +08:00
oparg = CURRENT_OPARG();
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
right = _stack_item_1;
left = _stack_item_0;
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
PyObject *left_o = PyStackRef_AsPyObjectBorrow(left);
PyObject *right_o = PyStackRef_AsPyObjectBorrow(right);
GH-131798: Remove JIT guards for dict, frozenset, list, set, and tuple (GH-132289)
2025-04-09 14:32:21 -07:00
assert(PyAnySet_CheckExact(right_o));
gh-116381: Remove bad specializations, add fail stats (GH-116464) * Remove bad specializations, add fail stats
2024-03-08 00:21:21 +08:00
STAT_INC(CONTAINS_OP, hit);
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
stack_pointer[0] = left;
stack_pointer[1] = right;
stack_pointer += 2;
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
GH-119866: Spill the stack around escaping calls. (GH-124392) * Spill the evaluation around escaping calls in the generated interpreter and JIT. * The code generator tracks live, cached values so they can be saved to memory when needed. * Spills the stack pointer around escaping calls, so that the exact stack is visible to the cycle GC.
2024-10-07 14:56:39 +01:00
_PyFrame_SetStackPointer(frame, stack_pointer);
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
int res = _PySet_Contains((PySetObject *)right_o, left_o);
GH-128682: Account for escapes in `DECREF_INPUTS` (GH-129953) * Handle escapes in DECREF_INPUTS * Mark a few more functions as escaping * Replace DECREF_INPUTS with PyStackRef_CLOSE where possible
2025-02-12 17:44:59 +00:00
_PyStackRef tmp = right;
right = PyStackRef_NULL;
stack_pointer[-1] = right;
PyStackRef_CLOSE(tmp);
tmp = left;
left = PyStackRef_NULL;
stack_pointer[-2] = left;
PyStackRef_CLOSE(tmp);
GH-119866: Spill the stack around escaping calls. (GH-124392) * Spill the evaluation around escaping calls in the generated interpreter and JIT. * The code generator tracks live, cached values so they can be saved to memory when needed. * Spills the stack pointer around escaping calls, so that the exact stack is visible to the cycle GC.
2024-10-07 14:56:39 +01:00
stack_pointer = _PyFrame_GetStackPointer(frame);
GH-128682: Account for escapes in `DECREF_INPUTS` (GH-129953) * Handle escapes in DECREF_INPUTS * Mark a few more functions as escaping * Replace DECREF_INPUTS with PyStackRef_CLOSE where possible
2025-02-12 17:44:59 +00:00
stack_pointer += -2;
GH-141794: Limit size of generated machine code. (GH-142228) * Factor out bodies of the largest uops, to reduce jit code size. * Factor out common assert, also reducing jit code size. * Limit size of jitted code for a single executor to 1MB.
2025-12-03 17:43:35 +00:00
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
GH-128682: Make `PyStackRef_CLOSE` escaping. (GH-129404)
2025-02-03 12:41:32 +00:00
if (res < 0) {
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(0);
GH-128682: Make `PyStackRef_CLOSE` escaping. (GH-129404)
2025-02-03 12:41:32 +00:00
JUMP_TO_ERROR();
}
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
b = (res ^ oparg) ? PyStackRef_True : PyStackRef_False;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_tos_cache0 = b;
_tos_cache1 = PyStackRef_ZERO_BITS;
_tos_cache2 = PyStackRef_ZERO_BITS;
SET_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-116381: Specialize CONTAINS_OP (GH-116385) * Specialize CONTAINS_OP * 📜🤖 Added by blurb_it. * Add PyAPI_FUNC for JIT --------- Co-authored-by: blurb-it[bot] <43283697+blurb-it[bot]@users.noreply.github.com>
2024-03-07 03:30:11 +08:00
break;
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
case _CONTAINS_OP_DICT_r21: {
CHECK_CURRENT_CACHED_VALUES(2);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
_PyStackRef right;
_PyStackRef left;
_PyStackRef b;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef _stack_item_0 = _tos_cache0;
_PyStackRef _stack_item_1 = _tos_cache1;
gh-116381: Specialize CONTAINS_OP (GH-116385) * Specialize CONTAINS_OP * 📜🤖 Added by blurb_it. * Add PyAPI_FUNC for JIT --------- Co-authored-by: blurb-it[bot] <43283697+blurb-it[bot]@users.noreply.github.com>
2024-03-07 03:30:11 +08:00
oparg = CURRENT_OPARG();
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
right = _stack_item_1;
left = _stack_item_0;
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
PyObject *left_o = PyStackRef_AsPyObjectBorrow(left);
PyObject *right_o = PyStackRef_AsPyObjectBorrow(right);
GH-131798: Remove JIT guards for dict, frozenset, list, set, and tuple (GH-132289)
2025-04-09 14:32:21 -07:00
assert(PyDict_CheckExact(right_o));
gh-116381: Remove bad specializations, add fail stats (GH-116464) * Remove bad specializations, add fail stats
2024-03-08 00:21:21 +08:00
STAT_INC(CONTAINS_OP, hit);
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
stack_pointer[0] = left;
stack_pointer[1] = right;
stack_pointer += 2;
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
GH-119866: Spill the stack around escaping calls. (GH-124392) * Spill the evaluation around escaping calls in the generated interpreter and JIT. * The code generator tracks live, cached values so they can be saved to memory when needed. * Spills the stack pointer around escaping calls, so that the exact stack is visible to the cycle GC.
2024-10-07 14:56:39 +01:00
_PyFrame_SetStackPointer(frame, stack_pointer);
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
int res = PyDict_Contains(right_o, left_o);
GH-128682: Account for escapes in `DECREF_INPUTS` (GH-129953) * Handle escapes in DECREF_INPUTS * Mark a few more functions as escaping * Replace DECREF_INPUTS with PyStackRef_CLOSE where possible
2025-02-12 17:44:59 +00:00
_PyStackRef tmp = right;
right = PyStackRef_NULL;
stack_pointer[-1] = right;
PyStackRef_CLOSE(tmp);
tmp = left;
left = PyStackRef_NULL;
stack_pointer[-2] = left;
PyStackRef_CLOSE(tmp);
GH-119866: Spill the stack around escaping calls. (GH-124392) * Spill the evaluation around escaping calls in the generated interpreter and JIT. * The code generator tracks live, cached values so they can be saved to memory when needed. * Spills the stack pointer around escaping calls, so that the exact stack is visible to the cycle GC.
2024-10-07 14:56:39 +01:00
stack_pointer = _PyFrame_GetStackPointer(frame);
GH-128682: Account for escapes in `DECREF_INPUTS` (GH-129953) * Handle escapes in DECREF_INPUTS * Mark a few more functions as escaping * Replace DECREF_INPUTS with PyStackRef_CLOSE where possible
2025-02-12 17:44:59 +00:00
stack_pointer += -2;
GH-141794: Limit size of generated machine code. (GH-142228) * Factor out bodies of the largest uops, to reduce jit code size. * Factor out common assert, also reducing jit code size. * Limit size of jitted code for a single executor to 1MB.
2025-12-03 17:43:35 +00:00
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
GH-128682: Make `PyStackRef_CLOSE` escaping. (GH-129404)
2025-02-03 12:41:32 +00:00
if (res < 0) {
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(0);
GH-128682: Make `PyStackRef_CLOSE` escaping. (GH-129404)
2025-02-03 12:41:32 +00:00
JUMP_TO_ERROR();
}
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
b = (res ^ oparg) ? PyStackRef_True : PyStackRef_False;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_tos_cache0 = b;
_tos_cache1 = PyStackRef_ZERO_BITS;
_tos_cache2 = PyStackRef_ZERO_BITS;
SET_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-116381: Specialize CONTAINS_OP (GH-116385) * Specialize CONTAINS_OP * 📜🤖 Added by blurb_it. * Add PyAPI_FUNC for JIT --------- Co-authored-by: blurb-it[bot] <43283697+blurb-it[bot]@users.noreply.github.com>
2024-03-07 03:30:11 +08:00
break;
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
case _CHECK_EG_MATCH_r22: {
CHECK_CURRENT_CACHED_VALUES(2);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
_PyStackRef match_type_st;
_PyStackRef exc_value_st;
_PyStackRef rest;
_PyStackRef match;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef _stack_item_0 = _tos_cache0;
_PyStackRef _stack_item_1 = _tos_cache1;
match_type_st = _stack_item_1;
exc_value_st = _stack_item_0;
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
PyObject *exc_value = PyStackRef_AsPyObjectBorrow(exc_value_st);
PyObject *match_type = PyStackRef_AsPyObjectBorrow(match_type_st);
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
stack_pointer[0] = exc_value_st;
stack_pointer[1] = match_type_st;
stack_pointer += 2;
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
GH-119866: Spill the stack around escaping calls. (GH-124392) * Spill the evaluation around escaping calls in the generated interpreter and JIT. * The code generator tracks live, cached values so they can be saved to memory when needed. * Spills the stack pointer around escaping calls, so that the exact stack is visible to the cycle GC.
2024-10-07 14:56:39 +01:00
_PyFrame_SetStackPointer(frame, stack_pointer);
GH-120024: Refactor code a bit so that escaping calls can be wrapped in spill code in code generator (GH-122693)
2024-08-06 08:40:39 +01:00
int err = _PyEval_CheckExceptStarTypeValid(tstate, match_type);
GH-119866: Spill the stack around escaping calls. (GH-124392) * Spill the evaluation around escaping calls in the generated interpreter and JIT. * The code generator tracks live, cached values so they can be saved to memory when needed. * Spills the stack pointer around escaping calls, so that the exact stack is visible to the cycle GC.
2024-10-07 14:56:39 +01:00
stack_pointer = _PyFrame_GetStackPointer(frame);
GH-120024: Refactor code a bit so that escaping calls can be wrapped in spill code in code generator (GH-122693)
2024-08-06 08:40:39 +01:00
if (err < 0) {
GH-128682: Account for escapes in `DECREF_INPUTS` (GH-129953) * Handle escapes in DECREF_INPUTS * Mark a few more functions as escaping * Replace DECREF_INPUTS with PyStackRef_CLOSE where possible
2025-02-12 17:44:59 +00:00
_PyFrame_SetStackPointer(frame, stack_pointer);
_PyStackRef tmp = match_type_st;
match_type_st = PyStackRef_NULL;
stack_pointer[-1] = match_type_st;
PyStackRef_CLOSE(tmp);
tmp = exc_value_st;
exc_value_st = PyStackRef_NULL;
stack_pointer[-2] = exc_value_st;
PyStackRef_CLOSE(tmp);
stack_pointer = _PyFrame_GetStackPointer(frame);
GH-128682: Make `PyStackRef_CLOSE` escaping. (GH-129404)
2025-02-03 12:41:32 +00:00
stack_pointer += -2;
GH-141794: Limit size of generated machine code. (GH-142228) * Factor out bodies of the largest uops, to reduce jit code size. * Factor out common assert, also reducing jit code size. * Limit size of jitted code for a single executor to 1MB.
2025-12-03 17:43:35 +00:00
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(0);
GH-128682: Make `PyStackRef_CLOSE` escaping. (GH-129404)
2025-02-03 12:41:32 +00:00
JUMP_TO_ERROR();
gh-104584: Baby steps towards generating and executing traces (#105924) Added a new, experimental, tracing optimizer and interpreter (a.k.a. "tier 2"). This currently pessimizes, so don't use yet -- this is infrastructure so we can experiment with optimizing passes. To enable it, pass ``-Xuops`` or set ``PYTHONUOPS=1``. To get debug output, set ``PYTHONUOPSDEBUG=N`` where ``N`` is a debug level (0-4, where 0 is no debug output and 4 is excessively verbose). All of this code is likely to change dramatically before the 3.13 feature freeze. But this is a first step.
2023-06-26 19:02:57 -07:00
}
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
PyObject *match_o = NULL;
PyObject *rest_o = NULL;
GH-119866: Spill the stack around escaping calls. (GH-124392) * Spill the evaluation around escaping calls in the generated interpreter and JIT. * The code generator tracks live, cached values so they can be saved to memory when needed. * Spills the stack pointer around escaping calls, so that the exact stack is visible to the cycle GC.
2024-10-07 14:56:39 +01:00
_PyFrame_SetStackPointer(frame, stack_pointer);
gh-128799: Add frame of except* to traceback when wrapping a naked exception (#128971)
2025-01-25 13:00:23 +00:00
int res = _PyEval_ExceptionGroupMatch(frame, exc_value, match_type,
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
&match_o, &rest_o);
GH-128682: Account for escapes in `DECREF_INPUTS` (GH-129953) * Handle escapes in DECREF_INPUTS * Mark a few more functions as escaping * Replace DECREF_INPUTS with PyStackRef_CLOSE where possible
2025-02-12 17:44:59 +00:00
_PyStackRef tmp = match_type_st;
match_type_st = PyStackRef_NULL;
stack_pointer[-1] = match_type_st;
PyStackRef_CLOSE(tmp);
tmp = exc_value_st;
exc_value_st = PyStackRef_NULL;
stack_pointer[-2] = exc_value_st;
PyStackRef_CLOSE(tmp);
GH-119866: Spill the stack around escaping calls. (GH-124392) * Spill the evaluation around escaping calls in the generated interpreter and JIT. * The code generator tracks live, cached values so they can be saved to memory when needed. * Spills the stack pointer around escaping calls, so that the exact stack is visible to the cycle GC.
2024-10-07 14:56:39 +01:00
stack_pointer = _PyFrame_GetStackPointer(frame);
GH-128682: Account for escapes in `DECREF_INPUTS` (GH-129953) * Handle escapes in DECREF_INPUTS * Mark a few more functions as escaping * Replace DECREF_INPUTS with PyStackRef_CLOSE where possible
2025-02-12 17:44:59 +00:00
stack_pointer += -2;
GH-141794: Limit size of generated machine code. (GH-142228) * Factor out bodies of the largest uops, to reduce jit code size. * Factor out common assert, also reducing jit code size. * Limit size of jitted code for a single executor to 1MB.
2025-12-03 17:43:35 +00:00
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
GH-128682: Make `PyStackRef_CLOSE` escaping. (GH-129404)
2025-02-03 12:41:32 +00:00
if (res < 0) {
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(0);
GH-128682: Make `PyStackRef_CLOSE` escaping. (GH-129404)
2025-02-03 12:41:32 +00:00
JUMP_TO_ERROR();
}
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
assert((match_o == NULL) == (rest_o == NULL));
GH-128682: Make `PyStackRef_CLOSE` escaping. (GH-129404)
2025-02-03 12:41:32 +00:00
if (match_o == NULL) {
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(0);
GH-128682: Make `PyStackRef_CLOSE` escaping. (GH-129404)
2025-02-03 12:41:32 +00:00
JUMP_TO_ERROR();
}
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
if (!Py_IsNone(match_o)) {
GH-119866: Spill the stack around escaping calls. (GH-124392) * Spill the evaluation around escaping calls in the generated interpreter and JIT. * The code generator tracks live, cached values so they can be saved to memory when needed. * Spills the stack pointer around escaping calls, so that the exact stack is visible to the cycle GC.
2024-10-07 14:56:39 +01:00
_PyFrame_SetStackPointer(frame, stack_pointer);
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
PyErr_SetHandledException(match_o);
GH-119866: Spill the stack around escaping calls. (GH-124392) * Spill the evaluation around escaping calls in the generated interpreter and JIT. * The code generator tracks live, cached values so they can be saved to memory when needed. * Spills the stack pointer around escaping calls, so that the exact stack is visible to the cycle GC.
2024-10-07 14:56:39 +01:00
stack_pointer = _PyFrame_GetStackPointer(frame);
gh-104584: Baby steps towards generating and executing traces (#105924) Added a new, experimental, tracing optimizer and interpreter (a.k.a. "tier 2"). This currently pessimizes, so don't use yet -- this is infrastructure so we can experiment with optimizing passes. To enable it, pass ``-Xuops`` or set ``PYTHONUOPS=1``. To get debug output, set ``PYTHONUOPSDEBUG=N`` where ``N`` is a debug level (0-4, where 0 is no debug output and 4 is excessively verbose). All of this code is likely to change dramatically before the 3.13 feature freeze. But this is a first step.
2023-06-26 19:02:57 -07:00
}
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
rest = PyStackRef_FromPyObjectSteal(rest_o);
match = PyStackRef_FromPyObjectSteal(match_o);
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_tos_cache1 = match;
_tos_cache0 = rest;
_tos_cache2 = PyStackRef_ZERO_BITS;
SET_CURRENT_CACHED_VALUES(2);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-104584: Baby steps towards generating and executing traces (#105924) Added a new, experimental, tracing optimizer and interpreter (a.k.a. "tier 2"). This currently pessimizes, so don't use yet -- this is infrastructure so we can experiment with optimizing passes. To enable it, pass ``-Xuops`` or set ``PYTHONUOPS=1``. To get debug output, set ``PYTHONUOPSDEBUG=N`` where ``N`` is a debug level (0-4, where 0 is no debug output and 4 is excessively verbose). All of this code is likely to change dramatically before the 3.13 feature freeze. But this is a first step.
2023-06-26 19:02:57 -07:00
break;
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
case _CHECK_EXC_MATCH_r22: {
CHECK_CURRENT_CACHED_VALUES(2);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
_PyStackRef right;
_PyStackRef left;
_PyStackRef b;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef _stack_item_0 = _tos_cache0;
_PyStackRef _stack_item_1 = _tos_cache1;
right = _stack_item_1;
left = _stack_item_0;
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
PyObject *left_o = PyStackRef_AsPyObjectBorrow(left);
PyObject *right_o = PyStackRef_AsPyObjectBorrow(right);
assert(PyExceptionInstance_Check(left_o));
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
stack_pointer[0] = left;
stack_pointer[1] = right;
stack_pointer += 2;
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
GH-119866: Spill the stack around escaping calls. (GH-124392) * Spill the evaluation around escaping calls in the generated interpreter and JIT. * The code generator tracks live, cached values so they can be saved to memory when needed. * Spills the stack pointer around escaping calls, so that the exact stack is visible to the cycle GC.
2024-10-07 14:56:39 +01:00
_PyFrame_SetStackPointer(frame, stack_pointer);
GH-120024: Move three more escaping calls out of conditional statements (GH-122734)
2024-08-06 14:14:52 +01:00
int err = _PyEval_CheckExceptTypeValid(tstate, right_o);
GH-119866: Spill the stack around escaping calls. (GH-124392) * Spill the evaluation around escaping calls in the generated interpreter and JIT. * The code generator tracks live, cached values so they can be saved to memory when needed. * Spills the stack pointer around escaping calls, so that the exact stack is visible to the cycle GC.
2024-10-07 14:56:39 +01:00
stack_pointer = _PyFrame_GetStackPointer(frame);
GH-120024: Move three more escaping calls out of conditional statements (GH-122734)
2024-08-06 14:14:52 +01:00
if (err < 0) {
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(0);
GH-128682: Make `PyStackRef_CLOSE` escaping. (GH-129404)
2025-02-03 12:41:32 +00:00
JUMP_TO_ERROR();
gh-104584: Baby steps towards generating and executing traces (#105924) Added a new, experimental, tracing optimizer and interpreter (a.k.a. "tier 2"). This currently pessimizes, so don't use yet -- this is infrastructure so we can experiment with optimizing passes. To enable it, pass ``-Xuops`` or set ``PYTHONUOPS=1``. To get debug output, set ``PYTHONUOPSDEBUG=N`` where ``N`` is a debug level (0-4, where 0 is no debug output and 4 is excessively verbose). All of this code is likely to change dramatically before the 3.13 feature freeze. But this is a first step.
2023-06-26 19:02:57 -07:00
}
GH-119866: Spill the stack around escaping calls. (GH-124392) * Spill the evaluation around escaping calls in the generated interpreter and JIT. * The code generator tracks live, cached values so they can be saved to memory when needed. * Spills the stack pointer around escaping calls, so that the exact stack is visible to the cycle GC.
2024-10-07 14:56:39 +01:00
_PyFrame_SetStackPointer(frame, stack_pointer);
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
int res = PyErr_GivenExceptionMatches(left_o, right_o);
GH-119866: Spill the stack around escaping calls. (GH-124392) * Spill the evaluation around escaping calls in the generated interpreter and JIT. * The code generator tracks live, cached values so they can be saved to memory when needed. * Spills the stack pointer around escaping calls, so that the exact stack is visible to the cycle GC.
2024-10-07 14:56:39 +01:00
stack_pointer = _PyFrame_GetStackPointer(frame);
GH-128682: Account for escapes in `DECREF_INPUTS` (GH-129953) * Handle escapes in DECREF_INPUTS * Mark a few more functions as escaping * Replace DECREF_INPUTS with PyStackRef_CLOSE where possible
2025-02-12 17:44:59 +00:00
stack_pointer += -1;
GH-141794: Limit size of generated machine code. (GH-142228) * Factor out bodies of the largest uops, to reduce jit code size. * Factor out common assert, also reducing jit code size. * Limit size of jitted code for a single executor to 1MB.
2025-12-03 17:43:35 +00:00
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
GH-128682: Account for escapes in `DECREF_INPUTS` (GH-129953) * Handle escapes in DECREF_INPUTS * Mark a few more functions as escaping * Replace DECREF_INPUTS with PyStackRef_CLOSE where possible
2025-02-12 17:44:59 +00:00
_PyFrame_SetStackPointer(frame, stack_pointer);
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
PyStackRef_CLOSE(right);
GH-128682: Account for escapes in `DECREF_INPUTS` (GH-129953) * Handle escapes in DECREF_INPUTS * Mark a few more functions as escaping * Replace DECREF_INPUTS with PyStackRef_CLOSE where possible
2025-02-12 17:44:59 +00:00
stack_pointer = _PyFrame_GetStackPointer(frame);
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
b = res ? PyStackRef_True : PyStackRef_False;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_tos_cache1 = b;
_tos_cache0 = left;
_tos_cache2 = PyStackRef_ZERO_BITS;
SET_CURRENT_CACHED_VALUES(2);
stack_pointer += -1;
GH-141794: Limit size of generated machine code. (GH-142228) * Factor out bodies of the largest uops, to reduce jit code size. * Factor out common assert, also reducing jit code size. * Limit size of jitted code for a single executor to 1MB.
2025-12-03 17:43:35 +00:00
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-104584: Baby steps towards generating and executing traces (#105924) Added a new, experimental, tracing optimizer and interpreter (a.k.a. "tier 2"). This currently pessimizes, so don't use yet -- this is infrastructure so we can experiment with optimizing passes. To enable it, pass ``-Xuops`` or set ``PYTHONUOPS=1``. To get debug output, set ``PYTHONUOPSDEBUG=N`` where ``N`` is a debug level (0-4, where 0 is no debug output and 4 is excessively verbose). All of this code is likely to change dramatically before the 3.13 feature freeze. But this is a first step.
2023-06-26 19:02:57 -07:00
break;
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
case _IMPORT_NAME_r21: {
CHECK_CURRENT_CACHED_VALUES(2);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-118093: Add tier two support to several instructions (GH-121884)
2024-07-18 14:24:58 -07:00
_PyStackRef fromlist;
_PyStackRef level;
_PyStackRef res;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef _stack_item_0 = _tos_cache0;
_PyStackRef _stack_item_1 = _tos_cache1;
GH-118093: Add tier two support to several instructions (GH-121884)
2024-07-18 14:24:58 -07:00
oparg = CURRENT_OPARG();
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
fromlist = _stack_item_1;
level = _stack_item_0;
GH-118093: Add tier two support to several instructions (GH-121884)
2024-07-18 14:24:58 -07:00
PyObject *name = GETITEM(FRAME_CO_NAMES, oparg);
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
stack_pointer[0] = level;
stack_pointer[1] = fromlist;
stack_pointer += 2;
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
GH-119866: Spill the stack around escaping calls. (GH-124392) * Spill the evaluation around escaping calls in the generated interpreter and JIT. * The code generator tracks live, cached values so they can be saved to memory when needed. * Spills the stack pointer around escaping calls, so that the exact stack is visible to the cycle GC.
2024-10-07 14:56:39 +01:00
_PyFrame_SetStackPointer(frame, stack_pointer);
GH-118093: Add tier two support to several instructions (GH-121884)
2024-07-18 14:24:58 -07:00
PyObject *res_o = _PyEval_ImportName(tstate, frame, name,
PyStackRef_AsPyObjectBorrow(fromlist),
PyStackRef_AsPyObjectBorrow(level));
GH-128682: Account for escapes in `DECREF_INPUTS` (GH-129953) * Handle escapes in DECREF_INPUTS * Mark a few more functions as escaping * Replace DECREF_INPUTS with PyStackRef_CLOSE where possible
2025-02-12 17:44:59 +00:00
_PyStackRef tmp = fromlist;
fromlist = PyStackRef_NULL;
stack_pointer[-1] = fromlist;
PyStackRef_CLOSE(tmp);
tmp = level;
level = PyStackRef_NULL;
stack_pointer[-2] = level;
PyStackRef_CLOSE(tmp);
GH-119866: Spill the stack around escaping calls. (GH-124392) * Spill the evaluation around escaping calls in the generated interpreter and JIT. * The code generator tracks live, cached values so they can be saved to memory when needed. * Spills the stack pointer around escaping calls, so that the exact stack is visible to the cycle GC.
2024-10-07 14:56:39 +01:00
stack_pointer = _PyFrame_GetStackPointer(frame);
GH-128682: Account for escapes in `DECREF_INPUTS` (GH-129953) * Handle escapes in DECREF_INPUTS * Mark a few more functions as escaping * Replace DECREF_INPUTS with PyStackRef_CLOSE where possible
2025-02-12 17:44:59 +00:00
stack_pointer += -2;
GH-141794: Limit size of generated machine code. (GH-142228) * Factor out bodies of the largest uops, to reduce jit code size. * Factor out common assert, also reducing jit code size. * Limit size of jitted code for a single executor to 1MB.
2025-12-03 17:43:35 +00:00
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
GH-128682: Make `PyStackRef_CLOSE` escaping. (GH-129404)
2025-02-03 12:41:32 +00:00
if (res_o == NULL) {
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(0);
GH-128682: Make `PyStackRef_CLOSE` escaping. (GH-129404)
2025-02-03 12:41:32 +00:00
JUMP_TO_ERROR();
}
GH-118093: Add tier two support to several instructions (GH-121884)
2024-07-18 14:24:58 -07:00
res = PyStackRef_FromPyObjectSteal(res_o);
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_tos_cache0 = res;
_tos_cache1 = PyStackRef_ZERO_BITS;
_tos_cache2 = PyStackRef_ZERO_BITS;
SET_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-118093: Add tier two support to several instructions (GH-121884)
2024-07-18 14:24:58 -07:00
break;
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
case _IMPORT_FROM_r12: {
CHECK_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-118093: Add tier two support to several instructions (GH-121884)
2024-07-18 14:24:58 -07:00
_PyStackRef from;
_PyStackRef res;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef _stack_item_0 = _tos_cache0;
GH-118093: Add tier two support to several instructions (GH-121884)
2024-07-18 14:24:58 -07:00
oparg = CURRENT_OPARG();
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
from = _stack_item_0;
GH-118093: Add tier two support to several instructions (GH-121884)
2024-07-18 14:24:58 -07:00
PyObject *name = GETITEM(FRAME_CO_NAMES, oparg);
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
stack_pointer[0] = from;
stack_pointer += 1;
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
GH-119866: Spill the stack around escaping calls. (GH-124392) * Spill the evaluation around escaping calls in the generated interpreter and JIT. * The code generator tracks live, cached values so they can be saved to memory when needed. * Spills the stack pointer around escaping calls, so that the exact stack is visible to the cycle GC.
2024-10-07 14:56:39 +01:00
_PyFrame_SetStackPointer(frame, stack_pointer);
GH-118093: Add tier two support to several instructions (GH-121884)
2024-07-18 14:24:58 -07:00
PyObject *res_o = _PyEval_ImportFrom(tstate, PyStackRef_AsPyObjectBorrow(from), name);
GH-119866: Spill the stack around escaping calls. (GH-124392) * Spill the evaluation around escaping calls in the generated interpreter and JIT. * The code generator tracks live, cached values so they can be saved to memory when needed. * Spills the stack pointer around escaping calls, so that the exact stack is visible to the cycle GC.
2024-10-07 14:56:39 +01:00
stack_pointer = _PyFrame_GetStackPointer(frame);
GH-128682: Make `PyStackRef_CLOSE` escaping. (GH-129404)
2025-02-03 12:41:32 +00:00
if (res_o == NULL) {
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(0);
GH-128682: Make `PyStackRef_CLOSE` escaping. (GH-129404)
2025-02-03 12:41:32 +00:00
JUMP_TO_ERROR();
}
GH-118093: Add tier two support to several instructions (GH-121884)
2024-07-18 14:24:58 -07:00
res = PyStackRef_FromPyObjectSteal(res_o);
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_tos_cache1 = res;
_tos_cache0 = from;
_tos_cache2 = PyStackRef_ZERO_BITS;
SET_CURRENT_CACHED_VALUES(2);
stack_pointer += -1;
GH-141794: Limit size of generated machine code. (GH-142228) * Factor out bodies of the largest uops, to reduce jit code size. * Factor out common assert, also reducing jit code size. * Limit size of jitted code for a single executor to 1MB.
2025-12-03 17:43:35 +00:00
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-118093: Add tier two support to several instructions (GH-121884)
2024-07-18 14:24:58 -07:00
break;
}
GH-116422: Tier2 hot/cold splitting (GH-116813) Splits the "cold" path, deopts and exits, from the "hot" path, reducing the size of most jitted instructions, at the cost of slower exits.
2024-03-26 09:35:11 +00:00
/* _POP_JUMP_IF_FALSE is not a viable micro-op for tier 2 because it is replaced */
GH-111485: Factor out tier 2 code generation from the rest of the interpreter code generator (GH-112968)
2023-12-12 12:12:17 +00:00
GH-116422: Tier2 hot/cold splitting (GH-116813) Splits the "cold" path, deopts and exits, from the "hot" path, reducing the size of most jitted instructions, at the cost of slower exits.
2024-03-26 09:35:11 +00:00
/* _POP_JUMP_IF_TRUE is not a viable micro-op for tier 2 because it is replaced */
GH-111485: Factor out tier 2 code generation from the rest of the interpreter code generator (GH-112968)
2023-12-12 12:12:17 +00:00
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
case _IS_NONE_r11: {
CHECK_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
_PyStackRef value;
_PyStackRef b;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef _stack_item_0 = _tos_cache0;
value = _stack_item_0;
gh-127022: Simplify `PyStackRef_FromPyObjectSteal` (#127024) This gets rid of the immortal check in `PyStackRef_FromPyObjectSteal()`. Overall, this improves performance about 2% in the free threading build. This also renames `PyStackRef_Is()` to `PyStackRef_IsExactly()` because the macro requires that the tag bits of the arguments match, which is only true in certain special cases.
2024-11-22 17:55:33 +00:00
if (PyStackRef_IsNone(value)) {
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
b = PyStackRef_True;
GH-106529: Define POP_JUMP_IF_NONE in terms of POP_JUMP_IF_TRUE (GH-106599)
2023-07-11 11:33:59 +01:00
}
else {
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
b = PyStackRef_False;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
stack_pointer[0] = value;
stack_pointer += 1;
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
GH-128682: Account for escapes in `DECREF_INPUTS` (GH-129953) * Handle escapes in DECREF_INPUTS * Mark a few more functions as escaping * Replace DECREF_INPUTS with PyStackRef_CLOSE where possible
2025-02-12 17:44:59 +00:00
_PyFrame_SetStackPointer(frame, stack_pointer);
_PyStackRef tmp = value;
value = b;
stack_pointer[-1] = value;
PyStackRef_CLOSE(tmp);
stack_pointer = _PyFrame_GetStackPointer(frame);
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
stack_pointer += -1;
GH-106529: Define POP_JUMP_IF_NONE in terms of POP_JUMP_IF_TRUE (GH-106599)
2023-07-11 11:33:59 +01:00
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_tos_cache0 = b;
_tos_cache1 = PyStackRef_ZERO_BITS;
_tos_cache2 = PyStackRef_ZERO_BITS;
SET_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-106529: Define POP_JUMP_IF_NONE in terms of POP_JUMP_IF_TRUE (GH-106599)
2023-07-11 11:33:59 +01:00
break;
}
gh-139109: A new tracing JIT compiler frontend for CPython (GH-140310) This PR changes the current JIT model from trace projection to trace recording. Benchmarking: better pyperformance (about 1.7% overall) geomean versus current https://raw.githubusercontent.com/facebookexperimental/free-threading-benchmarking/refs/heads/main/results/bm-20251108-3.15.0a1%2B-7e2bc1d-JIT/bm-20251108-vultr-x86_64-Fidget%252dSpinner-tracing_jit-3.15.0a1%2B-7e2bc1d-vs-base.svg, 100% faster Richards on the most improved benchmark versus the current JIT. Slowdown of about 10-15% on the worst benchmark versus the current JIT. **Note: the fastest version isn't the one merged, as it relies on fixing bugs in the specializing interpreter, which is left to another PR**. The speedup in the merged version is about 1.1%. https://raw.githubusercontent.com/facebookexperimental/free-threading-benchmarking/refs/heads/main/results/bm-20251112-3.15.0a1%2B-f8a764a-JIT/bm-20251112-vultr-x86_64-Fidget%252dSpinner-tracing_jit-3.15.0a1%2B-f8a764a-vs-base.svg Stats: 50% more uops executed, 30% more traces entered the last time we ran them. It also suggests our trace lengths for a real trace recording JIT are too short, as a lot of trace too long aborts https://github.com/facebookexperimental/free-threading-benchmarking/blob/main/results/bm-20251023-3.15.0a1%2B-eb73378-CLANG%2CJIT/bm-20251023-vultr-x86_64-Fidget%252dSpinner-tracing_jit-3.15.0a1%2B-eb73378-pystats-vs-base.md . This new JIT frontend is already able to record/execute significantly more instructions than the previous JIT frontend. In this PR, we are now able to record through custom dunders, simple object creation, generators, etc. None of these were done by the old JIT frontend. Some custom dunders uops were discovered to be broken as part of this work gh-140277 The optimizer stack space check is disabled, as it's no longer valid to deal with underflow. Pros: * Ignoring the generated tracer code as it's automatically created, this is only additional 1k lines of code. The maintenance burden is handled by the DSL and code generator. * `optimizer.c` is now significantly simpler, as we don't have to do strange things to recover the bytecode from a trace. * The new JIT frontend is able to handle a lot more control-flow than the old one. * Tracing is very low overhead. We use the tail calling interpreter/computed goto interpreter to switch between tracing mode and non-tracing mode. I call this mechanism dual dispatch, as we have two dispatch tables dispatching to each other. Specialization is still enabled while tracing. * Better handling of polymorphism. We leverage the specializing interpreter for this. Cons: * (For now) requires tail calling interpreter or computed gotos. This means no Windows JIT for now :(. Not to fret, tail calling is coming soon to Windows though https://github.com/python/cpython/pull/139962 Design: * After each instruction, the `record_previous_inst` function/label is executed. This does as the name suggests. * The tracing interpreter lowers bytecode to uops directly so that it can obtain "fresh" values at the point of lowering. * The tracing version behaves nearly identical to the normal interpreter, in fact it even has specialization! This allows it to run without much of a slowdown when tracing. The actual cost of tracing is only a function call and writes to memory. * The tracing interpreter uses the specializing interpreter's deopt to naturally form the side exit chains. This allows it to side exit chain effectively, without repeating much code. We force a re-specializing when tracing a deopt. * The tracing interpreter can even handle goto errors/exceptions, but I chose to disable them for now as it's not tested. * Because we do not share interpreter dispatch, there is should be no significant slowdown to the original specializing interpreter on tailcall and computed got with JIT disabled. With JIT enabled, there might be a slowdown in the form of the JIT trying to trace. * Things that could have dynamic instruction pointer effects are guarded on. The guard deopts to a new instruction --- `_DYNAMIC_EXIT`.
2025-11-14 02:08:32 +08:00
/* _JUMP_BACKWARD_NO_INTERRUPT is not a viable micro-op for tier 2 because it is replaced */
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
case _GET_LEN_r12: {
CHECK_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
_PyStackRef obj;
_PyStackRef len;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef _stack_item_0 = _tos_cache0;
obj = _stack_item_0;
stack_pointer[0] = obj;
stack_pointer += 1;
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
GH-119866: Spill the stack around escaping calls. (GH-124392) * Spill the evaluation around escaping calls in the generated interpreter and JIT. * The code generator tracks live, cached values so they can be saved to memory when needed. * Spills the stack pointer around escaping calls, so that the exact stack is visible to the cycle GC.
2024-10-07 14:56:39 +01:00
_PyFrame_SetStackPointer(frame, stack_pointer);
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
Py_ssize_t len_i = PyObject_Length(PyStackRef_AsPyObjectBorrow(obj));
GH-119866: Spill the stack around escaping calls. (GH-124392) * Spill the evaluation around escaping calls in the generated interpreter and JIT. * The code generator tracks live, cached values so they can be saved to memory when needed. * Spills the stack pointer around escaping calls, so that the exact stack is visible to the cycle GC.
2024-10-07 14:56:39 +01:00
stack_pointer = _PyFrame_GetStackPointer(frame);
GH-128682: Make `PyStackRef_CLOSE` escaping. (GH-129404)
2025-02-03 12:41:32 +00:00
if (len_i < 0) {
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(0);
GH-128682: Make `PyStackRef_CLOSE` escaping. (GH-129404)
2025-02-03 12:41:32 +00:00
JUMP_TO_ERROR();
}
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
PyObject *len_o = PyLong_FromSsize_t(len_i);
GH-128682: Make `PyStackRef_CLOSE` escaping. (GH-129404)
2025-02-03 12:41:32 +00:00
if (len_o == NULL) {
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(0);
GH-128682: Make `PyStackRef_CLOSE` escaping. (GH-129404)
2025-02-03 12:41:32 +00:00
JUMP_TO_ERROR();
}
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
len = PyStackRef_FromPyObjectSteal(len_o);
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_tos_cache1 = len;
_tos_cache0 = obj;
_tos_cache2 = PyStackRef_ZERO_BITS;
SET_CURRENT_CACHED_VALUES(2);
stack_pointer += -1;
GH-141794: Limit size of generated machine code. (GH-142228) * Factor out bodies of the largest uops, to reduce jit code size. * Factor out common assert, also reducing jit code size. * Limit size of jitted code for a single executor to 1MB.
2025-12-03 17:43:35 +00:00
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-104584: Baby steps towards generating and executing traces (#105924) Added a new, experimental, tracing optimizer and interpreter (a.k.a. "tier 2"). This currently pessimizes, so don't use yet -- this is infrastructure so we can experiment with optimizing passes. To enable it, pass ``-Xuops`` or set ``PYTHONUOPS=1``. To get debug output, set ``PYTHONUOPSDEBUG=N`` where ``N`` is a debug level (0-4, where 0 is no debug output and 4 is excessively verbose). All of this code is likely to change dramatically before the 3.13 feature freeze. But this is a first step.
2023-06-26 19:02:57 -07:00
break;
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
case _MATCH_CLASS_r31: {
CHECK_CURRENT_CACHED_VALUES(3);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
_PyStackRef names;
_PyStackRef type;
_PyStackRef subject;
_PyStackRef attrs;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef _stack_item_0 = _tos_cache0;
_PyStackRef _stack_item_1 = _tos_cache1;
_PyStackRef _stack_item_2 = _tos_cache2;
GH-111485: Factor out tier 2 code generation from the rest of the interpreter code generator (GH-112968)
2023-12-12 12:12:17 +00:00
oparg = CURRENT_OPARG();
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
names = _stack_item_2;
type = _stack_item_1;
subject = _stack_item_0;
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
assert(PyTuple_CheckExact(PyStackRef_AsPyObjectBorrow(names)));
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
stack_pointer[0] = subject;
stack_pointer[1] = type;
stack_pointer[2] = names;
stack_pointer += 3;
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
GH-119866: Spill the stack around escaping calls. (GH-124392) * Spill the evaluation around escaping calls in the generated interpreter and JIT. * The code generator tracks live, cached values so they can be saved to memory when needed. * Spills the stack pointer around escaping calls, so that the exact stack is visible to the cycle GC.
2024-10-07 14:56:39 +01:00
_PyFrame_SetStackPointer(frame, stack_pointer);
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
PyObject *attrs_o = _PyEval_MatchClass(tstate,
PyStackRef_AsPyObjectBorrow(subject),
PyStackRef_AsPyObjectBorrow(type), oparg,
PyStackRef_AsPyObjectBorrow(names));
GH-128682: Account for escapes in `DECREF_INPUTS` (GH-129953) * Handle escapes in DECREF_INPUTS * Mark a few more functions as escaping * Replace DECREF_INPUTS with PyStackRef_CLOSE where possible
2025-02-12 17:44:59 +00:00
_PyStackRef tmp = names;
names = PyStackRef_NULL;
stack_pointer[-1] = names;
PyStackRef_CLOSE(tmp);
tmp = type;
type = PyStackRef_NULL;
stack_pointer[-2] = type;
PyStackRef_CLOSE(tmp);
tmp = subject;
subject = PyStackRef_NULL;
stack_pointer[-3] = subject;
PyStackRef_CLOSE(tmp);
GH-119866: Spill the stack around escaping calls. (GH-124392) * Spill the evaluation around escaping calls in the generated interpreter and JIT. * The code generator tracks live, cached values so they can be saved to memory when needed. * Spills the stack pointer around escaping calls, so that the exact stack is visible to the cycle GC.
2024-10-07 14:56:39 +01:00
stack_pointer = _PyFrame_GetStackPointer(frame);
GH-128682: Account for escapes in `DECREF_INPUTS` (GH-129953) * Handle escapes in DECREF_INPUTS * Mark a few more functions as escaping * Replace DECREF_INPUTS with PyStackRef_CLOSE where possible
2025-02-12 17:44:59 +00:00
stack_pointer += -3;
GH-141794: Limit size of generated machine code. (GH-142228) * Factor out bodies of the largest uops, to reduce jit code size. * Factor out common assert, also reducing jit code size. * Limit size of jitted code for a single executor to 1MB.
2025-12-03 17:43:35 +00:00
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
if (attrs_o) {
GH-131498: Cases generator: Parse down to C statement level. (GH-131948) * Parse down to statement level in the cases generator * Add handling for #if macros, treating them much like normal ifs.
2025-04-02 16:31:59 +01:00
assert(PyTuple_CheckExact(attrs_o));
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
attrs = PyStackRef_FromPyObjectSteal(attrs_o);
gh-104584: Baby steps towards generating and executing traces (#105924) Added a new, experimental, tracing optimizer and interpreter (a.k.a. "tier 2"). This currently pessimizes, so don't use yet -- this is infrastructure so we can experiment with optimizing passes. To enable it, pass ``-Xuops`` or set ``PYTHONUOPS=1``. To get debug output, set ``PYTHONUOPSDEBUG=N`` where ``N`` is a debug level (0-4, where 0 is no debug output and 4 is excessively verbose). All of this code is likely to change dramatically before the 3.13 feature freeze. But this is a first step.
2023-06-26 19:02:57 -07:00
}
else {
GH-128682: Make `PyStackRef_CLOSE` escaping. (GH-129404)
2025-02-03 12:41:32 +00:00
if (_PyErr_Occurred(tstate)) {
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(0);
GH-128682: Make `PyStackRef_CLOSE` escaping. (GH-129404)
2025-02-03 12:41:32 +00:00
JUMP_TO_ERROR();
}
GH-131498: Cases generator: Parse down to C statement level. (GH-131948) * Parse down to statement level in the cases generator * Add handling for #if macros, treating them much like normal ifs.
2025-04-02 16:31:59 +01:00
attrs = PyStackRef_None;
gh-104584: Baby steps towards generating and executing traces (#105924) Added a new, experimental, tracing optimizer and interpreter (a.k.a. "tier 2"). This currently pessimizes, so don't use yet -- this is infrastructure so we can experiment with optimizing passes. To enable it, pass ``-Xuops`` or set ``PYTHONUOPS=1``. To get debug output, set ``PYTHONUOPSDEBUG=N`` where ``N`` is a debug level (0-4, where 0 is no debug output and 4 is excessively verbose). All of this code is likely to change dramatically before the 3.13 feature freeze. But this is a first step.
2023-06-26 19:02:57 -07:00
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_tos_cache0 = attrs;
_tos_cache1 = PyStackRef_ZERO_BITS;
_tos_cache2 = PyStackRef_ZERO_BITS;
SET_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-104584: Baby steps towards generating and executing traces (#105924) Added a new, experimental, tracing optimizer and interpreter (a.k.a. "tier 2"). This currently pessimizes, so don't use yet -- this is infrastructure so we can experiment with optimizing passes. To enable it, pass ``-Xuops`` or set ``PYTHONUOPS=1``. To get debug output, set ``PYTHONUOPSDEBUG=N`` where ``N`` is a debug level (0-4, where 0 is no debug output and 4 is excessively verbose). All of this code is likely to change dramatically before the 3.13 feature freeze. But this is a first step.
2023-06-26 19:02:57 -07:00
break;
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
case _MATCH_MAPPING_r02: {
CHECK_CURRENT_CACHED_VALUES(0);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
_PyStackRef subject;
_PyStackRef res;
gh-106812: Refactor cases_generator to allow uops with array stack effects (#107564) Introducing a new file, stacking.py, that takes over several responsibilities related to symbolic evaluation of push/pop operations, with more generality.
2023-08-04 09:35:56 -07:00
subject = stack_pointer[-1];
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
int match = PyStackRef_TYPE(subject)->tp_flags & Py_TPFLAGS_MAPPING;
res = match ? PyStackRef_True : PyStackRef_False;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_tos_cache1 = res;
_tos_cache0 = subject;
SET_CURRENT_CACHED_VALUES(2);
stack_pointer += -1;
GH-141794: Limit size of generated machine code. (GH-142228) * Factor out bodies of the largest uops, to reduce jit code size. * Factor out common assert, also reducing jit code size. * Limit size of jitted code for a single executor to 1MB.
2025-12-03 17:43:35 +00:00
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _MATCH_MAPPING_r12: {
CHECK_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef subject;
_PyStackRef res;
_PyStackRef _stack_item_0 = _tos_cache0;
subject = _stack_item_0;
int match = PyStackRef_TYPE(subject)->tp_flags & Py_TPFLAGS_MAPPING;
res = match ? PyStackRef_True : PyStackRef_False;
_tos_cache1 = res;
_tos_cache0 = subject;
SET_CURRENT_CACHED_VALUES(2);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _MATCH_MAPPING_r23: {
CHECK_CURRENT_CACHED_VALUES(2);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef subject;
_PyStackRef res;
_PyStackRef _stack_item_0 = _tos_cache0;
_PyStackRef _stack_item_1 = _tos_cache1;
subject = _stack_item_1;
int match = PyStackRef_TYPE(subject)->tp_flags & Py_TPFLAGS_MAPPING;
res = match ? PyStackRef_True : PyStackRef_False;
_tos_cache2 = res;
_tos_cache1 = subject;
_tos_cache0 = _stack_item_0;
SET_CURRENT_CACHED_VALUES(3);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-104584: Baby steps towards generating and executing traces (#105924) Added a new, experimental, tracing optimizer and interpreter (a.k.a. "tier 2"). This currently pessimizes, so don't use yet -- this is infrastructure so we can experiment with optimizing passes. To enable it, pass ``-Xuops`` or set ``PYTHONUOPS=1``. To get debug output, set ``PYTHONUOPSDEBUG=N`` where ``N`` is a debug level (0-4, where 0 is no debug output and 4 is excessively verbose). All of this code is likely to change dramatically before the 3.13 feature freeze. But this is a first step.
2023-06-26 19:02:57 -07:00
break;
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
case _MATCH_SEQUENCE_r02: {
CHECK_CURRENT_CACHED_VALUES(0);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
_PyStackRef subject;
_PyStackRef res;
gh-106812: Refactor cases_generator to allow uops with array stack effects (#107564) Introducing a new file, stacking.py, that takes over several responsibilities related to symbolic evaluation of push/pop operations, with more generality.
2023-08-04 09:35:56 -07:00
subject = stack_pointer[-1];
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
int match = PyStackRef_TYPE(subject)->tp_flags & Py_TPFLAGS_SEQUENCE;
res = match ? PyStackRef_True : PyStackRef_False;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_tos_cache1 = res;
_tos_cache0 = subject;
SET_CURRENT_CACHED_VALUES(2);
stack_pointer += -1;
GH-141794: Limit size of generated machine code. (GH-142228) * Factor out bodies of the largest uops, to reduce jit code size. * Factor out common assert, also reducing jit code size. * Limit size of jitted code for a single executor to 1MB.
2025-12-03 17:43:35 +00:00
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _MATCH_SEQUENCE_r12: {
CHECK_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef subject;
_PyStackRef res;
_PyStackRef _stack_item_0 = _tos_cache0;
subject = _stack_item_0;
int match = PyStackRef_TYPE(subject)->tp_flags & Py_TPFLAGS_SEQUENCE;
res = match ? PyStackRef_True : PyStackRef_False;
_tos_cache1 = res;
_tos_cache0 = subject;
SET_CURRENT_CACHED_VALUES(2);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _MATCH_SEQUENCE_r23: {
CHECK_CURRENT_CACHED_VALUES(2);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef subject;
_PyStackRef res;
_PyStackRef _stack_item_0 = _tos_cache0;
_PyStackRef _stack_item_1 = _tos_cache1;
subject = _stack_item_1;
int match = PyStackRef_TYPE(subject)->tp_flags & Py_TPFLAGS_SEQUENCE;
res = match ? PyStackRef_True : PyStackRef_False;
_tos_cache2 = res;
_tos_cache1 = subject;
_tos_cache0 = _stack_item_0;
SET_CURRENT_CACHED_VALUES(3);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-104584: Baby steps towards generating and executing traces (#105924) Added a new, experimental, tracing optimizer and interpreter (a.k.a. "tier 2"). This currently pessimizes, so don't use yet -- this is infrastructure so we can experiment with optimizing passes. To enable it, pass ``-Xuops`` or set ``PYTHONUOPS=1``. To get debug output, set ``PYTHONUOPSDEBUG=N`` where ``N`` is a debug level (0-4, where 0 is no debug output and 4 is excessively verbose). All of this code is likely to change dramatically before the 3.13 feature freeze. But this is a first step.
2023-06-26 19:02:57 -07:00
break;
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
case _MATCH_KEYS_r23: {
CHECK_CURRENT_CACHED_VALUES(2);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
_PyStackRef keys;
_PyStackRef subject;
_PyStackRef values_or_none;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef _stack_item_0 = _tos_cache0;
_PyStackRef _stack_item_1 = _tos_cache1;
keys = _stack_item_1;
subject = _stack_item_0;
stack_pointer[0] = subject;
stack_pointer[1] = keys;
stack_pointer += 2;
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
GH-119866: Spill the stack around escaping calls. (GH-124392) * Spill the evaluation around escaping calls in the generated interpreter and JIT. * The code generator tracks live, cached values so they can be saved to memory when needed. * Spills the stack pointer around escaping calls, so that the exact stack is visible to the cycle GC.
2024-10-07 14:56:39 +01:00
_PyFrame_SetStackPointer(frame, stack_pointer);
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
PyObject *values_or_none_o = _PyEval_MatchKeys(tstate,
PyStackRef_AsPyObjectBorrow(subject), PyStackRef_AsPyObjectBorrow(keys));
GH-119866: Spill the stack around escaping calls. (GH-124392) * Spill the evaluation around escaping calls in the generated interpreter and JIT. * The code generator tracks live, cached values so they can be saved to memory when needed. * Spills the stack pointer around escaping calls, so that the exact stack is visible to the cycle GC.
2024-10-07 14:56:39 +01:00
stack_pointer = _PyFrame_GetStackPointer(frame);
GH-128682: Make `PyStackRef_CLOSE` escaping. (GH-129404)
2025-02-03 12:41:32 +00:00
if (values_or_none_o == NULL) {
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(0);
GH-128682: Make `PyStackRef_CLOSE` escaping. (GH-129404)
2025-02-03 12:41:32 +00:00
JUMP_TO_ERROR();
}
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
values_or_none = PyStackRef_FromPyObjectSteal(values_or_none_o);
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_tos_cache2 = values_or_none;
_tos_cache1 = keys;
_tos_cache0 = subject;
SET_CURRENT_CACHED_VALUES(3);
stack_pointer += -2;
GH-141794: Limit size of generated machine code. (GH-142228) * Factor out bodies of the largest uops, to reduce jit code size. * Factor out common assert, also reducing jit code size. * Limit size of jitted code for a single executor to 1MB.
2025-12-03 17:43:35 +00:00
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-104584: Baby steps towards generating and executing traces (#105924) Added a new, experimental, tracing optimizer and interpreter (a.k.a. "tier 2"). This currently pessimizes, so don't use yet -- this is infrastructure so we can experiment with optimizing passes. To enable it, pass ``-Xuops`` or set ``PYTHONUOPS=1``. To get debug output, set ``PYTHONUOPSDEBUG=N`` where ``N`` is a debug level (0-4, where 0 is no debug output and 4 is excessively verbose). All of this code is likely to change dramatically before the 3.13 feature freeze. But this is a first step.
2023-06-26 19:02:57 -07:00
break;
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
case _GET_ITER_r12: {
CHECK_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
_PyStackRef iterable;
_PyStackRef iter;
GH-132554: "Virtual" iterators (GH-132555) * FOR_ITER now pushes either the iterator and NULL or leaves the iterable and pushes tagged zero * NEXT_ITER uses the tagged int as the index into the sequence or, if TOS is NULL, iterates as before.
2025-05-27 15:59:45 +01:00
_PyStackRef index_or_null;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef _stack_item_0 = _tos_cache0;
iterable = _stack_item_0;
GH-132554: Add stats for GET_ITER (GH-132592) * Add stats for GET_ITER * Look for common iterable types, not iterator types * Add stats for self iter and fix naming in summary
2025-04-29 09:00:14 +01:00
#ifdef Py_STATS
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
stack_pointer[0] = iterable;
stack_pointer += 1;
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
GH-132554: Add stats for GET_ITER (GH-132592) * Add stats for GET_ITER * Look for common iterable types, not iterator types * Add stats for self iter and fix naming in summary
2025-04-29 09:00:14 +01:00
_PyFrame_SetStackPointer(frame, stack_pointer);
_Py_GatherStats_GetIter(iterable);
stack_pointer = _PyFrame_GetStackPointer(frame);
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
stack_pointer += -1;
GH-132554: Add stats for GET_ITER (GH-132592) * Add stats for GET_ITER * Look for common iterable types, not iterator types * Add stats for self iter and fix naming in summary
2025-04-29 09:00:14 +01:00
#endif
GH-132554: "Virtual" iterators (GH-132555) * FOR_ITER now pushes either the iterator and NULL or leaves the iterable and pushes tagged zero * NEXT_ITER uses the tagged int as the index into the sequence or, if TOS is NULL, iterates as before.
2025-05-27 15:59:45 +01:00
PyTypeObject *tp = PyStackRef_TYPE(iterable);
if (tp == &PyTuple_Type || tp == &PyList_Type) {
iter = iterable;
index_or_null = PyStackRef_TagInt(0);
GH-128682: Make `PyStackRef_CLOSE` escaping. (GH-129404)
2025-02-03 12:41:32 +00:00
}
GH-132554: "Virtual" iterators (GH-132555) * FOR_ITER now pushes either the iterator and NULL or leaves the iterable and pushes tagged zero * NEXT_ITER uses the tagged int as the index into the sequence or, if TOS is NULL, iterates as before.
2025-05-27 15:59:45 +01:00
else {
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
stack_pointer[0] = iterable;
stack_pointer += 1;
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
GH-132554: "Virtual" iterators (GH-132555) * FOR_ITER now pushes either the iterator and NULL or leaves the iterable and pushes tagged zero * NEXT_ITER uses the tagged int as the index into the sequence or, if TOS is NULL, iterates as before.
2025-05-27 15:59:45 +01:00
_PyFrame_SetStackPointer(frame, stack_pointer);
PyObject *iter_o = PyObject_GetIter(PyStackRef_AsPyObjectBorrow(iterable));
stack_pointer = _PyFrame_GetStackPointer(frame);
stack_pointer += -1;
GH-141794: Limit size of generated machine code. (GH-142228) * Factor out bodies of the largest uops, to reduce jit code size. * Factor out common assert, also reducing jit code size. * Limit size of jitted code for a single executor to 1MB.
2025-12-03 17:43:35 +00:00
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
GH-132554: "Virtual" iterators (GH-132555) * FOR_ITER now pushes either the iterator and NULL or leaves the iterable and pushes tagged zero * NEXT_ITER uses the tagged int as the index into the sequence or, if TOS is NULL, iterates as before.
2025-05-27 15:59:45 +01:00
_PyFrame_SetStackPointer(frame, stack_pointer);
PyStackRef_CLOSE(iterable);
stack_pointer = _PyFrame_GetStackPointer(frame);
if (iter_o == NULL) {
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(0);
GH-132554: "Virtual" iterators (GH-132555) * FOR_ITER now pushes either the iterator and NULL or leaves the iterable and pushes tagged zero * NEXT_ITER uses the tagged int as the index into the sequence or, if TOS is NULL, iterates as before.
2025-05-27 15:59:45 +01:00
JUMP_TO_ERROR();
}
iter = PyStackRef_FromPyObjectSteal(iter_o);
index_or_null = PyStackRef_NULL;
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_tos_cache1 = index_or_null;
_tos_cache0 = iter;
_tos_cache2 = PyStackRef_ZERO_BITS;
SET_CURRENT_CACHED_VALUES(2);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-104584: Baby steps towards generating and executing traces (#105924) Added a new, experimental, tracing optimizer and interpreter (a.k.a. "tier 2"). This currently pessimizes, so don't use yet -- this is infrastructure so we can experiment with optimizing passes. To enable it, pass ``-Xuops`` or set ``PYTHONUOPS=1``. To get debug output, set ``PYTHONUOPSDEBUG=N`` where ``N`` is a debug level (0-4, where 0 is no debug output and 4 is excessively verbose). All of this code is likely to change dramatically before the 3.13 feature freeze. But this is a first step.
2023-06-26 19:02:57 -07:00
break;
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
case _GET_YIELD_FROM_ITER_r11: {
CHECK_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
_PyStackRef iterable;
_PyStackRef iter;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef _stack_item_0 = _tos_cache0;
iterable = _stack_item_0;
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
PyObject *iterable_o = PyStackRef_AsPyObjectBorrow(iterable);
if (PyCoro_CheckExact(iterable_o)) {
gh-104584: Baby steps towards generating and executing traces (#105924) Added a new, experimental, tracing optimizer and interpreter (a.k.a. "tier 2"). This currently pessimizes, so don't use yet -- this is infrastructure so we can experiment with optimizing passes. To enable it, pass ``-Xuops`` or set ``PYTHONUOPS=1``. To get debug output, set ``PYTHONUOPSDEBUG=N`` where ``N`` is a debug level (0-4, where 0 is no debug output and 4 is excessively verbose). All of this code is likely to change dramatically before the 3.13 feature freeze. But this is a first step.
2023-06-26 19:02:57 -07:00
if (!(_PyFrame_GetCode(frame)->co_flags & (CO_COROUTINE | CO_ITERABLE_COROUTINE))) {
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
stack_pointer[0] = iterable;
stack_pointer += 1;
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
GH-119866: Spill the stack around escaping calls. (GH-124392) * Spill the evaluation around escaping calls in the generated interpreter and JIT. * The code generator tracks live, cached values so they can be saved to memory when needed. * Spills the stack pointer around escaping calls, so that the exact stack is visible to the cycle GC.
2024-10-07 14:56:39 +01:00
_PyFrame_SetStackPointer(frame, stack_pointer);
gh-104584: Baby steps towards generating and executing traces (#105924) Added a new, experimental, tracing optimizer and interpreter (a.k.a. "tier 2"). This currently pessimizes, so don't use yet -- this is infrastructure so we can experiment with optimizing passes. To enable it, pass ``-Xuops`` or set ``PYTHONUOPS=1``. To get debug output, set ``PYTHONUOPSDEBUG=N`` where ``N`` is a debug level (0-4, where 0 is no debug output and 4 is excessively verbose). All of this code is likely to change dramatically before the 3.13 feature freeze. But this is a first step.
2023-06-26 19:02:57 -07:00
_PyErr_SetString(tstate, PyExc_TypeError,
"cannot 'yield from' a coroutine object "
"in a non-coroutine generator");
GH-119866: Spill the stack around escaping calls. (GH-124392) * Spill the evaluation around escaping calls in the generated interpreter and JIT. * The code generator tracks live, cached values so they can be saved to memory when needed. * Spills the stack pointer around escaping calls, so that the exact stack is visible to the cycle GC.
2024-10-07 14:56:39 +01:00
stack_pointer = _PyFrame_GetStackPointer(frame);
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(0);
GH-116422: Tier2 hot/cold splitting (GH-116813) Splits the "cold" path, deopts and exits, from the "hot" path, reducing the size of most jitted instructions, at the cost of slower exits.
2024-03-26 09:35:11 +00:00
JUMP_TO_ERROR();
gh-104584: Baby steps towards generating and executing traces (#105924) Added a new, experimental, tracing optimizer and interpreter (a.k.a. "tier 2"). This currently pessimizes, so don't use yet -- this is infrastructure so we can experiment with optimizing passes. To enable it, pass ``-Xuops`` or set ``PYTHONUOPS=1``. To get debug output, set ``PYTHONUOPSDEBUG=N`` where ``N`` is a debug level (0-4, where 0 is no debug output and 4 is excessively verbose). All of this code is likely to change dramatically before the 3.13 feature freeze. But this is a first step.
2023-06-26 19:02:57 -07:00
}
iter = iterable;
}
GH-131498: Cases generator: Parse down to C statement level. (GH-131948) * Parse down to statement level in the cases generator * Add handling for #if macros, treating them much like normal ifs.
2025-04-02 16:31:59 +01:00
else if (PyGen_CheckExact(iterable_o)) {
iter = iterable;
}
gh-104584: Baby steps towards generating and executing traces (#105924) Added a new, experimental, tracing optimizer and interpreter (a.k.a. "tier 2"). This currently pessimizes, so don't use yet -- this is infrastructure so we can experiment with optimizing passes. To enable it, pass ``-Xuops`` or set ``PYTHONUOPS=1``. To get debug output, set ``PYTHONUOPSDEBUG=N`` where ``N`` is a debug level (0-4, where 0 is no debug output and 4 is excessively verbose). All of this code is likely to change dramatically before the 3.13 feature freeze. But this is a first step.
2023-06-26 19:02:57 -07:00
else {
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
stack_pointer[0] = iterable;
stack_pointer += 1;
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
GH-131498: Cases generator: Parse down to C statement level. (GH-131948) * Parse down to statement level in the cases generator * Add handling for #if macros, treating them much like normal ifs.
2025-04-02 16:31:59 +01:00
_PyFrame_SetStackPointer(frame, stack_pointer);
PyObject *iter_o = PyObject_GetIter(iterable_o);
stack_pointer = _PyFrame_GetStackPointer(frame);
if (iter_o == NULL) {
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(0);
GH-131498: Cases generator: Parse down to C statement level. (GH-131948) * Parse down to statement level in the cases generator * Add handling for #if macros, treating them much like normal ifs.
2025-04-02 16:31:59 +01:00
JUMP_TO_ERROR();
gh-104584: Baby steps towards generating and executing traces (#105924) Added a new, experimental, tracing optimizer and interpreter (a.k.a. "tier 2"). This currently pessimizes, so don't use yet -- this is infrastructure so we can experiment with optimizing passes. To enable it, pass ``-Xuops`` or set ``PYTHONUOPS=1``. To get debug output, set ``PYTHONUOPSDEBUG=N`` where ``N`` is a debug level (0-4, where 0 is no debug output and 4 is excessively verbose). All of this code is likely to change dramatically before the 3.13 feature freeze. But this is a first step.
2023-06-26 19:02:57 -07:00
}
GH-131498: Cases generator: Parse down to C statement level. (GH-131948) * Parse down to statement level in the cases generator * Add handling for #if macros, treating them much like normal ifs.
2025-04-02 16:31:59 +01:00
iter = PyStackRef_FromPyObjectSteal(iter_o);
_PyFrame_SetStackPointer(frame, stack_pointer);
_PyStackRef tmp = iterable;
iterable = iter;
stack_pointer[-1] = iterable;
PyStackRef_CLOSE(tmp);
stack_pointer = _PyFrame_GetStackPointer(frame);
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
stack_pointer += -1;
gh-104584: Baby steps towards generating and executing traces (#105924) Added a new, experimental, tracing optimizer and interpreter (a.k.a. "tier 2"). This currently pessimizes, so don't use yet -- this is infrastructure so we can experiment with optimizing passes. To enable it, pass ``-Xuops`` or set ``PYTHONUOPS=1``. To get debug output, set ``PYTHONUOPSDEBUG=N`` where ``N`` is a debug level (0-4, where 0 is no debug output and 4 is excessively verbose). All of this code is likely to change dramatically before the 3.13 feature freeze. But this is a first step.
2023-06-26 19:02:57 -07:00
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_tos_cache0 = iter;
_tos_cache1 = PyStackRef_ZERO_BITS;
_tos_cache2 = PyStackRef_ZERO_BITS;
SET_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-104584: Baby steps towards generating and executing traces (#105924) Added a new, experimental, tracing optimizer and interpreter (a.k.a. "tier 2"). This currently pessimizes, so don't use yet -- this is infrastructure so we can experiment with optimizing passes. To enable it, pass ``-Xuops`` or set ``PYTHONUOPS=1``. To get debug output, set ``PYTHONUOPSDEBUG=N`` where ``N`` is a debug level (0-4, where 0 is no debug output and 4 is excessively verbose). All of this code is likely to change dramatically before the 3.13 feature freeze. But this is a first step.
2023-06-26 19:02:57 -07:00
break;
gh-106529: Make FOR_ITER a viable uop (#112134) This uses the new mechanism whereby certain uops are replaced by others during translation, using the `_PyUop_Replacements` table. We further special-case the `_FOR_ITER_TIER_TWO` uop to update the deoptimization target to point just past the corresponding `END_FOR` opcode. Two tiny code cleanups are also part of this PR.
2023-11-20 10:08:53 -08:00
}
GH-116422: Tier2 hot/cold splitting (GH-116813) Splits the "cold" path, deopts and exits, from the "hot" path, reducing the size of most jitted instructions, at the cost of slower exits.
2024-03-26 09:35:11 +00:00
/* _FOR_ITER is not a viable micro-op for tier 2 because it is replaced */
GH-111485: Factor out tier 2 code generation from the rest of the interpreter code generator (GH-112968)
2023-12-12 12:12:17 +00:00
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
case _FOR_ITER_TIER_TWO_r23: {
CHECK_CURRENT_CACHED_VALUES(2);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-132554: "Virtual" iterators (GH-132555) * FOR_ITER now pushes either the iterator and NULL or leaves the iterable and pushes tagged zero * NEXT_ITER uses the tagged int as the index into the sequence or, if TOS is NULL, iterates as before.
2025-05-27 15:59:45 +01:00
_PyStackRef null_or_index;
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
_PyStackRef iter;
_PyStackRef next;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef _stack_item_0 = _tos_cache0;
_PyStackRef _stack_item_1 = _tos_cache1;
null_or_index = _stack_item_1;
iter = _stack_item_0;
stack_pointer[0] = iter;
stack_pointer[1] = null_or_index;
stack_pointer += 2;
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
GH-119866: Spill the stack around escaping calls. (GH-124392) * Spill the evaluation around escaping calls in the generated interpreter and JIT. * The code generator tracks live, cached values so they can be saved to memory when needed. * Spills the stack pointer around escaping calls, so that the exact stack is visible to the cycle GC.
2024-10-07 14:56:39 +01:00
_PyFrame_SetStackPointer(frame, stack_pointer);
GH-132554: Fix tier2 `FOR_ITER` implementation and optimizations (GH-135137)
2025-06-05 18:53:57 +01:00
_PyStackRef item = _PyForIter_VirtualIteratorNext(tstate, frame, iter, &null_or_index);
GH-119866: Spill the stack around escaping calls. (GH-124392) * Spill the evaluation around escaping calls in the generated interpreter and JIT. * The code generator tracks live, cached values so they can be saved to memory when needed. * Spills the stack pointer around escaping calls, so that the exact stack is visible to the cycle GC.
2024-10-07 14:56:39 +01:00
stack_pointer = _PyFrame_GetStackPointer(frame);
GH-132554: Fix tier2 `FOR_ITER` implementation and optimizations (GH-135137)
2025-06-05 18:53:57 +01:00
if (!PyStackRef_IsValid(item)) {
if (PyStackRef_IsError(item)) {
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
stack_pointer[-1] = null_or_index;
SET_CURRENT_CACHED_VALUES(0);
GH-132554: Fix tier2 `FOR_ITER` implementation and optimizations (GH-135137)
2025-06-05 18:53:57 +01:00
JUMP_TO_ERROR();
gh-106529: Make FOR_ITER a viable uop (#112134) This uses the new mechanism whereby certain uops are replaced by others during translation, using the `_PyUop_Replacements` table. We further special-case the `_FOR_ITER_TIER_TWO` uop to update the deoptimization target to point just past the corresponding `END_FOR` opcode. Two tiny code cleanups are also part of this PR.
2023-11-20 10:08:53 -08:00
}
GH-117457: Correct pystats uop "miss" counts (GH-117477)
2024-04-04 18:49:18 -04:00
if (true) {
UOP_STAT_INC(uopcode, miss);
GH-142718: JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. (GH-142762) JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. * Make sure that stack is in correct state at side exits with TOS cached values * Simplify choice of cached items for side exits
2025-12-15 19:01:29 +00:00
_tos_cache1 = null_or_index;
_tos_cache0 = iter;
SET_CURRENT_CACHED_VALUES(2);
stack_pointer += -2;
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
GH-117457: Correct pystats uop "miss" counts (GH-117477)
2024-04-04 18:49:18 -04:00
JUMP_TO_JUMP_TARGET();
}
gh-106529: Make FOR_ITER a viable uop (#112134) This uses the new mechanism whereby certain uops are replaced by others during translation, using the `_PyUop_Replacements` table. We further special-case the `_FOR_ITER_TIER_TWO` uop to update the deoptimization target to point just past the corresponding `END_FOR` opcode. Two tiny code cleanups are also part of this PR.
2023-11-20 10:08:53 -08:00
}
GH-132554: Fix tier2 `FOR_ITER` implementation and optimizations (GH-135137)
2025-06-05 18:53:57 +01:00
next = item;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_tos_cache2 = next;
_tos_cache1 = null_or_index;
_tos_cache0 = iter;
SET_CURRENT_CACHED_VALUES(3);
stack_pointer += -2;
GH-141794: Limit size of generated machine code. (GH-142228) * Factor out bodies of the largest uops, to reduce jit code size. * Factor out common assert, also reducing jit code size. * Limit size of jitted code for a single executor to 1MB.
2025-12-03 17:43:35 +00:00
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-106529: Make FOR_ITER a viable uop (#112134) This uses the new mechanism whereby certain uops are replaced by others during translation, using the `_PyUop_Replacements` table. We further special-case the `_FOR_ITER_TIER_TWO` uop to update the deoptimization target to point just past the corresponding `END_FOR` opcode. Two tiny code cleanups are also part of this PR.
2023-11-20 10:08:53 -08:00
break;
gh-104584: Baby steps towards generating and executing traces (#105924) Added a new, experimental, tracing optimizer and interpreter (a.k.a. "tier 2"). This currently pessimizes, so don't use yet -- this is infrastructure so we can experiment with optimizing passes. To enable it, pass ``-Xuops`` or set ``PYTHONUOPS=1``. To get debug output, set ``PYTHONUOPSDEBUG=N`` where ``N`` is a debug level (0-4, where 0 is no debug output and 4 is excessively verbose). All of this code is likely to change dramatically before the 3.13 feature freeze. But this is a first step.
2023-06-26 19:02:57 -07:00
}
GH-116422: Tier2 hot/cold splitting (GH-116813) Splits the "cold" path, deopts and exits, from the "hot" path, reducing the size of most jitted instructions, at the cost of slower exits.
2024-03-26 09:35:11 +00:00
/* _INSTRUMENTED_FOR_ITER is not a viable micro-op for tier 2 because it is instrumented */
GH-111485: Factor out tier 2 code generation from the rest of the interpreter code generator (GH-112968)
2023-12-12 12:12:17 +00:00
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
case _ITER_CHECK_LIST_r02: {
CHECK_CURRENT_CACHED_VALUES(0);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-132554: "Virtual" iterators (GH-132555) * FOR_ITER now pushes either the iterator and NULL or leaves the iterable and pushes tagged zero * NEXT_ITER uses the tagged int as the index into the sequence or, if TOS is NULL, iterates as before.
2025-05-27 15:59:45 +01:00
_PyStackRef null_or_index;
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
_PyStackRef iter;
GH-132554: "Virtual" iterators (GH-132555) * FOR_ITER now pushes either the iterator and NULL or leaves the iterable and pushes tagged zero * NEXT_ITER uses the tagged int as the index into the sequence or, if TOS is NULL, iterates as before.
2025-05-27 15:59:45 +01:00
null_or_index = stack_pointer[-1];
iter = stack_pointer[-2];
gh-115999: Add free-threaded specialization for FOR_ITER (#128798) Add free-threaded versions of existing specialization for FOR_ITER (list, tuples, fast range iterators and generators), without significantly affecting their thread-safety. (Iterating over shared lists/tuples/ranges should be fine like before. Reusing iterators between threads is not fine, like before. Sharing generators between threads is a recipe for significant crashes, like before.)
2025-03-12 16:21:46 +01:00
PyObject *iter_o = PyStackRef_AsPyObjectBorrow(iter);
GH-132554: "Virtual" iterators (GH-132555) * FOR_ITER now pushes either the iterator and NULL or leaves the iterable and pushes tagged zero * NEXT_ITER uses the tagged int as the index into the sequence or, if TOS is NULL, iterates as before.
2025-05-27 15:59:45 +01:00
if (Py_TYPE(iter_o) != &PyList_Type) {
GH-117457: Correct pystats uop "miss" counts (GH-117477)
2024-04-04 18:49:18 -04:00
UOP_STAT_INC(uopcode, miss);
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(0);
JUMP_TO_JUMP_TARGET();
}
assert(PyStackRef_IsTaggedInt(null_or_index));
#ifdef Py_GIL_DISABLED
if (!_Py_IsOwnedByCurrentThread(iter_o) && !_PyObject_GC_IS_SHARED(iter_o)) {
UOP_STAT_INC(uopcode, miss);
SET_CURRENT_CACHED_VALUES(0);
JUMP_TO_JUMP_TARGET();
}
#endif
_tos_cache1 = null_or_index;
_tos_cache0 = iter;
SET_CURRENT_CACHED_VALUES(2);
stack_pointer += -2;
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _ITER_CHECK_LIST_r12: {
CHECK_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef null_or_index;
_PyStackRef iter;
_PyStackRef _stack_item_0 = _tos_cache0;
null_or_index = _stack_item_0;
iter = stack_pointer[-1];
PyObject *iter_o = PyStackRef_AsPyObjectBorrow(iter);
if (Py_TYPE(iter_o) != &PyList_Type) {
UOP_STAT_INC(uopcode, miss);
GH-142718: JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. (GH-142762) JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. * Make sure that stack is in correct state at side exits with TOS cached values * Simplify choice of cached items for side exits
2025-12-15 19:01:29 +00:00
_tos_cache0 = null_or_index;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(1);
JUMP_TO_JUMP_TARGET();
}
assert(PyStackRef_IsTaggedInt(null_or_index));
#ifdef Py_GIL_DISABLED
if (!_Py_IsOwnedByCurrentThread(iter_o) && !_PyObject_GC_IS_SHARED(iter_o)) {
UOP_STAT_INC(uopcode, miss);
GH-142718: JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. (GH-142762) JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. * Make sure that stack is in correct state at side exits with TOS cached values * Simplify choice of cached items for side exits
2025-12-15 19:01:29 +00:00
_tos_cache0 = null_or_index;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(1);
JUMP_TO_JUMP_TARGET();
}
#endif
_tos_cache1 = null_or_index;
_tos_cache0 = iter;
SET_CURRENT_CACHED_VALUES(2);
stack_pointer += -1;
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _ITER_CHECK_LIST_r22: {
CHECK_CURRENT_CACHED_VALUES(2);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef null_or_index;
_PyStackRef iter;
_PyStackRef _stack_item_0 = _tos_cache0;
_PyStackRef _stack_item_1 = _tos_cache1;
null_or_index = _stack_item_1;
iter = _stack_item_0;
PyObject *iter_o = PyStackRef_AsPyObjectBorrow(iter);
if (Py_TYPE(iter_o) != &PyList_Type) {
UOP_STAT_INC(uopcode, miss);
GH-142718: JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. (GH-142762) JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. * Make sure that stack is in correct state at side exits with TOS cached values * Simplify choice of cached items for side exits
2025-12-15 19:01:29 +00:00
_tos_cache1 = null_or_index;
_tos_cache0 = iter;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(2);
JUMP_TO_JUMP_TARGET();
}
assert(PyStackRef_IsTaggedInt(null_or_index));
#ifdef Py_GIL_DISABLED
if (!_Py_IsOwnedByCurrentThread(iter_o) && !_PyObject_GC_IS_SHARED(iter_o)) {
UOP_STAT_INC(uopcode, miss);
GH-142718: JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. (GH-142762) JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. * Make sure that stack is in correct state at side exits with TOS cached values * Simplify choice of cached items for side exits
2025-12-15 19:01:29 +00:00
_tos_cache1 = null_or_index;
_tos_cache0 = iter;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(2);
JUMP_TO_JUMP_TARGET();
}
#endif
_tos_cache1 = null_or_index;
_tos_cache0 = iter;
SET_CURRENT_CACHED_VALUES(2);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _ITER_CHECK_LIST_r33: {
CHECK_CURRENT_CACHED_VALUES(3);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef null_or_index;
_PyStackRef iter;
_PyStackRef _stack_item_0 = _tos_cache0;
_PyStackRef _stack_item_1 = _tos_cache1;
_PyStackRef _stack_item_2 = _tos_cache2;
null_or_index = _stack_item_2;
iter = _stack_item_1;
PyObject *iter_o = PyStackRef_AsPyObjectBorrow(iter);
if (Py_TYPE(iter_o) != &PyList_Type) {
UOP_STAT_INC(uopcode, miss);
GH-142718: JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. (GH-142762) JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. * Make sure that stack is in correct state at side exits with TOS cached values * Simplify choice of cached items for side exits
2025-12-15 19:01:29 +00:00
_tos_cache2 = null_or_index;
_tos_cache1 = iter;
_tos_cache0 = _stack_item_0;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(3);
GH-117457: Correct pystats uop "miss" counts (GH-117477)
2024-04-04 18:49:18 -04:00
JUMP_TO_JUMP_TARGET();
}
GH-132554: "Virtual" iterators (GH-132555) * FOR_ITER now pushes either the iterator and NULL or leaves the iterable and pushes tagged zero * NEXT_ITER uses the tagged int as the index into the sequence or, if TOS is NULL, iterates as before.
2025-05-27 15:59:45 +01:00
assert(PyStackRef_IsTaggedInt(null_or_index));
gh-115999: Add free-threaded specialization for FOR_ITER (#128798) Add free-threaded versions of existing specialization for FOR_ITER (list, tuples, fast range iterators and generators), without significantly affecting their thread-safety. (Iterating over shared lists/tuples/ranges should be fine like before. Reusing iterators between threads is not fine, like before. Sharing generators between threads is a recipe for significant crashes, like before.)
2025-03-12 16:21:46 +01:00
#ifdef Py_GIL_DISABLED
GH-132554: "Virtual" iterators (GH-132555) * FOR_ITER now pushes either the iterator and NULL or leaves the iterable and pushes tagged zero * NEXT_ITER uses the tagged int as the index into the sequence or, if TOS is NULL, iterates as before.
2025-05-27 15:59:45 +01:00
if (!_Py_IsOwnedByCurrentThread(iter_o) && !_PyObject_GC_IS_SHARED(iter_o)) {
gh-115999: Add free-threaded specialization for FOR_ITER (#128798) Add free-threaded versions of existing specialization for FOR_ITER (list, tuples, fast range iterators and generators), without significantly affecting their thread-safety. (Iterating over shared lists/tuples/ranges should be fine like before. Reusing iterators between threads is not fine, like before. Sharing generators between threads is a recipe for significant crashes, like before.)
2025-03-12 16:21:46 +01:00
UOP_STAT_INC(uopcode, miss);
GH-142718: JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. (GH-142762) JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. * Make sure that stack is in correct state at side exits with TOS cached values * Simplify choice of cached items for side exits
2025-12-15 19:01:29 +00:00
_tos_cache2 = null_or_index;
_tos_cache1 = iter;
_tos_cache0 = _stack_item_0;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(3);
JUMP_TO_JUMP_TARGET();
}
#endif
_tos_cache2 = null_or_index;
_tos_cache1 = iter;
_tos_cache0 = _stack_item_0;
SET_CURRENT_CACHED_VALUES(3);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
/* _ITER_JUMP_LIST is not a viable micro-op for tier 2 because it is replaced */
case _GUARD_NOT_EXHAUSTED_LIST_r02: {
CHECK_CURRENT_CACHED_VALUES(0);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef null_or_index;
_PyStackRef iter;
null_or_index = stack_pointer[-1];
iter = stack_pointer[-2];
#ifndef Py_GIL_DISABLED
PyObject *list_o = PyStackRef_AsPyObjectBorrow(iter);
assert(Py_TYPE(list_o) == &PyList_Type);
if ((size_t)PyStackRef_UntagInt(null_or_index) >= (size_t)PyList_GET_SIZE(list_o)) {
UOP_STAT_INC(uopcode, miss);
SET_CURRENT_CACHED_VALUES(0);
JUMP_TO_JUMP_TARGET();
}
#endif
_tos_cache1 = null_or_index;
_tos_cache0 = iter;
SET_CURRENT_CACHED_VALUES(2);
stack_pointer += -2;
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _GUARD_NOT_EXHAUSTED_LIST_r12: {
CHECK_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef null_or_index;
_PyStackRef iter;
_PyStackRef _stack_item_0 = _tos_cache0;
null_or_index = _stack_item_0;
iter = stack_pointer[-1];
#ifndef Py_GIL_DISABLED
PyObject *list_o = PyStackRef_AsPyObjectBorrow(iter);
assert(Py_TYPE(list_o) == &PyList_Type);
if ((size_t)PyStackRef_UntagInt(null_or_index) >= (size_t)PyList_GET_SIZE(list_o)) {
UOP_STAT_INC(uopcode, miss);
GH-142718: JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. (GH-142762) JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. * Make sure that stack is in correct state at side exits with TOS cached values * Simplify choice of cached items for side exits
2025-12-15 19:01:29 +00:00
_tos_cache0 = null_or_index;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(1);
gh-115999: Add free-threaded specialization for FOR_ITER (#128798) Add free-threaded versions of existing specialization for FOR_ITER (list, tuples, fast range iterators and generators), without significantly affecting their thread-safety. (Iterating over shared lists/tuples/ranges should be fine like before. Reusing iterators between threads is not fine, like before. Sharing generators between threads is a recipe for significant crashes, like before.)
2025-03-12 16:21:46 +01:00
JUMP_TO_JUMP_TARGET();
}
#endif
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_tos_cache1 = null_or_index;
_tos_cache0 = iter;
SET_CURRENT_CACHED_VALUES(2);
stack_pointer += -1;
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-106529: Split FOR_ITER_{LIST,TUPLE} into uops (#106696) Also rename `_ITER_EXHAUSTED_XXX` to `_IS_ITER_EXHAUSTED_XXX` to make it clear this is a test.
2023-07-13 17:27:35 -07:00
break;
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
case _GUARD_NOT_EXHAUSTED_LIST_r22: {
CHECK_CURRENT_CACHED_VALUES(2);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef null_or_index;
_PyStackRef iter;
_PyStackRef _stack_item_0 = _tos_cache0;
_PyStackRef _stack_item_1 = _tos_cache1;
null_or_index = _stack_item_1;
iter = _stack_item_0;
#ifndef Py_GIL_DISABLED
PyObject *list_o = PyStackRef_AsPyObjectBorrow(iter);
assert(Py_TYPE(list_o) == &PyList_Type);
if ((size_t)PyStackRef_UntagInt(null_or_index) >= (size_t)PyList_GET_SIZE(list_o)) {
UOP_STAT_INC(uopcode, miss);
GH-142718: JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. (GH-142762) JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. * Make sure that stack is in correct state at side exits with TOS cached values * Simplify choice of cached items for side exits
2025-12-15 19:01:29 +00:00
_tos_cache1 = null_or_index;
_tos_cache0 = iter;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(2);
JUMP_TO_JUMP_TARGET();
}
#endif
_tos_cache1 = null_or_index;
_tos_cache0 = iter;
SET_CURRENT_CACHED_VALUES(2);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
GH-111485: Factor out tier 2 code generation from the rest of the interpreter code generator (GH-112968)
2023-12-12 12:12:17 +00:00
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
case _GUARD_NOT_EXHAUSTED_LIST_r33: {
CHECK_CURRENT_CACHED_VALUES(3);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-132554: "Virtual" iterators (GH-132555) * FOR_ITER now pushes either the iterator and NULL or leaves the iterable and pushes tagged zero * NEXT_ITER uses the tagged int as the index into the sequence or, if TOS is NULL, iterates as before.
2025-05-27 15:59:45 +01:00
_PyStackRef null_or_index;
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
_PyStackRef iter;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef _stack_item_0 = _tos_cache0;
_PyStackRef _stack_item_1 = _tos_cache1;
_PyStackRef _stack_item_2 = _tos_cache2;
null_or_index = _stack_item_2;
iter = _stack_item_1;
gh-115999: Add free-threaded specialization for FOR_ITER (#128798) Add free-threaded versions of existing specialization for FOR_ITER (list, tuples, fast range iterators and generators), without significantly affecting their thread-safety. (Iterating over shared lists/tuples/ranges should be fine like before. Reusing iterators between threads is not fine, like before. Sharing generators between threads is a recipe for significant crashes, like before.)
2025-03-12 16:21:46 +01:00
#ifndef Py_GIL_DISABLED
GH-132554: "Virtual" iterators (GH-132555) * FOR_ITER now pushes either the iterator and NULL or leaves the iterable and pushes tagged zero * NEXT_ITER uses the tagged int as the index into the sequence or, if TOS is NULL, iterates as before.
2025-05-27 15:59:45 +01:00
PyObject *list_o = PyStackRef_AsPyObjectBorrow(iter);
assert(Py_TYPE(list_o) == &PyList_Type);
if ((size_t)PyStackRef_UntagInt(null_or_index) >= (size_t)PyList_GET_SIZE(list_o)) {
GH-117457: Correct pystats uop "miss" counts (GH-117477)
2024-04-04 18:49:18 -04:00
UOP_STAT_INC(uopcode, miss);
GH-142718: JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. (GH-142762) JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. * Make sure that stack is in correct state at side exits with TOS cached values * Simplify choice of cached items for side exits
2025-12-15 19:01:29 +00:00
_tos_cache2 = null_or_index;
_tos_cache1 = iter;
_tos_cache0 = _stack_item_0;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(3);
GH-117457: Correct pystats uop "miss" counts (GH-117477)
2024-04-04 18:49:18 -04:00
JUMP_TO_JUMP_TARGET();
}
gh-115999: Add free-threaded specialization for FOR_ITER (#128798) Add free-threaded versions of existing specialization for FOR_ITER (list, tuples, fast range iterators and generators), without significantly affecting their thread-safety. (Iterating over shared lists/tuples/ranges should be fine like before. Reusing iterators between threads is not fine, like before. Sharing generators between threads is a recipe for significant crashes, like before.)
2025-03-12 16:21:46 +01:00
#endif
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_tos_cache2 = null_or_index;
_tos_cache1 = iter;
_tos_cache0 = _stack_item_0;
SET_CURRENT_CACHED_VALUES(3);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-106529: Split FOR_ITER_{LIST,TUPLE} into uops (#106696) Also rename `_ITER_EXHAUSTED_XXX` to `_IS_ITER_EXHAUSTED_XXX` to make it clear this is a test.
2023-07-13 17:27:35 -07:00
break;
}
gh-115999: Add free-threaded specialization for FOR_ITER (#128798) Add free-threaded versions of existing specialization for FOR_ITER (list, tuples, fast range iterators and generators), without significantly affecting their thread-safety. (Iterating over shared lists/tuples/ranges should be fine like before. Reusing iterators between threads is not fine, like before. Sharing generators between threads is a recipe for significant crashes, like before.)
2025-03-12 16:21:46 +01:00
/* _ITER_NEXT_LIST is not a viable micro-op for tier 2 because it is replaced */
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
case _ITER_NEXT_LIST_TIER_TWO_r23: {
CHECK_CURRENT_CACHED_VALUES(2);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-132554: "Virtual" iterators (GH-132555) * FOR_ITER now pushes either the iterator and NULL or leaves the iterable and pushes tagged zero * NEXT_ITER uses the tagged int as the index into the sequence or, if TOS is NULL, iterates as before.
2025-05-27 15:59:45 +01:00
_PyStackRef null_or_index;
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
_PyStackRef iter;
_PyStackRef next;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef _stack_item_0 = _tos_cache0;
_PyStackRef _stack_item_1 = _tos_cache1;
null_or_index = _stack_item_1;
iter = _stack_item_0;
GH-132554: "Virtual" iterators (GH-132555) * FOR_ITER now pushes either the iterator and NULL or leaves the iterable and pushes tagged zero * NEXT_ITER uses the tagged int as the index into the sequence or, if TOS is NULL, iterates as before.
2025-05-27 15:59:45 +01:00
PyObject *list_o = PyStackRef_AsPyObjectBorrow(iter);
assert(PyList_CheckExact(list_o));
gh-115999: Add free-threaded specialization for FOR_ITER (#128798) Add free-threaded versions of existing specialization for FOR_ITER (list, tuples, fast range iterators and generators), without significantly affecting their thread-safety. (Iterating over shared lists/tuples/ranges should be fine like before. Reusing iterators between threads is not fine, like before. Sharing generators between threads is a recipe for significant crashes, like before.)
2025-03-12 16:21:46 +01:00
#ifdef Py_GIL_DISABLED
GH-132554: "Virtual" iterators (GH-132555) * FOR_ITER now pushes either the iterator and NULL or leaves the iterable and pushes tagged zero * NEXT_ITER uses the tagged int as the index into the sequence or, if TOS is NULL, iterates as before.
2025-05-27 15:59:45 +01:00
assert(_Py_IsOwnedByCurrentThread((PyObject *)list_o) ||
_PyObject_GC_IS_SHARED(list_o));
gh-115999: Add free-threaded specialization for FOR_ITER (#128798) Add free-threaded versions of existing specialization for FOR_ITER (list, tuples, fast range iterators and generators), without significantly affecting their thread-safety. (Iterating over shared lists/tuples/ranges should be fine like before. Reusing iterators between threads is not fine, like before. Sharing generators between threads is a recipe for significant crashes, like before.)
2025-03-12 16:21:46 +01:00
STAT_INC(FOR_ITER, hit);
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
stack_pointer[0] = iter;
stack_pointer[1] = null_or_index;
stack_pointer += 2;
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
gh-115999: Add free-threaded specialization for FOR_ITER (#128798) Add free-threaded versions of existing specialization for FOR_ITER (list, tuples, fast range iterators and generators), without significantly affecting their thread-safety. (Iterating over shared lists/tuples/ranges should be fine like before. Reusing iterators between threads is not fine, like before. Sharing generators between threads is a recipe for significant crashes, like before.)
2025-03-12 16:21:46 +01:00
_PyFrame_SetStackPointer(frame, stack_pointer);
GH-132554: "Virtual" iterators (GH-132555) * FOR_ITER now pushes either the iterator and NULL or leaves the iterable and pushes tagged zero * NEXT_ITER uses the tagged int as the index into the sequence or, if TOS is NULL, iterates as before.
2025-05-27 15:59:45 +01:00
int result = _PyList_GetItemRefNoLock((PyListObject *)list_o, PyStackRef_UntagInt(null_or_index), &next);
gh-115999: Add free-threaded specialization for FOR_ITER (#128798) Add free-threaded versions of existing specialization for FOR_ITER (list, tuples, fast range iterators and generators), without significantly affecting their thread-safety. (Iterating over shared lists/tuples/ranges should be fine like before. Reusing iterators between threads is not fine, like before. Sharing generators between threads is a recipe for significant crashes, like before.)
2025-03-12 16:21:46 +01:00
stack_pointer = _PyFrame_GetStackPointer(frame);
GH-132554: "Virtual" iterators (GH-132555) * FOR_ITER now pushes either the iterator and NULL or leaves the iterable and pushes tagged zero * NEXT_ITER uses the tagged int as the index into the sequence or, if TOS is NULL, iterates as before.
2025-05-27 15:59:45 +01:00
if (result <= 0) {
gh-115999: Add free-threaded specialization for FOR_ITER (#128798) Add free-threaded versions of existing specialization for FOR_ITER (list, tuples, fast range iterators and generators), without significantly affecting their thread-safety. (Iterating over shared lists/tuples/ranges should be fine like before. Reusing iterators between threads is not fine, like before. Sharing generators between threads is a recipe for significant crashes, like before.)
2025-03-12 16:21:46 +01:00
UOP_STAT_INC(uopcode, miss);
GH-142718: JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. (GH-142762) JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. * Make sure that stack is in correct state at side exits with TOS cached values * Simplify choice of cached items for side exits
2025-12-15 19:01:29 +00:00
_tos_cache1 = null_or_index;
_tos_cache0 = iter;
SET_CURRENT_CACHED_VALUES(2);
stack_pointer += -2;
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
gh-115999: Add free-threaded specialization for FOR_ITER (#128798) Add free-threaded versions of existing specialization for FOR_ITER (list, tuples, fast range iterators and generators), without significantly affecting their thread-safety. (Iterating over shared lists/tuples/ranges should be fine like before. Reusing iterators between threads is not fine, like before. Sharing generators between threads is a recipe for significant crashes, like before.)
2025-03-12 16:21:46 +01:00
JUMP_TO_JUMP_TARGET();
}
#else
GH-132554: "Virtual" iterators (GH-132555) * FOR_ITER now pushes either the iterator and NULL or leaves the iterable and pushes tagged zero * NEXT_ITER uses the tagged int as the index into the sequence or, if TOS is NULL, iterates as before.
2025-05-27 15:59:45 +01:00
assert(PyStackRef_UntagInt(null_or_index) < PyList_GET_SIZE(list_o));
next = PyStackRef_FromPyObjectNew(PyList_GET_ITEM(list_o, PyStackRef_UntagInt(null_or_index)));
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
stack_pointer += 2;
gh-115999: Add free-threaded specialization for FOR_ITER (#128798) Add free-threaded versions of existing specialization for FOR_ITER (list, tuples, fast range iterators and generators), without significantly affecting their thread-safety. (Iterating over shared lists/tuples/ranges should be fine like before. Reusing iterators between threads is not fine, like before. Sharing generators between threads is a recipe for significant crashes, like before.)
2025-03-12 16:21:46 +01:00
#endif
GH-132554: "Virtual" iterators (GH-132555) * FOR_ITER now pushes either the iterator and NULL or leaves the iterable and pushes tagged zero * NEXT_ITER uses the tagged int as the index into the sequence or, if TOS is NULL, iterates as before.
2025-05-27 15:59:45 +01:00
null_or_index = PyStackRef_IncrementTaggedIntNoOverflow(null_or_index);
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_tos_cache2 = next;
_tos_cache1 = null_or_index;
_tos_cache0 = iter;
SET_CURRENT_CACHED_VALUES(3);
stack_pointer += -2;
GH-141794: Limit size of generated machine code. (GH-142228) * Factor out bodies of the largest uops, to reduce jit code size. * Factor out common assert, also reducing jit code size. * Limit size of jitted code for a single executor to 1MB.
2025-12-03 17:43:35 +00:00
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-106529: Split FOR_ITER_{LIST,TUPLE} into uops (#106696) Also rename `_ITER_EXHAUSTED_XXX` to `_IS_ITER_EXHAUSTED_XXX` to make it clear this is a test.
2023-07-13 17:27:35 -07:00
break;
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
case _ITER_CHECK_TUPLE_r02: {
CHECK_CURRENT_CACHED_VALUES(0);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-132554: "Virtual" iterators (GH-132555) * FOR_ITER now pushes either the iterator and NULL or leaves the iterable and pushes tagged zero * NEXT_ITER uses the tagged int as the index into the sequence or, if TOS is NULL, iterates as before.
2025-05-27 15:59:45 +01:00
_PyStackRef null_or_index;
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
_PyStackRef iter;
GH-132554: "Virtual" iterators (GH-132555) * FOR_ITER now pushes either the iterator and NULL or leaves the iterable and pushes tagged zero * NEXT_ITER uses the tagged int as the index into the sequence or, if TOS is NULL, iterates as before.
2025-05-27 15:59:45 +01:00
null_or_index = stack_pointer[-1];
iter = stack_pointer[-2];
gh-115999: Add free-threaded specialization for FOR_ITER (#128798) Add free-threaded versions of existing specialization for FOR_ITER (list, tuples, fast range iterators and generators), without significantly affecting their thread-safety. (Iterating over shared lists/tuples/ranges should be fine like before. Reusing iterators between threads is not fine, like before. Sharing generators between threads is a recipe for significant crashes, like before.)
2025-03-12 16:21:46 +01:00
PyObject *iter_o = PyStackRef_AsPyObjectBorrow(iter);
GH-132554: "Virtual" iterators (GH-132555) * FOR_ITER now pushes either the iterator and NULL or leaves the iterable and pushes tagged zero * NEXT_ITER uses the tagged int as the index into the sequence or, if TOS is NULL, iterates as before.
2025-05-27 15:59:45 +01:00
if (Py_TYPE(iter_o) != &PyTuple_Type) {
GH-117457: Correct pystats uop "miss" counts (GH-117477)
2024-04-04 18:49:18 -04:00
UOP_STAT_INC(uopcode, miss);
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(0);
JUMP_TO_JUMP_TARGET();
}
assert(PyStackRef_IsTaggedInt(null_or_index));
_tos_cache1 = null_or_index;
_tos_cache0 = iter;
SET_CURRENT_CACHED_VALUES(2);
stack_pointer += -2;
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _ITER_CHECK_TUPLE_r12: {
CHECK_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef null_or_index;
_PyStackRef iter;
_PyStackRef _stack_item_0 = _tos_cache0;
null_or_index = _stack_item_0;
iter = stack_pointer[-1];
PyObject *iter_o = PyStackRef_AsPyObjectBorrow(iter);
if (Py_TYPE(iter_o) != &PyTuple_Type) {
UOP_STAT_INC(uopcode, miss);
GH-142718: JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. (GH-142762) JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. * Make sure that stack is in correct state at side exits with TOS cached values * Simplify choice of cached items for side exits
2025-12-15 19:01:29 +00:00
_tos_cache0 = null_or_index;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(1);
JUMP_TO_JUMP_TARGET();
}
assert(PyStackRef_IsTaggedInt(null_or_index));
_tos_cache1 = null_or_index;
_tos_cache0 = iter;
SET_CURRENT_CACHED_VALUES(2);
stack_pointer += -1;
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _ITER_CHECK_TUPLE_r22: {
CHECK_CURRENT_CACHED_VALUES(2);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef null_or_index;
_PyStackRef iter;
_PyStackRef _stack_item_0 = _tos_cache0;
_PyStackRef _stack_item_1 = _tos_cache1;
null_or_index = _stack_item_1;
iter = _stack_item_0;
PyObject *iter_o = PyStackRef_AsPyObjectBorrow(iter);
if (Py_TYPE(iter_o) != &PyTuple_Type) {
UOP_STAT_INC(uopcode, miss);
GH-142718: JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. (GH-142762) JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. * Make sure that stack is in correct state at side exits with TOS cached values * Simplify choice of cached items for side exits
2025-12-15 19:01:29 +00:00
_tos_cache1 = null_or_index;
_tos_cache0 = iter;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(2);
JUMP_TO_JUMP_TARGET();
}
assert(PyStackRef_IsTaggedInt(null_or_index));
_tos_cache1 = null_or_index;
_tos_cache0 = iter;
SET_CURRENT_CACHED_VALUES(2);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _ITER_CHECK_TUPLE_r33: {
CHECK_CURRENT_CACHED_VALUES(3);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef null_or_index;
_PyStackRef iter;
_PyStackRef _stack_item_0 = _tos_cache0;
_PyStackRef _stack_item_1 = _tos_cache1;
_PyStackRef _stack_item_2 = _tos_cache2;
null_or_index = _stack_item_2;
iter = _stack_item_1;
PyObject *iter_o = PyStackRef_AsPyObjectBorrow(iter);
if (Py_TYPE(iter_o) != &PyTuple_Type) {
UOP_STAT_INC(uopcode, miss);
GH-142718: JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. (GH-142762) JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. * Make sure that stack is in correct state at side exits with TOS cached values * Simplify choice of cached items for side exits
2025-12-15 19:01:29 +00:00
_tos_cache2 = null_or_index;
_tos_cache1 = iter;
_tos_cache0 = _stack_item_0;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(3);
GH-117457: Correct pystats uop "miss" counts (GH-117477)
2024-04-04 18:49:18 -04:00
JUMP_TO_JUMP_TARGET();
}
GH-132554: "Virtual" iterators (GH-132555) * FOR_ITER now pushes either the iterator and NULL or leaves the iterable and pushes tagged zero * NEXT_ITER uses the tagged int as the index into the sequence or, if TOS is NULL, iterates as before.
2025-05-27 15:59:45 +01:00
assert(PyStackRef_IsTaggedInt(null_or_index));
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_tos_cache2 = null_or_index;
_tos_cache1 = iter;
_tos_cache0 = _stack_item_0;
SET_CURRENT_CACHED_VALUES(3);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-106529: Split FOR_ITER_{LIST,TUPLE} into uops (#106696) Also rename `_ITER_EXHAUSTED_XXX` to `_IS_ITER_EXHAUSTED_XXX` to make it clear this is a test.
2023-07-13 17:27:35 -07:00
break;
}
GH-116422: Tier2 hot/cold splitting (GH-116813) Splits the "cold" path, deopts and exits, from the "hot" path, reducing the size of most jitted instructions, at the cost of slower exits.
2024-03-26 09:35:11 +00:00
/* _ITER_JUMP_TUPLE is not a viable micro-op for tier 2 because it is replaced */
GH-111485: Factor out tier 2 code generation from the rest of the interpreter code generator (GH-112968)
2023-12-12 12:12:17 +00:00
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
case _GUARD_NOT_EXHAUSTED_TUPLE_r02: {
CHECK_CURRENT_CACHED_VALUES(0);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-132554: "Virtual" iterators (GH-132555) * FOR_ITER now pushes either the iterator and NULL or leaves the iterable and pushes tagged zero * NEXT_ITER uses the tagged int as the index into the sequence or, if TOS is NULL, iterates as before.
2025-05-27 15:59:45 +01:00
_PyStackRef null_or_index;
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
_PyStackRef iter;
GH-132554: "Virtual" iterators (GH-132555) * FOR_ITER now pushes either the iterator and NULL or leaves the iterable and pushes tagged zero * NEXT_ITER uses the tagged int as the index into the sequence or, if TOS is NULL, iterates as before.
2025-05-27 15:59:45 +01:00
null_or_index = stack_pointer[-1];
iter = stack_pointer[-2];
PyObject *tuple_o = PyStackRef_AsPyObjectBorrow(iter);
assert(Py_TYPE(tuple_o) == &PyTuple_Type);
if ((size_t)PyStackRef_UntagInt(null_or_index) >= (size_t)PyTuple_GET_SIZE(tuple_o)) {
GH-117457: Correct pystats uop "miss" counts (GH-117477)
2024-04-04 18:49:18 -04:00
UOP_STAT_INC(uopcode, miss);
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(0);
JUMP_TO_JUMP_TARGET();
}
_tos_cache1 = null_or_index;
_tos_cache0 = iter;
SET_CURRENT_CACHED_VALUES(2);
stack_pointer += -2;
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _GUARD_NOT_EXHAUSTED_TUPLE_r12: {
CHECK_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef null_or_index;
_PyStackRef iter;
_PyStackRef _stack_item_0 = _tos_cache0;
null_or_index = _stack_item_0;
iter = stack_pointer[-1];
PyObject *tuple_o = PyStackRef_AsPyObjectBorrow(iter);
assert(Py_TYPE(tuple_o) == &PyTuple_Type);
if ((size_t)PyStackRef_UntagInt(null_or_index) >= (size_t)PyTuple_GET_SIZE(tuple_o)) {
UOP_STAT_INC(uopcode, miss);
GH-142718: JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. (GH-142762) JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. * Make sure that stack is in correct state at side exits with TOS cached values * Simplify choice of cached items for side exits
2025-12-15 19:01:29 +00:00
_tos_cache0 = null_or_index;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(1);
JUMP_TO_JUMP_TARGET();
}
_tos_cache1 = null_or_index;
_tos_cache0 = iter;
SET_CURRENT_CACHED_VALUES(2);
stack_pointer += -1;
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _GUARD_NOT_EXHAUSTED_TUPLE_r22: {
CHECK_CURRENT_CACHED_VALUES(2);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef null_or_index;
_PyStackRef iter;
_PyStackRef _stack_item_0 = _tos_cache0;
_PyStackRef _stack_item_1 = _tos_cache1;
null_or_index = _stack_item_1;
iter = _stack_item_0;
PyObject *tuple_o = PyStackRef_AsPyObjectBorrow(iter);
assert(Py_TYPE(tuple_o) == &PyTuple_Type);
if ((size_t)PyStackRef_UntagInt(null_or_index) >= (size_t)PyTuple_GET_SIZE(tuple_o)) {
UOP_STAT_INC(uopcode, miss);
GH-142718: JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. (GH-142762) JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. * Make sure that stack is in correct state at side exits with TOS cached values * Simplify choice of cached items for side exits
2025-12-15 19:01:29 +00:00
_tos_cache1 = null_or_index;
_tos_cache0 = iter;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(2);
JUMP_TO_JUMP_TARGET();
}
_tos_cache1 = null_or_index;
_tos_cache0 = iter;
SET_CURRENT_CACHED_VALUES(2);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _GUARD_NOT_EXHAUSTED_TUPLE_r33: {
CHECK_CURRENT_CACHED_VALUES(3);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef null_or_index;
_PyStackRef iter;
_PyStackRef _stack_item_0 = _tos_cache0;
_PyStackRef _stack_item_1 = _tos_cache1;
_PyStackRef _stack_item_2 = _tos_cache2;
null_or_index = _stack_item_2;
iter = _stack_item_1;
PyObject *tuple_o = PyStackRef_AsPyObjectBorrow(iter);
assert(Py_TYPE(tuple_o) == &PyTuple_Type);
if ((size_t)PyStackRef_UntagInt(null_or_index) >= (size_t)PyTuple_GET_SIZE(tuple_o)) {
UOP_STAT_INC(uopcode, miss);
GH-142718: JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. (GH-142762) JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. * Make sure that stack is in correct state at side exits with TOS cached values * Simplify choice of cached items for side exits
2025-12-15 19:01:29 +00:00
_tos_cache2 = null_or_index;
_tos_cache1 = iter;
_tos_cache0 = _stack_item_0;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(3);
GH-117457: Correct pystats uop "miss" counts (GH-117477)
2024-04-04 18:49:18 -04:00
JUMP_TO_JUMP_TARGET();
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_tos_cache2 = null_or_index;
_tos_cache1 = iter;
_tos_cache0 = _stack_item_0;
SET_CURRENT_CACHED_VALUES(3);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-106529: Split FOR_ITER_{LIST,TUPLE} into uops (#106696) Also rename `_ITER_EXHAUSTED_XXX` to `_IS_ITER_EXHAUSTED_XXX` to make it clear this is a test.
2023-07-13 17:27:35 -07:00
break;
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
case _ITER_NEXT_TUPLE_r03: {
CHECK_CURRENT_CACHED_VALUES(0);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-132554: "Virtual" iterators (GH-132555) * FOR_ITER now pushes either the iterator and NULL or leaves the iterable and pushes tagged zero * NEXT_ITER uses the tagged int as the index into the sequence or, if TOS is NULL, iterates as before.
2025-05-27 15:59:45 +01:00
_PyStackRef null_or_index;
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
_PyStackRef iter;
_PyStackRef next;
GH-132554: "Virtual" iterators (GH-132555) * FOR_ITER now pushes either the iterator and NULL or leaves the iterable and pushes tagged zero * NEXT_ITER uses the tagged int as the index into the sequence or, if TOS is NULL, iterates as before.
2025-05-27 15:59:45 +01:00
null_or_index = stack_pointer[-1];
iter = stack_pointer[-2];
PyObject *tuple_o = PyStackRef_AsPyObjectBorrow(iter);
assert(Py_TYPE(tuple_o) == &PyTuple_Type);
uintptr_t i = PyStackRef_UntagInt(null_or_index);
assert((size_t)i < (size_t)PyTuple_GET_SIZE(tuple_o));
next = PyStackRef_FromPyObjectNew(PyTuple_GET_ITEM(tuple_o, i));
null_or_index = PyStackRef_IncrementTaggedIntNoOverflow(null_or_index);
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_tos_cache2 = next;
_tos_cache1 = null_or_index;
_tos_cache0 = iter;
SET_CURRENT_CACHED_VALUES(3);
stack_pointer += -2;
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _ITER_NEXT_TUPLE_r13: {
CHECK_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef null_or_index;
_PyStackRef iter;
_PyStackRef next;
_PyStackRef _stack_item_0 = _tos_cache0;
null_or_index = _stack_item_0;
iter = stack_pointer[-1];
PyObject *tuple_o = PyStackRef_AsPyObjectBorrow(iter);
assert(Py_TYPE(tuple_o) == &PyTuple_Type);
uintptr_t i = PyStackRef_UntagInt(null_or_index);
assert((size_t)i < (size_t)PyTuple_GET_SIZE(tuple_o));
next = PyStackRef_FromPyObjectNew(PyTuple_GET_ITEM(tuple_o, i));
null_or_index = PyStackRef_IncrementTaggedIntNoOverflow(null_or_index);
_tos_cache2 = next;
_tos_cache1 = null_or_index;
_tos_cache0 = iter;
SET_CURRENT_CACHED_VALUES(3);
stack_pointer += -1;
GH-141794: Limit size of generated machine code. (GH-142228) * Factor out bodies of the largest uops, to reduce jit code size. * Factor out common assert, also reducing jit code size. * Limit size of jitted code for a single executor to 1MB.
2025-12-03 17:43:35 +00:00
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _ITER_NEXT_TUPLE_r23: {
CHECK_CURRENT_CACHED_VALUES(2);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef null_or_index;
_PyStackRef iter;
_PyStackRef next;
_PyStackRef _stack_item_0 = _tos_cache0;
_PyStackRef _stack_item_1 = _tos_cache1;
null_or_index = _stack_item_1;
iter = _stack_item_0;
PyObject *tuple_o = PyStackRef_AsPyObjectBorrow(iter);
assert(Py_TYPE(tuple_o) == &PyTuple_Type);
uintptr_t i = PyStackRef_UntagInt(null_or_index);
assert((size_t)i < (size_t)PyTuple_GET_SIZE(tuple_o));
next = PyStackRef_FromPyObjectNew(PyTuple_GET_ITEM(tuple_o, i));
null_or_index = PyStackRef_IncrementTaggedIntNoOverflow(null_or_index);
_tos_cache2 = next;
_tos_cache1 = null_or_index;
_tos_cache0 = iter;
SET_CURRENT_CACHED_VALUES(3);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-106529: Split FOR_ITER_{LIST,TUPLE} into uops (#106696) Also rename `_ITER_EXHAUSTED_XXX` to `_IS_ITER_EXHAUSTED_XXX` to make it clear this is a test.
2023-07-13 17:27:35 -07:00
break;
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
case _ITER_CHECK_RANGE_r02: {
CHECK_CURRENT_CACHED_VALUES(0);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
_PyStackRef iter;
GH-132554: "Virtual" iterators (GH-132555) * FOR_ITER now pushes either the iterator and NULL or leaves the iterable and pushes tagged zero * NEXT_ITER uses the tagged int as the index into the sequence or, if TOS is NULL, iterates as before.
2025-05-27 15:59:45 +01:00
iter = stack_pointer[-2];
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
_PyRangeIterObject *r = (_PyRangeIterObject *)PyStackRef_AsPyObjectBorrow(iter);
GH-117457: Correct pystats uop "miss" counts (GH-117477)
2024-04-04 18:49:18 -04:00
if (Py_TYPE(r) != &PyRangeIter_Type) {
UOP_STAT_INC(uopcode, miss);
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(0);
JUMP_TO_JUMP_TARGET();
}
#ifdef Py_GIL_DISABLED
if (!_PyObject_IsUniquelyReferenced((PyObject *)r)) {
UOP_STAT_INC(uopcode, miss);
SET_CURRENT_CACHED_VALUES(0);
JUMP_TO_JUMP_TARGET();
}
#endif
_tos_cache1 = stack_pointer[-1];
_tos_cache0 = iter;
SET_CURRENT_CACHED_VALUES(2);
stack_pointer += -2;
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _ITER_CHECK_RANGE_r12: {
CHECK_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef iter;
_PyStackRef _stack_item_0 = _tos_cache0;
iter = stack_pointer[-1];
_PyRangeIterObject *r = (_PyRangeIterObject *)PyStackRef_AsPyObjectBorrow(iter);
if (Py_TYPE(r) != &PyRangeIter_Type) {
UOP_STAT_INC(uopcode, miss);
GH-142718: JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. (GH-142762) JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. * Make sure that stack is in correct state at side exits with TOS cached values * Simplify choice of cached items for side exits
2025-12-15 19:01:29 +00:00
_tos_cache0 = _stack_item_0;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(1);
JUMP_TO_JUMP_TARGET();
}
#ifdef Py_GIL_DISABLED
if (!_PyObject_IsUniquelyReferenced((PyObject *)r)) {
UOP_STAT_INC(uopcode, miss);
GH-142718: JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. (GH-142762) JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. * Make sure that stack is in correct state at side exits with TOS cached values * Simplify choice of cached items for side exits
2025-12-15 19:01:29 +00:00
_tos_cache0 = _stack_item_0;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(1);
JUMP_TO_JUMP_TARGET();
}
#endif
_tos_cache1 = _stack_item_0;
_tos_cache0 = iter;
SET_CURRENT_CACHED_VALUES(2);
stack_pointer += -1;
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _ITER_CHECK_RANGE_r22: {
CHECK_CURRENT_CACHED_VALUES(2);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef iter;
_PyStackRef _stack_item_0 = _tos_cache0;
_PyStackRef _stack_item_1 = _tos_cache1;
iter = _stack_item_0;
_PyRangeIterObject *r = (_PyRangeIterObject *)PyStackRef_AsPyObjectBorrow(iter);
if (Py_TYPE(r) != &PyRangeIter_Type) {
UOP_STAT_INC(uopcode, miss);
GH-142718: JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. (GH-142762) JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. * Make sure that stack is in correct state at side exits with TOS cached values * Simplify choice of cached items for side exits
2025-12-15 19:01:29 +00:00
_tos_cache1 = _stack_item_1;
_tos_cache0 = iter;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(2);
JUMP_TO_JUMP_TARGET();
}
#ifdef Py_GIL_DISABLED
if (!_PyObject_IsUniquelyReferenced((PyObject *)r)) {
UOP_STAT_INC(uopcode, miss);
GH-142718: JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. (GH-142762) JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. * Make sure that stack is in correct state at side exits with TOS cached values * Simplify choice of cached items for side exits
2025-12-15 19:01:29 +00:00
_tos_cache1 = _stack_item_1;
_tos_cache0 = iter;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(2);
JUMP_TO_JUMP_TARGET();
}
#endif
_tos_cache1 = _stack_item_1;
_tos_cache0 = iter;
SET_CURRENT_CACHED_VALUES(2);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _ITER_CHECK_RANGE_r33: {
CHECK_CURRENT_CACHED_VALUES(3);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef iter;
_PyStackRef _stack_item_0 = _tos_cache0;
_PyStackRef _stack_item_1 = _tos_cache1;
_PyStackRef _stack_item_2 = _tos_cache2;
iter = _stack_item_1;
_PyRangeIterObject *r = (_PyRangeIterObject *)PyStackRef_AsPyObjectBorrow(iter);
if (Py_TYPE(r) != &PyRangeIter_Type) {
UOP_STAT_INC(uopcode, miss);
GH-142718: JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. (GH-142762) JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. * Make sure that stack is in correct state at side exits with TOS cached values * Simplify choice of cached items for side exits
2025-12-15 19:01:29 +00:00
_tos_cache2 = _stack_item_2;
_tos_cache1 = iter;
_tos_cache0 = _stack_item_0;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(3);
GH-117457: Correct pystats uop "miss" counts (GH-117477)
2024-04-04 18:49:18 -04:00
JUMP_TO_JUMP_TARGET();
}
gh-115999: Add free-threaded specialization for FOR_ITER (#128798) Add free-threaded versions of existing specialization for FOR_ITER (list, tuples, fast range iterators and generators), without significantly affecting their thread-safety. (Iterating over shared lists/tuples/ranges should be fine like before. Reusing iterators between threads is not fine, like before. Sharing generators between threads is a recipe for significant crashes, like before.)
2025-03-12 16:21:46 +01:00
#ifdef Py_GIL_DISABLED
if (!_PyObject_IsUniquelyReferenced((PyObject *)r)) {
UOP_STAT_INC(uopcode, miss);
GH-142718: JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. (GH-142762) JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. * Make sure that stack is in correct state at side exits with TOS cached values * Simplify choice of cached items for side exits
2025-12-15 19:01:29 +00:00
_tos_cache2 = _stack_item_2;
_tos_cache1 = iter;
_tos_cache0 = _stack_item_0;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(3);
gh-115999: Add free-threaded specialization for FOR_ITER (#128798) Add free-threaded versions of existing specialization for FOR_ITER (list, tuples, fast range iterators and generators), without significantly affecting their thread-safety. (Iterating over shared lists/tuples/ranges should be fine like before. Reusing iterators between threads is not fine, like before. Sharing generators between threads is a recipe for significant crashes, like before.)
2025-03-12 16:21:46 +01:00
JUMP_TO_JUMP_TARGET();
}
#endif
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_tos_cache2 = _stack_item_2;
_tos_cache1 = iter;
_tos_cache0 = _stack_item_0;
SET_CURRENT_CACHED_VALUES(3);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-106529: Split FOR_ITER_RANGE into uops (#106638) For an example of what this does for Tier 1 and Tier 2, see https://github.com/python/cpython/issues/106529#issuecomment-1631649920
2023-07-12 10:23:59 -07:00
break;
}
GH-116422: Tier2 hot/cold splitting (GH-116813) Splits the "cold" path, deopts and exits, from the "hot" path, reducing the size of most jitted instructions, at the cost of slower exits.
2024-03-26 09:35:11 +00:00
/* _ITER_JUMP_RANGE is not a viable micro-op for tier 2 because it is replaced */
GH-111485: Factor out tier 2 code generation from the rest of the interpreter code generator (GH-112968)
2023-12-12 12:12:17 +00:00
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
case _GUARD_NOT_EXHAUSTED_RANGE_r02: {
CHECK_CURRENT_CACHED_VALUES(0);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
_PyStackRef iter;
GH-132554: "Virtual" iterators (GH-132555) * FOR_ITER now pushes either the iterator and NULL or leaves the iterable and pushes tagged zero * NEXT_ITER uses the tagged int as the index into the sequence or, if TOS is NULL, iterates as before.
2025-05-27 15:59:45 +01:00
iter = stack_pointer[-2];
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
_PyRangeIterObject *r = (_PyRangeIterObject *)PyStackRef_AsPyObjectBorrow(iter);
gh-106529: Split FOR_ITER_RANGE into uops (#106638) For an example of what this does for Tier 1 and Tier 2, see https://github.com/python/cpython/issues/106529#issuecomment-1631649920
2023-07-12 10:23:59 -07:00
assert(Py_TYPE(r) == &PyRangeIter_Type);
GH-117457: Correct pystats uop "miss" counts (GH-117477)
2024-04-04 18:49:18 -04:00
if (r->len <= 0) {
UOP_STAT_INC(uopcode, miss);
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(0);
JUMP_TO_JUMP_TARGET();
}
_tos_cache1 = stack_pointer[-1];
_tos_cache0 = iter;
SET_CURRENT_CACHED_VALUES(2);
stack_pointer += -2;
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _GUARD_NOT_EXHAUSTED_RANGE_r12: {
CHECK_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef iter;
_PyStackRef _stack_item_0 = _tos_cache0;
iter = stack_pointer[-1];
_PyRangeIterObject *r = (_PyRangeIterObject *)PyStackRef_AsPyObjectBorrow(iter);
assert(Py_TYPE(r) == &PyRangeIter_Type);
if (r->len <= 0) {
UOP_STAT_INC(uopcode, miss);
GH-142718: JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. (GH-142762) JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. * Make sure that stack is in correct state at side exits with TOS cached values * Simplify choice of cached items for side exits
2025-12-15 19:01:29 +00:00
_tos_cache0 = _stack_item_0;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(1);
JUMP_TO_JUMP_TARGET();
}
_tos_cache1 = _stack_item_0;
_tos_cache0 = iter;
SET_CURRENT_CACHED_VALUES(2);
stack_pointer += -1;
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _GUARD_NOT_EXHAUSTED_RANGE_r22: {
CHECK_CURRENT_CACHED_VALUES(2);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef iter;
_PyStackRef _stack_item_0 = _tos_cache0;
_PyStackRef _stack_item_1 = _tos_cache1;
iter = _stack_item_0;
_PyRangeIterObject *r = (_PyRangeIterObject *)PyStackRef_AsPyObjectBorrow(iter);
assert(Py_TYPE(r) == &PyRangeIter_Type);
if (r->len <= 0) {
UOP_STAT_INC(uopcode, miss);
GH-142718: JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. (GH-142762) JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. * Make sure that stack is in correct state at side exits with TOS cached values * Simplify choice of cached items for side exits
2025-12-15 19:01:29 +00:00
_tos_cache1 = _stack_item_1;
_tos_cache0 = iter;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(2);
JUMP_TO_JUMP_TARGET();
}
_tos_cache1 = _stack_item_1;
_tos_cache0 = iter;
SET_CURRENT_CACHED_VALUES(2);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _GUARD_NOT_EXHAUSTED_RANGE_r33: {
CHECK_CURRENT_CACHED_VALUES(3);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef iter;
_PyStackRef _stack_item_0 = _tos_cache0;
_PyStackRef _stack_item_1 = _tos_cache1;
_PyStackRef _stack_item_2 = _tos_cache2;
iter = _stack_item_1;
_PyRangeIterObject *r = (_PyRangeIterObject *)PyStackRef_AsPyObjectBorrow(iter);
assert(Py_TYPE(r) == &PyRangeIter_Type);
if (r->len <= 0) {
UOP_STAT_INC(uopcode, miss);
GH-142718: JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. (GH-142762) JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. * Make sure that stack is in correct state at side exits with TOS cached values * Simplify choice of cached items for side exits
2025-12-15 19:01:29 +00:00
_tos_cache2 = _stack_item_2;
_tos_cache1 = iter;
_tos_cache0 = _stack_item_0;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(3);
GH-117457: Correct pystats uop "miss" counts (GH-117477)
2024-04-04 18:49:18 -04:00
JUMP_TO_JUMP_TARGET();
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_tos_cache2 = _stack_item_2;
_tos_cache1 = iter;
_tos_cache0 = _stack_item_0;
SET_CURRENT_CACHED_VALUES(3);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-106529: Split FOR_ITER_RANGE into uops (#106638) For an example of what this does for Tier 1 and Tier 2, see https://github.com/python/cpython/issues/106529#issuecomment-1631649920
2023-07-12 10:23:59 -07:00
break;
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
case _ITER_NEXT_RANGE_r03: {
CHECK_CURRENT_CACHED_VALUES(0);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
_PyStackRef iter;
_PyStackRef next;
GH-132554: "Virtual" iterators (GH-132555) * FOR_ITER now pushes either the iterator and NULL or leaves the iterable and pushes tagged zero * NEXT_ITER uses the tagged int as the index into the sequence or, if TOS is NULL, iterates as before.
2025-05-27 15:59:45 +01:00
iter = stack_pointer[-2];
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
_PyRangeIterObject *r = (_PyRangeIterObject *)PyStackRef_AsPyObjectBorrow(iter);
gh-106529: Split FOR_ITER_RANGE into uops (#106638) For an example of what this does for Tier 1 and Tier 2, see https://github.com/python/cpython/issues/106529#issuecomment-1631649920
2023-07-12 10:23:59 -07:00
assert(Py_TYPE(r) == &PyRangeIter_Type);
gh-115999: Add free-threaded specialization for FOR_ITER (#128798) Add free-threaded versions of existing specialization for FOR_ITER (list, tuples, fast range iterators and generators), without significantly affecting their thread-safety. (Iterating over shared lists/tuples/ranges should be fine like before. Reusing iterators between threads is not fine, like before. Sharing generators between threads is a recipe for significant crashes, like before.)
2025-03-12 16:21:46 +01:00
#ifdef Py_GIL_DISABLED
assert(_PyObject_IsUniquelyReferenced((PyObject *)r));
#endif
gh-106529: Split FOR_ITER_RANGE into uops (#106638) For an example of what this does for Tier 1 and Tier 2, see https://github.com/python/cpython/issues/106529#issuecomment-1631649920
2023-07-12 10:23:59 -07:00
assert(r->len > 0);
long value = r->start;
r->start = value + r->step;
r->len--;
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
PyObject *res = PyLong_FromLong(value);
GH-128682: Make `PyStackRef_CLOSE` escaping. (GH-129404)
2025-02-03 12:41:32 +00:00
if (res == NULL) {
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(0);
JUMP_TO_ERROR();
}
next = PyStackRef_FromPyObjectSteal(res);
_tos_cache2 = next;
_tos_cache1 = stack_pointer[-1];
_tos_cache0 = iter;
SET_CURRENT_CACHED_VALUES(3);
stack_pointer += -2;
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _ITER_NEXT_RANGE_r13: {
CHECK_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef iter;
_PyStackRef next;
_PyStackRef _stack_item_0 = _tos_cache0;
iter = stack_pointer[-1];
_PyRangeIterObject *r = (_PyRangeIterObject *)PyStackRef_AsPyObjectBorrow(iter);
assert(Py_TYPE(r) == &PyRangeIter_Type);
#ifdef Py_GIL_DISABLED
assert(_PyObject_IsUniquelyReferenced((PyObject *)r));
#endif
assert(r->len > 0);
long value = r->start;
r->start = value + r->step;
r->len--;
PyObject *res = PyLong_FromLong(value);
if (res == NULL) {
stack_pointer[0] = _stack_item_0;
stack_pointer += 1;
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
SET_CURRENT_CACHED_VALUES(0);
JUMP_TO_ERROR();
}
next = PyStackRef_FromPyObjectSteal(res);
_tos_cache2 = next;
_tos_cache1 = _stack_item_0;
_tos_cache0 = iter;
SET_CURRENT_CACHED_VALUES(3);
stack_pointer += -1;
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _ITER_NEXT_RANGE_r23: {
CHECK_CURRENT_CACHED_VALUES(2);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef iter;
_PyStackRef next;
_PyStackRef _stack_item_0 = _tos_cache0;
_PyStackRef _stack_item_1 = _tos_cache1;
iter = _stack_item_0;
_PyRangeIterObject *r = (_PyRangeIterObject *)PyStackRef_AsPyObjectBorrow(iter);
assert(Py_TYPE(r) == &PyRangeIter_Type);
#ifdef Py_GIL_DISABLED
assert(_PyObject_IsUniquelyReferenced((PyObject *)r));
#endif
assert(r->len > 0);
long value = r->start;
r->start = value + r->step;
r->len--;
PyObject *res = PyLong_FromLong(value);
if (res == NULL) {
stack_pointer[0] = iter;
stack_pointer[1] = _stack_item_1;
stack_pointer += 2;
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
SET_CURRENT_CACHED_VALUES(0);
GH-128682: Make `PyStackRef_CLOSE` escaping. (GH-129404)
2025-02-03 12:41:32 +00:00
JUMP_TO_ERROR();
}
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
next = PyStackRef_FromPyObjectSteal(res);
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_tos_cache2 = next;
_tos_cache1 = _stack_item_1;
_tos_cache0 = iter;
SET_CURRENT_CACHED_VALUES(3);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-106529: Split FOR_ITER_RANGE into uops (#106638) For an example of what this does for Tier 1 and Tier 2, see https://github.com/python/cpython/issues/106529#issuecomment-1631649920
2023-07-12 10:23:59 -07:00
break;
}
gh-120321: Make gi_frame_state transitions atomic in FT build (gh-142599) This makes generator frame state transitions atomic in the free threading build, which avoids segfaults when trying to execute a generator from multiple threads concurrently. There are still a few operations that aren't thread-safe and may crash if performed concurrently on the same generator/coroutine: * Accessing gi_yieldfrom/cr_await/ag_await * Accessing gi_frame/cr_frame/ag_frame * Async generator operations
2025-12-19 14:10:37 -05:00
case _FOR_ITER_GEN_FRAME_r03: {
CHECK_CURRENT_CACHED_VALUES(0);
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
_PyStackRef iter;
_PyStackRef gen_frame;
oparg = CURRENT_OPARG();
iter = stack_pointer[-2];
PyGenObject *gen = (PyGenObject *)PyStackRef_AsPyObjectBorrow(iter);
if (Py_TYPE(gen) != &PyGen_Type) {
UOP_STAT_INC(uopcode, miss);
SET_CURRENT_CACHED_VALUES(0);
JUMP_TO_JUMP_TARGET();
}
if (!gen_try_set_executing((PyGenObject *)gen)) {
UOP_STAT_INC(uopcode, miss);
SET_CURRENT_CACHED_VALUES(0);
JUMP_TO_JUMP_TARGET();
}
STAT_INC(FOR_ITER, hit);
_PyInterpreterFrame *pushed_frame = &gen->gi_iframe;
_PyFrame_StackPush(pushed_frame, PyStackRef_None);
gen->gi_exc_state.previous_item = tstate->exc_info;
tstate->exc_info = &gen->gi_exc_state;
pushed_frame->previous = frame;
frame->return_offset = (uint16_t)( 2u + oparg);
gen_frame = PyStackRef_Wrap(pushed_frame);
_tos_cache2 = gen_frame;
_tos_cache1 = stack_pointer[-1];
_tos_cache0 = iter;
SET_CURRENT_CACHED_VALUES(3);
stack_pointer += -2;
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
break;
}
case _FOR_ITER_GEN_FRAME_r13: {
CHECK_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
_PyStackRef iter;
GH-135379: Remove types from stack items in code generator. (GH-135384) * Make casts explicit in the instruction definitions
2025-06-11 15:52:25 +01:00
_PyStackRef gen_frame;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef _stack_item_0 = _tos_cache0;
GH-118095: Add dynamic exit support and FOR_ITER_GEN support to tier 2 (GH-118279)
2024-04-26 18:08:50 +01:00
oparg = CURRENT_OPARG();
gh-120321: Make gi_frame_state transitions atomic in FT build (gh-142599) This makes generator frame state transitions atomic in the free threading build, which avoids segfaults when trying to execute a generator from multiple threads concurrently. There are still a few operations that aren't thread-safe and may crash if performed concurrently on the same generator/coroutine: * Accessing gi_yieldfrom/cr_await/ag_await * Accessing gi_frame/cr_frame/ag_frame * Async generator operations
2025-12-19 14:10:37 -05:00
iter = stack_pointer[-1];
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
PyGenObject *gen = (PyGenObject *)PyStackRef_AsPyObjectBorrow(iter);
GH-118095: Add dynamic exit support and FOR_ITER_GEN support to tier 2 (GH-118279)
2024-04-26 18:08:50 +01:00
if (Py_TYPE(gen) != &PyGen_Type) {
UOP_STAT_INC(uopcode, miss);
gh-120321: Make gi_frame_state transitions atomic in FT build (gh-142599) This makes generator frame state transitions atomic in the free threading build, which avoids segfaults when trying to execute a generator from multiple threads concurrently. There are still a few operations that aren't thread-safe and may crash if performed concurrently on the same generator/coroutine: * Accessing gi_yieldfrom/cr_await/ag_await * Accessing gi_frame/cr_frame/ag_frame * Async generator operations
2025-12-19 14:10:37 -05:00
_tos_cache0 = _stack_item_0;
SET_CURRENT_CACHED_VALUES(1);
GH-118095: Add dynamic exit support and FOR_ITER_GEN support to tier 2 (GH-118279)
2024-04-26 18:08:50 +01:00
JUMP_TO_JUMP_TARGET();
}
gh-120321: Make gi_frame_state transitions atomic in FT build (gh-142599) This makes generator frame state transitions atomic in the free threading build, which avoids segfaults when trying to execute a generator from multiple threads concurrently. There are still a few operations that aren't thread-safe and may crash if performed concurrently on the same generator/coroutine: * Accessing gi_yieldfrom/cr_await/ag_await * Accessing gi_frame/cr_frame/ag_frame * Async generator operations
2025-12-19 14:10:37 -05:00
if (!gen_try_set_executing((PyGenObject *)gen)) {
UOP_STAT_INC(uopcode, miss);
_tos_cache0 = _stack_item_0;
SET_CURRENT_CACHED_VALUES(1);
JUMP_TO_JUMP_TARGET();
}
STAT_INC(FOR_ITER, hit);
_PyInterpreterFrame *pushed_frame = &gen->gi_iframe;
_PyFrame_StackPush(pushed_frame, PyStackRef_None);
gen->gi_exc_state.previous_item = tstate->exc_info;
tstate->exc_info = &gen->gi_exc_state;
pushed_frame->previous = frame;
frame->return_offset = (uint16_t)( 2u + oparg);
gen_frame = PyStackRef_Wrap(pushed_frame);
_tos_cache2 = gen_frame;
_tos_cache1 = _stack_item_0;
_tos_cache0 = iter;
SET_CURRENT_CACHED_VALUES(3);
stack_pointer += -1;
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
break;
}
GH-131498: Cases generator: Parse down to C statement level. (GH-131948) * Parse down to statement level in the cases generator * Add handling for #if macros, treating them much like normal ifs.
2025-04-02 16:31:59 +01:00
gh-120321: Make gi_frame_state transitions atomic in FT build (gh-142599) This makes generator frame state transitions atomic in the free threading build, which avoids segfaults when trying to execute a generator from multiple threads concurrently. There are still a few operations that aren't thread-safe and may crash if performed concurrently on the same generator/coroutine: * Accessing gi_yieldfrom/cr_await/ag_await * Accessing gi_frame/cr_frame/ag_frame * Async generator operations
2025-12-19 14:10:37 -05:00
case _FOR_ITER_GEN_FRAME_r23: {
CHECK_CURRENT_CACHED_VALUES(2);
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
_PyStackRef iter;
_PyStackRef gen_frame;
_PyStackRef _stack_item_0 = _tos_cache0;
_PyStackRef _stack_item_1 = _tos_cache1;
oparg = CURRENT_OPARG();
iter = _stack_item_0;
PyGenObject *gen = (PyGenObject *)PyStackRef_AsPyObjectBorrow(iter);
if (Py_TYPE(gen) != &PyGen_Type) {
gh-115999: Add free-threaded specialization for FOR_ITER (#128798) Add free-threaded versions of existing specialization for FOR_ITER (list, tuples, fast range iterators and generators), without significantly affecting their thread-safety. (Iterating over shared lists/tuples/ranges should be fine like before. Reusing iterators between threads is not fine, like before. Sharing generators between threads is a recipe for significant crashes, like before.)
2025-03-12 16:21:46 +01:00
UOP_STAT_INC(uopcode, miss);
GH-142718: JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. (GH-142762) JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. * Make sure that stack is in correct state at side exits with TOS cached values * Simplify choice of cached items for side exits
2025-12-15 19:01:29 +00:00
_tos_cache1 = _stack_item_1;
_tos_cache0 = iter;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(2);
gh-115999: Add free-threaded specialization for FOR_ITER (#128798) Add free-threaded versions of existing specialization for FOR_ITER (list, tuples, fast range iterators and generators), without significantly affecting their thread-safety. (Iterating over shared lists/tuples/ranges should be fine like before. Reusing iterators between threads is not fine, like before. Sharing generators between threads is a recipe for significant crashes, like before.)
2025-03-12 16:21:46 +01:00
JUMP_TO_JUMP_TARGET();
}
gh-120321: Make gi_frame_state transitions atomic in FT build (gh-142599) This makes generator frame state transitions atomic in the free threading build, which avoids segfaults when trying to execute a generator from multiple threads concurrently. There are still a few operations that aren't thread-safe and may crash if performed concurrently on the same generator/coroutine: * Accessing gi_yieldfrom/cr_await/ag_await * Accessing gi_frame/cr_frame/ag_frame * Async generator operations
2025-12-19 14:10:37 -05:00
if (!gen_try_set_executing((PyGenObject *)gen)) {
GH-118095: Add dynamic exit support and FOR_ITER_GEN support to tier 2 (GH-118279)
2024-04-26 18:08:50 +01:00
UOP_STAT_INC(uopcode, miss);
GH-142718: JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. (GH-142762) JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. * Make sure that stack is in correct state at side exits with TOS cached values * Simplify choice of cached items for side exits
2025-12-15 19:01:29 +00:00
_tos_cache1 = _stack_item_1;
_tos_cache0 = iter;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(2);
GH-118095: Add dynamic exit support and FOR_ITER_GEN support to tier 2 (GH-118279)
2024-04-26 18:08:50 +01:00
JUMP_TO_JUMP_TARGET();
}
STAT_INC(FOR_ITER, hit);
GH-135379: Remove types from stack items in code generator. (GH-135384) * Make casts explicit in the instruction definitions
2025-06-11 15:52:25 +01:00
_PyInterpreterFrame *pushed_frame = &gen->gi_iframe;
_PyFrame_StackPush(pushed_frame, PyStackRef_None);
GH-118095: Add dynamic exit support and FOR_ITER_GEN support to tier 2 (GH-118279)
2024-04-26 18:08:50 +01:00
gen->gi_exc_state.previous_item = tstate->exc_info;
tstate->exc_info = &gen->gi_exc_state;
GH-135379: Remove types from stack items in code generator. (GH-135384) * Make casts explicit in the instruction definitions
2025-06-11 15:52:25 +01:00
pushed_frame->previous = frame;
gh-138264: Fix gcc 14 compiler warnings (GH-138265)
2025-09-03 11:26:56 +03:00
frame->return_offset = (uint16_t)( 2u + oparg);
GH-135379: Remove types from stack items in code generator. (GH-135384) * Make casts explicit in the instruction definitions
2025-06-11 15:52:25 +01:00
gen_frame = PyStackRef_Wrap(pushed_frame);
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_tos_cache2 = gen_frame;
_tos_cache1 = _stack_item_1;
_tos_cache0 = iter;
SET_CURRENT_CACHED_VALUES(3);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-118095: Add dynamic exit support and FOR_ITER_GEN support to tier 2 (GH-118279)
2024-04-26 18:08:50 +01:00
break;
}
GH-111485: Factor out tier 2 code generation from the rest of the interpreter code generator (GH-112968)
2023-12-12 12:12:17 +00:00
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
case _INSERT_NULL_r10: {
CHECK_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-131586: Avoid refcount contention in context managers (gh-131851) This avoid reference count contention in the free threading build when calling special methods like `__enter__` and `__exit__`.
2025-04-21 15:54:25 -04:00
_PyStackRef self;
_PyStackRef *method_and_self;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef _stack_item_0 = _tos_cache0;
self = _stack_item_0;
method_and_self = &stack_pointer[0];
gh-131586: Avoid refcount contention in context managers (gh-131851) This avoid reference count contention in the free threading build when calling special methods like `__enter__` and `__exit__`.
2025-04-21 15:54:25 -04:00
method_and_self[1] = self;
method_and_self[0] = PyStackRef_NULL;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(0);
stack_pointer += 2;
GH-141794: Limit size of generated machine code. (GH-142228) * Factor out bodies of the largest uops, to reduce jit code size. * Factor out common assert, also reducing jit code size. * Limit size of jitted code for a single executor to 1MB.
2025-12-03 17:43:35 +00:00
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-131586: Avoid refcount contention in context managers (gh-131851) This avoid reference count contention in the free threading build when calling special methods like `__enter__` and `__exit__`.
2025-04-21 15:54:25 -04:00
break;
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
case _LOAD_SPECIAL_r00: {
CHECK_CURRENT_CACHED_VALUES(0);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-131586: Avoid refcount contention in context managers (gh-131851) This avoid reference count contention in the free threading build when calling special methods like `__enter__` and `__exit__`.
2025-04-21 15:54:25 -04:00
_PyStackRef *method_and_self;
GH-120507: Lower the `BEFORE_WITH` and `BEFORE_ASYNC_WITH` instructions. (#120640) * Remove BEFORE_WITH and BEFORE_ASYNC_WITH instructions. * Add LOAD_SPECIAL instruction * Reimplement `with` and `async with` statements using LOAD_SPECIAL
2024-06-18 12:17:46 +01:00
oparg = CURRENT_OPARG();
gh-131586: Avoid refcount contention in context managers (gh-131851) This avoid reference count contention in the free threading build when calling special methods like `__enter__` and `__exit__`.
2025-04-21 15:54:25 -04:00
method_and_self = &stack_pointer[-2];
GH-120507: Lower the `BEFORE_WITH` and `BEFORE_ASYNC_WITH` instructions. (#120640) * Remove BEFORE_WITH and BEFORE_ASYNC_WITH instructions. * Add LOAD_SPECIAL instruction * Reimplement `with` and `async with` statements using LOAD_SPECIAL
2024-06-18 12:17:46 +01:00
PyObject *name = _Py_SpecialMethods[oparg].name;
GH-119866: Spill the stack around escaping calls. (GH-124392) * Spill the evaluation around escaping calls in the generated interpreter and JIT. * The code generator tracks live, cached values so they can be saved to memory when needed. * Spills the stack pointer around escaping calls, so that the exact stack is visible to the cycle GC.
2024-10-07 14:56:39 +01:00
_PyFrame_SetStackPointer(frame, stack_pointer);
gh-131586: Avoid refcount contention in context managers (gh-131851) This avoid reference count contention in the free threading build when calling special methods like `__enter__` and `__exit__`.
2025-04-21 15:54:25 -04:00
int err = _PyObject_LookupSpecialMethod(name, method_and_self);
GH-119866: Spill the stack around escaping calls. (GH-124392) * Spill the evaluation around escaping calls in the generated interpreter and JIT. * The code generator tracks live, cached values so they can be saved to memory when needed. * Spills the stack pointer around escaping calls, so that the exact stack is visible to the cycle GC.
2024-10-07 14:56:39 +01:00
stack_pointer = _PyFrame_GetStackPointer(frame);
gh-131586: Avoid refcount contention in context managers (gh-131851) This avoid reference count contention in the free threading build when calling special methods like `__enter__` and `__exit__`.
2025-04-21 15:54:25 -04:00
if (err <= 0) {
if (err == 0) {
PyObject *owner = PyStackRef_AsPyObjectBorrow(method_and_self[1]);
GH-119866: Spill the stack around escaping calls. (GH-124392) * Spill the evaluation around escaping calls in the generated interpreter and JIT. * The code generator tracks live, cached values so they can be saved to memory when needed. * Spills the stack pointer around escaping calls, so that the exact stack is visible to the cycle GC.
2024-10-07 14:56:39 +01:00
_PyFrame_SetStackPointer(frame, stack_pointer);
gh-131586: Avoid refcount contention in context managers (gh-131851) This avoid reference count contention in the free threading build when calling special methods like `__enter__` and `__exit__`.
2025-04-21 15:54:25 -04:00
const char *errfmt = _PyEval_SpecialMethodCanSuggest(owner, oparg)
gh-128398: improve error messages when incorrectly using `with` and `async with` (#132218) Improve the error message with a suggestion when an object supporting the synchronous (resp. asynchronous) context manager protocol is entered using `async with` (resp. `with`) instead of `with` (resp. `async with`).
2025-04-19 10:44:01 +02:00
? _Py_SpecialMethods[oparg].error_suggestion
: _Py_SpecialMethods[oparg].error;
stack_pointer = _PyFrame_GetStackPointer(frame);
assert(!_PyErr_Occurred(tstate));
assert(errfmt != NULL);
_PyFrame_SetStackPointer(frame, stack_pointer);
gh-131586: Avoid refcount contention in context managers (gh-131851) This avoid reference count contention in the free threading build when calling special methods like `__enter__` and `__exit__`.
2025-04-21 15:54:25 -04:00
_PyErr_Format(tstate, PyExc_TypeError, errfmt, owner);
GH-119866: Spill the stack around escaping calls. (GH-124392) * Spill the evaluation around escaping calls in the generated interpreter and JIT. * The code generator tracks live, cached values so they can be saved to memory when needed. * Spills the stack pointer around escaping calls, so that the exact stack is visible to the cycle GC.
2024-10-07 14:56:39 +01:00
stack_pointer = _PyFrame_GetStackPointer(frame);
GH-120507: Lower the `BEFORE_WITH` and `BEFORE_ASYNC_WITH` instructions. (#120640) * Remove BEFORE_WITH and BEFORE_ASYNC_WITH instructions. * Add LOAD_SPECIAL instruction * Reimplement `with` and `async with` statements using LOAD_SPECIAL
2024-06-18 12:17:46 +01:00
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(0);
GH-128682: Make `PyStackRef_CLOSE` escaping. (GH-129404)
2025-02-03 12:41:32 +00:00
JUMP_TO_ERROR();
GH-120507: Lower the `BEFORE_WITH` and `BEFORE_ASYNC_WITH` instructions. (#120640) * Remove BEFORE_WITH and BEFORE_ASYNC_WITH instructions. * Add LOAD_SPECIAL instruction * Reimplement `with` and `async with` statements using LOAD_SPECIAL
2024-06-18 12:17:46 +01:00
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_tos_cache0 = PyStackRef_ZERO_BITS;
_tos_cache1 = PyStackRef_ZERO_BITS;
_tos_cache2 = PyStackRef_ZERO_BITS;
SET_CURRENT_CACHED_VALUES(0);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-120507: Lower the `BEFORE_WITH` and `BEFORE_ASYNC_WITH` instructions. (#120640) * Remove BEFORE_WITH and BEFORE_ASYNC_WITH instructions. * Add LOAD_SPECIAL instruction * Reimplement `with` and `async with` statements using LOAD_SPECIAL
2024-06-18 12:17:46 +01:00
break;
}
GH-111485: Remove some special cases from the code generator and bytecodes.c (GH-111540)
2023-10-31 13:21:07 +00:00
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
case _WITH_EXCEPT_START_r33: {
CHECK_CURRENT_CACHED_VALUES(3);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
_PyStackRef val;
_PyStackRef lasti;
_PyStackRef exit_self;
_PyStackRef exit_func;
_PyStackRef res;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef _stack_item_0 = _tos_cache0;
_PyStackRef _stack_item_1 = _tos_cache1;
_PyStackRef _stack_item_2 = _tos_cache2;
val = _stack_item_2;
lasti = _stack_item_0;
exit_self = stack_pointer[-1];
exit_func = stack_pointer[-2];
gh-104584: Baby steps towards generating and executing traces (#105924) Added a new, experimental, tracing optimizer and interpreter (a.k.a. "tier 2"). This currently pessimizes, so don't use yet -- this is infrastructure so we can experiment with optimizing passes. To enable it, pass ``-Xuops`` or set ``PYTHONUOPS=1``. To get debug output, set ``PYTHONUOPSDEBUG=N`` where ``N`` is a debug level (0-4, where 0 is no debug output and 4 is excessively verbose). All of this code is likely to change dramatically before the 3.13 feature freeze. But this is a first step.
2023-06-26 19:02:57 -07:00
PyObject *exc, *tb;
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
PyObject *val_o = PyStackRef_AsPyObjectBorrow(val);
PyObject *exit_func_o = PyStackRef_AsPyObjectBorrow(exit_func);
assert(val_o && PyExceptionInstance_Check(val_o));
exc = PyExceptionInstance_Class(val_o);
gh-130595: Keep traceback alive for WITH_EXCEPT_START (#130562) Keep traceback alive for WITH_EXCEPT_START
2025-02-26 10:41:26 -08:00
PyObject *original_tb = tb = PyException_GetTraceback(val_o);
GH-106895: Raise a `ValueError` when attempting to disable events that cannot be disabled. (GH-107337)
2023-07-27 15:27:11 +01:00
if (tb == NULL) {
tb = Py_None;
}
GH-132508: Use tagged integers on the evaluation stack for the last instruction offset (GH-132545)
2025-04-29 18:00:35 +01:00
assert(PyStackRef_IsTaggedInt(lasti));
GH-131498: Cases generator: Parse down to C statement level. (GH-131948) * Parse down to statement level in the cases generator * Add handling for #if macros, treating them much like normal ifs.
2025-04-02 16:31:59 +01:00
(void)lasti;
GH-139922: Tail calling for MSVC (VS 2026) (GH-143068) Co-authored-by: Ken Jin <28750310+Fidget-Spinner@users.noreply.github.com> Co-authored-by: Brandt Bucher <brandt@python.org> Co-authored-by: Hugo van Kemenade <1324225+hugovk@users.noreply.github.com>
2025-12-23 00:01:34 +01:00
PyObject* res_o;
{
PyObject *stack[5] = {NULL, PyStackRef_AsPyObjectBorrow(exit_self), exc, val_o, tb};
int has_self = !PyStackRef_IsNull(exit_self);
stack_pointer[0] = lasti;
stack_pointer[1] = _stack_item_1;
stack_pointer[2] = val;
stack_pointer += 3;
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
_PyFrame_SetStackPointer(frame, stack_pointer);
res_o = PyObject_Vectorcall(exit_func_o, stack + 2 - has_self,
(3 + has_self) | PY_VECTORCALL_ARGUMENTS_OFFSET, NULL);
stack_pointer = _PyFrame_GetStackPointer(frame);
}
GH-119866: Spill the stack around escaping calls. (GH-124392) * Spill the evaluation around escaping calls in the generated interpreter and JIT. * The code generator tracks live, cached values so they can be saved to memory when needed. * Spills the stack pointer around escaping calls, so that the exact stack is visible to the cycle GC.
2024-10-07 14:56:39 +01:00
_PyFrame_SetStackPointer(frame, stack_pointer);
gh-130595: Fix leak in WITH_EXCEPT_START error case (GH-130626) Co-authored-by: Ken Jin <kenjin@python.org>
2025-02-28 09:58:50 +01:00
Py_XDECREF(original_tb);
GH-119866: Spill the stack around escaping calls. (GH-124392) * Spill the evaluation around escaping calls in the generated interpreter and JIT. * The code generator tracks live, cached values so they can be saved to memory when needed. * Spills the stack pointer around escaping calls, so that the exact stack is visible to the cycle GC.
2024-10-07 14:56:39 +01:00
stack_pointer = _PyFrame_GetStackPointer(frame);
GH-128682: Make `PyStackRef_CLOSE` escaping. (GH-129404)
2025-02-03 12:41:32 +00:00
if (res_o == NULL) {
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(0);
GH-128682: Make `PyStackRef_CLOSE` escaping. (GH-129404)
2025-02-03 12:41:32 +00:00
JUMP_TO_ERROR();
}
GH-121459: Streamline PyObject* to PyStackRef conversions by disallowing NULL pointers. (GH-124894)
2024-10-07 18:13:04 +01:00
res = PyStackRef_FromPyObjectSteal(res_o);
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_tos_cache2 = res;
_tos_cache1 = val;
_tos_cache0 = _stack_item_1;
SET_CURRENT_CACHED_VALUES(3);
stack_pointer += -2;
GH-141794: Limit size of generated machine code. (GH-142228) * Factor out bodies of the largest uops, to reduce jit code size. * Factor out common assert, also reducing jit code size. * Limit size of jitted code for a single executor to 1MB.
2025-12-03 17:43:35 +00:00
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-104584: Baby steps towards generating and executing traces (#105924) Added a new, experimental, tracing optimizer and interpreter (a.k.a. "tier 2"). This currently pessimizes, so don't use yet -- this is infrastructure so we can experiment with optimizing passes. To enable it, pass ``-Xuops`` or set ``PYTHONUOPS=1``. To get debug output, set ``PYTHONUOPSDEBUG=N`` where ``N`` is a debug level (0-4, where 0 is no debug output and 4 is excessively verbose). All of this code is likely to change dramatically before the 3.13 feature freeze. But this is a first step.
2023-06-26 19:02:57 -07:00
break;
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
case _PUSH_EXC_INFO_r02: {
CHECK_CURRENT_CACHED_VALUES(0);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-119866: Spill the stack around escaping calls. (GH-124392) * Spill the evaluation around escaping calls in the generated interpreter and JIT. * The code generator tracks live, cached values so they can be saved to memory when needed. * Spills the stack pointer around escaping calls, so that the exact stack is visible to the cycle GC.
2024-10-07 14:56:39 +01:00
_PyStackRef exc;
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
_PyStackRef prev_exc;
GH-119866: Spill the stack around escaping calls. (GH-124392) * Spill the evaluation around escaping calls in the generated interpreter and JIT. * The code generator tracks live, cached values so they can be saved to memory when needed. * Spills the stack pointer around escaping calls, so that the exact stack is visible to the cycle GC.
2024-10-07 14:56:39 +01:00
_PyStackRef new_exc;
exc = stack_pointer[-1];
gh-104584: Baby steps towards generating and executing traces (#105924) Added a new, experimental, tracing optimizer and interpreter (a.k.a. "tier 2"). This currently pessimizes, so don't use yet -- this is infrastructure so we can experiment with optimizing passes. To enable it, pass ``-Xuops`` or set ``PYTHONUOPS=1``. To get debug output, set ``PYTHONUOPSDEBUG=N`` where ``N`` is a debug level (0-4, where 0 is no debug output and 4 is excessively verbose). All of this code is likely to change dramatically before the 3.13 feature freeze. But this is a first step.
2023-06-26 19:02:57 -07:00
_PyErr_StackItem *exc_info = tstate->exc_info;
if (exc_info->exc_value != NULL) {
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
prev_exc = PyStackRef_FromPyObjectSteal(exc_info->exc_value);
gh-104584: Baby steps towards generating and executing traces (#105924) Added a new, experimental, tracing optimizer and interpreter (a.k.a. "tier 2"). This currently pessimizes, so don't use yet -- this is infrastructure so we can experiment with optimizing passes. To enable it, pass ``-Xuops`` or set ``PYTHONUOPS=1``. To get debug output, set ``PYTHONUOPSDEBUG=N`` where ``N`` is a debug level (0-4, where 0 is no debug output and 4 is excessively verbose). All of this code is likely to change dramatically before the 3.13 feature freeze. But this is a first step.
2023-06-26 19:02:57 -07:00
}
else {
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
prev_exc = PyStackRef_None;
gh-104584: Baby steps towards generating and executing traces (#105924) Added a new, experimental, tracing optimizer and interpreter (a.k.a. "tier 2"). This currently pessimizes, so don't use yet -- this is infrastructure so we can experiment with optimizing passes. To enable it, pass ``-Xuops`` or set ``PYTHONUOPS=1``. To get debug output, set ``PYTHONUOPSDEBUG=N`` where ``N`` is a debug level (0-4, where 0 is no debug output and 4 is excessively verbose). All of this code is likely to change dramatically before the 3.13 feature freeze. But this is a first step.
2023-06-26 19:02:57 -07:00
}
GH-119866: Spill the stack around escaping calls. (GH-124392) * Spill the evaluation around escaping calls in the generated interpreter and JIT. * The code generator tracks live, cached values so they can be saved to memory when needed. * Spills the stack pointer around escaping calls, so that the exact stack is visible to the cycle GC.
2024-10-07 14:56:39 +01:00
assert(PyStackRef_ExceptionInstanceCheck(exc));
exc_info->exc_value = PyStackRef_AsPyObjectNew(exc);
new_exc = exc;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_tos_cache1 = new_exc;
_tos_cache0 = prev_exc;
SET_CURRENT_CACHED_VALUES(2);
stack_pointer += -1;
GH-141794: Limit size of generated machine code. (GH-142228) * Factor out bodies of the largest uops, to reduce jit code size. * Factor out common assert, also reducing jit code size. * Limit size of jitted code for a single executor to 1MB.
2025-12-03 17:43:35 +00:00
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _PUSH_EXC_INFO_r12: {
CHECK_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef exc;
_PyStackRef prev_exc;
_PyStackRef new_exc;
_PyStackRef _stack_item_0 = _tos_cache0;
exc = _stack_item_0;
_PyErr_StackItem *exc_info = tstate->exc_info;
if (exc_info->exc_value != NULL) {
prev_exc = PyStackRef_FromPyObjectSteal(exc_info->exc_value);
}
else {
prev_exc = PyStackRef_None;
}
assert(PyStackRef_ExceptionInstanceCheck(exc));
exc_info->exc_value = PyStackRef_AsPyObjectNew(exc);
new_exc = exc;
_tos_cache1 = new_exc;
_tos_cache0 = prev_exc;
SET_CURRENT_CACHED_VALUES(2);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _PUSH_EXC_INFO_r23: {
CHECK_CURRENT_CACHED_VALUES(2);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef exc;
_PyStackRef prev_exc;
_PyStackRef new_exc;
_PyStackRef _stack_item_0 = _tos_cache0;
_PyStackRef _stack_item_1 = _tos_cache1;
exc = _stack_item_1;
_PyErr_StackItem *exc_info = tstate->exc_info;
if (exc_info->exc_value != NULL) {
prev_exc = PyStackRef_FromPyObjectSteal(exc_info->exc_value);
}
else {
prev_exc = PyStackRef_None;
}
assert(PyStackRef_ExceptionInstanceCheck(exc));
exc_info->exc_value = PyStackRef_AsPyObjectNew(exc);
new_exc = exc;
_tos_cache2 = new_exc;
_tos_cache1 = prev_exc;
_tos_cache0 = _stack_item_0;
SET_CURRENT_CACHED_VALUES(3);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-104584: Baby steps towards generating and executing traces (#105924) Added a new, experimental, tracing optimizer and interpreter (a.k.a. "tier 2"). This currently pessimizes, so don't use yet -- this is infrastructure so we can experiment with optimizing passes. To enable it, pass ``-Xuops`` or set ``PYTHONUOPS=1``. To get debug output, set ``PYTHONUOPSDEBUG=N`` where ``N`` is a debug level (0-4, where 0 is no debug output and 4 is excessively verbose). All of this code is likely to change dramatically before the 3.13 feature freeze. But this is a first step.
2023-06-26 19:02:57 -07:00
break;
}
gh-106581: Split CALL_BOUND_METHOD_EXACT_ARGS into uops (#108462) Instead of using `GO_TO_INSTRUCTION(CALL_PY_EXACT_ARGS)` we just add the macro elements of the latter to the macro for the former. This requires lengthening the uops array in struct opcode_macro_expansion. (It also required changes to stacking.py that were merged already.)
2023-08-24 17:36:00 -07:00
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
case _GUARD_DORV_VALUES_INST_ATTR_FROM_DICT_r01: {
CHECK_CURRENT_CACHED_VALUES(0);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
_PyStackRef owner;
gh-104909: Split some more insts into ops (#109943) These are the most popular specializations of `LOAD_ATTR` and `STORE_ATTR` that weren't already viable uops: * Split LOAD_ATTR_METHOD_WITH_VALUES * Split LOAD_ATTR_METHOD_NO_DICT * Split LOAD_ATTR_SLOT * Split STORE_ATTR_SLOT * Split STORE_ATTR_INSTANCE_VALUE Also: * Add `-v` flag to code generator which prints a list of non-viable uops (easter-egg: it can print execution counts -- see source) * Double _Py_UOP_MAX_TRACE_LENGTH to 128 I had dropped one of the DEOPT_IF() calls! :-(
2023-09-27 15:27:44 -07:00
owner = stack_pointer[-1];
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
PyObject *owner_o = PyStackRef_AsPyObjectBorrow(owner);
assert(Py_TYPE(owner_o)->tp_flags & Py_TPFLAGS_INLINE_VALUES);
gh-115999: Specialize `LOAD_ATTR` for instance and class receivers in free-threaded builds (#128164) Finish specialization for LOAD_ATTR in the free-threaded build by adding support for class and instance receivers.
2025-01-14 11:56:11 -08:00
PyDictValues *ivs = _PyObject_InlineValues(owner_o);
if (!FT_ATOMIC_LOAD_UINT8(ivs->valid)) {
GH-117457: Correct pystats uop "miss" counts (GH-117477)
2024-04-04 18:49:18 -04:00
UOP_STAT_INC(uopcode, miss);
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(0);
JUMP_TO_JUMP_TARGET();
}
_tos_cache0 = owner;
SET_CURRENT_CACHED_VALUES(1);
stack_pointer += -1;
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _GUARD_DORV_VALUES_INST_ATTR_FROM_DICT_r11: {
CHECK_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef owner;
_PyStackRef _stack_item_0 = _tos_cache0;
owner = _stack_item_0;
PyObject *owner_o = PyStackRef_AsPyObjectBorrow(owner);
assert(Py_TYPE(owner_o)->tp_flags & Py_TPFLAGS_INLINE_VALUES);
PyDictValues *ivs = _PyObject_InlineValues(owner_o);
if (!FT_ATOMIC_LOAD_UINT8(ivs->valid)) {
UOP_STAT_INC(uopcode, miss);
GH-142718: JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. (GH-142762) JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. * Make sure that stack is in correct state at side exits with TOS cached values * Simplify choice of cached items for side exits
2025-12-15 19:01:29 +00:00
_tos_cache0 = owner;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(1);
JUMP_TO_JUMP_TARGET();
}
_tos_cache0 = owner;
SET_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _GUARD_DORV_VALUES_INST_ATTR_FROM_DICT_r22: {
CHECK_CURRENT_CACHED_VALUES(2);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef owner;
_PyStackRef _stack_item_0 = _tos_cache0;
_PyStackRef _stack_item_1 = _tos_cache1;
owner = _stack_item_1;
PyObject *owner_o = PyStackRef_AsPyObjectBorrow(owner);
assert(Py_TYPE(owner_o)->tp_flags & Py_TPFLAGS_INLINE_VALUES);
PyDictValues *ivs = _PyObject_InlineValues(owner_o);
if (!FT_ATOMIC_LOAD_UINT8(ivs->valid)) {
UOP_STAT_INC(uopcode, miss);
GH-142718: JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. (GH-142762) JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. * Make sure that stack is in correct state at side exits with TOS cached values * Simplify choice of cached items for side exits
2025-12-15 19:01:29 +00:00
_tos_cache1 = owner;
_tos_cache0 = _stack_item_0;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(2);
JUMP_TO_JUMP_TARGET();
}
_tos_cache1 = owner;
_tos_cache0 = _stack_item_0;
SET_CURRENT_CACHED_VALUES(2);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _GUARD_DORV_VALUES_INST_ATTR_FROM_DICT_r33: {
CHECK_CURRENT_CACHED_VALUES(3);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef owner;
_PyStackRef _stack_item_0 = _tos_cache0;
_PyStackRef _stack_item_1 = _tos_cache1;
_PyStackRef _stack_item_2 = _tos_cache2;
owner = _stack_item_2;
PyObject *owner_o = PyStackRef_AsPyObjectBorrow(owner);
assert(Py_TYPE(owner_o)->tp_flags & Py_TPFLAGS_INLINE_VALUES);
PyDictValues *ivs = _PyObject_InlineValues(owner_o);
if (!FT_ATOMIC_LOAD_UINT8(ivs->valid)) {
UOP_STAT_INC(uopcode, miss);
GH-142718: JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. (GH-142762) JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. * Make sure that stack is in correct state at side exits with TOS cached values * Simplify choice of cached items for side exits
2025-12-15 19:01:29 +00:00
_tos_cache2 = owner;
_tos_cache1 = _stack_item_1;
_tos_cache0 = _stack_item_0;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(3);
GH-117457: Correct pystats uop "miss" counts (GH-117477)
2024-04-04 18:49:18 -04:00
JUMP_TO_JUMP_TARGET();
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_tos_cache2 = owner;
_tos_cache1 = _stack_item_1;
_tos_cache0 = _stack_item_0;
SET_CURRENT_CACHED_VALUES(3);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-104909: Split some more insts into ops (#109943) These are the most popular specializations of `LOAD_ATTR` and `STORE_ATTR` that weren't already viable uops: * Split LOAD_ATTR_METHOD_WITH_VALUES * Split LOAD_ATTR_METHOD_NO_DICT * Split LOAD_ATTR_SLOT * Split STORE_ATTR_SLOT * Split STORE_ATTR_INSTANCE_VALUE Also: * Add `-v` flag to code generator which prints a list of non-viable uops (easter-egg: it can print execution counts -- see source) * Double _Py_UOP_MAX_TRACE_LENGTH to 128 I had dropped one of the DEOPT_IF() calls! :-(
2023-09-27 15:27:44 -07:00
break;
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
case _GUARD_KEYS_VERSION_r01: {
CHECK_CURRENT_CACHED_VALUES(0);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
_PyStackRef owner;
gh-104909: Split some more insts into ops (#109943) These are the most popular specializations of `LOAD_ATTR` and `STORE_ATTR` that weren't already viable uops: * Split LOAD_ATTR_METHOD_WITH_VALUES * Split LOAD_ATTR_METHOD_NO_DICT * Split LOAD_ATTR_SLOT * Split STORE_ATTR_SLOT * Split STORE_ATTR_INSTANCE_VALUE Also: * Add `-v` flag to code generator which prints a list of non-viable uops (easter-egg: it can print execution counts -- see source) * Double _Py_UOP_MAX_TRACE_LENGTH to 128 I had dropped one of the DEOPT_IF() calls! :-(
2023-09-27 15:27:44 -07:00
owner = stack_pointer[-1];
GH-140683: JIT: Improve machine code for loading smaller constants on AArch64. (GH-142511) * Use movz and movk instructions for loading 16 and 32 bit operands and oparg. * Loading of 64 bit operands is unchanged.
2025-12-11 12:33:39 +00:00
uint32_t keys_version = (uint32_t)CURRENT_OPERAND0_32();
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
PyTypeObject *owner_cls = Py_TYPE(PyStackRef_AsPyObjectBorrow(owner));
gh-104909: Split some more insts into ops (#109943) These are the most popular specializations of `LOAD_ATTR` and `STORE_ATTR` that weren't already viable uops: * Split LOAD_ATTR_METHOD_WITH_VALUES * Split LOAD_ATTR_METHOD_NO_DICT * Split LOAD_ATTR_SLOT * Split STORE_ATTR_SLOT * Split STORE_ATTR_INSTANCE_VALUE Also: * Add `-v` flag to code generator which prints a list of non-viable uops (easter-egg: it can print execution counts -- see source) * Double _Py_UOP_MAX_TRACE_LENGTH to 128 I had dropped one of the DEOPT_IF() calls! :-(
2023-09-27 15:27:44 -07:00
PyHeapTypeObject *owner_heap_type = (PyHeapTypeObject *)owner_cls;
gh-115999: Specialize `LOAD_ATTR` for instance and class receivers in free-threaded builds (#128164) Finish specialization for LOAD_ATTR in the free-threaded build by adding support for class and instance receivers.
2025-01-14 11:56:11 -08:00
PyDictKeysObject *keys = owner_heap_type->ht_cached_keys;
if (FT_ATOMIC_LOAD_UINT32_RELAXED(keys->dk_version) != keys_version) {
GH-117457: Correct pystats uop "miss" counts (GH-117477)
2024-04-04 18:49:18 -04:00
UOP_STAT_INC(uopcode, miss);
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(0);
JUMP_TO_JUMP_TARGET();
}
_tos_cache0 = owner;
SET_CURRENT_CACHED_VALUES(1);
stack_pointer += -1;
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _GUARD_KEYS_VERSION_r11: {
CHECK_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef owner;
_PyStackRef _stack_item_0 = _tos_cache0;
owner = _stack_item_0;
GH-140683: JIT: Improve machine code for loading smaller constants on AArch64. (GH-142511) * Use movz and movk instructions for loading 16 and 32 bit operands and oparg. * Loading of 64 bit operands is unchanged.
2025-12-11 12:33:39 +00:00
uint32_t keys_version = (uint32_t)CURRENT_OPERAND0_32();
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
PyTypeObject *owner_cls = Py_TYPE(PyStackRef_AsPyObjectBorrow(owner));
PyHeapTypeObject *owner_heap_type = (PyHeapTypeObject *)owner_cls;
PyDictKeysObject *keys = owner_heap_type->ht_cached_keys;
if (FT_ATOMIC_LOAD_UINT32_RELAXED(keys->dk_version) != keys_version) {
UOP_STAT_INC(uopcode, miss);
GH-142718: JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. (GH-142762) JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. * Make sure that stack is in correct state at side exits with TOS cached values * Simplify choice of cached items for side exits
2025-12-15 19:01:29 +00:00
_tos_cache0 = owner;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(1);
JUMP_TO_JUMP_TARGET();
}
_tos_cache0 = owner;
SET_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _GUARD_KEYS_VERSION_r22: {
CHECK_CURRENT_CACHED_VALUES(2);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef owner;
_PyStackRef _stack_item_0 = _tos_cache0;
_PyStackRef _stack_item_1 = _tos_cache1;
owner = _stack_item_1;
GH-140683: JIT: Improve machine code for loading smaller constants on AArch64. (GH-142511) * Use movz and movk instructions for loading 16 and 32 bit operands and oparg. * Loading of 64 bit operands is unchanged.
2025-12-11 12:33:39 +00:00
uint32_t keys_version = (uint32_t)CURRENT_OPERAND0_32();
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
PyTypeObject *owner_cls = Py_TYPE(PyStackRef_AsPyObjectBorrow(owner));
PyHeapTypeObject *owner_heap_type = (PyHeapTypeObject *)owner_cls;
PyDictKeysObject *keys = owner_heap_type->ht_cached_keys;
if (FT_ATOMIC_LOAD_UINT32_RELAXED(keys->dk_version) != keys_version) {
UOP_STAT_INC(uopcode, miss);
GH-142718: JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. (GH-142762) JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. * Make sure that stack is in correct state at side exits with TOS cached values * Simplify choice of cached items for side exits
2025-12-15 19:01:29 +00:00
_tos_cache1 = owner;
_tos_cache0 = _stack_item_0;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(2);
JUMP_TO_JUMP_TARGET();
}
_tos_cache1 = owner;
_tos_cache0 = _stack_item_0;
SET_CURRENT_CACHED_VALUES(2);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _GUARD_KEYS_VERSION_r33: {
CHECK_CURRENT_CACHED_VALUES(3);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef owner;
_PyStackRef _stack_item_0 = _tos_cache0;
_PyStackRef _stack_item_1 = _tos_cache1;
_PyStackRef _stack_item_2 = _tos_cache2;
owner = _stack_item_2;
GH-140683: JIT: Improve machine code for loading smaller constants on AArch64. (GH-142511) * Use movz and movk instructions for loading 16 and 32 bit operands and oparg. * Loading of 64 bit operands is unchanged.
2025-12-11 12:33:39 +00:00
uint32_t keys_version = (uint32_t)CURRENT_OPERAND0_32();
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
PyTypeObject *owner_cls = Py_TYPE(PyStackRef_AsPyObjectBorrow(owner));
PyHeapTypeObject *owner_heap_type = (PyHeapTypeObject *)owner_cls;
PyDictKeysObject *keys = owner_heap_type->ht_cached_keys;
if (FT_ATOMIC_LOAD_UINT32_RELAXED(keys->dk_version) != keys_version) {
UOP_STAT_INC(uopcode, miss);
GH-142718: JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. (GH-142762) JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. * Make sure that stack is in correct state at side exits with TOS cached values * Simplify choice of cached items for side exits
2025-12-15 19:01:29 +00:00
_tos_cache2 = owner;
_tos_cache1 = _stack_item_1;
_tos_cache0 = _stack_item_0;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(3);
GH-117457: Correct pystats uop "miss" counts (GH-117477)
2024-04-04 18:49:18 -04:00
JUMP_TO_JUMP_TARGET();
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_tos_cache2 = owner;
_tos_cache1 = _stack_item_1;
_tos_cache0 = _stack_item_0;
SET_CURRENT_CACHED_VALUES(3);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-104909: Split some more insts into ops (#109943) These are the most popular specializations of `LOAD_ATTR` and `STORE_ATTR` that weren't already viable uops: * Split LOAD_ATTR_METHOD_WITH_VALUES * Split LOAD_ATTR_METHOD_NO_DICT * Split LOAD_ATTR_SLOT * Split STORE_ATTR_SLOT * Split STORE_ATTR_INSTANCE_VALUE Also: * Add `-v` flag to code generator which prints a list of non-viable uops (easter-egg: it can print execution counts -- see source) * Double _Py_UOP_MAX_TRACE_LENGTH to 128 I had dropped one of the DEOPT_IF() calls! :-(
2023-09-27 15:27:44 -07:00
break;
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
case _LOAD_ATTR_METHOD_WITH_VALUES_r02: {
CHECK_CURRENT_CACHED_VALUES(0);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
_PyStackRef owner;
_PyStackRef attr;
GH-128914: Remove all but one conditional stack effects (GH-129226) * Remove all 'if (0)' and 'if (1)' conditional stack effects * Use array instead of conditional for BUILD_SLICE args * Refactor LOAD_GLOBAL to use a common conditional uop * Remove conditional stack effects from LOAD_ATTR specializations * Replace conditional stack effects in LOAD_ATTR with a 0 or 1 sized array. * Remove conditional stack effects from CALL_FUNCTION_EX
2025-01-27 16:24:48 +00:00
_PyStackRef self;
Revert "GH-128914: Remove conditional stack effects from `bytecodes.c` and the code generators (GH-128918)" (GH-129202) The commit introduced a ~2.5-3% regression in the free threading build. This reverts commit ab61d3f4303d14a413bc9ae6557c730ffdf7579e.
2025-01-23 04:26:25 -05:00
oparg = CURRENT_OPARG();
gh-104909: Split some more insts into ops (#109943) These are the most popular specializations of `LOAD_ATTR` and `STORE_ATTR` that weren't already viable uops: * Split LOAD_ATTR_METHOD_WITH_VALUES * Split LOAD_ATTR_METHOD_NO_DICT * Split LOAD_ATTR_SLOT * Split STORE_ATTR_SLOT * Split STORE_ATTR_INSTANCE_VALUE Also: * Add `-v` flag to code generator which prints a list of non-viable uops (easter-egg: it can print execution counts -- see source) * Double _Py_UOP_MAX_TRACE_LENGTH to 128 I had dropped one of the DEOPT_IF() calls! :-(
2023-09-27 15:27:44 -07:00
owner = stack_pointer[-1];
GH-140683: JIT: Improve machine code for loading smaller constants on AArch64. (GH-142511) * Use movz and movk instructions for loading 16 and 32 bit operands and oparg. * Loading of 64 bit operands is unchanged.
2025-12-11 12:33:39 +00:00
PyObject *descr = (PyObject *)CURRENT_OPERAND0_64();
Revert "GH-128914: Remove conditional stack effects from `bytecodes.c` and the code generators (GH-128918)" (GH-129202) The commit introduced a ~2.5-3% regression in the free threading build. This reverts commit ab61d3f4303d14a413bc9ae6557c730ffdf7579e.
2025-01-23 04:26:25 -05:00
assert(oparg & 1);
gh-104909: Split some more insts into ops (#109943) These are the most popular specializations of `LOAD_ATTR` and `STORE_ATTR` that weren't already viable uops: * Split LOAD_ATTR_METHOD_WITH_VALUES * Split LOAD_ATTR_METHOD_NO_DICT * Split LOAD_ATTR_SLOT * Split STORE_ATTR_SLOT * Split STORE_ATTR_INSTANCE_VALUE Also: * Add `-v` flag to code generator which prints a list of non-viable uops (easter-egg: it can print execution counts -- see source) * Double _Py_UOP_MAX_TRACE_LENGTH to 128 I had dropped one of the DEOPT_IF() calls! :-(
2023-09-27 15:27:44 -07:00
STAT_INC(LOAD_ATTR, hit);
assert(descr != NULL);
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
assert(_PyType_HasFeature(Py_TYPE(descr), Py_TPFLAGS_METHOD_DESCRIPTOR));
attr = PyStackRef_FromPyObjectNew(descr);
gh-118926: Spill deferred references to stack in cases generator (#122748) This automatically spills the results from `_PyStackRef_FromPyObjectNew` to the in-memory stack so that the deferred references are visible to the GC before we make any possibly escaping call. Co-authored-by: Ken Jin <kenjin@python.org>
2024-08-07 13:23:53 -04:00
self = owner;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_tos_cache1 = self;
_tos_cache0 = attr;
SET_CURRENT_CACHED_VALUES(2);
stack_pointer += -1;
GH-141794: Limit size of generated machine code. (GH-142228) * Factor out bodies of the largest uops, to reduce jit code size. * Factor out common assert, also reducing jit code size. * Limit size of jitted code for a single executor to 1MB.
2025-12-03 17:43:35 +00:00
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _LOAD_ATTR_METHOD_WITH_VALUES_r12: {
CHECK_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef owner;
_PyStackRef attr;
_PyStackRef self;
_PyStackRef _stack_item_0 = _tos_cache0;
oparg = CURRENT_OPARG();
owner = _stack_item_0;
GH-140683: JIT: Improve machine code for loading smaller constants on AArch64. (GH-142511) * Use movz and movk instructions for loading 16 and 32 bit operands and oparg. * Loading of 64 bit operands is unchanged.
2025-12-11 12:33:39 +00:00
PyObject *descr = (PyObject *)CURRENT_OPERAND0_64();
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
assert(oparg & 1);
STAT_INC(LOAD_ATTR, hit);
assert(descr != NULL);
assert(_PyType_HasFeature(Py_TYPE(descr), Py_TPFLAGS_METHOD_DESCRIPTOR));
attr = PyStackRef_FromPyObjectNew(descr);
self = owner;
_tos_cache1 = self;
_tos_cache0 = attr;
SET_CURRENT_CACHED_VALUES(2);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _LOAD_ATTR_METHOD_WITH_VALUES_r23: {
CHECK_CURRENT_CACHED_VALUES(2);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef owner;
_PyStackRef attr;
_PyStackRef self;
_PyStackRef _stack_item_0 = _tos_cache0;
_PyStackRef _stack_item_1 = _tos_cache1;
oparg = CURRENT_OPARG();
owner = _stack_item_1;
GH-140683: JIT: Improve machine code for loading smaller constants on AArch64. (GH-142511) * Use movz and movk instructions for loading 16 and 32 bit operands and oparg. * Loading of 64 bit operands is unchanged.
2025-12-11 12:33:39 +00:00
PyObject *descr = (PyObject *)CURRENT_OPERAND0_64();
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
assert(oparg & 1);
STAT_INC(LOAD_ATTR, hit);
assert(descr != NULL);
assert(_PyType_HasFeature(Py_TYPE(descr), Py_TPFLAGS_METHOD_DESCRIPTOR));
attr = PyStackRef_FromPyObjectNew(descr);
self = owner;
_tos_cache2 = self;
_tos_cache1 = attr;
_tos_cache0 = _stack_item_0;
SET_CURRENT_CACHED_VALUES(3);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-104909: Split some more insts into ops (#109943) These are the most popular specializations of `LOAD_ATTR` and `STORE_ATTR` that weren't already viable uops: * Split LOAD_ATTR_METHOD_WITH_VALUES * Split LOAD_ATTR_METHOD_NO_DICT * Split LOAD_ATTR_SLOT * Split STORE_ATTR_SLOT * Split STORE_ATTR_INSTANCE_VALUE Also: * Add `-v` flag to code generator which prints a list of non-viable uops (easter-egg: it can print execution counts -- see source) * Double _Py_UOP_MAX_TRACE_LENGTH to 128 I had dropped one of the DEOPT_IF() calls! :-(
2023-09-27 15:27:44 -07:00
break;
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
case _LOAD_ATTR_METHOD_NO_DICT_r02: {
CHECK_CURRENT_CACHED_VALUES(0);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
_PyStackRef owner;
_PyStackRef attr;
GH-128914: Remove all but one conditional stack effects (GH-129226) * Remove all 'if (0)' and 'if (1)' conditional stack effects * Use array instead of conditional for BUILD_SLICE args * Refactor LOAD_GLOBAL to use a common conditional uop * Remove conditional stack effects from LOAD_ATTR specializations * Replace conditional stack effects in LOAD_ATTR with a 0 or 1 sized array. * Remove conditional stack effects from CALL_FUNCTION_EX
2025-01-27 16:24:48 +00:00
_PyStackRef self;
Revert "GH-128914: Remove conditional stack effects from `bytecodes.c` and the code generators (GH-128918)" (GH-129202) The commit introduced a ~2.5-3% regression in the free threading build. This reverts commit ab61d3f4303d14a413bc9ae6557c730ffdf7579e.
2025-01-23 04:26:25 -05:00
oparg = CURRENT_OPARG();
gh-104909: Split some more insts into ops (#109943) These are the most popular specializations of `LOAD_ATTR` and `STORE_ATTR` that weren't already viable uops: * Split LOAD_ATTR_METHOD_WITH_VALUES * Split LOAD_ATTR_METHOD_NO_DICT * Split LOAD_ATTR_SLOT * Split STORE_ATTR_SLOT * Split STORE_ATTR_INSTANCE_VALUE Also: * Add `-v` flag to code generator which prints a list of non-viable uops (easter-egg: it can print execution counts -- see source) * Double _Py_UOP_MAX_TRACE_LENGTH to 128 I had dropped one of the DEOPT_IF() calls! :-(
2023-09-27 15:27:44 -07:00
owner = stack_pointer[-1];
GH-140683: JIT: Improve machine code for loading smaller constants on AArch64. (GH-142511) * Use movz and movk instructions for loading 16 and 32 bit operands and oparg. * Loading of 64 bit operands is unchanged.
2025-12-11 12:33:39 +00:00
PyObject *descr = (PyObject *)CURRENT_OPERAND0_64();
Revert "GH-128914: Remove conditional stack effects from `bytecodes.c` and the code generators (GH-128918)" (GH-129202) The commit introduced a ~2.5-3% regression in the free threading build. This reverts commit ab61d3f4303d14a413bc9ae6557c730ffdf7579e.
2025-01-23 04:26:25 -05:00
assert(oparg & 1);
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
assert(Py_TYPE(PyStackRef_AsPyObjectBorrow(owner))->tp_dictoffset == 0);
gh-104909: Split some more insts into ops (#109943) These are the most popular specializations of `LOAD_ATTR` and `STORE_ATTR` that weren't already viable uops: * Split LOAD_ATTR_METHOD_WITH_VALUES * Split LOAD_ATTR_METHOD_NO_DICT * Split LOAD_ATTR_SLOT * Split STORE_ATTR_SLOT * Split STORE_ATTR_INSTANCE_VALUE Also: * Add `-v` flag to code generator which prints a list of non-viable uops (easter-egg: it can print execution counts -- see source) * Double _Py_UOP_MAX_TRACE_LENGTH to 128 I had dropped one of the DEOPT_IF() calls! :-(
2023-09-27 15:27:44 -07:00
STAT_INC(LOAD_ATTR, hit);
assert(descr != NULL);
assert(_PyType_HasFeature(Py_TYPE(descr), Py_TPFLAGS_METHOD_DESCRIPTOR));
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
attr = PyStackRef_FromPyObjectNew(descr);
gh-118926: Spill deferred references to stack in cases generator (#122748) This automatically spills the results from `_PyStackRef_FromPyObjectNew` to the in-memory stack so that the deferred references are visible to the GC before we make any possibly escaping call. Co-authored-by: Ken Jin <kenjin@python.org>
2024-08-07 13:23:53 -04:00
self = owner;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_tos_cache1 = self;
_tos_cache0 = attr;
SET_CURRENT_CACHED_VALUES(2);
stack_pointer += -1;
GH-141794: Limit size of generated machine code. (GH-142228) * Factor out bodies of the largest uops, to reduce jit code size. * Factor out common assert, also reducing jit code size. * Limit size of jitted code for a single executor to 1MB.
2025-12-03 17:43:35 +00:00
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-104909: Split more LOAD_ATTR specializations (GH-110317) * Split LOAD_ATTR_MODULE * Split LOAD_ATTR_WITH_HINT * Split _GUARD_TYPE_VERSION out of the latter * Split LOAD_ATTR_CLASS * Split LOAD_ATTR_NONDESCRIPTOR_WITH_VALUES * Fix indent of DEOPT_IF in macros * Split LOAD_ATTR_METHOD_LAZY_DICT * Split LOAD_ATTR_NONDESCRIPTOR_NO_DICT * Fix omission of _CHECK_ATTR_METHOD_LAZY_DICT
2023-10-04 08:08:02 -07:00
break;
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
case _LOAD_ATTR_METHOD_NO_DICT_r12: {
CHECK_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
_PyStackRef owner;
_PyStackRef attr;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef self;
_PyStackRef _stack_item_0 = _tos_cache0;
Revert "GH-128914: Remove conditional stack effects from `bytecodes.c` and the code generators (GH-128918)" (GH-129202) The commit introduced a ~2.5-3% regression in the free threading build. This reverts commit ab61d3f4303d14a413bc9ae6557c730ffdf7579e.
2025-01-23 04:26:25 -05:00
oparg = CURRENT_OPARG();
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
owner = _stack_item_0;
GH-140683: JIT: Improve machine code for loading smaller constants on AArch64. (GH-142511) * Use movz and movk instructions for loading 16 and 32 bit operands and oparg. * Loading of 64 bit operands is unchanged.
2025-12-11 12:33:39 +00:00
PyObject *descr = (PyObject *)CURRENT_OPERAND0_64();
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
assert(oparg & 1);
assert(Py_TYPE(PyStackRef_AsPyObjectBorrow(owner))->tp_dictoffset == 0);
STAT_INC(LOAD_ATTR, hit);
assert(descr != NULL);
assert(_PyType_HasFeature(Py_TYPE(descr), Py_TPFLAGS_METHOD_DESCRIPTOR));
attr = PyStackRef_FromPyObjectNew(descr);
self = owner;
_tos_cache1 = self;
_tos_cache0 = attr;
SET_CURRENT_CACHED_VALUES(2);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _LOAD_ATTR_METHOD_NO_DICT_r23: {
CHECK_CURRENT_CACHED_VALUES(2);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef owner;
_PyStackRef attr;
_PyStackRef self;
_PyStackRef _stack_item_0 = _tos_cache0;
_PyStackRef _stack_item_1 = _tos_cache1;
oparg = CURRENT_OPARG();
owner = _stack_item_1;
GH-140683: JIT: Improve machine code for loading smaller constants on AArch64. (GH-142511) * Use movz and movk instructions for loading 16 and 32 bit operands and oparg. * Loading of 64 bit operands is unchanged.
2025-12-11 12:33:39 +00:00
PyObject *descr = (PyObject *)CURRENT_OPERAND0_64();
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
assert(oparg & 1);
assert(Py_TYPE(PyStackRef_AsPyObjectBorrow(owner))->tp_dictoffset == 0);
STAT_INC(LOAD_ATTR, hit);
assert(descr != NULL);
assert(_PyType_HasFeature(Py_TYPE(descr), Py_TPFLAGS_METHOD_DESCRIPTOR));
attr = PyStackRef_FromPyObjectNew(descr);
self = owner;
_tos_cache2 = self;
_tos_cache1 = attr;
_tos_cache0 = _stack_item_0;
SET_CURRENT_CACHED_VALUES(3);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _LOAD_ATTR_NONDESCRIPTOR_WITH_VALUES_r11: {
CHECK_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef owner;
_PyStackRef attr;
_PyStackRef _stack_item_0 = _tos_cache0;
oparg = CURRENT_OPARG();
owner = _stack_item_0;
GH-140683: JIT: Improve machine code for loading smaller constants on AArch64. (GH-142511) * Use movz and movk instructions for loading 16 and 32 bit operands and oparg. * Loading of 64 bit operands is unchanged.
2025-12-11 12:33:39 +00:00
PyObject *descr = (PyObject *)CURRENT_OPERAND0_64();
Revert "GH-128914: Remove conditional stack effects from `bytecodes.c` and the code generators (GH-128918)" (GH-129202) The commit introduced a ~2.5-3% regression in the free threading build. This reverts commit ab61d3f4303d14a413bc9ae6557c730ffdf7579e.
2025-01-23 04:26:25 -05:00
assert((oparg & 1) == 0);
gh-104909: Split more LOAD_ATTR specializations (GH-110317) * Split LOAD_ATTR_MODULE * Split LOAD_ATTR_WITH_HINT * Split _GUARD_TYPE_VERSION out of the latter * Split LOAD_ATTR_CLASS * Split LOAD_ATTR_NONDESCRIPTOR_WITH_VALUES * Fix indent of DEOPT_IF in macros * Split LOAD_ATTR_METHOD_LAZY_DICT * Split LOAD_ATTR_NONDESCRIPTOR_NO_DICT * Fix omission of _CHECK_ATTR_METHOD_LAZY_DICT
2023-10-04 08:08:02 -07:00
STAT_INC(LOAD_ATTR, hit);
assert(descr != NULL);
GH-128682: Account for escapes in `DECREF_INPUTS` (GH-129953) * Handle escapes in DECREF_INPUTS * Mark a few more functions as escaping * Replace DECREF_INPUTS with PyStackRef_CLOSE where possible
2025-02-12 17:44:59 +00:00
_PyFrame_SetStackPointer(frame, stack_pointer);
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
PyStackRef_CLOSE(owner);
GH-128682: Account for escapes in `DECREF_INPUTS` (GH-129953) * Handle escapes in DECREF_INPUTS * Mark a few more functions as escaping * Replace DECREF_INPUTS with PyStackRef_CLOSE where possible
2025-02-12 17:44:59 +00:00
stack_pointer = _PyFrame_GetStackPointer(frame);
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
attr = PyStackRef_FromPyObjectNew(descr);
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_tos_cache0 = attr;
_tos_cache1 = PyStackRef_ZERO_BITS;
_tos_cache2 = PyStackRef_ZERO_BITS;
SET_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-104909: Split more LOAD_ATTR specializations (GH-110317) * Split LOAD_ATTR_MODULE * Split LOAD_ATTR_WITH_HINT * Split _GUARD_TYPE_VERSION out of the latter * Split LOAD_ATTR_CLASS * Split LOAD_ATTR_NONDESCRIPTOR_WITH_VALUES * Fix indent of DEOPT_IF in macros * Split LOAD_ATTR_METHOD_LAZY_DICT * Split LOAD_ATTR_NONDESCRIPTOR_NO_DICT * Fix omission of _CHECK_ATTR_METHOD_LAZY_DICT
2023-10-04 08:08:02 -07:00
break;
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
case _LOAD_ATTR_NONDESCRIPTOR_NO_DICT_r11: {
CHECK_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
_PyStackRef owner;
_PyStackRef attr;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef _stack_item_0 = _tos_cache0;
Revert "GH-128914: Remove conditional stack effects from `bytecodes.c` and the code generators (GH-128918)" (GH-129202) The commit introduced a ~2.5-3% regression in the free threading build. This reverts commit ab61d3f4303d14a413bc9ae6557c730ffdf7579e.
2025-01-23 04:26:25 -05:00
oparg = CURRENT_OPARG();
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
owner = _stack_item_0;
GH-140683: JIT: Improve machine code for loading smaller constants on AArch64. (GH-142511) * Use movz and movk instructions for loading 16 and 32 bit operands and oparg. * Loading of 64 bit operands is unchanged.
2025-12-11 12:33:39 +00:00
PyObject *descr = (PyObject *)CURRENT_OPERAND0_64();
Revert "GH-128914: Remove conditional stack effects from `bytecodes.c` and the code generators (GH-128918)" (GH-129202) The commit introduced a ~2.5-3% regression in the free threading build. This reverts commit ab61d3f4303d14a413bc9ae6557c730ffdf7579e.
2025-01-23 04:26:25 -05:00
assert((oparg & 1) == 0);
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
assert(Py_TYPE(PyStackRef_AsPyObjectBorrow(owner))->tp_dictoffset == 0);
gh-104909: Split more LOAD_ATTR specializations (GH-110317) * Split LOAD_ATTR_MODULE * Split LOAD_ATTR_WITH_HINT * Split _GUARD_TYPE_VERSION out of the latter * Split LOAD_ATTR_CLASS * Split LOAD_ATTR_NONDESCRIPTOR_WITH_VALUES * Fix indent of DEOPT_IF in macros * Split LOAD_ATTR_METHOD_LAZY_DICT * Split LOAD_ATTR_NONDESCRIPTOR_NO_DICT * Fix omission of _CHECK_ATTR_METHOD_LAZY_DICT
2023-10-04 08:08:02 -07:00
STAT_INC(LOAD_ATTR, hit);
assert(descr != NULL);
GH-128682: Account for escapes in `DECREF_INPUTS` (GH-129953) * Handle escapes in DECREF_INPUTS * Mark a few more functions as escaping * Replace DECREF_INPUTS with PyStackRef_CLOSE where possible
2025-02-12 17:44:59 +00:00
_PyFrame_SetStackPointer(frame, stack_pointer);
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
PyStackRef_CLOSE(owner);
GH-128682: Account for escapes in `DECREF_INPUTS` (GH-129953) * Handle escapes in DECREF_INPUTS * Mark a few more functions as escaping * Replace DECREF_INPUTS with PyStackRef_CLOSE where possible
2025-02-12 17:44:59 +00:00
stack_pointer = _PyFrame_GetStackPointer(frame);
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
attr = PyStackRef_FromPyObjectNew(descr);
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_tos_cache0 = attr;
_tos_cache1 = PyStackRef_ZERO_BITS;
_tos_cache2 = PyStackRef_ZERO_BITS;
SET_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _CHECK_ATTR_METHOD_LAZY_DICT_r01: {
CHECK_CURRENT_CACHED_VALUES(0);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef owner;
owner = stack_pointer[-1];
GH-140683: JIT: Improve machine code for loading smaller constants on AArch64. (GH-142511) * Use movz and movk instructions for loading 16 and 32 bit operands and oparg. * Loading of 64 bit operands is unchanged.
2025-12-11 12:33:39 +00:00
uint16_t dictoffset = (uint16_t)CURRENT_OPERAND0_16();
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
char *ptr = ((char *)PyStackRef_AsPyObjectBorrow(owner)) + MANAGED_DICT_OFFSET + dictoffset;
PyObject *dict = FT_ATOMIC_LOAD_PTR_ACQUIRE(*(PyObject **)ptr);
if (dict != NULL) {
UOP_STAT_INC(uopcode, miss);
SET_CURRENT_CACHED_VALUES(0);
JUMP_TO_JUMP_TARGET();
}
_tos_cache0 = owner;
SET_CURRENT_CACHED_VALUES(1);
stack_pointer += -1;
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _CHECK_ATTR_METHOD_LAZY_DICT_r11: {
CHECK_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef owner;
_PyStackRef _stack_item_0 = _tos_cache0;
owner = _stack_item_0;
GH-140683: JIT: Improve machine code for loading smaller constants on AArch64. (GH-142511) * Use movz and movk instructions for loading 16 and 32 bit operands and oparg. * Loading of 64 bit operands is unchanged.
2025-12-11 12:33:39 +00:00
uint16_t dictoffset = (uint16_t)CURRENT_OPERAND0_16();
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
char *ptr = ((char *)PyStackRef_AsPyObjectBorrow(owner)) + MANAGED_DICT_OFFSET + dictoffset;
PyObject *dict = FT_ATOMIC_LOAD_PTR_ACQUIRE(*(PyObject **)ptr);
if (dict != NULL) {
UOP_STAT_INC(uopcode, miss);
GH-142718: JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. (GH-142762) JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. * Make sure that stack is in correct state at side exits with TOS cached values * Simplify choice of cached items for side exits
2025-12-15 19:01:29 +00:00
_tos_cache0 = owner;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(1);
JUMP_TO_JUMP_TARGET();
}
_tos_cache0 = owner;
SET_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _CHECK_ATTR_METHOD_LAZY_DICT_r22: {
CHECK_CURRENT_CACHED_VALUES(2);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef owner;
_PyStackRef _stack_item_0 = _tos_cache0;
_PyStackRef _stack_item_1 = _tos_cache1;
owner = _stack_item_1;
GH-140683: JIT: Improve machine code for loading smaller constants on AArch64. (GH-142511) * Use movz and movk instructions for loading 16 and 32 bit operands and oparg. * Loading of 64 bit operands is unchanged.
2025-12-11 12:33:39 +00:00
uint16_t dictoffset = (uint16_t)CURRENT_OPERAND0_16();
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
char *ptr = ((char *)PyStackRef_AsPyObjectBorrow(owner)) + MANAGED_DICT_OFFSET + dictoffset;
PyObject *dict = FT_ATOMIC_LOAD_PTR_ACQUIRE(*(PyObject **)ptr);
if (dict != NULL) {
UOP_STAT_INC(uopcode, miss);
GH-142718: JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. (GH-142762) JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. * Make sure that stack is in correct state at side exits with TOS cached values * Simplify choice of cached items for side exits
2025-12-15 19:01:29 +00:00
_tos_cache1 = owner;
_tos_cache0 = _stack_item_0;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(2);
JUMP_TO_JUMP_TARGET();
}
_tos_cache1 = owner;
_tos_cache0 = _stack_item_0;
SET_CURRENT_CACHED_VALUES(2);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _CHECK_ATTR_METHOD_LAZY_DICT_r33: {
CHECK_CURRENT_CACHED_VALUES(3);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef owner;
_PyStackRef _stack_item_0 = _tos_cache0;
_PyStackRef _stack_item_1 = _tos_cache1;
_PyStackRef _stack_item_2 = _tos_cache2;
owner = _stack_item_2;
GH-140683: JIT: Improve machine code for loading smaller constants on AArch64. (GH-142511) * Use movz and movk instructions for loading 16 and 32 bit operands and oparg. * Loading of 64 bit operands is unchanged.
2025-12-11 12:33:39 +00:00
uint16_t dictoffset = (uint16_t)CURRENT_OPERAND0_16();
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
char *ptr = ((char *)PyStackRef_AsPyObjectBorrow(owner)) + MANAGED_DICT_OFFSET + dictoffset;
PyObject *dict = FT_ATOMIC_LOAD_PTR_ACQUIRE(*(PyObject **)ptr);
if (dict != NULL) {
UOP_STAT_INC(uopcode, miss);
GH-142718: JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. (GH-142762) JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. * Make sure that stack is in correct state at side exits with TOS cached values * Simplify choice of cached items for side exits
2025-12-15 19:01:29 +00:00
_tos_cache2 = owner;
_tos_cache1 = _stack_item_1;
_tos_cache0 = _stack_item_0;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(3);
JUMP_TO_JUMP_TARGET();
}
_tos_cache2 = owner;
_tos_cache1 = _stack_item_1;
_tos_cache0 = _stack_item_0;
SET_CURRENT_CACHED_VALUES(3);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _LOAD_ATTR_METHOD_LAZY_DICT_r02: {
CHECK_CURRENT_CACHED_VALUES(0);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef owner;
_PyStackRef attr;
_PyStackRef self;
oparg = CURRENT_OPARG();
owner = stack_pointer[-1];
GH-140683: JIT: Improve machine code for loading smaller constants on AArch64. (GH-142511) * Use movz and movk instructions for loading 16 and 32 bit operands and oparg. * Loading of 64 bit operands is unchanged.
2025-12-11 12:33:39 +00:00
PyObject *descr = (PyObject *)CURRENT_OPERAND0_64();
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
assert(oparg & 1);
STAT_INC(LOAD_ATTR, hit);
assert(descr != NULL);
assert(_PyType_HasFeature(Py_TYPE(descr), Py_TPFLAGS_METHOD_DESCRIPTOR));
attr = PyStackRef_FromPyObjectNew(descr);
self = owner;
_tos_cache1 = self;
_tos_cache0 = attr;
SET_CURRENT_CACHED_VALUES(2);
stack_pointer += -1;
GH-141794: Limit size of generated machine code. (GH-142228) * Factor out bodies of the largest uops, to reduce jit code size. * Factor out common assert, also reducing jit code size. * Limit size of jitted code for a single executor to 1MB.
2025-12-03 17:43:35 +00:00
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-104909: Split more LOAD_ATTR specializations (GH-110317) * Split LOAD_ATTR_MODULE * Split LOAD_ATTR_WITH_HINT * Split _GUARD_TYPE_VERSION out of the latter * Split LOAD_ATTR_CLASS * Split LOAD_ATTR_NONDESCRIPTOR_WITH_VALUES * Fix indent of DEOPT_IF in macros * Split LOAD_ATTR_METHOD_LAZY_DICT * Split LOAD_ATTR_NONDESCRIPTOR_NO_DICT * Fix omission of _CHECK_ATTR_METHOD_LAZY_DICT
2023-10-04 08:08:02 -07:00
break;
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
case _LOAD_ATTR_METHOD_LAZY_DICT_r12: {
CHECK_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
_PyStackRef owner;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef attr;
_PyStackRef self;
_PyStackRef _stack_item_0 = _tos_cache0;
oparg = CURRENT_OPARG();
owner = _stack_item_0;
GH-140683: JIT: Improve machine code for loading smaller constants on AArch64. (GH-142511) * Use movz and movk instructions for loading 16 and 32 bit operands and oparg. * Loading of 64 bit operands is unchanged.
2025-12-11 12:33:39 +00:00
PyObject *descr = (PyObject *)CURRENT_OPERAND0_64();
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
assert(oparg & 1);
STAT_INC(LOAD_ATTR, hit);
assert(descr != NULL);
assert(_PyType_HasFeature(Py_TYPE(descr), Py_TPFLAGS_METHOD_DESCRIPTOR));
attr = PyStackRef_FromPyObjectNew(descr);
self = owner;
_tos_cache1 = self;
_tos_cache0 = attr;
SET_CURRENT_CACHED_VALUES(2);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-104909: Split more LOAD_ATTR specializations (GH-110317) * Split LOAD_ATTR_MODULE * Split LOAD_ATTR_WITH_HINT * Split _GUARD_TYPE_VERSION out of the latter * Split LOAD_ATTR_CLASS * Split LOAD_ATTR_NONDESCRIPTOR_WITH_VALUES * Fix indent of DEOPT_IF in macros * Split LOAD_ATTR_METHOD_LAZY_DICT * Split LOAD_ATTR_NONDESCRIPTOR_NO_DICT * Fix omission of _CHECK_ATTR_METHOD_LAZY_DICT
2023-10-04 08:08:02 -07:00
break;
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
case _LOAD_ATTR_METHOD_LAZY_DICT_r23: {
CHECK_CURRENT_CACHED_VALUES(2);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
_PyStackRef owner;
_PyStackRef attr;
GH-128914: Remove all but one conditional stack effects (GH-129226) * Remove all 'if (0)' and 'if (1)' conditional stack effects * Use array instead of conditional for BUILD_SLICE args * Refactor LOAD_GLOBAL to use a common conditional uop * Remove conditional stack effects from LOAD_ATTR specializations * Replace conditional stack effects in LOAD_ATTR with a 0 or 1 sized array. * Remove conditional stack effects from CALL_FUNCTION_EX
2025-01-27 16:24:48 +00:00
_PyStackRef self;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef _stack_item_0 = _tos_cache0;
_PyStackRef _stack_item_1 = _tos_cache1;
Revert "GH-128914: Remove conditional stack effects from `bytecodes.c` and the code generators (GH-128918)" (GH-129202) The commit introduced a ~2.5-3% regression in the free threading build. This reverts commit ab61d3f4303d14a413bc9ae6557c730ffdf7579e.
2025-01-23 04:26:25 -05:00
oparg = CURRENT_OPARG();
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
owner = _stack_item_1;
GH-140683: JIT: Improve machine code for loading smaller constants on AArch64. (GH-142511) * Use movz and movk instructions for loading 16 and 32 bit operands and oparg. * Loading of 64 bit operands is unchanged.
2025-12-11 12:33:39 +00:00
PyObject *descr = (PyObject *)CURRENT_OPERAND0_64();
Revert "GH-128914: Remove conditional stack effects from `bytecodes.c` and the code generators (GH-128918)" (GH-129202) The commit introduced a ~2.5-3% regression in the free threading build. This reverts commit ab61d3f4303d14a413bc9ae6557c730ffdf7579e.
2025-01-23 04:26:25 -05:00
assert(oparg & 1);
gh-104909: Split more LOAD_ATTR specializations (GH-110317) * Split LOAD_ATTR_MODULE * Split LOAD_ATTR_WITH_HINT * Split _GUARD_TYPE_VERSION out of the latter * Split LOAD_ATTR_CLASS * Split LOAD_ATTR_NONDESCRIPTOR_WITH_VALUES * Fix indent of DEOPT_IF in macros * Split LOAD_ATTR_METHOD_LAZY_DICT * Split LOAD_ATTR_NONDESCRIPTOR_NO_DICT * Fix omission of _CHECK_ATTR_METHOD_LAZY_DICT
2023-10-04 08:08:02 -07:00
STAT_INC(LOAD_ATTR, hit);
assert(descr != NULL);
assert(_PyType_HasFeature(Py_TYPE(descr), Py_TPFLAGS_METHOD_DESCRIPTOR));
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
attr = PyStackRef_FromPyObjectNew(descr);
gh-118926: Spill deferred references to stack in cases generator (#122748) This automatically spills the results from `_PyStackRef_FromPyObjectNew` to the in-memory stack so that the deferred references are visible to the GC before we make any possibly escaping call. Co-authored-by: Ken Jin <kenjin@python.org>
2024-08-07 13:23:53 -04:00
self = owner;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_tos_cache2 = self;
_tos_cache1 = attr;
_tos_cache0 = _stack_item_0;
SET_CURRENT_CACHED_VALUES(3);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-104909: Split some more insts into ops (#109943) These are the most popular specializations of `LOAD_ATTR` and `STORE_ATTR` that weren't already viable uops: * Split LOAD_ATTR_METHOD_WITH_VALUES * Split LOAD_ATTR_METHOD_NO_DICT * Split LOAD_ATTR_SLOT * Split STORE_ATTR_SLOT * Split STORE_ATTR_INSTANCE_VALUE Also: * Add `-v` flag to code generator which prints a list of non-viable uops (easter-egg: it can print execution counts -- see source) * Double _Py_UOP_MAX_TRACE_LENGTH to 128 I had dropped one of the DEOPT_IF() calls! :-(
2023-09-27 15:27:44 -07:00
break;
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
case _MAYBE_EXPAND_METHOD_r00: {
CHECK_CURRENT_CACHED_VALUES(0);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-131498: Replace single-element arrays with scalars in bytecodes.c (GH-132615)
2025-04-18 07:16:28 -07:00
_PyStackRef self_or_null;
_PyStackRef callable;
GH-122029: Break INSTRUMENTED_CALL into micro-ops, so that its behavior is consistent with CALL (GH-122177)
2024-07-26 14:35:57 +01:00
oparg = CURRENT_OPARG();
GH-131498: Replace single-element arrays with scalars in bytecodes.c (GH-132615)
2025-04-18 07:16:28 -07:00
self_or_null = stack_pointer[-1 - oparg];
callable = stack_pointer[-2 - oparg];
if (PyStackRef_TYPE(callable) == &PyMethod_Type && PyStackRef_IsNull(self_or_null)) {
PyObject *callable_o = PyStackRef_AsPyObjectBorrow(callable);
GH-122029: Break INSTRUMENTED_CALL into micro-ops, so that its behavior is consistent with CALL (GH-122177)
2024-07-26 14:35:57 +01:00
PyObject *self = ((PyMethodObject *)callable_o)->im_self;
GH-131498: Replace single-element arrays with scalars in bytecodes.c (GH-132615)
2025-04-18 07:16:28 -07:00
self_or_null = PyStackRef_FromPyObjectNew(self);
GH-122029: Break INSTRUMENTED_CALL into micro-ops, so that its behavior is consistent with CALL (GH-122177)
2024-07-26 14:35:57 +01:00
PyObject *method = ((PyMethodObject *)callable_o)->im_func;
GH-131498: Replace single-element arrays with scalars in bytecodes.c (GH-132615)
2025-04-18 07:16:28 -07:00
_PyStackRef temp = callable;
callable = PyStackRef_FromPyObjectNew(method);
stack_pointer[-2 - oparg] = callable;
stack_pointer[-1 - oparg] = self_or_null;
GH-128682: Make `PyStackRef_CLOSE` escaping. (GH-129404)
2025-02-03 12:41:32 +00:00
_PyFrame_SetStackPointer(frame, stack_pointer);
GH-119866: Spill the stack around escaping calls. (GH-124392) * Spill the evaluation around escaping calls in the generated interpreter and JIT. * The code generator tracks live, cached values so they can be saved to memory when needed. * Spills the stack pointer around escaping calls, so that the exact stack is visible to the cycle GC.
2024-10-07 14:56:39 +01:00
PyStackRef_CLOSE(temp);
GH-128682: Make `PyStackRef_CLOSE` escaping. (GH-129404)
2025-02-03 12:41:32 +00:00
stack_pointer = _PyFrame_GetStackPointer(frame);
GH-122029: Break INSTRUMENTED_CALL into micro-ops, so that its behavior is consistent with CALL (GH-122177)
2024-07-26 14:35:57 +01:00
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_tos_cache0 = PyStackRef_ZERO_BITS;
_tos_cache1 = PyStackRef_ZERO_BITS;
_tos_cache2 = PyStackRef_ZERO_BITS;
SET_CURRENT_CACHED_VALUES(0);
GH-131498: Replace single-element arrays with scalars in bytecodes.c (GH-132615)
2025-04-18 07:16:28 -07:00
stack_pointer[-2 - oparg] = callable;
stack_pointer[-1 - oparg] = self_or_null;
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-122029: Break INSTRUMENTED_CALL into micro-ops, so that its behavior is consistent with CALL (GH-122177)
2024-07-26 14:35:57 +01:00
break;
}
GH-111485: Factor out tier 2 code generation from the rest of the interpreter code generator (GH-112968)
2023-12-12 12:12:17 +00:00
GH-122029: Break INSTRUMENTED_CALL into micro-ops, so that its behavior is consistent with CALL (GH-122177)
2024-07-26 14:35:57 +01:00
/* _DO_CALL is not a viable micro-op for tier 2 because it uses the 'this_instr' variable */
GH-111485: Factor out tier 2 code generation from the rest of the interpreter code generator (GH-112968)
2023-12-12 12:12:17 +00:00
GH-122029: Break INSTRUMENTED_CALL into micro-ops, so that its behavior is consistent with CALL (GH-122177)
2024-07-26 14:35:57 +01:00
/* _MONITOR_CALL is not a viable micro-op for tier 2 because it uses the 'this_instr' variable */
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
case _PY_FRAME_GENERAL_r01: {
CHECK_CURRENT_CACHED_VALUES(0);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
_PyStackRef *args;
GH-131498: Replace single-element arrays with scalars in bytecodes.c (GH-132615)
2025-04-18 07:16:28 -07:00
_PyStackRef self_or_null;
_PyStackRef callable;
GH-135379: Remove types from stack items in code generator. (GH-135384) * Make casts explicit in the instruction definitions
2025-06-11 15:52:25 +01:00
_PyStackRef new_frame;
GH-118095: Use broader specializations of CALL in tier 1, for better tier 2 support of calls. (GH-118322) * Add CALL_PY_GENERAL, CALL_BOUND_METHOD_GENERAL and call CALL_NON_PY_GENERAL specializations. * Remove CALL_PY_WITH_DEFAULTS specialization * Use CALL_NON_PY_GENERAL in more cases when otherwise failing to specialize
2024-05-04 12:11:11 +01:00
oparg = CURRENT_OPARG();
args = &stack_pointer[-oparg];
GH-131498: Replace single-element arrays with scalars in bytecodes.c (GH-132615)
2025-04-18 07:16:28 -07:00
self_or_null = stack_pointer[-1 - oparg];
callable = stack_pointer[-2 - oparg];
PyObject *callable_o = PyStackRef_AsPyObjectBorrow(callable);
GH-118095: Use broader specializations of CALL in tier 1, for better tier 2 support of calls. (GH-118322) * Add CALL_PY_GENERAL, CALL_BOUND_METHOD_GENERAL and call CALL_NON_PY_GENERAL specializations. * Remove CALL_PY_WITH_DEFAULTS specialization * Use CALL_NON_PY_GENERAL in more cases when otherwise failing to specialize
2024-05-04 12:11:11 +01:00
int total_args = oparg;
GH-131498: Replace single-element arrays with scalars in bytecodes.c (GH-132615)
2025-04-18 07:16:28 -07:00
if (!PyStackRef_IsNull(self_or_null)) {
GH-118095: Use broader specializations of CALL in tier 1, for better tier 2 support of calls. (GH-118322) * Add CALL_PY_GENERAL, CALL_BOUND_METHOD_GENERAL and call CALL_NON_PY_GENERAL specializations. * Remove CALL_PY_WITH_DEFAULTS specialization * Use CALL_NON_PY_GENERAL in more cases when otherwise failing to specialize
2024-05-04 12:11:11 +01:00
args--;
total_args++;
}
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
assert(Py_TYPE(callable_o) == &PyFunction_Type);
int code_flags = ((PyCodeObject*)PyFunction_GET_CODE(callable_o))->co_flags;
PyObject *locals = code_flags & CO_OPTIMIZED ? NULL : Py_NewRef(PyFunction_GET_GLOBALS(callable_o));
GH-119866: Spill the stack around escaping calls. (GH-124392) * Spill the evaluation around escaping calls in the generated interpreter and JIT. * The code generator tracks live, cached values so they can be saved to memory when needed. * Spills the stack pointer around escaping calls, so that the exact stack is visible to the cycle GC.
2024-10-07 14:56:39 +01:00
_PyFrame_SetStackPointer(frame, stack_pointer);
_PyInterpreterFrame *temp = _PyEvalFramePushAndInit(
GH-131498: Replace single-element arrays with scalars in bytecodes.c (GH-132615)
2025-04-18 07:16:28 -07:00
tstate, callable, locals,
GH-118093: Make `CALL_ALLOC_AND_ENTER_INIT` suitable for tier 2. (GH-123140) * Convert CALL_ALLOC_AND_ENTER_INIT to micro-ops such that tier 2 supports it * Allow inexact arguments for CALL_ALLOC_AND_ENTER_INIT.
2024-08-20 16:52:58 +01:00
args, total_args, NULL, frame
GH-118095: Use broader specializations of CALL in tier 1, for better tier 2 support of calls. (GH-118322) * Add CALL_PY_GENERAL, CALL_BOUND_METHOD_GENERAL and call CALL_NON_PY_GENERAL specializations. * Remove CALL_PY_WITH_DEFAULTS specialization * Use CALL_NON_PY_GENERAL in more cases when otherwise failing to specialize
2024-05-04 12:11:11 +01:00
);
GH-119866: Spill the stack around escaping calls. (GH-124392) * Spill the evaluation around escaping calls in the generated interpreter and JIT. * The code generator tracks live, cached values so they can be saved to memory when needed. * Spills the stack pointer around escaping calls, so that the exact stack is visible to the cycle GC.
2024-10-07 14:56:39 +01:00
stack_pointer = _PyFrame_GetStackPointer(frame);
GH-118095: Use broader specializations of CALL in tier 1, for better tier 2 support of calls. (GH-118322) * Add CALL_PY_GENERAL, CALL_BOUND_METHOD_GENERAL and call CALL_NON_PY_GENERAL specializations. * Remove CALL_PY_WITH_DEFAULTS specialization * Use CALL_NON_PY_GENERAL in more cases when otherwise failing to specialize
2024-05-04 12:11:11 +01:00
stack_pointer += -2 - oparg;
GH-141794: Limit size of generated machine code. (GH-142228) * Factor out bodies of the largest uops, to reduce jit code size. * Factor out common assert, also reducing jit code size. * Limit size of jitted code for a single executor to 1MB.
2025-12-03 17:43:35 +00:00
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
GH-119866: Spill the stack around escaping calls. (GH-124392) * Spill the evaluation around escaping calls in the generated interpreter and JIT. * The code generator tracks live, cached values so they can be saved to memory when needed. * Spills the stack pointer around escaping calls, so that the exact stack is visible to the cycle GC.
2024-10-07 14:56:39 +01:00
if (temp == NULL) {
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(0);
GH-118095: Use broader specializations of CALL in tier 1, for better tier 2 support of calls. (GH-118322) * Add CALL_PY_GENERAL, CALL_BOUND_METHOD_GENERAL and call CALL_NON_PY_GENERAL specializations. * Remove CALL_PY_WITH_DEFAULTS specialization * Use CALL_NON_PY_GENERAL in more cases when otherwise failing to specialize
2024-05-04 12:11:11 +01:00
JUMP_TO_ERROR();
}
GH-135379: Remove types from stack items in code generator. (GH-135384) * Make casts explicit in the instruction definitions
2025-06-11 15:52:25 +01:00
new_frame = PyStackRef_Wrap(temp);
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_tos_cache0 = new_frame;
_tos_cache1 = PyStackRef_ZERO_BITS;
_tos_cache2 = PyStackRef_ZERO_BITS;
SET_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-118095: Use broader specializations of CALL in tier 1, for better tier 2 support of calls. (GH-118322) * Add CALL_PY_GENERAL, CALL_BOUND_METHOD_GENERAL and call CALL_NON_PY_GENERAL specializations. * Remove CALL_PY_WITH_DEFAULTS specialization * Use CALL_NON_PY_GENERAL in more cases when otherwise failing to specialize
2024-05-04 12:11:11 +01:00
break;
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
case _CHECK_FUNCTION_VERSION_r00: {
CHECK_CURRENT_CACHED_VALUES(0);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-131498: Replace single-element arrays with scalars in bytecodes.c (GH-132615)
2025-04-18 07:16:28 -07:00
_PyStackRef callable;
GH-118095: Use broader specializations of CALL in tier 1, for better tier 2 support of calls. (GH-118322) * Add CALL_PY_GENERAL, CALL_BOUND_METHOD_GENERAL and call CALL_NON_PY_GENERAL specializations. * Remove CALL_PY_WITH_DEFAULTS specialization * Use CALL_NON_PY_GENERAL in more cases when otherwise failing to specialize
2024-05-04 12:11:11 +01:00
oparg = CURRENT_OPARG();
GH-131498: Replace single-element arrays with scalars in bytecodes.c (GH-132615)
2025-04-18 07:16:28 -07:00
callable = stack_pointer[-2 - oparg];
GH-140683: JIT: Improve machine code for loading smaller constants on AArch64. (GH-142511) * Use movz and movk instructions for loading 16 and 32 bit operands and oparg. * Loading of 64 bit operands is unchanged.
2025-12-11 12:33:39 +00:00
uint32_t func_version = (uint32_t)CURRENT_OPERAND0_32();
GH-131498: Replace single-element arrays with scalars in bytecodes.c (GH-132615)
2025-04-18 07:16:28 -07:00
PyObject *callable_o = PyStackRef_AsPyObjectBorrow(callable);
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
if (!PyFunction_Check(callable_o)) {
GH-118095: Use broader specializations of CALL in tier 1, for better tier 2 support of calls. (GH-118322) * Add CALL_PY_GENERAL, CALL_BOUND_METHOD_GENERAL and call CALL_NON_PY_GENERAL specializations. * Remove CALL_PY_WITH_DEFAULTS specialization * Use CALL_NON_PY_GENERAL in more cases when otherwise failing to specialize
2024-05-04 12:11:11 +01:00
UOP_STAT_INC(uopcode, miss);
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(0);
JUMP_TO_JUMP_TARGET();
}
PyFunctionObject *func = (PyFunctionObject *)callable_o;
if (func->func_version != func_version) {
UOP_STAT_INC(uopcode, miss);
SET_CURRENT_CACHED_VALUES(0);
JUMP_TO_JUMP_TARGET();
}
SET_CURRENT_CACHED_VALUES(0);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _CHECK_FUNCTION_VERSION_INLINE_r00: {
CHECK_CURRENT_CACHED_VALUES(0);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-140683: JIT: Improve machine code for loading smaller constants on AArch64. (GH-142511) * Use movz and movk instructions for loading 16 and 32 bit operands and oparg. * Loading of 64 bit operands is unchanged.
2025-12-11 12:33:39 +00:00
uint32_t func_version = (uint32_t)CURRENT_OPERAND0_32();
PyObject *callable_o = (PyObject *)CURRENT_OPERAND1_64();
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
assert(PyFunction_Check(callable_o));
PyFunctionObject *func = (PyFunctionObject *)callable_o;
if (func->func_version != func_version) {
UOP_STAT_INC(uopcode, miss);
SET_CURRENT_CACHED_VALUES(0);
JUMP_TO_JUMP_TARGET();
}
SET_CURRENT_CACHED_VALUES(0);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _CHECK_FUNCTION_VERSION_INLINE_r11: {
CHECK_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef _stack_item_0 = _tos_cache0;
GH-140683: JIT: Improve machine code for loading smaller constants on AArch64. (GH-142511) * Use movz and movk instructions for loading 16 and 32 bit operands and oparg. * Loading of 64 bit operands is unchanged.
2025-12-11 12:33:39 +00:00
uint32_t func_version = (uint32_t)CURRENT_OPERAND0_32();
PyObject *callable_o = (PyObject *)CURRENT_OPERAND1_64();
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
assert(PyFunction_Check(callable_o));
PyFunctionObject *func = (PyFunctionObject *)callable_o;
if (func->func_version != func_version) {
UOP_STAT_INC(uopcode, miss);
GH-142718: JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. (GH-142762) JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. * Make sure that stack is in correct state at side exits with TOS cached values * Simplify choice of cached items for side exits
2025-12-15 19:01:29 +00:00
_tos_cache0 = _stack_item_0;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(1);
GH-118095: Use broader specializations of CALL in tier 1, for better tier 2 support of calls. (GH-118322) * Add CALL_PY_GENERAL, CALL_BOUND_METHOD_GENERAL and call CALL_NON_PY_GENERAL specializations. * Remove CALL_PY_WITH_DEFAULTS specialization * Use CALL_NON_PY_GENERAL in more cases when otherwise failing to specialize
2024-05-04 12:11:11 +01:00
JUMP_TO_JUMP_TARGET();
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_tos_cache0 = _stack_item_0;
SET_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _CHECK_FUNCTION_VERSION_INLINE_r22: {
CHECK_CURRENT_CACHED_VALUES(2);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef _stack_item_0 = _tos_cache0;
_PyStackRef _stack_item_1 = _tos_cache1;
GH-140683: JIT: Improve machine code for loading smaller constants on AArch64. (GH-142511) * Use movz and movk instructions for loading 16 and 32 bit operands and oparg. * Loading of 64 bit operands is unchanged.
2025-12-11 12:33:39 +00:00
uint32_t func_version = (uint32_t)CURRENT_OPERAND0_32();
PyObject *callable_o = (PyObject *)CURRENT_OPERAND1_64();
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
assert(PyFunction_Check(callable_o));
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
PyFunctionObject *func = (PyFunctionObject *)callable_o;
GH-118095: Use broader specializations of CALL in tier 1, for better tier 2 support of calls. (GH-118322) * Add CALL_PY_GENERAL, CALL_BOUND_METHOD_GENERAL and call CALL_NON_PY_GENERAL specializations. * Remove CALL_PY_WITH_DEFAULTS specialization * Use CALL_NON_PY_GENERAL in more cases when otherwise failing to specialize
2024-05-04 12:11:11 +01:00
if (func->func_version != func_version) {
UOP_STAT_INC(uopcode, miss);
GH-142718: JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. (GH-142762) JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. * Make sure that stack is in correct state at side exits with TOS cached values * Simplify choice of cached items for side exits
2025-12-15 19:01:29 +00:00
_tos_cache1 = _stack_item_1;
_tos_cache0 = _stack_item_0;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(2);
GH-118095: Use broader specializations of CALL in tier 1, for better tier 2 support of calls. (GH-118322) * Add CALL_PY_GENERAL, CALL_BOUND_METHOD_GENERAL and call CALL_NON_PY_GENERAL specializations. * Remove CALL_PY_WITH_DEFAULTS specialization * Use CALL_NON_PY_GENERAL in more cases when otherwise failing to specialize
2024-05-04 12:11:11 +01:00
JUMP_TO_JUMP_TARGET();
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_tos_cache1 = _stack_item_1;
_tos_cache0 = _stack_item_0;
SET_CURRENT_CACHED_VALUES(2);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-118095: Use broader specializations of CALL in tier 1, for better tier 2 support of calls. (GH-118322) * Add CALL_PY_GENERAL, CALL_BOUND_METHOD_GENERAL and call CALL_NON_PY_GENERAL specializations. * Remove CALL_PY_WITH_DEFAULTS specialization * Use CALL_NON_PY_GENERAL in more cases when otherwise failing to specialize
2024-05-04 12:11:11 +01:00
break;
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
case _CHECK_FUNCTION_VERSION_INLINE_r33: {
CHECK_CURRENT_CACHED_VALUES(3);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef _stack_item_0 = _tos_cache0;
_PyStackRef _stack_item_1 = _tos_cache1;
_PyStackRef _stack_item_2 = _tos_cache2;
GH-140683: JIT: Improve machine code for loading smaller constants on AArch64. (GH-142511) * Use movz and movk instructions for loading 16 and 32 bit operands and oparg. * Loading of 64 bit operands is unchanged.
2025-12-11 12:33:39 +00:00
uint32_t func_version = (uint32_t)CURRENT_OPERAND0_32();
PyObject *callable_o = (PyObject *)CURRENT_OPERAND1_64();
gh-120619: Strength reduce function guards, support 2-operand uop forms (GH-124846) Co-authored-by: Brandt Bucher <brandtbucher@gmail.com>
2024-11-09 11:35:33 +08:00
assert(PyFunction_Check(callable_o));
PyFunctionObject *func = (PyFunctionObject *)callable_o;
if (func->func_version != func_version) {
UOP_STAT_INC(uopcode, miss);
GH-142718: JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. (GH-142762) JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. * Make sure that stack is in correct state at side exits with TOS cached values * Simplify choice of cached items for side exits
2025-12-15 19:01:29 +00:00
_tos_cache2 = _stack_item_2;
_tos_cache1 = _stack_item_1;
_tos_cache0 = _stack_item_0;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(3);
gh-120619: Strength reduce function guards, support 2-operand uop forms (GH-124846) Co-authored-by: Brandt Bucher <brandtbucher@gmail.com>
2024-11-09 11:35:33 +08:00
JUMP_TO_JUMP_TARGET();
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_tos_cache2 = _stack_item_2;
_tos_cache1 = _stack_item_1;
_tos_cache0 = _stack_item_0;
SET_CURRENT_CACHED_VALUES(3);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-120619: Strength reduce function guards, support 2-operand uop forms (GH-124846) Co-authored-by: Brandt Bucher <brandtbucher@gmail.com>
2024-11-09 11:35:33 +08:00
break;
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
case _CHECK_METHOD_VERSION_r00: {
CHECK_CURRENT_CACHED_VALUES(0);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-131498: Replace single-element arrays with scalars in bytecodes.c (GH-132615)
2025-04-18 07:16:28 -07:00
_PyStackRef null;
_PyStackRef callable;
GH-118095: Use broader specializations of CALL in tier 1, for better tier 2 support of calls. (GH-118322) * Add CALL_PY_GENERAL, CALL_BOUND_METHOD_GENERAL and call CALL_NON_PY_GENERAL specializations. * Remove CALL_PY_WITH_DEFAULTS specialization * Use CALL_NON_PY_GENERAL in more cases when otherwise failing to specialize
2024-05-04 12:11:11 +01:00
oparg = CURRENT_OPARG();
GH-131498: Replace single-element arrays with scalars in bytecodes.c (GH-132615)
2025-04-18 07:16:28 -07:00
null = stack_pointer[-1 - oparg];
callable = stack_pointer[-2 - oparg];
GH-140683: JIT: Improve machine code for loading smaller constants on AArch64. (GH-142511) * Use movz and movk instructions for loading 16 and 32 bit operands and oparg. * Loading of 64 bit operands is unchanged.
2025-12-11 12:33:39 +00:00
uint32_t func_version = (uint32_t)CURRENT_OPERAND0_32();
GH-131498: Replace single-element arrays with scalars in bytecodes.c (GH-132615)
2025-04-18 07:16:28 -07:00
PyObject *callable_o = PyStackRef_AsPyObjectBorrow(callable);
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
if (Py_TYPE(callable_o) != &PyMethod_Type) {
GH-118095: Use broader specializations of CALL in tier 1, for better tier 2 support of calls. (GH-118322) * Add CALL_PY_GENERAL, CALL_BOUND_METHOD_GENERAL and call CALL_NON_PY_GENERAL specializations. * Remove CALL_PY_WITH_DEFAULTS specialization * Use CALL_NON_PY_GENERAL in more cases when otherwise failing to specialize
2024-05-04 12:11:11 +01:00
UOP_STAT_INC(uopcode, miss);
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(0);
GH-118095: Use broader specializations of CALL in tier 1, for better tier 2 support of calls. (GH-118322) * Add CALL_PY_GENERAL, CALL_BOUND_METHOD_GENERAL and call CALL_NON_PY_GENERAL specializations. * Remove CALL_PY_WITH_DEFAULTS specialization * Use CALL_NON_PY_GENERAL in more cases when otherwise failing to specialize
2024-05-04 12:11:11 +01:00
JUMP_TO_JUMP_TARGET();
}
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
PyObject *func = ((PyMethodObject *)callable_o)->im_func;
GH-118095: Use broader specializations of CALL in tier 1, for better tier 2 support of calls. (GH-118322) * Add CALL_PY_GENERAL, CALL_BOUND_METHOD_GENERAL and call CALL_NON_PY_GENERAL specializations. * Remove CALL_PY_WITH_DEFAULTS specialization * Use CALL_NON_PY_GENERAL in more cases when otherwise failing to specialize
2024-05-04 12:11:11 +01:00
if (!PyFunction_Check(func)) {
UOP_STAT_INC(uopcode, miss);
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(0);
GH-118095: Use broader specializations of CALL in tier 1, for better tier 2 support of calls. (GH-118322) * Add CALL_PY_GENERAL, CALL_BOUND_METHOD_GENERAL and call CALL_NON_PY_GENERAL specializations. * Remove CALL_PY_WITH_DEFAULTS specialization * Use CALL_NON_PY_GENERAL in more cases when otherwise failing to specialize
2024-05-04 12:11:11 +01:00
JUMP_TO_JUMP_TARGET();
}
if (((PyFunctionObject *)func)->func_version != func_version) {
UOP_STAT_INC(uopcode, miss);
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(0);
GH-118095: Use broader specializations of CALL in tier 1, for better tier 2 support of calls. (GH-118322) * Add CALL_PY_GENERAL, CALL_BOUND_METHOD_GENERAL and call CALL_NON_PY_GENERAL specializations. * Remove CALL_PY_WITH_DEFAULTS specialization * Use CALL_NON_PY_GENERAL in more cases when otherwise failing to specialize
2024-05-04 12:11:11 +01:00
JUMP_TO_JUMP_TARGET();
}
GH-131498: Replace single-element arrays with scalars in bytecodes.c (GH-132615)
2025-04-18 07:16:28 -07:00
if (!PyStackRef_IsNull(null)) {
GH-118095: Use broader specializations of CALL in tier 1, for better tier 2 support of calls. (GH-118322) * Add CALL_PY_GENERAL, CALL_BOUND_METHOD_GENERAL and call CALL_NON_PY_GENERAL specializations. * Remove CALL_PY_WITH_DEFAULTS specialization * Use CALL_NON_PY_GENERAL in more cases when otherwise failing to specialize
2024-05-04 12:11:11 +01:00
UOP_STAT_INC(uopcode, miss);
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(0);
GH-118095: Use broader specializations of CALL in tier 1, for better tier 2 support of calls. (GH-118322) * Add CALL_PY_GENERAL, CALL_BOUND_METHOD_GENERAL and call CALL_NON_PY_GENERAL specializations. * Remove CALL_PY_WITH_DEFAULTS specialization * Use CALL_NON_PY_GENERAL in more cases when otherwise failing to specialize
2024-05-04 12:11:11 +01:00
JUMP_TO_JUMP_TARGET();
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(0);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-118095: Use broader specializations of CALL in tier 1, for better tier 2 support of calls. (GH-118322) * Add CALL_PY_GENERAL, CALL_BOUND_METHOD_GENERAL and call CALL_NON_PY_GENERAL specializations. * Remove CALL_PY_WITH_DEFAULTS specialization * Use CALL_NON_PY_GENERAL in more cases when otherwise failing to specialize
2024-05-04 12:11:11 +01:00
break;
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
case _EXPAND_METHOD_r00: {
CHECK_CURRENT_CACHED_VALUES(0);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-131498: Replace single-element arrays with scalars in bytecodes.c (GH-132615)
2025-04-18 07:16:28 -07:00
_PyStackRef self_or_null;
_PyStackRef callable;
GH-118095: Use broader specializations of CALL in tier 1, for better tier 2 support of calls. (GH-118322) * Add CALL_PY_GENERAL, CALL_BOUND_METHOD_GENERAL and call CALL_NON_PY_GENERAL specializations. * Remove CALL_PY_WITH_DEFAULTS specialization * Use CALL_NON_PY_GENERAL in more cases when otherwise failing to specialize
2024-05-04 12:11:11 +01:00
oparg = CURRENT_OPARG();
GH-131498: Replace single-element arrays with scalars in bytecodes.c (GH-132615)
2025-04-18 07:16:28 -07:00
self_or_null = stack_pointer[-1 - oparg];
callable = stack_pointer[-2 - oparg];
PyObject *callable_o = PyStackRef_AsPyObjectBorrow(callable);
assert(PyStackRef_IsNull(self_or_null));
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
assert(Py_TYPE(callable_o) == &PyMethod_Type);
GH-131498: Replace single-element arrays with scalars in bytecodes.c (GH-132615)
2025-04-18 07:16:28 -07:00
self_or_null = PyStackRef_FromPyObjectNew(((PyMethodObject *)callable_o)->im_self);
_PyStackRef temp = callable;
callable = PyStackRef_FromPyObjectNew(((PyMethodObject *)callable_o)->im_func);
assert(PyStackRef_FunctionCheck(callable));
stack_pointer[-2 - oparg] = callable;
stack_pointer[-1 - oparg] = self_or_null;
GH-128682: Make `PyStackRef_CLOSE` escaping. (GH-129404)
2025-02-03 12:41:32 +00:00
_PyFrame_SetStackPointer(frame, stack_pointer);
GH-119866: Spill the stack around escaping calls. (GH-124392) * Spill the evaluation around escaping calls in the generated interpreter and JIT. * The code generator tracks live, cached values so they can be saved to memory when needed. * Spills the stack pointer around escaping calls, so that the exact stack is visible to the cycle GC.
2024-10-07 14:56:39 +01:00
PyStackRef_CLOSE(temp);
GH-128682: Make `PyStackRef_CLOSE` escaping. (GH-129404)
2025-02-03 12:41:32 +00:00
stack_pointer = _PyFrame_GetStackPointer(frame);
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_tos_cache0 = PyStackRef_ZERO_BITS;
_tos_cache1 = PyStackRef_ZERO_BITS;
_tos_cache2 = PyStackRef_ZERO_BITS;
SET_CURRENT_CACHED_VALUES(0);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-118095: Use broader specializations of CALL in tier 1, for better tier 2 support of calls. (GH-118322) * Add CALL_PY_GENERAL, CALL_BOUND_METHOD_GENERAL and call CALL_NON_PY_GENERAL specializations. * Remove CALL_PY_WITH_DEFAULTS specialization * Use CALL_NON_PY_GENERAL in more cases when otherwise failing to specialize
2024-05-04 12:11:11 +01:00
break;
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
case _CHECK_IS_NOT_PY_CALLABLE_r00: {
CHECK_CURRENT_CACHED_VALUES(0);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-131498: Replace single-element arrays with scalars in bytecodes.c (GH-132615)
2025-04-18 07:16:28 -07:00
_PyStackRef callable;
GH-118095: Use broader specializations of CALL in tier 1, for better tier 2 support of calls. (GH-118322) * Add CALL_PY_GENERAL, CALL_BOUND_METHOD_GENERAL and call CALL_NON_PY_GENERAL specializations. * Remove CALL_PY_WITH_DEFAULTS specialization * Use CALL_NON_PY_GENERAL in more cases when otherwise failing to specialize
2024-05-04 12:11:11 +01:00
oparg = CURRENT_OPARG();
GH-131498: Replace single-element arrays with scalars in bytecodes.c (GH-132615)
2025-04-18 07:16:28 -07:00
callable = stack_pointer[-2 - oparg];
PyObject *callable_o = PyStackRef_AsPyObjectBorrow(callable);
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
if (PyFunction_Check(callable_o)) {
GH-118095: Use broader specializations of CALL in tier 1, for better tier 2 support of calls. (GH-118322) * Add CALL_PY_GENERAL, CALL_BOUND_METHOD_GENERAL and call CALL_NON_PY_GENERAL specializations. * Remove CALL_PY_WITH_DEFAULTS specialization * Use CALL_NON_PY_GENERAL in more cases when otherwise failing to specialize
2024-05-04 12:11:11 +01:00
UOP_STAT_INC(uopcode, miss);
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(0);
GH-118095: Use broader specializations of CALL in tier 1, for better tier 2 support of calls. (GH-118322) * Add CALL_PY_GENERAL, CALL_BOUND_METHOD_GENERAL and call CALL_NON_PY_GENERAL specializations. * Remove CALL_PY_WITH_DEFAULTS specialization * Use CALL_NON_PY_GENERAL in more cases when otherwise failing to specialize
2024-05-04 12:11:11 +01:00
JUMP_TO_JUMP_TARGET();
}
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
if (Py_TYPE(callable_o) == &PyMethod_Type) {
GH-118095: Use broader specializations of CALL in tier 1, for better tier 2 support of calls. (GH-118322) * Add CALL_PY_GENERAL, CALL_BOUND_METHOD_GENERAL and call CALL_NON_PY_GENERAL specializations. * Remove CALL_PY_WITH_DEFAULTS specialization * Use CALL_NON_PY_GENERAL in more cases when otherwise failing to specialize
2024-05-04 12:11:11 +01:00
UOP_STAT_INC(uopcode, miss);
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(0);
GH-118095: Use broader specializations of CALL in tier 1, for better tier 2 support of calls. (GH-118322) * Add CALL_PY_GENERAL, CALL_BOUND_METHOD_GENERAL and call CALL_NON_PY_GENERAL specializations. * Remove CALL_PY_WITH_DEFAULTS specialization * Use CALL_NON_PY_GENERAL in more cases when otherwise failing to specialize
2024-05-04 12:11:11 +01:00
JUMP_TO_JUMP_TARGET();
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(0);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-118095: Use broader specializations of CALL in tier 1, for better tier 2 support of calls. (GH-118322) * Add CALL_PY_GENERAL, CALL_BOUND_METHOD_GENERAL and call CALL_NON_PY_GENERAL specializations. * Remove CALL_PY_WITH_DEFAULTS specialization * Use CALL_NON_PY_GENERAL in more cases when otherwise failing to specialize
2024-05-04 12:11:11 +01:00
break;
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
case _CALL_NON_PY_GENERAL_r01: {
CHECK_CURRENT_CACHED_VALUES(0);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
_PyStackRef *args;
GH-131498: Replace single-element arrays with scalars in bytecodes.c (GH-132615)
2025-04-18 07:16:28 -07:00
_PyStackRef self_or_null;
_PyStackRef callable;
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
_PyStackRef res;
GH-118095: Use broader specializations of CALL in tier 1, for better tier 2 support of calls. (GH-118322) * Add CALL_PY_GENERAL, CALL_BOUND_METHOD_GENERAL and call CALL_NON_PY_GENERAL specializations. * Remove CALL_PY_WITH_DEFAULTS specialization * Use CALL_NON_PY_GENERAL in more cases when otherwise failing to specialize
2024-05-04 12:11:11 +01:00
oparg = CURRENT_OPARG();
args = &stack_pointer[-oparg];
GH-131498: Replace single-element arrays with scalars in bytecodes.c (GH-132615)
2025-04-18 07:16:28 -07:00
self_or_null = stack_pointer[-1 - oparg];
callable = stack_pointer[-2 - oparg];
GH-118095: Use broader specializations of CALL in tier 1, for better tier 2 support of calls. (GH-118322) * Add CALL_PY_GENERAL, CALL_BOUND_METHOD_GENERAL and call CALL_NON_PY_GENERAL specializations. * Remove CALL_PY_WITH_DEFAULTS specialization * Use CALL_NON_PY_GENERAL in more cases when otherwise failing to specialize
2024-05-04 12:11:11 +01:00
#if TIER_ONE
assert(opcode != INSTRUMENTED_CALL);
#endif
int total_args = oparg;
GH-128682: Convert explicit loops closing arrays into `DECREF_INPUTS`. (GH-128822) * Mark Py_DECREF and Py_XDECREF as escaping * Remove explicit loops for clearing array inputs
2025-01-14 15:08:56 +00:00
_PyStackRef *arguments = args;
GH-131498: Replace single-element arrays with scalars in bytecodes.c (GH-132615)
2025-04-18 07:16:28 -07:00
if (!PyStackRef_IsNull(self_or_null)) {
GH-128682: Convert explicit loops closing arrays into `DECREF_INPUTS`. (GH-128822) * Mark Py_DECREF and Py_XDECREF as escaping * Remove explicit loops for clearing array inputs
2025-01-14 15:08:56 +00:00
arguments--;
GH-118095: Use broader specializations of CALL in tier 1, for better tier 2 support of calls. (GH-118322) * Add CALL_PY_GENERAL, CALL_BOUND_METHOD_GENERAL and call CALL_NON_PY_GENERAL specializations. * Remove CALL_PY_WITH_DEFAULTS specialization * Use CALL_NON_PY_GENERAL in more cases when otherwise failing to specialize
2024-05-04 12:11:11 +01:00
total_args++;
}
GH-119866: Spill the stack around escaping calls. (GH-124392) * Spill the evaluation around escaping calls in the generated interpreter and JIT. * The code generator tracks live, cached values so they can be saved to memory when needed. * Spills the stack pointer around escaping calls, so that the exact stack is visible to the cycle GC.
2024-10-07 14:56:39 +01:00
_PyFrame_SetStackPointer(frame, stack_pointer);
GH-141794: Limit size of generated machine code. (GH-142228) * Factor out bodies of the largest uops, to reduce jit code size. * Factor out common assert, also reducing jit code size. * Limit size of jitted code for a single executor to 1MB.
2025-12-03 17:43:35 +00:00
PyObject *res_o = _Py_VectorCall_StackRefSteal(
callable,
arguments,
total_args,
PyStackRef_NULL);
GH-119866: Spill the stack around escaping calls. (GH-124392) * Spill the evaluation around escaping calls in the generated interpreter and JIT. * The code generator tracks live, cached values so they can be saved to memory when needed. * Spills the stack pointer around escaping calls, so that the exact stack is visible to the cycle GC.
2024-10-07 14:56:39 +01:00
stack_pointer = _PyFrame_GetStackPointer(frame);
GH-128682: Make `PyStackRef_CLOSE` escaping. (GH-129404)
2025-02-03 12:41:32 +00:00
if (res_o == NULL) {
GH-141794: Limit size of generated machine code. (GH-142228) * Factor out bodies of the largest uops, to reduce jit code size. * Factor out common assert, also reducing jit code size. * Limit size of jitted code for a single executor to 1MB.
2025-12-03 17:43:35 +00:00
stack_pointer += -2 - oparg;
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(0);
GH-128682: Make `PyStackRef_CLOSE` escaping. (GH-129404)
2025-02-03 12:41:32 +00:00
JUMP_TO_ERROR();
}
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
res = PyStackRef_FromPyObjectSteal(res_o);
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_tos_cache0 = res;
_tos_cache1 = PyStackRef_ZERO_BITS;
_tos_cache2 = PyStackRef_ZERO_BITS;
SET_CURRENT_CACHED_VALUES(1);
stack_pointer += -2 - oparg;
GH-141794: Limit size of generated machine code. (GH-142228) * Factor out bodies of the largest uops, to reduce jit code size. * Factor out common assert, also reducing jit code size. * Limit size of jitted code for a single executor to 1MB.
2025-12-03 17:43:35 +00:00
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-118095: Use broader specializations of CALL in tier 1, for better tier 2 support of calls. (GH-118322) * Add CALL_PY_GENERAL, CALL_BOUND_METHOD_GENERAL and call CALL_NON_PY_GENERAL specializations. * Remove CALL_PY_WITH_DEFAULTS specialization * Use CALL_NON_PY_GENERAL in more cases when otherwise failing to specialize
2024-05-04 12:11:11 +01:00
break;
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
case _CHECK_CALL_BOUND_METHOD_EXACT_ARGS_r00: {
CHECK_CURRENT_CACHED_VALUES(0);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-131498: Replace single-element arrays with scalars in bytecodes.c (GH-132615)
2025-04-18 07:16:28 -07:00
_PyStackRef null;
_PyStackRef callable;
GH-111485: Factor out tier 2 code generation from the rest of the interpreter code generator (GH-112968)
2023-12-12 12:12:17 +00:00
oparg = CURRENT_OPARG();
GH-131498: Replace single-element arrays with scalars in bytecodes.c (GH-132615)
2025-04-18 07:16:28 -07:00
null = stack_pointer[-1 - oparg];
callable = stack_pointer[-2 - oparg];
if (!PyStackRef_IsNull(null)) {
GH-117457: Correct pystats uop "miss" counts (GH-117477)
2024-04-04 18:49:18 -04:00
UOP_STAT_INC(uopcode, miss);
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(0);
GH-117457: Correct pystats uop "miss" counts (GH-117477)
2024-04-04 18:49:18 -04:00
JUMP_TO_JUMP_TARGET();
}
GH-131498: Replace single-element arrays with scalars in bytecodes.c (GH-132615)
2025-04-18 07:16:28 -07:00
if (Py_TYPE(PyStackRef_AsPyObjectBorrow(callable)) != &PyMethod_Type) {
GH-117457: Correct pystats uop "miss" counts (GH-117477)
2024-04-04 18:49:18 -04:00
UOP_STAT_INC(uopcode, miss);
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(0);
GH-117457: Correct pystats uop "miss" counts (GH-117477)
2024-04-04 18:49:18 -04:00
JUMP_TO_JUMP_TARGET();
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(0);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-106581: Split CALL_BOUND_METHOD_EXACT_ARGS into uops (#108462) Instead of using `GO_TO_INSTRUCTION(CALL_PY_EXACT_ARGS)` we just add the macro elements of the latter to the macro for the former. This requires lengthening the uops array in struct opcode_macro_expansion. (It also required changes to stacking.py that were merged already.)
2023-08-24 17:36:00 -07:00
break;
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
case _INIT_CALL_BOUND_METHOD_EXACT_ARGS_r00: {
CHECK_CURRENT_CACHED_VALUES(0);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-131498: Replace single-element arrays with scalars in bytecodes.c (GH-132615)
2025-04-18 07:16:28 -07:00
_PyStackRef self_or_null;
_PyStackRef callable;
GH-111485: Factor out tier 2 code generation from the rest of the interpreter code generator (GH-112968)
2023-12-12 12:12:17 +00:00
oparg = CURRENT_OPARG();
GH-131498: Replace single-element arrays with scalars in bytecodes.c (GH-132615)
2025-04-18 07:16:28 -07:00
self_or_null = stack_pointer[-1 - oparg];
callable = stack_pointer[-2 - oparg];
assert(PyStackRef_IsNull(self_or_null));
PyObject *callable_o = PyStackRef_AsPyObjectBorrow(callable);
gh-106581: Split CALL_BOUND_METHOD_EXACT_ARGS into uops (#108462) Instead of using `GO_TO_INSTRUCTION(CALL_PY_EXACT_ARGS)` we just add the macro elements of the latter to the macro for the former. This requires lengthening the uops array in struct opcode_macro_expansion. (It also required changes to stacking.py that were merged already.)
2023-08-24 17:36:00 -07:00
STAT_INC(CALL, hit);
GH-131498: Replace single-element arrays with scalars in bytecodes.c (GH-132615)
2025-04-18 07:16:28 -07:00
self_or_null = PyStackRef_FromPyObjectNew(((PyMethodObject *)callable_o)->im_self);
_PyStackRef temp = callable;
callable = PyStackRef_FromPyObjectNew(((PyMethodObject *)callable_o)->im_func);
stack_pointer[-2 - oparg] = callable;
stack_pointer[-1 - oparg] = self_or_null;
GH-128682: Make `PyStackRef_CLOSE` escaping. (GH-129404)
2025-02-03 12:41:32 +00:00
_PyFrame_SetStackPointer(frame, stack_pointer);
GH-119866: Spill the stack around escaping calls. (GH-124392) * Spill the evaluation around escaping calls in the generated interpreter and JIT. * The code generator tracks live, cached values so they can be saved to memory when needed. * Spills the stack pointer around escaping calls, so that the exact stack is visible to the cycle GC.
2024-10-07 14:56:39 +01:00
PyStackRef_CLOSE(temp);
GH-128682: Make `PyStackRef_CLOSE` escaping. (GH-129404)
2025-02-03 12:41:32 +00:00
stack_pointer = _PyFrame_GetStackPointer(frame);
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_tos_cache0 = PyStackRef_ZERO_BITS;
_tos_cache1 = PyStackRef_ZERO_BITS;
_tos_cache2 = PyStackRef_ZERO_BITS;
SET_CURRENT_CACHED_VALUES(0);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _CHECK_PEP_523_r00: {
CHECK_CURRENT_CACHED_VALUES(0);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
if (tstate->interp->eval_frame) {
UOP_STAT_INC(uopcode, miss);
SET_CURRENT_CACHED_VALUES(0);
JUMP_TO_JUMP_TARGET();
}
SET_CURRENT_CACHED_VALUES(0);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _CHECK_PEP_523_r11: {
CHECK_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef _stack_item_0 = _tos_cache0;
if (tstate->interp->eval_frame) {
UOP_STAT_INC(uopcode, miss);
GH-142718: JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. (GH-142762) JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. * Make sure that stack is in correct state at side exits with TOS cached values * Simplify choice of cached items for side exits
2025-12-15 19:01:29 +00:00
_tos_cache0 = _stack_item_0;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(1);
JUMP_TO_JUMP_TARGET();
}
_tos_cache0 = _stack_item_0;
SET_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _CHECK_PEP_523_r22: {
CHECK_CURRENT_CACHED_VALUES(2);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef _stack_item_0 = _tos_cache0;
_PyStackRef _stack_item_1 = _tos_cache1;
if (tstate->interp->eval_frame) {
UOP_STAT_INC(uopcode, miss);
GH-142718: JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. (GH-142762) JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. * Make sure that stack is in correct state at side exits with TOS cached values * Simplify choice of cached items for side exits
2025-12-15 19:01:29 +00:00
_tos_cache1 = _stack_item_1;
_tos_cache0 = _stack_item_0;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(2);
JUMP_TO_JUMP_TARGET();
}
_tos_cache1 = _stack_item_1;
_tos_cache0 = _stack_item_0;
SET_CURRENT_CACHED_VALUES(2);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-106581: Split CALL_BOUND_METHOD_EXACT_ARGS into uops (#108462) Instead of using `GO_TO_INSTRUCTION(CALL_PY_EXACT_ARGS)` we just add the macro elements of the latter to the macro for the former. This requires lengthening the uops array in struct opcode_macro_expansion. (It also required changes to stacking.py that were merged already.)
2023-08-24 17:36:00 -07:00
break;
}
gh-104584: Baby steps towards generating and executing traces (#105924) Added a new, experimental, tracing optimizer and interpreter (a.k.a. "tier 2"). This currently pessimizes, so don't use yet -- this is infrastructure so we can experiment with optimizing passes. To enable it, pass ``-Xuops`` or set ``PYTHONUOPS=1``. To get debug output, set ``PYTHONUOPSDEBUG=N`` where ``N`` is a debug level (0-4, where 0 is no debug output and 4 is excessively verbose). All of this code is likely to change dramatically before the 3.13 feature freeze. But this is a first step.
2023-06-26 19:02:57 -07:00
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
case _CHECK_PEP_523_r33: {
CHECK_CURRENT_CACHED_VALUES(3);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef _stack_item_0 = _tos_cache0;
_PyStackRef _stack_item_1 = _tos_cache1;
_PyStackRef _stack_item_2 = _tos_cache2;
GH-117457: Correct pystats uop "miss" counts (GH-117477)
2024-04-04 18:49:18 -04:00
if (tstate->interp->eval_frame) {
UOP_STAT_INC(uopcode, miss);
GH-142718: JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. (GH-142762) JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. * Make sure that stack is in correct state at side exits with TOS cached values * Simplify choice of cached items for side exits
2025-12-15 19:01:29 +00:00
_tos_cache2 = _stack_item_2;
_tos_cache1 = _stack_item_1;
_tos_cache0 = _stack_item_0;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(3);
GH-117457: Correct pystats uop "miss" counts (GH-117477)
2024-04-04 18:49:18 -04:00
JUMP_TO_JUMP_TARGET();
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_tos_cache2 = _stack_item_2;
_tos_cache1 = _stack_item_1;
_tos_cache0 = _stack_item_0;
SET_CURRENT_CACHED_VALUES(3);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-106581: Split `CALL_PY_EXACT_ARGS` into uops (#107760) * Split `CALL_PY_EXACT_ARGS` into uops This is only the first step for doing `CALL` in Tier 2. The next step involves tracing into the called code object and back. After that we'll have to do the remaining `CALL` specialization. Finally we'll have to deal with `KW_NAMES`. Note: this moves setting `frame->return_offset` directly in front of `DISPATCH_INLINED()`, to make it easier to move it into `_PUSH_FRAME`.
2023-08-16 16:26:43 -07:00
break;
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
case _CHECK_FUNCTION_EXACT_ARGS_r00: {
CHECK_CURRENT_CACHED_VALUES(0);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-131498: Replace single-element arrays with scalars in bytecodes.c (GH-132615)
2025-04-18 07:16:28 -07:00
_PyStackRef self_or_null;
_PyStackRef callable;
GH-111485: Factor out tier 2 code generation from the rest of the interpreter code generator (GH-112968)
2023-12-12 12:12:17 +00:00
oparg = CURRENT_OPARG();
GH-131498: Replace single-element arrays with scalars in bytecodes.c (GH-132615)
2025-04-18 07:16:28 -07:00
self_or_null = stack_pointer[-1 - oparg];
callable = stack_pointer[-2 - oparg];
PyObject *callable_o = PyStackRef_AsPyObjectBorrow(callable);
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
assert(PyFunction_Check(callable_o));
PyFunctionObject *func = (PyFunctionObject *)callable_o;
gh-106581: Split `CALL_PY_EXACT_ARGS` into uops (#107760) * Split `CALL_PY_EXACT_ARGS` into uops This is only the first step for doing `CALL` in Tier 2. The next step involves tracing into the called code object and back. After that we'll have to do the remaining `CALL` specialization. Finally we'll have to deal with `KW_NAMES`. Note: this moves setting `frame->return_offset` directly in front of `DISPATCH_INLINED()`, to make it easier to move it into `_PUSH_FRAME`.
2023-08-16 16:26:43 -07:00
PyCodeObject *code = (PyCodeObject *)func->func_code;
GH-131498: Replace single-element arrays with scalars in bytecodes.c (GH-132615)
2025-04-18 07:16:28 -07:00
if (code->co_argcount != oparg + (!PyStackRef_IsNull(self_or_null))) {
GH-117457: Correct pystats uop "miss" counts (GH-117477)
2024-04-04 18:49:18 -04:00
UOP_STAT_INC(uopcode, miss);
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(0);
GH-117457: Correct pystats uop "miss" counts (GH-117477)
2024-04-04 18:49:18 -04:00
JUMP_TO_JUMP_TARGET();
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(0);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-106581: Split `CALL_PY_EXACT_ARGS` into uops (#107760) * Split `CALL_PY_EXACT_ARGS` into uops This is only the first step for doing `CALL` in Tier 2. The next step involves tracing into the called code object and back. After that we'll have to do the remaining `CALL` specialization. Finally we'll have to deal with `KW_NAMES`. Note: this moves setting `frame->return_offset` directly in front of `DISPATCH_INLINED()`, to make it easier to move it into `_PUSH_FRAME`.
2023-08-16 16:26:43 -07:00
break;
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
case _CHECK_STACK_SPACE_r00: {
CHECK_CURRENT_CACHED_VALUES(0);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-131498: Replace single-element arrays with scalars in bytecodes.c (GH-132615)
2025-04-18 07:16:28 -07:00
_PyStackRef callable;
GH-111485: Factor out tier 2 code generation from the rest of the interpreter code generator (GH-112968)
2023-12-12 12:12:17 +00:00
oparg = CURRENT_OPARG();
GH-131498: Replace single-element arrays with scalars in bytecodes.c (GH-132615)
2025-04-18 07:16:28 -07:00
callable = stack_pointer[-2 - oparg];
PyObject *callable_o = PyStackRef_AsPyObjectBorrow(callable);
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
PyFunctionObject *func = (PyFunctionObject *)callable_o;
gh-106581: Split `CALL_PY_EXACT_ARGS` into uops (#107760) * Split `CALL_PY_EXACT_ARGS` into uops This is only the first step for doing `CALL` in Tier 2. The next step involves tracing into the called code object and back. After that we'll have to do the remaining `CALL` specialization. Finally we'll have to deal with `KW_NAMES`. Note: this moves setting `frame->return_offset` directly in front of `DISPATCH_INLINED()`, to make it easier to move it into `_PUSH_FRAME`.
2023-08-16 16:26:43 -07:00
PyCodeObject *code = (PyCodeObject *)func->func_code;
GH-117457: Correct pystats uop "miss" counts (GH-117477)
2024-04-04 18:49:18 -04:00
if (!_PyThreadState_HasStackSpace(tstate, code->co_framesize)) {
UOP_STAT_INC(uopcode, miss);
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(0);
JUMP_TO_JUMP_TARGET();
}
SET_CURRENT_CACHED_VALUES(0);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _CHECK_RECURSION_REMAINING_r00: {
CHECK_CURRENT_CACHED_VALUES(0);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
if (tstate->py_recursion_remaining <= 1) {
UOP_STAT_INC(uopcode, miss);
SET_CURRENT_CACHED_VALUES(0);
JUMP_TO_JUMP_TARGET();
}
SET_CURRENT_CACHED_VALUES(0);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _CHECK_RECURSION_REMAINING_r11: {
CHECK_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef _stack_item_0 = _tos_cache0;
if (tstate->py_recursion_remaining <= 1) {
UOP_STAT_INC(uopcode, miss);
GH-142718: JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. (GH-142762) JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. * Make sure that stack is in correct state at side exits with TOS cached values * Simplify choice of cached items for side exits
2025-12-15 19:01:29 +00:00
_tos_cache0 = _stack_item_0;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(1);
JUMP_TO_JUMP_TARGET();
}
_tos_cache0 = _stack_item_0;
SET_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _CHECK_RECURSION_REMAINING_r22: {
CHECK_CURRENT_CACHED_VALUES(2);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef _stack_item_0 = _tos_cache0;
_PyStackRef _stack_item_1 = _tos_cache1;
if (tstate->py_recursion_remaining <= 1) {
UOP_STAT_INC(uopcode, miss);
GH-142718: JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. (GH-142762) JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. * Make sure that stack is in correct state at side exits with TOS cached values * Simplify choice of cached items for side exits
2025-12-15 19:01:29 +00:00
_tos_cache1 = _stack_item_1;
_tos_cache0 = _stack_item_0;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(2);
GH-117457: Correct pystats uop "miss" counts (GH-117477)
2024-04-04 18:49:18 -04:00
JUMP_TO_JUMP_TARGET();
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_tos_cache1 = _stack_item_1;
_tos_cache0 = _stack_item_0;
SET_CURRENT_CACHED_VALUES(2);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-132744: Check recursion limit in CALL_PY_GENERAL (GH-132746)
2025-05-03 00:36:29 +08:00
break;
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
case _CHECK_RECURSION_REMAINING_r33: {
CHECK_CURRENT_CACHED_VALUES(3);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef _stack_item_0 = _tos_cache0;
_PyStackRef _stack_item_1 = _tos_cache1;
_PyStackRef _stack_item_2 = _tos_cache2;
GH-117457: Correct pystats uop "miss" counts (GH-117477)
2024-04-04 18:49:18 -04:00
if (tstate->py_recursion_remaining <= 1) {
UOP_STAT_INC(uopcode, miss);
GH-142718: JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. (GH-142762) JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. * Make sure that stack is in correct state at side exits with TOS cached values * Simplify choice of cached items for side exits
2025-12-15 19:01:29 +00:00
_tos_cache2 = _stack_item_2;
_tos_cache1 = _stack_item_1;
_tos_cache0 = _stack_item_0;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(3);
GH-117457: Correct pystats uop "miss" counts (GH-117477)
2024-04-04 18:49:18 -04:00
JUMP_TO_JUMP_TARGET();
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_tos_cache2 = _stack_item_2;
_tos_cache1 = _stack_item_1;
_tos_cache0 = _stack_item_0;
SET_CURRENT_CACHED_VALUES(3);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-106581: Split `CALL_PY_EXACT_ARGS` into uops (#107760) * Split `CALL_PY_EXACT_ARGS` into uops This is only the first step for doing `CALL` in Tier 2. The next step involves tracing into the called code object and back. After that we'll have to do the remaining `CALL` specialization. Finally we'll have to deal with `KW_NAMES`. Note: this moves setting `frame->return_offset` directly in front of `DISPATCH_INLINED()`, to make it easier to move it into `_PUSH_FRAME`.
2023-08-16 16:26:43 -07:00
break;
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
case _INIT_CALL_PY_EXACT_ARGS_0_r01: {
CHECK_CURRENT_CACHED_VALUES(0);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
_PyStackRef *args;
GH-131498: Replace single-element arrays with scalars in bytecodes.c (GH-132615)
2025-04-18 07:16:28 -07:00
_PyStackRef self_or_null;
_PyStackRef callable;
GH-135379: Remove types from stack items in code generator. (GH-135384) * Make casts explicit in the instruction definitions
2025-06-11 15:52:25 +01:00
_PyStackRef new_frame;
GH-115457: Support splitting and replication of micro ops. (GH-115558)
2024-02-20 10:50:59 +00:00
oparg = 0;
assert(oparg == CURRENT_OPARG());
args = &stack_pointer[-oparg];
GH-131498: Replace single-element arrays with scalars in bytecodes.c (GH-132615)
2025-04-18 07:16:28 -07:00
self_or_null = stack_pointer[-1 - oparg];
callable = stack_pointer[-2 - oparg];
int has_self = !PyStackRef_IsNull(self_or_null);
GH-115457: Support splitting and replication of micro ops. (GH-115558)
2024-02-20 10:50:59 +00:00
STAT_INC(CALL, hit);
GH-135379: Remove types from stack items in code generator. (GH-135384) * Make casts explicit in the instruction definitions
2025-06-11 15:52:25 +01:00
_PyInterpreterFrame *pushed_frame = _PyFrame_PushUnchecked(tstate, callable, oparg + has_self, frame);
_PyStackRef *first_non_self_local = pushed_frame->localsplus + has_self;
pushed_frame->localsplus[0] = self_or_null;
GH-115802: Reduce the size of _INIT_CALL_PY_EXACT_ARGS. (GH-116856)
2024-03-15 17:16:30 +00:00
for (int i = 0; i < oparg; i++) {
first_non_self_local[i] = args[i];
GH-115457: Support splitting and replication of micro ops. (GH-115558)
2024-02-20 10:50:59 +00:00
}
GH-135379: Remove types from stack items in code generator. (GH-135384) * Make casts explicit in the instruction definitions
2025-06-11 15:52:25 +01:00
new_frame = PyStackRef_Wrap(pushed_frame);
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_tos_cache0 = new_frame;
SET_CURRENT_CACHED_VALUES(1);
stack_pointer += -2 - oparg;
GH-141794: Limit size of generated machine code. (GH-142228) * Factor out bodies of the largest uops, to reduce jit code size. * Factor out common assert, also reducing jit code size. * Limit size of jitted code for a single executor to 1MB.
2025-12-03 17:43:35 +00:00
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-115457: Support splitting and replication of micro ops. (GH-115558)
2024-02-20 10:50:59 +00:00
break;
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
case _INIT_CALL_PY_EXACT_ARGS_1_r01: {
CHECK_CURRENT_CACHED_VALUES(0);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
_PyStackRef *args;
GH-131498: Replace single-element arrays with scalars in bytecodes.c (GH-132615)
2025-04-18 07:16:28 -07:00
_PyStackRef self_or_null;
_PyStackRef callable;
GH-135379: Remove types from stack items in code generator. (GH-135384) * Make casts explicit in the instruction definitions
2025-06-11 15:52:25 +01:00
_PyStackRef new_frame;
GH-115457: Support splitting and replication of micro ops. (GH-115558)
2024-02-20 10:50:59 +00:00
oparg = 1;
assert(oparg == CURRENT_OPARG());
args = &stack_pointer[-oparg];
GH-131498: Replace single-element arrays with scalars in bytecodes.c (GH-132615)
2025-04-18 07:16:28 -07:00
self_or_null = stack_pointer[-1 - oparg];
callable = stack_pointer[-2 - oparg];
int has_self = !PyStackRef_IsNull(self_or_null);
GH-115457: Support splitting and replication of micro ops. (GH-115558)
2024-02-20 10:50:59 +00:00
STAT_INC(CALL, hit);
GH-135379: Remove types from stack items in code generator. (GH-135384) * Make casts explicit in the instruction definitions
2025-06-11 15:52:25 +01:00
_PyInterpreterFrame *pushed_frame = _PyFrame_PushUnchecked(tstate, callable, oparg + has_self, frame);
_PyStackRef *first_non_self_local = pushed_frame->localsplus + has_self;
pushed_frame->localsplus[0] = self_or_null;
GH-115802: Reduce the size of _INIT_CALL_PY_EXACT_ARGS. (GH-116856)
2024-03-15 17:16:30 +00:00
for (int i = 0; i < oparg; i++) {
first_non_self_local[i] = args[i];
GH-115457: Support splitting and replication of micro ops. (GH-115558)
2024-02-20 10:50:59 +00:00
}
GH-135379: Remove types from stack items in code generator. (GH-135384) * Make casts explicit in the instruction definitions
2025-06-11 15:52:25 +01:00
new_frame = PyStackRef_Wrap(pushed_frame);
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_tos_cache0 = new_frame;
SET_CURRENT_CACHED_VALUES(1);
stack_pointer += -2 - oparg;
GH-141794: Limit size of generated machine code. (GH-142228) * Factor out bodies of the largest uops, to reduce jit code size. * Factor out common assert, also reducing jit code size. * Limit size of jitted code for a single executor to 1MB.
2025-12-03 17:43:35 +00:00
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-115457: Support splitting and replication of micro ops. (GH-115558)
2024-02-20 10:50:59 +00:00
break;
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
case _INIT_CALL_PY_EXACT_ARGS_2_r01: {
CHECK_CURRENT_CACHED_VALUES(0);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
_PyStackRef *args;
GH-131498: Replace single-element arrays with scalars in bytecodes.c (GH-132615)
2025-04-18 07:16:28 -07:00
_PyStackRef self_or_null;
_PyStackRef callable;
GH-135379: Remove types from stack items in code generator. (GH-135384) * Make casts explicit in the instruction definitions
2025-06-11 15:52:25 +01:00
_PyStackRef new_frame;
GH-115457: Support splitting and replication of micro ops. (GH-115558)
2024-02-20 10:50:59 +00:00
oparg = 2;
assert(oparg == CURRENT_OPARG());
args = &stack_pointer[-oparg];
GH-131498: Replace single-element arrays with scalars in bytecodes.c (GH-132615)
2025-04-18 07:16:28 -07:00
self_or_null = stack_pointer[-1 - oparg];
callable = stack_pointer[-2 - oparg];
int has_self = !PyStackRef_IsNull(self_or_null);
GH-115457: Support splitting and replication of micro ops. (GH-115558)
2024-02-20 10:50:59 +00:00
STAT_INC(CALL, hit);
GH-135379: Remove types from stack items in code generator. (GH-135384) * Make casts explicit in the instruction definitions
2025-06-11 15:52:25 +01:00
_PyInterpreterFrame *pushed_frame = _PyFrame_PushUnchecked(tstate, callable, oparg + has_self, frame);
_PyStackRef *first_non_self_local = pushed_frame->localsplus + has_self;
pushed_frame->localsplus[0] = self_or_null;
GH-115802: Reduce the size of _INIT_CALL_PY_EXACT_ARGS. (GH-116856)
2024-03-15 17:16:30 +00:00
for (int i = 0; i < oparg; i++) {
first_non_self_local[i] = args[i];
GH-115457: Support splitting and replication of micro ops. (GH-115558)
2024-02-20 10:50:59 +00:00
}
GH-135379: Remove types from stack items in code generator. (GH-135384) * Make casts explicit in the instruction definitions
2025-06-11 15:52:25 +01:00
new_frame = PyStackRef_Wrap(pushed_frame);
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_tos_cache0 = new_frame;
SET_CURRENT_CACHED_VALUES(1);
stack_pointer += -2 - oparg;
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _INIT_CALL_PY_EXACT_ARGS_3_r01: {
CHECK_CURRENT_CACHED_VALUES(0);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef *args;
_PyStackRef self_or_null;
_PyStackRef callable;
_PyStackRef new_frame;
oparg = 3;
assert(oparg == CURRENT_OPARG());
args = &stack_pointer[-oparg];
self_or_null = stack_pointer[-1 - oparg];
callable = stack_pointer[-2 - oparg];
int has_self = !PyStackRef_IsNull(self_or_null);
STAT_INC(CALL, hit);
_PyInterpreterFrame *pushed_frame = _PyFrame_PushUnchecked(tstate, callable, oparg + has_self, frame);
_PyStackRef *first_non_self_local = pushed_frame->localsplus + has_self;
pushed_frame->localsplus[0] = self_or_null;
for (int i = 0; i < oparg; i++) {
first_non_self_local[i] = args[i];
}
new_frame = PyStackRef_Wrap(pushed_frame);
_tos_cache0 = new_frame;
SET_CURRENT_CACHED_VALUES(1);
stack_pointer += -2 - oparg;
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _INIT_CALL_PY_EXACT_ARGS_4_r01: {
CHECK_CURRENT_CACHED_VALUES(0);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef *args;
_PyStackRef self_or_null;
_PyStackRef callable;
_PyStackRef new_frame;
oparg = 4;
assert(oparg == CURRENT_OPARG());
args = &stack_pointer[-oparg];
self_or_null = stack_pointer[-1 - oparg];
callable = stack_pointer[-2 - oparg];
int has_self = !PyStackRef_IsNull(self_or_null);
STAT_INC(CALL, hit);
_PyInterpreterFrame *pushed_frame = _PyFrame_PushUnchecked(tstate, callable, oparg + has_self, frame);
_PyStackRef *first_non_self_local = pushed_frame->localsplus + has_self;
pushed_frame->localsplus[0] = self_or_null;
for (int i = 0; i < oparg; i++) {
first_non_self_local[i] = args[i];
}
new_frame = PyStackRef_Wrap(pushed_frame);
_tos_cache0 = new_frame;
SET_CURRENT_CACHED_VALUES(1);
stack_pointer += -2 - oparg;
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _INIT_CALL_PY_EXACT_ARGS_r01: {
CHECK_CURRENT_CACHED_VALUES(0);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef *args;
_PyStackRef self_or_null;
_PyStackRef callable;
_PyStackRef new_frame;
oparg = CURRENT_OPARG();
args = &stack_pointer[-oparg];
self_or_null = stack_pointer[-1 - oparg];
callable = stack_pointer[-2 - oparg];
int has_self = !PyStackRef_IsNull(self_or_null);
STAT_INC(CALL, hit);
_PyInterpreterFrame *pushed_frame = _PyFrame_PushUnchecked(tstate, callable, oparg + has_self, frame);
_PyStackRef *first_non_self_local = pushed_frame->localsplus + has_self;
pushed_frame->localsplus[0] = self_or_null;
for (int i = 0; i < oparg; i++) {
first_non_self_local[i] = args[i];
}
new_frame = PyStackRef_Wrap(pushed_frame);
_tos_cache0 = new_frame;
SET_CURRENT_CACHED_VALUES(1);
stack_pointer += -2 - oparg;
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _PUSH_FRAME_r10: {
CHECK_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef new_frame;
_PyStackRef _stack_item_0 = _tos_cache0;
new_frame = _stack_item_0;
assert(tstate->interp->eval_frame == NULL);
_PyInterpreterFrame *temp = PyStackRef_Unwrap(new_frame);
_PyFrame_SetStackPointer(frame, stack_pointer);
assert(temp->previous == frame || temp->previous->previous == frame);
CALL_STAT_INC(inlined_py_calls);
frame = tstate->current_frame = temp;
tstate->py_recursion_remaining--;
LOAD_SP();
LOAD_IP(0);
LLTRACE_RESUME_FRAME();
SET_CURRENT_CACHED_VALUES(0);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _GUARD_NOS_NULL_r02: {
CHECK_CURRENT_CACHED_VALUES(0);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef null;
null = stack_pointer[-2];
if (!PyStackRef_IsNull(null)) {
UOP_STAT_INC(uopcode, miss);
SET_CURRENT_CACHED_VALUES(0);
JUMP_TO_JUMP_TARGET();
}
_tos_cache1 = stack_pointer[-1];
_tos_cache0 = null;
SET_CURRENT_CACHED_VALUES(2);
stack_pointer += -2;
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _GUARD_NOS_NULL_r12: {
CHECK_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef null;
_PyStackRef _stack_item_0 = _tos_cache0;
null = stack_pointer[-1];
if (!PyStackRef_IsNull(null)) {
UOP_STAT_INC(uopcode, miss);
GH-142718: JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. (GH-142762) JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. * Make sure that stack is in correct state at side exits with TOS cached values * Simplify choice of cached items for side exits
2025-12-15 19:01:29 +00:00
_tos_cache0 = _stack_item_0;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(1);
JUMP_TO_JUMP_TARGET();
}
_tos_cache1 = _stack_item_0;
_tos_cache0 = null;
SET_CURRENT_CACHED_VALUES(2);
stack_pointer += -1;
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _GUARD_NOS_NULL_r22: {
CHECK_CURRENT_CACHED_VALUES(2);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef null;
_PyStackRef _stack_item_0 = _tos_cache0;
_PyStackRef _stack_item_1 = _tos_cache1;
null = _stack_item_0;
if (!PyStackRef_IsNull(null)) {
UOP_STAT_INC(uopcode, miss);
GH-142718: JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. (GH-142762) JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. * Make sure that stack is in correct state at side exits with TOS cached values * Simplify choice of cached items for side exits
2025-12-15 19:01:29 +00:00
_tos_cache1 = _stack_item_1;
_tos_cache0 = null;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(2);
JUMP_TO_JUMP_TARGET();
}
_tos_cache1 = _stack_item_1;
_tos_cache0 = null;
SET_CURRENT_CACHED_VALUES(2);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _GUARD_NOS_NULL_r33: {
CHECK_CURRENT_CACHED_VALUES(3);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef null;
_PyStackRef _stack_item_0 = _tos_cache0;
_PyStackRef _stack_item_1 = _tos_cache1;
_PyStackRef _stack_item_2 = _tos_cache2;
null = _stack_item_1;
if (!PyStackRef_IsNull(null)) {
UOP_STAT_INC(uopcode, miss);
GH-142718: JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. (GH-142762) JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. * Make sure that stack is in correct state at side exits with TOS cached values * Simplify choice of cached items for side exits
2025-12-15 19:01:29 +00:00
_tos_cache2 = _stack_item_2;
_tos_cache1 = null;
_tos_cache0 = _stack_item_0;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(3);
JUMP_TO_JUMP_TARGET();
}
_tos_cache2 = _stack_item_2;
_tos_cache1 = null;
_tos_cache0 = _stack_item_0;
SET_CURRENT_CACHED_VALUES(3);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _GUARD_NOS_NOT_NULL_r02: {
CHECK_CURRENT_CACHED_VALUES(0);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef nos;
nos = stack_pointer[-2];
PyObject *o = PyStackRef_AsPyObjectBorrow(nos);
if (o == NULL) {
UOP_STAT_INC(uopcode, miss);
SET_CURRENT_CACHED_VALUES(0);
JUMP_TO_JUMP_TARGET();
}
_tos_cache1 = stack_pointer[-1];
_tos_cache0 = nos;
SET_CURRENT_CACHED_VALUES(2);
stack_pointer += -2;
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _GUARD_NOS_NOT_NULL_r12: {
CHECK_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef nos;
_PyStackRef _stack_item_0 = _tos_cache0;
nos = stack_pointer[-1];
PyObject *o = PyStackRef_AsPyObjectBorrow(nos);
if (o == NULL) {
UOP_STAT_INC(uopcode, miss);
GH-142718: JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. (GH-142762) JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. * Make sure that stack is in correct state at side exits with TOS cached values * Simplify choice of cached items for side exits
2025-12-15 19:01:29 +00:00
_tos_cache0 = _stack_item_0;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(1);
JUMP_TO_JUMP_TARGET();
}
_tos_cache1 = _stack_item_0;
_tos_cache0 = nos;
SET_CURRENT_CACHED_VALUES(2);
stack_pointer += -1;
GH-141794: Limit size of generated machine code. (GH-142228) * Factor out bodies of the largest uops, to reduce jit code size. * Factor out common assert, also reducing jit code size. * Limit size of jitted code for a single executor to 1MB.
2025-12-03 17:43:35 +00:00
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _GUARD_NOS_NOT_NULL_r22: {
CHECK_CURRENT_CACHED_VALUES(2);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef nos;
_PyStackRef _stack_item_0 = _tos_cache0;
_PyStackRef _stack_item_1 = _tos_cache1;
nos = _stack_item_0;
PyObject *o = PyStackRef_AsPyObjectBorrow(nos);
if (o == NULL) {
UOP_STAT_INC(uopcode, miss);
GH-142718: JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. (GH-142762) JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. * Make sure that stack is in correct state at side exits with TOS cached values * Simplify choice of cached items for side exits
2025-12-15 19:01:29 +00:00
_tos_cache1 = _stack_item_1;
_tos_cache0 = nos;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(2);
JUMP_TO_JUMP_TARGET();
}
_tos_cache1 = _stack_item_1;
_tos_cache0 = nos;
SET_CURRENT_CACHED_VALUES(2);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-115457: Support splitting and replication of micro ops. (GH-115558)
2024-02-20 10:50:59 +00:00
break;
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
case _GUARD_NOS_NOT_NULL_r33: {
CHECK_CURRENT_CACHED_VALUES(3);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef nos;
_PyStackRef _stack_item_0 = _tos_cache0;
_PyStackRef _stack_item_1 = _tos_cache1;
_PyStackRef _stack_item_2 = _tos_cache2;
nos = _stack_item_1;
PyObject *o = PyStackRef_AsPyObjectBorrow(nos);
if (o == NULL) {
UOP_STAT_INC(uopcode, miss);
GH-142718: JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. (GH-142762) JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. * Make sure that stack is in correct state at side exits with TOS cached values * Simplify choice of cached items for side exits
2025-12-15 19:01:29 +00:00
_tos_cache2 = _stack_item_2;
_tos_cache1 = nos;
_tos_cache0 = _stack_item_0;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(3);
JUMP_TO_JUMP_TARGET();
GH-115457: Support splitting and replication of micro ops. (GH-115558)
2024-02-20 10:50:59 +00:00
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_tos_cache2 = _stack_item_2;
_tos_cache1 = nos;
_tos_cache0 = _stack_item_0;
SET_CURRENT_CACHED_VALUES(3);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-115457: Support splitting and replication of micro ops. (GH-115558)
2024-02-20 10:50:59 +00:00
break;
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
case _GUARD_THIRD_NULL_r03: {
CHECK_CURRENT_CACHED_VALUES(0);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef null;
null = stack_pointer[-3];
if (!PyStackRef_IsNull(null)) {
UOP_STAT_INC(uopcode, miss);
SET_CURRENT_CACHED_VALUES(0);
JUMP_TO_JUMP_TARGET();
GH-115457: Support splitting and replication of micro ops. (GH-115558)
2024-02-20 10:50:59 +00:00
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_tos_cache2 = stack_pointer[-1];
_tos_cache1 = stack_pointer[-2];
_tos_cache0 = null;
SET_CURRENT_CACHED_VALUES(3);
stack_pointer += -3;
GH-141794: Limit size of generated machine code. (GH-142228) * Factor out bodies of the largest uops, to reduce jit code size. * Factor out common assert, also reducing jit code size. * Limit size of jitted code for a single executor to 1MB.
2025-12-03 17:43:35 +00:00
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-115457: Support splitting and replication of micro ops. (GH-115558)
2024-02-20 10:50:59 +00:00
break;
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
case _GUARD_THIRD_NULL_r13: {
CHECK_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef null;
_PyStackRef _stack_item_0 = _tos_cache0;
null = stack_pointer[-2];
if (!PyStackRef_IsNull(null)) {
UOP_STAT_INC(uopcode, miss);
GH-142718: JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. (GH-142762) JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. * Make sure that stack is in correct state at side exits with TOS cached values * Simplify choice of cached items for side exits
2025-12-15 19:01:29 +00:00
_tos_cache0 = _stack_item_0;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(1);
JUMP_TO_JUMP_TARGET();
gh-106581: Split `CALL_PY_EXACT_ARGS` into uops (#107760) * Split `CALL_PY_EXACT_ARGS` into uops This is only the first step for doing `CALL` in Tier 2. The next step involves tracing into the called code object and back. After that we'll have to do the remaining `CALL` specialization. Finally we'll have to deal with `KW_NAMES`. Note: this moves setting `frame->return_offset` directly in front of `DISPATCH_INLINED()`, to make it easier to move it into `_PUSH_FRAME`.
2023-08-16 16:26:43 -07:00
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_tos_cache2 = _stack_item_0;
_tos_cache1 = stack_pointer[-1];
_tos_cache0 = null;
SET_CURRENT_CACHED_VALUES(3);
stack_pointer += -2;
GH-141794: Limit size of generated machine code. (GH-142228) * Factor out bodies of the largest uops, to reduce jit code size. * Factor out common assert, also reducing jit code size. * Limit size of jitted code for a single executor to 1MB.
2025-12-03 17:43:35 +00:00
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-106581: Split `CALL_PY_EXACT_ARGS` into uops (#107760) * Split `CALL_PY_EXACT_ARGS` into uops This is only the first step for doing `CALL` in Tier 2. The next step involves tracing into the called code object and back. After that we'll have to do the remaining `CALL` specialization. Finally we'll have to deal with `KW_NAMES`. Note: this moves setting `frame->return_offset` directly in front of `DISPATCH_INLINED()`, to make it easier to move it into `_PUSH_FRAME`.
2023-08-16 16:26:43 -07:00
break;
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
case _GUARD_THIRD_NULL_r23: {
CHECK_CURRENT_CACHED_VALUES(2);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef null;
_PyStackRef _stack_item_0 = _tos_cache0;
_PyStackRef _stack_item_1 = _tos_cache1;
null = stack_pointer[-1];
if (!PyStackRef_IsNull(null)) {
UOP_STAT_INC(uopcode, miss);
GH-142718: JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. (GH-142762) JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. * Make sure that stack is in correct state at side exits with TOS cached values * Simplify choice of cached items for side exits
2025-12-15 19:01:29 +00:00
_tos_cache1 = _stack_item_1;
_tos_cache0 = _stack_item_0;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(2);
JUMP_TO_JUMP_TARGET();
}
_tos_cache2 = _stack_item_1;
_tos_cache1 = _stack_item_0;
_tos_cache0 = null;
SET_CURRENT_CACHED_VALUES(3);
GH-111485: Factor out tier 2 code generation from the rest of the interpreter code generator (GH-112968)
2023-12-12 12:12:17 +00:00
stack_pointer += -1;
GH-141794: Limit size of generated machine code. (GH-142228) * Factor out bodies of the largest uops, to reduce jit code size. * Factor out common assert, also reducing jit code size. * Limit size of jitted code for a single executor to 1MB.
2025-12-03 17:43:35 +00:00
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-106581: Split `CALL_PY_EXACT_ARGS` into uops (#107760) * Split `CALL_PY_EXACT_ARGS` into uops This is only the first step for doing `CALL` in Tier 2. The next step involves tracing into the called code object and back. After that we'll have to do the remaining `CALL` specialization. Finally we'll have to deal with `KW_NAMES`. Note: this moves setting `frame->return_offset` directly in front of `DISPATCH_INLINED()`, to make it easier to move it into `_PUSH_FRAME`.
2023-08-16 16:26:43 -07:00
break;
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
case _GUARD_THIRD_NULL_r33: {
CHECK_CURRENT_CACHED_VALUES(3);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-131798: JIT: Split CALL_TYPE_1 into several uops (GH-132419)
2025-04-22 18:30:38 +02:00
_PyStackRef null;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef _stack_item_0 = _tos_cache0;
_PyStackRef _stack_item_1 = _tos_cache1;
_PyStackRef _stack_item_2 = _tos_cache2;
null = _stack_item_0;
GH-131798: JIT: Split CALL_TYPE_1 into several uops (GH-132419)
2025-04-22 18:30:38 +02:00
if (!PyStackRef_IsNull(null)) {
UOP_STAT_INC(uopcode, miss);
GH-142718: JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. (GH-142762) JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. * Make sure that stack is in correct state at side exits with TOS cached values * Simplify choice of cached items for side exits
2025-12-15 19:01:29 +00:00
_tos_cache2 = _stack_item_2;
_tos_cache1 = _stack_item_1;
_tos_cache0 = null;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(3);
GH-131798: JIT: Split CALL_TYPE_1 into several uops (GH-132419)
2025-04-22 18:30:38 +02:00
JUMP_TO_JUMP_TARGET();
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_tos_cache2 = _stack_item_2;
_tos_cache1 = _stack_item_1;
_tos_cache0 = null;
SET_CURRENT_CACHED_VALUES(3);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-131798: JIT: Split CALL_TYPE_1 into several uops (GH-132419)
2025-04-22 18:30:38 +02:00
break;
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
case _GUARD_CALLABLE_TYPE_1_r03: {
CHECK_CURRENT_CACHED_VALUES(0);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef callable;
callable = stack_pointer[-3];
PyObject *callable_o = PyStackRef_AsPyObjectBorrow(callable);
if (callable_o != (PyObject *)&PyType_Type) {
UOP_STAT_INC(uopcode, miss);
SET_CURRENT_CACHED_VALUES(0);
JUMP_TO_JUMP_TARGET();
}
_tos_cache2 = stack_pointer[-1];
_tos_cache1 = stack_pointer[-2];
_tos_cache0 = callable;
SET_CURRENT_CACHED_VALUES(3);
stack_pointer += -3;
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _GUARD_CALLABLE_TYPE_1_r13: {
CHECK_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef callable;
_PyStackRef _stack_item_0 = _tos_cache0;
callable = stack_pointer[-2];
PyObject *callable_o = PyStackRef_AsPyObjectBorrow(callable);
if (callable_o != (PyObject *)&PyType_Type) {
GH-131798: Split CALL_LIST_APPEND into several uops (GH-134240)
2025-05-19 15:48:55 -04:00
UOP_STAT_INC(uopcode, miss);
GH-142718: JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. (GH-142762) JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. * Make sure that stack is in correct state at side exits with TOS cached values * Simplify choice of cached items for side exits
2025-12-15 19:01:29 +00:00
_tos_cache0 = _stack_item_0;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(1);
GH-131798: Split CALL_LIST_APPEND into several uops (GH-134240)
2025-05-19 15:48:55 -04:00
JUMP_TO_JUMP_TARGET();
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_tos_cache2 = _stack_item_0;
_tos_cache1 = stack_pointer[-1];
_tos_cache0 = callable;
SET_CURRENT_CACHED_VALUES(3);
stack_pointer += -2;
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-131798: Split CALL_LIST_APPEND into several uops (GH-134240)
2025-05-19 15:48:55 -04:00
break;
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
case _GUARD_CALLABLE_TYPE_1_r23: {
CHECK_CURRENT_CACHED_VALUES(2);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef callable;
_PyStackRef _stack_item_0 = _tos_cache0;
_PyStackRef _stack_item_1 = _tos_cache1;
callable = stack_pointer[-1];
PyObject *callable_o = PyStackRef_AsPyObjectBorrow(callable);
if (callable_o != (PyObject *)&PyType_Type) {
GH-131798: Split up and optimize CALL_ISINSTANCE (GH-133339)
2025-05-08 23:26:30 +02:00
UOP_STAT_INC(uopcode, miss);
GH-142718: JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. (GH-142762) JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. * Make sure that stack is in correct state at side exits with TOS cached values * Simplify choice of cached items for side exits
2025-12-15 19:01:29 +00:00
_tos_cache1 = _stack_item_1;
_tos_cache0 = _stack_item_0;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(2);
GH-131798: Split up and optimize CALL_ISINSTANCE (GH-133339)
2025-05-08 23:26:30 +02:00
JUMP_TO_JUMP_TARGET();
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_tos_cache2 = _stack_item_1;
_tos_cache1 = _stack_item_0;
_tos_cache0 = callable;
SET_CURRENT_CACHED_VALUES(3);
stack_pointer += -1;
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-131798: Split up and optimize CALL_ISINSTANCE (GH-133339)
2025-05-08 23:26:30 +02:00
break;
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
case _GUARD_CALLABLE_TYPE_1_r33: {
CHECK_CURRENT_CACHED_VALUES(3);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-131798: JIT: Split CALL_TYPE_1 into several uops (GH-132419)
2025-04-22 18:30:38 +02:00
_PyStackRef callable;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef _stack_item_0 = _tos_cache0;
_PyStackRef _stack_item_1 = _tos_cache1;
_PyStackRef _stack_item_2 = _tos_cache2;
callable = _stack_item_0;
GH-131798: JIT: Split CALL_TYPE_1 into several uops (GH-132419)
2025-04-22 18:30:38 +02:00
PyObject *callable_o = PyStackRef_AsPyObjectBorrow(callable);
if (callable_o != (PyObject *)&PyType_Type) {
UOP_STAT_INC(uopcode, miss);
GH-142718: JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. (GH-142762) JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. * Make sure that stack is in correct state at side exits with TOS cached values * Simplify choice of cached items for side exits
2025-12-15 19:01:29 +00:00
_tos_cache2 = _stack_item_2;
_tos_cache1 = _stack_item_1;
_tos_cache0 = callable;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(3);
GH-131798: JIT: Split CALL_TYPE_1 into several uops (GH-132419)
2025-04-22 18:30:38 +02:00
JUMP_TO_JUMP_TARGET();
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_tos_cache2 = _stack_item_2;
_tos_cache1 = _stack_item_1;
_tos_cache0 = callable;
SET_CURRENT_CACHED_VALUES(3);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-131798: JIT: Split CALL_TYPE_1 into several uops (GH-132419)
2025-04-22 18:30:38 +02:00
break;
}
gh-134584: Eliminate redundant refcounting from `_CALL_TYPE_1` (GH-135818)
2025-12-23 18:01:10 +01:00
case _CALL_TYPE_1_r02: {
CHECK_CURRENT_CACHED_VALUES(0);
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
_PyStackRef arg;
_PyStackRef res;
_PyStackRef a;
oparg = CURRENT_OPARG();
arg = stack_pointer[-1];
PyObject *arg_o = PyStackRef_AsPyObjectBorrow(arg);
assert(oparg == 1);
STAT_INC(CALL, hit);
a = arg;
res = PyStackRef_FromPyObjectNew(Py_TYPE(arg_o));
_tos_cache1 = a;
_tos_cache0 = res;
SET_CURRENT_CACHED_VALUES(2);
stack_pointer += -3;
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
break;
}
case _CALL_TYPE_1_r12: {
CHECK_CURRENT_CACHED_VALUES(1);
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
_PyStackRef arg;
_PyStackRef res;
_PyStackRef a;
_PyStackRef _stack_item_0 = _tos_cache0;
oparg = CURRENT_OPARG();
arg = _stack_item_0;
PyObject *arg_o = PyStackRef_AsPyObjectBorrow(arg);
assert(oparg == 1);
STAT_INC(CALL, hit);
a = arg;
res = PyStackRef_FromPyObjectNew(Py_TYPE(arg_o));
_tos_cache1 = a;
_tos_cache0 = res;
SET_CURRENT_CACHED_VALUES(2);
stack_pointer += -2;
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
break;
}
case _CALL_TYPE_1_r22: {
CHECK_CURRENT_CACHED_VALUES(2);
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
_PyStackRef arg;
_PyStackRef res;
_PyStackRef a;
_PyStackRef _stack_item_0 = _tos_cache0;
_PyStackRef _stack_item_1 = _tos_cache1;
oparg = CURRENT_OPARG();
arg = _stack_item_1;
PyObject *arg_o = PyStackRef_AsPyObjectBorrow(arg);
assert(oparg == 1);
STAT_INC(CALL, hit);
a = arg;
res = PyStackRef_FromPyObjectNew(Py_TYPE(arg_o));
_tos_cache1 = a;
_tos_cache0 = res;
SET_CURRENT_CACHED_VALUES(2);
stack_pointer += -1;
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
break;
}
case _CALL_TYPE_1_r32: {
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
CHECK_CURRENT_CACHED_VALUES(3);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
_PyStackRef arg;
_PyStackRef res;
gh-134584: Eliminate redundant refcounting from `_CALL_TYPE_1` (GH-135818)
2025-12-23 18:01:10 +01:00
_PyStackRef a;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef _stack_item_0 = _tos_cache0;
_PyStackRef _stack_item_1 = _tos_cache1;
_PyStackRef _stack_item_2 = _tos_cache2;
GH-111485: Factor out tier 2 code generation from the rest of the interpreter code generator (GH-112968)
2023-12-12 12:12:17 +00:00
oparg = CURRENT_OPARG();
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
arg = _stack_item_2;
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
PyObject *arg_o = PyStackRef_AsPyObjectBorrow(arg);
gh-106581: Add 10 new opcodes by allowing `assert(kwnames == NULL)` (#106707) By turning `assert(kwnames == NULL)` into a macro that is not in the "forbidden" list, many instructions that formerly were skipped because they contained such an assert (but no other mention of `kwnames`) are now supported in Tier 2. This covers 10 instructions in total (all specializations of `CALL` that invoke some C code): - `CALL_NO_KW_TYPE_1` - `CALL_NO_KW_STR_1` - `CALL_NO_KW_TUPLE_1` - `CALL_NO_KW_BUILTIN_O` - `CALL_NO_KW_BUILTIN_FAST` - `CALL_NO_KW_LEN` - `CALL_NO_KW_ISINSTANCE` - `CALL_NO_KW_METHOD_DESCRIPTOR_O` - `CALL_NO_KW_METHOD_DESCRIPTOR_NOARGS` - `CALL_NO_KW_METHOD_DESCRIPTOR_FAST`
2023-07-17 11:02:58 -07:00
assert(oparg == 1);
STAT_INC(CALL, hit);
gh-134584: Eliminate redundant refcounting from `_CALL_TYPE_1` (GH-135818)
2025-12-23 18:01:10 +01:00
a = arg;
GH-127705: Use `_PyStackRef`s in the default build. (GH-127875)
2025-03-10 14:06:56 +00:00
res = PyStackRef_FromPyObjectNew(Py_TYPE(arg_o));
gh-134584: Eliminate redundant refcounting from `_CALL_TYPE_1` (GH-135818)
2025-12-23 18:01:10 +01:00
_tos_cache1 = a;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_tos_cache0 = res;
gh-134584: Eliminate redundant refcounting from `_CALL_TYPE_1` (GH-135818)
2025-12-23 18:01:10 +01:00
SET_CURRENT_CACHED_VALUES(2);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-106581: Add 10 new opcodes by allowing `assert(kwnames == NULL)` (#106707) By turning `assert(kwnames == NULL)` into a macro that is not in the "forbidden" list, many instructions that formerly were skipped because they contained such an assert (but no other mention of `kwnames`) are now supported in Tier 2. This covers 10 instructions in total (all specializations of `CALL` that invoke some C code): - `CALL_NO_KW_TYPE_1` - `CALL_NO_KW_STR_1` - `CALL_NO_KW_TUPLE_1` - `CALL_NO_KW_BUILTIN_O` - `CALL_NO_KW_BUILTIN_FAST` - `CALL_NO_KW_LEN` - `CALL_NO_KW_ISINSTANCE` - `CALL_NO_KW_METHOD_DESCRIPTOR_O` - `CALL_NO_KW_METHOD_DESCRIPTOR_NOARGS` - `CALL_NO_KW_METHOD_DESCRIPTOR_FAST`
2023-07-17 11:02:58 -07:00
break;
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
case _GUARD_CALLABLE_STR_1_r03: {
CHECK_CURRENT_CACHED_VALUES(0);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-131798: Split up and optimize CALL_STR_1 in the JIT (GH-132849)
2025-04-24 21:54:46 +02:00
_PyStackRef callable;
callable = stack_pointer[-3];
PyObject *callable_o = PyStackRef_AsPyObjectBorrow(callable);
if (callable_o != (PyObject *)&PyUnicode_Type) {
UOP_STAT_INC(uopcode, miss);
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(0);
JUMP_TO_JUMP_TARGET();
}
_tos_cache2 = stack_pointer[-1];
_tos_cache1 = stack_pointer[-2];
_tos_cache0 = callable;
SET_CURRENT_CACHED_VALUES(3);
stack_pointer += -3;
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _GUARD_CALLABLE_STR_1_r13: {
CHECK_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef callable;
_PyStackRef _stack_item_0 = _tos_cache0;
callable = stack_pointer[-2];
PyObject *callable_o = PyStackRef_AsPyObjectBorrow(callable);
if (callable_o != (PyObject *)&PyUnicode_Type) {
UOP_STAT_INC(uopcode, miss);
GH-142718: JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. (GH-142762) JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. * Make sure that stack is in correct state at side exits with TOS cached values * Simplify choice of cached items for side exits
2025-12-15 19:01:29 +00:00
_tos_cache0 = _stack_item_0;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(1);
JUMP_TO_JUMP_TARGET();
}
_tos_cache2 = _stack_item_0;
_tos_cache1 = stack_pointer[-1];
_tos_cache0 = callable;
SET_CURRENT_CACHED_VALUES(3);
stack_pointer += -2;
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _GUARD_CALLABLE_STR_1_r23: {
CHECK_CURRENT_CACHED_VALUES(2);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef callable;
_PyStackRef _stack_item_0 = _tos_cache0;
_PyStackRef _stack_item_1 = _tos_cache1;
callable = stack_pointer[-1];
PyObject *callable_o = PyStackRef_AsPyObjectBorrow(callable);
if (callable_o != (PyObject *)&PyUnicode_Type) {
UOP_STAT_INC(uopcode, miss);
GH-142718: JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. (GH-142762) JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. * Make sure that stack is in correct state at side exits with TOS cached values * Simplify choice of cached items for side exits
2025-12-15 19:01:29 +00:00
_tos_cache1 = _stack_item_1;
_tos_cache0 = _stack_item_0;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(2);
JUMP_TO_JUMP_TARGET();
}
_tos_cache2 = _stack_item_1;
_tos_cache1 = _stack_item_0;
_tos_cache0 = callable;
SET_CURRENT_CACHED_VALUES(3);
stack_pointer += -1;
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _GUARD_CALLABLE_STR_1_r33: {
CHECK_CURRENT_CACHED_VALUES(3);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef callable;
_PyStackRef _stack_item_0 = _tos_cache0;
_PyStackRef _stack_item_1 = _tos_cache1;
_PyStackRef _stack_item_2 = _tos_cache2;
callable = _stack_item_0;
PyObject *callable_o = PyStackRef_AsPyObjectBorrow(callable);
if (callable_o != (PyObject *)&PyUnicode_Type) {
UOP_STAT_INC(uopcode, miss);
GH-142718: JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. (GH-142762) JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. * Make sure that stack is in correct state at side exits with TOS cached values * Simplify choice of cached items for side exits
2025-12-15 19:01:29 +00:00
_tos_cache2 = _stack_item_2;
_tos_cache1 = _stack_item_1;
_tos_cache0 = callable;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(3);
GH-131798: Split up and optimize CALL_STR_1 in the JIT (GH-132849)
2025-04-24 21:54:46 +02:00
JUMP_TO_JUMP_TARGET();
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_tos_cache2 = _stack_item_2;
_tos_cache1 = _stack_item_1;
_tos_cache0 = callable;
SET_CURRENT_CACHED_VALUES(3);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-131798: Split up and optimize CALL_STR_1 in the JIT (GH-132849)
2025-04-24 21:54:46 +02:00
break;
}
gh-134584: Eliminate redundant refcounting from `_CALL_STR_1` (GH-136070) Signed-off-by: Manjusaka <me@manjusaka.me>
2025-12-14 17:33:05 +08:00
case _CALL_STR_1_r32: {
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
CHECK_CURRENT_CACHED_VALUES(3);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
_PyStackRef arg;
_PyStackRef res;
gh-134584: Eliminate redundant refcounting from `_CALL_STR_1` (GH-136070) Signed-off-by: Manjusaka <me@manjusaka.me>
2025-12-14 17:33:05 +08:00
_PyStackRef a;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef _stack_item_0 = _tos_cache0;
_PyStackRef _stack_item_1 = _tos_cache1;
_PyStackRef _stack_item_2 = _tos_cache2;
GH-111485: Factor out tier 2 code generation from the rest of the interpreter code generator (GH-112968)
2023-12-12 12:12:17 +00:00
oparg = CURRENT_OPARG();
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
arg = _stack_item_2;
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
PyObject *arg_o = PyStackRef_AsPyObjectBorrow(arg);
gh-106581: Add 10 new opcodes by allowing `assert(kwnames == NULL)` (#106707) By turning `assert(kwnames == NULL)` into a macro that is not in the "forbidden" list, many instructions that formerly were skipped because they contained such an assert (but no other mention of `kwnames`) are now supported in Tier 2. This covers 10 instructions in total (all specializations of `CALL` that invoke some C code): - `CALL_NO_KW_TYPE_1` - `CALL_NO_KW_STR_1` - `CALL_NO_KW_TUPLE_1` - `CALL_NO_KW_BUILTIN_O` - `CALL_NO_KW_BUILTIN_FAST` - `CALL_NO_KW_LEN` - `CALL_NO_KW_ISINSTANCE` - `CALL_NO_KW_METHOD_DESCRIPTOR_O` - `CALL_NO_KW_METHOD_DESCRIPTOR_NOARGS` - `CALL_NO_KW_METHOD_DESCRIPTOR_FAST`
2023-07-17 11:02:58 -07:00
assert(oparg == 1);
STAT_INC(CALL, hit);
gh-134584: Eliminate redundant refcounting from `_CALL_STR_1` (GH-136070) Signed-off-by: Manjusaka <me@manjusaka.me>
2025-12-14 17:33:05 +08:00
stack_pointer[0] = _stack_item_0;
stack_pointer[1] = _stack_item_1;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
stack_pointer[2] = arg;
stack_pointer += 3;
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
GH-119866: Spill the stack around escaping calls. (GH-124392) * Spill the evaluation around escaping calls in the generated interpreter and JIT. * The code generator tracks live, cached values so they can be saved to memory when needed. * Spills the stack pointer around escaping calls, so that the exact stack is visible to the cycle GC.
2024-10-07 14:56:39 +01:00
_PyFrame_SetStackPointer(frame, stack_pointer);
gh-121459: Fix a couple of uses of `PyStackRef_FromPyObjectSteal` (#125711) * Fix usage of PyStackRef_FromPyObjectSteal in CALL_TUPLE_1 This was missed in gh-124894 * Fix usage of PyStackRef_FromPyObjectSteal in _CALL_STR_1 This was missed in gh-124894 * Regenerate code
2024-10-21 11:08:13 -07:00
PyObject *res_o = PyObject_Str(arg_o);
GH-119866: Spill the stack around escaping calls. (GH-124392) * Spill the evaluation around escaping calls in the generated interpreter and JIT. * The code generator tracks live, cached values so they can be saved to memory when needed. * Spills the stack pointer around escaping calls, so that the exact stack is visible to the cycle GC.
2024-10-07 14:56:39 +01:00
stack_pointer = _PyFrame_GetStackPointer(frame);
GH-128682: Make `PyStackRef_CLOSE` escaping. (GH-129404)
2025-02-03 12:41:32 +00:00
if (res_o == NULL) {
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(0);
GH-128682: Make `PyStackRef_CLOSE` escaping. (GH-129404)
2025-02-03 12:41:32 +00:00
JUMP_TO_ERROR();
}
gh-134584: Eliminate redundant refcounting from `_CALL_STR_1` (GH-136070) Signed-off-by: Manjusaka <me@manjusaka.me>
2025-12-14 17:33:05 +08:00
a = arg;
gh-121459: Fix a couple of uses of `PyStackRef_FromPyObjectSteal` (#125711) * Fix usage of PyStackRef_FromPyObjectSteal in CALL_TUPLE_1 This was missed in gh-124894 * Fix usage of PyStackRef_FromPyObjectSteal in _CALL_STR_1 This was missed in gh-124894 * Regenerate code
2024-10-21 11:08:13 -07:00
res = PyStackRef_FromPyObjectSteal(res_o);
gh-134584: Eliminate redundant refcounting from `_CALL_STR_1` (GH-136070) Signed-off-by: Manjusaka <me@manjusaka.me>
2025-12-14 17:33:05 +08:00
_tos_cache1 = a;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_tos_cache0 = res;
_tos_cache2 = PyStackRef_ZERO_BITS;
gh-134584: Eliminate redundant refcounting from `_CALL_STR_1` (GH-136070) Signed-off-by: Manjusaka <me@manjusaka.me>
2025-12-14 17:33:05 +08:00
SET_CURRENT_CACHED_VALUES(2);
stack_pointer += -3;
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-106581: Add 10 new opcodes by allowing `assert(kwnames == NULL)` (#106707) By turning `assert(kwnames == NULL)` into a macro that is not in the "forbidden" list, many instructions that formerly were skipped because they contained such an assert (but no other mention of `kwnames`) are now supported in Tier 2. This covers 10 instructions in total (all specializations of `CALL` that invoke some C code): - `CALL_NO_KW_TYPE_1` - `CALL_NO_KW_STR_1` - `CALL_NO_KW_TUPLE_1` - `CALL_NO_KW_BUILTIN_O` - `CALL_NO_KW_BUILTIN_FAST` - `CALL_NO_KW_LEN` - `CALL_NO_KW_ISINSTANCE` - `CALL_NO_KW_METHOD_DESCRIPTOR_O` - `CALL_NO_KW_METHOD_DESCRIPTOR_NOARGS` - `CALL_NO_KW_METHOD_DESCRIPTOR_FAST`
2023-07-17 11:02:58 -07:00
break;
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
case _GUARD_CALLABLE_TUPLE_1_r03: {
CHECK_CURRENT_CACHED_VALUES(0);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-131798: Split up and optimize CALL_TUPLE_1 in the JIT (GH-132851)
2025-04-25 00:55:03 +02:00
_PyStackRef callable;
callable = stack_pointer[-3];
PyObject *callable_o = PyStackRef_AsPyObjectBorrow(callable);
if (callable_o != (PyObject *)&PyTuple_Type) {
UOP_STAT_INC(uopcode, miss);
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(0);
JUMP_TO_JUMP_TARGET();
}
_tos_cache2 = stack_pointer[-1];
_tos_cache1 = stack_pointer[-2];
_tos_cache0 = callable;
SET_CURRENT_CACHED_VALUES(3);
stack_pointer += -3;
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _GUARD_CALLABLE_TUPLE_1_r13: {
CHECK_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef callable;
_PyStackRef _stack_item_0 = _tos_cache0;
callable = stack_pointer[-2];
PyObject *callable_o = PyStackRef_AsPyObjectBorrow(callable);
if (callable_o != (PyObject *)&PyTuple_Type) {
UOP_STAT_INC(uopcode, miss);
GH-142718: JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. (GH-142762) JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. * Make sure that stack is in correct state at side exits with TOS cached values * Simplify choice of cached items for side exits
2025-12-15 19:01:29 +00:00
_tos_cache0 = _stack_item_0;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(1);
JUMP_TO_JUMP_TARGET();
}
_tos_cache2 = _stack_item_0;
_tos_cache1 = stack_pointer[-1];
_tos_cache0 = callable;
SET_CURRENT_CACHED_VALUES(3);
stack_pointer += -2;
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _GUARD_CALLABLE_TUPLE_1_r23: {
CHECK_CURRENT_CACHED_VALUES(2);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef callable;
_PyStackRef _stack_item_0 = _tos_cache0;
_PyStackRef _stack_item_1 = _tos_cache1;
callable = stack_pointer[-1];
PyObject *callable_o = PyStackRef_AsPyObjectBorrow(callable);
if (callable_o != (PyObject *)&PyTuple_Type) {
UOP_STAT_INC(uopcode, miss);
GH-142718: JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. (GH-142762) JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. * Make sure that stack is in correct state at side exits with TOS cached values * Simplify choice of cached items for side exits
2025-12-15 19:01:29 +00:00
_tos_cache1 = _stack_item_1;
_tos_cache0 = _stack_item_0;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(2);
JUMP_TO_JUMP_TARGET();
}
_tos_cache2 = _stack_item_1;
_tos_cache1 = _stack_item_0;
_tos_cache0 = callable;
SET_CURRENT_CACHED_VALUES(3);
stack_pointer += -1;
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _GUARD_CALLABLE_TUPLE_1_r33: {
CHECK_CURRENT_CACHED_VALUES(3);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef callable;
_PyStackRef _stack_item_0 = _tos_cache0;
_PyStackRef _stack_item_1 = _tos_cache1;
_PyStackRef _stack_item_2 = _tos_cache2;
callable = _stack_item_0;
PyObject *callable_o = PyStackRef_AsPyObjectBorrow(callable);
if (callable_o != (PyObject *)&PyTuple_Type) {
UOP_STAT_INC(uopcode, miss);
GH-142718: JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. (GH-142762) JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. * Make sure that stack is in correct state at side exits with TOS cached values * Simplify choice of cached items for side exits
2025-12-15 19:01:29 +00:00
_tos_cache2 = _stack_item_2;
_tos_cache1 = _stack_item_1;
_tos_cache0 = callable;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(3);
GH-131798: Split up and optimize CALL_TUPLE_1 in the JIT (GH-132851)
2025-04-25 00:55:03 +02:00
JUMP_TO_JUMP_TARGET();
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_tos_cache2 = _stack_item_2;
_tos_cache1 = _stack_item_1;
_tos_cache0 = callable;
SET_CURRENT_CACHED_VALUES(3);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-131798: Split up and optimize CALL_TUPLE_1 in the JIT (GH-132851)
2025-04-25 00:55:03 +02:00
break;
}
gh-134584: Cleanups for GH-135860 (GH-142604)
2025-12-13 22:38:10 +08:00
case _CALL_TUPLE_1_r32: {
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
CHECK_CURRENT_CACHED_VALUES(3);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
_PyStackRef arg;
_PyStackRef res;
gh-134584: Cleanups for GH-135860 (GH-142604)
2025-12-13 22:38:10 +08:00
_PyStackRef a;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef _stack_item_0 = _tos_cache0;
_PyStackRef _stack_item_1 = _tos_cache1;
_PyStackRef _stack_item_2 = _tos_cache2;
GH-111485: Factor out tier 2 code generation from the rest of the interpreter code generator (GH-112968)
2023-12-12 12:12:17 +00:00
oparg = CURRENT_OPARG();
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
arg = _stack_item_2;
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
PyObject *arg_o = PyStackRef_AsPyObjectBorrow(arg);
gh-106581: Add 10 new opcodes by allowing `assert(kwnames == NULL)` (#106707) By turning `assert(kwnames == NULL)` into a macro that is not in the "forbidden" list, many instructions that formerly were skipped because they contained such an assert (but no other mention of `kwnames`) are now supported in Tier 2. This covers 10 instructions in total (all specializations of `CALL` that invoke some C code): - `CALL_NO_KW_TYPE_1` - `CALL_NO_KW_STR_1` - `CALL_NO_KW_TUPLE_1` - `CALL_NO_KW_BUILTIN_O` - `CALL_NO_KW_BUILTIN_FAST` - `CALL_NO_KW_LEN` - `CALL_NO_KW_ISINSTANCE` - `CALL_NO_KW_METHOD_DESCRIPTOR_O` - `CALL_NO_KW_METHOD_DESCRIPTOR_NOARGS` - `CALL_NO_KW_METHOD_DESCRIPTOR_FAST`
2023-07-17 11:02:58 -07:00
assert(oparg == 1);
STAT_INC(CALL, hit);
gh-134584: Eliminate redundant refcounting from `_CALL_TUPLE_1` (GH-135860)
2025-12-11 16:31:28 +02:00
stack_pointer[0] = _stack_item_0;
stack_pointer[1] = _stack_item_1;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
stack_pointer[2] = arg;
stack_pointer += 3;
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
GH-119866: Spill the stack around escaping calls. (GH-124392) * Spill the evaluation around escaping calls in the generated interpreter and JIT. * The code generator tracks live, cached values so they can be saved to memory when needed. * Spills the stack pointer around escaping calls, so that the exact stack is visible to the cycle GC.
2024-10-07 14:56:39 +01:00
_PyFrame_SetStackPointer(frame, stack_pointer);
gh-121459: Fix a couple of uses of `PyStackRef_FromPyObjectSteal` (#125711) * Fix usage of PyStackRef_FromPyObjectSteal in CALL_TUPLE_1 This was missed in gh-124894 * Fix usage of PyStackRef_FromPyObjectSteal in _CALL_STR_1 This was missed in gh-124894 * Regenerate code
2024-10-21 11:08:13 -07:00
PyObject *res_o = PySequence_Tuple(arg_o);
GH-119866: Spill the stack around escaping calls. (GH-124392) * Spill the evaluation around escaping calls in the generated interpreter and JIT. * The code generator tracks live, cached values so they can be saved to memory when needed. * Spills the stack pointer around escaping calls, so that the exact stack is visible to the cycle GC.
2024-10-07 14:56:39 +01:00
stack_pointer = _PyFrame_GetStackPointer(frame);
GH-128682: Make `PyStackRef_CLOSE` escaping. (GH-129404)
2025-02-03 12:41:32 +00:00
if (res_o == NULL) {
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(0);
GH-128682: Make `PyStackRef_CLOSE` escaping. (GH-129404)
2025-02-03 12:41:32 +00:00
JUMP_TO_ERROR();
}
gh-134584: Cleanups for GH-135860 (GH-142604)
2025-12-13 22:38:10 +08:00
a = arg;
gh-121459: Fix a couple of uses of `PyStackRef_FromPyObjectSteal` (#125711) * Fix usage of PyStackRef_FromPyObjectSteal in CALL_TUPLE_1 This was missed in gh-124894 * Fix usage of PyStackRef_FromPyObjectSteal in _CALL_STR_1 This was missed in gh-124894 * Regenerate code
2024-10-21 11:08:13 -07:00
res = PyStackRef_FromPyObjectSteal(res_o);
gh-134584: Cleanups for GH-135860 (GH-142604)
2025-12-13 22:38:10 +08:00
_tos_cache1 = a;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_tos_cache0 = res;
_tos_cache2 = PyStackRef_ZERO_BITS;
gh-134584: Cleanups for GH-135860 (GH-142604)
2025-12-13 22:38:10 +08:00
SET_CURRENT_CACHED_VALUES(2);
stack_pointer += -3;
gh-134584: Eliminate redundant refcounting from `_CALL_TUPLE_1` (GH-135860)
2025-12-11 16:31:28 +02:00
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-106581: Add 10 new opcodes by allowing `assert(kwnames == NULL)` (#106707) By turning `assert(kwnames == NULL)` into a macro that is not in the "forbidden" list, many instructions that formerly were skipped because they contained such an assert (but no other mention of `kwnames`) are now supported in Tier 2. This covers 10 instructions in total (all specializations of `CALL` that invoke some C code): - `CALL_NO_KW_TYPE_1` - `CALL_NO_KW_STR_1` - `CALL_NO_KW_TUPLE_1` - `CALL_NO_KW_BUILTIN_O` - `CALL_NO_KW_BUILTIN_FAST` - `CALL_NO_KW_LEN` - `CALL_NO_KW_ISINSTANCE` - `CALL_NO_KW_METHOD_DESCRIPTOR_O` - `CALL_NO_KW_METHOD_DESCRIPTOR_NOARGS` - `CALL_NO_KW_METHOD_DESCRIPTOR_FAST`
2023-07-17 11:02:58 -07:00
break;
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
case _CHECK_AND_ALLOCATE_OBJECT_r00: {
CHECK_CURRENT_CACHED_VALUES(0);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-131498: Replace single-element arrays with scalars in bytecodes.c (GH-132615)
2025-04-18 07:16:28 -07:00
_PyStackRef self_or_null;
_PyStackRef callable;
GH-118093: Make `CALL_ALLOC_AND_ENTER_INIT` suitable for tier 2. (GH-123140) * Convert CALL_ALLOC_AND_ENTER_INIT to micro-ops such that tier 2 supports it * Allow inexact arguments for CALL_ALLOC_AND_ENTER_INIT.
2024-08-20 16:52:58 +01:00
oparg = CURRENT_OPARG();
GH-131498: Replace single-element arrays with scalars in bytecodes.c (GH-132615)
2025-04-18 07:16:28 -07:00
self_or_null = stack_pointer[-1 - oparg];
callable = stack_pointer[-2 - oparg];
GH-140683: JIT: Improve machine code for loading smaller constants on AArch64. (GH-142511) * Use movz and movk instructions for loading 16 and 32 bit operands and oparg. * Loading of 64 bit operands is unchanged.
2025-12-11 12:33:39 +00:00
uint32_t type_version = (uint32_t)CURRENT_OPERAND0_32();
GH-131498: Replace single-element arrays with scalars in bytecodes.c (GH-132615)
2025-04-18 07:16:28 -07:00
PyObject *callable_o = PyStackRef_AsPyObjectBorrow(callable);
if (!PyStackRef_IsNull(self_or_null)) {
GH-118093: Make `CALL_ALLOC_AND_ENTER_INIT` suitable for tier 2. (GH-123140) * Convert CALL_ALLOC_AND_ENTER_INIT to micro-ops such that tier 2 supports it * Allow inexact arguments for CALL_ALLOC_AND_ENTER_INIT.
2024-08-20 16:52:58 +01:00
UOP_STAT_INC(uopcode, miss);
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(0);
GH-118093: Make `CALL_ALLOC_AND_ENTER_INIT` suitable for tier 2. (GH-123140) * Convert CALL_ALLOC_AND_ENTER_INIT to micro-ops such that tier 2 supports it * Allow inexact arguments for CALL_ALLOC_AND_ENTER_INIT.
2024-08-20 16:52:58 +01:00
JUMP_TO_JUMP_TARGET();
}
if (!PyType_Check(callable_o)) {
UOP_STAT_INC(uopcode, miss);
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(0);
GH-118093: Make `CALL_ALLOC_AND_ENTER_INIT` suitable for tier 2. (GH-123140) * Convert CALL_ALLOC_AND_ENTER_INIT to micro-ops such that tier 2 supports it * Allow inexact arguments for CALL_ALLOC_AND_ENTER_INIT.
2024-08-20 16:52:58 +01:00
JUMP_TO_JUMP_TARGET();
}
PyTypeObject *tp = (PyTypeObject *)callable_o;
gh-115999: Enable specialization of `CALL` instructions in free-threaded builds (#127123) The CALL family of instructions were mostly thread-safe already and only required a small number of changes, which are documented below. A few changes were needed to make CALL_ALLOC_AND_ENTER_INIT thread-safe: Added _PyType_LookupRefAndVersion, which returns the type version corresponding to the returned ref. Added _PyType_CacheInitForSpecialization, which takes an init method and the corresponding type version and only populates the specialization cache if the current type version matches the supplied version. This prevents potentially caching a stale value in free-threaded builds if we race with an update to __init__. Only cache __init__ functions that are deferred in free-threaded builds. This ensures that the reference to __init__ that is stored in the specialization cache is valid if the type version guard in _CHECK_AND_ALLOCATE_OBJECT passes. Fix a bug in _CREATE_INIT_FRAME where the frame is pushed to the stack on failure. A few other miscellaneous changes were also needed: Use {LOCK,UNLOCK}_OBJECT in LIST_APPEND. This ensures that the list's per-object lock is held while we are appending to it. Add missing co_tlbc for _Py_InitCleanup. Stop/start the world around setting the eval frame hook. This allows us to read interp->eval_frame non-atomically and preserves the behavior of _CHECK_PEP_523 documented below.
2024-12-03 11:20:20 -08:00
if (FT_ATOMIC_LOAD_UINT32_RELAXED(tp->tp_version_tag) != type_version) {
GH-118093: Make `CALL_ALLOC_AND_ENTER_INIT` suitable for tier 2. (GH-123140) * Convert CALL_ALLOC_AND_ENTER_INIT to micro-ops such that tier 2 supports it * Allow inexact arguments for CALL_ALLOC_AND_ENTER_INIT.
2024-08-20 16:52:58 +01:00
UOP_STAT_INC(uopcode, miss);
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(0);
GH-118093: Make `CALL_ALLOC_AND_ENTER_INIT` suitable for tier 2. (GH-123140) * Convert CALL_ALLOC_AND_ENTER_INIT to micro-ops such that tier 2 supports it * Allow inexact arguments for CALL_ALLOC_AND_ENTER_INIT.
2024-08-20 16:52:58 +01:00
JUMP_TO_JUMP_TARGET();
}
gh-128262: Allow specialization of calls to classes with __slots__ (GH-128263)
2024-12-31 12:24:17 +08:00
assert(tp->tp_new == PyBaseObject_Type.tp_new);
assert(tp->tp_flags & Py_TPFLAGS_HEAPTYPE);
assert(tp->tp_alloc == PyType_GenericAlloc);
GH-118093: Make `CALL_ALLOC_AND_ENTER_INIT` suitable for tier 2. (GH-123140) * Convert CALL_ALLOC_AND_ENTER_INIT to micro-ops such that tier 2 supports it * Allow inexact arguments for CALL_ALLOC_AND_ENTER_INIT.
2024-08-20 16:52:58 +01:00
PyHeapTypeObject *cls = (PyHeapTypeObject *)callable_o;
gh-115999: Enable specialization of `CALL` instructions in free-threaded builds (#127123) The CALL family of instructions were mostly thread-safe already and only required a small number of changes, which are documented below. A few changes were needed to make CALL_ALLOC_AND_ENTER_INIT thread-safe: Added _PyType_LookupRefAndVersion, which returns the type version corresponding to the returned ref. Added _PyType_CacheInitForSpecialization, which takes an init method and the corresponding type version and only populates the specialization cache if the current type version matches the supplied version. This prevents potentially caching a stale value in free-threaded builds if we race with an update to __init__. Only cache __init__ functions that are deferred in free-threaded builds. This ensures that the reference to __init__ that is stored in the specialization cache is valid if the type version guard in _CHECK_AND_ALLOCATE_OBJECT passes. Fix a bug in _CREATE_INIT_FRAME where the frame is pushed to the stack on failure. A few other miscellaneous changes were also needed: Use {LOCK,UNLOCK}_OBJECT in LIST_APPEND. This ensures that the list's per-object lock is held while we are appending to it. Add missing co_tlbc for _Py_InitCleanup. Stop/start the world around setting the eval frame hook. This allows us to read interp->eval_frame non-atomically and preserves the behavior of _CHECK_PEP_523 documented below.
2024-12-03 11:20:20 -08:00
PyFunctionObject *init_func = (PyFunctionObject *)FT_ATOMIC_LOAD_PTR_ACQUIRE(cls->_spec_cache.init);
GH-118093: Make `CALL_ALLOC_AND_ENTER_INIT` suitable for tier 2. (GH-123140) * Convert CALL_ALLOC_AND_ENTER_INIT to micro-ops such that tier 2 supports it * Allow inexact arguments for CALL_ALLOC_AND_ENTER_INIT.
2024-08-20 16:52:58 +01:00
PyCodeObject *code = (PyCodeObject *)init_func->func_code;
if (!_PyThreadState_HasStackSpace(tstate, code->co_framesize + _Py_InitCleanup.co_framesize)) {
UOP_STAT_INC(uopcode, miss);
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(0);
GH-118093: Make `CALL_ALLOC_AND_ENTER_INIT` suitable for tier 2. (GH-123140) * Convert CALL_ALLOC_AND_ENTER_INIT to micro-ops such that tier 2 supports it * Allow inexact arguments for CALL_ALLOC_AND_ENTER_INIT.
2024-08-20 16:52:58 +01:00
JUMP_TO_JUMP_TARGET();
}
STAT_INC(CALL, hit);
gh-128262: Allow specialization of calls to classes with __slots__ (GH-128263)
2024-12-31 12:24:17 +08:00
_PyFrame_SetStackPointer(frame, stack_pointer);
PyObject *self_o = PyType_GenericAlloc(tp, 0);
stack_pointer = _PyFrame_GetStackPointer(frame);
GH-119866: Spill the stack around escaping calls. (GH-124392) * Spill the evaluation around escaping calls in the generated interpreter and JIT. * The code generator tracks live, cached values so they can be saved to memory when needed. * Spills the stack pointer around escaping calls, so that the exact stack is visible to the cycle GC.
2024-10-07 14:56:39 +01:00
if (self_o == NULL) {
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(0);
GH-118093: Make `CALL_ALLOC_AND_ENTER_INIT` suitable for tier 2. (GH-123140) * Convert CALL_ALLOC_AND_ENTER_INIT to micro-ops such that tier 2 supports it * Allow inexact arguments for CALL_ALLOC_AND_ENTER_INIT.
2024-08-20 16:52:58 +01:00
JUMP_TO_ERROR();
}
GH-131498: Replace single-element arrays with scalars in bytecodes.c (GH-132615)
2025-04-18 07:16:28 -07:00
self_or_null = PyStackRef_FromPyObjectSteal(self_o);
_PyStackRef temp = callable;
callable = PyStackRef_FromPyObjectNew(init_func);
stack_pointer[-2 - oparg] = callable;
stack_pointer[-1 - oparg] = self_or_null;
GH-128682: Make `PyStackRef_CLOSE` escaping. (GH-129404)
2025-02-03 12:41:32 +00:00
_PyFrame_SetStackPointer(frame, stack_pointer);
GH-119866: Spill the stack around escaping calls. (GH-124392) * Spill the evaluation around escaping calls in the generated interpreter and JIT. * The code generator tracks live, cached values so they can be saved to memory when needed. * Spills the stack pointer around escaping calls, so that the exact stack is visible to the cycle GC.
2024-10-07 14:56:39 +01:00
PyStackRef_CLOSE(temp);
GH-128682: Make `PyStackRef_CLOSE` escaping. (GH-129404)
2025-02-03 12:41:32 +00:00
stack_pointer = _PyFrame_GetStackPointer(frame);
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_tos_cache0 = PyStackRef_ZERO_BITS;
_tos_cache1 = PyStackRef_ZERO_BITS;
_tos_cache2 = PyStackRef_ZERO_BITS;
SET_CURRENT_CACHED_VALUES(0);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-118093: Make `CALL_ALLOC_AND_ENTER_INIT` suitable for tier 2. (GH-123140) * Convert CALL_ALLOC_AND_ENTER_INIT to micro-ops such that tier 2 supports it * Allow inexact arguments for CALL_ALLOC_AND_ENTER_INIT.
2024-08-20 16:52:58 +01:00
break;
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
case _CREATE_INIT_FRAME_r01: {
CHECK_CURRENT_CACHED_VALUES(0);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-118093: Make `CALL_ALLOC_AND_ENTER_INIT` suitable for tier 2. (GH-123140) * Convert CALL_ALLOC_AND_ENTER_INIT to micro-ops such that tier 2 supports it * Allow inexact arguments for CALL_ALLOC_AND_ENTER_INIT.
2024-08-20 16:52:58 +01:00
_PyStackRef *args;
GH-131498: Replace single-element arrays with scalars in bytecodes.c (GH-132615)
2025-04-18 07:16:28 -07:00
_PyStackRef self;
_PyStackRef init;
GH-135379: Remove types from stack items in code generator. (GH-135384) * Make casts explicit in the instruction definitions
2025-06-11 15:52:25 +01:00
_PyStackRef init_frame;
GH-118093: Make `CALL_ALLOC_AND_ENTER_INIT` suitable for tier 2. (GH-123140) * Convert CALL_ALLOC_AND_ENTER_INIT to micro-ops such that tier 2 supports it * Allow inexact arguments for CALL_ALLOC_AND_ENTER_INIT.
2024-08-20 16:52:58 +01:00
oparg = CURRENT_OPARG();
args = &stack_pointer[-oparg];
GH-131498: Replace single-element arrays with scalars in bytecodes.c (GH-132615)
2025-04-18 07:16:28 -07:00
self = stack_pointer[-1 - oparg];
init = stack_pointer[-2 - oparg];
GH-119866: Spill the stack around escaping calls. (GH-124392) * Spill the evaluation around escaping calls in the generated interpreter and JIT. * The code generator tracks live, cached values so they can be saved to memory when needed. * Spills the stack pointer around escaping calls, so that the exact stack is visible to the cycle GC.
2024-10-07 14:56:39 +01:00
_PyFrame_SetStackPointer(frame, stack_pointer);
GH-118093: Make `CALL_ALLOC_AND_ENTER_INIT` suitable for tier 2. (GH-123140) * Convert CALL_ALLOC_AND_ENTER_INIT to micro-ops such that tier 2 supports it * Allow inexact arguments for CALL_ALLOC_AND_ENTER_INIT.
2024-08-20 16:52:58 +01:00
_PyInterpreterFrame *shim = _PyFrame_PushTrampolineUnchecked(
tstate, (PyCodeObject *)&_Py_InitCleanup, 1, frame);
GH-129709: Clean up tier two (GH-129710)
2025-02-07 09:52:49 -08:00
stack_pointer = _PyFrame_GetStackPointer(frame);
gh-115999: Implement thread-local bytecode and enable specialization for `BINARY_OP` (#123926) Each thread specializes a thread-local copy of the bytecode, created on the first RESUME, in free-threaded builds. All copies of the bytecode for a code object are stored in the co_tlbc array on the code object. Threads reserve a globally unique index identifying its copy of the bytecode in all co_tlbc arrays at thread creation and release the index at thread destruction. The first entry in every co_tlbc array always points to the "main" copy of the bytecode that is stored at the end of the code object. This ensures that no bytecode is copied for programs that do not use threads. Thread-local bytecode can be disabled at runtime by providing either -X tlbc=0 or PYTHON_TLBC=0. Disabling thread-local bytecode also disables specialization. Concurrent modifications to the bytecode made by the specializing interpreter and instrumentation use atomics, with specialization taking care not to overwrite an instruction that was instrumented concurrently.
2024-11-04 11:13:32 -08:00
assert(_PyFrame_GetBytecode(shim)[0].op.code == EXIT_INIT_CHECK);
gh-115999: Enable specialization of `CALL` instructions in free-threaded builds (#127123) The CALL family of instructions were mostly thread-safe already and only required a small number of changes, which are documented below. A few changes were needed to make CALL_ALLOC_AND_ENTER_INIT thread-safe: Added _PyType_LookupRefAndVersion, which returns the type version corresponding to the returned ref. Added _PyType_CacheInitForSpecialization, which takes an init method and the corresponding type version and only populates the specialization cache if the current type version matches the supplied version. This prevents potentially caching a stale value in free-threaded builds if we race with an update to __init__. Only cache __init__ functions that are deferred in free-threaded builds. This ensures that the reference to __init__ that is stored in the specialization cache is valid if the type version guard in _CHECK_AND_ALLOCATE_OBJECT passes. Fix a bug in _CREATE_INIT_FRAME where the frame is pushed to the stack on failure. A few other miscellaneous changes were also needed: Use {LOCK,UNLOCK}_OBJECT in LIST_APPEND. This ensures that the list's per-object lock is held while we are appending to it. Add missing co_tlbc for _Py_InitCleanup. Stop/start the world around setting the eval frame hook. This allows us to read interp->eval_frame non-atomically and preserves the behavior of _CHECK_PEP_523 documented below.
2024-12-03 11:20:20 -08:00
assert(_PyFrame_GetBytecode(shim)[1].op.code == RETURN_VALUE);
GH-131498: Replace single-element arrays with scalars in bytecodes.c (GH-132615)
2025-04-18 07:16:28 -07:00
shim->localsplus[0] = PyStackRef_DUP(self);
GH-119866: Spill the stack around escaping calls. (GH-124392) * Spill the evaluation around escaping calls in the generated interpreter and JIT. * The code generator tracks live, cached values so they can be saved to memory when needed. * Spills the stack pointer around escaping calls, so that the exact stack is visible to the cycle GC.
2024-10-07 14:56:39 +01:00
_PyFrame_SetStackPointer(frame, stack_pointer);
gh-115999: Enable specialization of `CALL` instructions in free-threaded builds (#127123) The CALL family of instructions were mostly thread-safe already and only required a small number of changes, which are documented below. A few changes were needed to make CALL_ALLOC_AND_ENTER_INIT thread-safe: Added _PyType_LookupRefAndVersion, which returns the type version corresponding to the returned ref. Added _PyType_CacheInitForSpecialization, which takes an init method and the corresponding type version and only populates the specialization cache if the current type version matches the supplied version. This prevents potentially caching a stale value in free-threaded builds if we race with an update to __init__. Only cache __init__ functions that are deferred in free-threaded builds. This ensures that the reference to __init__ that is stored in the specialization cache is valid if the type version guard in _CHECK_AND_ALLOCATE_OBJECT passes. Fix a bug in _CREATE_INIT_FRAME where the frame is pushed to the stack on failure. A few other miscellaneous changes were also needed: Use {LOCK,UNLOCK}_OBJECT in LIST_APPEND. This ensures that the list's per-object lock is held while we are appending to it. Add missing co_tlbc for _Py_InitCleanup. Stop/start the world around setting the eval frame hook. This allows us to read interp->eval_frame non-atomically and preserves the behavior of _CHECK_PEP_523 documented below.
2024-12-03 11:20:20 -08:00
_PyInterpreterFrame *temp = _PyEvalFramePushAndInit(
GH-131498: Replace single-element arrays with scalars in bytecodes.c (GH-132615)
2025-04-18 07:16:28 -07:00
tstate, init, NULL, args-1, oparg+1, NULL, shim);
GH-119866: Spill the stack around escaping calls. (GH-124392) * Spill the evaluation around escaping calls in the generated interpreter and JIT. * The code generator tracks live, cached values so they can be saved to memory when needed. * Spills the stack pointer around escaping calls, so that the exact stack is visible to the cycle GC.
2024-10-07 14:56:39 +01:00
stack_pointer = _PyFrame_GetStackPointer(frame);
gh-115999: Enable specialization of `CALL` instructions in free-threaded builds (#127123) The CALL family of instructions were mostly thread-safe already and only required a small number of changes, which are documented below. A few changes were needed to make CALL_ALLOC_AND_ENTER_INIT thread-safe: Added _PyType_LookupRefAndVersion, which returns the type version corresponding to the returned ref. Added _PyType_CacheInitForSpecialization, which takes an init method and the corresponding type version and only populates the specialization cache if the current type version matches the supplied version. This prevents potentially caching a stale value in free-threaded builds if we race with an update to __init__. Only cache __init__ functions that are deferred in free-threaded builds. This ensures that the reference to __init__ that is stored in the specialization cache is valid if the type version guard in _CHECK_AND_ALLOCATE_OBJECT passes. Fix a bug in _CREATE_INIT_FRAME where the frame is pushed to the stack on failure. A few other miscellaneous changes were also needed: Use {LOCK,UNLOCK}_OBJECT in LIST_APPEND. This ensures that the list's per-object lock is held while we are appending to it. Add missing co_tlbc for _Py_InitCleanup. Stop/start the world around setting the eval frame hook. This allows us to read interp->eval_frame non-atomically and preserves the behavior of _CHECK_PEP_523 documented below.
2024-12-03 11:20:20 -08:00
stack_pointer += -2 - oparg;
GH-141794: Limit size of generated machine code. (GH-142228) * Factor out bodies of the largest uops, to reduce jit code size. * Factor out common assert, also reducing jit code size. * Limit size of jitted code for a single executor to 1MB.
2025-12-03 17:43:35 +00:00
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
gh-115999: Enable specialization of `CALL` instructions in free-threaded builds (#127123) The CALL family of instructions were mostly thread-safe already and only required a small number of changes, which are documented below. A few changes were needed to make CALL_ALLOC_AND_ENTER_INIT thread-safe: Added _PyType_LookupRefAndVersion, which returns the type version corresponding to the returned ref. Added _PyType_CacheInitForSpecialization, which takes an init method and the corresponding type version and only populates the specialization cache if the current type version matches the supplied version. This prevents potentially caching a stale value in free-threaded builds if we race with an update to __init__. Only cache __init__ functions that are deferred in free-threaded builds. This ensures that the reference to __init__ that is stored in the specialization cache is valid if the type version guard in _CHECK_AND_ALLOCATE_OBJECT passes. Fix a bug in _CREATE_INIT_FRAME where the frame is pushed to the stack on failure. A few other miscellaneous changes were also needed: Use {LOCK,UNLOCK}_OBJECT in LIST_APPEND. This ensures that the list's per-object lock is held while we are appending to it. Add missing co_tlbc for _Py_InitCleanup. Stop/start the world around setting the eval frame hook. This allows us to read interp->eval_frame non-atomically and preserves the behavior of _CHECK_PEP_523 documented below.
2024-12-03 11:20:20 -08:00
if (temp == NULL) {
GH-128682: Account for escapes in `DECREF_INPUTS` (GH-129953) * Handle escapes in DECREF_INPUTS * Mark a few more functions as escaping * Replace DECREF_INPUTS with PyStackRef_CLOSE where possible
2025-02-12 17:44:59 +00:00
_PyFrame_SetStackPointer(frame, stack_pointer);
GH-123185: Check for `NULL` after calling `_PyEvalFramePushAndInit` (GH-123194)
2024-08-21 12:44:56 +01:00
_PyEval_FrameClearAndPop(tstate, shim);
GH-128682: Account for escapes in `DECREF_INPUTS` (GH-129953) * Handle escapes in DECREF_INPUTS * Mark a few more functions as escaping * Replace DECREF_INPUTS with PyStackRef_CLOSE where possible
2025-02-12 17:44:59 +00:00
stack_pointer = _PyFrame_GetStackPointer(frame);
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(0);
GH-123185: Check for `NULL` after calling `_PyEvalFramePushAndInit` (GH-123194)
2024-08-21 12:44:56 +01:00
JUMP_TO_ERROR();
}
frame->return_offset = 1 + INLINE_CACHE_ENTRIES_CALL;
GH-118093: Make `CALL_ALLOC_AND_ENTER_INIT` suitable for tier 2. (GH-123140) * Convert CALL_ALLOC_AND_ENTER_INIT to micro-ops such that tier 2 supports it * Allow inexact arguments for CALL_ALLOC_AND_ENTER_INIT.
2024-08-20 16:52:58 +01:00
tstate->py_recursion_remaining--;
GH-135379: Remove types from stack items in code generator. (GH-135384) * Make casts explicit in the instruction definitions
2025-06-11 15:52:25 +01:00
init_frame = PyStackRef_Wrap(temp);
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_tos_cache0 = init_frame;
_tos_cache1 = PyStackRef_ZERO_BITS;
_tos_cache2 = PyStackRef_ZERO_BITS;
SET_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-118093: Make `CALL_ALLOC_AND_ENTER_INIT` suitable for tier 2. (GH-123140) * Convert CALL_ALLOC_AND_ENTER_INIT to micro-ops such that tier 2 supports it * Allow inexact arguments for CALL_ALLOC_AND_ENTER_INIT.
2024-08-20 16:52:58 +01:00
break;
}
GH-111485: Factor out tier 2 code generation from the rest of the interpreter code generator (GH-112968)
2023-12-12 12:12:17 +00:00
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
case _EXIT_INIT_CHECK_r10: {
CHECK_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
_PyStackRef should_be_none;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef _stack_item_0 = _tos_cache0;
should_be_none = _stack_item_0;
gh-127022: Simplify `PyStackRef_FromPyObjectSteal` (#127024) This gets rid of the immortal check in `PyStackRef_FromPyObjectSteal()`. Overall, this improves performance about 2% in the free threading build. This also renames `PyStackRef_Is()` to `PyStackRef_IsExactly()` because the macro requires that the tag bits of the arguments match, which is only true in certain special cases.
2024-11-22 17:55:33 +00:00
if (!PyStackRef_IsNone(should_be_none)) {
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
stack_pointer[0] = should_be_none;
stack_pointer += 1;
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
GH-119866: Spill the stack around escaping calls. (GH-124392) * Spill the evaluation around escaping calls in the generated interpreter and JIT. * The code generator tracks live, cached values so they can be saved to memory when needed. * Spills the stack pointer around escaping calls, so that the exact stack is visible to the cycle GC.
2024-10-07 14:56:39 +01:00
_PyFrame_SetStackPointer(frame, stack_pointer);
gh-104584: Baby steps towards generating and executing traces (#105924) Added a new, experimental, tracing optimizer and interpreter (a.k.a. "tier 2"). This currently pessimizes, so don't use yet -- this is infrastructure so we can experiment with optimizing passes. To enable it, pass ``-Xuops`` or set ``PYTHONUOPS=1``. To get debug output, set ``PYTHONUOPSDEBUG=N`` where ``N`` is a debug level (0-4, where 0 is no debug output and 4 is excessively verbose). All of this code is likely to change dramatically before the 3.13 feature freeze. But this is a first step.
2023-06-26 19:02:57 -07:00
PyErr_Format(PyExc_TypeError,
GH-111485: Factor out tier 2 code generation from the rest of the interpreter code generator (GH-112968)
2023-12-12 12:12:17 +00:00
"__init__() should return None, not '%.200s'",
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
Py_TYPE(PyStackRef_AsPyObjectBorrow(should_be_none))->tp_name);
GH-119866: Spill the stack around escaping calls. (GH-124392) * Spill the evaluation around escaping calls in the generated interpreter and JIT. * The code generator tracks live, cached values so they can be saved to memory when needed. * Spills the stack pointer around escaping calls, so that the exact stack is visible to the cycle GC.
2024-10-07 14:56:39 +01:00
stack_pointer = _PyFrame_GetStackPointer(frame);
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(0);
GH-116422: Tier2 hot/cold splitting (GH-116813) Splits the "cold" path, deopts and exits, from the "hot" path, reducing the size of most jitted instructions, at the cost of slower exits.
2024-03-26 09:35:11 +00:00
JUMP_TO_ERROR();
gh-104584: Baby steps towards generating and executing traces (#105924) Added a new, experimental, tracing optimizer and interpreter (a.k.a. "tier 2"). This currently pessimizes, so don't use yet -- this is infrastructure so we can experiment with optimizing passes. To enable it, pass ``-Xuops`` or set ``PYTHONUOPS=1``. To get debug output, set ``PYTHONUOPSDEBUG=N`` where ``N`` is a debug level (0-4, where 0 is no debug output and 4 is excessively verbose). All of this code is likely to change dramatically before the 3.13 feature freeze. But this is a first step.
2023-06-26 19:02:57 -07:00
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_tos_cache0 = PyStackRef_ZERO_BITS;
_tos_cache1 = PyStackRef_ZERO_BITS;
_tos_cache2 = PyStackRef_ZERO_BITS;
SET_CURRENT_CACHED_VALUES(0);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-104584: Baby steps towards generating and executing traces (#105924) Added a new, experimental, tracing optimizer and interpreter (a.k.a. "tier 2"). This currently pessimizes, so don't use yet -- this is infrastructure so we can experiment with optimizing passes. To enable it, pass ``-Xuops`` or set ``PYTHONUOPS=1``. To get debug output, set ``PYTHONUOPSDEBUG=N`` where ``N`` is a debug level (0-4, where 0 is no debug output and 4 is excessively verbose). All of this code is likely to change dramatically before the 3.13 feature freeze. But this is a first step.
2023-06-26 19:02:57 -07:00
break;
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
case _CALL_BUILTIN_CLASS_r01: {
CHECK_CURRENT_CACHED_VALUES(0);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
_PyStackRef *args;
GH-131498: Replace single-element arrays with scalars in bytecodes.c (GH-132615)
2025-04-18 07:16:28 -07:00
_PyStackRef self_or_null;
_PyStackRef callable;
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
_PyStackRef res;
GH-111485: Factor out tier 2 code generation from the rest of the interpreter code generator (GH-112968)
2023-12-12 12:12:17 +00:00
oparg = CURRENT_OPARG();
args = &stack_pointer[-oparg];
GH-131498: Replace single-element arrays with scalars in bytecodes.c (GH-132615)
2025-04-18 07:16:28 -07:00
self_or_null = stack_pointer[-1 - oparg];
callable = stack_pointer[-2 - oparg];
PyObject *callable_o = PyStackRef_AsPyObjectBorrow(callable);
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
if (!PyType_Check(callable_o)) {
GH-117457: Correct pystats uop "miss" counts (GH-117477)
2024-04-04 18:49:18 -04:00
UOP_STAT_INC(uopcode, miss);
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(0);
GH-117457: Correct pystats uop "miss" counts (GH-117477)
2024-04-04 18:49:18 -04:00
JUMP_TO_JUMP_TARGET();
}
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
PyTypeObject *tp = (PyTypeObject *)callable_o;
GH-128682: Stronger checking of `PyStackRef_CLOSE` and `DEAD`. (GH-128683)
2025-01-13 12:37:48 +00:00
int total_args = oparg;
_PyStackRef *arguments = args;
GH-131498: Replace single-element arrays with scalars in bytecodes.c (GH-132615)
2025-04-18 07:16:28 -07:00
if (!PyStackRef_IsNull(self_or_null)) {
GH-128682: Stronger checking of `PyStackRef_CLOSE` and `DEAD`. (GH-128683)
2025-01-13 12:37:48 +00:00
arguments--;
total_args++;
}
GH-117457: Correct pystats uop "miss" counts (GH-117477)
2024-04-04 18:49:18 -04:00
if (tp->tp_vectorcall == NULL) {
UOP_STAT_INC(uopcode, miss);
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(0);
GH-117457: Correct pystats uop "miss" counts (GH-117477)
2024-04-04 18:49:18 -04:00
JUMP_TO_JUMP_TARGET();
}
GH-105848: Replace KW_NAMES + CALL with LOAD_CONST + CALL_KW (GH-109300)
2023-09-13 10:25:45 -07:00
STAT_INC(CALL, hit);
GH-128682: Account for escapes in `DECREF_INPUTS` (GH-129953) * Handle escapes in DECREF_INPUTS * Mark a few more functions as escaping * Replace DECREF_INPUTS with PyStackRef_CLOSE where possible
2025-02-12 17:44:59 +00:00
_PyFrame_SetStackPointer(frame, stack_pointer);
GH-141794: Limit size of generated machine code. (GH-142228) * Factor out bodies of the largest uops, to reduce jit code size. * Factor out common assert, also reducing jit code size. * Limit size of jitted code for a single executor to 1MB.
2025-12-03 17:43:35 +00:00
PyObject *res_o = _Py_CallBuiltinClass_StackRefSteal(
callable,
arguments,
total_args);
GH-128682: Account for escapes in `DECREF_INPUTS` (GH-129953) * Handle escapes in DECREF_INPUTS * Mark a few more functions as escaping * Replace DECREF_INPUTS with PyStackRef_CLOSE where possible
2025-02-12 17:44:59 +00:00
stack_pointer = _PyFrame_GetStackPointer(frame);
GH-128682: Make `PyStackRef_CLOSE` escaping. (GH-129404)
2025-02-03 12:41:32 +00:00
if (res_o == NULL) {
GH-141794: Limit size of generated machine code. (GH-142228) * Factor out bodies of the largest uops, to reduce jit code size. * Factor out common assert, also reducing jit code size. * Limit size of jitted code for a single executor to 1MB.
2025-12-03 17:43:35 +00:00
stack_pointer += -2 - oparg;
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(0);
GH-128682: Make `PyStackRef_CLOSE` escaping. (GH-129404)
2025-02-03 12:41:32 +00:00
JUMP_TO_ERROR();
}
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
res = PyStackRef_FromPyObjectSteal(res_o);
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_tos_cache0 = res;
_tos_cache1 = PyStackRef_ZERO_BITS;
_tos_cache2 = PyStackRef_ZERO_BITS;
SET_CURRENT_CACHED_VALUES(1);
stack_pointer += -2 - oparg;
GH-141794: Limit size of generated machine code. (GH-142228) * Factor out bodies of the largest uops, to reduce jit code size. * Factor out common assert, also reducing jit code size. * Limit size of jitted code for a single executor to 1MB.
2025-12-03 17:43:35 +00:00
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-105848: Replace KW_NAMES + CALL with LOAD_CONST + CALL_KW (GH-109300)
2023-09-13 10:25:45 -07:00
break;
}
gh-134584: Eliminate redundant refcounting from `_CALL_BUILTION_O` (GH-142695) Co-authored-by: Ken Jin <kenjin4096@gmail.com>
2025-12-14 18:28:02 +08:00
case _CALL_BUILTIN_O_r03: {
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
CHECK_CURRENT_CACHED_VALUES(0);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
_PyStackRef *args;
GH-131498: Replace single-element arrays with scalars in bytecodes.c (GH-132615)
2025-04-18 07:16:28 -07:00
_PyStackRef self_or_null;
_PyStackRef callable;
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
_PyStackRef res;
gh-134584: Eliminate redundant refcounting from `_CALL_BUILTION_O` (GH-142695) Co-authored-by: Ken Jin <kenjin4096@gmail.com>
2025-12-14 18:28:02 +08:00
_PyStackRef a;
_PyStackRef c;
GH-111485: Factor out tier 2 code generation from the rest of the interpreter code generator (GH-112968)
2023-12-12 12:12:17 +00:00
oparg = CURRENT_OPARG();
args = &stack_pointer[-oparg];
GH-131498: Replace single-element arrays with scalars in bytecodes.c (GH-132615)
2025-04-18 07:16:28 -07:00
self_or_null = stack_pointer[-1 - oparg];
callable = stack_pointer[-2 - oparg];
PyObject *callable_o = PyStackRef_AsPyObjectBorrow(callable);
gh-106581: Add 10 new opcodes by allowing `assert(kwnames == NULL)` (#106707) By turning `assert(kwnames == NULL)` into a macro that is not in the "forbidden" list, many instructions that formerly were skipped because they contained such an assert (but no other mention of `kwnames`) are now supported in Tier 2. This covers 10 instructions in total (all specializations of `CALL` that invoke some C code): - `CALL_NO_KW_TYPE_1` - `CALL_NO_KW_STR_1` - `CALL_NO_KW_TUPLE_1` - `CALL_NO_KW_BUILTIN_O` - `CALL_NO_KW_BUILTIN_FAST` - `CALL_NO_KW_LEN` - `CALL_NO_KW_ISINSTANCE` - `CALL_NO_KW_METHOD_DESCRIPTOR_O` - `CALL_NO_KW_METHOD_DESCRIPTOR_NOARGS` - `CALL_NO_KW_METHOD_DESCRIPTOR_FAST`
2023-07-17 11:02:58 -07:00
int total_args = oparg;
GH-131498: Replace single-element arrays with scalars in bytecodes.c (GH-132615)
2025-04-18 07:16:28 -07:00
if (!PyStackRef_IsNull(self_or_null)) {
gh-106581: Add 10 new opcodes by allowing `assert(kwnames == NULL)` (#106707) By turning `assert(kwnames == NULL)` into a macro that is not in the "forbidden" list, many instructions that formerly were skipped because they contained such an assert (but no other mention of `kwnames`) are now supported in Tier 2. This covers 10 instructions in total (all specializations of `CALL` that invoke some C code): - `CALL_NO_KW_TYPE_1` - `CALL_NO_KW_STR_1` - `CALL_NO_KW_TUPLE_1` - `CALL_NO_KW_BUILTIN_O` - `CALL_NO_KW_BUILTIN_FAST` - `CALL_NO_KW_LEN` - `CALL_NO_KW_ISINSTANCE` - `CALL_NO_KW_METHOD_DESCRIPTOR_O` - `CALL_NO_KW_METHOD_DESCRIPTOR_NOARGS` - `CALL_NO_KW_METHOD_DESCRIPTOR_FAST`
2023-07-17 11:02:58 -07:00
args--;
total_args++;
}
GH-117457: Correct pystats uop "miss" counts (GH-117477)
2024-04-04 18:49:18 -04:00
if (total_args != 1) {
UOP_STAT_INC(uopcode, miss);
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(0);
GH-117457: Correct pystats uop "miss" counts (GH-117477)
2024-04-04 18:49:18 -04:00
JUMP_TO_JUMP_TARGET();
}
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
if (!PyCFunction_CheckExact(callable_o)) {
GH-117457: Correct pystats uop "miss" counts (GH-117477)
2024-04-04 18:49:18 -04:00
UOP_STAT_INC(uopcode, miss);
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(0);
GH-117457: Correct pystats uop "miss" counts (GH-117477)
2024-04-04 18:49:18 -04:00
JUMP_TO_JUMP_TARGET();
}
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
if (PyCFunction_GET_FLAGS(callable_o) != METH_O) {
GH-117457: Correct pystats uop "miss" counts (GH-117477)
2024-04-04 18:49:18 -04:00
UOP_STAT_INC(uopcode, miss);
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(0);
GH-117457: Correct pystats uop "miss" counts (GH-117477)
2024-04-04 18:49:18 -04:00
JUMP_TO_JUMP_TARGET();
}
GH-130396: Use computed stack limits on linux (GH-130398) * Implement C recursion protection with limit pointers for Linux, MacOS and Windows * Remove calls to PyOS_CheckStack * Add stack protection to parser * Make tests more robust to low stacks * Improve error messages for stack overflow
2025-02-25 09:24:48 +00:00
if (_Py_ReachedRecursionLimit(tstate)) {
GH-117457: Correct pystats uop "miss" counts (GH-117477)
2024-04-04 18:49:18 -04:00
UOP_STAT_INC(uopcode, miss);
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(0);
GH-117457: Correct pystats uop "miss" counts (GH-117477)
2024-04-04 18:49:18 -04:00
JUMP_TO_JUMP_TARGET();
}
gh-106581: Add 10 new opcodes by allowing `assert(kwnames == NULL)` (#106707) By turning `assert(kwnames == NULL)` into a macro that is not in the "forbidden" list, many instructions that formerly were skipped because they contained such an assert (but no other mention of `kwnames`) are now supported in Tier 2. This covers 10 instructions in total (all specializations of `CALL` that invoke some C code): - `CALL_NO_KW_TYPE_1` - `CALL_NO_KW_STR_1` - `CALL_NO_KW_TUPLE_1` - `CALL_NO_KW_BUILTIN_O` - `CALL_NO_KW_BUILTIN_FAST` - `CALL_NO_KW_LEN` - `CALL_NO_KW_ISINSTANCE` - `CALL_NO_KW_METHOD_DESCRIPTOR_O` - `CALL_NO_KW_METHOD_DESCRIPTOR_NOARGS` - `CALL_NO_KW_METHOD_DESCRIPTOR_FAST`
2023-07-17 11:02:58 -07:00
STAT_INC(CALL, hit);
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
PyCFunction cfunc = PyCFunction_GET_FUNCTION(callable_o);
_PyStackRef arg = args[0];
GH-119866: Spill the stack around escaping calls. (GH-124392) * Spill the evaluation around escaping calls in the generated interpreter and JIT. * The code generator tracks live, cached values so they can be saved to memory when needed. * Spills the stack pointer around escaping calls, so that the exact stack is visible to the cycle GC.
2024-10-07 14:56:39 +01:00
_PyFrame_SetStackPointer(frame, stack_pointer);
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
PyObject *res_o = _PyCFunction_TrampolineCall(cfunc, PyCFunction_GET_SELF(callable_o), PyStackRef_AsPyObjectBorrow(arg));
GH-119866: Spill the stack around escaping calls. (GH-124392) * Spill the evaluation around escaping calls in the generated interpreter and JIT. * The code generator tracks live, cached values so they can be saved to memory when needed. * Spills the stack pointer around escaping calls, so that the exact stack is visible to the cycle GC.
2024-10-07 14:56:39 +01:00
stack_pointer = _PyFrame_GetStackPointer(frame);
gh-106581: Add 10 new opcodes by allowing `assert(kwnames == NULL)` (#106707) By turning `assert(kwnames == NULL)` into a macro that is not in the "forbidden" list, many instructions that formerly were skipped because they contained such an assert (but no other mention of `kwnames`) are now supported in Tier 2. This covers 10 instructions in total (all specializations of `CALL` that invoke some C code): - `CALL_NO_KW_TYPE_1` - `CALL_NO_KW_STR_1` - `CALL_NO_KW_TUPLE_1` - `CALL_NO_KW_BUILTIN_O` - `CALL_NO_KW_BUILTIN_FAST` - `CALL_NO_KW_LEN` - `CALL_NO_KW_ISINSTANCE` - `CALL_NO_KW_METHOD_DESCRIPTOR_O` - `CALL_NO_KW_METHOD_DESCRIPTOR_NOARGS` - `CALL_NO_KW_METHOD_DESCRIPTOR_FAST`
2023-07-17 11:02:58 -07:00
_Py_LeaveRecursiveCallTstate(tstate);
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
assert((res_o != NULL) ^ (_PyErr_Occurred(tstate) != NULL));
GH-128682: Make `PyStackRef_CLOSE` escaping. (GH-129404)
2025-02-03 12:41:32 +00:00
if (res_o == NULL) {
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(0);
GH-128682: Make `PyStackRef_CLOSE` escaping. (GH-129404)
2025-02-03 12:41:32 +00:00
JUMP_TO_ERROR();
}
gh-134584: Eliminate redundant refcounting from `_CALL_BUILTION_O` (GH-142695) Co-authored-by: Ken Jin <kenjin4096@gmail.com>
2025-12-14 18:28:02 +08:00
a = arg;
c = callable;
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
res = PyStackRef_FromPyObjectSteal(res_o);
gh-134584: Eliminate redundant refcounting from `_CALL_BUILTION_O` (GH-142695) Co-authored-by: Ken Jin <kenjin4096@gmail.com>
2025-12-14 18:28:02 +08:00
_tos_cache2 = c;
_tos_cache1 = a;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_tos_cache0 = res;
gh-134584: Eliminate redundant refcounting from `_CALL_BUILTION_O` (GH-142695) Co-authored-by: Ken Jin <kenjin4096@gmail.com>
2025-12-14 18:28:02 +08:00
SET_CURRENT_CACHED_VALUES(3);
stack_pointer += -2 - oparg;
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-106581: Add 10 new opcodes by allowing `assert(kwnames == NULL)` (#106707) By turning `assert(kwnames == NULL)` into a macro that is not in the "forbidden" list, many instructions that formerly were skipped because they contained such an assert (but no other mention of `kwnames`) are now supported in Tier 2. This covers 10 instructions in total (all specializations of `CALL` that invoke some C code): - `CALL_NO_KW_TYPE_1` - `CALL_NO_KW_STR_1` - `CALL_NO_KW_TUPLE_1` - `CALL_NO_KW_BUILTIN_O` - `CALL_NO_KW_BUILTIN_FAST` - `CALL_NO_KW_LEN` - `CALL_NO_KW_ISINSTANCE` - `CALL_NO_KW_METHOD_DESCRIPTOR_O` - `CALL_NO_KW_METHOD_DESCRIPTOR_NOARGS` - `CALL_NO_KW_METHOD_DESCRIPTOR_FAST`
2023-07-17 11:02:58 -07:00
break;
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
case _CALL_BUILTIN_FAST_r01: {
CHECK_CURRENT_CACHED_VALUES(0);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
_PyStackRef *args;
GH-131498: Replace single-element arrays with scalars in bytecodes.c (GH-132615)
2025-04-18 07:16:28 -07:00
_PyStackRef self_or_null;
_PyStackRef callable;
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
_PyStackRef res;
GH-111485: Factor out tier 2 code generation from the rest of the interpreter code generator (GH-112968)
2023-12-12 12:12:17 +00:00
oparg = CURRENT_OPARG();
args = &stack_pointer[-oparg];
GH-131498: Replace single-element arrays with scalars in bytecodes.c (GH-132615)
2025-04-18 07:16:28 -07:00
self_or_null = stack_pointer[-1 - oparg];
callable = stack_pointer[-2 - oparg];
gh-106581: Add 10 new opcodes by allowing `assert(kwnames == NULL)` (#106707) By turning `assert(kwnames == NULL)` into a macro that is not in the "forbidden" list, many instructions that formerly were skipped because they contained such an assert (but no other mention of `kwnames`) are now supported in Tier 2. This covers 10 instructions in total (all specializations of `CALL` that invoke some C code): - `CALL_NO_KW_TYPE_1` - `CALL_NO_KW_STR_1` - `CALL_NO_KW_TUPLE_1` - `CALL_NO_KW_BUILTIN_O` - `CALL_NO_KW_BUILTIN_FAST` - `CALL_NO_KW_LEN` - `CALL_NO_KW_ISINSTANCE` - `CALL_NO_KW_METHOD_DESCRIPTOR_O` - `CALL_NO_KW_METHOD_DESCRIPTOR_NOARGS` - `CALL_NO_KW_METHOD_DESCRIPTOR_FAST`
2023-07-17 11:02:58 -07:00
int total_args = oparg;
GH-128682: Stronger checking of `PyStackRef_CLOSE` and `DEAD`. (GH-128683)
2025-01-13 12:37:48 +00:00
_PyStackRef *arguments = args;
GH-131498: Replace single-element arrays with scalars in bytecodes.c (GH-132615)
2025-04-18 07:16:28 -07:00
if (!PyStackRef_IsNull(self_or_null)) {
GH-128682: Stronger checking of `PyStackRef_CLOSE` and `DEAD`. (GH-128683)
2025-01-13 12:37:48 +00:00
arguments--;
gh-106581: Add 10 new opcodes by allowing `assert(kwnames == NULL)` (#106707) By turning `assert(kwnames == NULL)` into a macro that is not in the "forbidden" list, many instructions that formerly were skipped because they contained such an assert (but no other mention of `kwnames`) are now supported in Tier 2. This covers 10 instructions in total (all specializations of `CALL` that invoke some C code): - `CALL_NO_KW_TYPE_1` - `CALL_NO_KW_STR_1` - `CALL_NO_KW_TUPLE_1` - `CALL_NO_KW_BUILTIN_O` - `CALL_NO_KW_BUILTIN_FAST` - `CALL_NO_KW_LEN` - `CALL_NO_KW_ISINSTANCE` - `CALL_NO_KW_METHOD_DESCRIPTOR_O` - `CALL_NO_KW_METHOD_DESCRIPTOR_NOARGS` - `CALL_NO_KW_METHOD_DESCRIPTOR_FAST`
2023-07-17 11:02:58 -07:00
total_args++;
}
GH-141794: Limit size of generated machine code. (GH-142228) * Factor out bodies of the largest uops, to reduce jit code size. * Factor out common assert, also reducing jit code size. * Limit size of jitted code for a single executor to 1MB.
2025-12-03 17:43:35 +00:00
PyObject *callable_o = PyStackRef_AsPyObjectBorrow(callable);
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
if (!PyCFunction_CheckExact(callable_o)) {
GH-117457: Correct pystats uop "miss" counts (GH-117477)
2024-04-04 18:49:18 -04:00
UOP_STAT_INC(uopcode, miss);
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(0);
GH-117457: Correct pystats uop "miss" counts (GH-117477)
2024-04-04 18:49:18 -04:00
JUMP_TO_JUMP_TARGET();
}
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
if (PyCFunction_GET_FLAGS(callable_o) != METH_FASTCALL) {
GH-117457: Correct pystats uop "miss" counts (GH-117477)
2024-04-04 18:49:18 -04:00
UOP_STAT_INC(uopcode, miss);
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(0);
GH-117457: Correct pystats uop "miss" counts (GH-117477)
2024-04-04 18:49:18 -04:00
JUMP_TO_JUMP_TARGET();
}
gh-106581: Add 10 new opcodes by allowing `assert(kwnames == NULL)` (#106707) By turning `assert(kwnames == NULL)` into a macro that is not in the "forbidden" list, many instructions that formerly were skipped because they contained such an assert (but no other mention of `kwnames`) are now supported in Tier 2. This covers 10 instructions in total (all specializations of `CALL` that invoke some C code): - `CALL_NO_KW_TYPE_1` - `CALL_NO_KW_STR_1` - `CALL_NO_KW_TUPLE_1` - `CALL_NO_KW_BUILTIN_O` - `CALL_NO_KW_BUILTIN_FAST` - `CALL_NO_KW_LEN` - `CALL_NO_KW_ISINSTANCE` - `CALL_NO_KW_METHOD_DESCRIPTOR_O` - `CALL_NO_KW_METHOD_DESCRIPTOR_NOARGS` - `CALL_NO_KW_METHOD_DESCRIPTOR_FAST`
2023-07-17 11:02:58 -07:00
STAT_INC(CALL, hit);
GH-119866: Spill the stack around escaping calls. (GH-124392) * Spill the evaluation around escaping calls in the generated interpreter and JIT. * The code generator tracks live, cached values so they can be saved to memory when needed. * Spills the stack pointer around escaping calls, so that the exact stack is visible to the cycle GC.
2024-10-07 14:56:39 +01:00
_PyFrame_SetStackPointer(frame, stack_pointer);
GH-141794: Limit size of generated machine code. (GH-142228) * Factor out bodies of the largest uops, to reduce jit code size. * Factor out common assert, also reducing jit code size. * Limit size of jitted code for a single executor to 1MB.
2025-12-03 17:43:35 +00:00
PyObject *res_o = _Py_BuiltinCallFast_StackRefSteal(
callable,
arguments,
total_args
);
GH-128682: Account for escapes in `DECREF_INPUTS` (GH-129953) * Handle escapes in DECREF_INPUTS * Mark a few more functions as escaping * Replace DECREF_INPUTS with PyStackRef_CLOSE where possible
2025-02-12 17:44:59 +00:00
stack_pointer = _PyFrame_GetStackPointer(frame);
GH-128682: Make `PyStackRef_CLOSE` escaping. (GH-129404)
2025-02-03 12:41:32 +00:00
if (res_o == NULL) {
GH-141794: Limit size of generated machine code. (GH-142228) * Factor out bodies of the largest uops, to reduce jit code size. * Factor out common assert, also reducing jit code size. * Limit size of jitted code for a single executor to 1MB.
2025-12-03 17:43:35 +00:00
stack_pointer += -2 - oparg;
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(0);
GH-128682: Make `PyStackRef_CLOSE` escaping. (GH-129404)
2025-02-03 12:41:32 +00:00
JUMP_TO_ERROR();
}
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
res = PyStackRef_FromPyObjectSteal(res_o);
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_tos_cache0 = res;
_tos_cache1 = PyStackRef_ZERO_BITS;
_tos_cache2 = PyStackRef_ZERO_BITS;
SET_CURRENT_CACHED_VALUES(1);
stack_pointer += -2 - oparg;
GH-141794: Limit size of generated machine code. (GH-142228) * Factor out bodies of the largest uops, to reduce jit code size. * Factor out common assert, also reducing jit code size. * Limit size of jitted code for a single executor to 1MB.
2025-12-03 17:43:35 +00:00
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-106581: Add 10 new opcodes by allowing `assert(kwnames == NULL)` (#106707) By turning `assert(kwnames == NULL)` into a macro that is not in the "forbidden" list, many instructions that formerly were skipped because they contained such an assert (but no other mention of `kwnames`) are now supported in Tier 2. This covers 10 instructions in total (all specializations of `CALL` that invoke some C code): - `CALL_NO_KW_TYPE_1` - `CALL_NO_KW_STR_1` - `CALL_NO_KW_TUPLE_1` - `CALL_NO_KW_BUILTIN_O` - `CALL_NO_KW_BUILTIN_FAST` - `CALL_NO_KW_LEN` - `CALL_NO_KW_ISINSTANCE` - `CALL_NO_KW_METHOD_DESCRIPTOR_O` - `CALL_NO_KW_METHOD_DESCRIPTOR_NOARGS` - `CALL_NO_KW_METHOD_DESCRIPTOR_FAST`
2023-07-17 11:02:58 -07:00
break;
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
case _CALL_BUILTIN_FAST_WITH_KEYWORDS_r01: {
CHECK_CURRENT_CACHED_VALUES(0);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
_PyStackRef *args;
GH-131498: Replace single-element arrays with scalars in bytecodes.c (GH-132615)
2025-04-18 07:16:28 -07:00
_PyStackRef self_or_null;
_PyStackRef callable;
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
_PyStackRef res;
GH-111485: Factor out tier 2 code generation from the rest of the interpreter code generator (GH-112968)
2023-12-12 12:12:17 +00:00
oparg = CURRENT_OPARG();
args = &stack_pointer[-oparg];
GH-131498: Replace single-element arrays with scalars in bytecodes.c (GH-132615)
2025-04-18 07:16:28 -07:00
self_or_null = stack_pointer[-1 - oparg];
callable = stack_pointer[-2 - oparg];
GH-105848: Replace KW_NAMES + CALL with LOAD_CONST + CALL_KW (GH-109300)
2023-09-13 10:25:45 -07:00
int total_args = oparg;
GH-128682: Convert explicit loops closing arrays into `DECREF_INPUTS`. (GH-128822) * Mark Py_DECREF and Py_XDECREF as escaping * Remove explicit loops for clearing array inputs
2025-01-14 15:08:56 +00:00
_PyStackRef *arguments = args;
GH-131498: Replace single-element arrays with scalars in bytecodes.c (GH-132615)
2025-04-18 07:16:28 -07:00
if (!PyStackRef_IsNull(self_or_null)) {
GH-128682: Convert explicit loops closing arrays into `DECREF_INPUTS`. (GH-128822) * Mark Py_DECREF and Py_XDECREF as escaping * Remove explicit loops for clearing array inputs
2025-01-14 15:08:56 +00:00
arguments--;
GH-105848: Replace KW_NAMES + CALL with LOAD_CONST + CALL_KW (GH-109300)
2023-09-13 10:25:45 -07:00
total_args++;
}
GH-141794: Limit size of generated machine code. (GH-142228) * Factor out bodies of the largest uops, to reduce jit code size. * Factor out common assert, also reducing jit code size. * Limit size of jitted code for a single executor to 1MB.
2025-12-03 17:43:35 +00:00
PyObject *callable_o = PyStackRef_AsPyObjectBorrow(callable);
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
if (!PyCFunction_CheckExact(callable_o)) {
GH-117457: Correct pystats uop "miss" counts (GH-117477)
2024-04-04 18:49:18 -04:00
UOP_STAT_INC(uopcode, miss);
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(0);
GH-117457: Correct pystats uop "miss" counts (GH-117477)
2024-04-04 18:49:18 -04:00
JUMP_TO_JUMP_TARGET();
}
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
if (PyCFunction_GET_FLAGS(callable_o) != (METH_FASTCALL | METH_KEYWORDS)) {
GH-117457: Correct pystats uop "miss" counts (GH-117477)
2024-04-04 18:49:18 -04:00
UOP_STAT_INC(uopcode, miss);
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(0);
GH-117457: Correct pystats uop "miss" counts (GH-117477)
2024-04-04 18:49:18 -04:00
JUMP_TO_JUMP_TARGET();
}
GH-105848: Replace KW_NAMES + CALL with LOAD_CONST + CALL_KW (GH-109300)
2023-09-13 10:25:45 -07:00
STAT_INC(CALL, hit);
GH-119866: Spill the stack around escaping calls. (GH-124392) * Spill the evaluation around escaping calls in the generated interpreter and JIT. * The code generator tracks live, cached values so they can be saved to memory when needed. * Spills the stack pointer around escaping calls, so that the exact stack is visible to the cycle GC.
2024-10-07 14:56:39 +01:00
_PyFrame_SetStackPointer(frame, stack_pointer);
GH-141794: Limit size of generated machine code. (GH-142228) * Factor out bodies of the largest uops, to reduce jit code size. * Factor out common assert, also reducing jit code size. * Limit size of jitted code for a single executor to 1MB.
2025-12-03 17:43:35 +00:00
PyObject *res_o = _Py_BuiltinCallFastWithKeywords_StackRefSteal(callable, arguments, total_args);
GH-128682: Account for escapes in `DECREF_INPUTS` (GH-129953) * Handle escapes in DECREF_INPUTS * Mark a few more functions as escaping * Replace DECREF_INPUTS with PyStackRef_CLOSE where possible
2025-02-12 17:44:59 +00:00
stack_pointer = _PyFrame_GetStackPointer(frame);
GH-128682: Make `PyStackRef_CLOSE` escaping. (GH-129404)
2025-02-03 12:41:32 +00:00
if (res_o == NULL) {
GH-141794: Limit size of generated machine code. (GH-142228) * Factor out bodies of the largest uops, to reduce jit code size. * Factor out common assert, also reducing jit code size. * Limit size of jitted code for a single executor to 1MB.
2025-12-03 17:43:35 +00:00
stack_pointer += -2 - oparg;
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(0);
GH-128682: Make `PyStackRef_CLOSE` escaping. (GH-129404)
2025-02-03 12:41:32 +00:00
JUMP_TO_ERROR();
}
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
res = PyStackRef_FromPyObjectSteal(res_o);
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_tos_cache0 = res;
_tos_cache1 = PyStackRef_ZERO_BITS;
_tos_cache2 = PyStackRef_ZERO_BITS;
SET_CURRENT_CACHED_VALUES(1);
stack_pointer += -2 - oparg;
GH-141794: Limit size of generated machine code. (GH-142228) * Factor out bodies of the largest uops, to reduce jit code size. * Factor out common assert, also reducing jit code size. * Limit size of jitted code for a single executor to 1MB.
2025-12-03 17:43:35 +00:00
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-105848: Replace KW_NAMES + CALL with LOAD_CONST + CALL_KW (GH-109300)
2023-09-13 10:25:45 -07:00
break;
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
case _GUARD_CALLABLE_LEN_r03: {
CHECK_CURRENT_CACHED_VALUES(0);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-131498: Replace single-element arrays with scalars in bytecodes.c (GH-132615)
2025-04-18 07:16:28 -07:00
_PyStackRef callable;
GH-131798: Split CALL_LEN into several uops (GH-133180)
2025-05-05 22:31:48 +01:00
callable = stack_pointer[-3];
GH-131498: Replace single-element arrays with scalars in bytecodes.c (GH-132615)
2025-04-18 07:16:28 -07:00
PyObject *callable_o = PyStackRef_AsPyObjectBorrow(callable);
Use tstate->interp to get the interpreter state in bytecodes.c (GH-107506)
2023-07-31 14:18:38 -07:00
PyInterpreterState *interp = tstate->interp;
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
if (callable_o != interp->callable_cache.len) {
GH-117457: Correct pystats uop "miss" counts (GH-117477)
2024-04-04 18:49:18 -04:00
UOP_STAT_INC(uopcode, miss);
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(0);
JUMP_TO_JUMP_TARGET();
}
_tos_cache2 = stack_pointer[-1];
_tos_cache1 = stack_pointer[-2];
_tos_cache0 = callable;
SET_CURRENT_CACHED_VALUES(3);
stack_pointer += -3;
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _GUARD_CALLABLE_LEN_r13: {
CHECK_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef callable;
_PyStackRef _stack_item_0 = _tos_cache0;
callable = stack_pointer[-2];
PyObject *callable_o = PyStackRef_AsPyObjectBorrow(callable);
PyInterpreterState *interp = tstate->interp;
if (callable_o != interp->callable_cache.len) {
UOP_STAT_INC(uopcode, miss);
GH-142718: JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. (GH-142762) JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. * Make sure that stack is in correct state at side exits with TOS cached values * Simplify choice of cached items for side exits
2025-12-15 19:01:29 +00:00
_tos_cache0 = _stack_item_0;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(1);
JUMP_TO_JUMP_TARGET();
}
_tos_cache2 = _stack_item_0;
_tos_cache1 = stack_pointer[-1];
_tos_cache0 = callable;
SET_CURRENT_CACHED_VALUES(3);
stack_pointer += -2;
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _GUARD_CALLABLE_LEN_r23: {
CHECK_CURRENT_CACHED_VALUES(2);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef callable;
_PyStackRef _stack_item_0 = _tos_cache0;
_PyStackRef _stack_item_1 = _tos_cache1;
callable = stack_pointer[-1];
PyObject *callable_o = PyStackRef_AsPyObjectBorrow(callable);
PyInterpreterState *interp = tstate->interp;
if (callable_o != interp->callable_cache.len) {
UOP_STAT_INC(uopcode, miss);
GH-142718: JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. (GH-142762) JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. * Make sure that stack is in correct state at side exits with TOS cached values * Simplify choice of cached items for side exits
2025-12-15 19:01:29 +00:00
_tos_cache1 = _stack_item_1;
_tos_cache0 = _stack_item_0;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(2);
JUMP_TO_JUMP_TARGET();
}
_tos_cache2 = _stack_item_1;
_tos_cache1 = _stack_item_0;
_tos_cache0 = callable;
SET_CURRENT_CACHED_VALUES(3);
stack_pointer += -1;
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _GUARD_CALLABLE_LEN_r33: {
CHECK_CURRENT_CACHED_VALUES(3);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef callable;
_PyStackRef _stack_item_0 = _tos_cache0;
_PyStackRef _stack_item_1 = _tos_cache1;
_PyStackRef _stack_item_2 = _tos_cache2;
callable = _stack_item_0;
PyObject *callable_o = PyStackRef_AsPyObjectBorrow(callable);
PyInterpreterState *interp = tstate->interp;
if (callable_o != interp->callable_cache.len) {
UOP_STAT_INC(uopcode, miss);
GH-142718: JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. (GH-142762) JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. * Make sure that stack is in correct state at side exits with TOS cached values * Simplify choice of cached items for side exits
2025-12-15 19:01:29 +00:00
_tos_cache2 = _stack_item_2;
_tos_cache1 = _stack_item_1;
_tos_cache0 = callable;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(3);
GH-117457: Correct pystats uop "miss" counts (GH-117477)
2024-04-04 18:49:18 -04:00
JUMP_TO_JUMP_TARGET();
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_tos_cache2 = _stack_item_2;
_tos_cache1 = _stack_item_1;
_tos_cache0 = callable;
SET_CURRENT_CACHED_VALUES(3);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-131798: Split CALL_LEN into several uops (GH-133180)
2025-05-05 22:31:48 +01:00
break;
}
gh-134584: Eliminate redundant refcounting from ``_CALL_LEN`` (gh-136104)
2025-12-12 00:24:34 +09:00
case _CALL_LEN_r33: {
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
CHECK_CURRENT_CACHED_VALUES(3);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-131798: Split CALL_LEN into several uops (GH-133180)
2025-05-05 22:31:48 +01:00
_PyStackRef arg;
_PyStackRef callable;
_PyStackRef res;
gh-134584: Eliminate redundant refcounting from ``_CALL_LEN`` (gh-136104)
2025-12-12 00:24:34 +09:00
_PyStackRef a;
_PyStackRef c;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef _stack_item_0 = _tos_cache0;
_PyStackRef _stack_item_1 = _tos_cache1;
_PyStackRef _stack_item_2 = _tos_cache2;
arg = _stack_item_2;
callable = _stack_item_0;
gh-106581: Add 10 new opcodes by allowing `assert(kwnames == NULL)` (#106707) By turning `assert(kwnames == NULL)` into a macro that is not in the "forbidden" list, many instructions that formerly were skipped because they contained such an assert (but no other mention of `kwnames`) are now supported in Tier 2. This covers 10 instructions in total (all specializations of `CALL` that invoke some C code): - `CALL_NO_KW_TYPE_1` - `CALL_NO_KW_STR_1` - `CALL_NO_KW_TUPLE_1` - `CALL_NO_KW_BUILTIN_O` - `CALL_NO_KW_BUILTIN_FAST` - `CALL_NO_KW_LEN` - `CALL_NO_KW_ISINSTANCE` - `CALL_NO_KW_METHOD_DESCRIPTOR_O` - `CALL_NO_KW_METHOD_DESCRIPTOR_NOARGS` - `CALL_NO_KW_METHOD_DESCRIPTOR_FAST`
2023-07-17 11:02:58 -07:00
STAT_INC(CALL, hit);
GH-131798: Split CALL_LEN into several uops (GH-133180)
2025-05-05 22:31:48 +01:00
PyObject *arg_o = PyStackRef_AsPyObjectBorrow(arg);
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
stack_pointer[0] = callable;
gh-134584: Eliminate redundant refcounting from ``_CALL_LEN`` (gh-136104)
2025-12-12 00:24:34 +09:00
stack_pointer[1] = _stack_item_1;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
stack_pointer[2] = arg;
stack_pointer += 3;
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
GH-119866: Spill the stack around escaping calls. (GH-124392) * Spill the evaluation around escaping calls in the generated interpreter and JIT. * The code generator tracks live, cached values so they can be saved to memory when needed. * Spills the stack pointer around escaping calls, so that the exact stack is visible to the cycle GC.
2024-10-07 14:56:39 +01:00
_PyFrame_SetStackPointer(frame, stack_pointer);
GH-131798: Split CALL_LEN into several uops (GH-133180)
2025-05-05 22:31:48 +01:00
Py_ssize_t len_i = PyObject_Length(arg_o);
GH-119866: Spill the stack around escaping calls. (GH-124392) * Spill the evaluation around escaping calls in the generated interpreter and JIT. * The code generator tracks live, cached values so they can be saved to memory when needed. * Spills the stack pointer around escaping calls, so that the exact stack is visible to the cycle GC.
2024-10-07 14:56:39 +01:00
stack_pointer = _PyFrame_GetStackPointer(frame);
gh-106581: Add 10 new opcodes by allowing `assert(kwnames == NULL)` (#106707) By turning `assert(kwnames == NULL)` into a macro that is not in the "forbidden" list, many instructions that formerly were skipped because they contained such an assert (but no other mention of `kwnames`) are now supported in Tier 2. This covers 10 instructions in total (all specializations of `CALL` that invoke some C code): - `CALL_NO_KW_TYPE_1` - `CALL_NO_KW_STR_1` - `CALL_NO_KW_TUPLE_1` - `CALL_NO_KW_BUILTIN_O` - `CALL_NO_KW_BUILTIN_FAST` - `CALL_NO_KW_LEN` - `CALL_NO_KW_ISINSTANCE` - `CALL_NO_KW_METHOD_DESCRIPTOR_O` - `CALL_NO_KW_METHOD_DESCRIPTOR_NOARGS` - `CALL_NO_KW_METHOD_DESCRIPTOR_FAST`
2023-07-17 11:02:58 -07:00
if (len_i < 0) {
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(0);
GH-116422: Tier2 hot/cold splitting (GH-116813) Splits the "cold" path, deopts and exits, from the "hot" path, reducing the size of most jitted instructions, at the cost of slower exits.
2024-03-26 09:35:11 +00:00
JUMP_TO_ERROR();
gh-106581: Add 10 new opcodes by allowing `assert(kwnames == NULL)` (#106707) By turning `assert(kwnames == NULL)` into a macro that is not in the "forbidden" list, many instructions that formerly were skipped because they contained such an assert (but no other mention of `kwnames`) are now supported in Tier 2. This covers 10 instructions in total (all specializations of `CALL` that invoke some C code): - `CALL_NO_KW_TYPE_1` - `CALL_NO_KW_STR_1` - `CALL_NO_KW_TUPLE_1` - `CALL_NO_KW_BUILTIN_O` - `CALL_NO_KW_BUILTIN_FAST` - `CALL_NO_KW_LEN` - `CALL_NO_KW_ISINSTANCE` - `CALL_NO_KW_METHOD_DESCRIPTOR_O` - `CALL_NO_KW_METHOD_DESCRIPTOR_NOARGS` - `CALL_NO_KW_METHOD_DESCRIPTOR_FAST`
2023-07-17 11:02:58 -07:00
}
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
PyObject *res_o = PyLong_FromSsize_t(len_i);
assert((res_o != NULL) ^ (_PyErr_Occurred(tstate) != NULL));
if (res_o == NULL) {
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(0);
GH-129709: Clean up tier two (GH-129710)
2025-02-07 09:52:49 -08:00
JUMP_TO_ERROR();
GH-116422: Modify a few uops so that they can be supported by tier 2 with hot/cold splitting (GH-116832)
2024-03-15 10:48:00 +00:00
}
gh-134584: Eliminate redundant refcounting from ``_CALL_LEN`` (gh-136104)
2025-12-12 00:24:34 +09:00
a = arg;
c = callable;
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
res = PyStackRef_FromPyObjectSteal(res_o);
gh-134584: Eliminate redundant refcounting from ``_CALL_LEN`` (gh-136104)
2025-12-12 00:24:34 +09:00
_tos_cache2 = c;
_tos_cache1 = a;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_tos_cache0 = res;
gh-134584: Eliminate redundant refcounting from ``_CALL_LEN`` (gh-136104)
2025-12-12 00:24:34 +09:00
SET_CURRENT_CACHED_VALUES(3);
stack_pointer += -3;
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-106581: Add 10 new opcodes by allowing `assert(kwnames == NULL)` (#106707) By turning `assert(kwnames == NULL)` into a macro that is not in the "forbidden" list, many instructions that formerly were skipped because they contained such an assert (but no other mention of `kwnames`) are now supported in Tier 2. This covers 10 instructions in total (all specializations of `CALL` that invoke some C code): - `CALL_NO_KW_TYPE_1` - `CALL_NO_KW_STR_1` - `CALL_NO_KW_TUPLE_1` - `CALL_NO_KW_BUILTIN_O` - `CALL_NO_KW_BUILTIN_FAST` - `CALL_NO_KW_LEN` - `CALL_NO_KW_ISINSTANCE` - `CALL_NO_KW_METHOD_DESCRIPTOR_O` - `CALL_NO_KW_METHOD_DESCRIPTOR_NOARGS` - `CALL_NO_KW_METHOD_DESCRIPTOR_FAST`
2023-07-17 11:02:58 -07:00
break;
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
case _GUARD_CALLABLE_ISINSTANCE_r03: {
CHECK_CURRENT_CACHED_VALUES(0);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-131729: Code-gen better liveness analysis (GH-131732) * Rename 'defined' attribute to 'in_local' to more accurately reflect how it is used * Make death of variables explicit even for array variables. * Convert in_memory from boolean to stack offset * Don't apply liveness analysis to optimizer generated code * Fix RETURN_VALUE in optimizer
2025-03-26 15:21:35 +00:00
_PyStackRef callable;
GH-131798: Split up and optimize CALL_ISINSTANCE (GH-133339)
2025-05-08 23:26:30 +02:00
callable = stack_pointer[-4];
GH-131729: Code-gen better liveness analysis (GH-131732) * Rename 'defined' attribute to 'in_local' to more accurately reflect how it is used * Make death of variables explicit even for array variables. * Convert in_memory from boolean to stack offset * Don't apply liveness analysis to optimizer generated code * Fix RETURN_VALUE in optimizer
2025-03-26 15:21:35 +00:00
PyObject *callable_o = PyStackRef_AsPyObjectBorrow(callable);
Use tstate->interp to get the interpreter state in bytecodes.c (GH-107506)
2023-07-31 14:18:38 -07:00
PyInterpreterState *interp = tstate->interp;
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
if (callable_o != interp->callable_cache.isinstance) {
GH-117457: Correct pystats uop "miss" counts (GH-117477)
2024-04-04 18:49:18 -04:00
UOP_STAT_INC(uopcode, miss);
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(0);
JUMP_TO_JUMP_TARGET();
}
_tos_cache2 = stack_pointer[-1];
_tos_cache1 = stack_pointer[-2];
_tos_cache0 = stack_pointer[-3];
SET_CURRENT_CACHED_VALUES(3);
stack_pointer += -3;
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _GUARD_CALLABLE_ISINSTANCE_r13: {
CHECK_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef callable;
_PyStackRef _stack_item_0 = _tos_cache0;
callable = stack_pointer[-3];
PyObject *callable_o = PyStackRef_AsPyObjectBorrow(callable);
PyInterpreterState *interp = tstate->interp;
if (callable_o != interp->callable_cache.isinstance) {
UOP_STAT_INC(uopcode, miss);
GH-142718: JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. (GH-142762) JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. * Make sure that stack is in correct state at side exits with TOS cached values * Simplify choice of cached items for side exits
2025-12-15 19:01:29 +00:00
_tos_cache0 = _stack_item_0;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(1);
JUMP_TO_JUMP_TARGET();
}
_tos_cache2 = _stack_item_0;
_tos_cache1 = stack_pointer[-1];
_tos_cache0 = stack_pointer[-2];
SET_CURRENT_CACHED_VALUES(3);
stack_pointer += -2;
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _GUARD_CALLABLE_ISINSTANCE_r23: {
CHECK_CURRENT_CACHED_VALUES(2);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef callable;
_PyStackRef _stack_item_0 = _tos_cache0;
_PyStackRef _stack_item_1 = _tos_cache1;
callable = stack_pointer[-2];
PyObject *callable_o = PyStackRef_AsPyObjectBorrow(callable);
PyInterpreterState *interp = tstate->interp;
if (callable_o != interp->callable_cache.isinstance) {
UOP_STAT_INC(uopcode, miss);
GH-142718: JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. (GH-142762) JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. * Make sure that stack is in correct state at side exits with TOS cached values * Simplify choice of cached items for side exits
2025-12-15 19:01:29 +00:00
_tos_cache1 = _stack_item_1;
_tos_cache0 = _stack_item_0;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(2);
JUMP_TO_JUMP_TARGET();
}
_tos_cache2 = _stack_item_1;
_tos_cache1 = _stack_item_0;
_tos_cache0 = stack_pointer[-1];
SET_CURRENT_CACHED_VALUES(3);
stack_pointer += -1;
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _GUARD_CALLABLE_ISINSTANCE_r33: {
CHECK_CURRENT_CACHED_VALUES(3);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef callable;
_PyStackRef _stack_item_0 = _tos_cache0;
_PyStackRef _stack_item_1 = _tos_cache1;
_PyStackRef _stack_item_2 = _tos_cache2;
callable = stack_pointer[-1];
PyObject *callable_o = PyStackRef_AsPyObjectBorrow(callable);
PyInterpreterState *interp = tstate->interp;
if (callable_o != interp->callable_cache.isinstance) {
UOP_STAT_INC(uopcode, miss);
GH-142718: JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. (GH-142762) JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. * Make sure that stack is in correct state at side exits with TOS cached values * Simplify choice of cached items for side exits
2025-12-15 19:01:29 +00:00
_tos_cache2 = _stack_item_2;
_tos_cache1 = _stack_item_1;
_tos_cache0 = _stack_item_0;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(3);
GH-117457: Correct pystats uop "miss" counts (GH-117477)
2024-04-04 18:49:18 -04:00
JUMP_TO_JUMP_TARGET();
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_tos_cache2 = _stack_item_2;
_tos_cache1 = _stack_item_1;
_tos_cache0 = _stack_item_0;
SET_CURRENT_CACHED_VALUES(3);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-131798: Split up and optimize CALL_ISINSTANCE (GH-133339)
2025-05-08 23:26:30 +02:00
break;
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
case _CALL_ISINSTANCE_r31: {
CHECK_CURRENT_CACHED_VALUES(3);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-131798: Split up and optimize CALL_ISINSTANCE (GH-133339)
2025-05-08 23:26:30 +02:00
_PyStackRef cls;
_PyStackRef instance;
_PyStackRef null;
_PyStackRef callable;
_PyStackRef res;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef _stack_item_0 = _tos_cache0;
_PyStackRef _stack_item_1 = _tos_cache1;
_PyStackRef _stack_item_2 = _tos_cache2;
cls = _stack_item_2;
instance = _stack_item_1;
null = _stack_item_0;
callable = stack_pointer[-1];
gh-106581: Add 10 new opcodes by allowing `assert(kwnames == NULL)` (#106707) By turning `assert(kwnames == NULL)` into a macro that is not in the "forbidden" list, many instructions that formerly were skipped because they contained such an assert (but no other mention of `kwnames`) are now supported in Tier 2. This covers 10 instructions in total (all specializations of `CALL` that invoke some C code): - `CALL_NO_KW_TYPE_1` - `CALL_NO_KW_STR_1` - `CALL_NO_KW_TUPLE_1` - `CALL_NO_KW_BUILTIN_O` - `CALL_NO_KW_BUILTIN_FAST` - `CALL_NO_KW_LEN` - `CALL_NO_KW_ISINSTANCE` - `CALL_NO_KW_METHOD_DESCRIPTOR_O` - `CALL_NO_KW_METHOD_DESCRIPTOR_NOARGS` - `CALL_NO_KW_METHOD_DESCRIPTOR_FAST`
2023-07-17 11:02:58 -07:00
STAT_INC(CALL, hit);
GH-131798: Split up and optimize CALL_ISINSTANCE (GH-133339)
2025-05-08 23:26:30 +02:00
PyObject *inst_o = PyStackRef_AsPyObjectBorrow(instance);
PyObject *cls_o = PyStackRef_AsPyObjectBorrow(cls);
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
stack_pointer[0] = null;
stack_pointer[1] = instance;
stack_pointer[2] = cls;
stack_pointer += 3;
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
GH-119866: Spill the stack around escaping calls. (GH-124392) * Spill the evaluation around escaping calls in the generated interpreter and JIT. * The code generator tracks live, cached values so they can be saved to memory when needed. * Spills the stack pointer around escaping calls, so that the exact stack is visible to the cycle GC.
2024-10-07 14:56:39 +01:00
_PyFrame_SetStackPointer(frame, stack_pointer);
GH-131798: Split up and optimize CALL_ISINSTANCE (GH-133339)
2025-05-08 23:26:30 +02:00
int retval = PyObject_IsInstance(inst_o, cls_o);
GH-119866: Spill the stack around escaping calls. (GH-124392) * Spill the evaluation around escaping calls in the generated interpreter and JIT. * The code generator tracks live, cached values so they can be saved to memory when needed. * Spills the stack pointer around escaping calls, so that the exact stack is visible to the cycle GC.
2024-10-07 14:56:39 +01:00
stack_pointer = _PyFrame_GetStackPointer(frame);
gh-106581: Add 10 new opcodes by allowing `assert(kwnames == NULL)` (#106707) By turning `assert(kwnames == NULL)` into a macro that is not in the "forbidden" list, many instructions that formerly were skipped because they contained such an assert (but no other mention of `kwnames`) are now supported in Tier 2. This covers 10 instructions in total (all specializations of `CALL` that invoke some C code): - `CALL_NO_KW_TYPE_1` - `CALL_NO_KW_STR_1` - `CALL_NO_KW_TUPLE_1` - `CALL_NO_KW_BUILTIN_O` - `CALL_NO_KW_BUILTIN_FAST` - `CALL_NO_KW_LEN` - `CALL_NO_KW_ISINSTANCE` - `CALL_NO_KW_METHOD_DESCRIPTOR_O` - `CALL_NO_KW_METHOD_DESCRIPTOR_NOARGS` - `CALL_NO_KW_METHOD_DESCRIPTOR_FAST`
2023-07-17 11:02:58 -07:00
if (retval < 0) {
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(0);
GH-116422: Tier2 hot/cold splitting (GH-116813) Splits the "cold" path, deopts and exits, from the "hot" path, reducing the size of most jitted instructions, at the cost of slower exits.
2024-03-26 09:35:11 +00:00
JUMP_TO_ERROR();
gh-106581: Add 10 new opcodes by allowing `assert(kwnames == NULL)` (#106707) By turning `assert(kwnames == NULL)` into a macro that is not in the "forbidden" list, many instructions that formerly were skipped because they contained such an assert (but no other mention of `kwnames`) are now supported in Tier 2. This covers 10 instructions in total (all specializations of `CALL` that invoke some C code): - `CALL_NO_KW_TYPE_1` - `CALL_NO_KW_STR_1` - `CALL_NO_KW_TUPLE_1` - `CALL_NO_KW_BUILTIN_O` - `CALL_NO_KW_BUILTIN_FAST` - `CALL_NO_KW_LEN` - `CALL_NO_KW_ISINSTANCE` - `CALL_NO_KW_METHOD_DESCRIPTOR_O` - `CALL_NO_KW_METHOD_DESCRIPTOR_NOARGS` - `CALL_NO_KW_METHOD_DESCRIPTOR_FAST`
2023-07-17 11:02:58 -07:00
}
GH-131798: Split up and optimize CALL_ISINSTANCE (GH-133339)
2025-05-08 23:26:30 +02:00
(void)null;
stack_pointer += -1;
GH-141794: Limit size of generated machine code. (GH-142228) * Factor out bodies of the largest uops, to reduce jit code size. * Factor out common assert, also reducing jit code size. * Limit size of jitted code for a single executor to 1MB.
2025-12-03 17:43:35 +00:00
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
GH-128682: Account for escapes in `DECREF_INPUTS` (GH-129953) * Handle escapes in DECREF_INPUTS * Mark a few more functions as escaping * Replace DECREF_INPUTS with PyStackRef_CLOSE where possible
2025-02-12 17:44:59 +00:00
_PyFrame_SetStackPointer(frame, stack_pointer);
GH-131798: Split up and optimize CALL_ISINSTANCE (GH-133339)
2025-05-08 23:26:30 +02:00
PyStackRef_CLOSE(cls);
GH-128682: Account for escapes in `DECREF_INPUTS` (GH-129953) * Handle escapes in DECREF_INPUTS * Mark a few more functions as escaping * Replace DECREF_INPUTS with PyStackRef_CLOSE where possible
2025-02-12 17:44:59 +00:00
stack_pointer = _PyFrame_GetStackPointer(frame);
GH-131798: Split up and optimize CALL_ISINSTANCE (GH-133339)
2025-05-08 23:26:30 +02:00
stack_pointer += -1;
GH-141794: Limit size of generated machine code. (GH-142228) * Factor out bodies of the largest uops, to reduce jit code size. * Factor out common assert, also reducing jit code size. * Limit size of jitted code for a single executor to 1MB.
2025-12-03 17:43:35 +00:00
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
GH-131798: Split up and optimize CALL_ISINSTANCE (GH-133339)
2025-05-08 23:26:30 +02:00
_PyFrame_SetStackPointer(frame, stack_pointer);
PyStackRef_CLOSE(instance);
stack_pointer = _PyFrame_GetStackPointer(frame);
stack_pointer += -2;
GH-141794: Limit size of generated machine code. (GH-142228) * Factor out bodies of the largest uops, to reduce jit code size. * Factor out common assert, also reducing jit code size. * Limit size of jitted code for a single executor to 1MB.
2025-12-03 17:43:35 +00:00
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
GH-131798: Split up and optimize CALL_ISINSTANCE (GH-133339)
2025-05-08 23:26:30 +02:00
_PyFrame_SetStackPointer(frame, stack_pointer);
PyStackRef_CLOSE(callable);
stack_pointer = _PyFrame_GetStackPointer(frame);
res = retval ? PyStackRef_True : PyStackRef_False;
assert((!PyStackRef_IsNull(res)) ^ (_PyErr_Occurred(tstate) != NULL));
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_tos_cache0 = res;
_tos_cache1 = PyStackRef_ZERO_BITS;
_tos_cache2 = PyStackRef_ZERO_BITS;
SET_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-106581: Add 10 new opcodes by allowing `assert(kwnames == NULL)` (#106707) By turning `assert(kwnames == NULL)` into a macro that is not in the "forbidden" list, many instructions that formerly were skipped because they contained such an assert (but no other mention of `kwnames`) are now supported in Tier 2. This covers 10 instructions in total (all specializations of `CALL` that invoke some C code): - `CALL_NO_KW_TYPE_1` - `CALL_NO_KW_STR_1` - `CALL_NO_KW_TUPLE_1` - `CALL_NO_KW_BUILTIN_O` - `CALL_NO_KW_BUILTIN_FAST` - `CALL_NO_KW_LEN` - `CALL_NO_KW_ISINSTANCE` - `CALL_NO_KW_METHOD_DESCRIPTOR_O` - `CALL_NO_KW_METHOD_DESCRIPTOR_NOARGS` - `CALL_NO_KW_METHOD_DESCRIPTOR_FAST`
2023-07-17 11:02:58 -07:00
break;
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
case _GUARD_CALLABLE_LIST_APPEND_r03: {
CHECK_CURRENT_CACHED_VALUES(0);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-131798: Split CALL_LIST_APPEND into several uops (GH-134240)
2025-05-19 15:48:55 -04:00
_PyStackRef callable;
callable = stack_pointer[-3];
PyObject *callable_o = PyStackRef_AsPyObjectBorrow(callable);
PyInterpreterState *interp = tstate->interp;
if (callable_o != interp->callable_cache.list_append) {
UOP_STAT_INC(uopcode, miss);
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(0);
JUMP_TO_JUMP_TARGET();
}
_tos_cache2 = stack_pointer[-1];
_tos_cache1 = stack_pointer[-2];
_tos_cache0 = callable;
SET_CURRENT_CACHED_VALUES(3);
stack_pointer += -3;
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _GUARD_CALLABLE_LIST_APPEND_r13: {
CHECK_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef callable;
_PyStackRef _stack_item_0 = _tos_cache0;
callable = stack_pointer[-2];
PyObject *callable_o = PyStackRef_AsPyObjectBorrow(callable);
PyInterpreterState *interp = tstate->interp;
if (callable_o != interp->callable_cache.list_append) {
UOP_STAT_INC(uopcode, miss);
GH-142718: JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. (GH-142762) JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. * Make sure that stack is in correct state at side exits with TOS cached values * Simplify choice of cached items for side exits
2025-12-15 19:01:29 +00:00
_tos_cache0 = _stack_item_0;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(1);
JUMP_TO_JUMP_TARGET();
}
_tos_cache2 = _stack_item_0;
_tos_cache1 = stack_pointer[-1];
_tos_cache0 = callable;
SET_CURRENT_CACHED_VALUES(3);
stack_pointer += -2;
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _GUARD_CALLABLE_LIST_APPEND_r23: {
CHECK_CURRENT_CACHED_VALUES(2);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef callable;
_PyStackRef _stack_item_0 = _tos_cache0;
_PyStackRef _stack_item_1 = _tos_cache1;
callable = stack_pointer[-1];
PyObject *callable_o = PyStackRef_AsPyObjectBorrow(callable);
PyInterpreterState *interp = tstate->interp;
if (callable_o != interp->callable_cache.list_append) {
UOP_STAT_INC(uopcode, miss);
GH-142718: JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. (GH-142762) JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. * Make sure that stack is in correct state at side exits with TOS cached values * Simplify choice of cached items for side exits
2025-12-15 19:01:29 +00:00
_tos_cache1 = _stack_item_1;
_tos_cache0 = _stack_item_0;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(2);
JUMP_TO_JUMP_TARGET();
}
_tos_cache2 = _stack_item_1;
_tos_cache1 = _stack_item_0;
_tos_cache0 = callable;
SET_CURRENT_CACHED_VALUES(3);
stack_pointer += -1;
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _GUARD_CALLABLE_LIST_APPEND_r33: {
CHECK_CURRENT_CACHED_VALUES(3);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef callable;
_PyStackRef _stack_item_0 = _tos_cache0;
_PyStackRef _stack_item_1 = _tos_cache1;
_PyStackRef _stack_item_2 = _tos_cache2;
callable = _stack_item_0;
PyObject *callable_o = PyStackRef_AsPyObjectBorrow(callable);
PyInterpreterState *interp = tstate->interp;
if (callable_o != interp->callable_cache.list_append) {
UOP_STAT_INC(uopcode, miss);
GH-142718: JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. (GH-142762) JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. * Make sure that stack is in correct state at side exits with TOS cached values * Simplify choice of cached items for side exits
2025-12-15 19:01:29 +00:00
_tos_cache2 = _stack_item_2;
_tos_cache1 = _stack_item_1;
_tos_cache0 = callable;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(3);
GH-131798: Split CALL_LIST_APPEND into several uops (GH-134240)
2025-05-19 15:48:55 -04:00
JUMP_TO_JUMP_TARGET();
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_tos_cache2 = _stack_item_2;
_tos_cache1 = _stack_item_1;
_tos_cache0 = callable;
SET_CURRENT_CACHED_VALUES(3);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-131798: Split CALL_LIST_APPEND into several uops (GH-134240)
2025-05-19 15:48:55 -04:00
break;
}
gh-143092: Make CALL_LIST_APPEND and BINARY_OP_INPLACE_ADD_UNICODE normal instructions (GH-143124) These super instructions need many special cases in the interpreter, specializer, and JIT. It's best we convert them to normal instructions.
2025-12-25 06:03:00 +08:00
case _CALL_LIST_APPEND_r03: {
gh-134584: Eliminate redundant refcounting from _CALL_LIST_APPEND (GH-142711) Signed-off-by: Manjusaka <me@manjusaka.me> Co-authored-by: Ken Jin <kenjin4096@gmail.com>
2025-12-15 23:49:36 +08:00
CHECK_CURRENT_CACHED_VALUES(0);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-134584: Eliminate redundant refcounting from _CALL_LIST_APPEND (GH-142711) Signed-off-by: Manjusaka <me@manjusaka.me> Co-authored-by: Ken Jin <kenjin4096@gmail.com>
2025-12-15 23:49:36 +08:00
_PyStackRef arg;
_PyStackRef self;
_PyStackRef callable;
gh-143092: Make CALL_LIST_APPEND and BINARY_OP_INPLACE_ADD_UNICODE normal instructions (GH-143124) These super instructions need many special cases in the interpreter, specializer, and JIT. It's best we convert them to normal instructions.
2025-12-25 06:03:00 +08:00
_PyStackRef none;
gh-134584: Eliminate redundant refcounting from _CALL_LIST_APPEND (GH-142711) Signed-off-by: Manjusaka <me@manjusaka.me> Co-authored-by: Ken Jin <kenjin4096@gmail.com>
2025-12-15 23:49:36 +08:00
_PyStackRef c;
_PyStackRef s;
oparg = CURRENT_OPARG();
arg = stack_pointer[-1];
self = stack_pointer[-2];
callable = stack_pointer[-3];
assert(oparg == 1);
PyObject *self_o = PyStackRef_AsPyObjectBorrow(self);
if (!LOCK_OBJECT(self_o)) {
UOP_STAT_INC(uopcode, miss);
SET_CURRENT_CACHED_VALUES(0);
JUMP_TO_JUMP_TARGET();
}
STAT_INC(CALL, hit);
int err = _PyList_AppendTakeRef((PyListObject *)self_o, PyStackRef_AsPyObjectSteal(arg));
UNLOCK_OBJECT(self_o);
if (err) {
SET_CURRENT_CACHED_VALUES(0);
JUMP_TO_ERROR();
}
c = callable;
s = self;
gh-143092: Make CALL_LIST_APPEND and BINARY_OP_INPLACE_ADD_UNICODE normal instructions (GH-143124) These super instructions need many special cases in the interpreter, specializer, and JIT. It's best we convert them to normal instructions.
2025-12-25 06:03:00 +08:00
none = PyStackRef_None;
_tos_cache2 = s;
_tos_cache1 = c;
_tos_cache0 = none;
SET_CURRENT_CACHED_VALUES(3);
gh-134584: Eliminate redundant refcounting from _CALL_LIST_APPEND (GH-142711) Signed-off-by: Manjusaka <me@manjusaka.me> Co-authored-by: Ken Jin <kenjin4096@gmail.com>
2025-12-15 23:49:36 +08:00
stack_pointer += -3;
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-134584: Eliminate redundant refcounting from _CALL_LIST_APPEND (GH-142711) Signed-off-by: Manjusaka <me@manjusaka.me> Co-authored-by: Ken Jin <kenjin4096@gmail.com>
2025-12-15 23:49:36 +08:00
break;
}
gh-143092: Make CALL_LIST_APPEND and BINARY_OP_INPLACE_ADD_UNICODE normal instructions (GH-143124) These super instructions need many special cases in the interpreter, specializer, and JIT. It's best we convert them to normal instructions.
2025-12-25 06:03:00 +08:00
case _CALL_LIST_APPEND_r13: {
gh-134584: Eliminate redundant refcounting from _CALL_LIST_APPEND (GH-142711) Signed-off-by: Manjusaka <me@manjusaka.me> Co-authored-by: Ken Jin <kenjin4096@gmail.com>
2025-12-15 23:49:36 +08:00
CHECK_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-134584: Eliminate redundant refcounting from _CALL_LIST_APPEND (GH-142711) Signed-off-by: Manjusaka <me@manjusaka.me> Co-authored-by: Ken Jin <kenjin4096@gmail.com>
2025-12-15 23:49:36 +08:00
_PyStackRef arg;
_PyStackRef self;
_PyStackRef callable;
gh-143092: Make CALL_LIST_APPEND and BINARY_OP_INPLACE_ADD_UNICODE normal instructions (GH-143124) These super instructions need many special cases in the interpreter, specializer, and JIT. It's best we convert them to normal instructions.
2025-12-25 06:03:00 +08:00
_PyStackRef none;
gh-134584: Eliminate redundant refcounting from _CALL_LIST_APPEND (GH-142711) Signed-off-by: Manjusaka <me@manjusaka.me> Co-authored-by: Ken Jin <kenjin4096@gmail.com>
2025-12-15 23:49:36 +08:00
_PyStackRef c;
_PyStackRef s;
_PyStackRef _stack_item_0 = _tos_cache0;
oparg = CURRENT_OPARG();
arg = _stack_item_0;
self = stack_pointer[-1];
callable = stack_pointer[-2];
assert(oparg == 1);
PyObject *self_o = PyStackRef_AsPyObjectBorrow(self);
if (!LOCK_OBJECT(self_o)) {
UOP_STAT_INC(uopcode, miss);
GH-142718: JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. (GH-142762) JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. * Make sure that stack is in correct state at side exits with TOS cached values * Simplify choice of cached items for side exits
2025-12-15 19:01:29 +00:00
_tos_cache0 = arg;
gh-134584: Eliminate redundant refcounting from _CALL_LIST_APPEND (GH-142711) Signed-off-by: Manjusaka <me@manjusaka.me> Co-authored-by: Ken Jin <kenjin4096@gmail.com>
2025-12-15 23:49:36 +08:00
SET_CURRENT_CACHED_VALUES(1);
JUMP_TO_JUMP_TARGET();
}
STAT_INC(CALL, hit);
int err = _PyList_AppendTakeRef((PyListObject *)self_o, PyStackRef_AsPyObjectSteal(arg));
UNLOCK_OBJECT(self_o);
if (err) {
stack_pointer += 1;
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
SET_CURRENT_CACHED_VALUES(0);
JUMP_TO_ERROR();
}
c = callable;
s = self;
gh-143092: Make CALL_LIST_APPEND and BINARY_OP_INPLACE_ADD_UNICODE normal instructions (GH-143124) These super instructions need many special cases in the interpreter, specializer, and JIT. It's best we convert them to normal instructions.
2025-12-25 06:03:00 +08:00
none = PyStackRef_None;
_tos_cache2 = s;
_tos_cache1 = c;
_tos_cache0 = none;
SET_CURRENT_CACHED_VALUES(3);
gh-134584: Eliminate redundant refcounting from _CALL_LIST_APPEND (GH-142711) Signed-off-by: Manjusaka <me@manjusaka.me> Co-authored-by: Ken Jin <kenjin4096@gmail.com>
2025-12-15 23:49:36 +08:00
stack_pointer += -2;
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-134584: Eliminate redundant refcounting from _CALL_LIST_APPEND (GH-142711) Signed-off-by: Manjusaka <me@manjusaka.me> Co-authored-by: Ken Jin <kenjin4096@gmail.com>
2025-12-15 23:49:36 +08:00
break;
}
gh-143092: Make CALL_LIST_APPEND and BINARY_OP_INPLACE_ADD_UNICODE normal instructions (GH-143124) These super instructions need many special cases in the interpreter, specializer, and JIT. It's best we convert them to normal instructions.
2025-12-25 06:03:00 +08:00
case _CALL_LIST_APPEND_r23: {
gh-134584: Eliminate redundant refcounting from _CALL_LIST_APPEND (GH-142711) Signed-off-by: Manjusaka <me@manjusaka.me> Co-authored-by: Ken Jin <kenjin4096@gmail.com>
2025-12-15 23:49:36 +08:00
CHECK_CURRENT_CACHED_VALUES(2);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-134584: Eliminate redundant refcounting from _CALL_LIST_APPEND (GH-142711) Signed-off-by: Manjusaka <me@manjusaka.me> Co-authored-by: Ken Jin <kenjin4096@gmail.com>
2025-12-15 23:49:36 +08:00
_PyStackRef arg;
_PyStackRef self;
_PyStackRef callable;
gh-143092: Make CALL_LIST_APPEND and BINARY_OP_INPLACE_ADD_UNICODE normal instructions (GH-143124) These super instructions need many special cases in the interpreter, specializer, and JIT. It's best we convert them to normal instructions.
2025-12-25 06:03:00 +08:00
_PyStackRef none;
gh-134584: Eliminate redundant refcounting from _CALL_LIST_APPEND (GH-142711) Signed-off-by: Manjusaka <me@manjusaka.me> Co-authored-by: Ken Jin <kenjin4096@gmail.com>
2025-12-15 23:49:36 +08:00
_PyStackRef c;
_PyStackRef s;
_PyStackRef _stack_item_0 = _tos_cache0;
_PyStackRef _stack_item_1 = _tos_cache1;
oparg = CURRENT_OPARG();
arg = _stack_item_1;
self = _stack_item_0;
callable = stack_pointer[-1];
assert(oparg == 1);
PyObject *self_o = PyStackRef_AsPyObjectBorrow(self);
if (!LOCK_OBJECT(self_o)) {
UOP_STAT_INC(uopcode, miss);
GH-142718: JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. (GH-142762) JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. * Make sure that stack is in correct state at side exits with TOS cached values * Simplify choice of cached items for side exits
2025-12-15 19:01:29 +00:00
_tos_cache1 = arg;
_tos_cache0 = self;
gh-134584: Eliminate redundant refcounting from _CALL_LIST_APPEND (GH-142711) Signed-off-by: Manjusaka <me@manjusaka.me> Co-authored-by: Ken Jin <kenjin4096@gmail.com>
2025-12-15 23:49:36 +08:00
SET_CURRENT_CACHED_VALUES(2);
JUMP_TO_JUMP_TARGET();
}
STAT_INC(CALL, hit);
int err = _PyList_AppendTakeRef((PyListObject *)self_o, PyStackRef_AsPyObjectSteal(arg));
UNLOCK_OBJECT(self_o);
if (err) {
stack_pointer[0] = self;
stack_pointer += 2;
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
SET_CURRENT_CACHED_VALUES(0);
JUMP_TO_ERROR();
}
c = callable;
s = self;
gh-143092: Make CALL_LIST_APPEND and BINARY_OP_INPLACE_ADD_UNICODE normal instructions (GH-143124) These super instructions need many special cases in the interpreter, specializer, and JIT. It's best we convert them to normal instructions.
2025-12-25 06:03:00 +08:00
none = PyStackRef_None;
_tos_cache2 = s;
_tos_cache1 = c;
_tos_cache0 = none;
SET_CURRENT_CACHED_VALUES(3);
gh-134584: Eliminate redundant refcounting from _CALL_LIST_APPEND (GH-142711) Signed-off-by: Manjusaka <me@manjusaka.me> Co-authored-by: Ken Jin <kenjin4096@gmail.com>
2025-12-15 23:49:36 +08:00
stack_pointer += -1;
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-134584: Eliminate redundant refcounting from _CALL_LIST_APPEND (GH-142711) Signed-off-by: Manjusaka <me@manjusaka.me> Co-authored-by: Ken Jin <kenjin4096@gmail.com>
2025-12-15 23:49:36 +08:00
break;
}
gh-143092: Make CALL_LIST_APPEND and BINARY_OP_INPLACE_ADD_UNICODE normal instructions (GH-143124) These super instructions need many special cases in the interpreter, specializer, and JIT. It's best we convert them to normal instructions.
2025-12-25 06:03:00 +08:00
case _CALL_LIST_APPEND_r33: {
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
CHECK_CURRENT_CACHED_VALUES(3);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-118093: Add tier two support to several instructions (GH-121884)
2024-07-18 14:24:58 -07:00
_PyStackRef arg;
_PyStackRef self;
_PyStackRef callable;
gh-143092: Make CALL_LIST_APPEND and BINARY_OP_INPLACE_ADD_UNICODE normal instructions (GH-143124) These super instructions need many special cases in the interpreter, specializer, and JIT. It's best we convert them to normal instructions.
2025-12-25 06:03:00 +08:00
_PyStackRef none;
gh-134584: Eliminate redundant refcounting from _CALL_LIST_APPEND (GH-142711) Signed-off-by: Manjusaka <me@manjusaka.me> Co-authored-by: Ken Jin <kenjin4096@gmail.com>
2025-12-15 23:49:36 +08:00
_PyStackRef c;
_PyStackRef s;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef _stack_item_0 = _tos_cache0;
_PyStackRef _stack_item_1 = _tos_cache1;
_PyStackRef _stack_item_2 = _tos_cache2;
GH-118093: Add tier two support to several instructions (GH-121884)
2024-07-18 14:24:58 -07:00
oparg = CURRENT_OPARG();
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
arg = _stack_item_2;
self = _stack_item_1;
callable = _stack_item_0;
GH-118093: Add tier two support to several instructions (GH-121884)
2024-07-18 14:24:58 -07:00
assert(oparg == 1);
PyObject *self_o = PyStackRef_AsPyObjectBorrow(self);
gh-115999: Enable specialization of `CALL` instructions in free-threaded builds (#127123) The CALL family of instructions were mostly thread-safe already and only required a small number of changes, which are documented below. A few changes were needed to make CALL_ALLOC_AND_ENTER_INIT thread-safe: Added _PyType_LookupRefAndVersion, which returns the type version corresponding to the returned ref. Added _PyType_CacheInitForSpecialization, which takes an init method and the corresponding type version and only populates the specialization cache if the current type version matches the supplied version. This prevents potentially caching a stale value in free-threaded builds if we race with an update to __init__. Only cache __init__ functions that are deferred in free-threaded builds. This ensures that the reference to __init__ that is stored in the specialization cache is valid if the type version guard in _CHECK_AND_ALLOCATE_OBJECT passes. Fix a bug in _CREATE_INIT_FRAME where the frame is pushed to the stack on failure. A few other miscellaneous changes were also needed: Use {LOCK,UNLOCK}_OBJECT in LIST_APPEND. This ensures that the list's per-object lock is held while we are appending to it. Add missing co_tlbc for _Py_InitCleanup. Stop/start the world around setting the eval frame hook. This allows us to read interp->eval_frame non-atomically and preserves the behavior of _CHECK_PEP_523 documented below.
2024-12-03 11:20:20 -08:00
if (!LOCK_OBJECT(self_o)) {
UOP_STAT_INC(uopcode, miss);
GH-142718: JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. (GH-142762) JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. * Make sure that stack is in correct state at side exits with TOS cached values * Simplify choice of cached items for side exits
2025-12-15 19:01:29 +00:00
_tos_cache2 = arg;
_tos_cache1 = self;
_tos_cache0 = callable;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(3);
gh-115999: Enable specialization of `CALL` instructions in free-threaded builds (#127123) The CALL family of instructions were mostly thread-safe already and only required a small number of changes, which are documented below. A few changes were needed to make CALL_ALLOC_AND_ENTER_INIT thread-safe: Added _PyType_LookupRefAndVersion, which returns the type version corresponding to the returned ref. Added _PyType_CacheInitForSpecialization, which takes an init method and the corresponding type version and only populates the specialization cache if the current type version matches the supplied version. This prevents potentially caching a stale value in free-threaded builds if we race with an update to __init__. Only cache __init__ functions that are deferred in free-threaded builds. This ensures that the reference to __init__ that is stored in the specialization cache is valid if the type version guard in _CHECK_AND_ALLOCATE_OBJECT passes. Fix a bug in _CREATE_INIT_FRAME where the frame is pushed to the stack on failure. A few other miscellaneous changes were also needed: Use {LOCK,UNLOCK}_OBJECT in LIST_APPEND. This ensures that the list's per-object lock is held while we are appending to it. Add missing co_tlbc for _Py_InitCleanup. Stop/start the world around setting the eval frame hook. This allows us to read interp->eval_frame non-atomically and preserves the behavior of _CHECK_PEP_523 documented below.
2024-12-03 11:20:20 -08:00
JUMP_TO_JUMP_TARGET();
}
GH-118093: Add tier two support to several instructions (GH-121884)
2024-07-18 14:24:58 -07:00
STAT_INC(CALL, hit);
int err = _PyList_AppendTakeRef((PyListObject *)self_o, PyStackRef_AsPyObjectSteal(arg));
gh-115999: Enable specialization of `CALL` instructions in free-threaded builds (#127123) The CALL family of instructions were mostly thread-safe already and only required a small number of changes, which are documented below. A few changes were needed to make CALL_ALLOC_AND_ENTER_INIT thread-safe: Added _PyType_LookupRefAndVersion, which returns the type version corresponding to the returned ref. Added _PyType_CacheInitForSpecialization, which takes an init method and the corresponding type version and only populates the specialization cache if the current type version matches the supplied version. This prevents potentially caching a stale value in free-threaded builds if we race with an update to __init__. Only cache __init__ functions that are deferred in free-threaded builds. This ensures that the reference to __init__ that is stored in the specialization cache is valid if the type version guard in _CHECK_AND_ALLOCATE_OBJECT passes. Fix a bug in _CREATE_INIT_FRAME where the frame is pushed to the stack on failure. A few other miscellaneous changes were also needed: Use {LOCK,UNLOCK}_OBJECT in LIST_APPEND. This ensures that the list's per-object lock is held while we are appending to it. Add missing co_tlbc for _Py_InitCleanup. Stop/start the world around setting the eval frame hook. This allows us to read interp->eval_frame non-atomically and preserves the behavior of _CHECK_PEP_523 documented below.
2024-12-03 11:20:20 -08:00
UNLOCK_OBJECT(self_o);
GH-128682: Make `PyStackRef_CLOSE` escaping. (GH-129404)
2025-02-03 12:41:32 +00:00
if (err) {
gh-134584: Eliminate redundant refcounting from _CALL_LIST_APPEND (GH-142711) Signed-off-by: Manjusaka <me@manjusaka.me> Co-authored-by: Ken Jin <kenjin4096@gmail.com>
2025-12-15 23:49:36 +08:00
stack_pointer[0] = callable;
stack_pointer[1] = self;
stack_pointer += 3;
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(0);
GH-128682: Make `PyStackRef_CLOSE` escaping. (GH-129404)
2025-02-03 12:41:32 +00:00
JUMP_TO_ERROR();
}
gh-134584: Eliminate redundant refcounting from _CALL_LIST_APPEND (GH-142711) Signed-off-by: Manjusaka <me@manjusaka.me> Co-authored-by: Ken Jin <kenjin4096@gmail.com>
2025-12-15 23:49:36 +08:00
c = callable;
s = self;
gh-143092: Make CALL_LIST_APPEND and BINARY_OP_INPLACE_ADD_UNICODE normal instructions (GH-143124) These super instructions need many special cases in the interpreter, specializer, and JIT. It's best we convert them to normal instructions.
2025-12-25 06:03:00 +08:00
none = PyStackRef_None;
_tos_cache2 = s;
_tos_cache1 = c;
_tos_cache0 = none;
SET_CURRENT_CACHED_VALUES(3);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-118093: Add tier two support to several instructions (GH-121884)
2024-07-18 14:24:58 -07:00
break;
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
case _CALL_METHOD_DESCRIPTOR_O_r01: {
CHECK_CURRENT_CACHED_VALUES(0);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
_PyStackRef *args;
GH-131498: Replace single-element arrays with scalars in bytecodes.c (GH-132615)
2025-04-18 07:16:28 -07:00
_PyStackRef self_or_null;
_PyStackRef callable;
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
_PyStackRef res;
GH-111485: Factor out tier 2 code generation from the rest of the interpreter code generator (GH-112968)
2023-12-12 12:12:17 +00:00
oparg = CURRENT_OPARG();
args = &stack_pointer[-oparg];
GH-131498: Replace single-element arrays with scalars in bytecodes.c (GH-132615)
2025-04-18 07:16:28 -07:00
self_or_null = stack_pointer[-1 - oparg];
callable = stack_pointer[-2 - oparg];
PyObject *callable_o = PyStackRef_AsPyObjectBorrow(callable);
gh-106581: Add 10 new opcodes by allowing `assert(kwnames == NULL)` (#106707) By turning `assert(kwnames == NULL)` into a macro that is not in the "forbidden" list, many instructions that formerly were skipped because they contained such an assert (but no other mention of `kwnames`) are now supported in Tier 2. This covers 10 instructions in total (all specializations of `CALL` that invoke some C code): - `CALL_NO_KW_TYPE_1` - `CALL_NO_KW_STR_1` - `CALL_NO_KW_TUPLE_1` - `CALL_NO_KW_BUILTIN_O` - `CALL_NO_KW_BUILTIN_FAST` - `CALL_NO_KW_LEN` - `CALL_NO_KW_ISINSTANCE` - `CALL_NO_KW_METHOD_DESCRIPTOR_O` - `CALL_NO_KW_METHOD_DESCRIPTOR_NOARGS` - `CALL_NO_KW_METHOD_DESCRIPTOR_FAST`
2023-07-17 11:02:58 -07:00
int total_args = oparg;
GH-128682: Make `PyStackRef_CLOSE` escaping. (GH-129404)
2025-02-03 12:41:32 +00:00
_PyStackRef *arguments = args;
GH-131498: Replace single-element arrays with scalars in bytecodes.c (GH-132615)
2025-04-18 07:16:28 -07:00
if (!PyStackRef_IsNull(self_or_null)) {
GH-128682: Make `PyStackRef_CLOSE` escaping. (GH-129404)
2025-02-03 12:41:32 +00:00
arguments--;
gh-106581: Add 10 new opcodes by allowing `assert(kwnames == NULL)` (#106707) By turning `assert(kwnames == NULL)` into a macro that is not in the "forbidden" list, many instructions that formerly were skipped because they contained such an assert (but no other mention of `kwnames`) are now supported in Tier 2. This covers 10 instructions in total (all specializations of `CALL` that invoke some C code): - `CALL_NO_KW_TYPE_1` - `CALL_NO_KW_STR_1` - `CALL_NO_KW_TUPLE_1` - `CALL_NO_KW_BUILTIN_O` - `CALL_NO_KW_BUILTIN_FAST` - `CALL_NO_KW_LEN` - `CALL_NO_KW_ISINSTANCE` - `CALL_NO_KW_METHOD_DESCRIPTOR_O` - `CALL_NO_KW_METHOD_DESCRIPTOR_NOARGS` - `CALL_NO_KW_METHOD_DESCRIPTOR_FAST`
2023-07-17 11:02:58 -07:00
total_args++;
}
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
PyMethodDescrObject *method = (PyMethodDescrObject *)callable_o;
GH-117457: Correct pystats uop "miss" counts (GH-117477)
2024-04-04 18:49:18 -04:00
if (total_args != 2) {
UOP_STAT_INC(uopcode, miss);
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(0);
GH-117457: Correct pystats uop "miss" counts (GH-117477)
2024-04-04 18:49:18 -04:00
JUMP_TO_JUMP_TARGET();
}
if (!Py_IS_TYPE(method, &PyMethodDescr_Type)) {
UOP_STAT_INC(uopcode, miss);
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(0);
GH-117457: Correct pystats uop "miss" counts (GH-117477)
2024-04-04 18:49:18 -04:00
JUMP_TO_JUMP_TARGET();
}
GH-105848: Simplify the arrangement of CALL's stack (GH-107788)
2023-08-09 11:19:39 -07:00
PyMethodDef *meth = method->d_method;
GH-117457: Correct pystats uop "miss" counts (GH-117477)
2024-04-04 18:49:18 -04:00
if (meth->ml_flags != METH_O) {
UOP_STAT_INC(uopcode, miss);
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(0);
GH-117457: Correct pystats uop "miss" counts (GH-117477)
2024-04-04 18:49:18 -04:00
JUMP_TO_JUMP_TARGET();
}
GH-130396: Use computed stack limits on linux (GH-130398) * Implement C recursion protection with limit pointers for Linux, MacOS and Windows * Remove calls to PyOS_CheckStack * Add stack protection to parser * Make tests more robust to low stacks * Improve error messages for stack overflow
2025-02-25 09:24:48 +00:00
if (_Py_ReachedRecursionLimit(tstate)) {
GH-117457: Correct pystats uop "miss" counts (GH-117477)
2024-04-04 18:49:18 -04:00
UOP_STAT_INC(uopcode, miss);
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(0);
GH-117457: Correct pystats uop "miss" counts (GH-117477)
2024-04-04 18:49:18 -04:00
JUMP_TO_JUMP_TARGET();
}
GH-128682: Make `PyStackRef_CLOSE` escaping. (GH-129404)
2025-02-03 12:41:32 +00:00
_PyStackRef arg_stackref = arguments[1];
_PyStackRef self_stackref = arguments[0];
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
if (!Py_IS_TYPE(PyStackRef_AsPyObjectBorrow(self_stackref),
GH-118093: Turn some DEOPT_IFs into EXIT_IFs (GH-122998)
2024-08-14 07:54:42 -07:00
method->d_common.d_type)) {
GH-117457: Correct pystats uop "miss" counts (GH-117477)
2024-04-04 18:49:18 -04:00
UOP_STAT_INC(uopcode, miss);
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(0);
GH-117457: Correct pystats uop "miss" counts (GH-117477)
2024-04-04 18:49:18 -04:00
JUMP_TO_JUMP_TARGET();
}
gh-106581: Add 10 new opcodes by allowing `assert(kwnames == NULL)` (#106707) By turning `assert(kwnames == NULL)` into a macro that is not in the "forbidden" list, many instructions that formerly were skipped because they contained such an assert (but no other mention of `kwnames`) are now supported in Tier 2. This covers 10 instructions in total (all specializations of `CALL` that invoke some C code): - `CALL_NO_KW_TYPE_1` - `CALL_NO_KW_STR_1` - `CALL_NO_KW_TUPLE_1` - `CALL_NO_KW_BUILTIN_O` - `CALL_NO_KW_BUILTIN_FAST` - `CALL_NO_KW_LEN` - `CALL_NO_KW_ISINSTANCE` - `CALL_NO_KW_METHOD_DESCRIPTOR_O` - `CALL_NO_KW_METHOD_DESCRIPTOR_NOARGS` - `CALL_NO_KW_METHOD_DESCRIPTOR_FAST`
2023-07-17 11:02:58 -07:00
STAT_INC(CALL, hit);
PyCFunction cfunc = meth->ml_meth;
GH-119866: Spill the stack around escaping calls. (GH-124392) * Spill the evaluation around escaping calls in the generated interpreter and JIT. * The code generator tracks live, cached values so they can be saved to memory when needed. * Spills the stack pointer around escaping calls, so that the exact stack is visible to the cycle GC.
2024-10-07 14:56:39 +01:00
_PyFrame_SetStackPointer(frame, stack_pointer);
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
PyObject *res_o = _PyCFunction_TrampolineCall(cfunc,
PyStackRef_AsPyObjectBorrow(self_stackref),
PyStackRef_AsPyObjectBorrow(arg_stackref));
GH-119866: Spill the stack around escaping calls. (GH-124392) * Spill the evaluation around escaping calls in the generated interpreter and JIT. * The code generator tracks live, cached values so they can be saved to memory when needed. * Spills the stack pointer around escaping calls, so that the exact stack is visible to the cycle GC.
2024-10-07 14:56:39 +01:00
stack_pointer = _PyFrame_GetStackPointer(frame);
gh-106581: Add 10 new opcodes by allowing `assert(kwnames == NULL)` (#106707) By turning `assert(kwnames == NULL)` into a macro that is not in the "forbidden" list, many instructions that formerly were skipped because they contained such an assert (but no other mention of `kwnames`) are now supported in Tier 2. This covers 10 instructions in total (all specializations of `CALL` that invoke some C code): - `CALL_NO_KW_TYPE_1` - `CALL_NO_KW_STR_1` - `CALL_NO_KW_TUPLE_1` - `CALL_NO_KW_BUILTIN_O` - `CALL_NO_KW_BUILTIN_FAST` - `CALL_NO_KW_LEN` - `CALL_NO_KW_ISINSTANCE` - `CALL_NO_KW_METHOD_DESCRIPTOR_O` - `CALL_NO_KW_METHOD_DESCRIPTOR_NOARGS` - `CALL_NO_KW_METHOD_DESCRIPTOR_FAST`
2023-07-17 11:02:58 -07:00
_Py_LeaveRecursiveCallTstate(tstate);
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
assert((res_o != NULL) ^ (_PyErr_Occurred(tstate) != NULL));
GH-128682: Account for escapes in `DECREF_INPUTS` (GH-129953) * Handle escapes in DECREF_INPUTS * Mark a few more functions as escaping * Replace DECREF_INPUTS with PyStackRef_CLOSE where possible
2025-02-12 17:44:59 +00:00
_PyFrame_SetStackPointer(frame, stack_pointer);
_PyStackRef tmp;
GH-128682: Make `PyStackRef_CLOSE` escaping. (GH-129404)
2025-02-03 12:41:32 +00:00
for (int _i = oparg; --_i >= 0;) {
GH-128682: Account for escapes in `DECREF_INPUTS` (GH-129953) * Handle escapes in DECREF_INPUTS * Mark a few more functions as escaping * Replace DECREF_INPUTS with PyStackRef_CLOSE where possible
2025-02-12 17:44:59 +00:00
tmp = args[_i];
args[_i] = PyStackRef_NULL;
PyStackRef_CLOSE(tmp);
GH-128682: Make `PyStackRef_CLOSE` escaping. (GH-129404)
2025-02-03 12:41:32 +00:00
}
GH-131498: Replace single-element arrays with scalars in bytecodes.c (GH-132615)
2025-04-18 07:16:28 -07:00
tmp = self_or_null;
self_or_null = PyStackRef_NULL;
stack_pointer[-1 - oparg] = self_or_null;
GH-128682: Account for escapes in `DECREF_INPUTS` (GH-129953) * Handle escapes in DECREF_INPUTS * Mark a few more functions as escaping * Replace DECREF_INPUTS with PyStackRef_CLOSE where possible
2025-02-12 17:44:59 +00:00
PyStackRef_XCLOSE(tmp);
GH-131498: Replace single-element arrays with scalars in bytecodes.c (GH-132615)
2025-04-18 07:16:28 -07:00
tmp = callable;
callable = PyStackRef_NULL;
stack_pointer[-2 - oparg] = callable;
GH-128682: Account for escapes in `DECREF_INPUTS` (GH-129953) * Handle escapes in DECREF_INPUTS * Mark a few more functions as escaping * Replace DECREF_INPUTS with PyStackRef_CLOSE where possible
2025-02-12 17:44:59 +00:00
PyStackRef_CLOSE(tmp);
stack_pointer = _PyFrame_GetStackPointer(frame);
stack_pointer += -2 - oparg;
GH-141794: Limit size of generated machine code. (GH-142228) * Factor out bodies of the largest uops, to reduce jit code size. * Factor out common assert, also reducing jit code size. * Limit size of jitted code for a single executor to 1MB.
2025-12-03 17:43:35 +00:00
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
GH-128682: Make `PyStackRef_CLOSE` escaping. (GH-129404)
2025-02-03 12:41:32 +00:00
if (res_o == NULL) {
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(0);
GH-128682: Make `PyStackRef_CLOSE` escaping. (GH-129404)
2025-02-03 12:41:32 +00:00
JUMP_TO_ERROR();
}
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
res = PyStackRef_FromPyObjectSteal(res_o);
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_tos_cache0 = res;
_tos_cache1 = PyStackRef_ZERO_BITS;
_tos_cache2 = PyStackRef_ZERO_BITS;
SET_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-106581: Add 10 new opcodes by allowing `assert(kwnames == NULL)` (#106707) By turning `assert(kwnames == NULL)` into a macro that is not in the "forbidden" list, many instructions that formerly were skipped because they contained such an assert (but no other mention of `kwnames`) are now supported in Tier 2. This covers 10 instructions in total (all specializations of `CALL` that invoke some C code): - `CALL_NO_KW_TYPE_1` - `CALL_NO_KW_STR_1` - `CALL_NO_KW_TUPLE_1` - `CALL_NO_KW_BUILTIN_O` - `CALL_NO_KW_BUILTIN_FAST` - `CALL_NO_KW_LEN` - `CALL_NO_KW_ISINSTANCE` - `CALL_NO_KW_METHOD_DESCRIPTOR_O` - `CALL_NO_KW_METHOD_DESCRIPTOR_NOARGS` - `CALL_NO_KW_METHOD_DESCRIPTOR_FAST`
2023-07-17 11:02:58 -07:00
break;
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
case _CALL_METHOD_DESCRIPTOR_FAST_WITH_KEYWORDS_r01: {
CHECK_CURRENT_CACHED_VALUES(0);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
_PyStackRef *args;
GH-131498: Replace single-element arrays with scalars in bytecodes.c (GH-132615)
2025-04-18 07:16:28 -07:00
_PyStackRef self_or_null;
_PyStackRef callable;
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
_PyStackRef res;
GH-111485: Factor out tier 2 code generation from the rest of the interpreter code generator (GH-112968)
2023-12-12 12:12:17 +00:00
oparg = CURRENT_OPARG();
args = &stack_pointer[-oparg];
GH-131498: Replace single-element arrays with scalars in bytecodes.c (GH-132615)
2025-04-18 07:16:28 -07:00
self_or_null = stack_pointer[-1 - oparg];
callable = stack_pointer[-2 - oparg];
PyObject *callable_o = PyStackRef_AsPyObjectBorrow(callable);
GH-105848: Replace KW_NAMES + CALL with LOAD_CONST + CALL_KW (GH-109300)
2023-09-13 10:25:45 -07:00
int total_args = oparg;
GH-128682: Convert explicit loops closing arrays into `DECREF_INPUTS`. (GH-128822) * Mark Py_DECREF and Py_XDECREF as escaping * Remove explicit loops for clearing array inputs
2025-01-14 15:08:56 +00:00
_PyStackRef *arguments = args;
GH-131498: Replace single-element arrays with scalars in bytecodes.c (GH-132615)
2025-04-18 07:16:28 -07:00
if (!PyStackRef_IsNull(self_or_null)) {
GH-128682: Convert explicit loops closing arrays into `DECREF_INPUTS`. (GH-128822) * Mark Py_DECREF and Py_XDECREF as escaping * Remove explicit loops for clearing array inputs
2025-01-14 15:08:56 +00:00
arguments--;
GH-105848: Replace KW_NAMES + CALL with LOAD_CONST + CALL_KW (GH-109300)
2023-09-13 10:25:45 -07:00
total_args++;
}
gh-131998: Fix `NULL` dereference when using an unbound method descriptor in a specialized code path (#132000) Co-authored-by: sobolevn <mail@sobolevn.me> Co-authored-by: Victor Stinner <vstinner@python.org> Co-authored-by: Mark Shannon <mark@hotpy.org>
2025-04-08 06:31:43 -04:00
if (total_args == 0) {
UOP_STAT_INC(uopcode, miss);
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(0);
gh-131998: Fix `NULL` dereference when using an unbound method descriptor in a specialized code path (#132000) Co-authored-by: sobolevn <mail@sobolevn.me> Co-authored-by: Victor Stinner <vstinner@python.org> Co-authored-by: Mark Shannon <mark@hotpy.org>
2025-04-08 06:31:43 -04:00
JUMP_TO_JUMP_TARGET();
}
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
PyMethodDescrObject *method = (PyMethodDescrObject *)callable_o;
GH-117457: Correct pystats uop "miss" counts (GH-117477)
2024-04-04 18:49:18 -04:00
if (!Py_IS_TYPE(method, &PyMethodDescr_Type)) {
UOP_STAT_INC(uopcode, miss);
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(0);
GH-117457: Correct pystats uop "miss" counts (GH-117477)
2024-04-04 18:49:18 -04:00
JUMP_TO_JUMP_TARGET();
}
GH-105848: Replace KW_NAMES + CALL with LOAD_CONST + CALL_KW (GH-109300)
2023-09-13 10:25:45 -07:00
PyMethodDef *meth = method->d_method;
GH-117457: Correct pystats uop "miss" counts (GH-117477)
2024-04-04 18:49:18 -04:00
if (meth->ml_flags != (METH_FASTCALL|METH_KEYWORDS)) {
UOP_STAT_INC(uopcode, miss);
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(0);
GH-117457: Correct pystats uop "miss" counts (GH-117477)
2024-04-04 18:49:18 -04:00
JUMP_TO_JUMP_TARGET();
}
GH-105848: Replace KW_NAMES + CALL with LOAD_CONST + CALL_KW (GH-109300)
2023-09-13 10:25:45 -07:00
PyTypeObject *d_type = method->d_common.d_type;
GH-128682: Convert explicit loops closing arrays into `DECREF_INPUTS`. (GH-128822) * Mark Py_DECREF and Py_XDECREF as escaping * Remove explicit loops for clearing array inputs
2025-01-14 15:08:56 +00:00
PyObject *self = PyStackRef_AsPyObjectBorrow(arguments[0]);
gh-131998: Fix `NULL` dereference when using an unbound method descriptor in a specialized code path (#132000) Co-authored-by: sobolevn <mail@sobolevn.me> Co-authored-by: Victor Stinner <vstinner@python.org> Co-authored-by: Mark Shannon <mark@hotpy.org>
2025-04-08 06:31:43 -04:00
assert(self != NULL);
GH-117457: Correct pystats uop "miss" counts (GH-117477)
2024-04-04 18:49:18 -04:00
if (!Py_IS_TYPE(self, d_type)) {
UOP_STAT_INC(uopcode, miss);
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(0);
GH-117457: Correct pystats uop "miss" counts (GH-117477)
2024-04-04 18:49:18 -04:00
JUMP_TO_JUMP_TARGET();
}
GH-105848: Replace KW_NAMES + CALL with LOAD_CONST + CALL_KW (GH-109300)
2023-09-13 10:25:45 -07:00
STAT_INC(CALL, hit);
GH-119866: Spill the stack around escaping calls. (GH-124392) * Spill the evaluation around escaping calls in the generated interpreter and JIT. * The code generator tracks live, cached values so they can be saved to memory when needed. * Spills the stack pointer around escaping calls, so that the exact stack is visible to the cycle GC.
2024-10-07 14:56:39 +01:00
_PyFrame_SetStackPointer(frame, stack_pointer);
GH-141794: Limit size of generated machine code. (GH-142228) * Factor out bodies of the largest uops, to reduce jit code size. * Factor out common assert, also reducing jit code size. * Limit size of jitted code for a single executor to 1MB.
2025-12-03 17:43:35 +00:00
PyObject *res_o = _PyCallMethodDescriptorFastWithKeywords_StackRefSteal(
callable,
meth,
self,
arguments,
total_args
);
GH-128682: Account for escapes in `DECREF_INPUTS` (GH-129953) * Handle escapes in DECREF_INPUTS * Mark a few more functions as escaping * Replace DECREF_INPUTS with PyStackRef_CLOSE where possible
2025-02-12 17:44:59 +00:00
stack_pointer = _PyFrame_GetStackPointer(frame);
GH-128682: Make `PyStackRef_CLOSE` escaping. (GH-129404)
2025-02-03 12:41:32 +00:00
if (res_o == NULL) {
GH-141794: Limit size of generated machine code. (GH-142228) * Factor out bodies of the largest uops, to reduce jit code size. * Factor out common assert, also reducing jit code size. * Limit size of jitted code for a single executor to 1MB.
2025-12-03 17:43:35 +00:00
stack_pointer += -2 - oparg;
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(0);
GH-128682: Make `PyStackRef_CLOSE` escaping. (GH-129404)
2025-02-03 12:41:32 +00:00
JUMP_TO_ERROR();
}
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
res = PyStackRef_FromPyObjectSteal(res_o);
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_tos_cache0 = res;
_tos_cache1 = PyStackRef_ZERO_BITS;
_tos_cache2 = PyStackRef_ZERO_BITS;
SET_CURRENT_CACHED_VALUES(1);
stack_pointer += -2 - oparg;
GH-141794: Limit size of generated machine code. (GH-142228) * Factor out bodies of the largest uops, to reduce jit code size. * Factor out common assert, also reducing jit code size. * Limit size of jitted code for a single executor to 1MB.
2025-12-03 17:43:35 +00:00
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-105848: Replace KW_NAMES + CALL with LOAD_CONST + CALL_KW (GH-109300)
2023-09-13 10:25:45 -07:00
break;
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
case _CALL_METHOD_DESCRIPTOR_NOARGS_r01: {
CHECK_CURRENT_CACHED_VALUES(0);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
_PyStackRef *args;
GH-131498: Replace single-element arrays with scalars in bytecodes.c (GH-132615)
2025-04-18 07:16:28 -07:00
_PyStackRef self_or_null;
_PyStackRef callable;
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
_PyStackRef res;
GH-111485: Factor out tier 2 code generation from the rest of the interpreter code generator (GH-112968)
2023-12-12 12:12:17 +00:00
oparg = CURRENT_OPARG();
args = &stack_pointer[-oparg];
GH-131498: Replace single-element arrays with scalars in bytecodes.c (GH-132615)
2025-04-18 07:16:28 -07:00
self_or_null = stack_pointer[-1 - oparg];
callable = stack_pointer[-2 - oparg];
gh-106581: Add 10 new opcodes by allowing `assert(kwnames == NULL)` (#106707) By turning `assert(kwnames == NULL)` into a macro that is not in the "forbidden" list, many instructions that formerly were skipped because they contained such an assert (but no other mention of `kwnames`) are now supported in Tier 2. This covers 10 instructions in total (all specializations of `CALL` that invoke some C code): - `CALL_NO_KW_TYPE_1` - `CALL_NO_KW_STR_1` - `CALL_NO_KW_TUPLE_1` - `CALL_NO_KW_BUILTIN_O` - `CALL_NO_KW_BUILTIN_FAST` - `CALL_NO_KW_LEN` - `CALL_NO_KW_ISINSTANCE` - `CALL_NO_KW_METHOD_DESCRIPTOR_O` - `CALL_NO_KW_METHOD_DESCRIPTOR_NOARGS` - `CALL_NO_KW_METHOD_DESCRIPTOR_FAST`
2023-07-17 11:02:58 -07:00
assert(oparg == 0 || oparg == 1);
GH-131498: Replace single-element arrays with scalars in bytecodes.c (GH-132615)
2025-04-18 07:16:28 -07:00
PyObject *callable_o = PyStackRef_AsPyObjectBorrow(callable);
gh-106581: Add 10 new opcodes by allowing `assert(kwnames == NULL)` (#106707) By turning `assert(kwnames == NULL)` into a macro that is not in the "forbidden" list, many instructions that formerly were skipped because they contained such an assert (but no other mention of `kwnames`) are now supported in Tier 2. This covers 10 instructions in total (all specializations of `CALL` that invoke some C code): - `CALL_NO_KW_TYPE_1` - `CALL_NO_KW_STR_1` - `CALL_NO_KW_TUPLE_1` - `CALL_NO_KW_BUILTIN_O` - `CALL_NO_KW_BUILTIN_FAST` - `CALL_NO_KW_LEN` - `CALL_NO_KW_ISINSTANCE` - `CALL_NO_KW_METHOD_DESCRIPTOR_O` - `CALL_NO_KW_METHOD_DESCRIPTOR_NOARGS` - `CALL_NO_KW_METHOD_DESCRIPTOR_FAST`
2023-07-17 11:02:58 -07:00
int total_args = oparg;
GH-131498: Replace single-element arrays with scalars in bytecodes.c (GH-132615)
2025-04-18 07:16:28 -07:00
if (!PyStackRef_IsNull(self_or_null)) {
gh-106581: Add 10 new opcodes by allowing `assert(kwnames == NULL)` (#106707) By turning `assert(kwnames == NULL)` into a macro that is not in the "forbidden" list, many instructions that formerly were skipped because they contained such an assert (but no other mention of `kwnames`) are now supported in Tier 2. This covers 10 instructions in total (all specializations of `CALL` that invoke some C code): - `CALL_NO_KW_TYPE_1` - `CALL_NO_KW_STR_1` - `CALL_NO_KW_TUPLE_1` - `CALL_NO_KW_BUILTIN_O` - `CALL_NO_KW_BUILTIN_FAST` - `CALL_NO_KW_LEN` - `CALL_NO_KW_ISINSTANCE` - `CALL_NO_KW_METHOD_DESCRIPTOR_O` - `CALL_NO_KW_METHOD_DESCRIPTOR_NOARGS` - `CALL_NO_KW_METHOD_DESCRIPTOR_FAST`
2023-07-17 11:02:58 -07:00
args--;
total_args++;
}
GH-117457: Correct pystats uop "miss" counts (GH-117477)
2024-04-04 18:49:18 -04:00
if (total_args != 1) {
UOP_STAT_INC(uopcode, miss);
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(0);
GH-117457: Correct pystats uop "miss" counts (GH-117477)
2024-04-04 18:49:18 -04:00
JUMP_TO_JUMP_TARGET();
}
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
PyMethodDescrObject *method = (PyMethodDescrObject *)callable_o;
GH-117457: Correct pystats uop "miss" counts (GH-117477)
2024-04-04 18:49:18 -04:00
if (!Py_IS_TYPE(method, &PyMethodDescr_Type)) {
UOP_STAT_INC(uopcode, miss);
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(0);
GH-117457: Correct pystats uop "miss" counts (GH-117477)
2024-04-04 18:49:18 -04:00
JUMP_TO_JUMP_TARGET();
}
GH-105848: Simplify the arrangement of CALL's stack (GH-107788)
2023-08-09 11:19:39 -07:00
PyMethodDef *meth = method->d_method;
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
_PyStackRef self_stackref = args[0];
PyObject *self = PyStackRef_AsPyObjectBorrow(self_stackref);
GH-117457: Correct pystats uop "miss" counts (GH-117477)
2024-04-04 18:49:18 -04:00
if (!Py_IS_TYPE(self, method->d_common.d_type)) {
UOP_STAT_INC(uopcode, miss);
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(0);
GH-117457: Correct pystats uop "miss" counts (GH-117477)
2024-04-04 18:49:18 -04:00
JUMP_TO_JUMP_TARGET();
}
if (meth->ml_flags != METH_NOARGS) {
UOP_STAT_INC(uopcode, miss);
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(0);
GH-117457: Correct pystats uop "miss" counts (GH-117477)
2024-04-04 18:49:18 -04:00
JUMP_TO_JUMP_TARGET();
}
GH-130396: Use computed stack limits on linux (GH-130398) * Implement C recursion protection with limit pointers for Linux, MacOS and Windows * Remove calls to PyOS_CheckStack * Add stack protection to parser * Make tests more robust to low stacks * Improve error messages for stack overflow
2025-02-25 09:24:48 +00:00
if (_Py_ReachedRecursionLimit(tstate)) {
GH-117457: Correct pystats uop "miss" counts (GH-117477)
2024-04-04 18:49:18 -04:00
UOP_STAT_INC(uopcode, miss);
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(0);
GH-117457: Correct pystats uop "miss" counts (GH-117477)
2024-04-04 18:49:18 -04:00
JUMP_TO_JUMP_TARGET();
}
gh-106581: Add 10 new opcodes by allowing `assert(kwnames == NULL)` (#106707) By turning `assert(kwnames == NULL)` into a macro that is not in the "forbidden" list, many instructions that formerly were skipped because they contained such an assert (but no other mention of `kwnames`) are now supported in Tier 2. This covers 10 instructions in total (all specializations of `CALL` that invoke some C code): - `CALL_NO_KW_TYPE_1` - `CALL_NO_KW_STR_1` - `CALL_NO_KW_TUPLE_1` - `CALL_NO_KW_BUILTIN_O` - `CALL_NO_KW_BUILTIN_FAST` - `CALL_NO_KW_LEN` - `CALL_NO_KW_ISINSTANCE` - `CALL_NO_KW_METHOD_DESCRIPTOR_O` - `CALL_NO_KW_METHOD_DESCRIPTOR_NOARGS` - `CALL_NO_KW_METHOD_DESCRIPTOR_FAST`
2023-07-17 11:02:58 -07:00
STAT_INC(CALL, hit);
PyCFunction cfunc = meth->ml_meth;
GH-119866: Spill the stack around escaping calls. (GH-124392) * Spill the evaluation around escaping calls in the generated interpreter and JIT. * The code generator tracks live, cached values so they can be saved to memory when needed. * Spills the stack pointer around escaping calls, so that the exact stack is visible to the cycle GC.
2024-10-07 14:56:39 +01:00
_PyFrame_SetStackPointer(frame, stack_pointer);
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
PyObject *res_o = _PyCFunction_TrampolineCall(cfunc, self, NULL);
GH-119866: Spill the stack around escaping calls. (GH-124392) * Spill the evaluation around escaping calls in the generated interpreter and JIT. * The code generator tracks live, cached values so they can be saved to memory when needed. * Spills the stack pointer around escaping calls, so that the exact stack is visible to the cycle GC.
2024-10-07 14:56:39 +01:00
stack_pointer = _PyFrame_GetStackPointer(frame);
gh-106581: Add 10 new opcodes by allowing `assert(kwnames == NULL)` (#106707) By turning `assert(kwnames == NULL)` into a macro that is not in the "forbidden" list, many instructions that formerly were skipped because they contained such an assert (but no other mention of `kwnames`) are now supported in Tier 2. This covers 10 instructions in total (all specializations of `CALL` that invoke some C code): - `CALL_NO_KW_TYPE_1` - `CALL_NO_KW_STR_1` - `CALL_NO_KW_TUPLE_1` - `CALL_NO_KW_BUILTIN_O` - `CALL_NO_KW_BUILTIN_FAST` - `CALL_NO_KW_LEN` - `CALL_NO_KW_ISINSTANCE` - `CALL_NO_KW_METHOD_DESCRIPTOR_O` - `CALL_NO_KW_METHOD_DESCRIPTOR_NOARGS` - `CALL_NO_KW_METHOD_DESCRIPTOR_FAST`
2023-07-17 11:02:58 -07:00
_Py_LeaveRecursiveCallTstate(tstate);
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
assert((res_o != NULL) ^ (_PyErr_Occurred(tstate) != NULL));
GH-128682: Make `PyStackRef_CLOSE` escaping. (GH-129404)
2025-02-03 12:41:32 +00:00
_PyFrame_SetStackPointer(frame, stack_pointer);
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
PyStackRef_CLOSE(self_stackref);
GH-128682: Make `PyStackRef_CLOSE` escaping. (GH-129404)
2025-02-03 12:41:32 +00:00
stack_pointer = _PyFrame_GetStackPointer(frame);
stack_pointer += -2 - oparg;
GH-141794: Limit size of generated machine code. (GH-142228) * Factor out bodies of the largest uops, to reduce jit code size. * Factor out common assert, also reducing jit code size. * Limit size of jitted code for a single executor to 1MB.
2025-12-03 17:43:35 +00:00
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
GH-128682: Make `PyStackRef_CLOSE` escaping. (GH-129404)
2025-02-03 12:41:32 +00:00
_PyFrame_SetStackPointer(frame, stack_pointer);
GH-131498: Replace single-element arrays with scalars in bytecodes.c (GH-132615)
2025-04-18 07:16:28 -07:00
PyStackRef_CLOSE(callable);
GH-128682: Make `PyStackRef_CLOSE` escaping. (GH-129404)
2025-02-03 12:41:32 +00:00
stack_pointer = _PyFrame_GetStackPointer(frame);
if (res_o == NULL) {
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(0);
GH-128682: Make `PyStackRef_CLOSE` escaping. (GH-129404)
2025-02-03 12:41:32 +00:00
JUMP_TO_ERROR();
}
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
res = PyStackRef_FromPyObjectSteal(res_o);
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_tos_cache0 = res;
_tos_cache1 = PyStackRef_ZERO_BITS;
_tos_cache2 = PyStackRef_ZERO_BITS;
SET_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-106581: Add 10 new opcodes by allowing `assert(kwnames == NULL)` (#106707) By turning `assert(kwnames == NULL)` into a macro that is not in the "forbidden" list, many instructions that formerly were skipped because they contained such an assert (but no other mention of `kwnames`) are now supported in Tier 2. This covers 10 instructions in total (all specializations of `CALL` that invoke some C code): - `CALL_NO_KW_TYPE_1` - `CALL_NO_KW_STR_1` - `CALL_NO_KW_TUPLE_1` - `CALL_NO_KW_BUILTIN_O` - `CALL_NO_KW_BUILTIN_FAST` - `CALL_NO_KW_LEN` - `CALL_NO_KW_ISINSTANCE` - `CALL_NO_KW_METHOD_DESCRIPTOR_O` - `CALL_NO_KW_METHOD_DESCRIPTOR_NOARGS` - `CALL_NO_KW_METHOD_DESCRIPTOR_FAST`
2023-07-17 11:02:58 -07:00
break;
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
case _CALL_METHOD_DESCRIPTOR_FAST_r01: {
CHECK_CURRENT_CACHED_VALUES(0);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
_PyStackRef *args;
GH-131498: Replace single-element arrays with scalars in bytecodes.c (GH-132615)
2025-04-18 07:16:28 -07:00
_PyStackRef self_or_null;
_PyStackRef callable;
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
_PyStackRef res;
GH-111485: Factor out tier 2 code generation from the rest of the interpreter code generator (GH-112968)
2023-12-12 12:12:17 +00:00
oparg = CURRENT_OPARG();
args = &stack_pointer[-oparg];
GH-131498: Replace single-element arrays with scalars in bytecodes.c (GH-132615)
2025-04-18 07:16:28 -07:00
self_or_null = stack_pointer[-1 - oparg];
callable = stack_pointer[-2 - oparg];
PyObject *callable_o = PyStackRef_AsPyObjectBorrow(callable);
gh-106581: Add 10 new opcodes by allowing `assert(kwnames == NULL)` (#106707) By turning `assert(kwnames == NULL)` into a macro that is not in the "forbidden" list, many instructions that formerly were skipped because they contained such an assert (but no other mention of `kwnames`) are now supported in Tier 2. This covers 10 instructions in total (all specializations of `CALL` that invoke some C code): - `CALL_NO_KW_TYPE_1` - `CALL_NO_KW_STR_1` - `CALL_NO_KW_TUPLE_1` - `CALL_NO_KW_BUILTIN_O` - `CALL_NO_KW_BUILTIN_FAST` - `CALL_NO_KW_LEN` - `CALL_NO_KW_ISINSTANCE` - `CALL_NO_KW_METHOD_DESCRIPTOR_O` - `CALL_NO_KW_METHOD_DESCRIPTOR_NOARGS` - `CALL_NO_KW_METHOD_DESCRIPTOR_FAST`
2023-07-17 11:02:58 -07:00
int total_args = oparg;
GH-128682: Convert explicit loops closing arrays into `DECREF_INPUTS`. (GH-128822) * Mark Py_DECREF and Py_XDECREF as escaping * Remove explicit loops for clearing array inputs
2025-01-14 15:08:56 +00:00
_PyStackRef *arguments = args;
GH-131498: Replace single-element arrays with scalars in bytecodes.c (GH-132615)
2025-04-18 07:16:28 -07:00
if (!PyStackRef_IsNull(self_or_null)) {
GH-128682: Convert explicit loops closing arrays into `DECREF_INPUTS`. (GH-128822) * Mark Py_DECREF and Py_XDECREF as escaping * Remove explicit loops for clearing array inputs
2025-01-14 15:08:56 +00:00
arguments--;
gh-106581: Add 10 new opcodes by allowing `assert(kwnames == NULL)` (#106707) By turning `assert(kwnames == NULL)` into a macro that is not in the "forbidden" list, many instructions that formerly were skipped because they contained such an assert (but no other mention of `kwnames`) are now supported in Tier 2. This covers 10 instructions in total (all specializations of `CALL` that invoke some C code): - `CALL_NO_KW_TYPE_1` - `CALL_NO_KW_STR_1` - `CALL_NO_KW_TUPLE_1` - `CALL_NO_KW_BUILTIN_O` - `CALL_NO_KW_BUILTIN_FAST` - `CALL_NO_KW_LEN` - `CALL_NO_KW_ISINSTANCE` - `CALL_NO_KW_METHOD_DESCRIPTOR_O` - `CALL_NO_KW_METHOD_DESCRIPTOR_NOARGS` - `CALL_NO_KW_METHOD_DESCRIPTOR_FAST`
2023-07-17 11:02:58 -07:00
total_args++;
}
gh-131998: Fix `NULL` dereference when using an unbound method descriptor in a specialized code path (#132000) Co-authored-by: sobolevn <mail@sobolevn.me> Co-authored-by: Victor Stinner <vstinner@python.org> Co-authored-by: Mark Shannon <mark@hotpy.org>
2025-04-08 06:31:43 -04:00
if (total_args == 0) {
UOP_STAT_INC(uopcode, miss);
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(0);
gh-131998: Fix `NULL` dereference when using an unbound method descriptor in a specialized code path (#132000) Co-authored-by: sobolevn <mail@sobolevn.me> Co-authored-by: Victor Stinner <vstinner@python.org> Co-authored-by: Mark Shannon <mark@hotpy.org>
2025-04-08 06:31:43 -04:00
JUMP_TO_JUMP_TARGET();
}
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
PyMethodDescrObject *method = (PyMethodDescrObject *)callable_o;
GH-117457: Correct pystats uop "miss" counts (GH-117477)
2024-04-04 18:49:18 -04:00
if (!Py_IS_TYPE(method, &PyMethodDescr_Type)) {
UOP_STAT_INC(uopcode, miss);
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(0);
GH-117457: Correct pystats uop "miss" counts (GH-117477)
2024-04-04 18:49:18 -04:00
JUMP_TO_JUMP_TARGET();
}
GH-105848: Simplify the arrangement of CALL's stack (GH-107788)
2023-08-09 11:19:39 -07:00
PyMethodDef *meth = method->d_method;
GH-117457: Correct pystats uop "miss" counts (GH-117477)
2024-04-04 18:49:18 -04:00
if (meth->ml_flags != METH_FASTCALL) {
UOP_STAT_INC(uopcode, miss);
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(0);
GH-117457: Correct pystats uop "miss" counts (GH-117477)
2024-04-04 18:49:18 -04:00
JUMP_TO_JUMP_TARGET();
}
GH-128682: Convert explicit loops closing arrays into `DECREF_INPUTS`. (GH-128822) * Mark Py_DECREF and Py_XDECREF as escaping * Remove explicit loops for clearing array inputs
2025-01-14 15:08:56 +00:00
PyObject *self = PyStackRef_AsPyObjectBorrow(arguments[0]);
gh-131998: Fix `NULL` dereference when using an unbound method descriptor in a specialized code path (#132000) Co-authored-by: sobolevn <mail@sobolevn.me> Co-authored-by: Victor Stinner <vstinner@python.org> Co-authored-by: Mark Shannon <mark@hotpy.org>
2025-04-08 06:31:43 -04:00
assert(self != NULL);
GH-117457: Correct pystats uop "miss" counts (GH-117477)
2024-04-04 18:49:18 -04:00
if (!Py_IS_TYPE(self, method->d_common.d_type)) {
UOP_STAT_INC(uopcode, miss);
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(0);
GH-117457: Correct pystats uop "miss" counts (GH-117477)
2024-04-04 18:49:18 -04:00
JUMP_TO_JUMP_TARGET();
}
gh-106581: Add 10 new opcodes by allowing `assert(kwnames == NULL)` (#106707) By turning `assert(kwnames == NULL)` into a macro that is not in the "forbidden" list, many instructions that formerly were skipped because they contained such an assert (but no other mention of `kwnames`) are now supported in Tier 2. This covers 10 instructions in total (all specializations of `CALL` that invoke some C code): - `CALL_NO_KW_TYPE_1` - `CALL_NO_KW_STR_1` - `CALL_NO_KW_TUPLE_1` - `CALL_NO_KW_BUILTIN_O` - `CALL_NO_KW_BUILTIN_FAST` - `CALL_NO_KW_LEN` - `CALL_NO_KW_ISINSTANCE` - `CALL_NO_KW_METHOD_DESCRIPTOR_O` - `CALL_NO_KW_METHOD_DESCRIPTOR_NOARGS` - `CALL_NO_KW_METHOD_DESCRIPTOR_FAST`
2023-07-17 11:02:58 -07:00
STAT_INC(CALL, hit);
GH-119866: Spill the stack around escaping calls. (GH-124392) * Spill the evaluation around escaping calls in the generated interpreter and JIT. * The code generator tracks live, cached values so they can be saved to memory when needed. * Spills the stack pointer around escaping calls, so that the exact stack is visible to the cycle GC.
2024-10-07 14:56:39 +01:00
_PyFrame_SetStackPointer(frame, stack_pointer);
GH-141794: Limit size of generated machine code. (GH-142228) * Factor out bodies of the largest uops, to reduce jit code size. * Factor out common assert, also reducing jit code size. * Limit size of jitted code for a single executor to 1MB.
2025-12-03 17:43:35 +00:00
PyObject *res_o = _PyCallMethodDescriptorFast_StackRefSteal(
callable,
meth,
self,
arguments,
total_args
);
GH-128682: Account for escapes in `DECREF_INPUTS` (GH-129953) * Handle escapes in DECREF_INPUTS * Mark a few more functions as escaping * Replace DECREF_INPUTS with PyStackRef_CLOSE where possible
2025-02-12 17:44:59 +00:00
stack_pointer = _PyFrame_GetStackPointer(frame);
GH-128682: Make `PyStackRef_CLOSE` escaping. (GH-129404)
2025-02-03 12:41:32 +00:00
if (res_o == NULL) {
GH-141794: Limit size of generated machine code. (GH-142228) * Factor out bodies of the largest uops, to reduce jit code size. * Factor out common assert, also reducing jit code size. * Limit size of jitted code for a single executor to 1MB.
2025-12-03 17:43:35 +00:00
stack_pointer += -2 - oparg;
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(0);
GH-128682: Make `PyStackRef_CLOSE` escaping. (GH-129404)
2025-02-03 12:41:32 +00:00
JUMP_TO_ERROR();
}
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
res = PyStackRef_FromPyObjectSteal(res_o);
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_tos_cache0 = res;
_tos_cache1 = PyStackRef_ZERO_BITS;
_tos_cache2 = PyStackRef_ZERO_BITS;
SET_CURRENT_CACHED_VALUES(1);
stack_pointer += -2 - oparg;
GH-141794: Limit size of generated machine code. (GH-142228) * Factor out bodies of the largest uops, to reduce jit code size. * Factor out common assert, also reducing jit code size. * Limit size of jitted code for a single executor to 1MB.
2025-12-03 17:43:35 +00:00
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-106581: Add 10 new opcodes by allowing `assert(kwnames == NULL)` (#106707) By turning `assert(kwnames == NULL)` into a macro that is not in the "forbidden" list, many instructions that formerly were skipped because they contained such an assert (but no other mention of `kwnames`) are now supported in Tier 2. This covers 10 instructions in total (all specializations of `CALL` that invoke some C code): - `CALL_NO_KW_TYPE_1` - `CALL_NO_KW_STR_1` - `CALL_NO_KW_TUPLE_1` - `CALL_NO_KW_BUILTIN_O` - `CALL_NO_KW_BUILTIN_FAST` - `CALL_NO_KW_LEN` - `CALL_NO_KW_ISINSTANCE` - `CALL_NO_KW_METHOD_DESCRIPTOR_O` - `CALL_NO_KW_METHOD_DESCRIPTOR_NOARGS` - `CALL_NO_KW_METHOD_DESCRIPTOR_FAST`
2023-07-17 11:02:58 -07:00
break;
}
GH-128563: Generate `opcode = ...` in instructions that need `opcode` (GH-129608) * Remove support for GO_TO_INSTRUCTION
2025-02-03 15:09:21 +00:00
/* _MONITOR_CALL_KW is not a viable micro-op for tier 2 because it uses the 'this_instr' variable */
GH-111485: Factor out tier 2 code generation from the rest of the interpreter code generator (GH-112968)
2023-12-12 12:12:17 +00:00
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
case _MAYBE_EXPAND_METHOD_KW_r11: {
CHECK_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-131498: Replace single-element arrays with scalars in bytecodes.c (GH-132615)
2025-04-18 07:16:28 -07:00
_PyStackRef self_or_null;
_PyStackRef callable;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef _stack_item_0 = _tos_cache0;
GH-119866: Spill the stack around escaping calls. (GH-124392) * Spill the evaluation around escaping calls in the generated interpreter and JIT. * The code generator tracks live, cached values so they can be saved to memory when needed. * Spills the stack pointer around escaping calls, so that the exact stack is visible to the cycle GC.
2024-10-07 14:56:39 +01:00
oparg = CURRENT_OPARG();
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
self_or_null = stack_pointer[-1 - oparg];
callable = stack_pointer[-2 - oparg];
GH-131498: Replace single-element arrays with scalars in bytecodes.c (GH-132615)
2025-04-18 07:16:28 -07:00
if (PyStackRef_TYPE(callable) == &PyMethod_Type && PyStackRef_IsNull(self_or_null)) {
PyObject *callable_o = PyStackRef_AsPyObjectBorrow(callable);
GH-119866: Spill the stack around escaping calls. (GH-124392) * Spill the evaluation around escaping calls in the generated interpreter and JIT. * The code generator tracks live, cached values so they can be saved to memory when needed. * Spills the stack pointer around escaping calls, so that the exact stack is visible to the cycle GC.
2024-10-07 14:56:39 +01:00
PyObject *self = ((PyMethodObject *)callable_o)->im_self;
GH-131498: Replace single-element arrays with scalars in bytecodes.c (GH-132615)
2025-04-18 07:16:28 -07:00
self_or_null = PyStackRef_FromPyObjectNew(self);
GH-119866: Spill the stack around escaping calls. (GH-124392) * Spill the evaluation around escaping calls in the generated interpreter and JIT. * The code generator tracks live, cached values so they can be saved to memory when needed. * Spills the stack pointer around escaping calls, so that the exact stack is visible to the cycle GC.
2024-10-07 14:56:39 +01:00
PyObject *method = ((PyMethodObject *)callable_o)->im_func;
GH-131498: Replace single-element arrays with scalars in bytecodes.c (GH-132615)
2025-04-18 07:16:28 -07:00
_PyStackRef temp = callable;
callable = PyStackRef_FromPyObjectNew(method);
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
stack_pointer[-2 - oparg] = callable;
stack_pointer[-1 - oparg] = self_or_null;
stack_pointer[0] = _stack_item_0;
stack_pointer += 1;
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
GH-128682: Make `PyStackRef_CLOSE` escaping. (GH-129404)
2025-02-03 12:41:32 +00:00
_PyFrame_SetStackPointer(frame, stack_pointer);
GH-119866: Spill the stack around escaping calls. (GH-124392) * Spill the evaluation around escaping calls in the generated interpreter and JIT. * The code generator tracks live, cached values so they can be saved to memory when needed. * Spills the stack pointer around escaping calls, so that the exact stack is visible to the cycle GC.
2024-10-07 14:56:39 +01:00
PyStackRef_CLOSE(temp);
GH-128682: Make `PyStackRef_CLOSE` escaping. (GH-129404)
2025-02-03 12:41:32 +00:00
stack_pointer = _PyFrame_GetStackPointer(frame);
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
stack_pointer += -1;
GH-119866: Spill the stack around escaping calls. (GH-124392) * Spill the evaluation around escaping calls in the generated interpreter and JIT. * The code generator tracks live, cached values so they can be saved to memory when needed. * Spills the stack pointer around escaping calls, so that the exact stack is visible to the cycle GC.
2024-10-07 14:56:39 +01:00
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_tos_cache0 = _stack_item_0;
_tos_cache1 = PyStackRef_ZERO_BITS;
_tos_cache2 = PyStackRef_ZERO_BITS;
SET_CURRENT_CACHED_VALUES(1);
stack_pointer[-2 - oparg] = callable;
stack_pointer[-1 - oparg] = self_or_null;
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-119866: Spill the stack around escaping calls. (GH-124392) * Spill the evaluation around escaping calls in the generated interpreter and JIT. * The code generator tracks live, cached values so they can be saved to memory when needed. * Spills the stack pointer around escaping calls, so that the exact stack is visible to the cycle GC.
2024-10-07 14:56:39 +01:00
break;
}
GH-120024: Remove `CHECK_EVAL_BREAKER` macro. (GH-122968) * Factor some instructions into micro-ops to isolate CHECK_EVAL_BREAKER for escape analysis * Eliminate CHECK_EVAL_BREAKER macro
2024-08-14 12:04:05 +01:00
/* _DO_CALL_KW is not a viable micro-op for tier 2 because it uses the 'this_instr' variable */
GH-111485: Factor out tier 2 code generation from the rest of the interpreter code generator (GH-112968)
2023-12-12 12:12:17 +00:00
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
case _PY_FRAME_KW_r11: {
CHECK_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-118093: Specialize `CALL_KW` (GH-123006)
2024-08-16 17:11:24 +01:00
_PyStackRef kwnames;
_PyStackRef *args;
GH-131498: Replace single-element arrays with scalars in bytecodes.c (GH-132615)
2025-04-18 07:16:28 -07:00
_PyStackRef self_or_null;
_PyStackRef callable;
GH-135379: Remove types from stack items in code generator. (GH-135384) * Make casts explicit in the instruction definitions
2025-06-11 15:52:25 +01:00
_PyStackRef new_frame;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef _stack_item_0 = _tos_cache0;
GH-118093: Specialize `CALL_KW` (GH-123006)
2024-08-16 17:11:24 +01:00
oparg = CURRENT_OPARG();
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
kwnames = _stack_item_0;
args = &stack_pointer[-oparg];
self_or_null = stack_pointer[-1 - oparg];
callable = stack_pointer[-2 - oparg];
GH-131498: Replace single-element arrays with scalars in bytecodes.c (GH-132615)
2025-04-18 07:16:28 -07:00
PyObject *callable_o = PyStackRef_AsPyObjectBorrow(callable);
GH-118093: Specialize `CALL_KW` (GH-123006)
2024-08-16 17:11:24 +01:00
int total_args = oparg;
GH-128682: Convert explicit loops closing arrays into `DECREF_INPUTS`. (GH-128822) * Mark Py_DECREF and Py_XDECREF as escaping * Remove explicit loops for clearing array inputs
2025-01-14 15:08:56 +00:00
_PyStackRef *arguments = args;
GH-131498: Replace single-element arrays with scalars in bytecodes.c (GH-132615)
2025-04-18 07:16:28 -07:00
if (!PyStackRef_IsNull(self_or_null)) {
GH-128682: Convert explicit loops closing arrays into `DECREF_INPUTS`. (GH-128822) * Mark Py_DECREF and Py_XDECREF as escaping * Remove explicit loops for clearing array inputs
2025-01-14 15:08:56 +00:00
arguments--;
GH-118093: Specialize `CALL_KW` (GH-123006)
2024-08-16 17:11:24 +01:00
total_args++;
}
PyObject *kwnames_o = PyStackRef_AsPyObjectBorrow(kwnames);
int positional_args = total_args - (int)PyTuple_GET_SIZE(kwnames_o);
assert(Py_TYPE(callable_o) == &PyFunction_Type);
int code_flags = ((PyCodeObject*)PyFunction_GET_CODE(callable_o))->co_flags;
PyObject *locals = code_flags & CO_OPTIMIZED ? NULL : Py_NewRef(PyFunction_GET_GLOBALS(callable_o));
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
stack_pointer[0] = kwnames;
stack_pointer += 1;
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
GH-119866: Spill the stack around escaping calls. (GH-124392) * Spill the evaluation around escaping calls in the generated interpreter and JIT. * The code generator tracks live, cached values so they can be saved to memory when needed. * Spills the stack pointer around escaping calls, so that the exact stack is visible to the cycle GC.
2024-10-07 14:56:39 +01:00
_PyFrame_SetStackPointer(frame, stack_pointer);
gh-115999: Specialize `CALL_KW` in free-threaded builds (#127713) * Enable specialization of CALL_KW * Fix bug pushing frame in _PY_FRAME_KW `_PY_FRAME_KW` pushes a pointer to the new frame onto the stack for consumption by the next uop. When pushing the frame fails, we do not want to push the result, `NULL`, to the stack because it is not a valid stackref. This works in the default build because `PyStackRef_NULL` and `NULL` are the same value, so the `PyStackRef_XCLOSE()` in the error handler ignores it. In the free-threaded build the values are not the same; `PyStackRef_XCLOSE()` will attempt to decref a null pointer.
2024-12-11 15:18:22 -08:00
_PyInterpreterFrame *temp = _PyEvalFramePushAndInit(
GH-131498: Replace single-element arrays with scalars in bytecodes.c (GH-132615)
2025-04-18 07:16:28 -07:00
tstate, callable, locals,
GH-128682: Convert explicit loops closing arrays into `DECREF_INPUTS`. (GH-128822) * Mark Py_DECREF and Py_XDECREF as escaping * Remove explicit loops for clearing array inputs
2025-01-14 15:08:56 +00:00
arguments, positional_args, kwnames_o, frame
GH-118093: Specialize `CALL_KW` (GH-123006)
2024-08-16 17:11:24 +01:00
);
GH-119866: Spill the stack around escaping calls. (GH-124392) * Spill the evaluation around escaping calls in the generated interpreter and JIT. * The code generator tracks live, cached values so they can be saved to memory when needed. * Spills the stack pointer around escaping calls, so that the exact stack is visible to the cycle GC.
2024-10-07 14:56:39 +01:00
stack_pointer = _PyFrame_GetStackPointer(frame);
GH-128682: Make `PyStackRef_CLOSE` escaping. (GH-129404)
2025-02-03 12:41:32 +00:00
stack_pointer += -1;
GH-141794: Limit size of generated machine code. (GH-142228) * Factor out bodies of the largest uops, to reduce jit code size. * Factor out common assert, also reducing jit code size. * Limit size of jitted code for a single executor to 1MB.
2025-12-03 17:43:35 +00:00
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
GH-128682: Make `PyStackRef_CLOSE` escaping. (GH-129404)
2025-02-03 12:41:32 +00:00
_PyFrame_SetStackPointer(frame, stack_pointer);
GH-118093: Specialize `CALL_KW` (GH-123006)
2024-08-16 17:11:24 +01:00
PyStackRef_CLOSE(kwnames);
GH-128682: Make `PyStackRef_CLOSE` escaping. (GH-129404)
2025-02-03 12:41:32 +00:00
stack_pointer = _PyFrame_GetStackPointer(frame);
stack_pointer += -2 - oparg;
GH-141794: Limit size of generated machine code. (GH-142228) * Factor out bodies of the largest uops, to reduce jit code size. * Factor out common assert, also reducing jit code size. * Limit size of jitted code for a single executor to 1MB.
2025-12-03 17:43:35 +00:00
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
gh-115999: Specialize `CALL_KW` in free-threaded builds (#127713) * Enable specialization of CALL_KW * Fix bug pushing frame in _PY_FRAME_KW `_PY_FRAME_KW` pushes a pointer to the new frame onto the stack for consumption by the next uop. When pushing the frame fails, we do not want to push the result, `NULL`, to the stack because it is not a valid stackref. This works in the default build because `PyStackRef_NULL` and `NULL` are the same value, so the `PyStackRef_XCLOSE()` in the error handler ignores it. In the free-threaded build the values are not the same; `PyStackRef_XCLOSE()` will attempt to decref a null pointer.
2024-12-11 15:18:22 -08:00
if (temp == NULL) {
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(0);
GH-118093: Specialize `CALL_KW` (GH-123006)
2024-08-16 17:11:24 +01:00
JUMP_TO_ERROR();
}
GH-135379: Remove types from stack items in code generator. (GH-135384) * Make casts explicit in the instruction definitions
2025-06-11 15:52:25 +01:00
new_frame = PyStackRef_Wrap(temp);
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_tos_cache0 = new_frame;
_tos_cache1 = PyStackRef_ZERO_BITS;
_tos_cache2 = PyStackRef_ZERO_BITS;
SET_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-118093: Specialize `CALL_KW` (GH-123006)
2024-08-16 17:11:24 +01:00
break;
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
case _CHECK_FUNCTION_VERSION_KW_r11: {
CHECK_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-131498: Replace single-element arrays with scalars in bytecodes.c (GH-132615)
2025-04-18 07:16:28 -07:00
_PyStackRef callable;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef _stack_item_0 = _tos_cache0;
GH-118093: Specialize `CALL_KW` (GH-123006)
2024-08-16 17:11:24 +01:00
oparg = CURRENT_OPARG();
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
callable = stack_pointer[-2 - oparg];
GH-140683: JIT: Improve machine code for loading smaller constants on AArch64. (GH-142511) * Use movz and movk instructions for loading 16 and 32 bit operands and oparg. * Loading of 64 bit operands is unchanged.
2025-12-11 12:33:39 +00:00
uint32_t func_version = (uint32_t)CURRENT_OPERAND0_32();
GH-131498: Replace single-element arrays with scalars in bytecodes.c (GH-132615)
2025-04-18 07:16:28 -07:00
PyObject *callable_o = PyStackRef_AsPyObjectBorrow(callable);
GH-118093: Specialize `CALL_KW` (GH-123006)
2024-08-16 17:11:24 +01:00
if (!PyFunction_Check(callable_o)) {
UOP_STAT_INC(uopcode, miss);
GH-142718: JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. (GH-142762) JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. * Make sure that stack is in correct state at side exits with TOS cached values * Simplify choice of cached items for side exits
2025-12-15 19:01:29 +00:00
_tos_cache0 = _stack_item_0;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(1);
GH-118093: Specialize `CALL_KW` (GH-123006)
2024-08-16 17:11:24 +01:00
JUMP_TO_JUMP_TARGET();
}
PyFunctionObject *func = (PyFunctionObject *)callable_o;
if (func->func_version != func_version) {
UOP_STAT_INC(uopcode, miss);
GH-142718: JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. (GH-142762) JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. * Make sure that stack is in correct state at side exits with TOS cached values * Simplify choice of cached items for side exits
2025-12-15 19:01:29 +00:00
_tos_cache0 = _stack_item_0;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(1);
GH-118093: Specialize `CALL_KW` (GH-123006)
2024-08-16 17:11:24 +01:00
JUMP_TO_JUMP_TARGET();
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_tos_cache0 = _stack_item_0;
SET_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-118093: Specialize `CALL_KW` (GH-123006)
2024-08-16 17:11:24 +01:00
break;
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
case _CHECK_METHOD_VERSION_KW_r11: {
CHECK_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-131498: Replace single-element arrays with scalars in bytecodes.c (GH-132615)
2025-04-18 07:16:28 -07:00
_PyStackRef null;
_PyStackRef callable;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef _stack_item_0 = _tos_cache0;
GH-118093: Specialize `CALL_KW` (GH-123006)
2024-08-16 17:11:24 +01:00
oparg = CURRENT_OPARG();
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
null = stack_pointer[-1 - oparg];
callable = stack_pointer[-2 - oparg];
GH-140683: JIT: Improve machine code for loading smaller constants on AArch64. (GH-142511) * Use movz and movk instructions for loading 16 and 32 bit operands and oparg. * Loading of 64 bit operands is unchanged.
2025-12-11 12:33:39 +00:00
uint32_t func_version = (uint32_t)CURRENT_OPERAND0_32();
GH-131498: Replace single-element arrays with scalars in bytecodes.c (GH-132615)
2025-04-18 07:16:28 -07:00
PyObject *callable_o = PyStackRef_AsPyObjectBorrow(callable);
GH-118093: Specialize `CALL_KW` (GH-123006)
2024-08-16 17:11:24 +01:00
if (Py_TYPE(callable_o) != &PyMethod_Type) {
UOP_STAT_INC(uopcode, miss);
GH-142718: JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. (GH-142762) JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. * Make sure that stack is in correct state at side exits with TOS cached values * Simplify choice of cached items for side exits
2025-12-15 19:01:29 +00:00
_tos_cache0 = _stack_item_0;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(1);
GH-118093: Specialize `CALL_KW` (GH-123006)
2024-08-16 17:11:24 +01:00
JUMP_TO_JUMP_TARGET();
}
PyObject *func = ((PyMethodObject *)callable_o)->im_func;
if (!PyFunction_Check(func)) {
UOP_STAT_INC(uopcode, miss);
GH-142718: JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. (GH-142762) JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. * Make sure that stack is in correct state at side exits with TOS cached values * Simplify choice of cached items for side exits
2025-12-15 19:01:29 +00:00
_tos_cache0 = _stack_item_0;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(1);
GH-118093: Specialize `CALL_KW` (GH-123006)
2024-08-16 17:11:24 +01:00
JUMP_TO_JUMP_TARGET();
}
if (((PyFunctionObject *)func)->func_version != func_version) {
UOP_STAT_INC(uopcode, miss);
GH-142718: JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. (GH-142762) JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. * Make sure that stack is in correct state at side exits with TOS cached values * Simplify choice of cached items for side exits
2025-12-15 19:01:29 +00:00
_tos_cache0 = _stack_item_0;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(1);
GH-118093: Specialize `CALL_KW` (GH-123006)
2024-08-16 17:11:24 +01:00
JUMP_TO_JUMP_TARGET();
}
GH-131498: Replace single-element arrays with scalars in bytecodes.c (GH-132615)
2025-04-18 07:16:28 -07:00
if (!PyStackRef_IsNull(null)) {
GH-118093: Specialize `CALL_KW` (GH-123006)
2024-08-16 17:11:24 +01:00
UOP_STAT_INC(uopcode, miss);
GH-142718: JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. (GH-142762) JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. * Make sure that stack is in correct state at side exits with TOS cached values * Simplify choice of cached items for side exits
2025-12-15 19:01:29 +00:00
_tos_cache0 = _stack_item_0;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(1);
GH-118093: Specialize `CALL_KW` (GH-123006)
2024-08-16 17:11:24 +01:00
JUMP_TO_JUMP_TARGET();
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_tos_cache0 = _stack_item_0;
SET_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-118093: Specialize `CALL_KW` (GH-123006)
2024-08-16 17:11:24 +01:00
break;
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
case _EXPAND_METHOD_KW_r11: {
CHECK_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-131498: Replace single-element arrays with scalars in bytecodes.c (GH-132615)
2025-04-18 07:16:28 -07:00
_PyStackRef self_or_null;
_PyStackRef callable;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef _stack_item_0 = _tos_cache0;
GH-118093: Specialize `CALL_KW` (GH-123006)
2024-08-16 17:11:24 +01:00
oparg = CURRENT_OPARG();
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
self_or_null = stack_pointer[-1 - oparg];
callable = stack_pointer[-2 - oparg];
GH-131498: Replace single-element arrays with scalars in bytecodes.c (GH-132615)
2025-04-18 07:16:28 -07:00
assert(PyStackRef_IsNull(self_or_null));
_PyStackRef callable_s = callable;
PyObject *callable_o = PyStackRef_AsPyObjectBorrow(callable);
GH-118093: Specialize `CALL_KW` (GH-123006)
2024-08-16 17:11:24 +01:00
assert(Py_TYPE(callable_o) == &PyMethod_Type);
GH-131498: Replace single-element arrays with scalars in bytecodes.c (GH-132615)
2025-04-18 07:16:28 -07:00
self_or_null = PyStackRef_FromPyObjectNew(((PyMethodObject *)callable_o)->im_self);
callable = PyStackRef_FromPyObjectNew(((PyMethodObject *)callable_o)->im_func);
assert(PyStackRef_FunctionCheck(callable));
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
stack_pointer[-2 - oparg] = callable;
stack_pointer[-1 - oparg] = self_or_null;
stack_pointer[0] = _stack_item_0;
stack_pointer += 1;
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
GH-128682: Make `PyStackRef_CLOSE` escaping. (GH-129404)
2025-02-03 12:41:32 +00:00
_PyFrame_SetStackPointer(frame, stack_pointer);
GH-119866: Spill the stack around escaping calls. (GH-124392) * Spill the evaluation around escaping calls in the generated interpreter and JIT. * The code generator tracks live, cached values so they can be saved to memory when needed. * Spills the stack pointer around escaping calls, so that the exact stack is visible to the cycle GC.
2024-10-07 14:56:39 +01:00
PyStackRef_CLOSE(callable_s);
GH-128682: Make `PyStackRef_CLOSE` escaping. (GH-129404)
2025-02-03 12:41:32 +00:00
stack_pointer = _PyFrame_GetStackPointer(frame);
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_tos_cache0 = _stack_item_0;
_tos_cache1 = PyStackRef_ZERO_BITS;
_tos_cache2 = PyStackRef_ZERO_BITS;
SET_CURRENT_CACHED_VALUES(1);
stack_pointer += -1;
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-118093: Specialize `CALL_KW` (GH-123006)
2024-08-16 17:11:24 +01:00
break;
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
case _CHECK_IS_NOT_PY_CALLABLE_KW_r11: {
CHECK_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-131498: Replace single-element arrays with scalars in bytecodes.c (GH-132615)
2025-04-18 07:16:28 -07:00
_PyStackRef callable;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef _stack_item_0 = _tos_cache0;
GH-118093: Specialize `CALL_KW` (GH-123006)
2024-08-16 17:11:24 +01:00
oparg = CURRENT_OPARG();
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
callable = stack_pointer[-2 - oparg];
GH-131498: Replace single-element arrays with scalars in bytecodes.c (GH-132615)
2025-04-18 07:16:28 -07:00
PyObject *callable_o = PyStackRef_AsPyObjectBorrow(callable);
GH-118093: Specialize `CALL_KW` (GH-123006)
2024-08-16 17:11:24 +01:00
if (PyFunction_Check(callable_o)) {
UOP_STAT_INC(uopcode, miss);
GH-142718: JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. (GH-142762) JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. * Make sure that stack is in correct state at side exits with TOS cached values * Simplify choice of cached items for side exits
2025-12-15 19:01:29 +00:00
_tos_cache0 = _stack_item_0;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(1);
GH-118093: Specialize `CALL_KW` (GH-123006)
2024-08-16 17:11:24 +01:00
JUMP_TO_JUMP_TARGET();
}
if (Py_TYPE(callable_o) == &PyMethod_Type) {
UOP_STAT_INC(uopcode, miss);
GH-142718: JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. (GH-142762) JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. * Make sure that stack is in correct state at side exits with TOS cached values * Simplify choice of cached items for side exits
2025-12-15 19:01:29 +00:00
_tos_cache0 = _stack_item_0;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(1);
GH-118093: Specialize `CALL_KW` (GH-123006)
2024-08-16 17:11:24 +01:00
JUMP_TO_JUMP_TARGET();
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_tos_cache0 = _stack_item_0;
SET_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-118093: Specialize `CALL_KW` (GH-123006)
2024-08-16 17:11:24 +01:00
break;
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
case _CALL_KW_NON_PY_r11: {
CHECK_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-118093: Specialize `CALL_KW` (GH-123006)
2024-08-16 17:11:24 +01:00
_PyStackRef kwnames;
_PyStackRef *args;
GH-131498: Replace single-element arrays with scalars in bytecodes.c (GH-132615)
2025-04-18 07:16:28 -07:00
_PyStackRef self_or_null;
_PyStackRef callable;
GH-118093: Specialize `CALL_KW` (GH-123006)
2024-08-16 17:11:24 +01:00
_PyStackRef res;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef _stack_item_0 = _tos_cache0;
GH-118093: Specialize `CALL_KW` (GH-123006)
2024-08-16 17:11:24 +01:00
oparg = CURRENT_OPARG();
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
kwnames = _stack_item_0;
args = &stack_pointer[-oparg];
self_or_null = stack_pointer[-1 - oparg];
callable = stack_pointer[-2 - oparg];
GH-118093: Specialize `CALL_KW` (GH-123006)
2024-08-16 17:11:24 +01:00
#if TIER_ONE
assert(opcode != INSTRUMENTED_CALL);
#endif
int total_args = oparg;
GH-128682: Convert explicit loops closing arrays into `DECREF_INPUTS`. (GH-128822) * Mark Py_DECREF and Py_XDECREF as escaping * Remove explicit loops for clearing array inputs
2025-01-14 15:08:56 +00:00
_PyStackRef *arguments = args;
GH-131498: Replace single-element arrays with scalars in bytecodes.c (GH-132615)
2025-04-18 07:16:28 -07:00
if (!PyStackRef_IsNull(self_or_null)) {
GH-128682: Convert explicit loops closing arrays into `DECREF_INPUTS`. (GH-128822) * Mark Py_DECREF and Py_XDECREF as escaping * Remove explicit loops for clearing array inputs
2025-01-14 15:08:56 +00:00
arguments--;
GH-118093: Specialize `CALL_KW` (GH-123006)
2024-08-16 17:11:24 +01:00
total_args++;
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
stack_pointer[0] = kwnames;
stack_pointer += 1;
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
GH-119866: Spill the stack around escaping calls. (GH-124392) * Spill the evaluation around escaping calls in the generated interpreter and JIT. * The code generator tracks live, cached values so they can be saved to memory when needed. * Spills the stack pointer around escaping calls, so that the exact stack is visible to the cycle GC.
2024-10-07 14:56:39 +01:00
_PyFrame_SetStackPointer(frame, stack_pointer);
GH-141794: Limit size of generated machine code. (GH-142228) * Factor out bodies of the largest uops, to reduce jit code size. * Factor out common assert, also reducing jit code size. * Limit size of jitted code for a single executor to 1MB.
2025-12-03 17:43:35 +00:00
PyObject *res_o = _Py_VectorCall_StackRefSteal(
callable,
arguments,
total_args,
kwnames);
GH-128682: Make `PyStackRef_CLOSE` escaping. (GH-129404)
2025-02-03 12:41:32 +00:00
stack_pointer = _PyFrame_GetStackPointer(frame);
if (res_o == NULL) {
GH-141794: Limit size of generated machine code. (GH-142228) * Factor out bodies of the largest uops, to reduce jit code size. * Factor out common assert, also reducing jit code size. * Limit size of jitted code for a single executor to 1MB.
2025-12-03 17:43:35 +00:00
stack_pointer += -3 - oparg;
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(0);
GH-128682: Make `PyStackRef_CLOSE` escaping. (GH-129404)
2025-02-03 12:41:32 +00:00
JUMP_TO_ERROR();
}
GH-118093: Specialize `CALL_KW` (GH-123006)
2024-08-16 17:11:24 +01:00
res = PyStackRef_FromPyObjectSteal(res_o);
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_tos_cache0 = res;
_tos_cache1 = PyStackRef_ZERO_BITS;
_tos_cache2 = PyStackRef_ZERO_BITS;
SET_CURRENT_CACHED_VALUES(1);
stack_pointer += -3 - oparg;
GH-141794: Limit size of generated machine code. (GH-142228) * Factor out bodies of the largest uops, to reduce jit code size. * Factor out common assert, also reducing jit code size. * Limit size of jitted code for a single executor to 1MB.
2025-12-03 17:43:35 +00:00
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-118093: Specialize `CALL_KW` (GH-123006)
2024-08-16 17:11:24 +01:00
break;
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
case _MAKE_CALLARGS_A_TUPLE_r33: {
CHECK_CURRENT_CACHED_VALUES(3);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-119866: Spill the stack around escaping calls. (GH-124392) * Spill the evaluation around escaping calls in the generated interpreter and JIT. * The code generator tracks live, cached values so they can be saved to memory when needed. * Spills the stack pointer around escaping calls, so that the exact stack is visible to the cycle GC.
2024-10-07 14:56:39 +01:00
_PyStackRef callargs;
_PyStackRef func;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef _stack_item_0 = _tos_cache0;
_PyStackRef _stack_item_1 = _tos_cache1;
_PyStackRef _stack_item_2 = _tos_cache2;
callargs = _stack_item_1;
func = stack_pointer[-1];
GH-119866: Spill the stack around escaping calls. (GH-124392) * Spill the evaluation around escaping calls in the generated interpreter and JIT. * The code generator tracks live, cached values so they can be saved to memory when needed. * Spills the stack pointer around escaping calls, so that the exact stack is visible to the cycle GC.
2024-10-07 14:56:39 +01:00
PyObject *callargs_o = PyStackRef_AsPyObjectBorrow(callargs);
GH-131498: Cases generator: Allow input and 'peek' variables to be modified (GH-132506)
2025-04-14 12:19:53 +01:00
if (!PyTuple_CheckExact(callargs_o)) {
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
stack_pointer[0] = _stack_item_0;
stack_pointer[1] = callargs;
stack_pointer[2] = _stack_item_2;
stack_pointer += 3;
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
GH-119866: Spill the stack around escaping calls. (GH-124392) * Spill the evaluation around escaping calls in the generated interpreter and JIT. * The code generator tracks live, cached values so they can be saved to memory when needed. * Spills the stack pointer around escaping calls, so that the exact stack is visible to the cycle GC.
2024-10-07 14:56:39 +01:00
_PyFrame_SetStackPointer(frame, stack_pointer);
int err = _Py_Check_ArgsIterable(tstate, PyStackRef_AsPyObjectBorrow(func), callargs_o);
stack_pointer = _PyFrame_GetStackPointer(frame);
if (err < 0) {
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(0);
GH-119866: Spill the stack around escaping calls. (GH-124392) * Spill the evaluation around escaping calls in the generated interpreter and JIT. * The code generator tracks live, cached values so they can be saved to memory when needed. * Spills the stack pointer around escaping calls, so that the exact stack is visible to the cycle GC.
2024-10-07 14:56:39 +01:00
JUMP_TO_ERROR();
}
_PyFrame_SetStackPointer(frame, stack_pointer);
PyObject *tuple_o = PySequence_Tuple(callargs_o);
stack_pointer = _PyFrame_GetStackPointer(frame);
if (tuple_o == NULL) {
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(0);
GH-119866: Spill the stack around escaping calls. (GH-124392) * Spill the evaluation around escaping calls in the generated interpreter and JIT. * The code generator tracks live, cached values so they can be saved to memory when needed. * Spills the stack pointer around escaping calls, so that the exact stack is visible to the cycle GC.
2024-10-07 14:56:39 +01:00
JUMP_TO_ERROR();
}
GH-131498: Cases generator: Allow input and 'peek' variables to be modified (GH-132506)
2025-04-14 12:19:53 +01:00
_PyStackRef temp = callargs;
callargs = PyStackRef_FromPyObjectSteal(tuple_o);
stack_pointer[-2] = callargs;
GH-128682: Make `PyStackRef_CLOSE` escaping. (GH-129404)
2025-02-03 12:41:32 +00:00
_PyFrame_SetStackPointer(frame, stack_pointer);
GH-131498: Cases generator: Allow input and 'peek' variables to be modified (GH-132506)
2025-04-14 12:19:53 +01:00
PyStackRef_CLOSE(temp);
GH-128682: Make `PyStackRef_CLOSE` escaping. (GH-129404)
2025-02-03 12:41:32 +00:00
stack_pointer = _PyFrame_GetStackPointer(frame);
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
stack_pointer += -3;
GH-119866: Spill the stack around escaping calls. (GH-124392) * Spill the evaluation around escaping calls in the generated interpreter and JIT. * The code generator tracks live, cached values so they can be saved to memory when needed. * Spills the stack pointer around escaping calls, so that the exact stack is visible to the cycle GC.
2024-10-07 14:56:39 +01:00
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_tos_cache2 = _stack_item_2;
_tos_cache1 = callargs;
_tos_cache0 = _stack_item_0;
SET_CURRENT_CACHED_VALUES(3);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-119866: Spill the stack around escaping calls. (GH-124392) * Spill the evaluation around escaping calls in the generated interpreter and JIT. * The code generator tracks live, cached values so they can be saved to memory when needed. * Spills the stack pointer around escaping calls, so that the exact stack is visible to the cycle GC.
2024-10-07 14:56:39 +01:00
break;
}
/* _DO_CALL_FUNCTION_EX is not a viable micro-op for tier 2 because it uses the 'this_instr' variable */
GH-111485: Factor out tier 2 code generation from the rest of the interpreter code generator (GH-112968)
2023-12-12 12:12:17 +00:00
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
case _MAKE_FUNCTION_r11: {
CHECK_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
_PyStackRef codeobj_st;
_PyStackRef func;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef _stack_item_0 = _tos_cache0;
codeobj_st = _stack_item_0;
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
PyObject *codeobj = PyStackRef_AsPyObjectBorrow(codeobj_st);
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
stack_pointer[0] = codeobj_st;
stack_pointer += 1;
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
GH-119866: Spill the stack around escaping calls. (GH-124392) * Spill the evaluation around escaping calls in the generated interpreter and JIT. * The code generator tracks live, cached values so they can be saved to memory when needed. * Spills the stack pointer around escaping calls, so that the exact stack is visible to the cycle GC.
2024-10-07 14:56:39 +01:00
_PyFrame_SetStackPointer(frame, stack_pointer);
gh-104584: Baby steps towards generating and executing traces (#105924) Added a new, experimental, tracing optimizer and interpreter (a.k.a. "tier 2"). This currently pessimizes, so don't use yet -- this is infrastructure so we can experiment with optimizing passes. To enable it, pass ``-Xuops`` or set ``PYTHONUOPS=1``. To get debug output, set ``PYTHONUOPSDEBUG=N`` where ``N`` is a debug level (0-4, where 0 is no debug output and 4 is excessively verbose). All of this code is likely to change dramatically before the 3.13 feature freeze. But this is a first step.
2023-06-26 19:02:57 -07:00
PyFunctionObject *func_obj = (PyFunctionObject *)
GH-111485: Factor out tier 2 code generation from the rest of the interpreter code generator (GH-112968)
2023-12-12 12:12:17 +00:00
PyFunction_New(codeobj, GLOBALS());
GH-119866: Spill the stack around escaping calls. (GH-124392) * Spill the evaluation around escaping calls in the generated interpreter and JIT. * The code generator tracks live, cached values so they can be saved to memory when needed. * Spills the stack pointer around escaping calls, so that the exact stack is visible to the cycle GC.
2024-10-07 14:56:39 +01:00
stack_pointer = _PyFrame_GetStackPointer(frame);
GH-128682: Make `PyStackRef_CLOSE` escaping. (GH-129404)
2025-02-03 12:41:32 +00:00
stack_pointer += -1;
GH-141794: Limit size of generated machine code. (GH-142228) * Factor out bodies of the largest uops, to reduce jit code size. * Factor out common assert, also reducing jit code size. * Limit size of jitted code for a single executor to 1MB.
2025-12-03 17:43:35 +00:00
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
GH-128682: Make `PyStackRef_CLOSE` escaping. (GH-129404)
2025-02-03 12:41:32 +00:00
_PyFrame_SetStackPointer(frame, stack_pointer);
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
PyStackRef_CLOSE(codeobj_st);
GH-128682: Make `PyStackRef_CLOSE` escaping. (GH-129404)
2025-02-03 12:41:32 +00:00
stack_pointer = _PyFrame_GetStackPointer(frame);
if (func_obj == NULL) {
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(0);
GH-128682: Make `PyStackRef_CLOSE` escaping. (GH-129404)
2025-02-03 12:41:32 +00:00
JUMP_TO_ERROR();
}
gh-106581: Project through calls (#108067) This finishes the work begun in gh-107760. When, while projecting a superblock, we encounter a call to a short, simple function, the superblock will now enter the function using `_PUSH_FRAME`, continue through it, and leave it using `_POP_FRAME`, and then continue through the original code. Multiple frame pushes and pops are even possible. It is also possible to stop appending to the superblock in the middle of a called function, when running out of space or encountering an unsupported bytecode.
2023-08-17 11:29:58 -07:00
_PyFunction_SetVersion(
GH-111485: Factor out tier 2 code generation from the rest of the interpreter code generator (GH-112968)
2023-12-12 12:12:17 +00:00
func_obj, ((PyCodeObject *)codeobj)->co_version);
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
func = PyStackRef_FromPyObjectSteal((PyObject *)func_obj);
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_tos_cache0 = func;
_tos_cache1 = PyStackRef_ZERO_BITS;
_tos_cache2 = PyStackRef_ZERO_BITS;
SET_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-104584: Baby steps towards generating and executing traces (#105924) Added a new, experimental, tracing optimizer and interpreter (a.k.a. "tier 2"). This currently pessimizes, so don't use yet -- this is infrastructure so we can experiment with optimizing passes. To enable it, pass ``-Xuops`` or set ``PYTHONUOPS=1``. To get debug output, set ``PYTHONUOPSDEBUG=N`` where ``N`` is a debug level (0-4, where 0 is no debug output and 4 is excessively verbose). All of this code is likely to change dramatically before the 3.13 feature freeze. But this is a first step.
2023-06-26 19:02:57 -07:00
break;
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
case _SET_FUNCTION_ATTRIBUTE_r01: {
CHECK_CURRENT_CACHED_VALUES(0);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-119866: Spill the stack around escaping calls. (GH-124392) * Spill the evaluation around escaping calls in the generated interpreter and JIT. * The code generator tracks live, cached values so they can be saved to memory when needed. * Spills the stack pointer around escaping calls, so that the exact stack is visible to the cycle GC.
2024-10-07 14:56:39 +01:00
_PyStackRef func_in;
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
_PyStackRef attr_st;
GH-119866: Spill the stack around escaping calls. (GH-124392) * Spill the evaluation around escaping calls in the generated interpreter and JIT. * The code generator tracks live, cached values so they can be saved to memory when needed. * Spills the stack pointer around escaping calls, so that the exact stack is visible to the cycle GC.
2024-10-07 14:56:39 +01:00
_PyStackRef func_out;
GH-111485: Factor out tier 2 code generation from the rest of the interpreter code generator (GH-112968)
2023-12-12 12:12:17 +00:00
oparg = CURRENT_OPARG();
GH-119866: Spill the stack around escaping calls. (GH-124392) * Spill the evaluation around escaping calls in the generated interpreter and JIT. * The code generator tracks live, cached values so they can be saved to memory when needed. * Spills the stack pointer around escaping calls, so that the exact stack is visible to the cycle GC.
2024-10-07 14:56:39 +01:00
func_in = stack_pointer[-1];
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
attr_st = stack_pointer[-2];
GH-119866: Spill the stack around escaping calls. (GH-124392) * Spill the evaluation around escaping calls in the generated interpreter and JIT. * The code generator tracks live, cached values so they can be saved to memory when needed. * Spills the stack pointer around escaping calls, so that the exact stack is visible to the cycle GC.
2024-10-07 14:56:39 +01:00
PyObject *func = PyStackRef_AsPyObjectBorrow(func_in);
PyObject *attr = PyStackRef_AsPyObjectSteal(attr_st);
func_out = func_in;
gh-104584: Baby steps towards generating and executing traces (#105924) Added a new, experimental, tracing optimizer and interpreter (a.k.a. "tier 2"). This currently pessimizes, so don't use yet -- this is infrastructure so we can experiment with optimizing passes. To enable it, pass ``-Xuops`` or set ``PYTHONUOPS=1``. To get debug output, set ``PYTHONUOPSDEBUG=N`` where ``N`` is a debug level (0-4, where 0 is no debug output and 4 is excessively verbose). All of this code is likely to change dramatically before the 3.13 feature freeze. But this is a first step.
2023-06-26 19:02:57 -07:00
assert(PyFunction_Check(func));
GH-119866: Spill the stack around escaping calls. (GH-124392) * Spill the evaluation around escaping calls in the generated interpreter and JIT. * The code generator tracks live, cached values so they can be saved to memory when needed. * Spills the stack pointer around escaping calls, so that the exact stack is visible to the cycle GC.
2024-10-07 14:56:39 +01:00
size_t offset = _Py_FunctionAttributeOffsets[oparg];
assert(offset != 0);
PyObject **ptr = (PyObject **)(((char *)func) + offset);
assert(*ptr == NULL);
*ptr = attr;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_tos_cache0 = func_out;
SET_CURRENT_CACHED_VALUES(1);
stack_pointer += -2;
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _SET_FUNCTION_ATTRIBUTE_r11: {
CHECK_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef func_in;
_PyStackRef attr_st;
_PyStackRef func_out;
_PyStackRef _stack_item_0 = _tos_cache0;
oparg = CURRENT_OPARG();
func_in = _stack_item_0;
attr_st = stack_pointer[-1];
PyObject *func = PyStackRef_AsPyObjectBorrow(func_in);
PyObject *attr = PyStackRef_AsPyObjectSteal(attr_st);
func_out = func_in;
assert(PyFunction_Check(func));
size_t offset = _Py_FunctionAttributeOffsets[oparg];
assert(offset != 0);
PyObject **ptr = (PyObject **)(((char *)func) + offset);
assert(*ptr == NULL);
*ptr = attr;
_tos_cache0 = func_out;
SET_CURRENT_CACHED_VALUES(1);
GH-111485: Factor out tier 2 code generation from the rest of the interpreter code generator (GH-112968)
2023-12-12 12:12:17 +00:00
stack_pointer += -1;
GH-141794: Limit size of generated machine code. (GH-142228) * Factor out bodies of the largest uops, to reduce jit code size. * Factor out common assert, also reducing jit code size. * Limit size of jitted code for a single executor to 1MB.
2025-12-03 17:43:35 +00:00
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _SET_FUNCTION_ATTRIBUTE_r21: {
CHECK_CURRENT_CACHED_VALUES(2);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef func_in;
_PyStackRef attr_st;
_PyStackRef func_out;
_PyStackRef _stack_item_0 = _tos_cache0;
_PyStackRef _stack_item_1 = _tos_cache1;
oparg = CURRENT_OPARG();
func_in = _stack_item_1;
attr_st = _stack_item_0;
PyObject *func = PyStackRef_AsPyObjectBorrow(func_in);
PyObject *attr = PyStackRef_AsPyObjectSteal(attr_st);
func_out = func_in;
assert(PyFunction_Check(func));
size_t offset = _Py_FunctionAttributeOffsets[oparg];
assert(offset != 0);
PyObject **ptr = (PyObject **)(((char *)func) + offset);
assert(*ptr == NULL);
*ptr = attr;
_tos_cache0 = func_out;
SET_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _SET_FUNCTION_ATTRIBUTE_r32: {
CHECK_CURRENT_CACHED_VALUES(3);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef func_in;
_PyStackRef attr_st;
_PyStackRef func_out;
_PyStackRef _stack_item_0 = _tos_cache0;
_PyStackRef _stack_item_1 = _tos_cache1;
_PyStackRef _stack_item_2 = _tos_cache2;
oparg = CURRENT_OPARG();
func_in = _stack_item_2;
attr_st = _stack_item_1;
PyObject *func = PyStackRef_AsPyObjectBorrow(func_in);
PyObject *attr = PyStackRef_AsPyObjectSteal(attr_st);
func_out = func_in;
assert(PyFunction_Check(func));
size_t offset = _Py_FunctionAttributeOffsets[oparg];
assert(offset != 0);
PyObject **ptr = (PyObject **)(((char *)func) + offset);
assert(*ptr == NULL);
*ptr = attr;
_tos_cache1 = func_out;
_tos_cache0 = _stack_item_0;
SET_CURRENT_CACHED_VALUES(2);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-104584: Baby steps towards generating and executing traces (#105924) Added a new, experimental, tracing optimizer and interpreter (a.k.a. "tier 2"). This currently pessimizes, so don't use yet -- this is infrastructure so we can experiment with optimizing passes. To enable it, pass ``-Xuops`` or set ``PYTHONUOPS=1``. To get debug output, set ``PYTHONUOPSDEBUG=N`` where ``N`` is a debug level (0-4, where 0 is no debug output and 4 is excessively verbose). All of this code is likely to change dramatically before the 3.13 feature freeze. But this is a first step.
2023-06-26 19:02:57 -07:00
break;
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
case _RETURN_GENERATOR_r01: {
CHECK_CURRENT_CACHED_VALUES(0);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
_PyStackRef res;
gh-123923: Defer refcounting for `f_funcobj` in `_PyInterpreterFrame` (#124026) Use a `_PyStackRef` and defer the reference to `f_funcobj` when possible. This avoids some reference count contention in the common case of executing the same code object from multiple threads concurrently in the free-threaded build.
2024-09-24 13:08:18 -07:00
assert(PyStackRef_FunctionCheck(frame->f_funcobj));
PyFunctionObject *func = (PyFunctionObject *)PyStackRef_AsPyObjectBorrow(frame->f_funcobj);
GH-119866: Spill the stack around escaping calls. (GH-124392) * Spill the evaluation around escaping calls in the generated interpreter and JIT. * The code generator tracks live, cached values so they can be saved to memory when needed. * Spills the stack pointer around escaping calls, so that the exact stack is visible to the cycle GC.
2024-10-07 14:56:39 +01:00
_PyFrame_SetStackPointer(frame, stack_pointer);
GH-118095: Handle `RETURN_GENERATOR` in tier 2 (GH-118180)
2024-04-25 11:32:47 +01:00
PyGenObject *gen = (PyGenObject *)_Py_MakeCoro(func);
GH-119866: Spill the stack around escaping calls. (GH-124392) * Spill the evaluation around escaping calls in the generated interpreter and JIT. * The code generator tracks live, cached values so they can be saved to memory when needed. * Spills the stack pointer around escaping calls, so that the exact stack is visible to the cycle GC.
2024-10-07 14:56:39 +01:00
stack_pointer = _PyFrame_GetStackPointer(frame);
GH-128682: Make `PyStackRef_CLOSE` escaping. (GH-129404)
2025-02-03 12:41:32 +00:00
if (gen == NULL) {
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(0);
GH-128682: Make `PyStackRef_CLOSE` escaping. (GH-129404)
2025-02-03 12:41:32 +00:00
JUMP_TO_ERROR();
}
GH-131498: Cases generator: manage stacks automatically (GH-132074)
2025-04-04 17:59:36 +01:00
assert(STACK_LEVEL() == 0);
GH-118095: Handle `RETURN_GENERATOR` in tier 2 (GH-118180)
2024-04-25 11:32:47 +01:00
_PyFrame_SetStackPointer(frame, stack_pointer);
gh-120834: fix type of *_iframe field in _PyGenObject_HEAD declaration (#120835)
2024-06-24 10:23:38 +01:00
_PyInterpreterFrame *gen_frame = &gen->gi_iframe;
GH-118095: Handle `RETURN_GENERATOR` in tier 2 (GH-118180)
2024-04-25 11:32:47 +01:00
frame->instr_ptr++;
_PyFrame_Copy(frame, gen_frame);
assert(frame->frame_obj == NULL);
gen->gi_frame_state = FRAME_CREATED;
gen_frame->owner = FRAME_OWNED_BY_GENERATOR;
_Py_LeaveRecursiveCallPy(tstate);
_PyInterpreterFrame *prev = frame->previous;
_PyThreadState_PopFrame(tstate, frame);
frame = tstate->current_frame = prev;
LOAD_IP(frame->return_offset);
GH-119866: Spill the stack around escaping calls. (GH-124392) * Spill the evaluation around escaping calls in the generated interpreter and JIT. * The code generator tracks live, cached values so they can be saved to memory when needed. * Spills the stack pointer around escaping calls, so that the exact stack is visible to the cycle GC.
2024-10-07 14:56:39 +01:00
stack_pointer = _PyFrame_GetStackPointer(frame);
GH-127705: Use `_PyStackRef`s in the default build. (GH-127875)
2025-03-10 14:06:56 +00:00
res = PyStackRef_FromPyObjectStealMortal((PyObject *)gen);
GH-118095: Handle `RETURN_GENERATOR` in tier 2 (GH-118180)
2024-04-25 11:32:47 +01:00
LLTRACE_RESUME_FRAME();
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_tos_cache0 = res;
_tos_cache1 = PyStackRef_ZERO_BITS;
_tos_cache2 = PyStackRef_ZERO_BITS;
SET_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-118095: Handle `RETURN_GENERATOR` in tier 2 (GH-118180)
2024-04-25 11:32:47 +01:00
break;
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
case _BUILD_SLICE_r01: {
CHECK_CURRENT_CACHED_VALUES(0);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-128914: Remove all but one conditional stack effects (GH-129226) * Remove all 'if (0)' and 'if (1)' conditional stack effects * Use array instead of conditional for BUILD_SLICE args * Refactor LOAD_GLOBAL to use a common conditional uop * Remove conditional stack effects from LOAD_ATTR specializations * Replace conditional stack effects in LOAD_ATTR with a 0 or 1 sized array. * Remove conditional stack effects from CALL_FUNCTION_EX
2025-01-27 16:24:48 +00:00
_PyStackRef *args;
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
_PyStackRef slice;
GH-111485: Factor out tier 2 code generation from the rest of the interpreter code generator (GH-112968)
2023-12-12 12:12:17 +00:00
oparg = CURRENT_OPARG();
GH-128914: Remove all but one conditional stack effects (GH-129226) * Remove all 'if (0)' and 'if (1)' conditional stack effects * Use array instead of conditional for BUILD_SLICE args * Refactor LOAD_GLOBAL to use a common conditional uop * Remove conditional stack effects from LOAD_ATTR specializations * Replace conditional stack effects in LOAD_ATTR with a 0 or 1 sized array. * Remove conditional stack effects from CALL_FUNCTION_EX
2025-01-27 16:24:48 +00:00
args = &stack_pointer[-oparg];
PyObject *start_o = PyStackRef_AsPyObjectBorrow(args[0]);
PyObject *stop_o = PyStackRef_AsPyObjectBorrow(args[1]);
PyObject *step_o = oparg == 3 ? PyStackRef_AsPyObjectBorrow(args[2]) : NULL;
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
PyObject *slice_o = PySlice_New(start_o, stop_o, step_o);
GH-128682: Account for escapes in `DECREF_INPUTS` (GH-129953) * Handle escapes in DECREF_INPUTS * Mark a few more functions as escaping * Replace DECREF_INPUTS with PyStackRef_CLOSE where possible
2025-02-12 17:44:59 +00:00
_PyFrame_SetStackPointer(frame, stack_pointer);
_PyStackRef tmp;
GH-128914: Remove all but one conditional stack effects (GH-129226) * Remove all 'if (0)' and 'if (1)' conditional stack effects * Use array instead of conditional for BUILD_SLICE args * Refactor LOAD_GLOBAL to use a common conditional uop * Remove conditional stack effects from LOAD_ATTR specializations * Replace conditional stack effects in LOAD_ATTR with a 0 or 1 sized array. * Remove conditional stack effects from CALL_FUNCTION_EX
2025-01-27 16:24:48 +00:00
for (int _i = oparg; --_i >= 0;) {
GH-128682: Account for escapes in `DECREF_INPUTS` (GH-129953) * Handle escapes in DECREF_INPUTS * Mark a few more functions as escaping * Replace DECREF_INPUTS with PyStackRef_CLOSE where possible
2025-02-12 17:44:59 +00:00
tmp = args[_i];
args[_i] = PyStackRef_NULL;
PyStackRef_CLOSE(tmp);
GH-128914: Remove all but one conditional stack effects (GH-129226) * Remove all 'if (0)' and 'if (1)' conditional stack effects * Use array instead of conditional for BUILD_SLICE args * Refactor LOAD_GLOBAL to use a common conditional uop * Remove conditional stack effects from LOAD_ATTR specializations * Replace conditional stack effects in LOAD_ATTR with a 0 or 1 sized array. * Remove conditional stack effects from CALL_FUNCTION_EX
2025-01-27 16:24:48 +00:00
}
GH-128682: Account for escapes in `DECREF_INPUTS` (GH-129953) * Handle escapes in DECREF_INPUTS * Mark a few more functions as escaping * Replace DECREF_INPUTS with PyStackRef_CLOSE where possible
2025-02-12 17:44:59 +00:00
stack_pointer = _PyFrame_GetStackPointer(frame);
stack_pointer += -oparg;
GH-141794: Limit size of generated machine code. (GH-142228) * Factor out bodies of the largest uops, to reduce jit code size. * Factor out common assert, also reducing jit code size. * Limit size of jitted code for a single executor to 1MB.
2025-12-03 17:43:35 +00:00
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
GH-128682: Make `PyStackRef_CLOSE` escaping. (GH-129404)
2025-02-03 12:41:32 +00:00
if (slice_o == NULL) {
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(0);
GH-128682: Make `PyStackRef_CLOSE` escaping. (GH-129404)
2025-02-03 12:41:32 +00:00
JUMP_TO_ERROR();
}
GH-127705: Use `_PyStackRef`s in the default build. (GH-127875)
2025-03-10 14:06:56 +00:00
slice = PyStackRef_FromPyObjectStealMortal(slice_o);
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_tos_cache0 = slice;
_tos_cache1 = PyStackRef_ZERO_BITS;
_tos_cache2 = PyStackRef_ZERO_BITS;
SET_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-104584: Baby steps towards generating and executing traces (#105924) Added a new, experimental, tracing optimizer and interpreter (a.k.a. "tier 2"). This currently pessimizes, so don't use yet -- this is infrastructure so we can experiment with optimizing passes. To enable it, pass ``-Xuops`` or set ``PYTHONUOPS=1``. To get debug output, set ``PYTHONUOPSDEBUG=N`` where ``N`` is a debug level (0-4, where 0 is no debug output and 4 is excessively verbose). All of this code is likely to change dramatically before the 3.13 feature freeze. But this is a first step.
2023-06-26 19:02:57 -07:00
break;
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
case _CONVERT_VALUE_r11: {
CHECK_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
_PyStackRef value;
_PyStackRef result;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef _stack_item_0 = _tos_cache0;
GH-111485: Factor out tier 2 code generation from the rest of the interpreter code generator (GH-112968)
2023-12-12 12:12:17 +00:00
oparg = CURRENT_OPARG();
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
value = _stack_item_0;
GH-115802: JIT "small" code for Windows (GH-115964)
2024-02-29 08:11:28 -08:00
conversion_func conv_fn;
gh-104584: Baby steps towards generating and executing traces (#105924) Added a new, experimental, tracing optimizer and interpreter (a.k.a. "tier 2"). This currently pessimizes, so don't use yet -- this is infrastructure so we can experiment with optimizing passes. To enable it, pass ``-Xuops`` or set ``PYTHONUOPS=1``. To get debug output, set ``PYTHONUOPSDEBUG=N`` where ``N`` is a debug level (0-4, where 0 is no debug output and 4 is excessively verbose). All of this code is likely to change dramatically before the 3.13 feature freeze. But this is a first step.
2023-06-26 19:02:57 -07:00
assert(oparg >= FVC_STR && oparg <= FVC_ASCII);
GH-115802: JIT "small" code for Windows (GH-115964)
2024-02-29 08:11:28 -08:00
conv_fn = _PyEval_ConversionFuncs[oparg];
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
stack_pointer[0] = value;
stack_pointer += 1;
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
GH-119866: Spill the stack around escaping calls. (GH-124392) * Spill the evaluation around escaping calls in the generated interpreter and JIT. * The code generator tracks live, cached values so they can be saved to memory when needed. * Spills the stack pointer around escaping calls, so that the exact stack is visible to the cycle GC.
2024-10-07 14:56:39 +01:00
_PyFrame_SetStackPointer(frame, stack_pointer);
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
PyObject *result_o = conv_fn(PyStackRef_AsPyObjectBorrow(value));
GH-119866: Spill the stack around escaping calls. (GH-124392) * Spill the evaluation around escaping calls in the generated interpreter and JIT. * The code generator tracks live, cached values so they can be saved to memory when needed. * Spills the stack pointer around escaping calls, so that the exact stack is visible to the cycle GC.
2024-10-07 14:56:39 +01:00
stack_pointer = _PyFrame_GetStackPointer(frame);
GH-128682: Make `PyStackRef_CLOSE` escaping. (GH-129404)
2025-02-03 12:41:32 +00:00
stack_pointer += -1;
GH-141794: Limit size of generated machine code. (GH-142228) * Factor out bodies of the largest uops, to reduce jit code size. * Factor out common assert, also reducing jit code size. * Limit size of jitted code for a single executor to 1MB.
2025-12-03 17:43:35 +00:00
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
GH-128682: Make `PyStackRef_CLOSE` escaping. (GH-129404)
2025-02-03 12:41:32 +00:00
_PyFrame_SetStackPointer(frame, stack_pointer);
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
PyStackRef_CLOSE(value);
GH-128682: Make `PyStackRef_CLOSE` escaping. (GH-129404)
2025-02-03 12:41:32 +00:00
stack_pointer = _PyFrame_GetStackPointer(frame);
if (result_o == NULL) {
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(0);
GH-128682: Make `PyStackRef_CLOSE` escaping. (GH-129404)
2025-02-03 12:41:32 +00:00
JUMP_TO_ERROR();
}
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
result = PyStackRef_FromPyObjectSteal(result_o);
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_tos_cache0 = result;
_tos_cache1 = PyStackRef_ZERO_BITS;
_tos_cache2 = PyStackRef_ZERO_BITS;
SET_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-104584: Baby steps towards generating and executing traces (#105924) Added a new, experimental, tracing optimizer and interpreter (a.k.a. "tier 2"). This currently pessimizes, so don't use yet -- this is infrastructure so we can experiment with optimizing passes. To enable it, pass ``-Xuops`` or set ``PYTHONUOPS=1``. To get debug output, set ``PYTHONUOPSDEBUG=N`` where ``N`` is a debug level (0-4, where 0 is no debug output and 4 is excessively verbose). All of this code is likely to change dramatically before the 3.13 feature freeze. But this is a first step.
2023-06-26 19:02:57 -07:00
break;
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
case _FORMAT_SIMPLE_r11: {
CHECK_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
_PyStackRef value;
_PyStackRef res;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef _stack_item_0 = _tos_cache0;
value = _stack_item_0;
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
PyObject *value_o = PyStackRef_AsPyObjectBorrow(value);
if (!PyUnicode_CheckExact(value_o)) {
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
stack_pointer[0] = value;
stack_pointer += 1;
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
GH-119866: Spill the stack around escaping calls. (GH-124392) * Spill the evaluation around escaping calls in the generated interpreter and JIT. * The code generator tracks live, cached values so they can be saved to memory when needed. * Spills the stack pointer around escaping calls, so that the exact stack is visible to the cycle GC.
2024-10-07 14:56:39 +01:00
_PyFrame_SetStackPointer(frame, stack_pointer);
GH-121459: Streamline PyObject* to PyStackRef conversions by disallowing NULL pointers. (GH-124894)
2024-10-07 18:13:04 +01:00
PyObject *res_o = PyObject_Format(value_o, NULL);
GH-119866: Spill the stack around escaping calls. (GH-124392) * Spill the evaluation around escaping calls in the generated interpreter and JIT. * The code generator tracks live, cached values so they can be saved to memory when needed. * Spills the stack pointer around escaping calls, so that the exact stack is visible to the cycle GC.
2024-10-07 14:56:39 +01:00
stack_pointer = _PyFrame_GetStackPointer(frame);
GH-128682: Make `PyStackRef_CLOSE` escaping. (GH-129404)
2025-02-03 12:41:32 +00:00
stack_pointer += -1;
GH-141794: Limit size of generated machine code. (GH-142228) * Factor out bodies of the largest uops, to reduce jit code size. * Factor out common assert, also reducing jit code size. * Limit size of jitted code for a single executor to 1MB.
2025-12-03 17:43:35 +00:00
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
GH-128682: Make `PyStackRef_CLOSE` escaping. (GH-129404)
2025-02-03 12:41:32 +00:00
_PyFrame_SetStackPointer(frame, stack_pointer);
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
PyStackRef_CLOSE(value);
GH-128682: Make `PyStackRef_CLOSE` escaping. (GH-129404)
2025-02-03 12:41:32 +00:00
stack_pointer = _PyFrame_GetStackPointer(frame);
if (res_o == NULL) {
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(0);
GH-128682: Make `PyStackRef_CLOSE` escaping. (GH-129404)
2025-02-03 12:41:32 +00:00
JUMP_TO_ERROR();
}
GH-121459: Streamline PyObject* to PyStackRef conversions by disallowing NULL pointers. (GH-124894)
2024-10-07 18:13:04 +01:00
res = PyStackRef_FromPyObjectSteal(res_o);
gh-104584: Baby steps towards generating and executing traces (#105924) Added a new, experimental, tracing optimizer and interpreter (a.k.a. "tier 2"). This currently pessimizes, so don't use yet -- this is infrastructure so we can experiment with optimizing passes. To enable it, pass ``-Xuops`` or set ``PYTHONUOPS=1``. To get debug output, set ``PYTHONUOPSDEBUG=N`` where ``N`` is a debug level (0-4, where 0 is no debug output and 4 is excessively verbose). All of this code is likely to change dramatically before the 3.13 feature freeze. But this is a first step.
2023-06-26 19:02:57 -07:00
}
else {
res = value;
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_tos_cache0 = res;
_tos_cache1 = PyStackRef_ZERO_BITS;
_tos_cache2 = PyStackRef_ZERO_BITS;
SET_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _FORMAT_WITH_SPEC_r21: {
CHECK_CURRENT_CACHED_VALUES(2);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef fmt_spec;
_PyStackRef value;
_PyStackRef res;
_PyStackRef _stack_item_0 = _tos_cache0;
_PyStackRef _stack_item_1 = _tos_cache1;
fmt_spec = _stack_item_1;
value = _stack_item_0;
stack_pointer[0] = value;
stack_pointer[1] = fmt_spec;
stack_pointer += 2;
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
_PyFrame_SetStackPointer(frame, stack_pointer);
PyObject *res_o = PyObject_Format(PyStackRef_AsPyObjectBorrow(value), PyStackRef_AsPyObjectBorrow(fmt_spec));
_PyStackRef tmp = fmt_spec;
fmt_spec = PyStackRef_NULL;
stack_pointer[-1] = fmt_spec;
PyStackRef_CLOSE(tmp);
tmp = value;
value = PyStackRef_NULL;
stack_pointer[-2] = value;
PyStackRef_CLOSE(tmp);
stack_pointer = _PyFrame_GetStackPointer(frame);
stack_pointer += -2;
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
if (res_o == NULL) {
SET_CURRENT_CACHED_VALUES(0);
JUMP_TO_ERROR();
}
res = PyStackRef_FromPyObjectSteal(res_o);
_tos_cache0 = res;
_tos_cache1 = PyStackRef_ZERO_BITS;
_tos_cache2 = PyStackRef_ZERO_BITS;
SET_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _COPY_1_r02: {
CHECK_CURRENT_CACHED_VALUES(0);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef bottom;
_PyStackRef top;
bottom = stack_pointer[-1];
top = PyStackRef_DUP(bottom);
_tos_cache1 = top;
_tos_cache0 = bottom;
SET_CURRENT_CACHED_VALUES(2);
stack_pointer += -1;
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _COPY_1_r12: {
CHECK_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef bottom;
_PyStackRef top;
_PyStackRef _stack_item_0 = _tos_cache0;
bottom = _stack_item_0;
top = PyStackRef_DUP(bottom);
_tos_cache1 = top;
_tos_cache0 = bottom;
SET_CURRENT_CACHED_VALUES(2);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _COPY_1_r23: {
CHECK_CURRENT_CACHED_VALUES(2);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef bottom;
_PyStackRef top;
_PyStackRef _stack_item_0 = _tos_cache0;
_PyStackRef _stack_item_1 = _tos_cache1;
bottom = _stack_item_1;
top = PyStackRef_DUP(bottom);
_tos_cache2 = top;
_tos_cache1 = bottom;
_tos_cache0 = _stack_item_0;
SET_CURRENT_CACHED_VALUES(3);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _COPY_2_r03: {
CHECK_CURRENT_CACHED_VALUES(0);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef bottom;
_PyStackRef top;
bottom = stack_pointer[-2];
top = PyStackRef_DUP(bottom);
_tos_cache2 = top;
_tos_cache1 = stack_pointer[-1];
_tos_cache0 = bottom;
SET_CURRENT_CACHED_VALUES(3);
stack_pointer += -2;
GH-141794: Limit size of generated machine code. (GH-142228) * Factor out bodies of the largest uops, to reduce jit code size. * Factor out common assert, also reducing jit code size. * Limit size of jitted code for a single executor to 1MB.
2025-12-03 17:43:35 +00:00
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-104584: Baby steps towards generating and executing traces (#105924) Added a new, experimental, tracing optimizer and interpreter (a.k.a. "tier 2"). This currently pessimizes, so don't use yet -- this is infrastructure so we can experiment with optimizing passes. To enable it, pass ``-Xuops`` or set ``PYTHONUOPS=1``. To get debug output, set ``PYTHONUOPSDEBUG=N`` where ``N`` is a debug level (0-4, where 0 is no debug output and 4 is excessively verbose). All of this code is likely to change dramatically before the 3.13 feature freeze. But this is a first step.
2023-06-26 19:02:57 -07:00
break;
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
case _COPY_2_r13: {
CHECK_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef bottom;
_PyStackRef top;
_PyStackRef _stack_item_0 = _tos_cache0;
bottom = stack_pointer[-1];
top = PyStackRef_DUP(bottom);
_tos_cache2 = top;
_tos_cache1 = _stack_item_0;
_tos_cache0 = bottom;
SET_CURRENT_CACHED_VALUES(3);
stack_pointer += -1;
GH-141794: Limit size of generated machine code. (GH-142228) * Factor out bodies of the largest uops, to reduce jit code size. * Factor out common assert, also reducing jit code size. * Limit size of jitted code for a single executor to 1MB.
2025-12-03 17:43:35 +00:00
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-104584: Baby steps towards generating and executing traces (#105924) Added a new, experimental, tracing optimizer and interpreter (a.k.a. "tier 2"). This currently pessimizes, so don't use yet -- this is infrastructure so we can experiment with optimizing passes. To enable it, pass ``-Xuops`` or set ``PYTHONUOPS=1``. To get debug output, set ``PYTHONUOPSDEBUG=N`` where ``N`` is a debug level (0-4, where 0 is no debug output and 4 is excessively verbose). All of this code is likely to change dramatically before the 3.13 feature freeze. But this is a first step.
2023-06-26 19:02:57 -07:00
break;
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
case _COPY_2_r23: {
CHECK_CURRENT_CACHED_VALUES(2);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Support limited scalar replacement for replicated uops in the JIT code generator. (GH-135563) * Use it to support efficient specializations of COPY and SWAP in the JIT.
2025-06-17 13:43:09 +01:00
_PyStackRef bottom;
_PyStackRef top;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef _stack_item_0 = _tos_cache0;
_PyStackRef _stack_item_1 = _tos_cache1;
bottom = _stack_item_0;
GH-135379: Support limited scalar replacement for replicated uops in the JIT code generator. (GH-135563) * Use it to support efficient specializations of COPY and SWAP in the JIT.
2025-06-17 13:43:09 +01:00
top = PyStackRef_DUP(bottom);
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_tos_cache2 = top;
_tos_cache1 = _stack_item_1;
_tos_cache0 = bottom;
SET_CURRENT_CACHED_VALUES(3);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _COPY_3_r03: {
CHECK_CURRENT_CACHED_VALUES(0);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef bottom;
_PyStackRef top;
bottom = stack_pointer[-3];
top = PyStackRef_DUP(bottom);
_tos_cache2 = top;
_tos_cache1 = stack_pointer[-1];
_tos_cache0 = stack_pointer[-2];
SET_CURRENT_CACHED_VALUES(3);
stack_pointer += -2;
GH-141794: Limit size of generated machine code. (GH-142228) * Factor out bodies of the largest uops, to reduce jit code size. * Factor out common assert, also reducing jit code size. * Limit size of jitted code for a single executor to 1MB.
2025-12-03 17:43:35 +00:00
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Support limited scalar replacement for replicated uops in the JIT code generator. (GH-135563) * Use it to support efficient specializations of COPY and SWAP in the JIT.
2025-06-17 13:43:09 +01:00
break;
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
case _COPY_3_r13: {
CHECK_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Support limited scalar replacement for replicated uops in the JIT code generator. (GH-135563) * Use it to support efficient specializations of COPY and SWAP in the JIT.
2025-06-17 13:43:09 +01:00
_PyStackRef bottom;
_PyStackRef top;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef _stack_item_0 = _tos_cache0;
GH-135379: Support limited scalar replacement for replicated uops in the JIT code generator. (GH-135563) * Use it to support efficient specializations of COPY and SWAP in the JIT.
2025-06-17 13:43:09 +01:00
bottom = stack_pointer[-2];
top = PyStackRef_DUP(bottom);
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_tos_cache2 = top;
_tos_cache1 = _stack_item_0;
_tos_cache0 = stack_pointer[-1];
SET_CURRENT_CACHED_VALUES(3);
stack_pointer += -1;
GH-141794: Limit size of generated machine code. (GH-142228) * Factor out bodies of the largest uops, to reduce jit code size. * Factor out common assert, also reducing jit code size. * Limit size of jitted code for a single executor to 1MB.
2025-12-03 17:43:35 +00:00
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Support limited scalar replacement for replicated uops in the JIT code generator. (GH-135563) * Use it to support efficient specializations of COPY and SWAP in the JIT.
2025-06-17 13:43:09 +01:00
break;
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
case _COPY_3_r23: {
CHECK_CURRENT_CACHED_VALUES(2);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Support limited scalar replacement for replicated uops in the JIT code generator. (GH-135563) * Use it to support efficient specializations of COPY and SWAP in the JIT.
2025-06-17 13:43:09 +01:00
_PyStackRef bottom;
_PyStackRef top;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef _stack_item_0 = _tos_cache0;
_PyStackRef _stack_item_1 = _tos_cache1;
bottom = stack_pointer[-1];
top = PyStackRef_DUP(bottom);
_tos_cache2 = top;
_tos_cache1 = _stack_item_1;
_tos_cache0 = _stack_item_0;
SET_CURRENT_CACHED_VALUES(3);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _COPY_3_r33: {
CHECK_CURRENT_CACHED_VALUES(3);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef bottom;
_PyStackRef top;
_PyStackRef _stack_item_0 = _tos_cache0;
_PyStackRef _stack_item_1 = _tos_cache1;
_PyStackRef _stack_item_2 = _tos_cache2;
bottom = _stack_item_0;
GH-135379: Support limited scalar replacement for replicated uops in the JIT code generator. (GH-135563) * Use it to support efficient specializations of COPY and SWAP in the JIT.
2025-06-17 13:43:09 +01:00
top = PyStackRef_DUP(bottom);
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_tos_cache2 = top;
_tos_cache1 = _stack_item_2;
_tos_cache0 = _stack_item_1;
SET_CURRENT_CACHED_VALUES(3);
stack_pointer[0] = bottom;
GH-135379: Support limited scalar replacement for replicated uops in the JIT code generator. (GH-135563) * Use it to support efficient specializations of COPY and SWAP in the JIT.
2025-06-17 13:43:09 +01:00
stack_pointer += 1;
GH-141794: Limit size of generated machine code. (GH-142228) * Factor out bodies of the largest uops, to reduce jit code size. * Factor out common assert, also reducing jit code size. * Limit size of jitted code for a single executor to 1MB.
2025-12-03 17:43:35 +00:00
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Support limited scalar replacement for replicated uops in the JIT code generator. (GH-135563) * Use it to support efficient specializations of COPY and SWAP in the JIT.
2025-06-17 13:43:09 +01:00
break;
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
case _COPY_r01: {
CHECK_CURRENT_CACHED_VALUES(0);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
_PyStackRef bottom;
_PyStackRef top;
GH-111485: Factor out tier 2 code generation from the rest of the interpreter code generator (GH-112968)
2023-12-12 12:12:17 +00:00
oparg = CURRENT_OPARG();
gh-106812: Refactor cases_generator to allow uops with array stack effects (#107564) Introducing a new file, stacking.py, that takes over several responsibilities related to symbolic evaluation of push/pop operations, with more generality.
2023-08-04 09:35:56 -07:00
bottom = stack_pointer[-1 - (oparg-1)];
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
top = PyStackRef_DUP(bottom);
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_tos_cache0 = top;
SET_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-104584: Baby steps towards generating and executing traces (#105924) Added a new, experimental, tracing optimizer and interpreter (a.k.a. "tier 2"). This currently pessimizes, so don't use yet -- this is infrastructure so we can experiment with optimizing passes. To enable it, pass ``-Xuops`` or set ``PYTHONUOPS=1``. To get debug output, set ``PYTHONUOPSDEBUG=N`` where ``N`` is a debug level (0-4, where 0 is no debug output and 4 is excessively verbose). All of this code is likely to change dramatically before the 3.13 feature freeze. But this is a first step.
2023-06-26 19:02:57 -07:00
break;
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
case _BINARY_OP_r21: {
CHECK_CURRENT_CACHED_VALUES(2);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
_PyStackRef rhs;
_PyStackRef lhs;
_PyStackRef res;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef _stack_item_0 = _tos_cache0;
_PyStackRef _stack_item_1 = _tos_cache1;
GH-111485: Factor out tier 2 code generation from the rest of the interpreter code generator (GH-112968)
2023-12-12 12:12:17 +00:00
oparg = CURRENT_OPARG();
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
rhs = _stack_item_1;
lhs = _stack_item_0;
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
PyObject *lhs_o = PyStackRef_AsPyObjectBorrow(lhs);
PyObject *rhs_o = PyStackRef_AsPyObjectBorrow(rhs);
GH-106701: Move _PyUopExecute to Python/executor.c (GH-106924)
2023-07-20 13:37:19 -07:00
assert(_PyEval_BinaryOps[oparg]);
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
stack_pointer[0] = lhs;
stack_pointer[1] = rhs;
stack_pointer += 2;
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
GH-119866: Spill the stack around escaping calls. (GH-124392) * Spill the evaluation around escaping calls in the generated interpreter and JIT. * The code generator tracks live, cached values so they can be saved to memory when needed. * Spills the stack pointer around escaping calls, so that the exact stack is visible to the cycle GC.
2024-10-07 14:56:39 +01:00
_PyFrame_SetStackPointer(frame, stack_pointer);
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
PyObject *res_o = _PyEval_BinaryOps[oparg](lhs_o, rhs_o);
GH-119866: Spill the stack around escaping calls. (GH-124392) * Spill the evaluation around escaping calls in the generated interpreter and JIT. * The code generator tracks live, cached values so they can be saved to memory when needed. * Spills the stack pointer around escaping calls, so that the exact stack is visible to the cycle GC.
2024-10-07 14:56:39 +01:00
stack_pointer = _PyFrame_GetStackPointer(frame);
GH-128682: Make `PyStackRef_CLOSE` escaping. (GH-129404)
2025-02-03 12:41:32 +00:00
if (res_o == NULL) {
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(0);
GH-128682: Make `PyStackRef_CLOSE` escaping. (GH-129404)
2025-02-03 12:41:32 +00:00
JUMP_TO_ERROR();
}
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
res = PyStackRef_FromPyObjectSteal(res_o);
GH-128682: Account for escapes in `DECREF_INPUTS` (GH-129953) * Handle escapes in DECREF_INPUTS * Mark a few more functions as escaping * Replace DECREF_INPUTS with PyStackRef_CLOSE where possible
2025-02-12 17:44:59 +00:00
_PyFrame_SetStackPointer(frame, stack_pointer);
_PyStackRef tmp = lhs;
lhs = res;
stack_pointer[-2] = lhs;
PyStackRef_CLOSE(tmp);
GH-131729: Code-gen better liveness analysis (GH-131732) * Rename 'defined' attribute to 'in_local' to more accurately reflect how it is used * Make death of variables explicit even for array variables. * Convert in_memory from boolean to stack offset * Don't apply liveness analysis to optimizer generated code * Fix RETURN_VALUE in optimizer
2025-03-26 15:21:35 +00:00
tmp = rhs;
rhs = PyStackRef_NULL;
stack_pointer[-1] = rhs;
PyStackRef_CLOSE(tmp);
GH-128682: Account for escapes in `DECREF_INPUTS` (GH-129953) * Handle escapes in DECREF_INPUTS * Mark a few more functions as escaping * Replace DECREF_INPUTS with PyStackRef_CLOSE where possible
2025-02-12 17:44:59 +00:00
stack_pointer = _PyFrame_GetStackPointer(frame);
GH-111485: Factor out tier 2 code generation from the rest of the interpreter code generator (GH-112968)
2023-12-12 12:12:17 +00:00
stack_pointer += -1;
GH-141794: Limit size of generated machine code. (GH-142228) * Factor out bodies of the largest uops, to reduce jit code size. * Factor out common assert, also reducing jit code size. * Limit size of jitted code for a single executor to 1MB.
2025-12-03 17:43:35 +00:00
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_tos_cache0 = res;
_tos_cache1 = PyStackRef_ZERO_BITS;
_tos_cache2 = PyStackRef_ZERO_BITS;
SET_CURRENT_CACHED_VALUES(1);
stack_pointer += -1;
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-104584: Allow unspecialized instructions in superblocks (#106497) This adds several of unspecialized opcodes to superblocks: TO_BOOL, BINARY_SUBSCR, STORE_SUBSCR, UNPACK_SEQUENCE, LOAD_GLOBAL, LOAD_ATTR, COMPARE_OP, BINARY_OP. While we may not want that eventually, for now this helps finding bugs. There is a rudimentary test checking for UNPACK_SEQUENCE. Once we're ready to undo this, that would be simple: just replace the call to variable_used_unspecialized with a call to variable_used (as shown in a comment). Or add individual opcdes to FORBIDDEN_NAMES_IN_UOPS.
2023-07-07 11:03:27 -07:00
break;
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
case _SWAP_2_r02: {
CHECK_CURRENT_CACHED_VALUES(0);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Support limited scalar replacement for replicated uops in the JIT code generator. (GH-135563) * Use it to support efficient specializations of COPY and SWAP in the JIT.
2025-06-17 13:43:09 +01:00
_PyStackRef top;
_PyStackRef bottom;
top = stack_pointer[-1];
bottom = stack_pointer[-2];
_PyStackRef temp = bottom;
bottom = top;
top = temp;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_tos_cache1 = top;
_tos_cache0 = bottom;
SET_CURRENT_CACHED_VALUES(2);
stack_pointer += -2;
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _SWAP_2_r12: {
CHECK_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef top;
_PyStackRef bottom;
_PyStackRef _stack_item_0 = _tos_cache0;
top = _stack_item_0;
bottom = stack_pointer[-1];
_PyStackRef temp = bottom;
bottom = top;
top = temp;
_tos_cache1 = top;
_tos_cache0 = bottom;
SET_CURRENT_CACHED_VALUES(2);
stack_pointer += -1;
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _SWAP_2_r22: {
CHECK_CURRENT_CACHED_VALUES(2);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef top;
_PyStackRef bottom;
_PyStackRef _stack_item_0 = _tos_cache0;
_PyStackRef _stack_item_1 = _tos_cache1;
top = _stack_item_1;
bottom = _stack_item_0;
_PyStackRef temp = bottom;
bottom = top;
top = temp;
_tos_cache1 = top;
_tos_cache0 = bottom;
SET_CURRENT_CACHED_VALUES(2);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _SWAP_2_r33: {
CHECK_CURRENT_CACHED_VALUES(3);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef top;
_PyStackRef bottom;
_PyStackRef _stack_item_0 = _tos_cache0;
_PyStackRef _stack_item_1 = _tos_cache1;
_PyStackRef _stack_item_2 = _tos_cache2;
top = _stack_item_2;
bottom = _stack_item_1;
_PyStackRef temp = bottom;
bottom = top;
top = temp;
_tos_cache2 = top;
_tos_cache1 = bottom;
_tos_cache0 = _stack_item_0;
SET_CURRENT_CACHED_VALUES(3);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Support limited scalar replacement for replicated uops in the JIT code generator. (GH-135563) * Use it to support efficient specializations of COPY and SWAP in the JIT.
2025-06-17 13:43:09 +01:00
break;
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
case _SWAP_3_r03: {
CHECK_CURRENT_CACHED_VALUES(0);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Support limited scalar replacement for replicated uops in the JIT code generator. (GH-135563) * Use it to support efficient specializations of COPY and SWAP in the JIT.
2025-06-17 13:43:09 +01:00
_PyStackRef top;
_PyStackRef bottom;
top = stack_pointer[-1];
bottom = stack_pointer[-3];
_PyStackRef temp = bottom;
bottom = top;
top = temp;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_tos_cache2 = top;
_tos_cache1 = stack_pointer[-2];
_tos_cache0 = bottom;
SET_CURRENT_CACHED_VALUES(3);
stack_pointer += -3;
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _SWAP_3_r13: {
CHECK_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef top;
_PyStackRef bottom;
_PyStackRef _stack_item_0 = _tos_cache0;
top = _stack_item_0;
bottom = stack_pointer[-2];
_PyStackRef temp = bottom;
bottom = top;
top = temp;
_tos_cache2 = top;
_tos_cache1 = stack_pointer[-1];
_tos_cache0 = bottom;
SET_CURRENT_CACHED_VALUES(3);
stack_pointer += -2;
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _SWAP_3_r23: {
CHECK_CURRENT_CACHED_VALUES(2);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef top;
_PyStackRef bottom;
_PyStackRef _stack_item_0 = _tos_cache0;
_PyStackRef _stack_item_1 = _tos_cache1;
top = _stack_item_1;
bottom = stack_pointer[-1];
_PyStackRef temp = bottom;
bottom = top;
top = temp;
_tos_cache2 = top;
_tos_cache1 = _stack_item_0;
_tos_cache0 = bottom;
SET_CURRENT_CACHED_VALUES(3);
stack_pointer += -1;
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _SWAP_3_r33: {
CHECK_CURRENT_CACHED_VALUES(3);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef top;
_PyStackRef bottom;
_PyStackRef _stack_item_0 = _tos_cache0;
_PyStackRef _stack_item_1 = _tos_cache1;
_PyStackRef _stack_item_2 = _tos_cache2;
top = _stack_item_2;
bottom = _stack_item_0;
_PyStackRef temp = bottom;
bottom = top;
top = temp;
_tos_cache2 = top;
_tos_cache1 = _stack_item_1;
_tos_cache0 = bottom;
SET_CURRENT_CACHED_VALUES(3);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Support limited scalar replacement for replicated uops in the JIT code generator. (GH-135563) * Use it to support efficient specializations of COPY and SWAP in the JIT.
2025-06-17 13:43:09 +01:00
break;
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
case _SWAP_r11: {
CHECK_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-131498: Cases generator: Allow input and 'peek' variables to be modified (GH-132506)
2025-04-14 12:19:53 +01:00
_PyStackRef top;
_PyStackRef bottom;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef _stack_item_0 = _tos_cache0;
GH-111485: Factor out tier 2 code generation from the rest of the interpreter code generator (GH-112968)
2023-12-12 12:12:17 +00:00
oparg = CURRENT_OPARG();
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
top = _stack_item_0;
bottom = stack_pointer[-1 - (oparg-2)];
GH-131498: Cases generator: Allow input and 'peek' variables to be modified (GH-132506)
2025-04-14 12:19:53 +01:00
_PyStackRef temp = bottom;
bottom = top;
top = temp;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_tos_cache0 = top;
SET_CURRENT_CACHED_VALUES(1);
stack_pointer[-1 - (oparg-2)] = bottom;
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-104584: Baby steps towards generating and executing traces (#105924) Added a new, experimental, tracing optimizer and interpreter (a.k.a. "tier 2"). This currently pessimizes, so don't use yet -- this is infrastructure so we can experiment with optimizing passes. To enable it, pass ``-Xuops`` or set ``PYTHONUOPS=1``. To get debug output, set ``PYTHONUOPSDEBUG=N`` where ``N`` is a debug level (0-4, where 0 is no debug output and 4 is excessively verbose). All of this code is likely to change dramatically before the 3.13 feature freeze. But this is a first step.
2023-06-26 19:02:57 -07:00
break;
}
gh-106701: Move the hand-written Tier 2 uops to bytecodes.c (#106702) This moves EXIT_TRACE, SAVE_IP, JUMP_TO_TOP, and _POP_JUMP_IF_{FALSE,TRUE} from ceval.c to bytecodes.c. They are no less special than before, but this way they are discoverable o the copy-and-patch tooling.
2023-07-13 12:14:51 -07:00
GH-121131: Clean up and fix some instrumented instructions. (GH-121132) * Add support for 'prev_instr' to code generator and refactor some INSTRUMENTED instructions
2024-07-26 12:24:12 +01:00
/* _INSTRUMENTED_LINE is not a viable micro-op for tier 2 because it is instrumented */
GH-116422: Tier2 hot/cold splitting (GH-116813) Splits the "cold" path, deopts and exits, from the "hot" path, reducing the size of most jitted instructions, at the cost of slower exits.
2024-03-26 09:35:11 +00:00
/* _INSTRUMENTED_INSTRUCTION is not a viable micro-op for tier 2 because it is instrumented */
GH-111485: Factor out tier 2 code generation from the rest of the interpreter code generator (GH-112968)
2023-12-12 12:12:17 +00:00
GH-116422: Tier2 hot/cold splitting (GH-116813) Splits the "cold" path, deopts and exits, from the "hot" path, reducing the size of most jitted instructions, at the cost of slower exits.
2024-03-26 09:35:11 +00:00
/* _INSTRUMENTED_JUMP_FORWARD is not a viable micro-op for tier 2 because it is instrumented */
GH-111485: Factor out tier 2 code generation from the rest of the interpreter code generator (GH-112968)
2023-12-12 12:12:17 +00:00
GH-120024: Remove `CHECK_EVAL_BREAKER` macro. (GH-122968) * Factor some instructions into micro-ops to isolate CHECK_EVAL_BREAKER for escape analysis * Eliminate CHECK_EVAL_BREAKER macro
2024-08-14 12:04:05 +01:00
/* _MONITOR_JUMP_BACKWARD is not a viable micro-op for tier 2 because it uses the 'this_instr' variable */
GH-111485: Factor out tier 2 code generation from the rest of the interpreter code generator (GH-112968)
2023-12-12 12:12:17 +00:00
GH-122548: Implement branch taken and not taken events for sys.monitoring (GH-122564)
2024-12-19 16:59:51 +00:00
/* _INSTRUMENTED_NOT_TAKEN is not a viable micro-op for tier 2 because it is instrumented */
GH-116422: Tier2 hot/cold splitting (GH-116813) Splits the "cold" path, deopts and exits, from the "hot" path, reducing the size of most jitted instructions, at the cost of slower exits.
2024-03-26 09:35:11 +00:00
/* _INSTRUMENTED_POP_JUMP_IF_TRUE is not a viable micro-op for tier 2 because it is instrumented */
GH-111485: Factor out tier 2 code generation from the rest of the interpreter code generator (GH-112968)
2023-12-12 12:12:17 +00:00
GH-116422: Tier2 hot/cold splitting (GH-116813) Splits the "cold" path, deopts and exits, from the "hot" path, reducing the size of most jitted instructions, at the cost of slower exits.
2024-03-26 09:35:11 +00:00
/* _INSTRUMENTED_POP_JUMP_IF_FALSE is not a viable micro-op for tier 2 because it is instrumented */
GH-111485: Factor out tier 2 code generation from the rest of the interpreter code generator (GH-112968)
2023-12-12 12:12:17 +00:00
GH-116422: Tier2 hot/cold splitting (GH-116813) Splits the "cold" path, deopts and exits, from the "hot" path, reducing the size of most jitted instructions, at the cost of slower exits.
2024-03-26 09:35:11 +00:00
/* _INSTRUMENTED_POP_JUMP_IF_NONE is not a viable micro-op for tier 2 because it is instrumented */
GH-111485: Factor out tier 2 code generation from the rest of the interpreter code generator (GH-112968)
2023-12-12 12:12:17 +00:00
GH-116422: Tier2 hot/cold splitting (GH-116813) Splits the "cold" path, deopts and exits, from the "hot" path, reducing the size of most jitted instructions, at the cost of slower exits.
2024-03-26 09:35:11 +00:00
/* _INSTRUMENTED_POP_JUMP_IF_NOT_NONE is not a viable micro-op for tier 2 because it is instrumented */
GH-111485: Factor out tier 2 code generation from the rest of the interpreter code generator (GH-112968)
2023-12-12 12:12:17 +00:00
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
case _GUARD_IS_TRUE_POP_r00: {
CHECK_CURRENT_CACHED_VALUES(0);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
_PyStackRef flag;
gh-106812: Refactor cases_generator to allow uops with array stack effects (#107564) Introducing a new file, stacking.py, that takes over several responsibilities related to symbolic evaluation of push/pop operations, with more generality.
2023-08-04 09:35:56 -07:00
flag = stack_pointer[-1];
gh-127022: Simplify `PyStackRef_FromPyObjectSteal` (#127024) This gets rid of the immortal check in `PyStackRef_FromPyObjectSteal()`. Overall, this improves performance about 2% in the free threading build. This also renames `PyStackRef_Is()` to `PyStackRef_IsExactly()` because the macro requires that the tag bits of the arguments match, which is only true in certain special cases.
2024-11-22 17:55:33 +00:00
int is_true = PyStackRef_IsTrue(flag);
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
if (!is_true) {
UOP_STAT_INC(uopcode, miss);
SET_CURRENT_CACHED_VALUES(0);
stack_pointer += -1;
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
JUMP_TO_JUMP_TARGET();
}
SET_CURRENT_CACHED_VALUES(0);
GH-111485: Factor out tier 2 code generation from the rest of the interpreter code generator (GH-112968)
2023-12-12 12:12:17 +00:00
stack_pointer += -1;
GH-141794: Limit size of generated machine code. (GH-142228) * Factor out bodies of the largest uops, to reduce jit code size. * Factor out common assert, also reducing jit code size. * Limit size of jitted code for a single executor to 1MB.
2025-12-03 17:43:35 +00:00
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _GUARD_IS_TRUE_POP_r10: {
CHECK_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef flag;
_PyStackRef _stack_item_0 = _tos_cache0;
flag = _stack_item_0;
int is_true = PyStackRef_IsTrue(flag);
if (!is_true) {
UOP_STAT_INC(uopcode, miss);
SET_CURRENT_CACHED_VALUES(0);
JUMP_TO_JUMP_TARGET();
}
SET_CURRENT_CACHED_VALUES(0);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _GUARD_IS_TRUE_POP_r21: {
CHECK_CURRENT_CACHED_VALUES(2);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef flag;
_PyStackRef _stack_item_0 = _tos_cache0;
_PyStackRef _stack_item_1 = _tos_cache1;
flag = _stack_item_1;
int is_true = PyStackRef_IsTrue(flag);
if (!is_true) {
UOP_STAT_INC(uopcode, miss);
_tos_cache0 = _stack_item_0;
SET_CURRENT_CACHED_VALUES(1);
JUMP_TO_JUMP_TARGET();
}
_tos_cache0 = _stack_item_0;
SET_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _GUARD_IS_TRUE_POP_r32: {
CHECK_CURRENT_CACHED_VALUES(3);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef flag;
_PyStackRef _stack_item_0 = _tos_cache0;
_PyStackRef _stack_item_1 = _tos_cache1;
_PyStackRef _stack_item_2 = _tos_cache2;
flag = _stack_item_2;
int is_true = PyStackRef_IsTrue(flag);
GH-119866: Spill the stack around escaping calls. (GH-124392) * Spill the evaluation around escaping calls in the generated interpreter and JIT. * The code generator tracks live, cached values so they can be saved to memory when needed. * Spills the stack pointer around escaping calls, so that the exact stack is visible to the cycle GC.
2024-10-07 14:56:39 +01:00
if (!is_true) {
GH-117457: Correct pystats uop "miss" counts (GH-117477)
2024-04-04 18:49:18 -04:00
UOP_STAT_INC(uopcode, miss);
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_tos_cache1 = _stack_item_1;
_tos_cache0 = _stack_item_0;
SET_CURRENT_CACHED_VALUES(2);
GH-117457: Correct pystats uop "miss" counts (GH-117477)
2024-04-04 18:49:18 -04:00
JUMP_TO_JUMP_TARGET();
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_tos_cache1 = _stack_item_1;
_tos_cache0 = _stack_item_0;
SET_CURRENT_CACHED_VALUES(2);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-106701: Move the hand-written Tier 2 uops to bytecodes.c (#106702) This moves EXIT_TRACE, SAVE_IP, JUMP_TO_TOP, and _POP_JUMP_IF_{FALSE,TRUE} from ceval.c to bytecodes.c. They are no less special than before, but this way they are discoverable o the copy-and-patch tooling.
2023-07-13 12:14:51 -07:00
break;
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
case _GUARD_IS_FALSE_POP_r00: {
CHECK_CURRENT_CACHED_VALUES(0);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
_PyStackRef flag;
gh-106812: Refactor cases_generator to allow uops with array stack effects (#107564) Introducing a new file, stacking.py, that takes over several responsibilities related to symbolic evaluation of push/pop operations, with more generality.
2023-08-04 09:35:56 -07:00
flag = stack_pointer[-1];
gh-127022: Simplify `PyStackRef_FromPyObjectSteal` (#127024) This gets rid of the immortal check in `PyStackRef_FromPyObjectSteal()`. Overall, this improves performance about 2% in the free threading build. This also renames `PyStackRef_Is()` to `PyStackRef_IsExactly()` because the macro requires that the tag bits of the arguments match, which is only true in certain special cases.
2024-11-22 17:55:33 +00:00
int is_false = PyStackRef_IsFalse(flag);
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
if (!is_false) {
UOP_STAT_INC(uopcode, miss);
SET_CURRENT_CACHED_VALUES(0);
stack_pointer += -1;
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
JUMP_TO_JUMP_TARGET();
}
SET_CURRENT_CACHED_VALUES(0);
GH-111485: Factor out tier 2 code generation from the rest of the interpreter code generator (GH-112968)
2023-12-12 12:12:17 +00:00
stack_pointer += -1;
GH-141794: Limit size of generated machine code. (GH-142228) * Factor out bodies of the largest uops, to reduce jit code size. * Factor out common assert, also reducing jit code size. * Limit size of jitted code for a single executor to 1MB.
2025-12-03 17:43:35 +00:00
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _GUARD_IS_FALSE_POP_r10: {
CHECK_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef flag;
_PyStackRef _stack_item_0 = _tos_cache0;
flag = _stack_item_0;
int is_false = PyStackRef_IsFalse(flag);
if (!is_false) {
UOP_STAT_INC(uopcode, miss);
SET_CURRENT_CACHED_VALUES(0);
JUMP_TO_JUMP_TARGET();
}
SET_CURRENT_CACHED_VALUES(0);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _GUARD_IS_FALSE_POP_r21: {
CHECK_CURRENT_CACHED_VALUES(2);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef flag;
_PyStackRef _stack_item_0 = _tos_cache0;
_PyStackRef _stack_item_1 = _tos_cache1;
flag = _stack_item_1;
int is_false = PyStackRef_IsFalse(flag);
if (!is_false) {
UOP_STAT_INC(uopcode, miss);
_tos_cache0 = _stack_item_0;
SET_CURRENT_CACHED_VALUES(1);
JUMP_TO_JUMP_TARGET();
}
_tos_cache0 = _stack_item_0;
SET_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _GUARD_IS_FALSE_POP_r32: {
CHECK_CURRENT_CACHED_VALUES(3);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef flag;
_PyStackRef _stack_item_0 = _tos_cache0;
_PyStackRef _stack_item_1 = _tos_cache1;
_PyStackRef _stack_item_2 = _tos_cache2;
flag = _stack_item_2;
int is_false = PyStackRef_IsFalse(flag);
GH-119866: Spill the stack around escaping calls. (GH-124392) * Spill the evaluation around escaping calls in the generated interpreter and JIT. * The code generator tracks live, cached values so they can be saved to memory when needed. * Spills the stack pointer around escaping calls, so that the exact stack is visible to the cycle GC.
2024-10-07 14:56:39 +01:00
if (!is_false) {
GH-117457: Correct pystats uop "miss" counts (GH-117477)
2024-04-04 18:49:18 -04:00
UOP_STAT_INC(uopcode, miss);
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_tos_cache1 = _stack_item_1;
_tos_cache0 = _stack_item_0;
SET_CURRENT_CACHED_VALUES(2);
GH-117457: Correct pystats uop "miss" counts (GH-117477)
2024-04-04 18:49:18 -04:00
JUMP_TO_JUMP_TARGET();
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_tos_cache1 = _stack_item_1;
_tos_cache0 = _stack_item_0;
SET_CURRENT_CACHED_VALUES(2);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-111848: Convert remaining jumps to deopts into tier 2 code. (GH-112045)
2023-11-14 15:30:33 +00:00
break;
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
case _GUARD_IS_NONE_POP_r00: {
CHECK_CURRENT_CACHED_VALUES(0);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
_PyStackRef val;
GH-111848: Convert remaining jumps to deopts into tier 2 code. (GH-112045)
2023-11-14 15:30:33 +00:00
val = stack_pointer[-1];
gh-127022: Simplify `PyStackRef_FromPyObjectSteal` (#127024) This gets rid of the immortal check in `PyStackRef_FromPyObjectSteal()`. Overall, this improves performance about 2% in the free threading build. This also renames `PyStackRef_Is()` to `PyStackRef_IsExactly()` because the macro requires that the tag bits of the arguments match, which is only true in certain special cases.
2024-11-22 17:55:33 +00:00
int is_none = PyStackRef_IsNone(val);
GH-119866: Spill the stack around escaping calls. (GH-124392) * Spill the evaluation around escaping calls in the generated interpreter and JIT. * The code generator tracks live, cached values so they can be saved to memory when needed. * Spills the stack pointer around escaping calls, so that the exact stack is visible to the cycle GC.
2024-10-07 14:56:39 +01:00
if (!is_none) {
stack_pointer += -1;
GH-141794: Limit size of generated machine code. (GH-142228) * Factor out bodies of the largest uops, to reduce jit code size. * Factor out common assert, also reducing jit code size. * Limit size of jitted code for a single executor to 1MB.
2025-12-03 17:43:35 +00:00
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
GH-128682: Make `PyStackRef_CLOSE` escaping. (GH-129404)
2025-02-03 12:41:32 +00:00
_PyFrame_SetStackPointer(frame, stack_pointer);
PyStackRef_CLOSE(val);
stack_pointer = _PyFrame_GetStackPointer(frame);
GH-117457: Correct pystats uop "miss" counts (GH-117477)
2024-04-04 18:49:18 -04:00
if (1) {
UOP_STAT_INC(uopcode, miss);
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(0);
GH-117457: Correct pystats uop "miss" counts (GH-117477)
2024-04-04 18:49:18 -04:00
JUMP_TO_JUMP_TARGET();
}
GH-112354: `_GUARD_IS_TRUE_POP` side-exits to target the next instruction, not themselves. (GH-114078)
2024-01-15 11:41:06 +00:00
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(0);
GH-119866: Spill the stack around escaping calls. (GH-124392) * Spill the evaluation around escaping calls in the generated interpreter and JIT. * The code generator tracks live, cached values so they can be saved to memory when needed. * Spills the stack pointer around escaping calls, so that the exact stack is visible to the cycle GC.
2024-10-07 14:56:39 +01:00
stack_pointer += -1;
GH-141794: Limit size of generated machine code. (GH-142228) * Factor out bodies of the largest uops, to reduce jit code size. * Factor out common assert, also reducing jit code size. * Limit size of jitted code for a single executor to 1MB.
2025-12-03 17:43:35 +00:00
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-111848: Convert remaining jumps to deopts into tier 2 code. (GH-112045)
2023-11-14 15:30:33 +00:00
break;
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
case _GUARD_IS_NONE_POP_r10: {
CHECK_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
_PyStackRef val;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef _stack_item_0 = _tos_cache0;
val = _stack_item_0;
int is_none = PyStackRef_IsNone(val);
if (!is_none) {
_PyFrame_SetStackPointer(frame, stack_pointer);
PyStackRef_CLOSE(val);
stack_pointer = _PyFrame_GetStackPointer(frame);
if (1) {
UOP_STAT_INC(uopcode, miss);
SET_CURRENT_CACHED_VALUES(0);
JUMP_TO_JUMP_TARGET();
}
}
SET_CURRENT_CACHED_VALUES(0);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _GUARD_IS_NONE_POP_r21: {
CHECK_CURRENT_CACHED_VALUES(2);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef val;
_PyStackRef _stack_item_0 = _tos_cache0;
_PyStackRef _stack_item_1 = _tos_cache1;
val = _stack_item_1;
int is_none = PyStackRef_IsNone(val);
if (!is_none) {
stack_pointer[0] = _stack_item_0;
stack_pointer += 1;
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
_PyFrame_SetStackPointer(frame, stack_pointer);
PyStackRef_CLOSE(val);
stack_pointer = _PyFrame_GetStackPointer(frame);
if (1) {
UOP_STAT_INC(uopcode, miss);
_tos_cache0 = _stack_item_0;
SET_CURRENT_CACHED_VALUES(1);
stack_pointer += -1;
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
JUMP_TO_JUMP_TARGET();
}
}
_tos_cache0 = _stack_item_0;
SET_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _GUARD_IS_NONE_POP_r32: {
CHECK_CURRENT_CACHED_VALUES(3);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef val;
_PyStackRef _stack_item_0 = _tos_cache0;
_PyStackRef _stack_item_1 = _tos_cache1;
_PyStackRef _stack_item_2 = _tos_cache2;
val = _stack_item_2;
int is_none = PyStackRef_IsNone(val);
if (!is_none) {
stack_pointer[0] = _stack_item_0;
stack_pointer[1] = _stack_item_1;
stack_pointer += 2;
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
_PyFrame_SetStackPointer(frame, stack_pointer);
PyStackRef_CLOSE(val);
stack_pointer = _PyFrame_GetStackPointer(frame);
if (1) {
UOP_STAT_INC(uopcode, miss);
_tos_cache1 = _stack_item_1;
_tos_cache0 = _stack_item_0;
SET_CURRENT_CACHED_VALUES(2);
stack_pointer += -2;
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
JUMP_TO_JUMP_TARGET();
}
}
_tos_cache1 = _stack_item_1;
_tos_cache0 = _stack_item_0;
SET_CURRENT_CACHED_VALUES(2);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _GUARD_IS_NOT_NONE_POP_r10: {
CHECK_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef val;
_PyStackRef _stack_item_0 = _tos_cache0;
val = _stack_item_0;
gh-127022: Simplify `PyStackRef_FromPyObjectSteal` (#127024) This gets rid of the immortal check in `PyStackRef_FromPyObjectSteal()`. Overall, this improves performance about 2% in the free threading build. This also renames `PyStackRef_Is()` to `PyStackRef_IsExactly()` because the macro requires that the tag bits of the arguments match, which is only true in certain special cases.
2024-11-22 17:55:33 +00:00
int is_none = PyStackRef_IsNone(val);
GH-128682: Make `PyStackRef_CLOSE` escaping. (GH-129404)
2025-02-03 12:41:32 +00:00
_PyFrame_SetStackPointer(frame, stack_pointer);
PyStackRef_CLOSE(val);
stack_pointer = _PyFrame_GetStackPointer(frame);
GH-119866: Spill the stack around escaping calls. (GH-124392) * Spill the evaluation around escaping calls in the generated interpreter and JIT. * The code generator tracks live, cached values so they can be saved to memory when needed. * Spills the stack pointer around escaping calls, so that the exact stack is visible to the cycle GC.
2024-10-07 14:56:39 +01:00
if (is_none) {
GH-117457: Correct pystats uop "miss" counts (GH-117477)
2024-04-04 18:49:18 -04:00
UOP_STAT_INC(uopcode, miss);
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(0);
JUMP_TO_JUMP_TARGET();
}
_tos_cache0 = PyStackRef_ZERO_BITS;
_tos_cache1 = PyStackRef_ZERO_BITS;
_tos_cache2 = PyStackRef_ZERO_BITS;
SET_CURRENT_CACHED_VALUES(0);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _JUMP_TO_TOP_r00: {
CHECK_CURRENT_CACHED_VALUES(0);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
JUMP_TO_JUMP_TARGET();
SET_CURRENT_CACHED_VALUES(0);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _SET_IP_r00: {
CHECK_CURRENT_CACHED_VALUES(0);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-140683: JIT: Improve machine code for loading smaller constants on AArch64. (GH-142511) * Use movz and movk instructions for loading 16 and 32 bit operands and oparg. * Loading of 64 bit operands is unchanged.
2025-12-11 12:33:39 +00:00
PyObject *instr_ptr = (PyObject *)CURRENT_OPERAND0_64();
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
frame->instr_ptr = (_Py_CODEUNIT *)instr_ptr;
SET_CURRENT_CACHED_VALUES(0);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _SET_IP_r11: {
CHECK_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef _stack_item_0 = _tos_cache0;
GH-140683: JIT: Improve machine code for loading smaller constants on AArch64. (GH-142511) * Use movz and movk instructions for loading 16 and 32 bit operands and oparg. * Loading of 64 bit operands is unchanged.
2025-12-11 12:33:39 +00:00
PyObject *instr_ptr = (PyObject *)CURRENT_OPERAND0_64();
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
frame->instr_ptr = (_Py_CODEUNIT *)instr_ptr;
_tos_cache0 = _stack_item_0;
SET_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _SET_IP_r22: {
CHECK_CURRENT_CACHED_VALUES(2);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef _stack_item_0 = _tos_cache0;
_PyStackRef _stack_item_1 = _tos_cache1;
GH-140683: JIT: Improve machine code for loading smaller constants on AArch64. (GH-142511) * Use movz and movk instructions for loading 16 and 32 bit operands and oparg. * Loading of 64 bit operands is unchanged.
2025-12-11 12:33:39 +00:00
PyObject *instr_ptr = (PyObject *)CURRENT_OPERAND0_64();
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
frame->instr_ptr = (_Py_CODEUNIT *)instr_ptr;
_tos_cache1 = _stack_item_1;
_tos_cache0 = _stack_item_0;
SET_CURRENT_CACHED_VALUES(2);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _SET_IP_r33: {
CHECK_CURRENT_CACHED_VALUES(3);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef _stack_item_0 = _tos_cache0;
_PyStackRef _stack_item_1 = _tos_cache1;
_PyStackRef _stack_item_2 = _tos_cache2;
GH-140683: JIT: Improve machine code for loading smaller constants on AArch64. (GH-142511) * Use movz and movk instructions for loading 16 and 32 bit operands and oparg. * Loading of 64 bit operands is unchanged.
2025-12-11 12:33:39 +00:00
PyObject *instr_ptr = (PyObject *)CURRENT_OPERAND0_64();
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
frame->instr_ptr = (_Py_CODEUNIT *)instr_ptr;
_tos_cache2 = _stack_item_2;
_tos_cache1 = _stack_item_1;
_tos_cache0 = _stack_item_0;
SET_CURRENT_CACHED_VALUES(3);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _CHECK_STACK_SPACE_OPERAND_r00: {
CHECK_CURRENT_CACHED_VALUES(0);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-140683: JIT: Improve machine code for loading smaller constants on AArch64. (GH-142511) * Use movz and movk instructions for loading 16 and 32 bit operands and oparg. * Loading of 64 bit operands is unchanged.
2025-12-11 12:33:39 +00:00
uint32_t framesize = (uint32_t)CURRENT_OPERAND0_32();
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
assert(framesize <= INT_MAX);
if (!_PyThreadState_HasStackSpace(tstate, framesize)) {
UOP_STAT_INC(uopcode, miss);
SET_CURRENT_CACHED_VALUES(0);
JUMP_TO_JUMP_TARGET();
}
if (tstate->py_recursion_remaining <= 1) {
UOP_STAT_INC(uopcode, miss);
SET_CURRENT_CACHED_VALUES(0);
GH-117457: Correct pystats uop "miss" counts (GH-117477)
2024-04-04 18:49:18 -04:00
JUMP_TO_JUMP_TARGET();
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(0);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-106701: Move the hand-written Tier 2 uops to bytecodes.c (#106702) This moves EXIT_TRACE, SAVE_IP, JUMP_TO_TOP, and _POP_JUMP_IF_{FALSE,TRUE} from ceval.c to bytecodes.c. They are no less special than before, but this way they are discoverable o the copy-and-patch tooling.
2023-07-13 12:14:51 -07:00
break;
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
case _CHECK_STACK_SPACE_OPERAND_r11: {
CHECK_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef _stack_item_0 = _tos_cache0;
GH-140683: JIT: Improve machine code for loading smaller constants on AArch64. (GH-142511) * Use movz and movk instructions for loading 16 and 32 bit operands and oparg. * Loading of 64 bit operands is unchanged.
2025-12-11 12:33:39 +00:00
uint32_t framesize = (uint32_t)CURRENT_OPERAND0_32();
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
assert(framesize <= INT_MAX);
if (!_PyThreadState_HasStackSpace(tstate, framesize)) {
UOP_STAT_INC(uopcode, miss);
GH-142718: JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. (GH-142762) JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. * Make sure that stack is in correct state at side exits with TOS cached values * Simplify choice of cached items for side exits
2025-12-15 19:01:29 +00:00
_tos_cache0 = _stack_item_0;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(1);
JUMP_TO_JUMP_TARGET();
}
if (tstate->py_recursion_remaining <= 1) {
UOP_STAT_INC(uopcode, miss);
GH-142718: JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. (GH-142762) JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. * Make sure that stack is in correct state at side exits with TOS cached values * Simplify choice of cached items for side exits
2025-12-15 19:01:29 +00:00
_tos_cache0 = _stack_item_0;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(1);
JUMP_TO_JUMP_TARGET();
}
_tos_cache0 = _stack_item_0;
SET_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-106701: Move the hand-written Tier 2 uops to bytecodes.c (#106702) This moves EXIT_TRACE, SAVE_IP, JUMP_TO_TOP, and _POP_JUMP_IF_{FALSE,TRUE} from ceval.c to bytecodes.c. They are no less special than before, but this way they are discoverable o the copy-and-patch tooling.
2023-07-13 12:14:51 -07:00
break;
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
case _CHECK_STACK_SPACE_OPERAND_r22: {
CHECK_CURRENT_CACHED_VALUES(2);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef _stack_item_0 = _tos_cache0;
_PyStackRef _stack_item_1 = _tos_cache1;
GH-140683: JIT: Improve machine code for loading smaller constants on AArch64. (GH-142511) * Use movz and movk instructions for loading 16 and 32 bit operands and oparg. * Loading of 64 bit operands is unchanged.
2025-12-11 12:33:39 +00:00
uint32_t framesize = (uint32_t)CURRENT_OPERAND0_32();
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
assert(framesize <= INT_MAX);
if (!_PyThreadState_HasStackSpace(tstate, framesize)) {
UOP_STAT_INC(uopcode, miss);
GH-142718: JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. (GH-142762) JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. * Make sure that stack is in correct state at side exits with TOS cached values * Simplify choice of cached items for side exits
2025-12-15 19:01:29 +00:00
_tos_cache1 = _stack_item_1;
_tos_cache0 = _stack_item_0;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(2);
JUMP_TO_JUMP_TARGET();
}
if (tstate->py_recursion_remaining <= 1) {
UOP_STAT_INC(uopcode, miss);
GH-142718: JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. (GH-142762) JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. * Make sure that stack is in correct state at side exits with TOS cached values * Simplify choice of cached items for side exits
2025-12-15 19:01:29 +00:00
_tos_cache1 = _stack_item_1;
_tos_cache0 = _stack_item_0;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(2);
JUMP_TO_JUMP_TARGET();
}
_tos_cache1 = _stack_item_1;
_tos_cache0 = _stack_item_0;
SET_CURRENT_CACHED_VALUES(2);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-109094: replace frame->prev_instr by frame->instr_ptr (#109095)
2023-10-26 14:43:10 +01:00
break;
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
case _CHECK_STACK_SPACE_OPERAND_r33: {
CHECK_CURRENT_CACHED_VALUES(3);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef _stack_item_0 = _tos_cache0;
_PyStackRef _stack_item_1 = _tos_cache1;
_PyStackRef _stack_item_2 = _tos_cache2;
GH-140683: JIT: Improve machine code for loading smaller constants on AArch64. (GH-142511) * Use movz and movk instructions for loading 16 and 32 bit operands and oparg. * Loading of 64 bit operands is unchanged.
2025-12-11 12:33:39 +00:00
uint32_t framesize = (uint32_t)CURRENT_OPERAND0_32();
gh-116168: Remove extra `_CHECK_STACK_SPACE` uops (#117242) This merges all `_CHECK_STACK_SPACE` uops in a trace into a single `_CHECK_STACK_SPACE_OPERAND` uop that checks whether there is enough stack space for all calls included in the entire trace.
2024-04-03 10:14:18 -07:00
assert(framesize <= INT_MAX);
gh-117457: Regen executor cases post PR #117477 (#117559)
2024-04-05 12:13:00 +02:00
if (!_PyThreadState_HasStackSpace(tstate, framesize)) {
UOP_STAT_INC(uopcode, miss);
GH-142718: JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. (GH-142762) JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. * Make sure that stack is in correct state at side exits with TOS cached values * Simplify choice of cached items for side exits
2025-12-15 19:01:29 +00:00
_tos_cache2 = _stack_item_2;
_tos_cache1 = _stack_item_1;
_tos_cache0 = _stack_item_0;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(3);
gh-117457: Regen executor cases post PR #117477 (#117559)
2024-04-05 12:13:00 +02:00
JUMP_TO_JUMP_TARGET();
}
if (tstate->py_recursion_remaining <= 1) {
UOP_STAT_INC(uopcode, miss);
GH-142718: JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. (GH-142762) JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. * Make sure that stack is in correct state at side exits with TOS cached values * Simplify choice of cached items for side exits
2025-12-15 19:01:29 +00:00
_tos_cache2 = _stack_item_2;
_tos_cache1 = _stack_item_1;
_tos_cache0 = _stack_item_0;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(3);
gh-117457: Regen executor cases post PR #117477 (#117559)
2024-04-05 12:13:00 +02:00
JUMP_TO_JUMP_TARGET();
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_tos_cache2 = _stack_item_2;
_tos_cache1 = _stack_item_1;
_tos_cache0 = _stack_item_0;
SET_CURRENT_CACHED_VALUES(3);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _SAVE_RETURN_OFFSET_r00: {
CHECK_CURRENT_CACHED_VALUES(0);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
oparg = CURRENT_OPARG();
#if TIER_ONE
frame->return_offset = (uint16_t)(next_instr - this_instr);
#endif
#if TIER_TWO
frame->return_offset = oparg;
#endif
SET_CURRENT_CACHED_VALUES(0);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _SAVE_RETURN_OFFSET_r11: {
CHECK_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef _stack_item_0 = _tos_cache0;
oparg = CURRENT_OPARG();
#if TIER_ONE
frame->return_offset = (uint16_t)(next_instr - this_instr);
#endif
#if TIER_TWO
frame->return_offset = oparg;
#endif
_tos_cache0 = _stack_item_0;
SET_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _SAVE_RETURN_OFFSET_r22: {
CHECK_CURRENT_CACHED_VALUES(2);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef _stack_item_0 = _tos_cache0;
_PyStackRef _stack_item_1 = _tos_cache1;
oparg = CURRENT_OPARG();
#if TIER_ONE
frame->return_offset = (uint16_t)(next_instr - this_instr);
#endif
#if TIER_TWO
frame->return_offset = oparg;
#endif
_tos_cache1 = _stack_item_1;
_tos_cache0 = _stack_item_0;
SET_CURRENT_CACHED_VALUES(2);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-116168: Remove extra `_CHECK_STACK_SPACE` uops (#117242) This merges all `_CHECK_STACK_SPACE` uops in a trace into a single `_CHECK_STACK_SPACE_OPERAND` uop that checks whether there is enough stack space for all calls included in the entire trace.
2024-04-03 10:14:18 -07:00
break;
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
case _SAVE_RETURN_OFFSET_r33: {
CHECK_CURRENT_CACHED_VALUES(3);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef _stack_item_0 = _tos_cache0;
_PyStackRef _stack_item_1 = _tos_cache1;
_PyStackRef _stack_item_2 = _tos_cache2;
gh-112287: Speed up Tier 2 (uop) interpreter a little (#112286) This makes the Tier 2 interpreter a little faster. I calculated by about 3%, though I hesitate to claim an exact number. This starts by doubling the trace size limit (to 512), making it more likely that loops fit in a trace. The rest of the approach is to only load `oparg` and `operand` in cases that use them. The code generator know when these are used. For `oparg`, it will conditionally emit ``` oparg = CURRENT_OPARG(); ``` at the top of the case block. (The `oparg` variable may be referenced multiple times by the instructions code block, so it must be in a variable.) For `operand`, it will use `CURRENT_OPERAND()` directly instead of referencing the `operand` variable, which no longer exists. (There is only one place where this will be used.)
2023-11-20 11:25:32 -08:00
oparg = CURRENT_OPARG();
gh-109094: replace frame->prev_instr by frame->instr_ptr (#109095)
2023-10-26 14:43:10 +01:00
#if TIER_ONE
GH-111485: Increment `next_instr` consistently at the start of the instruction. (GH-111486)
2023-10-31 10:09:54 +00:00
frame->return_offset = (uint16_t)(next_instr - this_instr);
gh-109094: replace frame->prev_instr by frame->instr_ptr (#109095)
2023-10-26 14:43:10 +01:00
#endif
#if TIER_TWO
frame->return_offset = oparg;
#endif
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_tos_cache2 = _stack_item_2;
_tos_cache1 = _stack_item_1;
_tos_cache0 = _stack_item_0;
SET_CURRENT_CACHED_VALUES(3);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-106701: Move the hand-written Tier 2 uops to bytecodes.c (#106702) This moves EXIT_TRACE, SAVE_IP, JUMP_TO_TOP, and _POP_JUMP_IF_{FALSE,TRUE} from ceval.c to bytecodes.c. They are no less special than before, but this way they are discoverable o the copy-and-patch tooling.
2023-07-13 12:14:51 -07:00
break;
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
case _EXIT_TRACE_r00: {
CHECK_CURRENT_CACHED_VALUES(0);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-140683: JIT: Improve machine code for loading smaller constants on AArch64. (GH-142511) * Use movz and movk instructions for loading 16 and 32 bit operands and oparg. * Loading of 64 bit operands is unchanged.
2025-12-11 12:33:39 +00:00
PyObject *exit_p = (PyObject *)CURRENT_OPERAND0_64();
GH-122294: Burn in the addresses of side exits (GH-122295)
2024-07-26 09:40:15 -07:00
_PyExitData *exit = (_PyExitData *)exit_p;
GH-116017: Get rid of _COLD_EXITs (GH-120960)
2024-07-01 13:17:40 -07:00
#if defined(Py_DEBUG) && !defined(_Py_JIT)
gh-139109: A new tracing JIT compiler frontend for CPython (GH-140310) This PR changes the current JIT model from trace projection to trace recording. Benchmarking: better pyperformance (about 1.7% overall) geomean versus current https://raw.githubusercontent.com/facebookexperimental/free-threading-benchmarking/refs/heads/main/results/bm-20251108-3.15.0a1%2B-7e2bc1d-JIT/bm-20251108-vultr-x86_64-Fidget%252dSpinner-tracing_jit-3.15.0a1%2B-7e2bc1d-vs-base.svg, 100% faster Richards on the most improved benchmark versus the current JIT. Slowdown of about 10-15% on the worst benchmark versus the current JIT. **Note: the fastest version isn't the one merged, as it relies on fixing bugs in the specializing interpreter, which is left to another PR**. The speedup in the merged version is about 1.1%. https://raw.githubusercontent.com/facebookexperimental/free-threading-benchmarking/refs/heads/main/results/bm-20251112-3.15.0a1%2B-f8a764a-JIT/bm-20251112-vultr-x86_64-Fidget%252dSpinner-tracing_jit-3.15.0a1%2B-f8a764a-vs-base.svg Stats: 50% more uops executed, 30% more traces entered the last time we ran them. It also suggests our trace lengths for a real trace recording JIT are too short, as a lot of trace too long aborts https://github.com/facebookexperimental/free-threading-benchmarking/blob/main/results/bm-20251023-3.15.0a1%2B-eb73378-CLANG%2CJIT/bm-20251023-vultr-x86_64-Fidget%252dSpinner-tracing_jit-3.15.0a1%2B-eb73378-pystats-vs-base.md . This new JIT frontend is already able to record/execute significantly more instructions than the previous JIT frontend. In this PR, we are now able to record through custom dunders, simple object creation, generators, etc. None of these were done by the old JIT frontend. Some custom dunders uops were discovered to be broken as part of this work gh-140277 The optimizer stack space check is disabled, as it's no longer valid to deal with underflow. Pros: * Ignoring the generated tracer code as it's automatically created, this is only additional 1k lines of code. The maintenance burden is handled by the DSL and code generator. * `optimizer.c` is now significantly simpler, as we don't have to do strange things to recover the bytecode from a trace. * The new JIT frontend is able to handle a lot more control-flow than the old one. * Tracing is very low overhead. We use the tail calling interpreter/computed goto interpreter to switch between tracing mode and non-tracing mode. I call this mechanism dual dispatch, as we have two dispatch tables dispatching to each other. Specialization is still enabled while tracing. * Better handling of polymorphism. We leverage the specializing interpreter for this. Cons: * (For now) requires tail calling interpreter or computed gotos. This means no Windows JIT for now :(. Not to fret, tail calling is coming soon to Windows though https://github.com/python/cpython/pull/139962 Design: * After each instruction, the `record_previous_inst` function/label is executed. This does as the name suggests. * The tracing interpreter lowers bytecode to uops directly so that it can obtain "fresh" values at the point of lowering. * The tracing version behaves nearly identical to the normal interpreter, in fact it even has specialization! This allows it to run without much of a slowdown when tracing. The actual cost of tracing is only a function call and writes to memory. * The tracing interpreter uses the specializing interpreter's deopt to naturally form the side exit chains. This allows it to side exit chain effectively, without repeating much code. We force a re-specializing when tracing a deopt. * The tracing interpreter can even handle goto errors/exceptions, but I chose to disable them for now as it's not tested. * Because we do not share interpreter dispatch, there is should be no significant slowdown to the original specializing interpreter on tailcall and computed got with JIT disabled. With JIT enabled, there might be a slowdown in the form of the JIT trying to trace. * Things that could have dynamic instruction pointer effects are guarded on. The guard deopts to a new instruction --- `_DYNAMIC_EXIT`.
2025-11-14 02:08:32 +08:00
const _Py_CODEUNIT *target = ((frame->owner == FRAME_OWNED_BY_INTERPRETER)
? _Py_INTERPRETER_TRAMPOLINE_INSTRUCTIONS_PTR : _PyFrame_GetBytecode(frame))
+ exit->target;
GH-116017: Get rid of _COLD_EXITs (GH-120960)
2024-07-01 13:17:40 -07:00
OPT_HIST(trace_uop_execution_counter, trace_run_length_hist);
gh-139109: A new tracing JIT compiler frontend for CPython (GH-140310) This PR changes the current JIT model from trace projection to trace recording. Benchmarking: better pyperformance (about 1.7% overall) geomean versus current https://raw.githubusercontent.com/facebookexperimental/free-threading-benchmarking/refs/heads/main/results/bm-20251108-3.15.0a1%2B-7e2bc1d-JIT/bm-20251108-vultr-x86_64-Fidget%252dSpinner-tracing_jit-3.15.0a1%2B-7e2bc1d-vs-base.svg, 100% faster Richards on the most improved benchmark versus the current JIT. Slowdown of about 10-15% on the worst benchmark versus the current JIT. **Note: the fastest version isn't the one merged, as it relies on fixing bugs in the specializing interpreter, which is left to another PR**. The speedup in the merged version is about 1.1%. https://raw.githubusercontent.com/facebookexperimental/free-threading-benchmarking/refs/heads/main/results/bm-20251112-3.15.0a1%2B-f8a764a-JIT/bm-20251112-vultr-x86_64-Fidget%252dSpinner-tracing_jit-3.15.0a1%2B-f8a764a-vs-base.svg Stats: 50% more uops executed, 30% more traces entered the last time we ran them. It also suggests our trace lengths for a real trace recording JIT are too short, as a lot of trace too long aborts https://github.com/facebookexperimental/free-threading-benchmarking/blob/main/results/bm-20251023-3.15.0a1%2B-eb73378-CLANG%2CJIT/bm-20251023-vultr-x86_64-Fidget%252dSpinner-tracing_jit-3.15.0a1%2B-eb73378-pystats-vs-base.md . This new JIT frontend is already able to record/execute significantly more instructions than the previous JIT frontend. In this PR, we are now able to record through custom dunders, simple object creation, generators, etc. None of these were done by the old JIT frontend. Some custom dunders uops were discovered to be broken as part of this work gh-140277 The optimizer stack space check is disabled, as it's no longer valid to deal with underflow. Pros: * Ignoring the generated tracer code as it's automatically created, this is only additional 1k lines of code. The maintenance burden is handled by the DSL and code generator. * `optimizer.c` is now significantly simpler, as we don't have to do strange things to recover the bytecode from a trace. * The new JIT frontend is able to handle a lot more control-flow than the old one. * Tracing is very low overhead. We use the tail calling interpreter/computed goto interpreter to switch between tracing mode and non-tracing mode. I call this mechanism dual dispatch, as we have two dispatch tables dispatching to each other. Specialization is still enabled while tracing. * Better handling of polymorphism. We leverage the specializing interpreter for this. Cons: * (For now) requires tail calling interpreter or computed gotos. This means no Windows JIT for now :(. Not to fret, tail calling is coming soon to Windows though https://github.com/python/cpython/pull/139962 Design: * After each instruction, the `record_previous_inst` function/label is executed. This does as the name suggests. * The tracing interpreter lowers bytecode to uops directly so that it can obtain "fresh" values at the point of lowering. * The tracing version behaves nearly identical to the normal interpreter, in fact it even has specialization! This allows it to run without much of a slowdown when tracing. The actual cost of tracing is only a function call and writes to memory. * The tracing interpreter uses the specializing interpreter's deopt to naturally form the side exit chains. This allows it to side exit chain effectively, without repeating much code. We force a re-specializing when tracing a deopt. * The tracing interpreter can even handle goto errors/exceptions, but I chose to disable them for now as it's not tested. * Because we do not share interpreter dispatch, there is should be no significant slowdown to the original specializing interpreter on tailcall and computed got with JIT disabled. With JIT enabled, there might be a slowdown in the form of the JIT trying to trace. * Things that could have dynamic instruction pointer effects are guarded on. The guard deopts to a new instruction --- `_DYNAMIC_EXIT`.
2025-11-14 02:08:32 +08:00
if (frame->lltrace >= 3) {
GH-119866: Spill the stack around escaping calls. (GH-124392) * Spill the evaluation around escaping calls in the generated interpreter and JIT. * The code generator tracks live, cached values so they can be saved to memory when needed. * Spills the stack pointer around escaping calls, so that the exact stack is visible to the cycle GC.
2024-10-07 14:56:39 +01:00
_PyFrame_SetStackPointer(frame, stack_pointer);
GH-116017: Get rid of _COLD_EXITs (GH-120960)
2024-07-01 13:17:40 -07:00
printf("SIDE EXIT: [UOp ");
_PyUOpPrint(&next_uop[-1]);
gh-139109: A new tracing JIT compiler frontend for CPython (GH-140310) This PR changes the current JIT model from trace projection to trace recording. Benchmarking: better pyperformance (about 1.7% overall) geomean versus current https://raw.githubusercontent.com/facebookexperimental/free-threading-benchmarking/refs/heads/main/results/bm-20251108-3.15.0a1%2B-7e2bc1d-JIT/bm-20251108-vultr-x86_64-Fidget%252dSpinner-tracing_jit-3.15.0a1%2B-7e2bc1d-vs-base.svg, 100% faster Richards on the most improved benchmark versus the current JIT. Slowdown of about 10-15% on the worst benchmark versus the current JIT. **Note: the fastest version isn't the one merged, as it relies on fixing bugs in the specializing interpreter, which is left to another PR**. The speedup in the merged version is about 1.1%. https://raw.githubusercontent.com/facebookexperimental/free-threading-benchmarking/refs/heads/main/results/bm-20251112-3.15.0a1%2B-f8a764a-JIT/bm-20251112-vultr-x86_64-Fidget%252dSpinner-tracing_jit-3.15.0a1%2B-f8a764a-vs-base.svg Stats: 50% more uops executed, 30% more traces entered the last time we ran them. It also suggests our trace lengths for a real trace recording JIT are too short, as a lot of trace too long aborts https://github.com/facebookexperimental/free-threading-benchmarking/blob/main/results/bm-20251023-3.15.0a1%2B-eb73378-CLANG%2CJIT/bm-20251023-vultr-x86_64-Fidget%252dSpinner-tracing_jit-3.15.0a1%2B-eb73378-pystats-vs-base.md . This new JIT frontend is already able to record/execute significantly more instructions than the previous JIT frontend. In this PR, we are now able to record through custom dunders, simple object creation, generators, etc. None of these were done by the old JIT frontend. Some custom dunders uops were discovered to be broken as part of this work gh-140277 The optimizer stack space check is disabled, as it's no longer valid to deal with underflow. Pros: * Ignoring the generated tracer code as it's automatically created, this is only additional 1k lines of code. The maintenance burden is handled by the DSL and code generator. * `optimizer.c` is now significantly simpler, as we don't have to do strange things to recover the bytecode from a trace. * The new JIT frontend is able to handle a lot more control-flow than the old one. * Tracing is very low overhead. We use the tail calling interpreter/computed goto interpreter to switch between tracing mode and non-tracing mode. I call this mechanism dual dispatch, as we have two dispatch tables dispatching to each other. Specialization is still enabled while tracing. * Better handling of polymorphism. We leverage the specializing interpreter for this. Cons: * (For now) requires tail calling interpreter or computed gotos. This means no Windows JIT for now :(. Not to fret, tail calling is coming soon to Windows though https://github.com/python/cpython/pull/139962 Design: * After each instruction, the `record_previous_inst` function/label is executed. This does as the name suggests. * The tracing interpreter lowers bytecode to uops directly so that it can obtain "fresh" values at the point of lowering. * The tracing version behaves nearly identical to the normal interpreter, in fact it even has specialization! This allows it to run without much of a slowdown when tracing. The actual cost of tracing is only a function call and writes to memory. * The tracing interpreter uses the specializing interpreter's deopt to naturally form the side exit chains. This allows it to side exit chain effectively, without repeating much code. We force a re-specializing when tracing a deopt. * The tracing interpreter can even handle goto errors/exceptions, but I chose to disable them for now as it's not tested. * Because we do not share interpreter dispatch, there is should be no significant slowdown to the original specializing interpreter on tailcall and computed got with JIT disabled. With JIT enabled, there might be a slowdown in the form of the JIT trying to trace. * Things that could have dynamic instruction pointer effects are guarded on. The guard deopts to a new instruction --- `_DYNAMIC_EXIT`.
2025-11-14 02:08:32 +08:00
printf(", exit %tu, temp %d, target %d -> %s, is_control_flow %d]\n",
GH-116968: Remove branch from advance_backoff_counter (GH-124469)
2024-10-07 11:46:33 +01:00
exit - current_executor->exits, exit->temperature.value_and_backoff,
gh-115999: Implement thread-local bytecode and enable specialization for `BINARY_OP` (#123926) Each thread specializes a thread-local copy of the bytecode, created on the first RESUME, in free-threaded builds. All copies of the bytecode for a code object are stored in the co_tlbc array on the code object. Threads reserve a globally unique index identifying its copy of the bytecode in all co_tlbc arrays at thread creation and release the index at thread destruction. The first entry in every co_tlbc array always points to the "main" copy of the bytecode that is stored at the end of the code object. This ensures that no bytecode is copied for programs that do not use threads. Thread-local bytecode can be disabled at runtime by providing either -X tlbc=0 or PYTHON_TLBC=0. Disabling thread-local bytecode also disables specialization. Concurrent modifications to the bytecode made by the specializing interpreter and instrumentation use atomics, with specialization taking care not to overwrite an instruction that was instrumented concurrently.
2024-11-04 11:13:32 -08:00
(int)(target - _PyFrame_GetBytecode(frame)),
gh-139109: A new tracing JIT compiler frontend for CPython (GH-140310) This PR changes the current JIT model from trace projection to trace recording. Benchmarking: better pyperformance (about 1.7% overall) geomean versus current https://raw.githubusercontent.com/facebookexperimental/free-threading-benchmarking/refs/heads/main/results/bm-20251108-3.15.0a1%2B-7e2bc1d-JIT/bm-20251108-vultr-x86_64-Fidget%252dSpinner-tracing_jit-3.15.0a1%2B-7e2bc1d-vs-base.svg, 100% faster Richards on the most improved benchmark versus the current JIT. Slowdown of about 10-15% on the worst benchmark versus the current JIT. **Note: the fastest version isn't the one merged, as it relies on fixing bugs in the specializing interpreter, which is left to another PR**. The speedup in the merged version is about 1.1%. https://raw.githubusercontent.com/facebookexperimental/free-threading-benchmarking/refs/heads/main/results/bm-20251112-3.15.0a1%2B-f8a764a-JIT/bm-20251112-vultr-x86_64-Fidget%252dSpinner-tracing_jit-3.15.0a1%2B-f8a764a-vs-base.svg Stats: 50% more uops executed, 30% more traces entered the last time we ran them. It also suggests our trace lengths for a real trace recording JIT are too short, as a lot of trace too long aborts https://github.com/facebookexperimental/free-threading-benchmarking/blob/main/results/bm-20251023-3.15.0a1%2B-eb73378-CLANG%2CJIT/bm-20251023-vultr-x86_64-Fidget%252dSpinner-tracing_jit-3.15.0a1%2B-eb73378-pystats-vs-base.md . This new JIT frontend is already able to record/execute significantly more instructions than the previous JIT frontend. In this PR, we are now able to record through custom dunders, simple object creation, generators, etc. None of these were done by the old JIT frontend. Some custom dunders uops were discovered to be broken as part of this work gh-140277 The optimizer stack space check is disabled, as it's no longer valid to deal with underflow. Pros: * Ignoring the generated tracer code as it's automatically created, this is only additional 1k lines of code. The maintenance burden is handled by the DSL and code generator. * `optimizer.c` is now significantly simpler, as we don't have to do strange things to recover the bytecode from a trace. * The new JIT frontend is able to handle a lot more control-flow than the old one. * Tracing is very low overhead. We use the tail calling interpreter/computed goto interpreter to switch between tracing mode and non-tracing mode. I call this mechanism dual dispatch, as we have two dispatch tables dispatching to each other. Specialization is still enabled while tracing. * Better handling of polymorphism. We leverage the specializing interpreter for this. Cons: * (For now) requires tail calling interpreter or computed gotos. This means no Windows JIT for now :(. Not to fret, tail calling is coming soon to Windows though https://github.com/python/cpython/pull/139962 Design: * After each instruction, the `record_previous_inst` function/label is executed. This does as the name suggests. * The tracing interpreter lowers bytecode to uops directly so that it can obtain "fresh" values at the point of lowering. * The tracing version behaves nearly identical to the normal interpreter, in fact it even has specialization! This allows it to run without much of a slowdown when tracing. The actual cost of tracing is only a function call and writes to memory. * The tracing interpreter uses the specializing interpreter's deopt to naturally form the side exit chains. This allows it to side exit chain effectively, without repeating much code. We force a re-specializing when tracing a deopt. * The tracing interpreter can even handle goto errors/exceptions, but I chose to disable them for now as it's not tested. * Because we do not share interpreter dispatch, there is should be no significant slowdown to the original specializing interpreter on tailcall and computed got with JIT disabled. With JIT enabled, there might be a slowdown in the form of the JIT trying to trace. * Things that could have dynamic instruction pointer effects are guarded on. The guard deopts to a new instruction --- `_DYNAMIC_EXIT`.
2025-11-14 02:08:32 +08:00
_PyOpcode_OpName[target->op.code], exit->is_control_flow);
GH-119866: Spill the stack around escaping calls. (GH-124392) * Spill the evaluation around escaping calls in the generated interpreter and JIT. * The code generator tracks live, cached values so they can be saved to memory when needed. * Spills the stack pointer around escaping calls, so that the exact stack is visible to the cycle GC.
2024-10-07 14:56:39 +01:00
stack_pointer = _PyFrame_GetStackPointer(frame);
GH-116017: Get rid of _COLD_EXITs (GH-120960)
2024-07-01 13:17:40 -07:00
}
#endif
GH-136410: Faster side exits by using a cold exit stub (GH-136411)
2025-08-01 16:26:07 +01:00
tstate->jit_exit = exit;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(0);
GH-137959: Replace shim code in jitted code with a single trampoline function. (GH-137961)
2025-08-21 10:40:53 +01:00
TIER2_TO_TIER2(exit->executor);
gh-106701: Move the hand-written Tier 2 uops to bytecodes.c (#106702) This moves EXIT_TRACE, SAVE_IP, JUMP_TO_TOP, and _POP_JUMP_IF_{FALSE,TRUE} from ceval.c to bytecodes.c. They are no less special than before, but this way they are discoverable o the copy-and-patch tooling.
2023-07-13 12:14:51 -07:00
}
gh-107557: Setup abstract interpretation (#107847) Co-authored-by: Guido van Rossum <gvanrossum@users.noreply.github.com> Co-authored-by: Jules <57632293+juliapoo@users.noreply.github.com>
2023-08-16 02:04:17 +08:00
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
case _EXIT_TRACE_r10: {
CHECK_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef _stack_item_0 = _tos_cache0;
GH-140683: JIT: Improve machine code for loading smaller constants on AArch64. (GH-142511) * Use movz and movk instructions for loading 16 and 32 bit operands and oparg. * Loading of 64 bit operands is unchanged.
2025-12-11 12:33:39 +00:00
PyObject *exit_p = (PyObject *)CURRENT_OPERAND0_64();
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyExitData *exit = (_PyExitData *)exit_p;
#if defined(Py_DEBUG) && !defined(_Py_JIT)
const _Py_CODEUNIT *target = ((frame->owner == FRAME_OWNED_BY_INTERPRETER)
? _Py_INTERPRETER_TRAMPOLINE_INSTRUCTIONS_PTR : _PyFrame_GetBytecode(frame))
+ exit->target;
OPT_HIST(trace_uop_execution_counter, trace_run_length_hist);
if (frame->lltrace >= 3) {
stack_pointer[0] = _stack_item_0;
stack_pointer += 1;
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
_PyFrame_SetStackPointer(frame, stack_pointer);
printf("SIDE EXIT: [UOp ");
_PyUOpPrint(&next_uop[-1]);
printf(", exit %tu, temp %d, target %d -> %s, is_control_flow %d]\n",
exit - current_executor->exits, exit->temperature.value_and_backoff,
(int)(target - _PyFrame_GetBytecode(frame)),
_PyOpcode_OpName[target->op.code], exit->is_control_flow);
stack_pointer = _PyFrame_GetStackPointer(frame);
stack_pointer += -1;
}
#endif
tstate->jit_exit = exit;
SET_CURRENT_CACHED_VALUES(0);
stack_pointer[0] = _stack_item_0;
stack_pointer += 1;
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
TIER2_TO_TIER2(exit->executor);
}
case _EXIT_TRACE_r20: {
CHECK_CURRENT_CACHED_VALUES(2);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef _stack_item_0 = _tos_cache0;
_PyStackRef _stack_item_1 = _tos_cache1;
GH-140683: JIT: Improve machine code for loading smaller constants on AArch64. (GH-142511) * Use movz and movk instructions for loading 16 and 32 bit operands and oparg. * Loading of 64 bit operands is unchanged.
2025-12-11 12:33:39 +00:00
PyObject *exit_p = (PyObject *)CURRENT_OPERAND0_64();
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyExitData *exit = (_PyExitData *)exit_p;
#if defined(Py_DEBUG) && !defined(_Py_JIT)
const _Py_CODEUNIT *target = ((frame->owner == FRAME_OWNED_BY_INTERPRETER)
? _Py_INTERPRETER_TRAMPOLINE_INSTRUCTIONS_PTR : _PyFrame_GetBytecode(frame))
+ exit->target;
OPT_HIST(trace_uop_execution_counter, trace_run_length_hist);
if (frame->lltrace >= 3) {
stack_pointer[0] = _stack_item_0;
stack_pointer[1] = _stack_item_1;
stack_pointer += 2;
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
_PyFrame_SetStackPointer(frame, stack_pointer);
printf("SIDE EXIT: [UOp ");
_PyUOpPrint(&next_uop[-1]);
printf(", exit %tu, temp %d, target %d -> %s, is_control_flow %d]\n",
exit - current_executor->exits, exit->temperature.value_and_backoff,
(int)(target - _PyFrame_GetBytecode(frame)),
_PyOpcode_OpName[target->op.code], exit->is_control_flow);
stack_pointer = _PyFrame_GetStackPointer(frame);
stack_pointer += -2;
}
#endif
tstate->jit_exit = exit;
SET_CURRENT_CACHED_VALUES(0);
stack_pointer[0] = _stack_item_0;
stack_pointer[1] = _stack_item_1;
stack_pointer += 2;
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
TIER2_TO_TIER2(exit->executor);
}
case _EXIT_TRACE_r30: {
CHECK_CURRENT_CACHED_VALUES(3);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef _stack_item_0 = _tos_cache0;
_PyStackRef _stack_item_1 = _tos_cache1;
_PyStackRef _stack_item_2 = _tos_cache2;
GH-140683: JIT: Improve machine code for loading smaller constants on AArch64. (GH-142511) * Use movz and movk instructions for loading 16 and 32 bit operands and oparg. * Loading of 64 bit operands is unchanged.
2025-12-11 12:33:39 +00:00
PyObject *exit_p = (PyObject *)CURRENT_OPERAND0_64();
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyExitData *exit = (_PyExitData *)exit_p;
#if defined(Py_DEBUG) && !defined(_Py_JIT)
const _Py_CODEUNIT *target = ((frame->owner == FRAME_OWNED_BY_INTERPRETER)
? _Py_INTERPRETER_TRAMPOLINE_INSTRUCTIONS_PTR : _PyFrame_GetBytecode(frame))
+ exit->target;
OPT_HIST(trace_uop_execution_counter, trace_run_length_hist);
if (frame->lltrace >= 3) {
stack_pointer[0] = _stack_item_0;
stack_pointer[1] = _stack_item_1;
stack_pointer[2] = _stack_item_2;
stack_pointer += 3;
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
_PyFrame_SetStackPointer(frame, stack_pointer);
printf("SIDE EXIT: [UOp ");
_PyUOpPrint(&next_uop[-1]);
printf(", exit %tu, temp %d, target %d -> %s, is_control_flow %d]\n",
exit - current_executor->exits, exit->temperature.value_and_backoff,
(int)(target - _PyFrame_GetBytecode(frame)),
_PyOpcode_OpName[target->op.code], exit->is_control_flow);
stack_pointer = _PyFrame_GetStackPointer(frame);
stack_pointer += -3;
}
#endif
tstate->jit_exit = exit;
SET_CURRENT_CACHED_VALUES(0);
stack_pointer[0] = _stack_item_0;
stack_pointer[1] = _stack_item_1;
stack_pointer[2] = _stack_item_2;
stack_pointer += 3;
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
TIER2_TO_TIER2(exit->executor);
}
case _DYNAMIC_EXIT_r00: {
CHECK_CURRENT_CACHED_VALUES(0);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-140683: JIT: Improve machine code for loading smaller constants on AArch64. (GH-142511) * Use movz and movk instructions for loading 16 and 32 bit operands and oparg. * Loading of 64 bit operands is unchanged.
2025-12-11 12:33:39 +00:00
PyObject *exit_p = (PyObject *)CURRENT_OPERAND0_64();
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
#if defined(Py_DEBUG) && !defined(_Py_JIT)
_PyExitData *exit = (_PyExitData *)exit_p;
_Py_CODEUNIT *target = frame->instr_ptr;
OPT_HIST(trace_uop_execution_counter, trace_run_length_hist);
if (frame->lltrace >= 3) {
_PyFrame_SetStackPointer(frame, stack_pointer);
printf("DYNAMIC EXIT: [UOp ");
_PyUOpPrint(&next_uop[-1]);
printf(", exit %tu, temp %d, target %d -> %s]\n",
exit - current_executor->exits, exit->temperature.value_and_backoff,
(int)(target - _PyFrame_GetBytecode(frame)),
_PyOpcode_OpName[target->op.code]);
stack_pointer = _PyFrame_GetStackPointer(frame);
}
#endif
SET_CURRENT_CACHED_VALUES(0);
GOTO_TIER_ONE(frame->instr_ptr);
}
case _DYNAMIC_EXIT_r10: {
CHECK_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef _stack_item_0 = _tos_cache0;
GH-140683: JIT: Improve machine code for loading smaller constants on AArch64. (GH-142511) * Use movz and movk instructions for loading 16 and 32 bit operands and oparg. * Loading of 64 bit operands is unchanged.
2025-12-11 12:33:39 +00:00
PyObject *exit_p = (PyObject *)CURRENT_OPERAND0_64();
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
#if defined(Py_DEBUG) && !defined(_Py_JIT)
_PyExitData *exit = (_PyExitData *)exit_p;
_Py_CODEUNIT *target = frame->instr_ptr;
OPT_HIST(trace_uop_execution_counter, trace_run_length_hist);
if (frame->lltrace >= 3) {
stack_pointer[0] = _stack_item_0;
stack_pointer += 1;
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
_PyFrame_SetStackPointer(frame, stack_pointer);
printf("DYNAMIC EXIT: [UOp ");
_PyUOpPrint(&next_uop[-1]);
printf(", exit %tu, temp %d, target %d -> %s]\n",
exit - current_executor->exits, exit->temperature.value_and_backoff,
(int)(target - _PyFrame_GetBytecode(frame)),
_PyOpcode_OpName[target->op.code]);
stack_pointer = _PyFrame_GetStackPointer(frame);
stack_pointer += -1;
}
#endif
SET_CURRENT_CACHED_VALUES(0);
stack_pointer[0] = _stack_item_0;
stack_pointer += 1;
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
GOTO_TIER_ONE(frame->instr_ptr);
}
case _DYNAMIC_EXIT_r20: {
CHECK_CURRENT_CACHED_VALUES(2);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef _stack_item_0 = _tos_cache0;
_PyStackRef _stack_item_1 = _tos_cache1;
GH-140683: JIT: Improve machine code for loading smaller constants on AArch64. (GH-142511) * Use movz and movk instructions for loading 16 and 32 bit operands and oparg. * Loading of 64 bit operands is unchanged.
2025-12-11 12:33:39 +00:00
PyObject *exit_p = (PyObject *)CURRENT_OPERAND0_64();
gh-139109: A new tracing JIT compiler frontend for CPython (GH-140310) This PR changes the current JIT model from trace projection to trace recording. Benchmarking: better pyperformance (about 1.7% overall) geomean versus current https://raw.githubusercontent.com/facebookexperimental/free-threading-benchmarking/refs/heads/main/results/bm-20251108-3.15.0a1%2B-7e2bc1d-JIT/bm-20251108-vultr-x86_64-Fidget%252dSpinner-tracing_jit-3.15.0a1%2B-7e2bc1d-vs-base.svg, 100% faster Richards on the most improved benchmark versus the current JIT. Slowdown of about 10-15% on the worst benchmark versus the current JIT. **Note: the fastest version isn't the one merged, as it relies on fixing bugs in the specializing interpreter, which is left to another PR**. The speedup in the merged version is about 1.1%. https://raw.githubusercontent.com/facebookexperimental/free-threading-benchmarking/refs/heads/main/results/bm-20251112-3.15.0a1%2B-f8a764a-JIT/bm-20251112-vultr-x86_64-Fidget%252dSpinner-tracing_jit-3.15.0a1%2B-f8a764a-vs-base.svg Stats: 50% more uops executed, 30% more traces entered the last time we ran them. It also suggests our trace lengths for a real trace recording JIT are too short, as a lot of trace too long aborts https://github.com/facebookexperimental/free-threading-benchmarking/blob/main/results/bm-20251023-3.15.0a1%2B-eb73378-CLANG%2CJIT/bm-20251023-vultr-x86_64-Fidget%252dSpinner-tracing_jit-3.15.0a1%2B-eb73378-pystats-vs-base.md . This new JIT frontend is already able to record/execute significantly more instructions than the previous JIT frontend. In this PR, we are now able to record through custom dunders, simple object creation, generators, etc. None of these were done by the old JIT frontend. Some custom dunders uops were discovered to be broken as part of this work gh-140277 The optimizer stack space check is disabled, as it's no longer valid to deal with underflow. Pros: * Ignoring the generated tracer code as it's automatically created, this is only additional 1k lines of code. The maintenance burden is handled by the DSL and code generator. * `optimizer.c` is now significantly simpler, as we don't have to do strange things to recover the bytecode from a trace. * The new JIT frontend is able to handle a lot more control-flow than the old one. * Tracing is very low overhead. We use the tail calling interpreter/computed goto interpreter to switch between tracing mode and non-tracing mode. I call this mechanism dual dispatch, as we have two dispatch tables dispatching to each other. Specialization is still enabled while tracing. * Better handling of polymorphism. We leverage the specializing interpreter for this. Cons: * (For now) requires tail calling interpreter or computed gotos. This means no Windows JIT for now :(. Not to fret, tail calling is coming soon to Windows though https://github.com/python/cpython/pull/139962 Design: * After each instruction, the `record_previous_inst` function/label is executed. This does as the name suggests. * The tracing interpreter lowers bytecode to uops directly so that it can obtain "fresh" values at the point of lowering. * The tracing version behaves nearly identical to the normal interpreter, in fact it even has specialization! This allows it to run without much of a slowdown when tracing. The actual cost of tracing is only a function call and writes to memory. * The tracing interpreter uses the specializing interpreter's deopt to naturally form the side exit chains. This allows it to side exit chain effectively, without repeating much code. We force a re-specializing when tracing a deopt. * The tracing interpreter can even handle goto errors/exceptions, but I chose to disable them for now as it's not tested. * Because we do not share interpreter dispatch, there is should be no significant slowdown to the original specializing interpreter on tailcall and computed got with JIT disabled. With JIT enabled, there might be a slowdown in the form of the JIT trying to trace. * Things that could have dynamic instruction pointer effects are guarded on. The guard deopts to a new instruction --- `_DYNAMIC_EXIT`.
2025-11-14 02:08:32 +08:00
#if defined(Py_DEBUG) && !defined(_Py_JIT)
_PyExitData *exit = (_PyExitData *)exit_p;
_Py_CODEUNIT *target = frame->instr_ptr;
OPT_HIST(trace_uop_execution_counter, trace_run_length_hist);
if (frame->lltrace >= 3) {
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
stack_pointer[0] = _stack_item_0;
stack_pointer[1] = _stack_item_1;
stack_pointer += 2;
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
gh-139109: A new tracing JIT compiler frontend for CPython (GH-140310) This PR changes the current JIT model from trace projection to trace recording. Benchmarking: better pyperformance (about 1.7% overall) geomean versus current https://raw.githubusercontent.com/facebookexperimental/free-threading-benchmarking/refs/heads/main/results/bm-20251108-3.15.0a1%2B-7e2bc1d-JIT/bm-20251108-vultr-x86_64-Fidget%252dSpinner-tracing_jit-3.15.0a1%2B-7e2bc1d-vs-base.svg, 100% faster Richards on the most improved benchmark versus the current JIT. Slowdown of about 10-15% on the worst benchmark versus the current JIT. **Note: the fastest version isn't the one merged, as it relies on fixing bugs in the specializing interpreter, which is left to another PR**. The speedup in the merged version is about 1.1%. https://raw.githubusercontent.com/facebookexperimental/free-threading-benchmarking/refs/heads/main/results/bm-20251112-3.15.0a1%2B-f8a764a-JIT/bm-20251112-vultr-x86_64-Fidget%252dSpinner-tracing_jit-3.15.0a1%2B-f8a764a-vs-base.svg Stats: 50% more uops executed, 30% more traces entered the last time we ran them. It also suggests our trace lengths for a real trace recording JIT are too short, as a lot of trace too long aborts https://github.com/facebookexperimental/free-threading-benchmarking/blob/main/results/bm-20251023-3.15.0a1%2B-eb73378-CLANG%2CJIT/bm-20251023-vultr-x86_64-Fidget%252dSpinner-tracing_jit-3.15.0a1%2B-eb73378-pystats-vs-base.md . This new JIT frontend is already able to record/execute significantly more instructions than the previous JIT frontend. In this PR, we are now able to record through custom dunders, simple object creation, generators, etc. None of these were done by the old JIT frontend. Some custom dunders uops were discovered to be broken as part of this work gh-140277 The optimizer stack space check is disabled, as it's no longer valid to deal with underflow. Pros: * Ignoring the generated tracer code as it's automatically created, this is only additional 1k lines of code. The maintenance burden is handled by the DSL and code generator. * `optimizer.c` is now significantly simpler, as we don't have to do strange things to recover the bytecode from a trace. * The new JIT frontend is able to handle a lot more control-flow than the old one. * Tracing is very low overhead. We use the tail calling interpreter/computed goto interpreter to switch between tracing mode and non-tracing mode. I call this mechanism dual dispatch, as we have two dispatch tables dispatching to each other. Specialization is still enabled while tracing. * Better handling of polymorphism. We leverage the specializing interpreter for this. Cons: * (For now) requires tail calling interpreter or computed gotos. This means no Windows JIT for now :(. Not to fret, tail calling is coming soon to Windows though https://github.com/python/cpython/pull/139962 Design: * After each instruction, the `record_previous_inst` function/label is executed. This does as the name suggests. * The tracing interpreter lowers bytecode to uops directly so that it can obtain "fresh" values at the point of lowering. * The tracing version behaves nearly identical to the normal interpreter, in fact it even has specialization! This allows it to run without much of a slowdown when tracing. The actual cost of tracing is only a function call and writes to memory. * The tracing interpreter uses the specializing interpreter's deopt to naturally form the side exit chains. This allows it to side exit chain effectively, without repeating much code. We force a re-specializing when tracing a deopt. * The tracing interpreter can even handle goto errors/exceptions, but I chose to disable them for now as it's not tested. * Because we do not share interpreter dispatch, there is should be no significant slowdown to the original specializing interpreter on tailcall and computed got with JIT disabled. With JIT enabled, there might be a slowdown in the form of the JIT trying to trace. * Things that could have dynamic instruction pointer effects are guarded on. The guard deopts to a new instruction --- `_DYNAMIC_EXIT`.
2025-11-14 02:08:32 +08:00
_PyFrame_SetStackPointer(frame, stack_pointer);
printf("DYNAMIC EXIT: [UOp ");
_PyUOpPrint(&next_uop[-1]);
printf(", exit %tu, temp %d, target %d -> %s]\n",
exit - current_executor->exits, exit->temperature.value_and_backoff,
(int)(target - _PyFrame_GetBytecode(frame)),
_PyOpcode_OpName[target->op.code]);
stack_pointer = _PyFrame_GetStackPointer(frame);
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
stack_pointer += -2;
gh-139109: A new tracing JIT compiler frontend for CPython (GH-140310) This PR changes the current JIT model from trace projection to trace recording. Benchmarking: better pyperformance (about 1.7% overall) geomean versus current https://raw.githubusercontent.com/facebookexperimental/free-threading-benchmarking/refs/heads/main/results/bm-20251108-3.15.0a1%2B-7e2bc1d-JIT/bm-20251108-vultr-x86_64-Fidget%252dSpinner-tracing_jit-3.15.0a1%2B-7e2bc1d-vs-base.svg, 100% faster Richards on the most improved benchmark versus the current JIT. Slowdown of about 10-15% on the worst benchmark versus the current JIT. **Note: the fastest version isn't the one merged, as it relies on fixing bugs in the specializing interpreter, which is left to another PR**. The speedup in the merged version is about 1.1%. https://raw.githubusercontent.com/facebookexperimental/free-threading-benchmarking/refs/heads/main/results/bm-20251112-3.15.0a1%2B-f8a764a-JIT/bm-20251112-vultr-x86_64-Fidget%252dSpinner-tracing_jit-3.15.0a1%2B-f8a764a-vs-base.svg Stats: 50% more uops executed, 30% more traces entered the last time we ran them. It also suggests our trace lengths for a real trace recording JIT are too short, as a lot of trace too long aborts https://github.com/facebookexperimental/free-threading-benchmarking/blob/main/results/bm-20251023-3.15.0a1%2B-eb73378-CLANG%2CJIT/bm-20251023-vultr-x86_64-Fidget%252dSpinner-tracing_jit-3.15.0a1%2B-eb73378-pystats-vs-base.md . This new JIT frontend is already able to record/execute significantly more instructions than the previous JIT frontend. In this PR, we are now able to record through custom dunders, simple object creation, generators, etc. None of these were done by the old JIT frontend. Some custom dunders uops were discovered to be broken as part of this work gh-140277 The optimizer stack space check is disabled, as it's no longer valid to deal with underflow. Pros: * Ignoring the generated tracer code as it's automatically created, this is only additional 1k lines of code. The maintenance burden is handled by the DSL and code generator. * `optimizer.c` is now significantly simpler, as we don't have to do strange things to recover the bytecode from a trace. * The new JIT frontend is able to handle a lot more control-flow than the old one. * Tracing is very low overhead. We use the tail calling interpreter/computed goto interpreter to switch between tracing mode and non-tracing mode. I call this mechanism dual dispatch, as we have two dispatch tables dispatching to each other. Specialization is still enabled while tracing. * Better handling of polymorphism. We leverage the specializing interpreter for this. Cons: * (For now) requires tail calling interpreter or computed gotos. This means no Windows JIT for now :(. Not to fret, tail calling is coming soon to Windows though https://github.com/python/cpython/pull/139962 Design: * After each instruction, the `record_previous_inst` function/label is executed. This does as the name suggests. * The tracing interpreter lowers bytecode to uops directly so that it can obtain "fresh" values at the point of lowering. * The tracing version behaves nearly identical to the normal interpreter, in fact it even has specialization! This allows it to run without much of a slowdown when tracing. The actual cost of tracing is only a function call and writes to memory. * The tracing interpreter uses the specializing interpreter's deopt to naturally form the side exit chains. This allows it to side exit chain effectively, without repeating much code. We force a re-specializing when tracing a deopt. * The tracing interpreter can even handle goto errors/exceptions, but I chose to disable them for now as it's not tested. * Because we do not share interpreter dispatch, there is should be no significant slowdown to the original specializing interpreter on tailcall and computed got with JIT disabled. With JIT enabled, there might be a slowdown in the form of the JIT trying to trace. * Things that could have dynamic instruction pointer effects are guarded on. The guard deopts to a new instruction --- `_DYNAMIC_EXIT`.
2025-11-14 02:08:32 +08:00
}
#endif
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(0);
stack_pointer[0] = _stack_item_0;
stack_pointer[1] = _stack_item_1;
stack_pointer += 2;
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
GOTO_TIER_ONE(frame->instr_ptr);
}
gh-139109: A new tracing JIT compiler frontend for CPython (GH-140310) This PR changes the current JIT model from trace projection to trace recording. Benchmarking: better pyperformance (about 1.7% overall) geomean versus current https://raw.githubusercontent.com/facebookexperimental/free-threading-benchmarking/refs/heads/main/results/bm-20251108-3.15.0a1%2B-7e2bc1d-JIT/bm-20251108-vultr-x86_64-Fidget%252dSpinner-tracing_jit-3.15.0a1%2B-7e2bc1d-vs-base.svg, 100% faster Richards on the most improved benchmark versus the current JIT. Slowdown of about 10-15% on the worst benchmark versus the current JIT. **Note: the fastest version isn't the one merged, as it relies on fixing bugs in the specializing interpreter, which is left to another PR**. The speedup in the merged version is about 1.1%. https://raw.githubusercontent.com/facebookexperimental/free-threading-benchmarking/refs/heads/main/results/bm-20251112-3.15.0a1%2B-f8a764a-JIT/bm-20251112-vultr-x86_64-Fidget%252dSpinner-tracing_jit-3.15.0a1%2B-f8a764a-vs-base.svg Stats: 50% more uops executed, 30% more traces entered the last time we ran them. It also suggests our trace lengths for a real trace recording JIT are too short, as a lot of trace too long aborts https://github.com/facebookexperimental/free-threading-benchmarking/blob/main/results/bm-20251023-3.15.0a1%2B-eb73378-CLANG%2CJIT/bm-20251023-vultr-x86_64-Fidget%252dSpinner-tracing_jit-3.15.0a1%2B-eb73378-pystats-vs-base.md . This new JIT frontend is already able to record/execute significantly more instructions than the previous JIT frontend. In this PR, we are now able to record through custom dunders, simple object creation, generators, etc. None of these were done by the old JIT frontend. Some custom dunders uops were discovered to be broken as part of this work gh-140277 The optimizer stack space check is disabled, as it's no longer valid to deal with underflow. Pros: * Ignoring the generated tracer code as it's automatically created, this is only additional 1k lines of code. The maintenance burden is handled by the DSL and code generator. * `optimizer.c` is now significantly simpler, as we don't have to do strange things to recover the bytecode from a trace. * The new JIT frontend is able to handle a lot more control-flow than the old one. * Tracing is very low overhead. We use the tail calling interpreter/computed goto interpreter to switch between tracing mode and non-tracing mode. I call this mechanism dual dispatch, as we have two dispatch tables dispatching to each other. Specialization is still enabled while tracing. * Better handling of polymorphism. We leverage the specializing interpreter for this. Cons: * (For now) requires tail calling interpreter or computed gotos. This means no Windows JIT for now :(. Not to fret, tail calling is coming soon to Windows though https://github.com/python/cpython/pull/139962 Design: * After each instruction, the `record_previous_inst` function/label is executed. This does as the name suggests. * The tracing interpreter lowers bytecode to uops directly so that it can obtain "fresh" values at the point of lowering. * The tracing version behaves nearly identical to the normal interpreter, in fact it even has specialization! This allows it to run without much of a slowdown when tracing. The actual cost of tracing is only a function call and writes to memory. * The tracing interpreter uses the specializing interpreter's deopt to naturally form the side exit chains. This allows it to side exit chain effectively, without repeating much code. We force a re-specializing when tracing a deopt. * The tracing interpreter can even handle goto errors/exceptions, but I chose to disable them for now as it's not tested. * Because we do not share interpreter dispatch, there is should be no significant slowdown to the original specializing interpreter on tailcall and computed got with JIT disabled. With JIT enabled, there might be a slowdown in the form of the JIT trying to trace. * Things that could have dynamic instruction pointer effects are guarded on. The guard deopts to a new instruction --- `_DYNAMIC_EXIT`.
2025-11-14 02:08:32 +08:00
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
case _DYNAMIC_EXIT_r30: {
CHECK_CURRENT_CACHED_VALUES(3);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef _stack_item_0 = _tos_cache0;
_PyStackRef _stack_item_1 = _tos_cache1;
_PyStackRef _stack_item_2 = _tos_cache2;
GH-140683: JIT: Improve machine code for loading smaller constants on AArch64. (GH-142511) * Use movz and movk instructions for loading 16 and 32 bit operands and oparg. * Loading of 64 bit operands is unchanged.
2025-12-11 12:33:39 +00:00
PyObject *exit_p = (PyObject *)CURRENT_OPERAND0_64();
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
#if defined(Py_DEBUG) && !defined(_Py_JIT)
_PyExitData *exit = (_PyExitData *)exit_p;
_Py_CODEUNIT *target = frame->instr_ptr;
OPT_HIST(trace_uop_execution_counter, trace_run_length_hist);
if (frame->lltrace >= 3) {
stack_pointer[0] = _stack_item_0;
stack_pointer[1] = _stack_item_1;
stack_pointer[2] = _stack_item_2;
stack_pointer += 3;
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
_PyFrame_SetStackPointer(frame, stack_pointer);
printf("DYNAMIC EXIT: [UOp ");
_PyUOpPrint(&next_uop[-1]);
printf(", exit %tu, temp %d, target %d -> %s]\n",
exit - current_executor->exits, exit->temperature.value_and_backoff,
(int)(target - _PyFrame_GetBytecode(frame)),
_PyOpcode_OpName[target->op.code]);
stack_pointer = _PyFrame_GetStackPointer(frame);
stack_pointer += -3;
}
#endif
SET_CURRENT_CACHED_VALUES(0);
stack_pointer[0] = _stack_item_0;
stack_pointer[1] = _stack_item_1;
stack_pointer[2] = _stack_item_2;
stack_pointer += 3;
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
gh-139109: A new tracing JIT compiler frontend for CPython (GH-140310) This PR changes the current JIT model from trace projection to trace recording. Benchmarking: better pyperformance (about 1.7% overall) geomean versus current https://raw.githubusercontent.com/facebookexperimental/free-threading-benchmarking/refs/heads/main/results/bm-20251108-3.15.0a1%2B-7e2bc1d-JIT/bm-20251108-vultr-x86_64-Fidget%252dSpinner-tracing_jit-3.15.0a1%2B-7e2bc1d-vs-base.svg, 100% faster Richards on the most improved benchmark versus the current JIT. Slowdown of about 10-15% on the worst benchmark versus the current JIT. **Note: the fastest version isn't the one merged, as it relies on fixing bugs in the specializing interpreter, which is left to another PR**. The speedup in the merged version is about 1.1%. https://raw.githubusercontent.com/facebookexperimental/free-threading-benchmarking/refs/heads/main/results/bm-20251112-3.15.0a1%2B-f8a764a-JIT/bm-20251112-vultr-x86_64-Fidget%252dSpinner-tracing_jit-3.15.0a1%2B-f8a764a-vs-base.svg Stats: 50% more uops executed, 30% more traces entered the last time we ran them. It also suggests our trace lengths for a real trace recording JIT are too short, as a lot of trace too long aborts https://github.com/facebookexperimental/free-threading-benchmarking/blob/main/results/bm-20251023-3.15.0a1%2B-eb73378-CLANG%2CJIT/bm-20251023-vultr-x86_64-Fidget%252dSpinner-tracing_jit-3.15.0a1%2B-eb73378-pystats-vs-base.md . This new JIT frontend is already able to record/execute significantly more instructions than the previous JIT frontend. In this PR, we are now able to record through custom dunders, simple object creation, generators, etc. None of these were done by the old JIT frontend. Some custom dunders uops were discovered to be broken as part of this work gh-140277 The optimizer stack space check is disabled, as it's no longer valid to deal with underflow. Pros: * Ignoring the generated tracer code as it's automatically created, this is only additional 1k lines of code. The maintenance burden is handled by the DSL and code generator. * `optimizer.c` is now significantly simpler, as we don't have to do strange things to recover the bytecode from a trace. * The new JIT frontend is able to handle a lot more control-flow than the old one. * Tracing is very low overhead. We use the tail calling interpreter/computed goto interpreter to switch between tracing mode and non-tracing mode. I call this mechanism dual dispatch, as we have two dispatch tables dispatching to each other. Specialization is still enabled while tracing. * Better handling of polymorphism. We leverage the specializing interpreter for this. Cons: * (For now) requires tail calling interpreter or computed gotos. This means no Windows JIT for now :(. Not to fret, tail calling is coming soon to Windows though https://github.com/python/cpython/pull/139962 Design: * After each instruction, the `record_previous_inst` function/label is executed. This does as the name suggests. * The tracing interpreter lowers bytecode to uops directly so that it can obtain "fresh" values at the point of lowering. * The tracing version behaves nearly identical to the normal interpreter, in fact it even has specialization! This allows it to run without much of a slowdown when tracing. The actual cost of tracing is only a function call and writes to memory. * The tracing interpreter uses the specializing interpreter's deopt to naturally form the side exit chains. This allows it to side exit chain effectively, without repeating much code. We force a re-specializing when tracing a deopt. * The tracing interpreter can even handle goto errors/exceptions, but I chose to disable them for now as it's not tested. * Because we do not share interpreter dispatch, there is should be no significant slowdown to the original specializing interpreter on tailcall and computed got with JIT disabled. With JIT enabled, there might be a slowdown in the form of the JIT trying to trace. * Things that could have dynamic instruction pointer effects are guarded on. The guard deopts to a new instruction --- `_DYNAMIC_EXIT`.
2025-11-14 02:08:32 +08:00
GOTO_TIER_ONE(frame->instr_ptr);
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
}
case _CHECK_VALIDITY_r00: {
CHECK_CURRENT_CACHED_VALUES(0);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
if (!current_executor->vm_data.valid) {
UOP_STAT_INC(uopcode, miss);
SET_CURRENT_CACHED_VALUES(0);
JUMP_TO_JUMP_TARGET();
}
SET_CURRENT_CACHED_VALUES(0);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _CHECK_VALIDITY_r11: {
CHECK_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef _stack_item_0 = _tos_cache0;
if (!current_executor->vm_data.valid) {
UOP_STAT_INC(uopcode, miss);
GH-142718: JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. (GH-142762) JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. * Make sure that stack is in correct state at side exits with TOS cached values * Simplify choice of cached items for side exits
2025-12-15 19:01:29 +00:00
_tos_cache0 = _stack_item_0;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(1);
JUMP_TO_JUMP_TARGET();
}
_tos_cache0 = _stack_item_0;
SET_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _CHECK_VALIDITY_r22: {
CHECK_CURRENT_CACHED_VALUES(2);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef _stack_item_0 = _tos_cache0;
_PyStackRef _stack_item_1 = _tos_cache1;
if (!current_executor->vm_data.valid) {
UOP_STAT_INC(uopcode, miss);
GH-142718: JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. (GH-142762) JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. * Make sure that stack is in correct state at side exits with TOS cached values * Simplify choice of cached items for side exits
2025-12-15 19:01:29 +00:00
_tos_cache1 = _stack_item_1;
_tos_cache0 = _stack_item_0;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(2);
JUMP_TO_JUMP_TARGET();
}
_tos_cache1 = _stack_item_1;
_tos_cache0 = _stack_item_0;
SET_CURRENT_CACHED_VALUES(2);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-139109: A new tracing JIT compiler frontend for CPython (GH-140310) This PR changes the current JIT model from trace projection to trace recording. Benchmarking: better pyperformance (about 1.7% overall) geomean versus current https://raw.githubusercontent.com/facebookexperimental/free-threading-benchmarking/refs/heads/main/results/bm-20251108-3.15.0a1%2B-7e2bc1d-JIT/bm-20251108-vultr-x86_64-Fidget%252dSpinner-tracing_jit-3.15.0a1%2B-7e2bc1d-vs-base.svg, 100% faster Richards on the most improved benchmark versus the current JIT. Slowdown of about 10-15% on the worst benchmark versus the current JIT. **Note: the fastest version isn't the one merged, as it relies on fixing bugs in the specializing interpreter, which is left to another PR**. The speedup in the merged version is about 1.1%. https://raw.githubusercontent.com/facebookexperimental/free-threading-benchmarking/refs/heads/main/results/bm-20251112-3.15.0a1%2B-f8a764a-JIT/bm-20251112-vultr-x86_64-Fidget%252dSpinner-tracing_jit-3.15.0a1%2B-f8a764a-vs-base.svg Stats: 50% more uops executed, 30% more traces entered the last time we ran them. It also suggests our trace lengths for a real trace recording JIT are too short, as a lot of trace too long aborts https://github.com/facebookexperimental/free-threading-benchmarking/blob/main/results/bm-20251023-3.15.0a1%2B-eb73378-CLANG%2CJIT/bm-20251023-vultr-x86_64-Fidget%252dSpinner-tracing_jit-3.15.0a1%2B-eb73378-pystats-vs-base.md . This new JIT frontend is already able to record/execute significantly more instructions than the previous JIT frontend. In this PR, we are now able to record through custom dunders, simple object creation, generators, etc. None of these were done by the old JIT frontend. Some custom dunders uops were discovered to be broken as part of this work gh-140277 The optimizer stack space check is disabled, as it's no longer valid to deal with underflow. Pros: * Ignoring the generated tracer code as it's automatically created, this is only additional 1k lines of code. The maintenance burden is handled by the DSL and code generator. * `optimizer.c` is now significantly simpler, as we don't have to do strange things to recover the bytecode from a trace. * The new JIT frontend is able to handle a lot more control-flow than the old one. * Tracing is very low overhead. We use the tail calling interpreter/computed goto interpreter to switch between tracing mode and non-tracing mode. I call this mechanism dual dispatch, as we have two dispatch tables dispatching to each other. Specialization is still enabled while tracing. * Better handling of polymorphism. We leverage the specializing interpreter for this. Cons: * (For now) requires tail calling interpreter or computed gotos. This means no Windows JIT for now :(. Not to fret, tail calling is coming soon to Windows though https://github.com/python/cpython/pull/139962 Design: * After each instruction, the `record_previous_inst` function/label is executed. This does as the name suggests. * The tracing interpreter lowers bytecode to uops directly so that it can obtain "fresh" values at the point of lowering. * The tracing version behaves nearly identical to the normal interpreter, in fact it even has specialization! This allows it to run without much of a slowdown when tracing. The actual cost of tracing is only a function call and writes to memory. * The tracing interpreter uses the specializing interpreter's deopt to naturally form the side exit chains. This allows it to side exit chain effectively, without repeating much code. We force a re-specializing when tracing a deopt. * The tracing interpreter can even handle goto errors/exceptions, but I chose to disable them for now as it's not tested. * Because we do not share interpreter dispatch, there is should be no significant slowdown to the original specializing interpreter on tailcall and computed got with JIT disabled. With JIT enabled, there might be a slowdown in the form of the JIT trying to trace. * Things that could have dynamic instruction pointer effects are guarded on. The guard deopts to a new instruction --- `_DYNAMIC_EXIT`.
2025-11-14 02:08:32 +08:00
break;
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
case _CHECK_VALIDITY_r33: {
CHECK_CURRENT_CACHED_VALUES(3);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef _stack_item_0 = _tos_cache0;
_PyStackRef _stack_item_1 = _tos_cache1;
_PyStackRef _stack_item_2 = _tos_cache2;
GH-117457: Correct pystats uop "miss" counts (GH-117477)
2024-04-04 18:49:18 -04:00
if (!current_executor->vm_data.valid) {
UOP_STAT_INC(uopcode, miss);
GH-142718: JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. (GH-142762) JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. * Make sure that stack is in correct state at side exits with TOS cached values * Simplify choice of cached items for side exits
2025-12-15 19:01:29 +00:00
_tos_cache2 = _stack_item_2;
_tos_cache1 = _stack_item_1;
_tos_cache0 = _stack_item_0;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(3);
GH-117457: Correct pystats uop "miss" counts (GH-117477)
2024-04-04 18:49:18 -04:00
JUMP_TO_JUMP_TARGET();
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_tos_cache2 = _stack_item_2;
_tos_cache1 = _stack_item_1;
_tos_cache0 = _stack_item_0;
SET_CURRENT_CACHED_VALUES(3);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-109369: Exit tier 2 if executor is invalid (GH-111657)
2023-11-09 11:19:51 +00:00
break;
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
case _LOAD_CONST_INLINE_r01: {
CHECK_CURRENT_CACHED_VALUES(0);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
_PyStackRef value;
GH-140683: JIT: Improve machine code for loading smaller constants on AArch64. (GH-142511) * Use movz and movk instructions for loading 16 and 32 bit operands and oparg. * Loading of 64 bit operands is unchanged.
2025-12-11 12:33:39 +00:00
PyObject *ptr = (PyObject *)CURRENT_OPERAND0_64();
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
value = PyStackRef_FromPyObjectNew(ptr);
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_tos_cache0 = value;
SET_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _LOAD_CONST_INLINE_r12: {
CHECK_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef value;
_PyStackRef _stack_item_0 = _tos_cache0;
GH-140683: JIT: Improve machine code for loading smaller constants on AArch64. (GH-142511) * Use movz and movk instructions for loading 16 and 32 bit operands and oparg. * Loading of 64 bit operands is unchanged.
2025-12-11 12:33:39 +00:00
PyObject *ptr = (PyObject *)CURRENT_OPERAND0_64();
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
value = PyStackRef_FromPyObjectNew(ptr);
_tos_cache1 = value;
_tos_cache0 = _stack_item_0;
SET_CURRENT_CACHED_VALUES(2);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _LOAD_CONST_INLINE_r23: {
CHECK_CURRENT_CACHED_VALUES(2);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef value;
_PyStackRef _stack_item_0 = _tos_cache0;
_PyStackRef _stack_item_1 = _tos_cache1;
GH-140683: JIT: Improve machine code for loading smaller constants on AArch64. (GH-142511) * Use movz and movk instructions for loading 16 and 32 bit operands and oparg. * Loading of 64 bit operands is unchanged.
2025-12-11 12:33:39 +00:00
PyObject *ptr = (PyObject *)CURRENT_OPERAND0_64();
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
value = PyStackRef_FromPyObjectNew(ptr);
_tos_cache2 = value;
_tos_cache1 = _stack_item_1;
_tos_cache0 = _stack_item_0;
SET_CURRENT_CACHED_VALUES(3);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _POP_TOP_LOAD_CONST_INLINE_r11: {
CHECK_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef pop;
_PyStackRef value;
_PyStackRef _stack_item_0 = _tos_cache0;
pop = _stack_item_0;
GH-140683: JIT: Improve machine code for loading smaller constants on AArch64. (GH-142511) * Use movz and movk instructions for loading 16 and 32 bit operands and oparg. * Loading of 64 bit operands is unchanged.
2025-12-11 12:33:39 +00:00
PyObject *ptr = (PyObject *)CURRENT_OPERAND0_64();
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyFrame_SetStackPointer(frame, stack_pointer);
PyStackRef_CLOSE(pop);
stack_pointer = _PyFrame_GetStackPointer(frame);
value = PyStackRef_FromPyObjectNew(ptr);
_tos_cache0 = value;
_tos_cache1 = PyStackRef_ZERO_BITS;
_tos_cache2 = PyStackRef_ZERO_BITS;
SET_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _LOAD_CONST_INLINE_BORROW_r01: {
CHECK_CURRENT_CACHED_VALUES(0);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef value;
GH-140683: JIT: Improve machine code for loading smaller constants on AArch64. (GH-142511) * Use movz and movk instructions for loading 16 and 32 bit operands and oparg. * Loading of 64 bit operands is unchanged.
2025-12-11 12:33:39 +00:00
PyObject *ptr = (PyObject *)CURRENT_OPERAND0_64();
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
value = PyStackRef_FromPyObjectBorrow(ptr);
_tos_cache0 = value;
SET_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-113710: Add a tier 2 peephole optimization pass. (GH-114487) * Convert _LOAD_CONST to inline versions * Remove PEP 523 checks
2024-01-24 12:08:31 +00:00
break;
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
case _LOAD_CONST_INLINE_BORROW_r12: {
CHECK_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-130296: Avoid stack transients in four instructions. (GH-130310) * Combine _GUARD_GLOBALS_VERSION_PUSH_KEYS and _LOAD_GLOBAL_MODULE_FROM_KEYS into _LOAD_GLOBAL_MODULE * Combine _GUARD_BUILTINS_VERSION_PUSH_KEYS and _LOAD_GLOBAL_BUILTINS_FROM_KEYS into _LOAD_GLOBAL_BUILTINS * Combine _CHECK_ATTR_MODULE_PUSH_KEYS and _LOAD_ATTR_MODULE_FROM_KEYS into _LOAD_ATTR_MODULE * Remove stack transient in LOAD_ATTR_WITH_HINT
2025-02-28 18:00:38 +00:00
_PyStackRef value;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef _stack_item_0 = _tos_cache0;
GH-140683: JIT: Improve machine code for loading smaller constants on AArch64. (GH-142511) * Use movz and movk instructions for loading 16 and 32 bit operands and oparg. * Loading of 64 bit operands is unchanged.
2025-12-11 12:33:39 +00:00
PyObject *ptr = (PyObject *)CURRENT_OPERAND0_64();
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
value = PyStackRef_FromPyObjectBorrow(ptr);
_tos_cache1 = value;
_tos_cache0 = _stack_item_0;
SET_CURRENT_CACHED_VALUES(2);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-130296: Avoid stack transients in four instructions. (GH-130310) * Combine _GUARD_GLOBALS_VERSION_PUSH_KEYS and _LOAD_GLOBAL_MODULE_FROM_KEYS into _LOAD_GLOBAL_MODULE * Combine _GUARD_BUILTINS_VERSION_PUSH_KEYS and _LOAD_GLOBAL_BUILTINS_FROM_KEYS into _LOAD_GLOBAL_BUILTINS * Combine _CHECK_ATTR_MODULE_PUSH_KEYS and _LOAD_ATTR_MODULE_FROM_KEYS into _LOAD_ATTR_MODULE * Remove stack transient in LOAD_ATTR_WITH_HINT
2025-02-28 18:00:38 +00:00
break;
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
case _LOAD_CONST_INLINE_BORROW_r23: {
CHECK_CURRENT_CACHED_VALUES(2);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
_PyStackRef value;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef _stack_item_0 = _tos_cache0;
_PyStackRef _stack_item_1 = _tos_cache1;
GH-140683: JIT: Improve machine code for loading smaller constants on AArch64. (GH-142511) * Use movz and movk instructions for loading 16 and 32 bit operands and oparg. * Loading of 64 bit operands is unchanged.
2025-12-11 12:33:39 +00:00
PyObject *ptr = (PyObject *)CURRENT_OPERAND0_64();
GH-134282: Always borrow references LOAD_CONST (GH-134284)
2025-05-20 11:24:11 -04:00
value = PyStackRef_FromPyObjectBorrow(ptr);
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_tos_cache2 = value;
_tos_cache1 = _stack_item_1;
_tos_cache0 = _stack_item_0;
SET_CURRENT_CACHED_VALUES(3);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-113860: All executors are now defined in terms of micro ops. Convert counter executor to use uops. (GH-113864)
2024-01-10 15:44:34 +00:00
break;
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
case _POP_CALL_r20: {
CHECK_CURRENT_CACHED_VALUES(2);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-131798: Optimize away isinstance calls in the JIT (GH-134369)
2025-05-22 12:52:47 -04:00
_PyStackRef null;
_PyStackRef callable;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef _stack_item_0 = _tos_cache0;
_PyStackRef _stack_item_1 = _tos_cache1;
null = _stack_item_1;
callable = _stack_item_0;
GH-131798: Optimize away isinstance calls in the JIT (GH-134369)
2025-05-22 12:52:47 -04:00
(void)null;
_PyFrame_SetStackPointer(frame, stack_pointer);
PyStackRef_CLOSE(callable);
stack_pointer = _PyFrame_GetStackPointer(frame);
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_tos_cache0 = PyStackRef_ZERO_BITS;
_tos_cache1 = PyStackRef_ZERO_BITS;
_tos_cache2 = PyStackRef_ZERO_BITS;
SET_CURRENT_CACHED_VALUES(0);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-131798: Optimize away isinstance calls in the JIT (GH-134369)
2025-05-22 12:52:47 -04:00
break;
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
case _POP_CALL_ONE_r30: {
CHECK_CURRENT_CACHED_VALUES(3);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-131798: Optimize away isinstance calls in the JIT (GH-134369)
2025-05-22 12:52:47 -04:00
_PyStackRef pop;
_PyStackRef null;
_PyStackRef callable;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef _stack_item_0 = _tos_cache0;
_PyStackRef _stack_item_1 = _tos_cache1;
_PyStackRef _stack_item_2 = _tos_cache2;
pop = _stack_item_2;
null = _stack_item_1;
callable = _stack_item_0;
stack_pointer[0] = callable;
stack_pointer[1] = null;
stack_pointer += 2;
GH-141794: Limit size of generated machine code. (GH-142228) * Factor out bodies of the largest uops, to reduce jit code size. * Factor out common assert, also reducing jit code size. * Limit size of jitted code for a single executor to 1MB.
2025-12-03 17:43:35 +00:00
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
GH-131798: Optimize away isinstance calls in the JIT (GH-134369)
2025-05-22 12:52:47 -04:00
_PyFrame_SetStackPointer(frame, stack_pointer);
PyStackRef_CLOSE(pop);
stack_pointer = _PyFrame_GetStackPointer(frame);
(void)null;
stack_pointer += -2;
GH-141794: Limit size of generated machine code. (GH-142228) * Factor out bodies of the largest uops, to reduce jit code size. * Factor out common assert, also reducing jit code size. * Limit size of jitted code for a single executor to 1MB.
2025-12-03 17:43:35 +00:00
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
GH-131798: Optimize away isinstance calls in the JIT (GH-134369)
2025-05-22 12:52:47 -04:00
_PyFrame_SetStackPointer(frame, stack_pointer);
PyStackRef_CLOSE(callable);
stack_pointer = _PyFrame_GetStackPointer(frame);
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_tos_cache0 = PyStackRef_ZERO_BITS;
_tos_cache1 = PyStackRef_ZERO_BITS;
_tos_cache2 = PyStackRef_ZERO_BITS;
SET_CURRENT_CACHED_VALUES(0);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-131798: Optimize away isinstance calls in the JIT (GH-134369)
2025-05-22 12:52:47 -04:00
break;
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
case _POP_CALL_TWO_r30: {
CHECK_CURRENT_CACHED_VALUES(3);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-131798: Optimize away isinstance calls in the JIT (GH-134369)
2025-05-22 12:52:47 -04:00
_PyStackRef pop2;
_PyStackRef pop1;
_PyStackRef null;
_PyStackRef callable;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef _stack_item_0 = _tos_cache0;
_PyStackRef _stack_item_1 = _tos_cache1;
_PyStackRef _stack_item_2 = _tos_cache2;
pop2 = _stack_item_2;
pop1 = _stack_item_1;
null = _stack_item_0;
callable = stack_pointer[-1];
stack_pointer[0] = null;
stack_pointer[1] = pop1;
stack_pointer += 2;
GH-141794: Limit size of generated machine code. (GH-142228) * Factor out bodies of the largest uops, to reduce jit code size. * Factor out common assert, also reducing jit code size. * Limit size of jitted code for a single executor to 1MB.
2025-12-03 17:43:35 +00:00
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
GH-131798: Optimize away isinstance calls in the JIT (GH-134369)
2025-05-22 12:52:47 -04:00
_PyFrame_SetStackPointer(frame, stack_pointer);
PyStackRef_CLOSE(pop2);
stack_pointer = _PyFrame_GetStackPointer(frame);
stack_pointer += -1;
GH-141794: Limit size of generated machine code. (GH-142228) * Factor out bodies of the largest uops, to reduce jit code size. * Factor out common assert, also reducing jit code size. * Limit size of jitted code for a single executor to 1MB.
2025-12-03 17:43:35 +00:00
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
GH-131798: Optimize away isinstance calls in the JIT (GH-134369)
2025-05-22 12:52:47 -04:00
_PyFrame_SetStackPointer(frame, stack_pointer);
PyStackRef_CLOSE(pop1);
stack_pointer = _PyFrame_GetStackPointer(frame);
(void)null;
stack_pointer += -2;
GH-141794: Limit size of generated machine code. (GH-142228) * Factor out bodies of the largest uops, to reduce jit code size. * Factor out common assert, also reducing jit code size. * Limit size of jitted code for a single executor to 1MB.
2025-12-03 17:43:35 +00:00
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
GH-131798: Optimize away isinstance calls in the JIT (GH-134369)
2025-05-22 12:52:47 -04:00
_PyFrame_SetStackPointer(frame, stack_pointer);
PyStackRef_CLOSE(callable);
stack_pointer = _PyFrame_GetStackPointer(frame);
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_tos_cache0 = PyStackRef_ZERO_BITS;
_tos_cache1 = PyStackRef_ZERO_BITS;
_tos_cache2 = PyStackRef_ZERO_BITS;
SET_CURRENT_CACHED_VALUES(0);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-131798: Optimize away isinstance calls in the JIT (GH-134369)
2025-05-22 12:52:47 -04:00
break;
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
case _POP_TOP_LOAD_CONST_INLINE_BORROW_r11: {
CHECK_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
_PyStackRef pop;
_PyStackRef value;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef _stack_item_0 = _tos_cache0;
pop = _stack_item_0;
GH-140683: JIT: Improve machine code for loading smaller constants on AArch64. (GH-142511) * Use movz and movk instructions for loading 16 and 32 bit operands and oparg. * Loading of 64 bit operands is unchanged.
2025-12-11 12:33:39 +00:00
PyObject *ptr = (PyObject *)CURRENT_OPERAND0_64();
GH-128682: Make `PyStackRef_CLOSE` escaping. (GH-129404)
2025-02-03 12:41:32 +00:00
_PyFrame_SetStackPointer(frame, stack_pointer);
gh-117139: Convert the evaluation stack to stack refs (#118450) This PR sets up tagged pointers for CPython. The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly. Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out. This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it. The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs. Please read Include/internal/pycore_stackref.h for more information! --------- Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
PyStackRef_CLOSE(pop);
GH-128682: Make `PyStackRef_CLOSE` escaping. (GH-129404)
2025-02-03 12:41:32 +00:00
stack_pointer = _PyFrame_GetStackPointer(frame);
GH-134282: Always borrow references LOAD_CONST (GH-134284)
2025-05-20 11:24:11 -04:00
value = PyStackRef_FromPyObjectBorrow(ptr);
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_tos_cache0 = value;
_tos_cache1 = PyStackRef_ZERO_BITS;
_tos_cache2 = PyStackRef_ZERO_BITS;
SET_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-130415: Optimize constant comparison in JIT builds (GH-131489)
2025-03-21 11:23:12 -07:00
break;
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
case _POP_TWO_LOAD_CONST_INLINE_BORROW_r21: {
CHECK_CURRENT_CACHED_VALUES(2);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-130415: Optimize constant comparison in JIT builds (GH-131489)
2025-03-21 11:23:12 -07:00
_PyStackRef pop2;
_PyStackRef pop1;
_PyStackRef value;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef _stack_item_0 = _tos_cache0;
_PyStackRef _stack_item_1 = _tos_cache1;
pop2 = _stack_item_1;
pop1 = _stack_item_0;
GH-140683: JIT: Improve machine code for loading smaller constants on AArch64. (GH-142511) * Use movz and movk instructions for loading 16 and 32 bit operands and oparg. * Loading of 64 bit operands is unchanged.
2025-12-11 12:33:39 +00:00
PyObject *ptr = (PyObject *)CURRENT_OPERAND0_64();
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
stack_pointer[0] = pop1;
stack_pointer += 1;
GH-141794: Limit size of generated machine code. (GH-142228) * Factor out bodies of the largest uops, to reduce jit code size. * Factor out common assert, also reducing jit code size. * Limit size of jitted code for a single executor to 1MB.
2025-12-03 17:43:35 +00:00
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
GH-130415: Optimize constant comparison in JIT builds (GH-131489)
2025-03-21 11:23:12 -07:00
_PyFrame_SetStackPointer(frame, stack_pointer);
PyStackRef_CLOSE(pop2);
stack_pointer = _PyFrame_GetStackPointer(frame);
stack_pointer += -1;
GH-141794: Limit size of generated machine code. (GH-142228) * Factor out bodies of the largest uops, to reduce jit code size. * Factor out common assert, also reducing jit code size. * Limit size of jitted code for a single executor to 1MB.
2025-12-03 17:43:35 +00:00
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
GH-130415: Optimize constant comparison in JIT builds (GH-131489)
2025-03-21 11:23:12 -07:00
_PyFrame_SetStackPointer(frame, stack_pointer);
PyStackRef_CLOSE(pop1);
stack_pointer = _PyFrame_GetStackPointer(frame);
GH-134282: Always borrow references LOAD_CONST (GH-134284)
2025-05-20 11:24:11 -04:00
value = PyStackRef_FromPyObjectBorrow(ptr);
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_tos_cache0 = value;
_tos_cache1 = PyStackRef_ZERO_BITS;
_tos_cache2 = PyStackRef_ZERO_BITS;
SET_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-115685: Type/values propagate for TO_BOOL in tier 2 (GH-115686)
2024-03-01 06:13:38 +08:00
break;
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
case _POP_CALL_LOAD_CONST_INLINE_BORROW_r21: {
CHECK_CURRENT_CACHED_VALUES(2);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-131798: Optimize away isinstance calls in the JIT (GH-134369)
2025-05-22 12:52:47 -04:00
_PyStackRef null;
_PyStackRef callable;
_PyStackRef value;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef _stack_item_0 = _tos_cache0;
_PyStackRef _stack_item_1 = _tos_cache1;
null = _stack_item_1;
callable = _stack_item_0;
GH-140683: JIT: Improve machine code for loading smaller constants on AArch64. (GH-142511) * Use movz and movk instructions for loading 16 and 32 bit operands and oparg. * Loading of 64 bit operands is unchanged.
2025-12-11 12:33:39 +00:00
PyObject *ptr = (PyObject *)CURRENT_OPERAND0_64();
GH-131798: Optimize away isinstance calls in the JIT (GH-134369)
2025-05-22 12:52:47 -04:00
(void)null;
_PyFrame_SetStackPointer(frame, stack_pointer);
PyStackRef_CLOSE(callable);
stack_pointer = _PyFrame_GetStackPointer(frame);
value = PyStackRef_FromPyObjectBorrow(ptr);
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_tos_cache0 = value;
_tos_cache1 = PyStackRef_ZERO_BITS;
_tos_cache2 = PyStackRef_ZERO_BITS;
SET_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-131798: Optimize away isinstance calls in the JIT (GH-134369)
2025-05-22 12:52:47 -04:00
break;
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
case _POP_CALL_ONE_LOAD_CONST_INLINE_BORROW_r31: {
CHECK_CURRENT_CACHED_VALUES(3);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-131798: Optimize away isinstance calls in the JIT (GH-134369)
2025-05-22 12:52:47 -04:00
_PyStackRef pop;
_PyStackRef null;
_PyStackRef callable;
_PyStackRef value;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef _stack_item_0 = _tos_cache0;
_PyStackRef _stack_item_1 = _tos_cache1;
_PyStackRef _stack_item_2 = _tos_cache2;
pop = _stack_item_2;
null = _stack_item_1;
callable = _stack_item_0;
GH-140683: JIT: Improve machine code for loading smaller constants on AArch64. (GH-142511) * Use movz and movk instructions for loading 16 and 32 bit operands and oparg. * Loading of 64 bit operands is unchanged.
2025-12-11 12:33:39 +00:00
PyObject *ptr = (PyObject *)CURRENT_OPERAND0_64();
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
stack_pointer[0] = callable;
stack_pointer[1] = null;
stack_pointer += 2;
GH-141794: Limit size of generated machine code. (GH-142228) * Factor out bodies of the largest uops, to reduce jit code size. * Factor out common assert, also reducing jit code size. * Limit size of jitted code for a single executor to 1MB.
2025-12-03 17:43:35 +00:00
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
GH-131798: Optimize away isinstance calls in the JIT (GH-134369)
2025-05-22 12:52:47 -04:00
_PyFrame_SetStackPointer(frame, stack_pointer);
PyStackRef_CLOSE(pop);
stack_pointer = _PyFrame_GetStackPointer(frame);
(void)null;
stack_pointer += -2;
GH-141794: Limit size of generated machine code. (GH-142228) * Factor out bodies of the largest uops, to reduce jit code size. * Factor out common assert, also reducing jit code size. * Limit size of jitted code for a single executor to 1MB.
2025-12-03 17:43:35 +00:00
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
GH-131798: Optimize away isinstance calls in the JIT (GH-134369)
2025-05-22 12:52:47 -04:00
_PyFrame_SetStackPointer(frame, stack_pointer);
PyStackRef_CLOSE(callable);
stack_pointer = _PyFrame_GetStackPointer(frame);
value = PyStackRef_FromPyObjectBorrow(ptr);
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_tos_cache0 = value;
_tos_cache1 = PyStackRef_ZERO_BITS;
_tos_cache2 = PyStackRef_ZERO_BITS;
SET_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-131798: Optimize away isinstance calls in the JIT (GH-134369)
2025-05-22 12:52:47 -04:00
break;
}
gh-134584: Eliminate redundant refcounting from `_CALL_TYPE_1` (GH-135818)
2025-12-23 18:01:10 +01:00
case _SHUFFLE_2_LOAD_CONST_INLINE_BORROW_r02: {
CHECK_CURRENT_CACHED_VALUES(0);
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
_PyStackRef arg;
_PyStackRef res;
_PyStackRef a;
arg = stack_pointer[-1];
PyObject *ptr = (PyObject *)CURRENT_OPERAND0_64();
res = PyStackRef_FromPyObjectBorrow(ptr);
a = arg;
_tos_cache1 = a;
_tos_cache0 = res;
SET_CURRENT_CACHED_VALUES(2);
stack_pointer += -3;
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
break;
}
case _SHUFFLE_2_LOAD_CONST_INLINE_BORROW_r12: {
CHECK_CURRENT_CACHED_VALUES(1);
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
_PyStackRef arg;
_PyStackRef res;
_PyStackRef a;
_PyStackRef _stack_item_0 = _tos_cache0;
arg = _stack_item_0;
PyObject *ptr = (PyObject *)CURRENT_OPERAND0_64();
res = PyStackRef_FromPyObjectBorrow(ptr);
a = arg;
_tos_cache1 = a;
_tos_cache0 = res;
SET_CURRENT_CACHED_VALUES(2);
stack_pointer += -2;
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
break;
}
case _SHUFFLE_2_LOAD_CONST_INLINE_BORROW_r22: {
CHECK_CURRENT_CACHED_VALUES(2);
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
_PyStackRef arg;
_PyStackRef res;
_PyStackRef a;
_PyStackRef _stack_item_0 = _tos_cache0;
_PyStackRef _stack_item_1 = _tos_cache1;
arg = _stack_item_1;
PyObject *ptr = (PyObject *)CURRENT_OPERAND0_64();
res = PyStackRef_FromPyObjectBorrow(ptr);
a = arg;
_tos_cache1 = a;
_tos_cache0 = res;
SET_CURRENT_CACHED_VALUES(2);
stack_pointer += -1;
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
break;
}
case _SHUFFLE_2_LOAD_CONST_INLINE_BORROW_r32: {
CHECK_CURRENT_CACHED_VALUES(3);
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
_PyStackRef arg;
_PyStackRef res;
_PyStackRef a;
_PyStackRef _stack_item_0 = _tos_cache0;
_PyStackRef _stack_item_1 = _tos_cache1;
_PyStackRef _stack_item_2 = _tos_cache2;
arg = _stack_item_2;
PyObject *ptr = (PyObject *)CURRENT_OPERAND0_64();
res = PyStackRef_FromPyObjectBorrow(ptr);
a = arg;
_tos_cache1 = a;
_tos_cache0 = res;
SET_CURRENT_CACHED_VALUES(2);
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
break;
}
gh-142961: Fix constant folding len(tuple) in JIT (GH-142963)
2025-12-20 01:43:36 +08:00
case _SHUFFLE_3_LOAD_CONST_INLINE_BORROW_r03: {
CHECK_CURRENT_CACHED_VALUES(0);
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
_PyStackRef arg;
_PyStackRef callable;
_PyStackRef res;
_PyStackRef a;
_PyStackRef c;
arg = stack_pointer[-1];
callable = stack_pointer[-3];
PyObject *ptr = (PyObject *)CURRENT_OPERAND0_64();
res = PyStackRef_FromPyObjectBorrow(ptr);
a = arg;
c = callable;
_tos_cache2 = c;
_tos_cache1 = a;
_tos_cache0 = res;
SET_CURRENT_CACHED_VALUES(3);
stack_pointer += -3;
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
break;
}
case _SHUFFLE_3_LOAD_CONST_INLINE_BORROW_r13: {
CHECK_CURRENT_CACHED_VALUES(1);
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
_PyStackRef arg;
_PyStackRef callable;
_PyStackRef res;
_PyStackRef a;
_PyStackRef c;
_PyStackRef _stack_item_0 = _tos_cache0;
arg = _stack_item_0;
callable = stack_pointer[-2];
PyObject *ptr = (PyObject *)CURRENT_OPERAND0_64();
res = PyStackRef_FromPyObjectBorrow(ptr);
a = arg;
c = callable;
_tos_cache2 = c;
_tos_cache1 = a;
_tos_cache0 = res;
SET_CURRENT_CACHED_VALUES(3);
stack_pointer += -2;
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
break;
}
case _SHUFFLE_3_LOAD_CONST_INLINE_BORROW_r23: {
CHECK_CURRENT_CACHED_VALUES(2);
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
_PyStackRef arg;
_PyStackRef callable;
_PyStackRef res;
_PyStackRef a;
_PyStackRef c;
_PyStackRef _stack_item_0 = _tos_cache0;
_PyStackRef _stack_item_1 = _tos_cache1;
arg = _stack_item_1;
callable = stack_pointer[-1];
PyObject *ptr = (PyObject *)CURRENT_OPERAND0_64();
res = PyStackRef_FromPyObjectBorrow(ptr);
a = arg;
c = callable;
_tos_cache2 = c;
_tos_cache1 = a;
_tos_cache0 = res;
SET_CURRENT_CACHED_VALUES(3);
stack_pointer += -1;
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
break;
}
case _SHUFFLE_3_LOAD_CONST_INLINE_BORROW_r33: {
CHECK_CURRENT_CACHED_VALUES(3);
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
_PyStackRef arg;
_PyStackRef callable;
_PyStackRef res;
_PyStackRef a;
_PyStackRef c;
_PyStackRef _stack_item_0 = _tos_cache0;
_PyStackRef _stack_item_1 = _tos_cache1;
_PyStackRef _stack_item_2 = _tos_cache2;
arg = _stack_item_2;
callable = _stack_item_0;
PyObject *ptr = (PyObject *)CURRENT_OPERAND0_64();
res = PyStackRef_FromPyObjectBorrow(ptr);
a = arg;
c = callable;
_tos_cache2 = c;
_tos_cache1 = a;
_tos_cache0 = res;
SET_CURRENT_CACHED_VALUES(3);
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
break;
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
case _POP_CALL_TWO_LOAD_CONST_INLINE_BORROW_r31: {
CHECK_CURRENT_CACHED_VALUES(3);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-131798: Add _POP_CALL_TWO_LOAD_CONST_INLINE_BORROW (GH-134268)
2025-05-19 18:00:53 -04:00
_PyStackRef pop2;
_PyStackRef pop1;
_PyStackRef null;
_PyStackRef callable;
_PyStackRef value;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef _stack_item_0 = _tos_cache0;
_PyStackRef _stack_item_1 = _tos_cache1;
_PyStackRef _stack_item_2 = _tos_cache2;
pop2 = _stack_item_2;
pop1 = _stack_item_1;
null = _stack_item_0;
callable = stack_pointer[-1];
GH-140683: JIT: Improve machine code for loading smaller constants on AArch64. (GH-142511) * Use movz and movk instructions for loading 16 and 32 bit operands and oparg. * Loading of 64 bit operands is unchanged.
2025-12-11 12:33:39 +00:00
PyObject *ptr = (PyObject *)CURRENT_OPERAND0_64();
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
stack_pointer[0] = null;
stack_pointer[1] = pop1;
stack_pointer += 2;
GH-141794: Limit size of generated machine code. (GH-142228) * Factor out bodies of the largest uops, to reduce jit code size. * Factor out common assert, also reducing jit code size. * Limit size of jitted code for a single executor to 1MB.
2025-12-03 17:43:35 +00:00
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
GH-131798: Add _POP_CALL_TWO_LOAD_CONST_INLINE_BORROW (GH-134268)
2025-05-19 18:00:53 -04:00
_PyFrame_SetStackPointer(frame, stack_pointer);
PyStackRef_CLOSE(pop2);
stack_pointer = _PyFrame_GetStackPointer(frame);
stack_pointer += -1;
GH-141794: Limit size of generated machine code. (GH-142228) * Factor out bodies of the largest uops, to reduce jit code size. * Factor out common assert, also reducing jit code size. * Limit size of jitted code for a single executor to 1MB.
2025-12-03 17:43:35 +00:00
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
GH-131798: Add _POP_CALL_TWO_LOAD_CONST_INLINE_BORROW (GH-134268)
2025-05-19 18:00:53 -04:00
_PyFrame_SetStackPointer(frame, stack_pointer);
PyStackRef_CLOSE(pop1);
stack_pointer = _PyFrame_GetStackPointer(frame);
(void)null;
stack_pointer += -2;
GH-141794: Limit size of generated machine code. (GH-142228) * Factor out bodies of the largest uops, to reduce jit code size. * Factor out common assert, also reducing jit code size. * Limit size of jitted code for a single executor to 1MB.
2025-12-03 17:43:35 +00:00
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
GH-131798: Add _POP_CALL_TWO_LOAD_CONST_INLINE_BORROW (GH-134268)
2025-05-19 18:00:53 -04:00
_PyFrame_SetStackPointer(frame, stack_pointer);
PyStackRef_CLOSE(callable);
stack_pointer = _PyFrame_GetStackPointer(frame);
GH-134282: Always borrow references LOAD_CONST (GH-134284)
2025-05-20 11:24:11 -04:00
value = PyStackRef_FromPyObjectBorrow(ptr);
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_tos_cache0 = value;
_tos_cache1 = PyStackRef_ZERO_BITS;
_tos_cache2 = PyStackRef_ZERO_BITS;
SET_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-131798: Add _POP_CALL_TWO_LOAD_CONST_INLINE_BORROW (GH-134268)
2025-05-19 18:00:53 -04:00
break;
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
case _LOAD_CONST_UNDER_INLINE_r02: {
CHECK_CURRENT_CACHED_VALUES(0);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-131798: Optimize cached class attributes and methods in the JIT (GH-134403)
2025-05-22 11:15:03 -04:00
_PyStackRef old;
_PyStackRef value;
_PyStackRef new;
old = stack_pointer[-1];
GH-140683: JIT: Improve machine code for loading smaller constants on AArch64. (GH-142511) * Use movz and movk instructions for loading 16 and 32 bit operands and oparg. * Loading of 64 bit operands is unchanged.
2025-12-11 12:33:39 +00:00
PyObject *ptr = (PyObject *)CURRENT_OPERAND0_64();
GH-131798: Optimize cached class attributes and methods in the JIT (GH-134403)
2025-05-22 11:15:03 -04:00
new = old;
value = PyStackRef_FromPyObjectNew(ptr);
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_tos_cache1 = new;
_tos_cache0 = value;
SET_CURRENT_CACHED_VALUES(2);
stack_pointer += -1;
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _LOAD_CONST_UNDER_INLINE_r12: {
CHECK_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef old;
_PyStackRef value;
_PyStackRef new;
_PyStackRef _stack_item_0 = _tos_cache0;
old = _stack_item_0;
GH-140683: JIT: Improve machine code for loading smaller constants on AArch64. (GH-142511) * Use movz and movk instructions for loading 16 and 32 bit operands and oparg. * Loading of 64 bit operands is unchanged.
2025-12-11 12:33:39 +00:00
PyObject *ptr = (PyObject *)CURRENT_OPERAND0_64();
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
new = old;
value = PyStackRef_FromPyObjectNew(ptr);
_tos_cache1 = new;
_tos_cache0 = value;
SET_CURRENT_CACHED_VALUES(2);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _LOAD_CONST_UNDER_INLINE_r23: {
CHECK_CURRENT_CACHED_VALUES(2);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef old;
_PyStackRef value;
_PyStackRef new;
_PyStackRef _stack_item_0 = _tos_cache0;
_PyStackRef _stack_item_1 = _tos_cache1;
old = _stack_item_1;
GH-140683: JIT: Improve machine code for loading smaller constants on AArch64. (GH-142511) * Use movz and movk instructions for loading 16 and 32 bit operands and oparg. * Loading of 64 bit operands is unchanged.
2025-12-11 12:33:39 +00:00
PyObject *ptr = (PyObject *)CURRENT_OPERAND0_64();
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
new = old;
value = PyStackRef_FromPyObjectNew(ptr);
_tos_cache2 = new;
_tos_cache1 = value;
_tos_cache0 = _stack_item_0;
SET_CURRENT_CACHED_VALUES(3);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _LOAD_CONST_UNDER_INLINE_BORROW_r02: {
CHECK_CURRENT_CACHED_VALUES(0);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef old;
_PyStackRef value;
_PyStackRef new;
old = stack_pointer[-1];
GH-140683: JIT: Improve machine code for loading smaller constants on AArch64. (GH-142511) * Use movz and movk instructions for loading 16 and 32 bit operands and oparg. * Loading of 64 bit operands is unchanged.
2025-12-11 12:33:39 +00:00
PyObject *ptr = (PyObject *)CURRENT_OPERAND0_64();
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
new = old;
value = PyStackRef_FromPyObjectBorrow(ptr);
_tos_cache1 = new;
_tos_cache0 = value;
SET_CURRENT_CACHED_VALUES(2);
stack_pointer += -1;
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _LOAD_CONST_UNDER_INLINE_BORROW_r12: {
CHECK_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef old;
_PyStackRef value;
_PyStackRef new;
_PyStackRef _stack_item_0 = _tos_cache0;
old = _stack_item_0;
GH-140683: JIT: Improve machine code for loading smaller constants on AArch64. (GH-142511) * Use movz and movk instructions for loading 16 and 32 bit operands and oparg. * Loading of 64 bit operands is unchanged.
2025-12-11 12:33:39 +00:00
PyObject *ptr = (PyObject *)CURRENT_OPERAND0_64();
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
new = old;
value = PyStackRef_FromPyObjectBorrow(ptr);
_tos_cache1 = new;
_tos_cache0 = value;
SET_CURRENT_CACHED_VALUES(2);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _LOAD_CONST_UNDER_INLINE_BORROW_r23: {
CHECK_CURRENT_CACHED_VALUES(2);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef old;
_PyStackRef value;
_PyStackRef new;
_PyStackRef _stack_item_0 = _tos_cache0;
_PyStackRef _stack_item_1 = _tos_cache1;
old = _stack_item_1;
GH-140683: JIT: Improve machine code for loading smaller constants on AArch64. (GH-142511) * Use movz and movk instructions for loading 16 and 32 bit operands and oparg. * Loading of 64 bit operands is unchanged.
2025-12-11 12:33:39 +00:00
PyObject *ptr = (PyObject *)CURRENT_OPERAND0_64();
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
new = old;
value = PyStackRef_FromPyObjectBorrow(ptr);
_tos_cache2 = new;
_tos_cache1 = value;
_tos_cache0 = _stack_item_0;
SET_CURRENT_CACHED_VALUES(3);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _START_EXECUTOR_r00: {
CHECK_CURRENT_CACHED_VALUES(0);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-140683: JIT: Improve machine code for loading smaller constants on AArch64. (GH-142511) * Use movz and movk instructions for loading 16 and 32 bit operands and oparg. * Loading of 64 bit operands is unchanged.
2025-12-11 12:33:39 +00:00
PyObject *executor = (PyObject *)CURRENT_OPERAND0_64();
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
#ifndef _Py_JIT
assert(current_executor == (_PyExecutorObject*)executor);
#endif
assert(tstate->jit_exit == NULL || tstate->jit_exit->executor == current_executor);
JIT: Move executor to a register (#143072)
2025-12-24 09:44:16 +00:00
tstate->current_executor = (PyObject *)current_executor;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
if (!current_executor->vm_data.valid) {
assert(tstate->jit_exit->executor == current_executor);
assert(tstate->current_executor == executor);
_PyFrame_SetStackPointer(frame, stack_pointer);
_PyExecutor_ClearExit(tstate->jit_exit);
stack_pointer = _PyFrame_GetStackPointer(frame);
if (true) {
UOP_STAT_INC(uopcode, miss);
SET_CURRENT_CACHED_VALUES(0);
JUMP_TO_JUMP_TARGET();
}
}
SET_CURRENT_CACHED_VALUES(0);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _MAKE_WARM_r00: {
CHECK_CURRENT_CACHED_VALUES(0);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
current_executor->vm_data.warm = true;
SET_CURRENT_CACHED_VALUES(0);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _MAKE_WARM_r11: {
CHECK_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef _stack_item_0 = _tos_cache0;
current_executor->vm_data.warm = true;
_tos_cache0 = _stack_item_0;
SET_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _MAKE_WARM_r22: {
CHECK_CURRENT_CACHED_VALUES(2);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef _stack_item_0 = _tos_cache0;
_PyStackRef _stack_item_1 = _tos_cache1;
current_executor->vm_data.warm = true;
_tos_cache1 = _stack_item_1;
_tos_cache0 = _stack_item_0;
SET_CURRENT_CACHED_VALUES(2);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _MAKE_WARM_r33: {
CHECK_CURRENT_CACHED_VALUES(3);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef _stack_item_0 = _tos_cache0;
_PyStackRef _stack_item_1 = _tos_cache1;
_PyStackRef _stack_item_2 = _tos_cache2;
current_executor->vm_data.warm = true;
_tos_cache2 = _stack_item_2;
_tos_cache1 = _stack_item_1;
_tos_cache0 = _stack_item_0;
SET_CURRENT_CACHED_VALUES(3);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _FATAL_ERROR_r00: {
CHECK_CURRENT_CACHED_VALUES(0);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
assert(0);
Py_FatalError("Fatal error uop executed.");
SET_CURRENT_CACHED_VALUES(0);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _FATAL_ERROR_r11: {
CHECK_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef _stack_item_0 = _tos_cache0;
assert(0);
Py_FatalError("Fatal error uop executed.");
_tos_cache0 = _stack_item_0;
SET_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _FATAL_ERROR_r22: {
CHECK_CURRENT_CACHED_VALUES(2);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef _stack_item_0 = _tos_cache0;
_PyStackRef _stack_item_1 = _tos_cache1;
assert(0);
Py_FatalError("Fatal error uop executed.");
_tos_cache1 = _stack_item_1;
_tos_cache0 = _stack_item_0;
SET_CURRENT_CACHED_VALUES(2);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _FATAL_ERROR_r33: {
CHECK_CURRENT_CACHED_VALUES(3);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef _stack_item_0 = _tos_cache0;
_PyStackRef _stack_item_1 = _tos_cache1;
_PyStackRef _stack_item_2 = _tos_cache2;
assert(0);
Py_FatalError("Fatal error uop executed.");
_tos_cache2 = _stack_item_2;
_tos_cache1 = _stack_item_1;
_tos_cache0 = _stack_item_0;
SET_CURRENT_CACHED_VALUES(3);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _DEOPT_r00: {
CHECK_CURRENT_CACHED_VALUES(0);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(0);
GOTO_TIER_ONE((frame->owner == FRAME_OWNED_BY_INTERPRETER)
? _Py_INTERPRETER_TRAMPOLINE_INSTRUCTIONS_PTR : _PyFrame_GetBytecode(frame) + CURRENT_TARGET());
}
case _DEOPT_r10: {
CHECK_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef _stack_item_0 = _tos_cache0;
stack_pointer[0] = _stack_item_0;
stack_pointer += 1;
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
SET_CURRENT_CACHED_VALUES(0);
GOTO_TIER_ONE((frame->owner == FRAME_OWNED_BY_INTERPRETER)
? _Py_INTERPRETER_TRAMPOLINE_INSTRUCTIONS_PTR : _PyFrame_GetBytecode(frame) + CURRENT_TARGET());
}
case _DEOPT_r20: {
CHECK_CURRENT_CACHED_VALUES(2);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef _stack_item_0 = _tos_cache0;
_PyStackRef _stack_item_1 = _tos_cache1;
stack_pointer[0] = _stack_item_0;
stack_pointer[1] = _stack_item_1;
stack_pointer += 2;
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
SET_CURRENT_CACHED_VALUES(0);
GOTO_TIER_ONE((frame->owner == FRAME_OWNED_BY_INTERPRETER)
? _Py_INTERPRETER_TRAMPOLINE_INSTRUCTIONS_PTR : _PyFrame_GetBytecode(frame) + CURRENT_TARGET());
}
case _DEOPT_r30: {
CHECK_CURRENT_CACHED_VALUES(3);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef _stack_item_0 = _tos_cache0;
_PyStackRef _stack_item_1 = _tos_cache1;
_PyStackRef _stack_item_2 = _tos_cache2;
stack_pointer[0] = _stack_item_0;
stack_pointer[1] = _stack_item_1;
stack_pointer[2] = _stack_item_2;
stack_pointer += 3;
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
SET_CURRENT_CACHED_VALUES(0);
GOTO_TIER_ONE((frame->owner == FRAME_OWNED_BY_INTERPRETER)
? _Py_INTERPRETER_TRAMPOLINE_INSTRUCTIONS_PTR : _PyFrame_GetBytecode(frame) + CURRENT_TARGET());
}
case _HANDLE_PENDING_AND_DEOPT_r00: {
CHECK_CURRENT_CACHED_VALUES(0);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyFrame_SetStackPointer(frame, stack_pointer);
int err = _Py_HandlePending(tstate);
stack_pointer = _PyFrame_GetStackPointer(frame);
SET_CURRENT_CACHED_VALUES(0);
GOTO_TIER_ONE(err ? NULL : _PyFrame_GetBytecode(frame) + CURRENT_TARGET());
}
case _HANDLE_PENDING_AND_DEOPT_r10: {
CHECK_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef _stack_item_0 = _tos_cache0;
stack_pointer[0] = _stack_item_0;
stack_pointer += 1;
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
_PyFrame_SetStackPointer(frame, stack_pointer);
int err = _Py_HandlePending(tstate);
stack_pointer = _PyFrame_GetStackPointer(frame);
SET_CURRENT_CACHED_VALUES(0);
GOTO_TIER_ONE(err ? NULL : _PyFrame_GetBytecode(frame) + CURRENT_TARGET());
}
case _HANDLE_PENDING_AND_DEOPT_r20: {
CHECK_CURRENT_CACHED_VALUES(2);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef _stack_item_0 = _tos_cache0;
_PyStackRef _stack_item_1 = _tos_cache1;
stack_pointer[0] = _stack_item_0;
stack_pointer[1] = _stack_item_1;
stack_pointer += 2;
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
_PyFrame_SetStackPointer(frame, stack_pointer);
int err = _Py_HandlePending(tstate);
stack_pointer = _PyFrame_GetStackPointer(frame);
SET_CURRENT_CACHED_VALUES(0);
GOTO_TIER_ONE(err ? NULL : _PyFrame_GetBytecode(frame) + CURRENT_TARGET());
}
case _HANDLE_PENDING_AND_DEOPT_r30: {
CHECK_CURRENT_CACHED_VALUES(3);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef _stack_item_0 = _tos_cache0;
_PyStackRef _stack_item_1 = _tos_cache1;
_PyStackRef _stack_item_2 = _tos_cache2;
stack_pointer[0] = _stack_item_0;
stack_pointer[1] = _stack_item_1;
stack_pointer[2] = _stack_item_2;
stack_pointer += 3;
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
_PyFrame_SetStackPointer(frame, stack_pointer);
int err = _Py_HandlePending(tstate);
stack_pointer = _PyFrame_GetStackPointer(frame);
SET_CURRENT_CACHED_VALUES(0);
GOTO_TIER_ONE(err ? NULL : _PyFrame_GetBytecode(frame) + CURRENT_TARGET());
}
case _ERROR_POP_N_r00: {
CHECK_CURRENT_CACHED_VALUES(0);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
oparg = CURRENT_OPARG();
GH-140683: JIT: Improve machine code for loading smaller constants on AArch64. (GH-142511) * Use movz and movk instructions for loading 16 and 32 bit operands and oparg. * Loading of 64 bit operands is unchanged.
2025-12-11 12:33:39 +00:00
uint32_t target = (uint32_t)CURRENT_OPERAND0_32();
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
assert(oparg == 0);
frame->instr_ptr = _PyFrame_GetBytecode(frame) + target;
SET_CURRENT_CACHED_VALUES(0);
GOTO_TIER_ONE(NULL);
}
case _SPILL_OR_RELOAD_r01: {
CHECK_CURRENT_CACHED_VALUES(0);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_tos_cache0 = stack_pointer[-1];
SET_CURRENT_CACHED_VALUES(1);
stack_pointer += -1;
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _SPILL_OR_RELOAD_r02: {
CHECK_CURRENT_CACHED_VALUES(0);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_tos_cache1 = stack_pointer[-1];
_tos_cache0 = stack_pointer[-2];
SET_CURRENT_CACHED_VALUES(2);
stack_pointer += -2;
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _SPILL_OR_RELOAD_r03: {
CHECK_CURRENT_CACHED_VALUES(0);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_tos_cache2 = stack_pointer[-1];
_tos_cache1 = stack_pointer[-2];
_tos_cache0 = stack_pointer[-3];
SET_CURRENT_CACHED_VALUES(3);
stack_pointer += -3;
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _SPILL_OR_RELOAD_r10: {
CHECK_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef _stack_item_0 = _tos_cache0;
SET_CURRENT_CACHED_VALUES(0);
stack_pointer[0] = _stack_item_0;
GH-131798: Optimize cached class attributes and methods in the JIT (GH-134403)
2025-05-22 11:15:03 -04:00
stack_pointer += 1;
GH-141794: Limit size of generated machine code. (GH-142228) * Factor out bodies of the largest uops, to reduce jit code size. * Factor out common assert, also reducing jit code size. * Limit size of jitted code for a single executor to 1MB.
2025-12-03 17:43:35 +00:00
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _SPILL_OR_RELOAD_r12: {
CHECK_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef _stack_item_0 = _tos_cache0;
_tos_cache1 = _stack_item_0;
_tos_cache0 = stack_pointer[-1];
SET_CURRENT_CACHED_VALUES(2);
stack_pointer += -1;
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _SPILL_OR_RELOAD_r13: {
CHECK_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef _stack_item_0 = _tos_cache0;
_tos_cache2 = _stack_item_0;
_tos_cache1 = stack_pointer[-1];
_tos_cache0 = stack_pointer[-2];
SET_CURRENT_CACHED_VALUES(3);
stack_pointer += -2;
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _SPILL_OR_RELOAD_r20: {
CHECK_CURRENT_CACHED_VALUES(2);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef _stack_item_0 = _tos_cache0;
_PyStackRef _stack_item_1 = _tos_cache1;
SET_CURRENT_CACHED_VALUES(0);
stack_pointer[0] = _stack_item_0;
stack_pointer[1] = _stack_item_1;
stack_pointer += 2;
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-131798: Optimize cached class attributes and methods in the JIT (GH-134403)
2025-05-22 11:15:03 -04:00
break;
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
case _SPILL_OR_RELOAD_r21: {
CHECK_CURRENT_CACHED_VALUES(2);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef _stack_item_0 = _tos_cache0;
_PyStackRef _stack_item_1 = _tos_cache1;
_tos_cache0 = _stack_item_1;
SET_CURRENT_CACHED_VALUES(1);
stack_pointer[0] = _stack_item_0;
GH-131798: Optimize cached class attributes and methods in the JIT (GH-134403)
2025-05-22 11:15:03 -04:00
stack_pointer += 1;
GH-141794: Limit size of generated machine code. (GH-142228) * Factor out bodies of the largest uops, to reduce jit code size. * Factor out common assert, also reducing jit code size. * Limit size of jitted code for a single executor to 1MB.
2025-12-03 17:43:35 +00:00
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-131798: Optimize cached class attributes and methods in the JIT (GH-134403)
2025-05-22 11:15:03 -04:00
break;
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
case _SPILL_OR_RELOAD_r23: {
CHECK_CURRENT_CACHED_VALUES(2);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef _stack_item_0 = _tos_cache0;
_PyStackRef _stack_item_1 = _tos_cache1;
_tos_cache2 = _stack_item_1;
_tos_cache1 = _stack_item_0;
_tos_cache0 = stack_pointer[-1];
SET_CURRENT_CACHED_VALUES(3);
stack_pointer += -1;
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-112354: Initial implementation of warm up on exits and trace-stitching (GH-114142)
2024-02-20 09:39:55 +00:00
break;
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
case _SPILL_OR_RELOAD_r30: {
CHECK_CURRENT_CACHED_VALUES(3);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef _stack_item_0 = _tos_cache0;
_PyStackRef _stack_item_1 = _tos_cache1;
_PyStackRef _stack_item_2 = _tos_cache2;
SET_CURRENT_CACHED_VALUES(0);
stack_pointer[0] = _stack_item_0;
stack_pointer[1] = _stack_item_1;
stack_pointer[2] = _stack_item_2;
stack_pointer += 3;
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-123516: Improve JIT memory consumption by invalidating cold executors (GH-124443) Co-authored-by: Bénédikt Tran <10796600+picnixz@users.noreply.github.com>
2024-09-26 17:35:42 -07:00
break;
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
case _SPILL_OR_RELOAD_r31: {
CHECK_CURRENT_CACHED_VALUES(3);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef _stack_item_0 = _tos_cache0;
_PyStackRef _stack_item_1 = _tos_cache1;
_PyStackRef _stack_item_2 = _tos_cache2;
_tos_cache0 = _stack_item_2;
SET_CURRENT_CACHED_VALUES(1);
stack_pointer[0] = _stack_item_0;
stack_pointer[1] = _stack_item_1;
stack_pointer += 2;
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-112354: Initial implementation of warm up on exits and trace-stitching (GH-114142)
2024-02-20 09:39:55 +00:00
break;
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
case _SPILL_OR_RELOAD_r32: {
CHECK_CURRENT_CACHED_VALUES(3);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef _stack_item_0 = _tos_cache0;
_PyStackRef _stack_item_1 = _tos_cache1;
_PyStackRef _stack_item_2 = _tos_cache2;
_tos_cache1 = _stack_item_2;
_tos_cache0 = _stack_item_1;
SET_CURRENT_CACHED_VALUES(2);
stack_pointer[0] = _stack_item_0;
stack_pointer += 1;
ASSERT_WITHIN_STACK_BOUNDS(__FILE__, __LINE__);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-116422: Tier2 hot/cold splitting (GH-116813) Splits the "cold" path, deopts and exits, from the "hot" path, reducing the size of most jitted instructions, at the cost of slower exits.
2024-03-26 09:35:11 +00:00
break;
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
case _TIER2_RESUME_CHECK_r00: {
CHECK_CURRENT_CACHED_VALUES(0);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
#if defined(__EMSCRIPTEN__)
if (_Py_emscripten_signal_clock == 0) {
UOP_STAT_INC(uopcode, miss);
SET_CURRENT_CACHED_VALUES(0);
JUMP_TO_JUMP_TARGET();
}
_Py_emscripten_signal_clock -= Py_EMSCRIPTEN_SIGNAL_HANDLING;
#endif
uintptr_t eval_breaker = _Py_atomic_load_uintptr_relaxed(&tstate->eval_breaker);
if (eval_breaker & _PY_EVAL_EVENTS_MASK) {
UOP_STAT_INC(uopcode, miss);
SET_CURRENT_CACHED_VALUES(0);
JUMP_TO_JUMP_TARGET();
}
assert(tstate->tracing || eval_breaker == FT_ATOMIC_LOAD_UINTPTR_ACQUIRE(_PyFrame_GetCode(frame)->_co_instrumentation_version));
SET_CURRENT_CACHED_VALUES(0);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-132532: Add new DSL macros to better declare semantics of exits at ends of instructions/uops. (GH-137098)
2025-08-09 15:41:28 +01:00
break;
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
case _TIER2_RESUME_CHECK_r11: {
CHECK_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef _stack_item_0 = _tos_cache0;
#if defined(__EMSCRIPTEN__)
if (_Py_emscripten_signal_clock == 0) {
UOP_STAT_INC(uopcode, miss);
GH-142718: JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. (GH-142762) JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. * Make sure that stack is in correct state at side exits with TOS cached values * Simplify choice of cached items for side exits
2025-12-15 19:01:29 +00:00
_tos_cache0 = _stack_item_0;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(1);
JUMP_TO_JUMP_TARGET();
}
_Py_emscripten_signal_clock -= Py_EMSCRIPTEN_SIGNAL_HANDLING;
#endif
uintptr_t eval_breaker = _Py_atomic_load_uintptr_relaxed(&tstate->eval_breaker);
if (eval_breaker & _PY_EVAL_EVENTS_MASK) {
UOP_STAT_INC(uopcode, miss);
GH-142718: JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. (GH-142762) JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. * Make sure that stack is in correct state at side exits with TOS cached values * Simplify choice of cached items for side exits
2025-12-15 19:01:29 +00:00
_tos_cache0 = _stack_item_0;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(1);
JUMP_TO_JUMP_TARGET();
}
assert(tstate->tracing || eval_breaker == FT_ATOMIC_LOAD_UINTPTR_ACQUIRE(_PyFrame_GetCode(frame)->_co_instrumentation_version));
_tos_cache0 = _stack_item_0;
SET_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _TIER2_RESUME_CHECK_r22: {
CHECK_CURRENT_CACHED_VALUES(2);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef _stack_item_0 = _tos_cache0;
_PyStackRef _stack_item_1 = _tos_cache1;
#if defined(__EMSCRIPTEN__)
if (_Py_emscripten_signal_clock == 0) {
UOP_STAT_INC(uopcode, miss);
GH-142718: JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. (GH-142762) JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. * Make sure that stack is in correct state at side exits with TOS cached values * Simplify choice of cached items for side exits
2025-12-15 19:01:29 +00:00
_tos_cache1 = _stack_item_1;
_tos_cache0 = _stack_item_0;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(2);
JUMP_TO_JUMP_TARGET();
}
_Py_emscripten_signal_clock -= Py_EMSCRIPTEN_SIGNAL_HANDLING;
#endif
uintptr_t eval_breaker = _Py_atomic_load_uintptr_relaxed(&tstate->eval_breaker);
if (eval_breaker & _PY_EVAL_EVENTS_MASK) {
UOP_STAT_INC(uopcode, miss);
GH-142718: JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. (GH-142762) JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. * Make sure that stack is in correct state at side exits with TOS cached values * Simplify choice of cached items for side exits
2025-12-15 19:01:29 +00:00
_tos_cache1 = _stack_item_1;
_tos_cache0 = _stack_item_0;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(2);
JUMP_TO_JUMP_TARGET();
}
assert(tstate->tracing || eval_breaker == FT_ATOMIC_LOAD_UINTPTR_ACQUIRE(_PyFrame_GetCode(frame)->_co_instrumentation_version));
_tos_cache1 = _stack_item_1;
_tos_cache0 = _stack_item_0;
SET_CURRENT_CACHED_VALUES(2);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-116422: Tier2 hot/cold splitting (GH-116813) Splits the "cold" path, deopts and exits, from the "hot" path, reducing the size of most jitted instructions, at the cost of slower exits.
2024-03-26 09:35:11 +00:00
break;
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
case _TIER2_RESUME_CHECK_r33: {
CHECK_CURRENT_CACHED_VALUES(3);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef _stack_item_0 = _tos_cache0;
_PyStackRef _stack_item_1 = _tos_cache1;
_PyStackRef _stack_item_2 = _tos_cache2;
GH-118095: Allow a variant of RESUME_CHECK in tier 2 (GH-118286)
2024-04-29 07:54:05 +01:00
#if defined(__EMSCRIPTEN__)
if (_Py_emscripten_signal_clock == 0) {
UOP_STAT_INC(uopcode, miss);
GH-142718: JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. (GH-142762) JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. * Make sure that stack is in correct state at side exits with TOS cached values * Simplify choice of cached items for side exits
2025-12-15 19:01:29 +00:00
_tos_cache2 = _stack_item_2;
_tos_cache1 = _stack_item_1;
_tos_cache0 = _stack_item_0;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(3);
GH-118095: Allow a variant of RESUME_CHECK in tier 2 (GH-118286)
2024-04-29 07:54:05 +01:00
JUMP_TO_JUMP_TARGET();
}
_Py_emscripten_signal_clock -= Py_EMSCRIPTEN_SIGNAL_HANDLING;
#endif
uintptr_t eval_breaker = _Py_atomic_load_uintptr_relaxed(&tstate->eval_breaker);
if (eval_breaker & _PY_EVAL_EVENTS_MASK) {
UOP_STAT_INC(uopcode, miss);
GH-142718: JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. (GH-142762) JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. * Make sure that stack is in correct state at side exits with TOS cached values * Simplify choice of cached items for side exits
2025-12-15 19:01:29 +00:00
_tos_cache2 = _stack_item_2;
_tos_cache1 = _stack_item_1;
_tos_cache0 = _stack_item_0;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(3);
GH-118095: Allow a variant of RESUME_CHECK in tier 2 (GH-118286)
2024-04-29 07:54:05 +01:00
JUMP_TO_JUMP_TARGET();
}
gh-107674: Improve performance of `sys.settrace` (GH-117133) * Check tracing in RESUME_CHECK * Only change to RESUME_CHECK if not tracing
2024-05-03 11:49:24 -07:00
assert(tstate->tracing || eval_breaker == FT_ATOMIC_LOAD_UINTPTR_ACQUIRE(_PyFrame_GetCode(frame)->_co_instrumentation_version));
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_tos_cache2 = _stack_item_2;
_tos_cache1 = _stack_item_1;
_tos_cache0 = _stack_item_0;
SET_CURRENT_CACHED_VALUES(3);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-118095: Allow a variant of RESUME_CHECK in tier 2 (GH-118286)
2024-04-29 07:54:05 +01:00
break;
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
case _COLD_EXIT_r00: {
CHECK_CURRENT_CACHED_VALUES(0);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-136410: Faster side exits by using a cold exit stub (GH-136411)
2025-08-01 16:26:07 +01:00
_PyExitData *exit = tstate->jit_exit;
assert(exit != NULL);
gh-139109: A new tracing JIT compiler frontend for CPython (GH-140310) This PR changes the current JIT model from trace projection to trace recording. Benchmarking: better pyperformance (about 1.7% overall) geomean versus current https://raw.githubusercontent.com/facebookexperimental/free-threading-benchmarking/refs/heads/main/results/bm-20251108-3.15.0a1%2B-7e2bc1d-JIT/bm-20251108-vultr-x86_64-Fidget%252dSpinner-tracing_jit-3.15.0a1%2B-7e2bc1d-vs-base.svg, 100% faster Richards on the most improved benchmark versus the current JIT. Slowdown of about 10-15% on the worst benchmark versus the current JIT. **Note: the fastest version isn't the one merged, as it relies on fixing bugs in the specializing interpreter, which is left to another PR**. The speedup in the merged version is about 1.1%. https://raw.githubusercontent.com/facebookexperimental/free-threading-benchmarking/refs/heads/main/results/bm-20251112-3.15.0a1%2B-f8a764a-JIT/bm-20251112-vultr-x86_64-Fidget%252dSpinner-tracing_jit-3.15.0a1%2B-f8a764a-vs-base.svg Stats: 50% more uops executed, 30% more traces entered the last time we ran them. It also suggests our trace lengths for a real trace recording JIT are too short, as a lot of trace too long aborts https://github.com/facebookexperimental/free-threading-benchmarking/blob/main/results/bm-20251023-3.15.0a1%2B-eb73378-CLANG%2CJIT/bm-20251023-vultr-x86_64-Fidget%252dSpinner-tracing_jit-3.15.0a1%2B-eb73378-pystats-vs-base.md . This new JIT frontend is already able to record/execute significantly more instructions than the previous JIT frontend. In this PR, we are now able to record through custom dunders, simple object creation, generators, etc. None of these were done by the old JIT frontend. Some custom dunders uops were discovered to be broken as part of this work gh-140277 The optimizer stack space check is disabled, as it's no longer valid to deal with underflow. Pros: * Ignoring the generated tracer code as it's automatically created, this is only additional 1k lines of code. The maintenance burden is handled by the DSL and code generator. * `optimizer.c` is now significantly simpler, as we don't have to do strange things to recover the bytecode from a trace. * The new JIT frontend is able to handle a lot more control-flow than the old one. * Tracing is very low overhead. We use the tail calling interpreter/computed goto interpreter to switch between tracing mode and non-tracing mode. I call this mechanism dual dispatch, as we have two dispatch tables dispatching to each other. Specialization is still enabled while tracing. * Better handling of polymorphism. We leverage the specializing interpreter for this. Cons: * (For now) requires tail calling interpreter or computed gotos. This means no Windows JIT for now :(. Not to fret, tail calling is coming soon to Windows though https://github.com/python/cpython/pull/139962 Design: * After each instruction, the `record_previous_inst` function/label is executed. This does as the name suggests. * The tracing interpreter lowers bytecode to uops directly so that it can obtain "fresh" values at the point of lowering. * The tracing version behaves nearly identical to the normal interpreter, in fact it even has specialization! This allows it to run without much of a slowdown when tracing. The actual cost of tracing is only a function call and writes to memory. * The tracing interpreter uses the specializing interpreter's deopt to naturally form the side exit chains. This allows it to side exit chain effectively, without repeating much code. We force a re-specializing when tracing a deopt. * The tracing interpreter can even handle goto errors/exceptions, but I chose to disable them for now as it's not tested. * Because we do not share interpreter dispatch, there is should be no significant slowdown to the original specializing interpreter on tailcall and computed got with JIT disabled. With JIT enabled, there might be a slowdown in the form of the JIT trying to trace. * Things that could have dynamic instruction pointer effects are guarded on. The guard deopts to a new instruction --- `_DYNAMIC_EXIT`.
2025-11-14 02:08:32 +08:00
assert(frame->owner < FRAME_OWNED_BY_INTERPRETER);
GH-136410: Faster side exits by using a cold exit stub (GH-136411)
2025-08-01 16:26:07 +01:00
_Py_CODEUNIT *target = _PyFrame_GetBytecode(frame) + exit->target;
_Py_BackoffCounter temperature = exit->temperature;
_PyExecutorObject *executor;
if (target->op.code == ENTER_EXECUTOR) {
PyCodeObject *code = _PyFrame_GetCode(frame);
executor = code->co_executors->executors[target->op.arg];
Py_INCREF(executor);
gh-139109: A new tracing JIT compiler frontend for CPython (GH-140310) This PR changes the current JIT model from trace projection to trace recording. Benchmarking: better pyperformance (about 1.7% overall) geomean versus current https://raw.githubusercontent.com/facebookexperimental/free-threading-benchmarking/refs/heads/main/results/bm-20251108-3.15.0a1%2B-7e2bc1d-JIT/bm-20251108-vultr-x86_64-Fidget%252dSpinner-tracing_jit-3.15.0a1%2B-7e2bc1d-vs-base.svg, 100% faster Richards on the most improved benchmark versus the current JIT. Slowdown of about 10-15% on the worst benchmark versus the current JIT. **Note: the fastest version isn't the one merged, as it relies on fixing bugs in the specializing interpreter, which is left to another PR**. The speedup in the merged version is about 1.1%. https://raw.githubusercontent.com/facebookexperimental/free-threading-benchmarking/refs/heads/main/results/bm-20251112-3.15.0a1%2B-f8a764a-JIT/bm-20251112-vultr-x86_64-Fidget%252dSpinner-tracing_jit-3.15.0a1%2B-f8a764a-vs-base.svg Stats: 50% more uops executed, 30% more traces entered the last time we ran them. It also suggests our trace lengths for a real trace recording JIT are too short, as a lot of trace too long aborts https://github.com/facebookexperimental/free-threading-benchmarking/blob/main/results/bm-20251023-3.15.0a1%2B-eb73378-CLANG%2CJIT/bm-20251023-vultr-x86_64-Fidget%252dSpinner-tracing_jit-3.15.0a1%2B-eb73378-pystats-vs-base.md . This new JIT frontend is already able to record/execute significantly more instructions than the previous JIT frontend. In this PR, we are now able to record through custom dunders, simple object creation, generators, etc. None of these were done by the old JIT frontend. Some custom dunders uops were discovered to be broken as part of this work gh-140277 The optimizer stack space check is disabled, as it's no longer valid to deal with underflow. Pros: * Ignoring the generated tracer code as it's automatically created, this is only additional 1k lines of code. The maintenance burden is handled by the DSL and code generator. * `optimizer.c` is now significantly simpler, as we don't have to do strange things to recover the bytecode from a trace. * The new JIT frontend is able to handle a lot more control-flow than the old one. * Tracing is very low overhead. We use the tail calling interpreter/computed goto interpreter to switch between tracing mode and non-tracing mode. I call this mechanism dual dispatch, as we have two dispatch tables dispatching to each other. Specialization is still enabled while tracing. * Better handling of polymorphism. We leverage the specializing interpreter for this. Cons: * (For now) requires tail calling interpreter or computed gotos. This means no Windows JIT for now :(. Not to fret, tail calling is coming soon to Windows though https://github.com/python/cpython/pull/139962 Design: * After each instruction, the `record_previous_inst` function/label is executed. This does as the name suggests. * The tracing interpreter lowers bytecode to uops directly so that it can obtain "fresh" values at the point of lowering. * The tracing version behaves nearly identical to the normal interpreter, in fact it even has specialization! This allows it to run without much of a slowdown when tracing. The actual cost of tracing is only a function call and writes to memory. * The tracing interpreter uses the specializing interpreter's deopt to naturally form the side exit chains. This allows it to side exit chain effectively, without repeating much code. We force a re-specializing when tracing a deopt. * The tracing interpreter can even handle goto errors/exceptions, but I chose to disable them for now as it's not tested. * Because we do not share interpreter dispatch, there is should be no significant slowdown to the original specializing interpreter on tailcall and computed got with JIT disabled. With JIT enabled, there might be a slowdown in the form of the JIT trying to trace. * Things that could have dynamic instruction pointer effects are guarded on. The guard deopts to a new instruction --- `_DYNAMIC_EXIT`.
2025-11-14 02:08:32 +08:00
assert(tstate->jit_exit == exit);
exit->executor = executor;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(0);
gh-139109: A new tracing JIT compiler frontend for CPython (GH-140310) This PR changes the current JIT model from trace projection to trace recording. Benchmarking: better pyperformance (about 1.7% overall) geomean versus current https://raw.githubusercontent.com/facebookexperimental/free-threading-benchmarking/refs/heads/main/results/bm-20251108-3.15.0a1%2B-7e2bc1d-JIT/bm-20251108-vultr-x86_64-Fidget%252dSpinner-tracing_jit-3.15.0a1%2B-7e2bc1d-vs-base.svg, 100% faster Richards on the most improved benchmark versus the current JIT. Slowdown of about 10-15% on the worst benchmark versus the current JIT. **Note: the fastest version isn't the one merged, as it relies on fixing bugs in the specializing interpreter, which is left to another PR**. The speedup in the merged version is about 1.1%. https://raw.githubusercontent.com/facebookexperimental/free-threading-benchmarking/refs/heads/main/results/bm-20251112-3.15.0a1%2B-f8a764a-JIT/bm-20251112-vultr-x86_64-Fidget%252dSpinner-tracing_jit-3.15.0a1%2B-f8a764a-vs-base.svg Stats: 50% more uops executed, 30% more traces entered the last time we ran them. It also suggests our trace lengths for a real trace recording JIT are too short, as a lot of trace too long aborts https://github.com/facebookexperimental/free-threading-benchmarking/blob/main/results/bm-20251023-3.15.0a1%2B-eb73378-CLANG%2CJIT/bm-20251023-vultr-x86_64-Fidget%252dSpinner-tracing_jit-3.15.0a1%2B-eb73378-pystats-vs-base.md . This new JIT frontend is already able to record/execute significantly more instructions than the previous JIT frontend. In this PR, we are now able to record through custom dunders, simple object creation, generators, etc. None of these were done by the old JIT frontend. Some custom dunders uops were discovered to be broken as part of this work gh-140277 The optimizer stack space check is disabled, as it's no longer valid to deal with underflow. Pros: * Ignoring the generated tracer code as it's automatically created, this is only additional 1k lines of code. The maintenance burden is handled by the DSL and code generator. * `optimizer.c` is now significantly simpler, as we don't have to do strange things to recover the bytecode from a trace. * The new JIT frontend is able to handle a lot more control-flow than the old one. * Tracing is very low overhead. We use the tail calling interpreter/computed goto interpreter to switch between tracing mode and non-tracing mode. I call this mechanism dual dispatch, as we have two dispatch tables dispatching to each other. Specialization is still enabled while tracing. * Better handling of polymorphism. We leverage the specializing interpreter for this. Cons: * (For now) requires tail calling interpreter or computed gotos. This means no Windows JIT for now :(. Not to fret, tail calling is coming soon to Windows though https://github.com/python/cpython/pull/139962 Design: * After each instruction, the `record_previous_inst` function/label is executed. This does as the name suggests. * The tracing interpreter lowers bytecode to uops directly so that it can obtain "fresh" values at the point of lowering. * The tracing version behaves nearly identical to the normal interpreter, in fact it even has specialization! This allows it to run without much of a slowdown when tracing. The actual cost of tracing is only a function call and writes to memory. * The tracing interpreter uses the specializing interpreter's deopt to naturally form the side exit chains. This allows it to side exit chain effectively, without repeating much code. We force a re-specializing when tracing a deopt. * The tracing interpreter can even handle goto errors/exceptions, but I chose to disable them for now as it's not tested. * Because we do not share interpreter dispatch, there is should be no significant slowdown to the original specializing interpreter on tailcall and computed got with JIT disabled. With JIT enabled, there might be a slowdown in the form of the JIT trying to trace. * Things that could have dynamic instruction pointer effects are guarded on. The guard deopts to a new instruction --- `_DYNAMIC_EXIT`.
2025-11-14 02:08:32 +08:00
TIER2_TO_TIER2(exit->executor);
GH-136410: Faster side exits by using a cold exit stub (GH-136411)
2025-08-01 16:26:07 +01:00
}
else {
gh-139109: A new tracing JIT compiler frontend for CPython (GH-140310) This PR changes the current JIT model from trace projection to trace recording. Benchmarking: better pyperformance (about 1.7% overall) geomean versus current https://raw.githubusercontent.com/facebookexperimental/free-threading-benchmarking/refs/heads/main/results/bm-20251108-3.15.0a1%2B-7e2bc1d-JIT/bm-20251108-vultr-x86_64-Fidget%252dSpinner-tracing_jit-3.15.0a1%2B-7e2bc1d-vs-base.svg, 100% faster Richards on the most improved benchmark versus the current JIT. Slowdown of about 10-15% on the worst benchmark versus the current JIT. **Note: the fastest version isn't the one merged, as it relies on fixing bugs in the specializing interpreter, which is left to another PR**. The speedup in the merged version is about 1.1%. https://raw.githubusercontent.com/facebookexperimental/free-threading-benchmarking/refs/heads/main/results/bm-20251112-3.15.0a1%2B-f8a764a-JIT/bm-20251112-vultr-x86_64-Fidget%252dSpinner-tracing_jit-3.15.0a1%2B-f8a764a-vs-base.svg Stats: 50% more uops executed, 30% more traces entered the last time we ran them. It also suggests our trace lengths for a real trace recording JIT are too short, as a lot of trace too long aborts https://github.com/facebookexperimental/free-threading-benchmarking/blob/main/results/bm-20251023-3.15.0a1%2B-eb73378-CLANG%2CJIT/bm-20251023-vultr-x86_64-Fidget%252dSpinner-tracing_jit-3.15.0a1%2B-eb73378-pystats-vs-base.md . This new JIT frontend is already able to record/execute significantly more instructions than the previous JIT frontend. In this PR, we are now able to record through custom dunders, simple object creation, generators, etc. None of these were done by the old JIT frontend. Some custom dunders uops were discovered to be broken as part of this work gh-140277 The optimizer stack space check is disabled, as it's no longer valid to deal with underflow. Pros: * Ignoring the generated tracer code as it's automatically created, this is only additional 1k lines of code. The maintenance burden is handled by the DSL and code generator. * `optimizer.c` is now significantly simpler, as we don't have to do strange things to recover the bytecode from a trace. * The new JIT frontend is able to handle a lot more control-flow than the old one. * Tracing is very low overhead. We use the tail calling interpreter/computed goto interpreter to switch between tracing mode and non-tracing mode. I call this mechanism dual dispatch, as we have two dispatch tables dispatching to each other. Specialization is still enabled while tracing. * Better handling of polymorphism. We leverage the specializing interpreter for this. Cons: * (For now) requires tail calling interpreter or computed gotos. This means no Windows JIT for now :(. Not to fret, tail calling is coming soon to Windows though https://github.com/python/cpython/pull/139962 Design: * After each instruction, the `record_previous_inst` function/label is executed. This does as the name suggests. * The tracing interpreter lowers bytecode to uops directly so that it can obtain "fresh" values at the point of lowering. * The tracing version behaves nearly identical to the normal interpreter, in fact it even has specialization! This allows it to run without much of a slowdown when tracing. The actual cost of tracing is only a function call and writes to memory. * The tracing interpreter uses the specializing interpreter's deopt to naturally form the side exit chains. This allows it to side exit chain effectively, without repeating much code. We force a re-specializing when tracing a deopt. * The tracing interpreter can even handle goto errors/exceptions, but I chose to disable them for now as it's not tested. * Because we do not share interpreter dispatch, there is should be no significant slowdown to the original specializing interpreter on tailcall and computed got with JIT disabled. With JIT enabled, there might be a slowdown in the form of the JIT trying to trace. * Things that could have dynamic instruction pointer effects are guarded on. The guard deopts to a new instruction --- `_DYNAMIC_EXIT`.
2025-11-14 02:08:32 +08:00
if (!backoff_counter_triggers(temperature)) {
exit->temperature = advance_backoff_counter(temperature);
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(0);
gh-139109: A new tracing JIT compiler frontend for CPython (GH-140310) This PR changes the current JIT model from trace projection to trace recording. Benchmarking: better pyperformance (about 1.7% overall) geomean versus current https://raw.githubusercontent.com/facebookexperimental/free-threading-benchmarking/refs/heads/main/results/bm-20251108-3.15.0a1%2B-7e2bc1d-JIT/bm-20251108-vultr-x86_64-Fidget%252dSpinner-tracing_jit-3.15.0a1%2B-7e2bc1d-vs-base.svg, 100% faster Richards on the most improved benchmark versus the current JIT. Slowdown of about 10-15% on the worst benchmark versus the current JIT. **Note: the fastest version isn't the one merged, as it relies on fixing bugs in the specializing interpreter, which is left to another PR**. The speedup in the merged version is about 1.1%. https://raw.githubusercontent.com/facebookexperimental/free-threading-benchmarking/refs/heads/main/results/bm-20251112-3.15.0a1%2B-f8a764a-JIT/bm-20251112-vultr-x86_64-Fidget%252dSpinner-tracing_jit-3.15.0a1%2B-f8a764a-vs-base.svg Stats: 50% more uops executed, 30% more traces entered the last time we ran them. It also suggests our trace lengths for a real trace recording JIT are too short, as a lot of trace too long aborts https://github.com/facebookexperimental/free-threading-benchmarking/blob/main/results/bm-20251023-3.15.0a1%2B-eb73378-CLANG%2CJIT/bm-20251023-vultr-x86_64-Fidget%252dSpinner-tracing_jit-3.15.0a1%2B-eb73378-pystats-vs-base.md . This new JIT frontend is already able to record/execute significantly more instructions than the previous JIT frontend. In this PR, we are now able to record through custom dunders, simple object creation, generators, etc. None of these were done by the old JIT frontend. Some custom dunders uops were discovered to be broken as part of this work gh-140277 The optimizer stack space check is disabled, as it's no longer valid to deal with underflow. Pros: * Ignoring the generated tracer code as it's automatically created, this is only additional 1k lines of code. The maintenance burden is handled by the DSL and code generator. * `optimizer.c` is now significantly simpler, as we don't have to do strange things to recover the bytecode from a trace. * The new JIT frontend is able to handle a lot more control-flow than the old one. * Tracing is very low overhead. We use the tail calling interpreter/computed goto interpreter to switch between tracing mode and non-tracing mode. I call this mechanism dual dispatch, as we have two dispatch tables dispatching to each other. Specialization is still enabled while tracing. * Better handling of polymorphism. We leverage the specializing interpreter for this. Cons: * (For now) requires tail calling interpreter or computed gotos. This means no Windows JIT for now :(. Not to fret, tail calling is coming soon to Windows though https://github.com/python/cpython/pull/139962 Design: * After each instruction, the `record_previous_inst` function/label is executed. This does as the name suggests. * The tracing interpreter lowers bytecode to uops directly so that it can obtain "fresh" values at the point of lowering. * The tracing version behaves nearly identical to the normal interpreter, in fact it even has specialization! This allows it to run without much of a slowdown when tracing. The actual cost of tracing is only a function call and writes to memory. * The tracing interpreter uses the specializing interpreter's deopt to naturally form the side exit chains. This allows it to side exit chain effectively, without repeating much code. We force a re-specializing when tracing a deopt. * The tracing interpreter can even handle goto errors/exceptions, but I chose to disable them for now as it's not tested. * Because we do not share interpreter dispatch, there is should be no significant slowdown to the original specializing interpreter on tailcall and computed got with JIT disabled. With JIT enabled, there might be a slowdown in the form of the JIT trying to trace. * Things that could have dynamic instruction pointer effects are guarded on. The guard deopts to a new instruction --- `_DYNAMIC_EXIT`.
2025-11-14 02:08:32 +08:00
GOTO_TIER_ONE(target);
}
GH-136410: Faster side exits by using a cold exit stub (GH-136411)
2025-08-01 16:26:07 +01:00
_PyExecutorObject *previous_executor = _PyExecutor_FromExit(exit);
assert(tstate->current_executor == (PyObject *)previous_executor);
gh-139109: A new tracing JIT compiler frontend for CPython (GH-140310) This PR changes the current JIT model from trace projection to trace recording. Benchmarking: better pyperformance (about 1.7% overall) geomean versus current https://raw.githubusercontent.com/facebookexperimental/free-threading-benchmarking/refs/heads/main/results/bm-20251108-3.15.0a1%2B-7e2bc1d-JIT/bm-20251108-vultr-x86_64-Fidget%252dSpinner-tracing_jit-3.15.0a1%2B-7e2bc1d-vs-base.svg, 100% faster Richards on the most improved benchmark versus the current JIT. Slowdown of about 10-15% on the worst benchmark versus the current JIT. **Note: the fastest version isn't the one merged, as it relies on fixing bugs in the specializing interpreter, which is left to another PR**. The speedup in the merged version is about 1.1%. https://raw.githubusercontent.com/facebookexperimental/free-threading-benchmarking/refs/heads/main/results/bm-20251112-3.15.0a1%2B-f8a764a-JIT/bm-20251112-vultr-x86_64-Fidget%252dSpinner-tracing_jit-3.15.0a1%2B-f8a764a-vs-base.svg Stats: 50% more uops executed, 30% more traces entered the last time we ran them. It also suggests our trace lengths for a real trace recording JIT are too short, as a lot of trace too long aborts https://github.com/facebookexperimental/free-threading-benchmarking/blob/main/results/bm-20251023-3.15.0a1%2B-eb73378-CLANG%2CJIT/bm-20251023-vultr-x86_64-Fidget%252dSpinner-tracing_jit-3.15.0a1%2B-eb73378-pystats-vs-base.md . This new JIT frontend is already able to record/execute significantly more instructions than the previous JIT frontend. In this PR, we are now able to record through custom dunders, simple object creation, generators, etc. None of these were done by the old JIT frontend. Some custom dunders uops were discovered to be broken as part of this work gh-140277 The optimizer stack space check is disabled, as it's no longer valid to deal with underflow. Pros: * Ignoring the generated tracer code as it's automatically created, this is only additional 1k lines of code. The maintenance burden is handled by the DSL and code generator. * `optimizer.c` is now significantly simpler, as we don't have to do strange things to recover the bytecode from a trace. * The new JIT frontend is able to handle a lot more control-flow than the old one. * Tracing is very low overhead. We use the tail calling interpreter/computed goto interpreter to switch between tracing mode and non-tracing mode. I call this mechanism dual dispatch, as we have two dispatch tables dispatching to each other. Specialization is still enabled while tracing. * Better handling of polymorphism. We leverage the specializing interpreter for this. Cons: * (For now) requires tail calling interpreter or computed gotos. This means no Windows JIT for now :(. Not to fret, tail calling is coming soon to Windows though https://github.com/python/cpython/pull/139962 Design: * After each instruction, the `record_previous_inst` function/label is executed. This does as the name suggests. * The tracing interpreter lowers bytecode to uops directly so that it can obtain "fresh" values at the point of lowering. * The tracing version behaves nearly identical to the normal interpreter, in fact it even has specialization! This allows it to run without much of a slowdown when tracing. The actual cost of tracing is only a function call and writes to memory. * The tracing interpreter uses the specializing interpreter's deopt to naturally form the side exit chains. This allows it to side exit chain effectively, without repeating much code. We force a re-specializing when tracing a deopt. * The tracing interpreter can even handle goto errors/exceptions, but I chose to disable them for now as it's not tested. * Because we do not share interpreter dispatch, there is should be no significant slowdown to the original specializing interpreter on tailcall and computed got with JIT disabled. With JIT enabled, there might be a slowdown in the form of the JIT trying to trace. * Things that could have dynamic instruction pointer effects are guarded on. The guard deopts to a new instruction --- `_DYNAMIC_EXIT`.
2025-11-14 02:08:32 +08:00
int chain_depth = previous_executor->vm_data.chain_depth + !exit->is_control_flow;
int succ = _PyJit_TryInitializeTracing(tstate, frame, target, target, target, STACK_LEVEL(), chain_depth, exit, target->op.arg);
exit->temperature = restart_backoff_counter(exit->temperature);
if (succ) {
GOTO_TIER_ONE_CONTINUE_TRACING(target);
GH-136410: Faster side exits by using a cold exit stub (GH-136411)
2025-08-01 16:26:07 +01:00
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(0);
gh-139109: A new tracing JIT compiler frontend for CPython (GH-140310) This PR changes the current JIT model from trace projection to trace recording. Benchmarking: better pyperformance (about 1.7% overall) geomean versus current https://raw.githubusercontent.com/facebookexperimental/free-threading-benchmarking/refs/heads/main/results/bm-20251108-3.15.0a1%2B-7e2bc1d-JIT/bm-20251108-vultr-x86_64-Fidget%252dSpinner-tracing_jit-3.15.0a1%2B-7e2bc1d-vs-base.svg, 100% faster Richards on the most improved benchmark versus the current JIT. Slowdown of about 10-15% on the worst benchmark versus the current JIT. **Note: the fastest version isn't the one merged, as it relies on fixing bugs in the specializing interpreter, which is left to another PR**. The speedup in the merged version is about 1.1%. https://raw.githubusercontent.com/facebookexperimental/free-threading-benchmarking/refs/heads/main/results/bm-20251112-3.15.0a1%2B-f8a764a-JIT/bm-20251112-vultr-x86_64-Fidget%252dSpinner-tracing_jit-3.15.0a1%2B-f8a764a-vs-base.svg Stats: 50% more uops executed, 30% more traces entered the last time we ran them. It also suggests our trace lengths for a real trace recording JIT are too short, as a lot of trace too long aborts https://github.com/facebookexperimental/free-threading-benchmarking/blob/main/results/bm-20251023-3.15.0a1%2B-eb73378-CLANG%2CJIT/bm-20251023-vultr-x86_64-Fidget%252dSpinner-tracing_jit-3.15.0a1%2B-eb73378-pystats-vs-base.md . This new JIT frontend is already able to record/execute significantly more instructions than the previous JIT frontend. In this PR, we are now able to record through custom dunders, simple object creation, generators, etc. None of these were done by the old JIT frontend. Some custom dunders uops were discovered to be broken as part of this work gh-140277 The optimizer stack space check is disabled, as it's no longer valid to deal with underflow. Pros: * Ignoring the generated tracer code as it's automatically created, this is only additional 1k lines of code. The maintenance burden is handled by the DSL and code generator. * `optimizer.c` is now significantly simpler, as we don't have to do strange things to recover the bytecode from a trace. * The new JIT frontend is able to handle a lot more control-flow than the old one. * Tracing is very low overhead. We use the tail calling interpreter/computed goto interpreter to switch between tracing mode and non-tracing mode. I call this mechanism dual dispatch, as we have two dispatch tables dispatching to each other. Specialization is still enabled while tracing. * Better handling of polymorphism. We leverage the specializing interpreter for this. Cons: * (For now) requires tail calling interpreter or computed gotos. This means no Windows JIT for now :(. Not to fret, tail calling is coming soon to Windows though https://github.com/python/cpython/pull/139962 Design: * After each instruction, the `record_previous_inst` function/label is executed. This does as the name suggests. * The tracing interpreter lowers bytecode to uops directly so that it can obtain "fresh" values at the point of lowering. * The tracing version behaves nearly identical to the normal interpreter, in fact it even has specialization! This allows it to run without much of a slowdown when tracing. The actual cost of tracing is only a function call and writes to memory. * The tracing interpreter uses the specializing interpreter's deopt to naturally form the side exit chains. This allows it to side exit chain effectively, without repeating much code. We force a re-specializing when tracing a deopt. * The tracing interpreter can even handle goto errors/exceptions, but I chose to disable them for now as it's not tested. * Because we do not share interpreter dispatch, there is should be no significant slowdown to the original specializing interpreter on tailcall and computed got with JIT disabled. With JIT enabled, there might be a slowdown in the form of the JIT trying to trace. * Things that could have dynamic instruction pointer effects are guarded on. The guard deopts to a new instruction --- `_DYNAMIC_EXIT`.
2025-11-14 02:08:32 +08:00
GOTO_TIER_ONE(target);
GH-136410: Faster side exits by using a cold exit stub (GH-136411)
2025-08-01 16:26:07 +01:00
}
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
case _COLD_DYNAMIC_EXIT_r00: {
CHECK_CURRENT_CACHED_VALUES(0);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-139109: A new tracing JIT compiler frontend for CPython (GH-140310) This PR changes the current JIT model from trace projection to trace recording. Benchmarking: better pyperformance (about 1.7% overall) geomean versus current https://raw.githubusercontent.com/facebookexperimental/free-threading-benchmarking/refs/heads/main/results/bm-20251108-3.15.0a1%2B-7e2bc1d-JIT/bm-20251108-vultr-x86_64-Fidget%252dSpinner-tracing_jit-3.15.0a1%2B-7e2bc1d-vs-base.svg, 100% faster Richards on the most improved benchmark versus the current JIT. Slowdown of about 10-15% on the worst benchmark versus the current JIT. **Note: the fastest version isn't the one merged, as it relies on fixing bugs in the specializing interpreter, which is left to another PR**. The speedup in the merged version is about 1.1%. https://raw.githubusercontent.com/facebookexperimental/free-threading-benchmarking/refs/heads/main/results/bm-20251112-3.15.0a1%2B-f8a764a-JIT/bm-20251112-vultr-x86_64-Fidget%252dSpinner-tracing_jit-3.15.0a1%2B-f8a764a-vs-base.svg Stats: 50% more uops executed, 30% more traces entered the last time we ran them. It also suggests our trace lengths for a real trace recording JIT are too short, as a lot of trace too long aborts https://github.com/facebookexperimental/free-threading-benchmarking/blob/main/results/bm-20251023-3.15.0a1%2B-eb73378-CLANG%2CJIT/bm-20251023-vultr-x86_64-Fidget%252dSpinner-tracing_jit-3.15.0a1%2B-eb73378-pystats-vs-base.md . This new JIT frontend is already able to record/execute significantly more instructions than the previous JIT frontend. In this PR, we are now able to record through custom dunders, simple object creation, generators, etc. None of these were done by the old JIT frontend. Some custom dunders uops were discovered to be broken as part of this work gh-140277 The optimizer stack space check is disabled, as it's no longer valid to deal with underflow. Pros: * Ignoring the generated tracer code as it's automatically created, this is only additional 1k lines of code. The maintenance burden is handled by the DSL and code generator. * `optimizer.c` is now significantly simpler, as we don't have to do strange things to recover the bytecode from a trace. * The new JIT frontend is able to handle a lot more control-flow than the old one. * Tracing is very low overhead. We use the tail calling interpreter/computed goto interpreter to switch between tracing mode and non-tracing mode. I call this mechanism dual dispatch, as we have two dispatch tables dispatching to each other. Specialization is still enabled while tracing. * Better handling of polymorphism. We leverage the specializing interpreter for this. Cons: * (For now) requires tail calling interpreter or computed gotos. This means no Windows JIT for now :(. Not to fret, tail calling is coming soon to Windows though https://github.com/python/cpython/pull/139962 Design: * After each instruction, the `record_previous_inst` function/label is executed. This does as the name suggests. * The tracing interpreter lowers bytecode to uops directly so that it can obtain "fresh" values at the point of lowering. * The tracing version behaves nearly identical to the normal interpreter, in fact it even has specialization! This allows it to run without much of a slowdown when tracing. The actual cost of tracing is only a function call and writes to memory. * The tracing interpreter uses the specializing interpreter's deopt to naturally form the side exit chains. This allows it to side exit chain effectively, without repeating much code. We force a re-specializing when tracing a deopt. * The tracing interpreter can even handle goto errors/exceptions, but I chose to disable them for now as it's not tested. * Because we do not share interpreter dispatch, there is should be no significant slowdown to the original specializing interpreter on tailcall and computed got with JIT disabled. With JIT enabled, there might be a slowdown in the form of the JIT trying to trace. * Things that could have dynamic instruction pointer effects are guarded on. The guard deopts to a new instruction --- `_DYNAMIC_EXIT`.
2025-11-14 02:08:32 +08:00
_Py_CODEUNIT *target = frame->instr_ptr;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(0);
gh-139109: A new tracing JIT compiler frontend for CPython (GH-140310) This PR changes the current JIT model from trace projection to trace recording. Benchmarking: better pyperformance (about 1.7% overall) geomean versus current https://raw.githubusercontent.com/facebookexperimental/free-threading-benchmarking/refs/heads/main/results/bm-20251108-3.15.0a1%2B-7e2bc1d-JIT/bm-20251108-vultr-x86_64-Fidget%252dSpinner-tracing_jit-3.15.0a1%2B-7e2bc1d-vs-base.svg, 100% faster Richards on the most improved benchmark versus the current JIT. Slowdown of about 10-15% on the worst benchmark versus the current JIT. **Note: the fastest version isn't the one merged, as it relies on fixing bugs in the specializing interpreter, which is left to another PR**. The speedup in the merged version is about 1.1%. https://raw.githubusercontent.com/facebookexperimental/free-threading-benchmarking/refs/heads/main/results/bm-20251112-3.15.0a1%2B-f8a764a-JIT/bm-20251112-vultr-x86_64-Fidget%252dSpinner-tracing_jit-3.15.0a1%2B-f8a764a-vs-base.svg Stats: 50% more uops executed, 30% more traces entered the last time we ran them. It also suggests our trace lengths for a real trace recording JIT are too short, as a lot of trace too long aborts https://github.com/facebookexperimental/free-threading-benchmarking/blob/main/results/bm-20251023-3.15.0a1%2B-eb73378-CLANG%2CJIT/bm-20251023-vultr-x86_64-Fidget%252dSpinner-tracing_jit-3.15.0a1%2B-eb73378-pystats-vs-base.md . This new JIT frontend is already able to record/execute significantly more instructions than the previous JIT frontend. In this PR, we are now able to record through custom dunders, simple object creation, generators, etc. None of these were done by the old JIT frontend. Some custom dunders uops were discovered to be broken as part of this work gh-140277 The optimizer stack space check is disabled, as it's no longer valid to deal with underflow. Pros: * Ignoring the generated tracer code as it's automatically created, this is only additional 1k lines of code. The maintenance burden is handled by the DSL and code generator. * `optimizer.c` is now significantly simpler, as we don't have to do strange things to recover the bytecode from a trace. * The new JIT frontend is able to handle a lot more control-flow than the old one. * Tracing is very low overhead. We use the tail calling interpreter/computed goto interpreter to switch between tracing mode and non-tracing mode. I call this mechanism dual dispatch, as we have two dispatch tables dispatching to each other. Specialization is still enabled while tracing. * Better handling of polymorphism. We leverage the specializing interpreter for this. Cons: * (For now) requires tail calling interpreter or computed gotos. This means no Windows JIT for now :(. Not to fret, tail calling is coming soon to Windows though https://github.com/python/cpython/pull/139962 Design: * After each instruction, the `record_previous_inst` function/label is executed. This does as the name suggests. * The tracing interpreter lowers bytecode to uops directly so that it can obtain "fresh" values at the point of lowering. * The tracing version behaves nearly identical to the normal interpreter, in fact it even has specialization! This allows it to run without much of a slowdown when tracing. The actual cost of tracing is only a function call and writes to memory. * The tracing interpreter uses the specializing interpreter's deopt to naturally form the side exit chains. This allows it to side exit chain effectively, without repeating much code. We force a re-specializing when tracing a deopt. * The tracing interpreter can even handle goto errors/exceptions, but I chose to disable them for now as it's not tested. * Because we do not share interpreter dispatch, there is should be no significant slowdown to the original specializing interpreter on tailcall and computed got with JIT disabled. With JIT enabled, there might be a slowdown in the form of the JIT trying to trace. * Things that could have dynamic instruction pointer effects are guarded on. The guard deopts to a new instruction --- `_DYNAMIC_EXIT`.
2025-11-14 02:08:32 +08:00
GOTO_TIER_ONE(target);
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
}
case _GUARD_IP__PUSH_FRAME_r00: {
CHECK_CURRENT_CACHED_VALUES(0);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
#define OFFSET_OF__PUSH_FRAME ((0))
GH-140683: JIT: Improve machine code for loading smaller constants on AArch64. (GH-142511) * Use movz and movk instructions for loading 16 and 32 bit operands and oparg. * Loading of 64 bit operands is unchanged.
2025-12-11 12:33:39 +00:00
PyObject *ip = (PyObject *)CURRENT_OPERAND0_64();
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_Py_CODEUNIT *target = frame->instr_ptr + OFFSET_OF__PUSH_FRAME;
if (target != (_Py_CODEUNIT *)ip) {
frame->instr_ptr += OFFSET_OF__PUSH_FRAME;
if (true) {
UOP_STAT_INC(uopcode, miss);
SET_CURRENT_CACHED_VALUES(0);
JUMP_TO_JUMP_TARGET();
}
}
SET_CURRENT_CACHED_VALUES(0);
#undef OFFSET_OF__PUSH_FRAME
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _GUARD_IP__PUSH_FRAME_r11: {
CHECK_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef _stack_item_0 = _tos_cache0;
#define OFFSET_OF__PUSH_FRAME ((0))
GH-140683: JIT: Improve machine code for loading smaller constants on AArch64. (GH-142511) * Use movz and movk instructions for loading 16 and 32 bit operands and oparg. * Loading of 64 bit operands is unchanged.
2025-12-11 12:33:39 +00:00
PyObject *ip = (PyObject *)CURRENT_OPERAND0_64();
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_Py_CODEUNIT *target = frame->instr_ptr + OFFSET_OF__PUSH_FRAME;
if (target != (_Py_CODEUNIT *)ip) {
frame->instr_ptr += OFFSET_OF__PUSH_FRAME;
if (true) {
UOP_STAT_INC(uopcode, miss);
GH-142718: JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. (GH-142762) JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. * Make sure that stack is in correct state at side exits with TOS cached values * Simplify choice of cached items for side exits
2025-12-15 19:01:29 +00:00
_tos_cache0 = _stack_item_0;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(1);
JUMP_TO_JUMP_TARGET();
}
}
_tos_cache0 = _stack_item_0;
SET_CURRENT_CACHED_VALUES(1);
#undef OFFSET_OF__PUSH_FRAME
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _GUARD_IP__PUSH_FRAME_r22: {
CHECK_CURRENT_CACHED_VALUES(2);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef _stack_item_0 = _tos_cache0;
_PyStackRef _stack_item_1 = _tos_cache1;
#define OFFSET_OF__PUSH_FRAME ((0))
GH-140683: JIT: Improve machine code for loading smaller constants on AArch64. (GH-142511) * Use movz and movk instructions for loading 16 and 32 bit operands and oparg. * Loading of 64 bit operands is unchanged.
2025-12-11 12:33:39 +00:00
PyObject *ip = (PyObject *)CURRENT_OPERAND0_64();
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_Py_CODEUNIT *target = frame->instr_ptr + OFFSET_OF__PUSH_FRAME;
if (target != (_Py_CODEUNIT *)ip) {
frame->instr_ptr += OFFSET_OF__PUSH_FRAME;
if (true) {
UOP_STAT_INC(uopcode, miss);
GH-142718: JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. (GH-142762) JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. * Make sure that stack is in correct state at side exits with TOS cached values * Simplify choice of cached items for side exits
2025-12-15 19:01:29 +00:00
_tos_cache1 = _stack_item_1;
_tos_cache0 = _stack_item_0;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(2);
JUMP_TO_JUMP_TARGET();
}
}
_tos_cache1 = _stack_item_1;
_tos_cache0 = _stack_item_0;
SET_CURRENT_CACHED_VALUES(2);
#undef OFFSET_OF__PUSH_FRAME
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-139109: A new tracing JIT compiler frontend for CPython (GH-140310) This PR changes the current JIT model from trace projection to trace recording. Benchmarking: better pyperformance (about 1.7% overall) geomean versus current https://raw.githubusercontent.com/facebookexperimental/free-threading-benchmarking/refs/heads/main/results/bm-20251108-3.15.0a1%2B-7e2bc1d-JIT/bm-20251108-vultr-x86_64-Fidget%252dSpinner-tracing_jit-3.15.0a1%2B-7e2bc1d-vs-base.svg, 100% faster Richards on the most improved benchmark versus the current JIT. Slowdown of about 10-15% on the worst benchmark versus the current JIT. **Note: the fastest version isn't the one merged, as it relies on fixing bugs in the specializing interpreter, which is left to another PR**. The speedup in the merged version is about 1.1%. https://raw.githubusercontent.com/facebookexperimental/free-threading-benchmarking/refs/heads/main/results/bm-20251112-3.15.0a1%2B-f8a764a-JIT/bm-20251112-vultr-x86_64-Fidget%252dSpinner-tracing_jit-3.15.0a1%2B-f8a764a-vs-base.svg Stats: 50% more uops executed, 30% more traces entered the last time we ran them. It also suggests our trace lengths for a real trace recording JIT are too short, as a lot of trace too long aborts https://github.com/facebookexperimental/free-threading-benchmarking/blob/main/results/bm-20251023-3.15.0a1%2B-eb73378-CLANG%2CJIT/bm-20251023-vultr-x86_64-Fidget%252dSpinner-tracing_jit-3.15.0a1%2B-eb73378-pystats-vs-base.md . This new JIT frontend is already able to record/execute significantly more instructions than the previous JIT frontend. In this PR, we are now able to record through custom dunders, simple object creation, generators, etc. None of these were done by the old JIT frontend. Some custom dunders uops were discovered to be broken as part of this work gh-140277 The optimizer stack space check is disabled, as it's no longer valid to deal with underflow. Pros: * Ignoring the generated tracer code as it's automatically created, this is only additional 1k lines of code. The maintenance burden is handled by the DSL and code generator. * `optimizer.c` is now significantly simpler, as we don't have to do strange things to recover the bytecode from a trace. * The new JIT frontend is able to handle a lot more control-flow than the old one. * Tracing is very low overhead. We use the tail calling interpreter/computed goto interpreter to switch between tracing mode and non-tracing mode. I call this mechanism dual dispatch, as we have two dispatch tables dispatching to each other. Specialization is still enabled while tracing. * Better handling of polymorphism. We leverage the specializing interpreter for this. Cons: * (For now) requires tail calling interpreter or computed gotos. This means no Windows JIT for now :(. Not to fret, tail calling is coming soon to Windows though https://github.com/python/cpython/pull/139962 Design: * After each instruction, the `record_previous_inst` function/label is executed. This does as the name suggests. * The tracing interpreter lowers bytecode to uops directly so that it can obtain "fresh" values at the point of lowering. * The tracing version behaves nearly identical to the normal interpreter, in fact it even has specialization! This allows it to run without much of a slowdown when tracing. The actual cost of tracing is only a function call and writes to memory. * The tracing interpreter uses the specializing interpreter's deopt to naturally form the side exit chains. This allows it to side exit chain effectively, without repeating much code. We force a re-specializing when tracing a deopt. * The tracing interpreter can even handle goto errors/exceptions, but I chose to disable them for now as it's not tested. * Because we do not share interpreter dispatch, there is should be no significant slowdown to the original specializing interpreter on tailcall and computed got with JIT disabled. With JIT enabled, there might be a slowdown in the form of the JIT trying to trace. * Things that could have dynamic instruction pointer effects are guarded on. The guard deopts to a new instruction --- `_DYNAMIC_EXIT`.
2025-11-14 02:08:32 +08:00
break;
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
case _GUARD_IP__PUSH_FRAME_r33: {
CHECK_CURRENT_CACHED_VALUES(3);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef _stack_item_0 = _tos_cache0;
_PyStackRef _stack_item_1 = _tos_cache1;
_PyStackRef _stack_item_2 = _tos_cache2;
gh-139109: A new tracing JIT compiler frontend for CPython (GH-140310) This PR changes the current JIT model from trace projection to trace recording. Benchmarking: better pyperformance (about 1.7% overall) geomean versus current https://raw.githubusercontent.com/facebookexperimental/free-threading-benchmarking/refs/heads/main/results/bm-20251108-3.15.0a1%2B-7e2bc1d-JIT/bm-20251108-vultr-x86_64-Fidget%252dSpinner-tracing_jit-3.15.0a1%2B-7e2bc1d-vs-base.svg, 100% faster Richards on the most improved benchmark versus the current JIT. Slowdown of about 10-15% on the worst benchmark versus the current JIT. **Note: the fastest version isn't the one merged, as it relies on fixing bugs in the specializing interpreter, which is left to another PR**. The speedup in the merged version is about 1.1%. https://raw.githubusercontent.com/facebookexperimental/free-threading-benchmarking/refs/heads/main/results/bm-20251112-3.15.0a1%2B-f8a764a-JIT/bm-20251112-vultr-x86_64-Fidget%252dSpinner-tracing_jit-3.15.0a1%2B-f8a764a-vs-base.svg Stats: 50% more uops executed, 30% more traces entered the last time we ran them. It also suggests our trace lengths for a real trace recording JIT are too short, as a lot of trace too long aborts https://github.com/facebookexperimental/free-threading-benchmarking/blob/main/results/bm-20251023-3.15.0a1%2B-eb73378-CLANG%2CJIT/bm-20251023-vultr-x86_64-Fidget%252dSpinner-tracing_jit-3.15.0a1%2B-eb73378-pystats-vs-base.md . This new JIT frontend is already able to record/execute significantly more instructions than the previous JIT frontend. In this PR, we are now able to record through custom dunders, simple object creation, generators, etc. None of these were done by the old JIT frontend. Some custom dunders uops were discovered to be broken as part of this work gh-140277 The optimizer stack space check is disabled, as it's no longer valid to deal with underflow. Pros: * Ignoring the generated tracer code as it's automatically created, this is only additional 1k lines of code. The maintenance burden is handled by the DSL and code generator. * `optimizer.c` is now significantly simpler, as we don't have to do strange things to recover the bytecode from a trace. * The new JIT frontend is able to handle a lot more control-flow than the old one. * Tracing is very low overhead. We use the tail calling interpreter/computed goto interpreter to switch between tracing mode and non-tracing mode. I call this mechanism dual dispatch, as we have two dispatch tables dispatching to each other. Specialization is still enabled while tracing. * Better handling of polymorphism. We leverage the specializing interpreter for this. Cons: * (For now) requires tail calling interpreter or computed gotos. This means no Windows JIT for now :(. Not to fret, tail calling is coming soon to Windows though https://github.com/python/cpython/pull/139962 Design: * After each instruction, the `record_previous_inst` function/label is executed. This does as the name suggests. * The tracing interpreter lowers bytecode to uops directly so that it can obtain "fresh" values at the point of lowering. * The tracing version behaves nearly identical to the normal interpreter, in fact it even has specialization! This allows it to run without much of a slowdown when tracing. The actual cost of tracing is only a function call and writes to memory. * The tracing interpreter uses the specializing interpreter's deopt to naturally form the side exit chains. This allows it to side exit chain effectively, without repeating much code. We force a re-specializing when tracing a deopt. * The tracing interpreter can even handle goto errors/exceptions, but I chose to disable them for now as it's not tested. * Because we do not share interpreter dispatch, there is should be no significant slowdown to the original specializing interpreter on tailcall and computed got with JIT disabled. With JIT enabled, there might be a slowdown in the form of the JIT trying to trace. * Things that could have dynamic instruction pointer effects are guarded on. The guard deopts to a new instruction --- `_DYNAMIC_EXIT`.
2025-11-14 02:08:32 +08:00
#define OFFSET_OF__PUSH_FRAME ((0))
GH-140683: JIT: Improve machine code for loading smaller constants on AArch64. (GH-142511) * Use movz and movk instructions for loading 16 and 32 bit operands and oparg. * Loading of 64 bit operands is unchanged.
2025-12-11 12:33:39 +00:00
PyObject *ip = (PyObject *)CURRENT_OPERAND0_64();
gh-139109: A new tracing JIT compiler frontend for CPython (GH-140310) This PR changes the current JIT model from trace projection to trace recording. Benchmarking: better pyperformance (about 1.7% overall) geomean versus current https://raw.githubusercontent.com/facebookexperimental/free-threading-benchmarking/refs/heads/main/results/bm-20251108-3.15.0a1%2B-7e2bc1d-JIT/bm-20251108-vultr-x86_64-Fidget%252dSpinner-tracing_jit-3.15.0a1%2B-7e2bc1d-vs-base.svg, 100% faster Richards on the most improved benchmark versus the current JIT. Slowdown of about 10-15% on the worst benchmark versus the current JIT. **Note: the fastest version isn't the one merged, as it relies on fixing bugs in the specializing interpreter, which is left to another PR**. The speedup in the merged version is about 1.1%. https://raw.githubusercontent.com/facebookexperimental/free-threading-benchmarking/refs/heads/main/results/bm-20251112-3.15.0a1%2B-f8a764a-JIT/bm-20251112-vultr-x86_64-Fidget%252dSpinner-tracing_jit-3.15.0a1%2B-f8a764a-vs-base.svg Stats: 50% more uops executed, 30% more traces entered the last time we ran them. It also suggests our trace lengths for a real trace recording JIT are too short, as a lot of trace too long aborts https://github.com/facebookexperimental/free-threading-benchmarking/blob/main/results/bm-20251023-3.15.0a1%2B-eb73378-CLANG%2CJIT/bm-20251023-vultr-x86_64-Fidget%252dSpinner-tracing_jit-3.15.0a1%2B-eb73378-pystats-vs-base.md . This new JIT frontend is already able to record/execute significantly more instructions than the previous JIT frontend. In this PR, we are now able to record through custom dunders, simple object creation, generators, etc. None of these were done by the old JIT frontend. Some custom dunders uops were discovered to be broken as part of this work gh-140277 The optimizer stack space check is disabled, as it's no longer valid to deal with underflow. Pros: * Ignoring the generated tracer code as it's automatically created, this is only additional 1k lines of code. The maintenance burden is handled by the DSL and code generator. * `optimizer.c` is now significantly simpler, as we don't have to do strange things to recover the bytecode from a trace. * The new JIT frontend is able to handle a lot more control-flow than the old one. * Tracing is very low overhead. We use the tail calling interpreter/computed goto interpreter to switch between tracing mode and non-tracing mode. I call this mechanism dual dispatch, as we have two dispatch tables dispatching to each other. Specialization is still enabled while tracing. * Better handling of polymorphism. We leverage the specializing interpreter for this. Cons: * (For now) requires tail calling interpreter or computed gotos. This means no Windows JIT for now :(. Not to fret, tail calling is coming soon to Windows though https://github.com/python/cpython/pull/139962 Design: * After each instruction, the `record_previous_inst` function/label is executed. This does as the name suggests. * The tracing interpreter lowers bytecode to uops directly so that it can obtain "fresh" values at the point of lowering. * The tracing version behaves nearly identical to the normal interpreter, in fact it even has specialization! This allows it to run without much of a slowdown when tracing. The actual cost of tracing is only a function call and writes to memory. * The tracing interpreter uses the specializing interpreter's deopt to naturally form the side exit chains. This allows it to side exit chain effectively, without repeating much code. We force a re-specializing when tracing a deopt. * The tracing interpreter can even handle goto errors/exceptions, but I chose to disable them for now as it's not tested. * Because we do not share interpreter dispatch, there is should be no significant slowdown to the original specializing interpreter on tailcall and computed got with JIT disabled. With JIT enabled, there might be a slowdown in the form of the JIT trying to trace. * Things that could have dynamic instruction pointer effects are guarded on. The guard deopts to a new instruction --- `_DYNAMIC_EXIT`.
2025-11-14 02:08:32 +08:00
_Py_CODEUNIT *target = frame->instr_ptr + OFFSET_OF__PUSH_FRAME;
if (target != (_Py_CODEUNIT *)ip) {
frame->instr_ptr += OFFSET_OF__PUSH_FRAME;
if (true) {
UOP_STAT_INC(uopcode, miss);
GH-142718: JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. (GH-142762) JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. * Make sure that stack is in correct state at side exits with TOS cached values * Simplify choice of cached items for side exits
2025-12-15 19:01:29 +00:00
_tos_cache2 = _stack_item_2;
_tos_cache1 = _stack_item_1;
_tos_cache0 = _stack_item_0;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(3);
gh-139109: A new tracing JIT compiler frontend for CPython (GH-140310) This PR changes the current JIT model from trace projection to trace recording. Benchmarking: better pyperformance (about 1.7% overall) geomean versus current https://raw.githubusercontent.com/facebookexperimental/free-threading-benchmarking/refs/heads/main/results/bm-20251108-3.15.0a1%2B-7e2bc1d-JIT/bm-20251108-vultr-x86_64-Fidget%252dSpinner-tracing_jit-3.15.0a1%2B-7e2bc1d-vs-base.svg, 100% faster Richards on the most improved benchmark versus the current JIT. Slowdown of about 10-15% on the worst benchmark versus the current JIT. **Note: the fastest version isn't the one merged, as it relies on fixing bugs in the specializing interpreter, which is left to another PR**. The speedup in the merged version is about 1.1%. https://raw.githubusercontent.com/facebookexperimental/free-threading-benchmarking/refs/heads/main/results/bm-20251112-3.15.0a1%2B-f8a764a-JIT/bm-20251112-vultr-x86_64-Fidget%252dSpinner-tracing_jit-3.15.0a1%2B-f8a764a-vs-base.svg Stats: 50% more uops executed, 30% more traces entered the last time we ran them. It also suggests our trace lengths for a real trace recording JIT are too short, as a lot of trace too long aborts https://github.com/facebookexperimental/free-threading-benchmarking/blob/main/results/bm-20251023-3.15.0a1%2B-eb73378-CLANG%2CJIT/bm-20251023-vultr-x86_64-Fidget%252dSpinner-tracing_jit-3.15.0a1%2B-eb73378-pystats-vs-base.md . This new JIT frontend is already able to record/execute significantly more instructions than the previous JIT frontend. In this PR, we are now able to record through custom dunders, simple object creation, generators, etc. None of these were done by the old JIT frontend. Some custom dunders uops were discovered to be broken as part of this work gh-140277 The optimizer stack space check is disabled, as it's no longer valid to deal with underflow. Pros: * Ignoring the generated tracer code as it's automatically created, this is only additional 1k lines of code. The maintenance burden is handled by the DSL and code generator. * `optimizer.c` is now significantly simpler, as we don't have to do strange things to recover the bytecode from a trace. * The new JIT frontend is able to handle a lot more control-flow than the old one. * Tracing is very low overhead. We use the tail calling interpreter/computed goto interpreter to switch between tracing mode and non-tracing mode. I call this mechanism dual dispatch, as we have two dispatch tables dispatching to each other. Specialization is still enabled while tracing. * Better handling of polymorphism. We leverage the specializing interpreter for this. Cons: * (For now) requires tail calling interpreter or computed gotos. This means no Windows JIT for now :(. Not to fret, tail calling is coming soon to Windows though https://github.com/python/cpython/pull/139962 Design: * After each instruction, the `record_previous_inst` function/label is executed. This does as the name suggests. * The tracing interpreter lowers bytecode to uops directly so that it can obtain "fresh" values at the point of lowering. * The tracing version behaves nearly identical to the normal interpreter, in fact it even has specialization! This allows it to run without much of a slowdown when tracing. The actual cost of tracing is only a function call and writes to memory. * The tracing interpreter uses the specializing interpreter's deopt to naturally form the side exit chains. This allows it to side exit chain effectively, without repeating much code. We force a re-specializing when tracing a deopt. * The tracing interpreter can even handle goto errors/exceptions, but I chose to disable them for now as it's not tested. * Because we do not share interpreter dispatch, there is should be no significant slowdown to the original specializing interpreter on tailcall and computed got with JIT disabled. With JIT enabled, there might be a slowdown in the form of the JIT trying to trace. * Things that could have dynamic instruction pointer effects are guarded on. The guard deopts to a new instruction --- `_DYNAMIC_EXIT`.
2025-11-14 02:08:32 +08:00
JUMP_TO_JUMP_TARGET();
}
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_tos_cache2 = _stack_item_2;
_tos_cache1 = _stack_item_1;
_tos_cache0 = _stack_item_0;
SET_CURRENT_CACHED_VALUES(3);
gh-139109: A new tracing JIT compiler frontend for CPython (GH-140310) This PR changes the current JIT model from trace projection to trace recording. Benchmarking: better pyperformance (about 1.7% overall) geomean versus current https://raw.githubusercontent.com/facebookexperimental/free-threading-benchmarking/refs/heads/main/results/bm-20251108-3.15.0a1%2B-7e2bc1d-JIT/bm-20251108-vultr-x86_64-Fidget%252dSpinner-tracing_jit-3.15.0a1%2B-7e2bc1d-vs-base.svg, 100% faster Richards on the most improved benchmark versus the current JIT. Slowdown of about 10-15% on the worst benchmark versus the current JIT. **Note: the fastest version isn't the one merged, as it relies on fixing bugs in the specializing interpreter, which is left to another PR**. The speedup in the merged version is about 1.1%. https://raw.githubusercontent.com/facebookexperimental/free-threading-benchmarking/refs/heads/main/results/bm-20251112-3.15.0a1%2B-f8a764a-JIT/bm-20251112-vultr-x86_64-Fidget%252dSpinner-tracing_jit-3.15.0a1%2B-f8a764a-vs-base.svg Stats: 50% more uops executed, 30% more traces entered the last time we ran them. It also suggests our trace lengths for a real trace recording JIT are too short, as a lot of trace too long aborts https://github.com/facebookexperimental/free-threading-benchmarking/blob/main/results/bm-20251023-3.15.0a1%2B-eb73378-CLANG%2CJIT/bm-20251023-vultr-x86_64-Fidget%252dSpinner-tracing_jit-3.15.0a1%2B-eb73378-pystats-vs-base.md . This new JIT frontend is already able to record/execute significantly more instructions than the previous JIT frontend. In this PR, we are now able to record through custom dunders, simple object creation, generators, etc. None of these were done by the old JIT frontend. Some custom dunders uops were discovered to be broken as part of this work gh-140277 The optimizer stack space check is disabled, as it's no longer valid to deal with underflow. Pros: * Ignoring the generated tracer code as it's automatically created, this is only additional 1k lines of code. The maintenance burden is handled by the DSL and code generator. * `optimizer.c` is now significantly simpler, as we don't have to do strange things to recover the bytecode from a trace. * The new JIT frontend is able to handle a lot more control-flow than the old one. * Tracing is very low overhead. We use the tail calling interpreter/computed goto interpreter to switch between tracing mode and non-tracing mode. I call this mechanism dual dispatch, as we have two dispatch tables dispatching to each other. Specialization is still enabled while tracing. * Better handling of polymorphism. We leverage the specializing interpreter for this. Cons: * (For now) requires tail calling interpreter or computed gotos. This means no Windows JIT for now :(. Not to fret, tail calling is coming soon to Windows though https://github.com/python/cpython/pull/139962 Design: * After each instruction, the `record_previous_inst` function/label is executed. This does as the name suggests. * The tracing interpreter lowers bytecode to uops directly so that it can obtain "fresh" values at the point of lowering. * The tracing version behaves nearly identical to the normal interpreter, in fact it even has specialization! This allows it to run without much of a slowdown when tracing. The actual cost of tracing is only a function call and writes to memory. * The tracing interpreter uses the specializing interpreter's deopt to naturally form the side exit chains. This allows it to side exit chain effectively, without repeating much code. We force a re-specializing when tracing a deopt. * The tracing interpreter can even handle goto errors/exceptions, but I chose to disable them for now as it's not tested. * Because we do not share interpreter dispatch, there is should be no significant slowdown to the original specializing interpreter on tailcall and computed got with JIT disabled. With JIT enabled, there might be a slowdown in the form of the JIT trying to trace. * Things that could have dynamic instruction pointer effects are guarded on. The guard deopts to a new instruction --- `_DYNAMIC_EXIT`.
2025-11-14 02:08:32 +08:00
#undef OFFSET_OF__PUSH_FRAME
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _GUARD_IP_YIELD_VALUE_r00: {
CHECK_CURRENT_CACHED_VALUES(0);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
#define OFFSET_OF_YIELD_VALUE ((1+INLINE_CACHE_ENTRIES_SEND))
GH-140683: JIT: Improve machine code for loading smaller constants on AArch64. (GH-142511) * Use movz and movk instructions for loading 16 and 32 bit operands and oparg. * Loading of 64 bit operands is unchanged.
2025-12-11 12:33:39 +00:00
PyObject *ip = (PyObject *)CURRENT_OPERAND0_64();
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_Py_CODEUNIT *target = frame->instr_ptr + OFFSET_OF_YIELD_VALUE;
if (target != (_Py_CODEUNIT *)ip) {
frame->instr_ptr += OFFSET_OF_YIELD_VALUE;
if (true) {
UOP_STAT_INC(uopcode, miss);
SET_CURRENT_CACHED_VALUES(0);
JUMP_TO_JUMP_TARGET();
}
}
SET_CURRENT_CACHED_VALUES(0);
#undef OFFSET_OF_YIELD_VALUE
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _GUARD_IP_YIELD_VALUE_r11: {
CHECK_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef _stack_item_0 = _tos_cache0;
#define OFFSET_OF_YIELD_VALUE ((1+INLINE_CACHE_ENTRIES_SEND))
GH-140683: JIT: Improve machine code for loading smaller constants on AArch64. (GH-142511) * Use movz and movk instructions for loading 16 and 32 bit operands and oparg. * Loading of 64 bit operands is unchanged.
2025-12-11 12:33:39 +00:00
PyObject *ip = (PyObject *)CURRENT_OPERAND0_64();
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_Py_CODEUNIT *target = frame->instr_ptr + OFFSET_OF_YIELD_VALUE;
if (target != (_Py_CODEUNIT *)ip) {
frame->instr_ptr += OFFSET_OF_YIELD_VALUE;
if (true) {
UOP_STAT_INC(uopcode, miss);
GH-142718: JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. (GH-142762) JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. * Make sure that stack is in correct state at side exits with TOS cached values * Simplify choice of cached items for side exits
2025-12-15 19:01:29 +00:00
_tos_cache0 = _stack_item_0;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(1);
JUMP_TO_JUMP_TARGET();
}
}
_tos_cache0 = _stack_item_0;
SET_CURRENT_CACHED_VALUES(1);
#undef OFFSET_OF_YIELD_VALUE
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _GUARD_IP_YIELD_VALUE_r22: {
CHECK_CURRENT_CACHED_VALUES(2);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef _stack_item_0 = _tos_cache0;
_PyStackRef _stack_item_1 = _tos_cache1;
#define OFFSET_OF_YIELD_VALUE ((1+INLINE_CACHE_ENTRIES_SEND))
GH-140683: JIT: Improve machine code for loading smaller constants on AArch64. (GH-142511) * Use movz and movk instructions for loading 16 and 32 bit operands and oparg. * Loading of 64 bit operands is unchanged.
2025-12-11 12:33:39 +00:00
PyObject *ip = (PyObject *)CURRENT_OPERAND0_64();
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_Py_CODEUNIT *target = frame->instr_ptr + OFFSET_OF_YIELD_VALUE;
if (target != (_Py_CODEUNIT *)ip) {
frame->instr_ptr += OFFSET_OF_YIELD_VALUE;
if (true) {
UOP_STAT_INC(uopcode, miss);
GH-142718: JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. (GH-142762) JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. * Make sure that stack is in correct state at side exits with TOS cached values * Simplify choice of cached items for side exits
2025-12-15 19:01:29 +00:00
_tos_cache1 = _stack_item_1;
_tos_cache0 = _stack_item_0;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(2);
JUMP_TO_JUMP_TARGET();
}
}
_tos_cache1 = _stack_item_1;
_tos_cache0 = _stack_item_0;
SET_CURRENT_CACHED_VALUES(2);
#undef OFFSET_OF_YIELD_VALUE
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-139109: A new tracing JIT compiler frontend for CPython (GH-140310) This PR changes the current JIT model from trace projection to trace recording. Benchmarking: better pyperformance (about 1.7% overall) geomean versus current https://raw.githubusercontent.com/facebookexperimental/free-threading-benchmarking/refs/heads/main/results/bm-20251108-3.15.0a1%2B-7e2bc1d-JIT/bm-20251108-vultr-x86_64-Fidget%252dSpinner-tracing_jit-3.15.0a1%2B-7e2bc1d-vs-base.svg, 100% faster Richards on the most improved benchmark versus the current JIT. Slowdown of about 10-15% on the worst benchmark versus the current JIT. **Note: the fastest version isn't the one merged, as it relies on fixing bugs in the specializing interpreter, which is left to another PR**. The speedup in the merged version is about 1.1%. https://raw.githubusercontent.com/facebookexperimental/free-threading-benchmarking/refs/heads/main/results/bm-20251112-3.15.0a1%2B-f8a764a-JIT/bm-20251112-vultr-x86_64-Fidget%252dSpinner-tracing_jit-3.15.0a1%2B-f8a764a-vs-base.svg Stats: 50% more uops executed, 30% more traces entered the last time we ran them. It also suggests our trace lengths for a real trace recording JIT are too short, as a lot of trace too long aborts https://github.com/facebookexperimental/free-threading-benchmarking/blob/main/results/bm-20251023-3.15.0a1%2B-eb73378-CLANG%2CJIT/bm-20251023-vultr-x86_64-Fidget%252dSpinner-tracing_jit-3.15.0a1%2B-eb73378-pystats-vs-base.md . This new JIT frontend is already able to record/execute significantly more instructions than the previous JIT frontend. In this PR, we are now able to record through custom dunders, simple object creation, generators, etc. None of these were done by the old JIT frontend. Some custom dunders uops were discovered to be broken as part of this work gh-140277 The optimizer stack space check is disabled, as it's no longer valid to deal with underflow. Pros: * Ignoring the generated tracer code as it's automatically created, this is only additional 1k lines of code. The maintenance burden is handled by the DSL and code generator. * `optimizer.c` is now significantly simpler, as we don't have to do strange things to recover the bytecode from a trace. * The new JIT frontend is able to handle a lot more control-flow than the old one. * Tracing is very low overhead. We use the tail calling interpreter/computed goto interpreter to switch between tracing mode and non-tracing mode. I call this mechanism dual dispatch, as we have two dispatch tables dispatching to each other. Specialization is still enabled while tracing. * Better handling of polymorphism. We leverage the specializing interpreter for this. Cons: * (For now) requires tail calling interpreter or computed gotos. This means no Windows JIT for now :(. Not to fret, tail calling is coming soon to Windows though https://github.com/python/cpython/pull/139962 Design: * After each instruction, the `record_previous_inst` function/label is executed. This does as the name suggests. * The tracing interpreter lowers bytecode to uops directly so that it can obtain "fresh" values at the point of lowering. * The tracing version behaves nearly identical to the normal interpreter, in fact it even has specialization! This allows it to run without much of a slowdown when tracing. The actual cost of tracing is only a function call and writes to memory. * The tracing interpreter uses the specializing interpreter's deopt to naturally form the side exit chains. This allows it to side exit chain effectively, without repeating much code. We force a re-specializing when tracing a deopt. * The tracing interpreter can even handle goto errors/exceptions, but I chose to disable them for now as it's not tested. * Because we do not share interpreter dispatch, there is should be no significant slowdown to the original specializing interpreter on tailcall and computed got with JIT disabled. With JIT enabled, there might be a slowdown in the form of the JIT trying to trace. * Things that could have dynamic instruction pointer effects are guarded on. The guard deopts to a new instruction --- `_DYNAMIC_EXIT`.
2025-11-14 02:08:32 +08:00
break;
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
case _GUARD_IP_YIELD_VALUE_r33: {
CHECK_CURRENT_CACHED_VALUES(3);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef _stack_item_0 = _tos_cache0;
_PyStackRef _stack_item_1 = _tos_cache1;
_PyStackRef _stack_item_2 = _tos_cache2;
gh-139109: A new tracing JIT compiler frontend for CPython (GH-140310) This PR changes the current JIT model from trace projection to trace recording. Benchmarking: better pyperformance (about 1.7% overall) geomean versus current https://raw.githubusercontent.com/facebookexperimental/free-threading-benchmarking/refs/heads/main/results/bm-20251108-3.15.0a1%2B-7e2bc1d-JIT/bm-20251108-vultr-x86_64-Fidget%252dSpinner-tracing_jit-3.15.0a1%2B-7e2bc1d-vs-base.svg, 100% faster Richards on the most improved benchmark versus the current JIT. Slowdown of about 10-15% on the worst benchmark versus the current JIT. **Note: the fastest version isn't the one merged, as it relies on fixing bugs in the specializing interpreter, which is left to another PR**. The speedup in the merged version is about 1.1%. https://raw.githubusercontent.com/facebookexperimental/free-threading-benchmarking/refs/heads/main/results/bm-20251112-3.15.0a1%2B-f8a764a-JIT/bm-20251112-vultr-x86_64-Fidget%252dSpinner-tracing_jit-3.15.0a1%2B-f8a764a-vs-base.svg Stats: 50% more uops executed, 30% more traces entered the last time we ran them. It also suggests our trace lengths for a real trace recording JIT are too short, as a lot of trace too long aborts https://github.com/facebookexperimental/free-threading-benchmarking/blob/main/results/bm-20251023-3.15.0a1%2B-eb73378-CLANG%2CJIT/bm-20251023-vultr-x86_64-Fidget%252dSpinner-tracing_jit-3.15.0a1%2B-eb73378-pystats-vs-base.md . This new JIT frontend is already able to record/execute significantly more instructions than the previous JIT frontend. In this PR, we are now able to record through custom dunders, simple object creation, generators, etc. None of these were done by the old JIT frontend. Some custom dunders uops were discovered to be broken as part of this work gh-140277 The optimizer stack space check is disabled, as it's no longer valid to deal with underflow. Pros: * Ignoring the generated tracer code as it's automatically created, this is only additional 1k lines of code. The maintenance burden is handled by the DSL and code generator. * `optimizer.c` is now significantly simpler, as we don't have to do strange things to recover the bytecode from a trace. * The new JIT frontend is able to handle a lot more control-flow than the old one. * Tracing is very low overhead. We use the tail calling interpreter/computed goto interpreter to switch between tracing mode and non-tracing mode. I call this mechanism dual dispatch, as we have two dispatch tables dispatching to each other. Specialization is still enabled while tracing. * Better handling of polymorphism. We leverage the specializing interpreter for this. Cons: * (For now) requires tail calling interpreter or computed gotos. This means no Windows JIT for now :(. Not to fret, tail calling is coming soon to Windows though https://github.com/python/cpython/pull/139962 Design: * After each instruction, the `record_previous_inst` function/label is executed. This does as the name suggests. * The tracing interpreter lowers bytecode to uops directly so that it can obtain "fresh" values at the point of lowering. * The tracing version behaves nearly identical to the normal interpreter, in fact it even has specialization! This allows it to run without much of a slowdown when tracing. The actual cost of tracing is only a function call and writes to memory. * The tracing interpreter uses the specializing interpreter's deopt to naturally form the side exit chains. This allows it to side exit chain effectively, without repeating much code. We force a re-specializing when tracing a deopt. * The tracing interpreter can even handle goto errors/exceptions, but I chose to disable them for now as it's not tested. * Because we do not share interpreter dispatch, there is should be no significant slowdown to the original specializing interpreter on tailcall and computed got with JIT disabled. With JIT enabled, there might be a slowdown in the form of the JIT trying to trace. * Things that could have dynamic instruction pointer effects are guarded on. The guard deopts to a new instruction --- `_DYNAMIC_EXIT`.
2025-11-14 02:08:32 +08:00
#define OFFSET_OF_YIELD_VALUE ((1+INLINE_CACHE_ENTRIES_SEND))
GH-140683: JIT: Improve machine code for loading smaller constants on AArch64. (GH-142511) * Use movz and movk instructions for loading 16 and 32 bit operands and oparg. * Loading of 64 bit operands is unchanged.
2025-12-11 12:33:39 +00:00
PyObject *ip = (PyObject *)CURRENT_OPERAND0_64();
gh-139109: A new tracing JIT compiler frontend for CPython (GH-140310) This PR changes the current JIT model from trace projection to trace recording. Benchmarking: better pyperformance (about 1.7% overall) geomean versus current https://raw.githubusercontent.com/facebookexperimental/free-threading-benchmarking/refs/heads/main/results/bm-20251108-3.15.0a1%2B-7e2bc1d-JIT/bm-20251108-vultr-x86_64-Fidget%252dSpinner-tracing_jit-3.15.0a1%2B-7e2bc1d-vs-base.svg, 100% faster Richards on the most improved benchmark versus the current JIT. Slowdown of about 10-15% on the worst benchmark versus the current JIT. **Note: the fastest version isn't the one merged, as it relies on fixing bugs in the specializing interpreter, which is left to another PR**. The speedup in the merged version is about 1.1%. https://raw.githubusercontent.com/facebookexperimental/free-threading-benchmarking/refs/heads/main/results/bm-20251112-3.15.0a1%2B-f8a764a-JIT/bm-20251112-vultr-x86_64-Fidget%252dSpinner-tracing_jit-3.15.0a1%2B-f8a764a-vs-base.svg Stats: 50% more uops executed, 30% more traces entered the last time we ran them. It also suggests our trace lengths for a real trace recording JIT are too short, as a lot of trace too long aborts https://github.com/facebookexperimental/free-threading-benchmarking/blob/main/results/bm-20251023-3.15.0a1%2B-eb73378-CLANG%2CJIT/bm-20251023-vultr-x86_64-Fidget%252dSpinner-tracing_jit-3.15.0a1%2B-eb73378-pystats-vs-base.md . This new JIT frontend is already able to record/execute significantly more instructions than the previous JIT frontend. In this PR, we are now able to record through custom dunders, simple object creation, generators, etc. None of these were done by the old JIT frontend. Some custom dunders uops were discovered to be broken as part of this work gh-140277 The optimizer stack space check is disabled, as it's no longer valid to deal with underflow. Pros: * Ignoring the generated tracer code as it's automatically created, this is only additional 1k lines of code. The maintenance burden is handled by the DSL and code generator. * `optimizer.c` is now significantly simpler, as we don't have to do strange things to recover the bytecode from a trace. * The new JIT frontend is able to handle a lot more control-flow than the old one. * Tracing is very low overhead. We use the tail calling interpreter/computed goto interpreter to switch between tracing mode and non-tracing mode. I call this mechanism dual dispatch, as we have two dispatch tables dispatching to each other. Specialization is still enabled while tracing. * Better handling of polymorphism. We leverage the specializing interpreter for this. Cons: * (For now) requires tail calling interpreter or computed gotos. This means no Windows JIT for now :(. Not to fret, tail calling is coming soon to Windows though https://github.com/python/cpython/pull/139962 Design: * After each instruction, the `record_previous_inst` function/label is executed. This does as the name suggests. * The tracing interpreter lowers bytecode to uops directly so that it can obtain "fresh" values at the point of lowering. * The tracing version behaves nearly identical to the normal interpreter, in fact it even has specialization! This allows it to run without much of a slowdown when tracing. The actual cost of tracing is only a function call and writes to memory. * The tracing interpreter uses the specializing interpreter's deopt to naturally form the side exit chains. This allows it to side exit chain effectively, without repeating much code. We force a re-specializing when tracing a deopt. * The tracing interpreter can even handle goto errors/exceptions, but I chose to disable them for now as it's not tested. * Because we do not share interpreter dispatch, there is should be no significant slowdown to the original specializing interpreter on tailcall and computed got with JIT disabled. With JIT enabled, there might be a slowdown in the form of the JIT trying to trace. * Things that could have dynamic instruction pointer effects are guarded on. The guard deopts to a new instruction --- `_DYNAMIC_EXIT`.
2025-11-14 02:08:32 +08:00
_Py_CODEUNIT *target = frame->instr_ptr + OFFSET_OF_YIELD_VALUE;
if (target != (_Py_CODEUNIT *)ip) {
frame->instr_ptr += OFFSET_OF_YIELD_VALUE;
if (true) {
UOP_STAT_INC(uopcode, miss);
GH-142718: JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. (GH-142762) JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. * Make sure that stack is in correct state at side exits with TOS cached values * Simplify choice of cached items for side exits
2025-12-15 19:01:29 +00:00
_tos_cache2 = _stack_item_2;
_tos_cache1 = _stack_item_1;
_tos_cache0 = _stack_item_0;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(3);
gh-139109: A new tracing JIT compiler frontend for CPython (GH-140310) This PR changes the current JIT model from trace projection to trace recording. Benchmarking: better pyperformance (about 1.7% overall) geomean versus current https://raw.githubusercontent.com/facebookexperimental/free-threading-benchmarking/refs/heads/main/results/bm-20251108-3.15.0a1%2B-7e2bc1d-JIT/bm-20251108-vultr-x86_64-Fidget%252dSpinner-tracing_jit-3.15.0a1%2B-7e2bc1d-vs-base.svg, 100% faster Richards on the most improved benchmark versus the current JIT. Slowdown of about 10-15% on the worst benchmark versus the current JIT. **Note: the fastest version isn't the one merged, as it relies on fixing bugs in the specializing interpreter, which is left to another PR**. The speedup in the merged version is about 1.1%. https://raw.githubusercontent.com/facebookexperimental/free-threading-benchmarking/refs/heads/main/results/bm-20251112-3.15.0a1%2B-f8a764a-JIT/bm-20251112-vultr-x86_64-Fidget%252dSpinner-tracing_jit-3.15.0a1%2B-f8a764a-vs-base.svg Stats: 50% more uops executed, 30% more traces entered the last time we ran them. It also suggests our trace lengths for a real trace recording JIT are too short, as a lot of trace too long aborts https://github.com/facebookexperimental/free-threading-benchmarking/blob/main/results/bm-20251023-3.15.0a1%2B-eb73378-CLANG%2CJIT/bm-20251023-vultr-x86_64-Fidget%252dSpinner-tracing_jit-3.15.0a1%2B-eb73378-pystats-vs-base.md . This new JIT frontend is already able to record/execute significantly more instructions than the previous JIT frontend. In this PR, we are now able to record through custom dunders, simple object creation, generators, etc. None of these were done by the old JIT frontend. Some custom dunders uops were discovered to be broken as part of this work gh-140277 The optimizer stack space check is disabled, as it's no longer valid to deal with underflow. Pros: * Ignoring the generated tracer code as it's automatically created, this is only additional 1k lines of code. The maintenance burden is handled by the DSL and code generator. * `optimizer.c` is now significantly simpler, as we don't have to do strange things to recover the bytecode from a trace. * The new JIT frontend is able to handle a lot more control-flow than the old one. * Tracing is very low overhead. We use the tail calling interpreter/computed goto interpreter to switch between tracing mode and non-tracing mode. I call this mechanism dual dispatch, as we have two dispatch tables dispatching to each other. Specialization is still enabled while tracing. * Better handling of polymorphism. We leverage the specializing interpreter for this. Cons: * (For now) requires tail calling interpreter or computed gotos. This means no Windows JIT for now :(. Not to fret, tail calling is coming soon to Windows though https://github.com/python/cpython/pull/139962 Design: * After each instruction, the `record_previous_inst` function/label is executed. This does as the name suggests. * The tracing interpreter lowers bytecode to uops directly so that it can obtain "fresh" values at the point of lowering. * The tracing version behaves nearly identical to the normal interpreter, in fact it even has specialization! This allows it to run without much of a slowdown when tracing. The actual cost of tracing is only a function call and writes to memory. * The tracing interpreter uses the specializing interpreter's deopt to naturally form the side exit chains. This allows it to side exit chain effectively, without repeating much code. We force a re-specializing when tracing a deopt. * The tracing interpreter can even handle goto errors/exceptions, but I chose to disable them for now as it's not tested. * Because we do not share interpreter dispatch, there is should be no significant slowdown to the original specializing interpreter on tailcall and computed got with JIT disabled. With JIT enabled, there might be a slowdown in the form of the JIT trying to trace. * Things that could have dynamic instruction pointer effects are guarded on. The guard deopts to a new instruction --- `_DYNAMIC_EXIT`.
2025-11-14 02:08:32 +08:00
JUMP_TO_JUMP_TARGET();
}
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_tos_cache2 = _stack_item_2;
_tos_cache1 = _stack_item_1;
_tos_cache0 = _stack_item_0;
SET_CURRENT_CACHED_VALUES(3);
gh-139109: A new tracing JIT compiler frontend for CPython (GH-140310) This PR changes the current JIT model from trace projection to trace recording. Benchmarking: better pyperformance (about 1.7% overall) geomean versus current https://raw.githubusercontent.com/facebookexperimental/free-threading-benchmarking/refs/heads/main/results/bm-20251108-3.15.0a1%2B-7e2bc1d-JIT/bm-20251108-vultr-x86_64-Fidget%252dSpinner-tracing_jit-3.15.0a1%2B-7e2bc1d-vs-base.svg, 100% faster Richards on the most improved benchmark versus the current JIT. Slowdown of about 10-15% on the worst benchmark versus the current JIT. **Note: the fastest version isn't the one merged, as it relies on fixing bugs in the specializing interpreter, which is left to another PR**. The speedup in the merged version is about 1.1%. https://raw.githubusercontent.com/facebookexperimental/free-threading-benchmarking/refs/heads/main/results/bm-20251112-3.15.0a1%2B-f8a764a-JIT/bm-20251112-vultr-x86_64-Fidget%252dSpinner-tracing_jit-3.15.0a1%2B-f8a764a-vs-base.svg Stats: 50% more uops executed, 30% more traces entered the last time we ran them. It also suggests our trace lengths for a real trace recording JIT are too short, as a lot of trace too long aborts https://github.com/facebookexperimental/free-threading-benchmarking/blob/main/results/bm-20251023-3.15.0a1%2B-eb73378-CLANG%2CJIT/bm-20251023-vultr-x86_64-Fidget%252dSpinner-tracing_jit-3.15.0a1%2B-eb73378-pystats-vs-base.md . This new JIT frontend is already able to record/execute significantly more instructions than the previous JIT frontend. In this PR, we are now able to record through custom dunders, simple object creation, generators, etc. None of these were done by the old JIT frontend. Some custom dunders uops were discovered to be broken as part of this work gh-140277 The optimizer stack space check is disabled, as it's no longer valid to deal with underflow. Pros: * Ignoring the generated tracer code as it's automatically created, this is only additional 1k lines of code. The maintenance burden is handled by the DSL and code generator. * `optimizer.c` is now significantly simpler, as we don't have to do strange things to recover the bytecode from a trace. * The new JIT frontend is able to handle a lot more control-flow than the old one. * Tracing is very low overhead. We use the tail calling interpreter/computed goto interpreter to switch between tracing mode and non-tracing mode. I call this mechanism dual dispatch, as we have two dispatch tables dispatching to each other. Specialization is still enabled while tracing. * Better handling of polymorphism. We leverage the specializing interpreter for this. Cons: * (For now) requires tail calling interpreter or computed gotos. This means no Windows JIT for now :(. Not to fret, tail calling is coming soon to Windows though https://github.com/python/cpython/pull/139962 Design: * After each instruction, the `record_previous_inst` function/label is executed. This does as the name suggests. * The tracing interpreter lowers bytecode to uops directly so that it can obtain "fresh" values at the point of lowering. * The tracing version behaves nearly identical to the normal interpreter, in fact it even has specialization! This allows it to run without much of a slowdown when tracing. The actual cost of tracing is only a function call and writes to memory. * The tracing interpreter uses the specializing interpreter's deopt to naturally form the side exit chains. This allows it to side exit chain effectively, without repeating much code. We force a re-specializing when tracing a deopt. * The tracing interpreter can even handle goto errors/exceptions, but I chose to disable them for now as it's not tested. * Because we do not share interpreter dispatch, there is should be no significant slowdown to the original specializing interpreter on tailcall and computed got with JIT disabled. With JIT enabled, there might be a slowdown in the form of the JIT trying to trace. * Things that could have dynamic instruction pointer effects are guarded on. The guard deopts to a new instruction --- `_DYNAMIC_EXIT`.
2025-11-14 02:08:32 +08:00
#undef OFFSET_OF_YIELD_VALUE
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _GUARD_IP_RETURN_VALUE_r00: {
CHECK_CURRENT_CACHED_VALUES(0);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
#define OFFSET_OF_RETURN_VALUE ((frame->return_offset))
GH-140683: JIT: Improve machine code for loading smaller constants on AArch64. (GH-142511) * Use movz and movk instructions for loading 16 and 32 bit operands and oparg. * Loading of 64 bit operands is unchanged.
2025-12-11 12:33:39 +00:00
PyObject *ip = (PyObject *)CURRENT_OPERAND0_64();
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_Py_CODEUNIT *target = frame->instr_ptr + OFFSET_OF_RETURN_VALUE;
if (target != (_Py_CODEUNIT *)ip) {
frame->instr_ptr += OFFSET_OF_RETURN_VALUE;
if (true) {
UOP_STAT_INC(uopcode, miss);
SET_CURRENT_CACHED_VALUES(0);
JUMP_TO_JUMP_TARGET();
}
}
SET_CURRENT_CACHED_VALUES(0);
#undef OFFSET_OF_RETURN_VALUE
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _GUARD_IP_RETURN_VALUE_r11: {
CHECK_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef _stack_item_0 = _tos_cache0;
#define OFFSET_OF_RETURN_VALUE ((frame->return_offset))
GH-140683: JIT: Improve machine code for loading smaller constants on AArch64. (GH-142511) * Use movz and movk instructions for loading 16 and 32 bit operands and oparg. * Loading of 64 bit operands is unchanged.
2025-12-11 12:33:39 +00:00
PyObject *ip = (PyObject *)CURRENT_OPERAND0_64();
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_Py_CODEUNIT *target = frame->instr_ptr + OFFSET_OF_RETURN_VALUE;
if (target != (_Py_CODEUNIT *)ip) {
frame->instr_ptr += OFFSET_OF_RETURN_VALUE;
if (true) {
UOP_STAT_INC(uopcode, miss);
GH-142718: JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. (GH-142762) JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. * Make sure that stack is in correct state at side exits with TOS cached values * Simplify choice of cached items for side exits
2025-12-15 19:01:29 +00:00
_tos_cache0 = _stack_item_0;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(1);
JUMP_TO_JUMP_TARGET();
}
}
_tos_cache0 = _stack_item_0;
SET_CURRENT_CACHED_VALUES(1);
#undef OFFSET_OF_RETURN_VALUE
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _GUARD_IP_RETURN_VALUE_r22: {
CHECK_CURRENT_CACHED_VALUES(2);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef _stack_item_0 = _tos_cache0;
_PyStackRef _stack_item_1 = _tos_cache1;
#define OFFSET_OF_RETURN_VALUE ((frame->return_offset))
GH-140683: JIT: Improve machine code for loading smaller constants on AArch64. (GH-142511) * Use movz and movk instructions for loading 16 and 32 bit operands and oparg. * Loading of 64 bit operands is unchanged.
2025-12-11 12:33:39 +00:00
PyObject *ip = (PyObject *)CURRENT_OPERAND0_64();
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_Py_CODEUNIT *target = frame->instr_ptr + OFFSET_OF_RETURN_VALUE;
if (target != (_Py_CODEUNIT *)ip) {
frame->instr_ptr += OFFSET_OF_RETURN_VALUE;
if (true) {
UOP_STAT_INC(uopcode, miss);
GH-142718: JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. (GH-142762) JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. * Make sure that stack is in correct state at side exits with TOS cached values * Simplify choice of cached items for side exits
2025-12-15 19:01:29 +00:00
_tos_cache1 = _stack_item_1;
_tos_cache0 = _stack_item_0;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(2);
JUMP_TO_JUMP_TARGET();
}
}
_tos_cache1 = _stack_item_1;
_tos_cache0 = _stack_item_0;
SET_CURRENT_CACHED_VALUES(2);
#undef OFFSET_OF_RETURN_VALUE
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-139109: A new tracing JIT compiler frontend for CPython (GH-140310) This PR changes the current JIT model from trace projection to trace recording. Benchmarking: better pyperformance (about 1.7% overall) geomean versus current https://raw.githubusercontent.com/facebookexperimental/free-threading-benchmarking/refs/heads/main/results/bm-20251108-3.15.0a1%2B-7e2bc1d-JIT/bm-20251108-vultr-x86_64-Fidget%252dSpinner-tracing_jit-3.15.0a1%2B-7e2bc1d-vs-base.svg, 100% faster Richards on the most improved benchmark versus the current JIT. Slowdown of about 10-15% on the worst benchmark versus the current JIT. **Note: the fastest version isn't the one merged, as it relies on fixing bugs in the specializing interpreter, which is left to another PR**. The speedup in the merged version is about 1.1%. https://raw.githubusercontent.com/facebookexperimental/free-threading-benchmarking/refs/heads/main/results/bm-20251112-3.15.0a1%2B-f8a764a-JIT/bm-20251112-vultr-x86_64-Fidget%252dSpinner-tracing_jit-3.15.0a1%2B-f8a764a-vs-base.svg Stats: 50% more uops executed, 30% more traces entered the last time we ran them. It also suggests our trace lengths for a real trace recording JIT are too short, as a lot of trace too long aborts https://github.com/facebookexperimental/free-threading-benchmarking/blob/main/results/bm-20251023-3.15.0a1%2B-eb73378-CLANG%2CJIT/bm-20251023-vultr-x86_64-Fidget%252dSpinner-tracing_jit-3.15.0a1%2B-eb73378-pystats-vs-base.md . This new JIT frontend is already able to record/execute significantly more instructions than the previous JIT frontend. In this PR, we are now able to record through custom dunders, simple object creation, generators, etc. None of these were done by the old JIT frontend. Some custom dunders uops were discovered to be broken as part of this work gh-140277 The optimizer stack space check is disabled, as it's no longer valid to deal with underflow. Pros: * Ignoring the generated tracer code as it's automatically created, this is only additional 1k lines of code. The maintenance burden is handled by the DSL and code generator. * `optimizer.c` is now significantly simpler, as we don't have to do strange things to recover the bytecode from a trace. * The new JIT frontend is able to handle a lot more control-flow than the old one. * Tracing is very low overhead. We use the tail calling interpreter/computed goto interpreter to switch between tracing mode and non-tracing mode. I call this mechanism dual dispatch, as we have two dispatch tables dispatching to each other. Specialization is still enabled while tracing. * Better handling of polymorphism. We leverage the specializing interpreter for this. Cons: * (For now) requires tail calling interpreter or computed gotos. This means no Windows JIT for now :(. Not to fret, tail calling is coming soon to Windows though https://github.com/python/cpython/pull/139962 Design: * After each instruction, the `record_previous_inst` function/label is executed. This does as the name suggests. * The tracing interpreter lowers bytecode to uops directly so that it can obtain "fresh" values at the point of lowering. * The tracing version behaves nearly identical to the normal interpreter, in fact it even has specialization! This allows it to run without much of a slowdown when tracing. The actual cost of tracing is only a function call and writes to memory. * The tracing interpreter uses the specializing interpreter's deopt to naturally form the side exit chains. This allows it to side exit chain effectively, without repeating much code. We force a re-specializing when tracing a deopt. * The tracing interpreter can even handle goto errors/exceptions, but I chose to disable them for now as it's not tested. * Because we do not share interpreter dispatch, there is should be no significant slowdown to the original specializing interpreter on tailcall and computed got with JIT disabled. With JIT enabled, there might be a slowdown in the form of the JIT trying to trace. * Things that could have dynamic instruction pointer effects are guarded on. The guard deopts to a new instruction --- `_DYNAMIC_EXIT`.
2025-11-14 02:08:32 +08:00
break;
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
case _GUARD_IP_RETURN_VALUE_r33: {
CHECK_CURRENT_CACHED_VALUES(3);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef _stack_item_0 = _tos_cache0;
_PyStackRef _stack_item_1 = _tos_cache1;
_PyStackRef _stack_item_2 = _tos_cache2;
gh-139109: A new tracing JIT compiler frontend for CPython (GH-140310) This PR changes the current JIT model from trace projection to trace recording. Benchmarking: better pyperformance (about 1.7% overall) geomean versus current https://raw.githubusercontent.com/facebookexperimental/free-threading-benchmarking/refs/heads/main/results/bm-20251108-3.15.0a1%2B-7e2bc1d-JIT/bm-20251108-vultr-x86_64-Fidget%252dSpinner-tracing_jit-3.15.0a1%2B-7e2bc1d-vs-base.svg, 100% faster Richards on the most improved benchmark versus the current JIT. Slowdown of about 10-15% on the worst benchmark versus the current JIT. **Note: the fastest version isn't the one merged, as it relies on fixing bugs in the specializing interpreter, which is left to another PR**. The speedup in the merged version is about 1.1%. https://raw.githubusercontent.com/facebookexperimental/free-threading-benchmarking/refs/heads/main/results/bm-20251112-3.15.0a1%2B-f8a764a-JIT/bm-20251112-vultr-x86_64-Fidget%252dSpinner-tracing_jit-3.15.0a1%2B-f8a764a-vs-base.svg Stats: 50% more uops executed, 30% more traces entered the last time we ran them. It also suggests our trace lengths for a real trace recording JIT are too short, as a lot of trace too long aborts https://github.com/facebookexperimental/free-threading-benchmarking/blob/main/results/bm-20251023-3.15.0a1%2B-eb73378-CLANG%2CJIT/bm-20251023-vultr-x86_64-Fidget%252dSpinner-tracing_jit-3.15.0a1%2B-eb73378-pystats-vs-base.md . This new JIT frontend is already able to record/execute significantly more instructions than the previous JIT frontend. In this PR, we are now able to record through custom dunders, simple object creation, generators, etc. None of these were done by the old JIT frontend. Some custom dunders uops were discovered to be broken as part of this work gh-140277 The optimizer stack space check is disabled, as it's no longer valid to deal with underflow. Pros: * Ignoring the generated tracer code as it's automatically created, this is only additional 1k lines of code. The maintenance burden is handled by the DSL and code generator. * `optimizer.c` is now significantly simpler, as we don't have to do strange things to recover the bytecode from a trace. * The new JIT frontend is able to handle a lot more control-flow than the old one. * Tracing is very low overhead. We use the tail calling interpreter/computed goto interpreter to switch between tracing mode and non-tracing mode. I call this mechanism dual dispatch, as we have two dispatch tables dispatching to each other. Specialization is still enabled while tracing. * Better handling of polymorphism. We leverage the specializing interpreter for this. Cons: * (For now) requires tail calling interpreter or computed gotos. This means no Windows JIT for now :(. Not to fret, tail calling is coming soon to Windows though https://github.com/python/cpython/pull/139962 Design: * After each instruction, the `record_previous_inst` function/label is executed. This does as the name suggests. * The tracing interpreter lowers bytecode to uops directly so that it can obtain "fresh" values at the point of lowering. * The tracing version behaves nearly identical to the normal interpreter, in fact it even has specialization! This allows it to run without much of a slowdown when tracing. The actual cost of tracing is only a function call and writes to memory. * The tracing interpreter uses the specializing interpreter's deopt to naturally form the side exit chains. This allows it to side exit chain effectively, without repeating much code. We force a re-specializing when tracing a deopt. * The tracing interpreter can even handle goto errors/exceptions, but I chose to disable them for now as it's not tested. * Because we do not share interpreter dispatch, there is should be no significant slowdown to the original specializing interpreter on tailcall and computed got with JIT disabled. With JIT enabled, there might be a slowdown in the form of the JIT trying to trace. * Things that could have dynamic instruction pointer effects are guarded on. The guard deopts to a new instruction --- `_DYNAMIC_EXIT`.
2025-11-14 02:08:32 +08:00
#define OFFSET_OF_RETURN_VALUE ((frame->return_offset))
GH-140683: JIT: Improve machine code for loading smaller constants on AArch64. (GH-142511) * Use movz and movk instructions for loading 16 and 32 bit operands and oparg. * Loading of 64 bit operands is unchanged.
2025-12-11 12:33:39 +00:00
PyObject *ip = (PyObject *)CURRENT_OPERAND0_64();
gh-139109: A new tracing JIT compiler frontend for CPython (GH-140310) This PR changes the current JIT model from trace projection to trace recording. Benchmarking: better pyperformance (about 1.7% overall) geomean versus current https://raw.githubusercontent.com/facebookexperimental/free-threading-benchmarking/refs/heads/main/results/bm-20251108-3.15.0a1%2B-7e2bc1d-JIT/bm-20251108-vultr-x86_64-Fidget%252dSpinner-tracing_jit-3.15.0a1%2B-7e2bc1d-vs-base.svg, 100% faster Richards on the most improved benchmark versus the current JIT. Slowdown of about 10-15% on the worst benchmark versus the current JIT. **Note: the fastest version isn't the one merged, as it relies on fixing bugs in the specializing interpreter, which is left to another PR**. The speedup in the merged version is about 1.1%. https://raw.githubusercontent.com/facebookexperimental/free-threading-benchmarking/refs/heads/main/results/bm-20251112-3.15.0a1%2B-f8a764a-JIT/bm-20251112-vultr-x86_64-Fidget%252dSpinner-tracing_jit-3.15.0a1%2B-f8a764a-vs-base.svg Stats: 50% more uops executed, 30% more traces entered the last time we ran them. It also suggests our trace lengths for a real trace recording JIT are too short, as a lot of trace too long aborts https://github.com/facebookexperimental/free-threading-benchmarking/blob/main/results/bm-20251023-3.15.0a1%2B-eb73378-CLANG%2CJIT/bm-20251023-vultr-x86_64-Fidget%252dSpinner-tracing_jit-3.15.0a1%2B-eb73378-pystats-vs-base.md . This new JIT frontend is already able to record/execute significantly more instructions than the previous JIT frontend. In this PR, we are now able to record through custom dunders, simple object creation, generators, etc. None of these were done by the old JIT frontend. Some custom dunders uops were discovered to be broken as part of this work gh-140277 The optimizer stack space check is disabled, as it's no longer valid to deal with underflow. Pros: * Ignoring the generated tracer code as it's automatically created, this is only additional 1k lines of code. The maintenance burden is handled by the DSL and code generator. * `optimizer.c` is now significantly simpler, as we don't have to do strange things to recover the bytecode from a trace. * The new JIT frontend is able to handle a lot more control-flow than the old one. * Tracing is very low overhead. We use the tail calling interpreter/computed goto interpreter to switch between tracing mode and non-tracing mode. I call this mechanism dual dispatch, as we have two dispatch tables dispatching to each other. Specialization is still enabled while tracing. * Better handling of polymorphism. We leverage the specializing interpreter for this. Cons: * (For now) requires tail calling interpreter or computed gotos. This means no Windows JIT for now :(. Not to fret, tail calling is coming soon to Windows though https://github.com/python/cpython/pull/139962 Design: * After each instruction, the `record_previous_inst` function/label is executed. This does as the name suggests. * The tracing interpreter lowers bytecode to uops directly so that it can obtain "fresh" values at the point of lowering. * The tracing version behaves nearly identical to the normal interpreter, in fact it even has specialization! This allows it to run without much of a slowdown when tracing. The actual cost of tracing is only a function call and writes to memory. * The tracing interpreter uses the specializing interpreter's deopt to naturally form the side exit chains. This allows it to side exit chain effectively, without repeating much code. We force a re-specializing when tracing a deopt. * The tracing interpreter can even handle goto errors/exceptions, but I chose to disable them for now as it's not tested. * Because we do not share interpreter dispatch, there is should be no significant slowdown to the original specializing interpreter on tailcall and computed got with JIT disabled. With JIT enabled, there might be a slowdown in the form of the JIT trying to trace. * Things that could have dynamic instruction pointer effects are guarded on. The guard deopts to a new instruction --- `_DYNAMIC_EXIT`.
2025-11-14 02:08:32 +08:00
_Py_CODEUNIT *target = frame->instr_ptr + OFFSET_OF_RETURN_VALUE;
if (target != (_Py_CODEUNIT *)ip) {
frame->instr_ptr += OFFSET_OF_RETURN_VALUE;
if (true) {
UOP_STAT_INC(uopcode, miss);
GH-142718: JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. (GH-142762) JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. * Make sure that stack is in correct state at side exits with TOS cached values * Simplify choice of cached items for side exits
2025-12-15 19:01:29 +00:00
_tos_cache2 = _stack_item_2;
_tos_cache1 = _stack_item_1;
_tos_cache0 = _stack_item_0;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(3);
gh-139109: A new tracing JIT compiler frontend for CPython (GH-140310) This PR changes the current JIT model from trace projection to trace recording. Benchmarking: better pyperformance (about 1.7% overall) geomean versus current https://raw.githubusercontent.com/facebookexperimental/free-threading-benchmarking/refs/heads/main/results/bm-20251108-3.15.0a1%2B-7e2bc1d-JIT/bm-20251108-vultr-x86_64-Fidget%252dSpinner-tracing_jit-3.15.0a1%2B-7e2bc1d-vs-base.svg, 100% faster Richards on the most improved benchmark versus the current JIT. Slowdown of about 10-15% on the worst benchmark versus the current JIT. **Note: the fastest version isn't the one merged, as it relies on fixing bugs in the specializing interpreter, which is left to another PR**. The speedup in the merged version is about 1.1%. https://raw.githubusercontent.com/facebookexperimental/free-threading-benchmarking/refs/heads/main/results/bm-20251112-3.15.0a1%2B-f8a764a-JIT/bm-20251112-vultr-x86_64-Fidget%252dSpinner-tracing_jit-3.15.0a1%2B-f8a764a-vs-base.svg Stats: 50% more uops executed, 30% more traces entered the last time we ran them. It also suggests our trace lengths for a real trace recording JIT are too short, as a lot of trace too long aborts https://github.com/facebookexperimental/free-threading-benchmarking/blob/main/results/bm-20251023-3.15.0a1%2B-eb73378-CLANG%2CJIT/bm-20251023-vultr-x86_64-Fidget%252dSpinner-tracing_jit-3.15.0a1%2B-eb73378-pystats-vs-base.md . This new JIT frontend is already able to record/execute significantly more instructions than the previous JIT frontend. In this PR, we are now able to record through custom dunders, simple object creation, generators, etc. None of these were done by the old JIT frontend. Some custom dunders uops were discovered to be broken as part of this work gh-140277 The optimizer stack space check is disabled, as it's no longer valid to deal with underflow. Pros: * Ignoring the generated tracer code as it's automatically created, this is only additional 1k lines of code. The maintenance burden is handled by the DSL and code generator. * `optimizer.c` is now significantly simpler, as we don't have to do strange things to recover the bytecode from a trace. * The new JIT frontend is able to handle a lot more control-flow than the old one. * Tracing is very low overhead. We use the tail calling interpreter/computed goto interpreter to switch between tracing mode and non-tracing mode. I call this mechanism dual dispatch, as we have two dispatch tables dispatching to each other. Specialization is still enabled while tracing. * Better handling of polymorphism. We leverage the specializing interpreter for this. Cons: * (For now) requires tail calling interpreter or computed gotos. This means no Windows JIT for now :(. Not to fret, tail calling is coming soon to Windows though https://github.com/python/cpython/pull/139962 Design: * After each instruction, the `record_previous_inst` function/label is executed. This does as the name suggests. * The tracing interpreter lowers bytecode to uops directly so that it can obtain "fresh" values at the point of lowering. * The tracing version behaves nearly identical to the normal interpreter, in fact it even has specialization! This allows it to run without much of a slowdown when tracing. The actual cost of tracing is only a function call and writes to memory. * The tracing interpreter uses the specializing interpreter's deopt to naturally form the side exit chains. This allows it to side exit chain effectively, without repeating much code. We force a re-specializing when tracing a deopt. * The tracing interpreter can even handle goto errors/exceptions, but I chose to disable them for now as it's not tested. * Because we do not share interpreter dispatch, there is should be no significant slowdown to the original specializing interpreter on tailcall and computed got with JIT disabled. With JIT enabled, there might be a slowdown in the form of the JIT trying to trace. * Things that could have dynamic instruction pointer effects are guarded on. The guard deopts to a new instruction --- `_DYNAMIC_EXIT`.
2025-11-14 02:08:32 +08:00
JUMP_TO_JUMP_TARGET();
}
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_tos_cache2 = _stack_item_2;
_tos_cache1 = _stack_item_1;
_tos_cache0 = _stack_item_0;
SET_CURRENT_CACHED_VALUES(3);
gh-139109: A new tracing JIT compiler frontend for CPython (GH-140310) This PR changes the current JIT model from trace projection to trace recording. Benchmarking: better pyperformance (about 1.7% overall) geomean versus current https://raw.githubusercontent.com/facebookexperimental/free-threading-benchmarking/refs/heads/main/results/bm-20251108-3.15.0a1%2B-7e2bc1d-JIT/bm-20251108-vultr-x86_64-Fidget%252dSpinner-tracing_jit-3.15.0a1%2B-7e2bc1d-vs-base.svg, 100% faster Richards on the most improved benchmark versus the current JIT. Slowdown of about 10-15% on the worst benchmark versus the current JIT. **Note: the fastest version isn't the one merged, as it relies on fixing bugs in the specializing interpreter, which is left to another PR**. The speedup in the merged version is about 1.1%. https://raw.githubusercontent.com/facebookexperimental/free-threading-benchmarking/refs/heads/main/results/bm-20251112-3.15.0a1%2B-f8a764a-JIT/bm-20251112-vultr-x86_64-Fidget%252dSpinner-tracing_jit-3.15.0a1%2B-f8a764a-vs-base.svg Stats: 50% more uops executed, 30% more traces entered the last time we ran them. It also suggests our trace lengths for a real trace recording JIT are too short, as a lot of trace too long aborts https://github.com/facebookexperimental/free-threading-benchmarking/blob/main/results/bm-20251023-3.15.0a1%2B-eb73378-CLANG%2CJIT/bm-20251023-vultr-x86_64-Fidget%252dSpinner-tracing_jit-3.15.0a1%2B-eb73378-pystats-vs-base.md . This new JIT frontend is already able to record/execute significantly more instructions than the previous JIT frontend. In this PR, we are now able to record through custom dunders, simple object creation, generators, etc. None of these were done by the old JIT frontend. Some custom dunders uops were discovered to be broken as part of this work gh-140277 The optimizer stack space check is disabled, as it's no longer valid to deal with underflow. Pros: * Ignoring the generated tracer code as it's automatically created, this is only additional 1k lines of code. The maintenance burden is handled by the DSL and code generator. * `optimizer.c` is now significantly simpler, as we don't have to do strange things to recover the bytecode from a trace. * The new JIT frontend is able to handle a lot more control-flow than the old one. * Tracing is very low overhead. We use the tail calling interpreter/computed goto interpreter to switch between tracing mode and non-tracing mode. I call this mechanism dual dispatch, as we have two dispatch tables dispatching to each other. Specialization is still enabled while tracing. * Better handling of polymorphism. We leverage the specializing interpreter for this. Cons: * (For now) requires tail calling interpreter or computed gotos. This means no Windows JIT for now :(. Not to fret, tail calling is coming soon to Windows though https://github.com/python/cpython/pull/139962 Design: * After each instruction, the `record_previous_inst` function/label is executed. This does as the name suggests. * The tracing interpreter lowers bytecode to uops directly so that it can obtain "fresh" values at the point of lowering. * The tracing version behaves nearly identical to the normal interpreter, in fact it even has specialization! This allows it to run without much of a slowdown when tracing. The actual cost of tracing is only a function call and writes to memory. * The tracing interpreter uses the specializing interpreter's deopt to naturally form the side exit chains. This allows it to side exit chain effectively, without repeating much code. We force a re-specializing when tracing a deopt. * The tracing interpreter can even handle goto errors/exceptions, but I chose to disable them for now as it's not tested. * Because we do not share interpreter dispatch, there is should be no significant slowdown to the original specializing interpreter on tailcall and computed got with JIT disabled. With JIT enabled, there might be a slowdown in the form of the JIT trying to trace. * Things that could have dynamic instruction pointer effects are guarded on. The guard deopts to a new instruction --- `_DYNAMIC_EXIT`.
2025-11-14 02:08:32 +08:00
#undef OFFSET_OF_RETURN_VALUE
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _GUARD_IP_RETURN_GENERATOR_r00: {
CHECK_CURRENT_CACHED_VALUES(0);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
#define OFFSET_OF_RETURN_GENERATOR ((frame->return_offset))
GH-140683: JIT: Improve machine code for loading smaller constants on AArch64. (GH-142511) * Use movz and movk instructions for loading 16 and 32 bit operands and oparg. * Loading of 64 bit operands is unchanged.
2025-12-11 12:33:39 +00:00
PyObject *ip = (PyObject *)CURRENT_OPERAND0_64();
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_Py_CODEUNIT *target = frame->instr_ptr + OFFSET_OF_RETURN_GENERATOR;
if (target != (_Py_CODEUNIT *)ip) {
frame->instr_ptr += OFFSET_OF_RETURN_GENERATOR;
if (true) {
UOP_STAT_INC(uopcode, miss);
SET_CURRENT_CACHED_VALUES(0);
JUMP_TO_JUMP_TARGET();
}
}
SET_CURRENT_CACHED_VALUES(0);
#undef OFFSET_OF_RETURN_GENERATOR
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _GUARD_IP_RETURN_GENERATOR_r11: {
CHECK_CURRENT_CACHED_VALUES(1);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef _stack_item_0 = _tos_cache0;
#define OFFSET_OF_RETURN_GENERATOR ((frame->return_offset))
GH-140683: JIT: Improve machine code for loading smaller constants on AArch64. (GH-142511) * Use movz and movk instructions for loading 16 and 32 bit operands and oparg. * Loading of 64 bit operands is unchanged.
2025-12-11 12:33:39 +00:00
PyObject *ip = (PyObject *)CURRENT_OPERAND0_64();
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_Py_CODEUNIT *target = frame->instr_ptr + OFFSET_OF_RETURN_GENERATOR;
if (target != (_Py_CODEUNIT *)ip) {
frame->instr_ptr += OFFSET_OF_RETURN_GENERATOR;
if (true) {
UOP_STAT_INC(uopcode, miss);
GH-142718: JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. (GH-142762) JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. * Make sure that stack is in correct state at side exits with TOS cached values * Simplify choice of cached items for side exits
2025-12-15 19:01:29 +00:00
_tos_cache0 = _stack_item_0;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(1);
JUMP_TO_JUMP_TARGET();
}
}
_tos_cache0 = _stack_item_0;
SET_CURRENT_CACHED_VALUES(1);
#undef OFFSET_OF_RETURN_GENERATOR
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
break;
}
case _GUARD_IP_RETURN_GENERATOR_r22: {
CHECK_CURRENT_CACHED_VALUES(2);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef _stack_item_0 = _tos_cache0;
_PyStackRef _stack_item_1 = _tos_cache1;
#define OFFSET_OF_RETURN_GENERATOR ((frame->return_offset))
GH-140683: JIT: Improve machine code for loading smaller constants on AArch64. (GH-142511) * Use movz and movk instructions for loading 16 and 32 bit operands and oparg. * Loading of 64 bit operands is unchanged.
2025-12-11 12:33:39 +00:00
PyObject *ip = (PyObject *)CURRENT_OPERAND0_64();
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_Py_CODEUNIT *target = frame->instr_ptr + OFFSET_OF_RETURN_GENERATOR;
if (target != (_Py_CODEUNIT *)ip) {
frame->instr_ptr += OFFSET_OF_RETURN_GENERATOR;
if (true) {
UOP_STAT_INC(uopcode, miss);
GH-142718: JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. (GH-142762) JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. * Make sure that stack is in correct state at side exits with TOS cached values * Simplify choice of cached items for side exits
2025-12-15 19:01:29 +00:00
_tos_cache1 = _stack_item_1;
_tos_cache0 = _stack_item_0;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(2);
JUMP_TO_JUMP_TARGET();
}
}
_tos_cache1 = _stack_item_1;
_tos_cache0 = _stack_item_0;
SET_CURRENT_CACHED_VALUES(2);
#undef OFFSET_OF_RETURN_GENERATOR
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-139109: A new tracing JIT compiler frontend for CPython (GH-140310) This PR changes the current JIT model from trace projection to trace recording. Benchmarking: better pyperformance (about 1.7% overall) geomean versus current https://raw.githubusercontent.com/facebookexperimental/free-threading-benchmarking/refs/heads/main/results/bm-20251108-3.15.0a1%2B-7e2bc1d-JIT/bm-20251108-vultr-x86_64-Fidget%252dSpinner-tracing_jit-3.15.0a1%2B-7e2bc1d-vs-base.svg, 100% faster Richards on the most improved benchmark versus the current JIT. Slowdown of about 10-15% on the worst benchmark versus the current JIT. **Note: the fastest version isn't the one merged, as it relies on fixing bugs in the specializing interpreter, which is left to another PR**. The speedup in the merged version is about 1.1%. https://raw.githubusercontent.com/facebookexperimental/free-threading-benchmarking/refs/heads/main/results/bm-20251112-3.15.0a1%2B-f8a764a-JIT/bm-20251112-vultr-x86_64-Fidget%252dSpinner-tracing_jit-3.15.0a1%2B-f8a764a-vs-base.svg Stats: 50% more uops executed, 30% more traces entered the last time we ran them. It also suggests our trace lengths for a real trace recording JIT are too short, as a lot of trace too long aborts https://github.com/facebookexperimental/free-threading-benchmarking/blob/main/results/bm-20251023-3.15.0a1%2B-eb73378-CLANG%2CJIT/bm-20251023-vultr-x86_64-Fidget%252dSpinner-tracing_jit-3.15.0a1%2B-eb73378-pystats-vs-base.md . This new JIT frontend is already able to record/execute significantly more instructions than the previous JIT frontend. In this PR, we are now able to record through custom dunders, simple object creation, generators, etc. None of these were done by the old JIT frontend. Some custom dunders uops were discovered to be broken as part of this work gh-140277 The optimizer stack space check is disabled, as it's no longer valid to deal with underflow. Pros: * Ignoring the generated tracer code as it's automatically created, this is only additional 1k lines of code. The maintenance burden is handled by the DSL and code generator. * `optimizer.c` is now significantly simpler, as we don't have to do strange things to recover the bytecode from a trace. * The new JIT frontend is able to handle a lot more control-flow than the old one. * Tracing is very low overhead. We use the tail calling interpreter/computed goto interpreter to switch between tracing mode and non-tracing mode. I call this mechanism dual dispatch, as we have two dispatch tables dispatching to each other. Specialization is still enabled while tracing. * Better handling of polymorphism. We leverage the specializing interpreter for this. Cons: * (For now) requires tail calling interpreter or computed gotos. This means no Windows JIT for now :(. Not to fret, tail calling is coming soon to Windows though https://github.com/python/cpython/pull/139962 Design: * After each instruction, the `record_previous_inst` function/label is executed. This does as the name suggests. * The tracing interpreter lowers bytecode to uops directly so that it can obtain "fresh" values at the point of lowering. * The tracing version behaves nearly identical to the normal interpreter, in fact it even has specialization! This allows it to run without much of a slowdown when tracing. The actual cost of tracing is only a function call and writes to memory. * The tracing interpreter uses the specializing interpreter's deopt to naturally form the side exit chains. This allows it to side exit chain effectively, without repeating much code. We force a re-specializing when tracing a deopt. * The tracing interpreter can even handle goto errors/exceptions, but I chose to disable them for now as it's not tested. * Because we do not share interpreter dispatch, there is should be no significant slowdown to the original specializing interpreter on tailcall and computed got with JIT disabled. With JIT enabled, there might be a slowdown in the form of the JIT trying to trace. * Things that could have dynamic instruction pointer effects are guarded on. The guard deopts to a new instruction --- `_DYNAMIC_EXIT`.
2025-11-14 02:08:32 +08:00
break;
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
case _GUARD_IP_RETURN_GENERATOR_r33: {
CHECK_CURRENT_CACHED_VALUES(3);
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_PyStackRef _stack_item_0 = _tos_cache0;
_PyStackRef _stack_item_1 = _tos_cache1;
_PyStackRef _stack_item_2 = _tos_cache2;
gh-139109: A new tracing JIT compiler frontend for CPython (GH-140310) This PR changes the current JIT model from trace projection to trace recording. Benchmarking: better pyperformance (about 1.7% overall) geomean versus current https://raw.githubusercontent.com/facebookexperimental/free-threading-benchmarking/refs/heads/main/results/bm-20251108-3.15.0a1%2B-7e2bc1d-JIT/bm-20251108-vultr-x86_64-Fidget%252dSpinner-tracing_jit-3.15.0a1%2B-7e2bc1d-vs-base.svg, 100% faster Richards on the most improved benchmark versus the current JIT. Slowdown of about 10-15% on the worst benchmark versus the current JIT. **Note: the fastest version isn't the one merged, as it relies on fixing bugs in the specializing interpreter, which is left to another PR**. The speedup in the merged version is about 1.1%. https://raw.githubusercontent.com/facebookexperimental/free-threading-benchmarking/refs/heads/main/results/bm-20251112-3.15.0a1%2B-f8a764a-JIT/bm-20251112-vultr-x86_64-Fidget%252dSpinner-tracing_jit-3.15.0a1%2B-f8a764a-vs-base.svg Stats: 50% more uops executed, 30% more traces entered the last time we ran them. It also suggests our trace lengths for a real trace recording JIT are too short, as a lot of trace too long aborts https://github.com/facebookexperimental/free-threading-benchmarking/blob/main/results/bm-20251023-3.15.0a1%2B-eb73378-CLANG%2CJIT/bm-20251023-vultr-x86_64-Fidget%252dSpinner-tracing_jit-3.15.0a1%2B-eb73378-pystats-vs-base.md . This new JIT frontend is already able to record/execute significantly more instructions than the previous JIT frontend. In this PR, we are now able to record through custom dunders, simple object creation, generators, etc. None of these were done by the old JIT frontend. Some custom dunders uops were discovered to be broken as part of this work gh-140277 The optimizer stack space check is disabled, as it's no longer valid to deal with underflow. Pros: * Ignoring the generated tracer code as it's automatically created, this is only additional 1k lines of code. The maintenance burden is handled by the DSL and code generator. * `optimizer.c` is now significantly simpler, as we don't have to do strange things to recover the bytecode from a trace. * The new JIT frontend is able to handle a lot more control-flow than the old one. * Tracing is very low overhead. We use the tail calling interpreter/computed goto interpreter to switch between tracing mode and non-tracing mode. I call this mechanism dual dispatch, as we have two dispatch tables dispatching to each other. Specialization is still enabled while tracing. * Better handling of polymorphism. We leverage the specializing interpreter for this. Cons: * (For now) requires tail calling interpreter or computed gotos. This means no Windows JIT for now :(. Not to fret, tail calling is coming soon to Windows though https://github.com/python/cpython/pull/139962 Design: * After each instruction, the `record_previous_inst` function/label is executed. This does as the name suggests. * The tracing interpreter lowers bytecode to uops directly so that it can obtain "fresh" values at the point of lowering. * The tracing version behaves nearly identical to the normal interpreter, in fact it even has specialization! This allows it to run without much of a slowdown when tracing. The actual cost of tracing is only a function call and writes to memory. * The tracing interpreter uses the specializing interpreter's deopt to naturally form the side exit chains. This allows it to side exit chain effectively, without repeating much code. We force a re-specializing when tracing a deopt. * The tracing interpreter can even handle goto errors/exceptions, but I chose to disable them for now as it's not tested. * Because we do not share interpreter dispatch, there is should be no significant slowdown to the original specializing interpreter on tailcall and computed got with JIT disabled. With JIT enabled, there might be a slowdown in the form of the JIT trying to trace. * Things that could have dynamic instruction pointer effects are guarded on. The guard deopts to a new instruction --- `_DYNAMIC_EXIT`.
2025-11-14 02:08:32 +08:00
#define OFFSET_OF_RETURN_GENERATOR ((frame->return_offset))
GH-140683: JIT: Improve machine code for loading smaller constants on AArch64. (GH-142511) * Use movz and movk instructions for loading 16 and 32 bit operands and oparg. * Loading of 64 bit operands is unchanged.
2025-12-11 12:33:39 +00:00
PyObject *ip = (PyObject *)CURRENT_OPERAND0_64();
gh-139109: A new tracing JIT compiler frontend for CPython (GH-140310) This PR changes the current JIT model from trace projection to trace recording. Benchmarking: better pyperformance (about 1.7% overall) geomean versus current https://raw.githubusercontent.com/facebookexperimental/free-threading-benchmarking/refs/heads/main/results/bm-20251108-3.15.0a1%2B-7e2bc1d-JIT/bm-20251108-vultr-x86_64-Fidget%252dSpinner-tracing_jit-3.15.0a1%2B-7e2bc1d-vs-base.svg, 100% faster Richards on the most improved benchmark versus the current JIT. Slowdown of about 10-15% on the worst benchmark versus the current JIT. **Note: the fastest version isn't the one merged, as it relies on fixing bugs in the specializing interpreter, which is left to another PR**. The speedup in the merged version is about 1.1%. https://raw.githubusercontent.com/facebookexperimental/free-threading-benchmarking/refs/heads/main/results/bm-20251112-3.15.0a1%2B-f8a764a-JIT/bm-20251112-vultr-x86_64-Fidget%252dSpinner-tracing_jit-3.15.0a1%2B-f8a764a-vs-base.svg Stats: 50% more uops executed, 30% more traces entered the last time we ran them. It also suggests our trace lengths for a real trace recording JIT are too short, as a lot of trace too long aborts https://github.com/facebookexperimental/free-threading-benchmarking/blob/main/results/bm-20251023-3.15.0a1%2B-eb73378-CLANG%2CJIT/bm-20251023-vultr-x86_64-Fidget%252dSpinner-tracing_jit-3.15.0a1%2B-eb73378-pystats-vs-base.md . This new JIT frontend is already able to record/execute significantly more instructions than the previous JIT frontend. In this PR, we are now able to record through custom dunders, simple object creation, generators, etc. None of these were done by the old JIT frontend. Some custom dunders uops were discovered to be broken as part of this work gh-140277 The optimizer stack space check is disabled, as it's no longer valid to deal with underflow. Pros: * Ignoring the generated tracer code as it's automatically created, this is only additional 1k lines of code. The maintenance burden is handled by the DSL and code generator. * `optimizer.c` is now significantly simpler, as we don't have to do strange things to recover the bytecode from a trace. * The new JIT frontend is able to handle a lot more control-flow than the old one. * Tracing is very low overhead. We use the tail calling interpreter/computed goto interpreter to switch between tracing mode and non-tracing mode. I call this mechanism dual dispatch, as we have two dispatch tables dispatching to each other. Specialization is still enabled while tracing. * Better handling of polymorphism. We leverage the specializing interpreter for this. Cons: * (For now) requires tail calling interpreter or computed gotos. This means no Windows JIT for now :(. Not to fret, tail calling is coming soon to Windows though https://github.com/python/cpython/pull/139962 Design: * After each instruction, the `record_previous_inst` function/label is executed. This does as the name suggests. * The tracing interpreter lowers bytecode to uops directly so that it can obtain "fresh" values at the point of lowering. * The tracing version behaves nearly identical to the normal interpreter, in fact it even has specialization! This allows it to run without much of a slowdown when tracing. The actual cost of tracing is only a function call and writes to memory. * The tracing interpreter uses the specializing interpreter's deopt to naturally form the side exit chains. This allows it to side exit chain effectively, without repeating much code. We force a re-specializing when tracing a deopt. * The tracing interpreter can even handle goto errors/exceptions, but I chose to disable them for now as it's not tested. * Because we do not share interpreter dispatch, there is should be no significant slowdown to the original specializing interpreter on tailcall and computed got with JIT disabled. With JIT enabled, there might be a slowdown in the form of the JIT trying to trace. * Things that could have dynamic instruction pointer effects are guarded on. The guard deopts to a new instruction --- `_DYNAMIC_EXIT`.
2025-11-14 02:08:32 +08:00
_Py_CODEUNIT *target = frame->instr_ptr + OFFSET_OF_RETURN_GENERATOR;
if (target != (_Py_CODEUNIT *)ip) {
frame->instr_ptr += OFFSET_OF_RETURN_GENERATOR;
if (true) {
UOP_STAT_INC(uopcode, miss);
GH-142718: JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. (GH-142762) JIT: Fix crash due to incorrect caching on side exits when exiting jitted code. * Make sure that stack is in correct state at side exits with TOS cached values * Simplify choice of cached items for side exits
2025-12-15 19:01:29 +00:00
_tos_cache2 = _stack_item_2;
_tos_cache1 = _stack_item_1;
_tos_cache0 = _stack_item_0;
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
SET_CURRENT_CACHED_VALUES(3);
gh-139109: A new tracing JIT compiler frontend for CPython (GH-140310) This PR changes the current JIT model from trace projection to trace recording. Benchmarking: better pyperformance (about 1.7% overall) geomean versus current https://raw.githubusercontent.com/facebookexperimental/free-threading-benchmarking/refs/heads/main/results/bm-20251108-3.15.0a1%2B-7e2bc1d-JIT/bm-20251108-vultr-x86_64-Fidget%252dSpinner-tracing_jit-3.15.0a1%2B-7e2bc1d-vs-base.svg, 100% faster Richards on the most improved benchmark versus the current JIT. Slowdown of about 10-15% on the worst benchmark versus the current JIT. **Note: the fastest version isn't the one merged, as it relies on fixing bugs in the specializing interpreter, which is left to another PR**. The speedup in the merged version is about 1.1%. https://raw.githubusercontent.com/facebookexperimental/free-threading-benchmarking/refs/heads/main/results/bm-20251112-3.15.0a1%2B-f8a764a-JIT/bm-20251112-vultr-x86_64-Fidget%252dSpinner-tracing_jit-3.15.0a1%2B-f8a764a-vs-base.svg Stats: 50% more uops executed, 30% more traces entered the last time we ran them. It also suggests our trace lengths for a real trace recording JIT are too short, as a lot of trace too long aborts https://github.com/facebookexperimental/free-threading-benchmarking/blob/main/results/bm-20251023-3.15.0a1%2B-eb73378-CLANG%2CJIT/bm-20251023-vultr-x86_64-Fidget%252dSpinner-tracing_jit-3.15.0a1%2B-eb73378-pystats-vs-base.md . This new JIT frontend is already able to record/execute significantly more instructions than the previous JIT frontend. In this PR, we are now able to record through custom dunders, simple object creation, generators, etc. None of these were done by the old JIT frontend. Some custom dunders uops were discovered to be broken as part of this work gh-140277 The optimizer stack space check is disabled, as it's no longer valid to deal with underflow. Pros: * Ignoring the generated tracer code as it's automatically created, this is only additional 1k lines of code. The maintenance burden is handled by the DSL and code generator. * `optimizer.c` is now significantly simpler, as we don't have to do strange things to recover the bytecode from a trace. * The new JIT frontend is able to handle a lot more control-flow than the old one. * Tracing is very low overhead. We use the tail calling interpreter/computed goto interpreter to switch between tracing mode and non-tracing mode. I call this mechanism dual dispatch, as we have two dispatch tables dispatching to each other. Specialization is still enabled while tracing. * Better handling of polymorphism. We leverage the specializing interpreter for this. Cons: * (For now) requires tail calling interpreter or computed gotos. This means no Windows JIT for now :(. Not to fret, tail calling is coming soon to Windows though https://github.com/python/cpython/pull/139962 Design: * After each instruction, the `record_previous_inst` function/label is executed. This does as the name suggests. * The tracing interpreter lowers bytecode to uops directly so that it can obtain "fresh" values at the point of lowering. * The tracing version behaves nearly identical to the normal interpreter, in fact it even has specialization! This allows it to run without much of a slowdown when tracing. The actual cost of tracing is only a function call and writes to memory. * The tracing interpreter uses the specializing interpreter's deopt to naturally form the side exit chains. This allows it to side exit chain effectively, without repeating much code. We force a re-specializing when tracing a deopt. * The tracing interpreter can even handle goto errors/exceptions, but I chose to disable them for now as it's not tested. * Because we do not share interpreter dispatch, there is should be no significant slowdown to the original specializing interpreter on tailcall and computed got with JIT disabled. With JIT enabled, there might be a slowdown in the form of the JIT trying to trace. * Things that could have dynamic instruction pointer effects are guarded on. The guard deopts to a new instruction --- `_DYNAMIC_EXIT`.
2025-11-14 02:08:32 +08:00
JUMP_TO_JUMP_TARGET();
}
}
GH-135379: Top of stack caching for the JIT. (GH-135465) Uses three registers to cache values at the top of the evaluation stack This significantly reduces memory traffic for smaller, more common uops.
2025-12-11 10:32:52 +00:00
_tos_cache2 = _stack_item_2;
_tos_cache1 = _stack_item_1;
_tos_cache0 = _stack_item_0;
SET_CURRENT_CACHED_VALUES(3);
gh-139109: A new tracing JIT compiler frontend for CPython (GH-140310) This PR changes the current JIT model from trace projection to trace recording. Benchmarking: better pyperformance (about 1.7% overall) geomean versus current https://raw.githubusercontent.com/facebookexperimental/free-threading-benchmarking/refs/heads/main/results/bm-20251108-3.15.0a1%2B-7e2bc1d-JIT/bm-20251108-vultr-x86_64-Fidget%252dSpinner-tracing_jit-3.15.0a1%2B-7e2bc1d-vs-base.svg, 100% faster Richards on the most improved benchmark versus the current JIT. Slowdown of about 10-15% on the worst benchmark versus the current JIT. **Note: the fastest version isn't the one merged, as it relies on fixing bugs in the specializing interpreter, which is left to another PR**. The speedup in the merged version is about 1.1%. https://raw.githubusercontent.com/facebookexperimental/free-threading-benchmarking/refs/heads/main/results/bm-20251112-3.15.0a1%2B-f8a764a-JIT/bm-20251112-vultr-x86_64-Fidget%252dSpinner-tracing_jit-3.15.0a1%2B-f8a764a-vs-base.svg Stats: 50% more uops executed, 30% more traces entered the last time we ran them. It also suggests our trace lengths for a real trace recording JIT are too short, as a lot of trace too long aborts https://github.com/facebookexperimental/free-threading-benchmarking/blob/main/results/bm-20251023-3.15.0a1%2B-eb73378-CLANG%2CJIT/bm-20251023-vultr-x86_64-Fidget%252dSpinner-tracing_jit-3.15.0a1%2B-eb73378-pystats-vs-base.md . This new JIT frontend is already able to record/execute significantly more instructions than the previous JIT frontend. In this PR, we are now able to record through custom dunders, simple object creation, generators, etc. None of these were done by the old JIT frontend. Some custom dunders uops were discovered to be broken as part of this work gh-140277 The optimizer stack space check is disabled, as it's no longer valid to deal with underflow. Pros: * Ignoring the generated tracer code as it's automatically created, this is only additional 1k lines of code. The maintenance burden is handled by the DSL and code generator. * `optimizer.c` is now significantly simpler, as we don't have to do strange things to recover the bytecode from a trace. * The new JIT frontend is able to handle a lot more control-flow than the old one. * Tracing is very low overhead. We use the tail calling interpreter/computed goto interpreter to switch between tracing mode and non-tracing mode. I call this mechanism dual dispatch, as we have two dispatch tables dispatching to each other. Specialization is still enabled while tracing. * Better handling of polymorphism. We leverage the specializing interpreter for this. Cons: * (For now) requires tail calling interpreter or computed gotos. This means no Windows JIT for now :(. Not to fret, tail calling is coming soon to Windows though https://github.com/python/cpython/pull/139962 Design: * After each instruction, the `record_previous_inst` function/label is executed. This does as the name suggests. * The tracing interpreter lowers bytecode to uops directly so that it can obtain "fresh" values at the point of lowering. * The tracing version behaves nearly identical to the normal interpreter, in fact it even has specialization! This allows it to run without much of a slowdown when tracing. The actual cost of tracing is only a function call and writes to memory. * The tracing interpreter uses the specializing interpreter's deopt to naturally form the side exit chains. This allows it to side exit chain effectively, without repeating much code. We force a re-specializing when tracing a deopt. * The tracing interpreter can even handle goto errors/exceptions, but I chose to disable them for now as it's not tested. * Because we do not share interpreter dispatch, there is should be no significant slowdown to the original specializing interpreter on tailcall and computed got with JIT disabled. With JIT enabled, there might be a slowdown in the form of the JIT trying to trace. * Things that could have dynamic instruction pointer effects are guarded on. The guard deopts to a new instruction --- `_DYNAMIC_EXIT`.
2025-11-14 02:08:32 +08:00
#undef OFFSET_OF_RETURN_GENERATOR
gh-134584: Remove custom float decref ops (GH-142576)
2025-12-16 03:38:58 +08:00
assert(WITHIN_STACK_BOUNDS_IGNORING_CACHE());
gh-139109: A new tracing JIT compiler frontend for CPython (GH-140310) This PR changes the current JIT model from trace projection to trace recording. Benchmarking: better pyperformance (about 1.7% overall) geomean versus current https://raw.githubusercontent.com/facebookexperimental/free-threading-benchmarking/refs/heads/main/results/bm-20251108-3.15.0a1%2B-7e2bc1d-JIT/bm-20251108-vultr-x86_64-Fidget%252dSpinner-tracing_jit-3.15.0a1%2B-7e2bc1d-vs-base.svg, 100% faster Richards on the most improved benchmark versus the current JIT. Slowdown of about 10-15% on the worst benchmark versus the current JIT. **Note: the fastest version isn't the one merged, as it relies on fixing bugs in the specializing interpreter, which is left to another PR**. The speedup in the merged version is about 1.1%. https://raw.githubusercontent.com/facebookexperimental/free-threading-benchmarking/refs/heads/main/results/bm-20251112-3.15.0a1%2B-f8a764a-JIT/bm-20251112-vultr-x86_64-Fidget%252dSpinner-tracing_jit-3.15.0a1%2B-f8a764a-vs-base.svg Stats: 50% more uops executed, 30% more traces entered the last time we ran them. It also suggests our trace lengths for a real trace recording JIT are too short, as a lot of trace too long aborts https://github.com/facebookexperimental/free-threading-benchmarking/blob/main/results/bm-20251023-3.15.0a1%2B-eb73378-CLANG%2CJIT/bm-20251023-vultr-x86_64-Fidget%252dSpinner-tracing_jit-3.15.0a1%2B-eb73378-pystats-vs-base.md . This new JIT frontend is already able to record/execute significantly more instructions than the previous JIT frontend. In this PR, we are now able to record through custom dunders, simple object creation, generators, etc. None of these were done by the old JIT frontend. Some custom dunders uops were discovered to be broken as part of this work gh-140277 The optimizer stack space check is disabled, as it's no longer valid to deal with underflow. Pros: * Ignoring the generated tracer code as it's automatically created, this is only additional 1k lines of code. The maintenance burden is handled by the DSL and code generator. * `optimizer.c` is now significantly simpler, as we don't have to do strange things to recover the bytecode from a trace. * The new JIT frontend is able to handle a lot more control-flow than the old one. * Tracing is very low overhead. We use the tail calling interpreter/computed goto interpreter to switch between tracing mode and non-tracing mode. I call this mechanism dual dispatch, as we have two dispatch tables dispatching to each other. Specialization is still enabled while tracing. * Better handling of polymorphism. We leverage the specializing interpreter for this. Cons: * (For now) requires tail calling interpreter or computed gotos. This means no Windows JIT for now :(. Not to fret, tail calling is coming soon to Windows though https://github.com/python/cpython/pull/139962 Design: * After each instruction, the `record_previous_inst` function/label is executed. This does as the name suggests. * The tracing interpreter lowers bytecode to uops directly so that it can obtain "fresh" values at the point of lowering. * The tracing version behaves nearly identical to the normal interpreter, in fact it even has specialization! This allows it to run without much of a slowdown when tracing. The actual cost of tracing is only a function call and writes to memory. * The tracing interpreter uses the specializing interpreter's deopt to naturally form the side exit chains. This allows it to side exit chain effectively, without repeating much code. We force a re-specializing when tracing a deopt. * The tracing interpreter can even handle goto errors/exceptions, but I chose to disable them for now as it's not tested. * Because we do not share interpreter dispatch, there is should be no significant slowdown to the original specializing interpreter on tailcall and computed got with JIT disabled. With JIT enabled, there might be a slowdown in the form of the JIT trying to trace. * Things that could have dynamic instruction pointer effects are guarded on. The guard deopts to a new instruction --- `_DYNAMIC_EXIT`.
2025-11-14 02:08:32 +08:00
break;
}
GH-139109: Support switch/case dispatch with the tracing interpreter. (GH-141703)
2025-11-18 13:31:48 +00:00
/* _TRACE_RECORD is not a viable micro-op for tier 2 because it uses the 'this_instr' variable */
gh-139109: A new tracing JIT compiler frontend for CPython (GH-140310) This PR changes the current JIT model from trace projection to trace recording. Benchmarking: better pyperformance (about 1.7% overall) geomean versus current https://raw.githubusercontent.com/facebookexperimental/free-threading-benchmarking/refs/heads/main/results/bm-20251108-3.15.0a1%2B-7e2bc1d-JIT/bm-20251108-vultr-x86_64-Fidget%252dSpinner-tracing_jit-3.15.0a1%2B-7e2bc1d-vs-base.svg, 100% faster Richards on the most improved benchmark versus the current JIT. Slowdown of about 10-15% on the worst benchmark versus the current JIT. **Note: the fastest version isn't the one merged, as it relies on fixing bugs in the specializing interpreter, which is left to another PR**. The speedup in the merged version is about 1.1%. https://raw.githubusercontent.com/facebookexperimental/free-threading-benchmarking/refs/heads/main/results/bm-20251112-3.15.0a1%2B-f8a764a-JIT/bm-20251112-vultr-x86_64-Fidget%252dSpinner-tracing_jit-3.15.0a1%2B-f8a764a-vs-base.svg Stats: 50% more uops executed, 30% more traces entered the last time we ran them. It also suggests our trace lengths for a real trace recording JIT are too short, as a lot of trace too long aborts https://github.com/facebookexperimental/free-threading-benchmarking/blob/main/results/bm-20251023-3.15.0a1%2B-eb73378-CLANG%2CJIT/bm-20251023-vultr-x86_64-Fidget%252dSpinner-tracing_jit-3.15.0a1%2B-eb73378-pystats-vs-base.md . This new JIT frontend is already able to record/execute significantly more instructions than the previous JIT frontend. In this PR, we are now able to record through custom dunders, simple object creation, generators, etc. None of these were done by the old JIT frontend. Some custom dunders uops were discovered to be broken as part of this work gh-140277 The optimizer stack space check is disabled, as it's no longer valid to deal with underflow. Pros: * Ignoring the generated tracer code as it's automatically created, this is only additional 1k lines of code. The maintenance burden is handled by the DSL and code generator. * `optimizer.c` is now significantly simpler, as we don't have to do strange things to recover the bytecode from a trace. * The new JIT frontend is able to handle a lot more control-flow than the old one. * Tracing is very low overhead. We use the tail calling interpreter/computed goto interpreter to switch between tracing mode and non-tracing mode. I call this mechanism dual dispatch, as we have two dispatch tables dispatching to each other. Specialization is still enabled while tracing. * Better handling of polymorphism. We leverage the specializing interpreter for this. Cons: * (For now) requires tail calling interpreter or computed gotos. This means no Windows JIT for now :(. Not to fret, tail calling is coming soon to Windows though https://github.com/python/cpython/pull/139962 Design: * After each instruction, the `record_previous_inst` function/label is executed. This does as the name suggests. * The tracing interpreter lowers bytecode to uops directly so that it can obtain "fresh" values at the point of lowering. * The tracing version behaves nearly identical to the normal interpreter, in fact it even has specialization! This allows it to run without much of a slowdown when tracing. The actual cost of tracing is only a function call and writes to memory. * The tracing interpreter uses the specializing interpreter's deopt to naturally form the side exit chains. This allows it to side exit chain effectively, without repeating much code. We force a re-specializing when tracing a deopt. * The tracing interpreter can even handle goto errors/exceptions, but I chose to disable them for now as it's not tested. * Because we do not share interpreter dispatch, there is should be no significant slowdown to the original specializing interpreter on tailcall and computed got with JIT disabled. With JIT enabled, there might be a slowdown in the form of the JIT trying to trace. * Things that could have dynamic instruction pointer effects are guarded on. The guard deopts to a new instruction --- `_DYNAMIC_EXIT`.
2025-11-14 02:08:32 +08:00
gh-111520: Integrate the Tier 2 interpreter in the Tier 1 interpreter (#111428) - There is no longer a separate Python/executor.c file. - Conventions in Python/bytecodes.c are slightly different -- don't use `goto error`, you must use `GOTO_ERROR(error)` (same for others like `unused_local_error`). - The `TIER_ONE` and `TIER_TWO` symbols are only valid in the generated (.c.h) files. - In Lib/test/support/__init__.py, `Py_C_RECURSION_LIMIT` is imported from `_testcapi`. - On Windows, in debug mode, stack allocation grows from 8MiB to 12MiB. - **Beware!** This changes the env vars to enable uops and their debugging to `PYTHON_UOPS` and `PYTHON_LLTRACE`.
2023-11-01 13:13:02 -07:00
#undef TIER_TWO
Reference in a new issue Copy permalink