gh-110167: Fix test_socket deadlock in doCleanups() (GH-110416)
Fix a deadlock in test_socket when server fails with a timeout but
the client is still running in its thread. Don't hold a lock to call
cleanup functions in doCleanups(). One of the cleanup function waits
until the client completes, whereas the client could deadlock if it
called addCleanup() in such situation.
doCleanups() is called when the server completed, but the client can
still be running in its thread especially if the server failed with a
timeout. Don't put a lock on doCleanups() to prevent deadlock between
addCleanup() called in the client and doCleanups() waiting for
self.done.wait of ThreadableTest._setUp().
(cherry picked from commit 318f5df271)
Co-authored-by: Victor Stinner <vstinner@python.org>
gh-110391: socket NetworkConnectionAttributesTest always declare cli (GH-110401)
NetworkConnectionAttributesTest of test_socket now always declare the
'cli' attribute, so clientTearDown() cannot fail with AttributeError.
(cherry picked from commit e37d4557c3)
Co-authored-by: Victor Stinner <vstinner@python.org>
gh-110259: Fix f-strings with multiline expressions and format specs (GH-110271)
(cherry picked from commit cc389ef627)
Signed-off-by: Pablo Galindo <pablogsal@gmail.com>
Co-authored-by: Pablo Galindo Salgado <Pablogsal@gmail.com>
Add back bltin-boolean-values ref tag (GH-110371)
To avoid breaking downstream intersphinx via numpydoc
(cherry picked from commit f7860295b1)
Co-authored-by: P. L. Lim <2090236+pllim@users.noreply.github.com>
gh-85984: Document change in return type of tty functions (GH-110028)
(cherry picked from commit f02f26e293)
Co-authored-by: Jelle Zijlstra <jelle.zijlstra@gmail.com>
Enhance TypedDict docs around required/optional keys (GH-109547)
As discussed in comments to GH-109544, the semantics of this attribute
are somewhat confusing. Add a note explaining its limitations and
steering users towards __required_keys__ and __optional_keys__ instead.
(cherry picked from commit f49958c886)
Co-authored-by: Jelle Zijlstra <jelle.zijlstra@gmail.com>
* gh-109972: Enhance test_gdb (#110026)
* Split test_pycfunction.py: add test_cfunction_full.py.
Split the function into the following 6 functions. In verbose
mode, these "pycfunction" tests now log each tested call.
* test_pycfunction_noargs()
* test_pycfunction_o()
* test_pycfunction_varargs()
* test_pycfunction_varargs_keywords()
* test_pycfunction_fastcall()
* test_pycfunction_fastcall_keywords()
* Move get_gdb_repr() to PrettyPrintTests.
* Replace DebuggerTests.get_sample_script() with SAMPLE_SCRIPT.
* Rename checkout_hook_path to CHECKOUT_HOOK_PATH.
* Rename gdb_version to GDB_VERSION_TEXT.
* Replace (gdb_major_version, gdb_minor_version) with GDB_VERSION.
* run_gdb() uses "backslashreplace" error handler instead of "replace".
* Add check_gdb() function to util.py.
* Enhance support.check_cflags_pgo(): check also for sysconfig
PGO_PROF_USE_FLAG (if available) in compiler flags.
* Move some SkipTest checks to test_gdb/__init__.py.
* Elaborate why gdb cannot be tested on Windows: gdb doesn't support
PDB debug symbol files.
(cherry picked from commit 757cbd4f29)
* gh-104736: Fix test_gdb tests on ppc64le with clang (#109360)
Fix test_gdb on Python built with LLVM clang 16 on Linux ppc64le (ex:
Fedora 38). Search patterns in gdb "bt" command output to detect
when gdb fails to retrieve the traceback. For example, skip a test if
"Backtrace stopped: frame did not save the PC" is found.
(cherry picked from commit 44d9a71ea2)
* gh-110166: Fix gdb CFunctionFullTests on ppc64le clang build (#110331)
CFunctionFullTests now also runs "bt" command before "py-bt-full",
similar to CFunctionTests which also runs "bt" command before
"py-bt". So test_gdb can skip the test if patterns like "?? ()" are
found in the gdb output.
(cherry picked from commit 1de9406f91)
* gh-109974: Fix threading lock_tests race conditions (#110057)
Fix race conditions in test_threading lock tests. Wait until a
condition is met rather than using time.sleep() with a hardcoded
number of seconds.
* Replace sleeping loops with support.sleeping_retry() which raises
an exception on timeout.
* Add wait_threads_blocked(nthread) which computes a sleep depending
on the number of threads. Remove _wait() function.
* test_set_and_clear(): use a way longer Event.wait() timeout.
* BarrierTests.test_repr(): wait until the 2 threads are waiting for
the barrier. Use a way longer timeout for Barrier.wait() timeout.
* test_thread_leak() no longer needs to count
len(threading.enumerate()): Bunch uses
threading_helper.wait_threads_exit() internally which does it in
wait_for_finished().
* Add BaseLockTests.wait_phase() which implements a timeout.
test_reacquire() and test_recursion_count() use wait_phase().
(cherry picked from commit 4e356ad183)
* gh-109974: Fix more threading lock_tests race conditions (#110089)
* Add context manager on Bunch class.
* Bunch now catchs exceptions on executed functions and re-raise them
at __exit__() as an ExceptionGroup.
* Rewrite BarrierProxy.test_default_timeout(). Use a single thread.
Only check that barrier.wait() blocks for at least default timeout
seconds.
* test_with(): inline _with() function.
(cherry picked from commit 743e3572ee)
* gh-108987: Fix _thread.start_new_thread() race condition (#109135)
Fix _thread.start_new_thread() race condition. If a thread is created
during Python finalization, the newly spawned thread now exits
immediately instead of trying to access freed memory and lead to a
crash.
thread_run() calls PyEval_AcquireThread() which checks if the thread
must exit. The problem was that tstate was dereferenced earlier in
_PyThreadState_Bind() which leads to a crash most of the time.
Move _PyThreadState_CheckConsistency() from thread_run() to
_PyThreadState_Bind().
(cherry picked from commit 517cd82ea7)
* gh-109795: `_thread.start_new_thread`: allocate thread bootstate using raw memory allocator (#109808)
(cherry picked from commit 1b8f2366b3)
---------
Co-authored-by: Radislav Chugunov <52372310+chgnrdv@users.noreply.github.com>
gh-110335: asyncio test_unix_events cleans multiprocessing (GH-110336)
test_unix_events tests using the multiprocessing module now call
multiprocessing.util._cleanup_tests().
(cherry picked from commit 1337765225)
Co-authored-by: Victor Stinner <vstinner@python.org>
* gh-109615: Fix support test_copy_python_src_ignore() (#109958)
Fix the test when run on an installed Python: use "abs_srcdir" of
sysconfig, and skip the test if the Python source code cannot be
found.
* Tools/patchcheck/patchcheck.py, Tools/freeze/test/freeze.py and
Lib/test/libregrtest/utils.py now first try to get "abs_srcdir"
from sysconfig, before getting "srcdir" from sysconfig.
* test.pythoninfo logs sysconfig "abs_srcdir".
(cherry picked from commit b89ed9df39)
* gh-109615: Fix support test_copy_python_src_ignore() on WASM (#109970)
Not only check if src_dir exists, but look also for Lib/os.py
landmark.
(cherry picked from commit cc54bcf17b)
* gh-109615: Look for 'Modules' as landmark for test_copy_python_src_ignore (GH-110108)
(cherry picked from commit 20bc5f7c28)
* gh-109748: Fix again venv test_zippath_from_non_installed_posix() (#110149)
Call also copy_python_src_ignore() on listdir() names.
shutil.copytree(): replace set() with an empty tuple. An empty tuple
becomes a constant in the compiler and checking if an item is in an
empty tuple is cheap.
(cherry picked from commit 0def8c712b)
---------
Co-authored-by: Steve Dower <steve.dower@python.org>
gh-109972: Split test_gdb.py into test_gdb package (#109977)
Split test_gdb.py file into a test_gdb package made of multiple
tests, so tests can now be run in parallel.
* Create Lib/test/test_gdb/ directory.
* Split test_gdb.py into multiple files in Lib/test/test_gdb/
directory.
* Move Lib/test/gdb_sample.py to Lib/test/test_gdb/ directory.
Update get_sample_script(): use __file__ to locate gdb_sample.py.
* Move gdb_has_frame_select() and HAS_PYUP_PYDOWN to test_misc.py.
* Explicitly skip test_gdb on Windows. Previously, test_gdb was
skipped even if gdb was available because of
gdb_has_frame_select().
(cherry picked from commit 8f324b7ecd)
gh-109917: Fix test instability in test_concurrent_futures (GH-110306)
The test had an instability issue due to the ordering of the dummy
queue operation and the real wakeup pipe operations. Both primitives
are thread safe but not done atomically as a single update and may
interleave arbitrarily. With the old order of operations this can lead
to an incorrect state where the dummy queue is full but the wakeup
pipe is empty. By swapping the order in clear() I think this can no
longer happen in any possible operation interleaving (famous last
words).
(cherry picked from commit a376a72bd9)
Co-authored-by: elfstrom <elfstrom@users.noreply.github.com>
Bump various dependencies in `Doc/requirements-oldest-sphinx.txt` (GH-110278)
This resolves a Dependabot security alert on the repository for urllib3==2.0.4.
(cherry picked from commit f1663a492e)
Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com>
gh-109653: Fix regression in the import time of `random` in Python 3.12 (GH-110221)
(cherry picked from commit 21a6263020)
Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com>
gh-108494: Document how to add a project in PCbuild/readme.txt (GH-110077)
(cherry picked from commit 6387b5313c)
Co-authored-by: Victor Stinner <vstinner@python.org>
gh-110088, gh-109878: Fix test_asyncio timeouts (#110092)
Fix test_asyncio timeouts: don't measure the maximum duration, a test
should not measure a CI performance. Only measure the minimum
duration when a task has a timeout or delay. Add CLOCK_RES to
test_asyncio.utils.
(cherry picked from commit db0a258e79)
gh-110031: Skip test_threading fork tests if ASAN (#110100)
Skip test_threading tests using thread+fork if Python is built with
Address Sanitizer (ASAN).
(cherry picked from commit 86e76ab8af)
gh-110178: Use fewer weakrefs in test_typing.py (GH-110194)
Confirmed that without the C changes from GH-108517, this test still segfaults with only 10 weakrefs.
(cherry picked from commit 732ad44cec)
Co-authored-by: Jelle Zijlstra <jelle.zijlstra@gmail.com>
gh-108963: using random to generate unique string in sys.intern test (#109491)
(cherry picked from commit 44b1e4ea48)
Co-authored-by: AN Long <aisk@users.noreply.github.com>
gh-109590: Update shutil.which on Windows to prefer a PATHEXT extension on executable files (GH-109995)
The default arguments for shutil.which() request an executable file, but extensionless files are not executable on Windows and should be ignored.
(cherry picked from commit 29b875bb93)
Co-authored-by: Charles Machalow <csm10495@gmail.com>
* gh-109649: Enhance os.cpu_count() documentation
* Doc: Specify that os.cpu_count() counts *logicial* CPUs.
* Doc: Specify that os.sched_getaffinity(0) is related to the calling
thread.
* Fix test_posix.test_sched_getaffinity(): restore the old CPU mask
when the test completes!
* Restore removed text
gh-110088: Fix asyncio test_prompt_cancellation() (GH-110157)
Don't measure the CI performance: don't test the maximum elapsed
time. The check failed on a slow CI.
(cherry picked from commit c62b49ecc8)
Co-authored-by: Victor Stinner <vstinner@python.org>
Add example for linear_regression() with proportional=True. (gh-110133)
(cherry picked from commit 613c0d4e86)
Co-authored-by: Raymond Hettinger <rhettinger@users.noreply.github.com>
gh-109047: concurrent.futures catches PythonFinalizationError (#109810)
concurrent.futures: The *executor manager thread* now catches
exceptions when adding an item to the *call queue*. During Python
finalization, creating a new thread can now raise RuntimeError. Catch
the exception and call terminate_broken() in this case.
Add test_python_finalization_error() to test_concurrent_futures.
concurrent.futures._ExecutorManagerThread changes:
* terminate_broken() no longer calls shutdown_workers() since the
call queue is no longer working anymore (read and write ends of
the queue pipe are closed).
* terminate_broken() now terminates child processes, not only
wait until they complete.
* _ExecutorManagerThread.terminate_broken() now holds shutdown_lock
to prevent race conditons with ProcessPoolExecutor.submit().
multiprocessing.Queue changes:
* Add _terminate_broken() method.
* _start_thread() sets _thread to None on exception to prevent
leaking "dangling threads" even if the thread was not started
yet.
(cherry picked from commit 6351842121)
gh-109592: test_eintr tolerates 20 ms when comparing timings (GH-110102)
(cherry picked from commit 9c73a9acec)
Co-authored-by: Victor Stinner <vstinner@python.org>
gh-110052: Fix faulthandler for freed tstate (#110069)
faulthandler now detected freed interp and freed tstate, and no
longer dereference them.
(cherry picked from commit 2e37a38bcb)
gh-110036: multiprocessing Popen.terminate() catches PermissionError (GH-110037)
On Windows, multiprocessing Popen.terminate() now catchs
PermissionError and get the process exit code. If the process is
still running, raise again the PermissionError. Otherwise, the
process terminated as expected: store its exit code.
(cherry picked from commit bd4518c60c)
Co-authored-by: Victor Stinner <vstinner@python.org>
gh-109960: Remove test_pty timeout of 10 seconds (GH-110058)
In 2003, test_pty got a hardcoded timeout of 10 seconds to prevent
hanging on AIX & HPUX "if run after test_openpty":
commit 7d8145268e. Since 2003, test_pty
was no longer reported to hang on AIX. But today, the test can fail
simply because a CI is busy running other tests in parallel.
The timeout of 10 seconds is no longer needed, just remove it.
Moreover, regrtest now has multiple built-in generic timeout
mecanisms.
(cherry picked from commit 5fdcea7440)
Co-authored-by: Victor Stinner <vstinner@python.org>
gh-109889: fix compiler's redundant NOP detection to look past NOPs with no lineno when looking for the next instruction's lineno (GH-109987)
(cherry picked from commit f580edcc6a)
Co-authored-by: Irit Katriel <1055913+iritkatriel@users.noreply.github.com>
