gh-135335: flush stdout/stderr in forkserver after preloading modules (GH-135338)
If a preloaded module writes to stdout or stderr, and the stream is buffered,
child processes will inherit the buffered data after forking. Attempt to
prevent this by flushing the streams after preload.
(cherry picked from commit 9877d191f4)
Co-authored-by: Duane Griffin <duaneg@dghda.com>
Co-authored-by: Mikhail Efimov <efimov.mikhail@gmail.com>
Co-authored-by: Victor Stinner <vstinner@python.org>
Use `ma_used` instead of `ma_keys->dk_nentries` for modification check
so that we only check if the dictionary is modified, not if new keys are
added to a different dictionary that shared the same keys object.
(cherry picked from commit d8994b0a77)
gh-132969: Fix error/hang when shutdown(wait=False) and task exited abnormally (GH-133222)
When shutdown is called with wait=False, the executor thread keeps running
even after the ProcessPoolExecutor's state is reset. The executor then tries
to replenish the worker processes pool resulting in an error and a potential hang
when it comes across a worker that has died. Fixed the issue by having
_adjust_process_count() return without doing anything if the ProcessPoolExecutor's
state has been reset.
Added unit tests to validate two scenarios:
max_workers < num_tasks (exception)
max_workers > num_tasks (exception + hang)
(cherry picked from commit 598aa7cc98)
Co-authored-by: Ajay Kamdar <140011370+ogbiggles@users.noreply.github.com>
gh-67022: Document bytes/str inconsistency in email.header.decode_header() and suggest email.headerregistry.HeaderRegistry as a sane alternative (GH-92900)
* gh-67022: Document bytes/str inconsistency in email.header.decode_header()
This function's possible return types have been surprising and error-prone
for the entirety of its Python 3.x history. It can return either:
1. `typing.List[typing.Tuple[bytes, typing.Optional[str]]]` of length >1
2. or `typing.List[typing.Tuple[str, None]]`, of length exactly 1
This means that any user of this function must be prepared to accept either
`bytes` or `str` for the first member of the 2-tuples it returns, which is a
very surprising behavior in Python 3.x, particularly given that the second
member of the tuple is supposed to represent the charset/encoding of the
first member.
This patch documents the behavior of this function, and adds test cases
to demonstrate it.
As discussed in bpo-22833, this cannot be changed in a backwards-compatible
way, and some users of this function depend precisely on the existing
behavior.
Add warnings about obsolescence of 'email.header.decode_header' and 'email.header.make_header' functions.
Recommend use of `email.headerregistry.HeaderRegistry` instead, as suggested
in https://github.com/python/cpython/pull/92900#discussion_r1112472177
(cherry picked from commit 60181f4ed0)
Co-authored-by: Dan Lenski <dlenski@gmail.com>
End-of-file errors are now handled according to the HTML5 specs --
comments and declarations are automatically closed, tags are ignored.
(cherry picked from commit 6eb6c5dbfb)
Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>
* gh-135276: Refresh `zipfile.Path` from zipp 3.23 (GH-135277)
Apply changes from zipp 3.23
(cherry picked from commit 8d6eb0c262)
Co-authored-by: Jason R. Coombs <jaraco@jaraco.com>
* Removed features slated for Python 3.15 only.
---------
Co-authored-by: Jason R. Coombs <jaraco@jaraco.com>
gh-134155: fix AttributeError in email._header_value_parser.get_address (GH-134194)
Append the defect to defects instead of to the parse tree.
(cherry picked from commit d9cad074d5)
Co-authored-by: Sergey Miryanov <sergey.miryanov@gmail.com>
Co-authored-by: Bénédikt Tran <10796600+picnixz@users.noreply.github.com>
Co-authored-by: Hugo van Kemenade <1324225+hugovk@users.noreply.github.com>
gh-135124: Change stdout errors in regrtest worker process (GH-135138)
Set sys.stdout encoder error handler to backslashreplace in regrtest
workers to avoid UnicodeEncodeError when printing a traceback
or any other non-encodable character.
Move the code from the Regrtest class to setup_process().
Call setup_process() earlier, before displaying regrtest headers.
(cherry picked from commit 3d396ab759)
Co-authored-by: Victor Stinner <vstinner@python.org>
The `test_ssl_in_multiple_threads` test failed because `test_check_hostname_idn()`
modified the global warnings filters via `warnings_helper.check_no_resource_warning()`.
Disable the warnings check in the multi-threaded test because `warnings_helper` isn't
thread-safe in 3.13 or earlier.
(cherry picked from commit 40c8be0008)
Co-authored-by: Thomas Grainger <tagrain@gmail.com>
* Fix for 3.13
---------
Co-authored-by: Thomas Grainger <tagrain@gmail.com>
Co-authored-by: Sam Gross <colesbury@gmail.com>
50 producer and 50 consumer threads need more than 5GB of memory.
(cherry picked from commit 7a79f52d83)
Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>
random.getrandbits() can now generate more that 2**31 bits.
random.randbytes() can now generate more that 256 MiB.
(cherry picked from commit 68784fed78)
The `textiowrapper_iternext` function called `_textiowrapper_writeflush`, but did not
use a critical section, making it racy in free-threaded builds.
(cherry picked from commit 44fb7c361c)
Co-authored-by: Duane Griffin <duaneg@dghda.com>
OpenSSL and HACL*-based hash functions constructors now support both `data` and `string` parameters.
Previously these constructor functions inconsistently supported sometimes `data` and sometimes `string`,
while the documentation expected `data` to be given in all cases.
(cherry picked from commit c6e63d9d35)
(cherry picked from commit 379d0bc956)
gh-134954: Hard-cap max file descriptors in subprocess test fd_status (GH-134955)
* Hard-cap max file descriptors in subprocess test fd_status
On some systems, `SC_OPEN_MAX` may return a very large value (i.e. 10**30), leading to the subprocess test timing out (or run forever).
Prevent this situation by applying a hard cap on how many file descriptors are checked.
* Fix typo in usage docstring
s/fd_stats/fd_status/
(cherry picked from commit f58873e4b2)
Co-authored-by: Itamar Oren <itamarost@gmail.com>
gh-133711: Fix test_readline.test_nonascii() for UTF-8 Mode (GH-134841)
Skip the test if the Python UTF-8 Mode is enabled and the LC_CTYPE
encoding is not UTF-8.
(cherry picked from commit 4635115c3f)
Co-authored-by: Victor Stinner <vstinner@python.org>
gh-133711: Fix test_regrtest for PYTHONUTF8=1 (GH-134839)
Use "backslashreplace" error handler to decode stdout and stderr.
Example:
vstinner@WIN C:\victor\python\main\build\test_python_worker_8360\x91>
"C:\victor\python\main\PCbuild\amd64\python_d.exe" -m test
--fast-ci --slow-ci --testdir
C:\Users\vstinner\AppData\Local\Temp\tmp0t59e8da
test_regrtest_noop1 test_regrtest_noop2 test_regrtest_noop3
test_regrtest_noop4
Notice the "\x91" byte at the end of the first line: it's the
non-ASCII U+00E6 character encoded to the OEM cp437 code page.
(cherry picked from commit 91618278e7)
Co-authored-by: Victor Stinner <vstinner@python.org>
* gh-132876: workaround broken ldexp() on Windows 10
ldexp() fails to round subnormal results before Windows 11,
so hide their bug.
(cherry picked from commit cf8941c603)
Co-authored-by: Tim Peters <tim.peters@gmail.com>
gh-128840: Limit the number of parts in IPv6 address parsing (GH-128841)
GH-128840: Limit the number of parts in IPv6 address parsing
Limit length of IP address string to 39
---------
(cherry picked from commit 47f1161d3a)
Co-authored-by: Seth Michael Larson <seth@python.org>
Co-authored-by: Hugo van Kemenade <1324225+hugovk@users.noreply.github.com>
Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>
Co-authored-by: Gregory P. Smith <greg@krypto.org>
gh-131357: Add some extra tests for empty bytes and bytearray (GH-134458)
(cherry picked from commit 7309eb60c0)
Co-authored-by: Sergey Miryanov <sergey.miryanov@gmail.com>
gh-134062: Fix hash collisions in IPv4Network and IPv6Network (GH-134063)
(cherry picked from commit f3fc0c16e0)
gh-134062: Fix hash collisions in IPv4Network and IPv6Network
gh-134062: Add hash collision regression test
Co-authored-by: Mike Salvatore <mike.s.salvatore@gmail.com>
Update `test_io` `_check_warn_on_dealloc` to use `self.` to dispatch to
different I/O implementations.
Update the `_pyio` implementation to match expected behavior, using the
same `_dealloc_warn` design as the C implementation uses to report the
topmost `__del__` object.
The FileIO one now matches all the others, so can use IOBase. There was
a missing check on closing (self._fd must be valid), add that check
(cherry picked from commit 5b0e827521)
Co-authored-by: Cody Maloney <cmaloney@users.noreply.github.com>
Test was only testing _io, expanded to cover _pyio.
(cherry picked from commit 06eaf4055c)
Co-authored-by: Cody Maloney <cmaloney@users.noreply.github.com>
Co-authored-by: Peter Bierma <zintensitydev@gmail.com>
