* gh-100001: Omit control characters in http.server stderr logs. (GH-100002)
Replace control characters in http.server.BaseHTTPRequestHandler.log_message with an escaped \xHH sequence to avoid causing problems for the terminal the output is printed to.
(cherry picked from commit d8ab0a4dfa)
Co-authored-by: Gregory P. Smith <greg@krypto.org>
* also escape \s (backport of PR #100038).
* add versionadded and remove extra 'to'
Co-authored-by: Gregory P. Smith <greg@krypto.org>
There was an unnecessary quadratic loop in idna decoding. This restores
the behavior to linear.
(cherry picked from commit d315722564)
(cherry picked from commit a6f6c3a3d6)
Co-authored-by: Miss Islington (bot) <31488909+miss-islington@users.noreply.github.com>
Co-authored-by: Gregory P. Smith <greg@krypto.org>
Linux abstract sockets are insecure as they lack any form of filesystem
permissions so their use allows anyone on the system to inject code into
the process.
This removes the default preference for abstract sockets in
multiprocessing introduced in Python 3.9+ via
https://github.com/python/cpython/pull/18866 while fixing
https://github.com/python/cpython/issues/84031.
Explicit use of an abstract socket by a user now generates a
RuntimeWarning. If we choose to keep this warning, it should be
backported to the 3.7 and 3.8 branches.
(cherry picked from commit 49f61068f4)
Co-authored-by: Gregory P. Smith <greg@krypto.org>
This is a port of the applicable part of XKCP's fix [1] for
CVE-2022-37454 and avoids the segmentation fault and the infinite
loop in the test cases published in [2].
[1]: fdc6fef075
[2]: https://mouha.be/sha-3-buffer-overflow/
Regression test added by: Gregory P. Smith [Google LLC] <greg@krypto.org>
(cherry picked from commit 0e4e058602)
Co-authored-by: Theo Buehler <botovq@users.noreply.github.com>
gh-96710: Make the test timing more lenient for the int/str DoS regression test. (GH-96717)
A regression would still absolutely fail and even a flaky pass isn't
harmful as it'd fail most of the time across our N system test runs.
Windows has a low resolution timer and CI systems are prone to odd
timing so this just gives more leeway to avoid flakiness.
(cherry picked from commit 11e3548fd1)
Co-authored-by: Gregory P. Smith <greg@krypto.org>
gh-68966: Make mailcap refuse to match unsafe filenames/types/params (GH-91993)
(cherry picked from commit b9509ba7a9)
Co-authored-by: Petr Viktorin <encukou@gmail.com>
The macOS 13 SDK includes support for the `mkfifoat` and `mknodat` system calls.
Using the `dir_fd` option with either `os.mkfifo` or `os.mknod` could result in a
segfault if cpython is built with the macOS 13 SDK but run on an earlier
version of macOS. Prevent this by adding runtime support for detection of
these system calls ("weaklinking") as is done for other newer syscalls on
macOS.
(cherry picked from commit 6d0a0191a4)
Co-authored-by: Ned Deily <nad@python.org>
gh-96848: Fix -X int_max_str_digits option parsing (GH-96988)
Fix command line parsing: reject "-X int_max_str_digits" option with
no value (invalid) when the PYTHONINTMAXSTRDIGITS environment
variable is set to a valid limit.
(cherry picked from commit 41351662bc)
Co-authored-by: Victor Stinner <vstinner@python.org>
gh-97616: list_resize() checks for integer overflow (GH-97617)
Fix multiplying a list by an integer (list *= int): detect the
integer overflow when the new allocated length is close to the
maximum size. Issue reported by Jordan Limor.
list_resize() now checks for integer overflow before multiplying the
new allocated length by the list item size (sizeof(PyObject*)).
(cherry picked from commit a5f092f3c4)
Co-authored-by: Victor Stinner <vstinner@python.org>
* Correctly pre-check for int-to-str conversion (#96537)
Converting a large enough `int` to a decimal string raises `ValueError` as expected. However, the raise comes _after_ the quadratic-time base-conversion algorithm has run to completion. For effective DOS prevention, we need some kind of check before entering the quadratic-time loop. Oops! =)
The quick fix: essentially we catch _most_ values that exceed the threshold up front. Those that slip through will still be on the small side (read: sufficiently fast), and will get caught by the existing check so that the limit remains exact.
The justification for the current check. The C code check is:
```c
max_str_digits / (3 * PyLong_SHIFT) <= (size_a - 11) / 10
```
In GitHub markdown math-speak, writing $M$ for `max_str_digits`, $L$ for `PyLong_SHIFT` and $s$ for `size_a`, that check is:
$$\left\lfloor\frac{M}{3L}\right\rfloor \le \left\lfloor\frac{s - 11}{10}\right\rfloor$$
From this it follows that
$$\frac{M}{3L} < \frac{s-1}{10}$$
hence that
$$\frac{L(s-1)}{M} > \frac{10}{3} > \log_2(10).$$
So
$$2^{L(s-1)} > 10^M.$$
But our input integer $a$ satisfies $|a| \ge 2^{L(s-1)}$, so $|a|$ is larger than $10^M$. This shows that we don't accidentally capture anything _below_ the intended limit in the check.
<!-- gh-issue-number: gh-95778 -->
* Issue: gh-95778
<!-- /gh-issue-number -->
Co-authored-by: Gregory P. Smith [Google LLC] <greg@krypto.org>
Co-authored-by: Christian Heimes <christian@python.org>
Co-authored-by: Mark Dickinson <dickinsm@gmail.com>
Three test cases were failing on FreeBSD with latest OpenSSL.
(cherry picked from commit 1bc86c2625)
Co-authored-by: Christian Heimes <christian@python.org>
When binding a unix socket to an empty address on Linux, the socket is
automatically bound to an available address in the abstract namespace.
>>> s = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)
>>> s.bind("")
>>> s.getsockname()
b'\x0075499'
Since python 3.9, the socket is bound to the one address:
>>> s.getsockname()
b'\x00'
And trying to bind multiple sockets will fail with:
Traceback (most recent call last):
File "/home/nsoffer/src/cpython/Lib/test/test_socket.py", line 5553, in testAutobind
s2.bind("")
OSError: [Errno 98] Address already in use
Added 2 tests:
- Auto binding empty address on Linux
- Failing to bind an empty address on other platforms
Fixes f6b3a07b7d (bpo-44493: Add missing terminated NUL in sockaddr_un's length (GH-26866)
(cherry picked from commit c22f134211)
Co-authored-by: Nir Soffer <nsoffer@redhat.com>
Fix an open redirection vulnerability in the `http.server` module when
an URI path starts with `//` that could produce a 301 Location header
with a misleading target. Vulnerability discovered, and logic fix
proposed, by Hamza Avvan (@hamzaavvan).
Test and comments authored by Gregory P. Smith [Google].
(cherry picked from commit 4abab6b603)
Co-authored-by: Gregory P. Smith <greg@krypto.org>
Suppress writing an XML declaration in open files in ElementTree.write()
with encoding='unicode' and xml_declaration=None.
If file patch is passed to ElementTree.write() with encoding='unicode',
always open a new file in UTF-8.
(cherry picked from commit d7db9dc3cc)
Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>
Also while there, clarify a few things about why we reduce the hash to 32 bits.
Co-authored-by: Eli Libman <eli@hyro.ai>
Co-authored-by: Yury Selivanov <yury@edgedb.com>
Co-authored-by: Łukasz Langa <lukasz@langa.pl>
(cherry picked from commit c1f5c903a7)
The `utc_to_seconds` call can fail, here's a minimal reproducer on
Linux:
TZ=UTC python -c "from datetime import *; datetime.fromtimestamp(253402300799 + 1)"
The old behavior still raised an error in a similar way, but only
because subsequent calculations happened to fail as well. Better to fail
fast.
This also refactors the tests to split out the `fromtimestamp` and
`utcfromtimestamp` tests, and to get us closer to the actual desired
limits of the functions. As part of this, we also changed the way we
detect platforms where the same limits don't necessarily apply (e.g.
Windows).
As part of refactoring the tests to hit this condition explicitly (even
though the user-facing behvior doesn't change in any way we plan to
guarantee), I noticed that there was a difference in the places that
`datetime.utcfromtimestamp` fails in the C and pure Python versions, which
was fixed by skipping the "probe for fold" logic for UTC specifically —
since UTC doesn't have any folds or gaps, we were never going to find a
fold value anyway. This should prevent some failures in the pure python
`utcfromtimestamp` method on timestamps close to 0001-01-01.
There are two separate news entries for this because one is a
potentially user-facing change, the other is an internal code
correctness change that, if anything, changes some error messages. The
two happen to be coupled because of the test refactoring, but they are
probably best thought of as independent changes.
Fixes GH-91581
(cherry picked from commit 83c0247d47)
Co-authored-by: Paul Ganssle <1377457+pganssle@users.noreply.github.com>
If Condition.notify() was interrupted just after it released the waiter lock,
but before removing it from the queue, the following calls of notify() failed
with RuntimeError: cannot release un-acquired lock.
(cherry picked from commit 70af994fee)
Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>
ElementTree method write() and function tostring() now use the text file's
encoding ("UTF-8" if not available) instead of locale encoding in XML
declaration when encoding="unicode" is specified.
(cherry picked from commit 707839b0fe)
Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>
Automerge-Triggered-By: GH:serhiy-storchaka
* [3.9] bpo-46785: Fix race condition between os.stat() and unlink on Windows (GH-31858).
(cherry picked from commit 39e6b8ae6a)
Co-authored-by: Itai Steinherz <itaisteinherz@gmail.com>
Do not spawn ProcessPool workers on demand when they spawn via fork.
This avoids potential deadlocks in the child processes due to forking from
a multithreaded process..
(cherry picked from commit ebb37fc3fd)
Co-authored-by: Gregory P. Smith <greg@krypto.org>
(cherry picked from commit b795376a62)
Co-authored-by: Gregory P. Smith <greg@krypto.org>
Co-authored-by: Gregory P. Smith <greg@krypto.org>
The function fetch_server_certificate is replaced by get_server_certificate in the module. I reflected the change in the module docstrings.
Co-authored-by: Jelle Zijlstra <jelle.zijlstra@gmail.com>
(cherry picked from commit feca9bbd1f)
Co-authored-by: Kossi GLOKPOR <83467320+glk0@users.noreply.github.com>
`IPv*Network` and `IPv*Interface` constructors accept a 2-tuple of
(address description, netmask) as the address parameter.
When the tuple-based address is used errors are not propagated
correctly through the `ipaddress.ip_*` helper because of the %-formatting now expecting several arguments:
In [7]: ipaddress.ip_network(("192.168.100.0", "fooo"))
...
TypeError: not all arguments converted during string formatting
Compared to:
In [8]: ipaddress.IPv4Network(("192.168.100.0", "foo"))
...
NetmaskValueError: 'foo' is not a valid netmask
Use an f-string to make sure the error is always properly formatted.
Co-authored-by: Jelle Zijlstra <jelle.zijlstra@gmail.com>
(cherry picked from commit 52dc9c3066)
Co-authored-by: Thomas Cellerier <thomascellerier@gmail.com>
Do not store `ProcessPoolExecutor` work item exception traceback that prevents
exception frame locals from being garbage collected.
(cherry picked from commit 9c204b148f)
Co-authored-by: themylogin <themylogin@gmail.com>
If the error handler returns position less or equal than the starting
position of non-encodable characters, most of built-in encoders didn't
properly re-size the output buffer. This led to out-of-bounds writes,
and segfaults.
(cherry picked from commit 18b07d773e)
Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>
Previously, pickling did not fail, but the result could not be unpickled.
(cherry picked from commit 6d0d547033)
(cherry picked from commit e8ff3c92f6)
It was raised if the charset itself contains characters not encodable
in UTF-8 (in particular \udcxx characters representing non-decodable
bytes in the source).
(cherry picked from commit e91dee87ed)
Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>
GH- Adding 'required' to names in Lib.argparse.Action
gh-91832:
Added 'required' to the list `names` in `Lib.argparse.Action`.
Changed constant strings that test the Action object.
Automerge-Triggered-By: GH:merwok
(cherry picked from commit 4ed3900041)
Co-authored-by: Abhigyan Bose <abhigyandeepbose@gmail.com>
