The IMAP4 client could consume an arbitrary amount of memory when trying
to connect to a malicious server, because it read a "literal" data with a
single read(size) call, and BufferedReader.read() allocates the bytes
object of the specified size before reading. Now the IMAP4 client reads data
by chunks, therefore the amount of used memory is limited by the
amount of the data actually been sent by the server.
(cherry picked from commit 735f25c5e3)
Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>
Co-authored-by: Gregory P. Smith <greg@krypto.org>
Co-authored-by: Łukasz Langa <lukasz@langa.pl>
* [3.11] gh-119461: Fix ThreadedVSOCKSocketStreamTest (GH-129171)
Fix ThreadedVSOCKSocketStreamTest: if get_cid() returns the host
address or the "any" address, use the local communication address
(loopback): VMADDR_CID_LOCAL.
On Linux 6.9, apparently, the /dev/vsock device is now available but
get_cid() returns VMADDR_CID_ANY (-1).
(cherry picked from commit 45db419c31)
Co-authored-by: Victor Stinner <vstinner@python.org>
(cherry picked from commit e94dbe4ed8)
(cherry picked from commit c750061047)
(cherry picked from commit cbfe3023e4)
* gh-119461: Restore the testSocket VSOCK skipUnless removed by PR #119465 (#129561)
Restore the skipUnless removed by #119465.
This test can only pass on virtual machines, not actual machines.
actual machines see:
```
self.cli.connect((cid, VSOCKPORT))
~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^
OSError: [Errno 19] No such device
```
Reproduced on (Linux) Ubuntu 24.04.1 running 6.8.0-52-generic.
---------
Co-authored-by: Victor Stinner <vstinner@python.org>
Co-authored-by: Gregory P. Smith <greg@krypto.org>
Co-authored-by: Gregory P. Smith <greg@krypto.org>
(cherry picked from commit 29f348e232)
Co-authored-by: JohnJamesUtley <81572567+JohnJamesUtley@users.noreply.github.com>
gh-125041: gh-90781: test_zlib: For s390x HW acceleration, skip checking the compressed bytes (GH-125042)
This backports two commits:
- GH-31096 skipped the tests unconditionally
- GH-125042 skips only the possibly-failing assertion
(cherry picked from commit cc5a225cdc)
* gh-107902: Don't test setting suid/sgid on systems that don't support them (GH-108368)
* gh-108948: Skip test_tarfile.test_modes() on EFTYPE error (#109697)
On FreeBSD, regular users cannot set the sticky bit. Skip the test if
chmod() fails with EFTYPE error.
---------
Co-authored-by: Victor Stinner <vstinner@python.org>
gh-100005: Skip test_script_as_dev_fd() on FreeBSD (GH-100006)
On FreeBSD, skip test_script_as_dev_fd() of test_cmd_line_script if
fdescfs is not mounted (at /dev/fd).
(cherry picked from commit 038b151963)
Co-authored-by: Victor Stinner <vstinner@python.org>
Co-authored-by: Petr Viktorin <encukou@gmail.com>
[3.11] gh-109396: Fix test_socket.test_hmac_sha1() in FIPS mode (GH-109423) (GH-109427)
gh-109396: Fix test_socket.test_hmac_sha1() in FIPS mode (GH-109423)
Use a longer key: FIPS mode requires at least of at least 112 bits.
The previous key was only 32 bits.
(cherry picked from commit e091b9f20f)
(cherry picked from commit f7bfac4b3d)
Co-authored-by: Victor Stinner <vstinner@python.org>
Co-authored-by: Petr Viktorin <encukou@gmail.com>
Detect email address parsing errors and return empty tuple to
indicate the parsing error (old API). Add an optional 'strict'
parameter to getaddresses() and parseaddr() functions. Patch by
Thomas Dwyer.
(cherry picked from commit 4a153a1d3b)
Co-authored-by: Victor Stinner <vstinner@python.org>
Co-Authored-By: Thomas Dwyer <github@tomd.tel>
Applies changes from zipp 3.20.1 and jaraco/zippGH-124
(cherry picked from commit 2231286d78)
(cherry picked from commit 17b77bb)
Co-authored-by: Jason R. Coombs <jaraco@jaraco.com>
Applies changes from zipp 3.20.1 and jaraco/zippGH-124
(cherry picked from commit 2231286d78)
(cherry picked from commit 17b77bb)
Co-authored-by: Jason R. Coombs <jaraco@jaraco.com>
Per RFC 2047:
> [...] these encoding schemes allow the
> encoding of arbitrary octet values, mail readers that implement this
> decoding should also ensure that display of the decoded data on the
> recipient's terminal will not cause unwanted side-effects
It seems that the "quoted-word" scheme is a valid way to include
a newline character in a header value, just like we already allow
undecodable bytes or control characters.
They do need to be properly quoted when serialized to text, though.
This should fail for custom fold() implementations that aren't careful
about newlines.
(cherry picked from commit 0976339818)
Co-authored-by: Petr Viktorin <encukou@gmail.com>
Co-authored-by: Bas Bloemsaat <bas@bloemsaat.org>
Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>
* Remove backtracking when parsing tarfile headers
* Rewrite PAX header parsing to be stricter
* Optimize parsing of GNU extended sparse headers v0.0
(cherry picked from commit 34ddb64d08)
Co-authored-by: Kirill Podoprigora <kirill.bast9@mail.ru>
Co-authored-by: Gregory P. Smith <greg@krypto.org>
(cherry picked from commit f071f01b7b)
Co-authored-by: Russell Keith-Magee <russell@keith-magee.com>
Co-authored-by: Gregory P. Smith <greg@krypto.org>
Authenticate socket connection for `socket.socketpair()` fallback when the platform does not have a native `socketpair` C API. We authenticate in-process using `getsocketname` and `getpeername` (thanks to Nathaniel J Smith for that suggestion).
(cherry picked from commit 78df1043db)
Co-authored-by: Seth Michael Larson <seth@python.org>
Co-authored-by: Gregory P. Smith <greg@krypto.org>
[3.10] gh-112769: test_zlib: test_zlib: Fix comparison of ZLIB_RUNTIME_VERSION with non-int suffix
zlib-ng defines the version as "1.3.0.zlib-ng".
(cherry picked from commit d384813)
Co-authored-by: Miro Hrončok miro@hroncok.cz
The _private_networks variables, used by various is_private
implementations, were missing some ranges and at the same time had
overly strict ranges (where there are more specific ranges considered
globally reachable by the IANA registries).
This patch updates the ranges with what was missing or otherwise
incorrect.
100.64.0.0/10 is left alone, for now, as it's been made special in [1].
The _address_exclude_many() call returns 8 networks for IPv4, 121
networks for IPv6.
[1] https://github.com/python/cpython/issues/61602
In 3.10 and below, is_private checks whether the network and broadcast
address are both private.
In later versions (where the test wss backported from), it checks
whether they both are in the same private network.
For 0.0.0.0/0, both 0.0.0.0 and 255.225.255.255 are private,
but one is in 0.0.0.0/8 ("This network") and the other in
255.255.255.255/32 ("Limited broadcast").
---------
Co-authored-by: Jakub Stasiak <jakub@stasiak.at>
This fixes XML unittest fallout from the https://github.com/python/cpython/issues/115398 security fix. When configured using `--with-system-expat` on systems with older pre 2.6.0 versions of libexpat, our unittests were failing.
Co-authored-by: Sebastian Pipping <sebastian@pipping.org>
Use of a proxy is intended to defer DNS for the hosts to the proxy itself, rather than a potential for information leak of the host doing DNS resolution itself for any reason. Proxy bypass lists are strictly name based. Most implementations of proxy support agree.
(cherry picked from commit c43b26d02e)
Co-authored-by: Weii Wang <weii.wang@canonical.com>
Allow controlling Expat >=2.6.0 reparse deferral (CVE-2023-52425) by adding five new methods:
- `xml.etree.ElementTree.XMLParser.flush`
- `xml.etree.ElementTree.XMLPullParser.flush`
- `xml.parsers.expat.xmlparser.GetReparseDeferralEnabled`
- `xml.parsers.expat.xmlparser.SetReparseDeferralEnabled`
- `xml.sax.expatreader.ExpatParser.flush`
Based on the "flush" idea from https://github.com/python/cpython/pull/115138#issuecomment-1932444270 .
Includes code suggested-by: Snild Dolkow <snild@sony.com>
and by core dev Serhiy Storchaka.
Co-authored-by: Gregory P. Smith <greg@krypto.org>
gh-115133: Fix tests for XMLPullParser with Expat 2.6.0 (GH-115164)
Feeding the parser by too small chunks defers parsing to prevent
CVE-2023-52425. Future versions of Expat may be more reactive.
(cherry picked from commit 4a08e7b343)
Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>
Raise BadZipFile when try to read an entry that overlaps with other entry or
central directory.
(cherry picked from commit 66363b9a7b)
Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>
* Fix a crash when pass UINT_MAX.
* Fix an integer overflow on 64-bit non-Windows platforms.
(cherry picked from commit 0daf555c6f)
Co-authored-by: Zackery Spytz <zspytz@gmail.com>
* In preauth tests of test_ssl, explicitly break reference cycles
invoving SingleConnectionTestServerThread to make sure that the
thread is deleted. Otherwise, the test marks the environment as
altered because the threading module sees a "dangling thread"
(SingleConnectionTestServerThread). This test leak was introduced
by the test added for the fix of issue gh-108310.
* Use support.SHORT_TIMEOUT instead of hardcoded 1.0 or 2.0 seconds
timeout.
* SingleConnectionTestServerThread.run() catchs TimeoutError
* Fix a race condition (missing synchronization) in
test_preauth_data_to_tls_client(): the server now waits until the
client connect() completed in call_after_accept().
* test_https_client_non_tls_response_ignored() calls server.join()
explicitly.
* Replace "localhost" with server.listener.getsockname()[0].
(cherry picked from commit 592bacb6fc)
Co-authored-by: Victor Stinner <vstinner@python.org>
Previously *consumed was not set in this case.
(cherry picked from commit b8b3e6afc0)
(cherry picked from commit f08e52ccb0)
Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>
gh-108310: Fix CVE-2023-40217: Check for & avoid the ssl pre-close flaw
Instances of `ssl.SSLSocket` were vulnerable to a bypass of the TLS handshake
and included protections (like certificate verification) and treating sent
unencrypted data as if it were post-handshake TLS encrypted data.
The vulnerability is caused when a socket is connected, data is sent by the
malicious peer and stored in a buffer, and then the malicious peer closes the
socket within a small timing window before the other peers’ TLS handshake can
begin. After this sequence of events the closed socket will not immediately
attempt a TLS handshake due to not being connected but will also allow the
buffered data to be read as if a successful TLS handshake had occurred.
Co-authored-by: Gregory P. Smith [Google LLC] <greg@krypto.org>
gh-104472: Skip `test_subprocess.ProcessTestCase.test_empty_env` if ASAN is enabled (GH-104667)
Skip test_subprocess.ProcessTestCase.test_empty_env if ASAN is enabled.
(cherry picked from commit c3f43bfb4b)
Co-authored-by: chgnrdv <52372310+chgnrdv@users.noreply.github.com>
gh-102153: Start stripping C0 control and space chars in `urlsplit` (GH-102508)
`urllib.parse.urlsplit` has already been respecting the WHATWG spec a bit GH-25595.
This adds more sanitizing to respect the "Remove any leading C0 control or space from input" [rule](https://url.spec.whatwg.org/GH-url-parsing:~:text=Remove%20any%20leading%20and%20trailing%20C0%20control%20or%20space%20from%20input.) in response to [CVE-2023-24329](https://nvd.nist.gov/vuln/detail/CVE-2023-24329).
I simplified the docs by eliding the state of the world explanatory
paragraph in this security release only backport. (people will see
that in the mainline /3/ docs)
---------
(cherry picked from commit 2f630e1ce1)
(cherry picked from commit 610cc0ab1b)
Co-authored-by: Miss Islington (bot) <31488909+miss-islington@users.noreply.github.com>
Co-authored-by: Illia Volochii <illia.volochii@gmail.com>
Co-authored-by: Gregory P. Smith [Google] <greg@krypto.org>
gh-104049: do not expose on-disk location from SimpleHTTPRequestHandler (GH-104067)
Do not expose the local server's on-disk location from `SimpleHTTPRequestHandler` when generating a directory index. (unnecessary information disclosure)
---------
(cherry picked from commit c7c3a60c88)
Co-authored-by: Ethan Furman <ethan@stoneleaf.us>
Co-authored-by: Gregory P. Smith <greg@krypto.org>
Co-authored-by: Jelle Zijlstra <jelle.zijlstra@gmail.com>
On macOS all file descriptors for a particular file in /dev/fd
share the same file offset, that is ``open("/dev/fd/9", "r")`` behaves
more like ``dup(9)`` than a regular open.
This causes problems when a user tries to run "/dev/fd/9" as a script
because zipimport changes the file offset to try to read a zipfile
directory. Therefore change zipimport to reset the file offset after
trying to read the zipfile directory.
(cherry picked from commit d08fb25769)
Co-authored-by: Ronald Oussoren <ronaldoussoren@mac.com>
* Regen zipimport
---------
Co-authored-by: Ronald Oussoren <ronaldoussoren@mac.com>
Co-authored-by: Shantanu <12621235+hauntsaninja@users.noreply.github.com>
Co-authored-by: Łukasz Langa <lukasz@langa.pl>
Previously, any data _after_ the zip64 extra would be removed.
With many new tests.
Fixes GH-88233
(cherry picked from commit 59e86caca8)
Co-authored-by: Tim Hatch <tim@timhatch.com>
GH-25309 enabled SSL_OP_IGNORE_UNEXPECTED_EOF by default, with a comment
that it restores OpenSSL 1.1.1 behavior, but this wasn't quite right.
That option causes OpenSSL to treat transport EOF as the same as
close_notify (i.e. SSL_ERROR_ZERO_RETURN), whereas Python actually has
distinct SSLEOFError and SSLZeroReturnError exceptions. (The latter is
usually mapped to a zero return from read.) In OpenSSL 1.1.1, the ssl
module would raise them for transport EOF and close_notify,
respectively. In OpenSSL 3.0, both act like close_notify.
Fix this by, instead, just detecting SSL_R_UNEXPECTED_EOF_WHILE_READING
and mapping that to the other exception type.
There doesn't seem to have been any unit test of this error, so fill in
the missing one. This had to be done with the BIO path because it's
actually slightly tricky to simulate a transport EOF with Python's fd
based APIs. (If you instruct the server to close the socket, it gets
confused, probably because the server's SSL object is still referencing
the now dead fd?)
(cherry picked from commit 420bbb783b)
Co-authored-by: David Benjamin <davidben@google.com>
