gh-126623: Update libexpat to 2.6.4, make future updates easier (GH-126792)
Update libexpat to 2.6.4, make future updates easier.
(cherry picked from commit 3c99969094)
Co-authored-by: Seth Michael Larson <seth@python.org>
gh-122792: Make IPv4-mapped IPv6 address properties consistent with IPv4 (GH-122793)
Make IPv4-mapped IPv6 address properties consistent with IPv4.
(cherry picked from commit 76a1c5d183)
Co-authored-by: Seth Michael Larson <seth@python.org>
Authenticate socket connection for `socket.socketpair()` fallback when the platform does not have a native `socketpair` C API. We authenticate in-process using `getsocketname` and `getpeername` (thanks to Nathaniel J Smith for that suggestion).
(cherry picked from commit 78df1043db)
Co-authored-by: Seth Michael Larson <seth@python.org>
Co-authored-by: Gregory P. Smith <greg@krypto.org>
gh-116741: Upgrade libexpat to 2.6.2 (GH-117296)
Upgrade libexpat to 2.6.2
(cherry picked from commit c9829eec08)
Co-authored-by: Seth Michael Larson <seth@python.org>
gh-117233: Detect support for several hashes at hashlib build time (GH-117234)
Detect libcrypto BLAKE2, Shake, SHA3, and Truncated-SHA512 support at hashlib build time
GH-GH- BLAKE2
While OpenSSL supports both "b" and "s" variants of the BLAKE2 hash
function, other cryptographic libraries may lack support for one or both
of the variants. This commit modifies `hashlib`'s C code to detect
whether or not the linked libcrypto supports each BLAKE2 variant, and
elides references to each variant's NID accordingly. In cases where the
underlying libcrypto doesn't fully support BLAKE2, CPython's
`./configure` script can be given the following flag to use CPython's
interned BLAKE2 implementation: `--with-builtin-hashlib-hashes=blake2`.
GH-GH- SHA3, Shake, & truncated SHA512.
Detect BLAKE2, SHA3, Shake, & truncated SHA512 support in the
OpenSSL-ish libcrypto library at build time. This helps allow hashlib's
`_hashopenssl` to be used with libraries that do not to support every
algorithm that upstream OpenSSL does. Such as AWS-LC & BoringSSL.
(cherry picked from commit b8eaad3009)
Co-authored-by: Will Childs-Klein <willck93@gmail.com>
Co-authored-by: Gregory P. Smith [Google LLC] <greg@krypto.org>
Allow controlling Expat >=2.6.0 reparse deferral (CVE-2023-52425) by adding five new methods:
- `xml.etree.ElementTree.XMLParser.flush`
- `xml.etree.ElementTree.XMLPullParser.flush`
- `xml.parsers.expat.xmlparser.GetReparseDeferralEnabled`
- `xml.parsers.expat.xmlparser.SetReparseDeferralEnabled`
- `xml.sax.expatreader.ExpatParser.flush`
Based on the "flush" idea from https://github.com/python/cpython/pull/115138#issuecomment-1932444270 .
- Please treat as a security fix related to CVE-2023-52425.
(cherry picked from commit 6a95676bb5)
(cherry picked from commit 73807eb634)
(cherry picked from commit eda2963378)
---------
Includes code suggested-by: Snild Dolkow <snild@sony.com>
and by core dev Serhiy Storchaka.
Co-authored-by: Gregory P. Smith <greg@krypto.org>
gh-114572: Fix locking in cert_store_stats and get_ca_certs (GH-114573)
* gh-114572: Fix locking in cert_store_stats and get_ca_certs
cert_store_stats and get_ca_certs query the SSLContext's X509_STORE with
X509_STORE_get0_objects, but reading the result requires a lock. See
https://github.com/openssl/openssl/pull/23224 for details.
Instead, use X509_STORE_get1_objects, newly added in that PR.
X509_STORE_get1_objects does not exist in current OpenSSLs, but we can
polyfill it with X509_STORE_lock and X509_STORE_unlock.
* Work around const-correctness problem
* Add missing X509_STORE_get1_objects failure check
* Add blurb
(cherry picked from commit bce693111b)
Co-authored-by: David Benjamin <davidben@google.com>
Skip .pth files with names starting with a dot or hidden file attribute.
(cherry picked from commit 74208ed0c4)
Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>
Instances of `ssl.SSLSocket` were vulnerable to a bypass of the TLS handshake
and included protections (like certificate verification) and treating sent
unencrypted data as if it were post-handshake TLS encrypted data.
The vulnerability is caused when a socket is connected, data is sent by the
malicious peer and stored in a buffer, and then the malicious peer closes the
socket within a small timing window before the other peers’ TLS handshake can
begin. After this sequence of events the closed socket will not immediately
attempt a TLS handshake due to not being connected but will also allow the
buffered data to be read as if a successful TLS handshake had occurred.
Co-authored-by: Gregory P. Smith [Google LLC] <greg@krypto.org>
gh-102988: Detect email address parsing errors and return empty tuple to indicate the parsing error (old API) (GH-105127)
Detect email address parsing errors and return empty tuple to indicate the parsing error (old API). This fixes or at least ameliorates CVE-2023-27043.
---------
(cherry picked from commit 18dfbd0357)
Co-authored-by: Thomas Dwyer <github@tomd.tel>
Co-authored-by: Gregory P. Smith <greg@krypto.org>
gh-103142: Upgrade binary builds and CI to OpenSSL 1.1.1u (GH-105174)
Upgrade builds to OpenSSL 1.1.1u.
This OpenSSL version addresses a pile if less-urgent CVEs since 1.1.1t.
The Mac/BuildScript/build-installer.py was already updated.
Also updates _ssl_data_111.h from OpenSSL 1.1.1u, _ssl_data_300.h from 3.0.9, and adds a new _ssl_data_31.h file from 3.1.1 along with the ssl.c code to use it.
Manual edits to the _ssl_data_300.h file prevent it from removing any existing definitions in case those exist in some peoples builds and were important (avoiding regressions during backporting).
backports of this prior to 3.12 will not include the openssl 3.1 header.
(cherry picked from commit ede89af605)
Co-authored-by: Gregory P. Smith [Google] <greg@krypto.org>
gh-99108: Refresh HACL* (GH-104808)
Refresh HACL* from upstream to improve SHA2 performance and fix a 32-bit issue in SHA3.
(cherry picked from commit 160321e530)
Co-authored-by: Jonathan Protzenko <protz@microsoft.com>
* Fix directory traversal security flaw in uu.decode()
* also check absolute paths and os.altsep
* Add a regression test.
---------
Co-authored-by: Gregory P. Smith <greg@krypto.org> [Google]
Replaces our built-in SHA3 implementation with a verified one from the HACL* project.
This implementation is used when OpenSSL does not provide SHA3 or is not present.
3.11 shiped with a very slow tiny sha3 implementation to get off of the <=3.10 reference implementation that wound up having serious bugs. This brings us back to a reasonably performing built-in implementation consistent with what we've just replaced our other guaranteed available standard hash algorithms with: code from the HACL* project.
---------
Co-authored-by: Gregory P. Smith <greg@krypto.org>
Do not expose the local server's on-disk location from `SimpleHTTPRequestHandler` when generating a directory index. (unnecessary information disclosure)
---------
Co-authored-by: Gregory P. Smith <greg@krypto.org>
Co-authored-by: Jelle Zijlstra <jelle.zijlstra@gmail.com>
Replace the builtin hashlib implementations of SHA2-384 and SHA2-512
originally from LibTomCrypt with formally verified, side-channel resistant
code from the [HACL*](https://github.com/hacl-star/hacl-star/) project.
The builtins remain a fallback only used when OpenSSL does not provide them.
replacing hashlib primitives (for the non-OpenSSL case) with verified implementations from HACL*. This is the first PR in the series, and focuses specifically on SHA2-256 and SHA2-224.
This PR imports Hacl_Streaming_SHA2 into the Python tree. This is the HACL* implementation of SHA2, which combines a core implementation of SHA2 along with a layer of buffer management that allows updating the digest with any number of bytes. This supersedes the previous implementation in the tree.
@franziskuskiefer was kind enough to benchmark the changes: in addition to being verified (thus providing significant safety and security improvements), this implementation also provides a sizeable performance boost!
```
---------------------------------------------------------------
Benchmark Time CPU Iterations
---------------------------------------------------------------
Sha2_256_Streaming 3163 ns 3160 ns 219353 // this PR
LibTomCrypt_Sha2_256 5057 ns 5056 ns 136234 // library used by Python currently
```
The changes in this PR are as follows:
- import the subset of HACL* that covers SHA2-256/224 into `Modules/_hacl`
- rewire sha256module.c to use the HACL* implementation
Co-authored-by: Gregory P. Smith [Google LLC] <greg@krypto.org>
Co-authored-by: Erlend E. Aasland <erlend.aasland@protonmail.com>
Replace control characters in http.server.BaseHTTPRequestHandler.log_message with an escaped \xHH sequence to avoid causing problems for the terminal the output is printed to.
There was an unnecessary quadratic loop in idna decoding. This restores
the behavior to linear.
This also adds an early length check in IDNA decoding to outright reject
huge inputs early on given the ultimate result is defined to be 63 or fewer
characters.
