This was added in 3.15; let's use a real sentinel instead of an ad-hoc list object.
(cherry picked from commit 884ac3e3ec)
Co-authored-by: Jelle Zijlstra <jelle.zijlstra@gmail.com>
* empty lines are always ignored instead of separating groups
* the "user-agent" line after a rule starts a new group
* groups matching the same user agent are now merged
* the rule with the longest match wins instead of the first matching rule
* in case of equal matches, the “Allow” rule wins over “Disallow”
* special characters “$” and “*” are now supported in rules
* prefer full match for user agent
Changes in the urllib.parse module:
* Add option missing_as_none in urlparse(), urlsplit() and urldefrag(). If
it is true, represent not defined components as None instead of an
empty string.
* Add option keep_empty in urlunparse() and urlunsplit(). If it is
true, keep empty non-None components in the resulting string.
* Don't fail trying to parse weird patterns.
* Don't fail trying to decode non-UTF-8 "robots.txt" files.
* No longer ignore trailing "?" in patterns and URLs.
* Distinguish raw special characters "?", "=" and "&" from the
percent-encoded ones.
* Remove tests that do nothing.
Follow-up to 66cdb2bd8a.
Add *resolve_host* keyword-only argument to `url2pathname()`, defaulting to
false. When set to true, we call `socket.gethostbyname()` to resolve the
URL hostname.
Co-authored-by: Bénédikt Tran <10796600+picnixz@users.noreply.github.com>
Co-authored-by: Adam Turner <9087854+AA-Turner@users.noreply.github.com>
Co-authored-by: Steve Dower <steve.dower@microsoft.com>
In `_is_local_authority()`, return early if the authority matches the
machine hostname from `socket.gethostname()`, rather than resolving the
names and matching IP addresses.
Add optional *add_scheme* argument to `urllib.request.pathname2url()`; when
set to true, a complete URL is returned. Likewise add optional
*require_scheme* argument to `url2pathname()`; when set to true, a complete
URL is accepted.
Co-authored-by: Bénédikt Tran <10796600+picnixz@users.noreply.github.com>
In `urllib.request.url2pathname()`, if the authority resolves to the
current host, discard it. If an authority is present but resolves somewhere
else, then on Windows we return a UNC path (as before), and on other
platforms we raise `URLError`.
Affects `pathlib.Path.from_uri()` in the same way.
Co-authored-by: Adam Turner <9087854+AA-Turner@users.noreply.github.com>
Co-authored-by: Bénédikt Tran <10796600+picnixz@users.noreply.github.com>
Deprecate the `nturl2path` module. Its functionality is merged into
`urllib.request`.
Add `tests.test_nturl2path` to exercise `nturl2path`, as it's no longer
covered by `test_urllib`.
* gh-105704: Disallow square brackets ( and ) in domain names for parsed URLs
* Use Sphinx references
Co-authored-by: Peter Bierma <zintensitydev@gmail.com>
* Add mismatched bracket test cases, fix news format
* Add more test coverage for ports
---------
Co-authored-by: Peter Bierma <zintensitydev@gmail.com>
support sha-256 digest authentication
Co-authored-by: Peter Bierma <zintensitydev@gmail.com>
Co-authored-by: Bénédikt Tran <10796600+picnixz@users.noreply.github.com>
Co-authored-by: Gregory P. Smith <greg@krypto.org>
The canonical `file:` URL (as generated by `pathname2url()`) is now used as the `url` attribute of the returned `addinfourl` object. The `addinfourl.url` attribute reflects the resolved URL for both `file:` or `http[s]:` URLs now.
When handed an absolute Windows path such as `C:\foo` or `//server/share`,
the `urllib.request.pathname2url()` function returns a URL with an
authority section, such as `///C:/foo` or `//server/share` (or before
GH-126205, `////server/share`). Only the `file:` prefix is omitted.
But when handed an absolute POSIX path such as `/etc/hosts`, or a Windows
path of the same form (rooted but lacking a drive), the function returns a
URL without an authority section, such as `/etc/hosts`.
This patch corrects the discrepancy by adding a `//` prefix before
drive-less, rooted paths when generating URLs.
Discard any 'localhost' authority from the beginning of a `file:` URI. As a
result, file URIs like `//localhost/etc/hosts` are correctly decoded as
`/etc/hosts`.
Adjust `urllib.request.url2pathname()` and `pathname2url()` to use the
filesystem encoding when quoting and unquoting file URIs, rather than
forcing use of UTF-8.
No changes are needed in the `nturl2path` module because Windows always
uses UTF-8, per PEP 529.
Discard two leading slashes from the beginning of a `file:` URI if they
introduce an empty authority section. As a result, file URIs like
`///etc/hosts` are correctly parsed as `/etc/hosts`.
Accepting objects with false values (like 0 and []) except empty strings
and byte-like objects and None in urllib.parse functions parse_qsl() and
parse_qs() is now deprecated.
* urljoin() with relative reference "?" sets empty query and removes fragment.
* Preserve empty components (authority, params, query, fragment) in urljoin().
* Preserve empty components (authority, params, query) in urldefrag().
Also refactor the code and get rid of double _coerce_args() and
_coerce_result() calls in urljoin(), urldefrag(), urlparse() and
urlunparse().
* pass the original string error message from the ftplib error to URLError()
* Update request.py
Change error string for ftp error to be consistent with other errors reported for ftp
* Add NEWS entry for change to urllib.request for ftp errors.
* Track the change in the ftp error message in the test.
Tools such as ruff can ignore "imported but unused" warnings if a
line ends with "# noqa: F401". It avoids the temptation to remove
an import which is used effectively.
* Restore support of None and other false values.
* Raise TypeError for non-zero integers and non-empty sequences.
The regressions were introduced in gh-74668
(bdba8ef42b).
Use of a proxy is intended to defer DNS for the hosts to the proxy itself, rather than a potential for information leak of the host doing DNS resolution itself for any reason. Proxy bypass lists are strictly name based. Most implementations of proxy support agree.
Small performance improvement of getproxies_environment() when there are many environment variables. In a benchmark with 5k environment variables not related to proxies, and 5 specifying proxies, we get a 10% walltime improvement.
* GH-104554: Add RTSPS support to `urllib/parse.py`
RTSPS is the permanent scheme defined in
https://www.iana.org/assignments/uri-schemes/uri-schemes.xhtml
alongside RTSP and RTSPU schemes.
* 📜🤖 Added by blurb_it.
---------
Co-authored-by: blurb-it[bot] <43283697+blurb-it[bot]@users.noreply.github.com>
* Adds checks to ensure that bracketed hosts found by urlsplit are of IPv6 or IPvFuture format
---------
Co-authored-by: Gregory P. Smith <greg@krypto.org>
