Email generators using email.policy.default may convert an RFC 2047
encoded-word to unencoded form during header refolding. In a structured
header, this could allow 'specials' chars outside a quoted-string,
leading to invalid address headers and enabling spoofing. This change
ensures a parsed encoded-word that contains specials is kept as an
encoded-word while the header is refolded.
[Better fix from @bitdancer.]
(cherry picked from commit 295b53df2a)
Co-authored-by: Mike Edmunds <medmunds@gmail.com>
Co-authored-by: R David Murray <rdmurray@bitdance.com>
Co-authored-by: Petr Viktorin <encukou@gmail.com>
Email generators using email.policy.default could incorrectly omit the
quote ('"') characters from a quoted-string during header refolding,
leading to invalid address headers and enabling header spoofing. This
change restores the quote characters on a bare-quoted-string as the
header is refolded, and escapes backslash and quote chars in the string.
(cherry picked from commit 5aaf416858)
Co-authored-by: Mike Edmunds <medmunds@gmail.com>
Co-authored-by: Petr Viktorin <encukou@gmail.com>
Detect email address parsing errors and return empty tuple to
indicate the parsing error (old API). Add an optional 'strict'
parameter to getaddresses() and parseaddr() functions. Patch by
Thomas Dwyer.
(cherry picked from commit 4a153a1d3b)
Co-authored-by: Victor Stinner <vstinner@python.org>
Co-authored-by: Thomas Dwyer <github@tomd.tel>
Per RFC 2047:
> [...] these encoding schemes allow the
> encoding of arbitrary octet values, mail readers that implement this
> decoding should also ensure that display of the decoded data on the
> recipient's terminal will not cause unwanted side-effects
It seems that the "quoted-word" scheme is a valid way to include
a newline character in a header value, just like we already allow
undecodable bytes or control characters.
They do need to be properly quoted when serialized to text, though.
Verify that email headers are well-formed.
This should fail for custom fold() implementations that aren't careful
about newlines.
(cherry picked from commit 0976339818)
Co-authored-by: Petr Viktorin <encukou@gmail.com>
Co-authored-by: Bas Bloemsaat <bas@bloemsaat.org>
Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>
Fix regression introduced in gh-100884: AttributeError when re-fold a long
address list.
Also fix more cases of incorrect encoding of the address separator in the
address list missed in gh-100884.
(cherry picked from commit 858b9e85fc)
ListSeparator should not be encoded. This could happen when a long line
pushes its separator to the next line, which would have been encoded.
(cherry picked from commit 09fab93c3d)
Co-authored-by: Thomas Weißschuh <thomas@t-8ch.de>
It now always encodes non-ASCII characters in headers if utf8 is false.
(cherry picked from commit 504334c7be)
Co-authored-by: Rito Takeuchi <licht-t@outlook.jp>
Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>
Determine the support of the Kyiv timezone by checking the result of
astimezone() which uses the system tz database and not the one
populated by zoneinfo.
(cherry picked from commit 931d7e052e)
Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>
It occurred when try to re-encode an unknown-8bit part combined with non-unknown-8bit part.
(cherry picked from commit e9d5b6ea2d)
Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>
It was raised if the charset itself contains characters not encodable
in UTF-8 (in particular \udcxx characters representing non-decodable
bytes in the source).
Similar to the rewrite of email/mime/image.py and associated test after the
deprecation of imghdr.py, thisrewrites email/mime/audio.py and associated
tests after the deprecation of sndhdr.py.
Closes#91885
* Rewrite imghdr inlining for clarity and completeness
* Move MIMEImage class back closer to the top of the file since it's the
important thing.
* Use a decorate to mark a given rule function and simplify the rule function
names for clarity.
* Copy over all the imghdr test data files into the email package's test data
directory. This way when imghdr is actually removed, it won't affect the
MIMEImage guessing tests.
* Rewrite and extend the MIMEImage tests to test for all supported
auto-detected MIME image subtypes.
* Remove the now redundant PyBanner048.gif data file.
* See https://github.com/python/cpython/pull/91461#discussion_r850313336
Co-authored-by: Oleg Iarygin <dralife@yandex.ru>
Co-authored-by: Oleg Iarygin <dralife@yandex.ru>
* Deprecate imghdr
Co-authored-by: Hugo van Kemenade <hugovk@users.noreply.github.com>
* Update Doc/whatsnew/3.11.rst
Co-authored-by: Hugo van Kemenade <hugovk@users.noreply.github.com>
* Inline `imghdr` into `email.mime.image`
Co-authored-by: Hugo van Kemenade <hugovk@users.noreply.github.com>
Co-authored-by: Barry Warsaw <barry@python.org>
Various date parsing utilities in the email module, such as
email.utils.parsedate(), are supposed to gracefully handle invalid
input, typically by raising an appropriate exception or by returning
None.
The internal email._parseaddr._parsedate_tz() helper used by some of
these date parsing routines tries to be robust against malformed input,
but unfortunately it can still crash ungracefully when a non-empty but
whitespace-only input is passed. This manifests as an unexpected
IndexError.
In practice, this can happen when parsing an email with only a newline
inside a ‘Date:’ header, which unfortunately happens occasionally in the
real world.
Here's a minimal example:
$ python
Python 3.9.6 (default, Jun 30 2021, 10:22:16)
[GCC 11.1.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import email.utils
>>> email.utils.parsedate('foo')
>>> email.utils.parsedate(' ')
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/lib/python3.9/email/_parseaddr.py", line 176, in parsedate
t = parsedate_tz(data)
File "/usr/lib/python3.9/email/_parseaddr.py", line 50, in parsedate_tz
res = _parsedate_tz(data)
File "/usr/lib/python3.9/email/_parseaddr.py", line 72, in _parsedate_tz
if data[0].endswith(',') or data[0].lower() in _daynames:
IndexError: list index out of range
The fix is rather straight-forward: guard against empty lists, after
splitting on whitespace, but before accessing the first element.
I am re-submitting an older PR which was abandoned but is still relevant, #10783 by @timb07.
The issue being solved () is still relevant. The original PR #10783 was closed as
the final request changes were not applied and since abandoned.
In this new PR I have re-used the original patch plus applied both comments from the review, by @maxking and @pganssle.
For reference, here is the original PR description:
In email.utils.parsedate_to_datetime(), a failure to parse the date, or invalid date components (such as hour outside 0..23) raises an exception. Document this behaviour, and add tests to test_email/test_utils.py to confirm this behaviour.
In email.headerregistry.DateHeader.parse(), check when parsedate_to_datetime() raises an exception and add a new defect InvalidDateDefect; preserve the invalid value as the string value of the header, but set the datetime attribute to None.
Add tests to test_email/test_headerregistry.py to confirm this behaviour; also added test to test_email/test_inversion.py to confirm emails with such defective date headers round trip successfully.
This pull request incorporates feedback gratefully received from @bitdancer, @brettcannon, @Mariatta and @warsaw, and replaces the earlier PR #2254.
Automerge-Triggered-By: GH:warsaw
This PR replaces #1977. The reason for the replacement is two-fold.
The fix itself is different is that if the CTE header doesn't exist in the original message, it is inserted. This is important because the new CTE could be quoted-printable whereas the original is implicit 8bit.
Also the tests are different. The test_nonascii_as_string_without_cte test in #1977 doesn't actually test the issue in that it passes without the fix. The test_nonascii_as_string_without_content_type_and_cte test is improved here, and even though it doesn't fail without the fix, it is included for completeness.
Automerge-Triggered-By: @warsaw
* bpo-39040: Fix parsing of email headers with encoded-words inside a quoted string.
It is fairly common to find malformed mime headers (especially content-disposition
headers) where the parameter values, instead of being encoded to RFC
standards, are "encoded" by doing RFC 2047 "encoded word" encoding, and
then enclosing the whole thing in quotes. The processing of these malformed
headers was incorrectly leaving the spaces between encoded words in the decoded
text (whitespace between adjacent encoded words is supposed to be stripped on
decoding). This changeset fixes the encoded word processing inside quoted strings
(bare-quoted-string) to do correct RFC 2047 decoding by stripping that
whitespace.
raw_data_manager (default for EmailPolicy, EmailMessage)
does correct wrapping of 'text' parts as long as the message contains
characters outside of 7bit US-ASCII set: base64 or qp
Content-Transfer-Encoding is applied if the lines would be too long
without it. It did not, however, do this for ascii-only text,
which could result in lines that were longer than
policy.max_line_length or even the rfc 998 maximum.
This changeset fixes the heuristic so that if lines are longer than
policy.max_line_length, it will always apply a
content-transfer-encoding so that the lines are wrapped correctly.
parse_message_id() was improperly using a token defined inside an exception
handler, which was raising `UnboundLocalError` on parsing an invalid value.
https://bugs.python.org/issue38698
