Stowage/cpython
2
0
Fork 0
mirror of https://github.com/python/cpython.git synced 2025-12-08 06:10:17 +00:00
Code Issues Projects Releases Packages Wiki Activity
cpython/Lib/test/test_netrc.py
Victor Stinner 79f7a4c0a4
[3.11] gh-108822: Backport libregrtest changes from the main branch (#108820) 
* Revert "[3.11] gh-101634: regrtest reports decoding error as failed test (#106169) (#106175)"

This reverts commit d5418e97fc.

* Revert "[3.11] bpo-46523: fix tests rerun when `setUp[Class|Module]` fails (GH-30895) (GH-103342)"

This reverts commit ecb09a8496.

* Revert "gh-95027: Fix regrtest stdout encoding on Windows (GH-98492)"

This reverts commit b2aa28eec5.

* Revert "[3.11] gh-94026: Buffer regrtest worker stdout in temporary file (GH-94253) (GH-94408)"

This reverts commit 0122ab235b.

* Revert "Run Tools/scripts/reindent.py (GH-94225)"

This reverts commit f0f3a424af.

* Revert "gh-94052: Don't re-run failed tests with --python option (GH-94054)"

This reverts commit 1347607db1.

* Revert "[3.11] gh-84461: Fix Emscripten umask and permission issues (GH-94002) (GH-94006)"

This reverts commit 1073184918.

* gh-93353: regrtest checks for leaked temporary files (#93776)

When running tests with -jN, create a temporary directory per process
and mark a test as "environment changed" if a test leaks a temporary
file or directory.

(cherry picked from commit e566ce5496)

* gh-93353: Fix regrtest for -jN with N >= 2 (GH-93813)

(cherry picked from commit 36934a16e8)

* gh-93353: regrtest supports checking tmp files with -j2 (#93909)

regrtest now also implements checking for leaked temporary files and
directories when using -jN for N >= 2. Use tempfile.mkdtemp() to
create the temporary directory. Skip this check on WASI.

(cherry picked from commit 4f85cec9e2)

* gh-84461: Fix Emscripten umask and permission issues (GH-94002)

- Emscripten's default umask is too strict, see
  https://github.com/emscripten-core/emscripten/issues/17269
- getuid/getgid and geteuid/getegid are stubs that always return 0
  (root). Disable effective uid/gid syscalls and fix tests that use
  chmod() current user.
- Cannot drop X bit from directory.

(cherry picked from commit 2702e408fd)

* gh-94052: Don't re-run failed tests with --python option (#94054)

(cherry picked from commit 0ff7b996f5)

* Run Tools/scripts/reindent.py (#94225)

Reindent files which were not properly formatted (PEP 8: 4 spaces).

Remove also some trailing spaces.

(cherry picked from commit e87ada48a9)

* gh-94026: Buffer regrtest worker stdout in temporary file (GH-94253)

Co-authored-by: Victor Stinner <vstinner@python.org>
(cherry picked from commit 199ba23324)

* gh-96465: Clear fractions hash lru_cache under refleak testing (GH-96689)

Automerge-Triggered-By: GH:zware
(cherry picked from commit 9c8f379433)

* gh-95027: Fix regrtest stdout encoding on Windows (#98492)

On Windows, when the Python test suite is run with the -jN option,
the ANSI code page is now used as the encoding for the stdout
temporary file, rather than using UTF-8 which can lead to decoding
errors.

(cherry picked from commit ec1f6f5f13)

* gh-98903: Test suite fails with exit code 4 if no tests ran (#98904)

The Python test suite now fails wit exit code 4 if no tests ran. It
should help detecting typos in test names and test methods.

* Add "EXITCODE_" constants to Lib/test/libregrtest/main.py.
* Fix a typo: "NO TEST RUN" becomes "NO TESTS RAN"

(cherry picked from commit c76db37c0d)

* gh-100086: Add build info to test.libregrtest (#100093)

The Python test runner (libregrtest) now logs Python build information like
"debug" vs "release" build, or LTO and PGO optimizations.

(cherry picked from commit 3c89202247)

* bpo-46523: fix tests rerun when `setUp[Class|Module]` fails (#30895)

Co-authored-by: Jelle Zijlstra <jelle.zijlstra@gmail.com>
Co-authored-by: Łukasz Langa <lukasz@langa.pl>
(cherry picked from commit 995386071f)

* gh-82054: allow test runner to split test_asyncio to execute in parallel by sharding. (#103927)

This runs test_asyncio sub-tests in parallel using sharding from Cinder. This suite is typically the longest-pole in runs because it is a test package with a lot of further sub-tests otherwise run serially. By breaking out the sub-tests as independent modules we can run a lot more in parallel.

After porting we can see the direct impact on a multicore system.

Without this change:
  Running make test is 5 min 26 seconds
With this change:
  Running make test takes 3 min 39 seconds

That'll vary based on system and parallelism. On a `-j 4` run similar to what CI and buildbot systems often do, it reduced the overall test suite completion latency by 10%.

The drawbacks are that this implementation is hacky and due to the sorting of the tests it obscures when the asyncio tests occur and involves changing CPython test infrastructure but, the wall time saved it is worth it, especially in low-core count CI runs as it pulls a long tail. The win for productivity and reserved CI resource usage is significant.

Future tests that deserve to be refactored into split up suites to benefit from are test_concurrent_futures and the way the _test_multiprocessing suite gets run for all start methods. As exposed by passing the -o flag to python -m test to get a list of the 10 longest running tests.

---------

Co-authored-by: Carl Meyer <carl@oddbird.net>
Co-authored-by: Gregory P. Smith <greg@krypto.org> [Google, LLC]
(cherry picked from commit 9e011e7c77)

* Display the sanitizer config in the regrtest header. (#105301)

Display the sanitizers present in libregrtest.

Having this in the CI output for tests with the relevant environment
variable displayed will help make it easier to do what we need to
create an equivalent local test run.

(cherry picked from commit 852348ab65)

* gh-101634: regrtest reports decoding error as failed test (#106169)

When running the Python test suite with -jN option, if a worker stdout
cannot be decoded from the locale encoding report a failed testn so the
exitcode is non-zero.

(cherry picked from commit 2ac3eec103)

* gh-108223: test.pythoninfo and libregrtest log Py_NOGIL (#108238)

Enable with --disable-gil --without-pydebug:

    $ make pythoninfo|grep NOGIL
    sysconfig[Py_NOGIL]: 1

    $ ./python -m test
    ...
    == Python build: nogil debug
    ...

(cherry picked from commit 5afe0c17ca)

* gh-90791: test.pythoninfo logs ASAN_OPTIONS env var (#108289)

* Cleanup libregrtest code logging ASAN_OPTIONS.
* Fix a typo on "ASAN_OPTIONS" vs "MSAN_OPTIONS".

(cherry picked from commit 3a1ac87f8f)

* gh-108388: regrtest splits test_asyncio package (#108393)

Currently, test_asyncio package is only splitted into sub-tests when
using command "./python -m test". With this change, it's also
splitted when passing it on the command line:
"./python -m test test_asyncio".

Remove the concept of "STDTESTS". Python is now mature enough to not
have to bother with that anymore. Removing STDTESTS simplify the
code.

(cherry picked from commit 174e9da083)

* regrtest computes statistics (#108793)

test_netrc, test_pep646_syntax and test_xml_etree now return results
in the test_main() function.

Changes:

* Rewrite TestResult as a dataclass with a new State class.
* Add test.support.TestStats class and Regrtest.stats_dict attribute.
* libregrtest.runtest functions now modify a TestResult instance
  in-place.
* libregrtest summary lists the number of run tests and skipped
  tests, and denied resources.
* Add TestResult.has_meaningful_duration() method.
* Compute TestResult duration in the upper function.
* Use time.perf_counter() instead of time.monotonic().
* Regrtest: rename 'resource_denieds' attribute to 'resource_denied'.
* Rename CHILD_ERROR to MULTIPROCESSING_ERROR.
* Use match/case syntadx to have different code depending on the
  test state.

Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com>
(cherry picked from commit d4e534cbb3)

* gh-108822: Add Changelog entry for regrtest statistics (#108821)

---------

Co-authored-by: Christian Heimes <christian@python.org>
Co-authored-by: Zachary Ware <zach@python.org>
Co-authored-by: Nikita Sobolev <mail@sobolevn.me>
Co-authored-by: Joshua Herman <zitterbewegung@gmail.com>
Co-authored-by: Gregory P. Smith <greg@krypto.org>
2023-09-03 19:21:53 +02:00

315 lines
12 KiB
Python
Raw Blame History

import netrc, os, unittest, sys, textwrap
from test.support import os_helper, run_unittest
try:
import pwd
except ImportError:
pwd = None
temp_filename = os_helper.TESTFN
class NetrcTestCase(unittest.TestCase):
def make_nrc(self, test_data):
test_data = textwrap.dedent(test_data)
mode = 'w'
if sys.platform != 'cygwin':
mode += 't'
with open(temp_filename, mode, encoding="utf-8") as fp:
fp.write(test_data)
try:
nrc = netrc.netrc(temp_filename)
finally:
os.unlink(temp_filename)
return nrc
def test_toplevel_non_ordered_tokens(self):
nrc = self.make_nrc("""\
machine host.domain.com password pass1 login log1 account acct1
default login log2 password pass2 account acct2
""")
self.assertEqual(nrc.hosts['host.domain.com'], ('log1', 'acct1', 'pass1'))
self.assertEqual(nrc.hosts['default'], ('log2', 'acct2', 'pass2'))
def test_toplevel_tokens(self):
nrc = self.make_nrc("""\
machine host.domain.com login log1 password pass1 account acct1
default login log2 password pass2 account acct2
""")
self.assertEqual(nrc.hosts['host.domain.com'], ('log1', 'acct1', 'pass1'))
self.assertEqual(nrc.hosts['default'], ('log2', 'acct2', 'pass2'))
def test_macros(self):
data = """\
macdef macro1
line1
line2
macdef macro2
line3
line4
"""
nrc = self.make_nrc(data)
self.assertEqual(nrc.macros, {'macro1': ['line1\n', 'line2\n'],
'macro2': ['line3\n', 'line4\n']})
# strip the last \n
self.assertRaises(netrc.NetrcParseError, self.make_nrc,
data.rstrip(' ')[:-1])
def test_optional_tokens(self):
data = (
"machine host.domain.com",
"machine host.domain.com login",
"machine host.domain.com account",
"machine host.domain.com password",
"machine host.domain.com login \"\" account",
"machine host.domain.com login \"\" password",
"machine host.domain.com account \"\" password"
)
for item in data:
nrc = self.make_nrc(item)
self.assertEqual(nrc.hosts['host.domain.com'], ('', '', ''))
data = (
"default",
"default login",
"default account",
"default password",
"default login \"\" account",
"default login \"\" password",
"default account \"\" password"
)
for item in data:
nrc = self.make_nrc(item)
self.assertEqual(nrc.hosts['default'], ('', '', ''))
def test_invalid_tokens(self):
data = (
"invalid host.domain.com",
"machine host.domain.com invalid",
"machine host.domain.com login log password pass account acct invalid",
"default host.domain.com invalid",
"default host.domain.com login log password pass account acct invalid"
)
for item in data:
self.assertRaises(netrc.NetrcParseError, self.make_nrc, item)
def _test_token_x(self, nrc, token, value):
nrc = self.make_nrc(nrc)
if token == 'login':
self.assertEqual(nrc.hosts['host.domain.com'], (value, 'acct', 'pass'))
elif token == 'account':
self.assertEqual(nrc.hosts['host.domain.com'], ('log', value, 'pass'))
elif token == 'password':
self.assertEqual(nrc.hosts['host.domain.com'], ('log', 'acct', value))
def test_token_value_quotes(self):
self._test_token_x("""\
machine host.domain.com login "log" password pass account acct
""", 'login', 'log')
self._test_token_x("""\
machine host.domain.com login log password pass account "acct"
""", 'account', 'acct')
self._test_token_x("""\
machine host.domain.com login log password "pass" account acct
""", 'password', 'pass')
def test_token_value_escape(self):
self._test_token_x("""\
machine host.domain.com login \\"log password pass account acct
""", 'login', '"log')
self._test_token_x("""\
machine host.domain.com login "\\"log" password pass account acct
""", 'login', '"log')
self._test_token_x("""\
machine host.domain.com login log password pass account \\"acct
""", 'account', '"acct')
self._test_token_x("""\
machine host.domain.com login log password pass account "\\"acct"
""", 'account', '"acct')
self._test_token_x("""\
machine host.domain.com login log password \\"pass account acct
""", 'password', '"pass')
self._test_token_x("""\
machine host.domain.com login log password "\\"pass" account acct
""", 'password', '"pass')
def test_token_value_whitespace(self):
self._test_token_x("""\
machine host.domain.com login "lo g" password pass account acct
""", 'login', 'lo g')
self._test_token_x("""\
machine host.domain.com login log password "pas s" account acct
""", 'password', 'pas s')
self._test_token_x("""\
machine host.domain.com login log password pass account "acc t"
""", 'account', 'acc t')
def test_token_value_non_ascii(self):
self._test_token_x("""\
machine host.domain.com login \xa1\xa2 password pass account acct
""", 'login', '\xa1\xa2')
self._test_token_x("""\
machine host.domain.com login log password pass account \xa1\xa2
""", 'account', '\xa1\xa2')
self._test_token_x("""\
machine host.domain.com login log password \xa1\xa2 account acct
""", 'password', '\xa1\xa2')
def test_token_value_leading_hash(self):
self._test_token_x("""\
machine host.domain.com login #log password pass account acct
""", 'login', '#log')
self._test_token_x("""\
machine host.domain.com login log password pass account #acct
""", 'account', '#acct')
self._test_token_x("""\
machine host.domain.com login log password #pass account acct
""", 'password', '#pass')
def test_token_value_trailing_hash(self):
self._test_token_x("""\
machine host.domain.com login log# password pass account acct
""", 'login', 'log#')
self._test_token_x("""\
machine host.domain.com login log password pass account acct#
""", 'account', 'acct#')
self._test_token_x("""\
machine host.domain.com login log password pass# account acct
""", 'password', 'pass#')
def test_token_value_internal_hash(self):
self._test_token_x("""\
machine host.domain.com login lo#g password pass account acct
""", 'login', 'lo#g')
self._test_token_x("""\
machine host.domain.com login log password pass account ac#ct
""", 'account', 'ac#ct')
self._test_token_x("""\
machine host.domain.com login log password pa#ss account acct
""", 'password', 'pa#ss')
def _test_comment(self, nrc, passwd='pass'):
nrc = self.make_nrc(nrc)
self.assertEqual(nrc.hosts['foo.domain.com'], ('bar', '', passwd))
self.assertEqual(nrc.hosts['bar.domain.com'], ('foo', '', 'pass'))
def test_comment_before_machine_line(self):
self._test_comment("""\
# comment
machine foo.domain.com login bar password pass
machine bar.domain.com login foo password pass
""")
def test_comment_before_machine_line_no_space(self):
self._test_comment("""\
#comment
machine foo.domain.com login bar password pass
machine bar.domain.com login foo password pass
""")
def test_comment_before_machine_line_hash_only(self):
self._test_comment("""\
#
machine foo.domain.com login bar password pass
machine bar.domain.com login foo password pass
""")
def test_comment_after_machine_line(self):
self._test_comment("""\
machine foo.domain.com login bar password pass
# comment
machine bar.domain.com login foo password pass
""")
self._test_comment("""\
machine foo.domain.com login bar password pass
machine bar.domain.com login foo password pass
# comment
""")
def test_comment_after_machine_line_no_space(self):
self._test_comment("""\
machine foo.domain.com login bar password pass
#comment
machine bar.domain.com login foo password pass
""")
self._test_comment("""\
machine foo.domain.com login bar password pass
machine bar.domain.com login foo password pass
#comment
""")
def test_comment_after_machine_line_hash_only(self):
self._test_comment("""\
machine foo.domain.com login bar password pass
#
machine bar.domain.com login foo password pass
""")
self._test_comment("""\
machine foo.domain.com login bar password pass
machine bar.domain.com login foo password pass
#
""")
def test_comment_at_end_of_machine_line(self):
self._test_comment("""\
machine foo.domain.com login bar password pass # comment
machine bar.domain.com login foo password pass
""")
def test_comment_at_end_of_machine_line_no_space(self):
self._test_comment("""\
machine foo.domain.com login bar password pass #comment
machine bar.domain.com login foo password pass
""")
def test_comment_at_end_of_machine_line_pass_has_hash(self):
self._test_comment("""\
machine foo.domain.com login bar password #pass #comment
machine bar.domain.com login foo password pass
""", '#pass')
@unittest.skipUnless(os.name == 'posix', 'POSIX only test')
@unittest.skipIf(pwd is None, 'security check requires pwd module')
@os_helper.skip_unless_working_chmod
def test_security(self):
# This test is incomplete since we are normally not run as root and
# therefore can't test the file ownership being wrong.
d = os_helper.TESTFN
os.mkdir(d)
self.addCleanup(os_helper.rmtree, d)
fn = os.path.join(d, '.netrc')
with open(fn, 'wt') as f:
f.write("""\
machine foo.domain.com login bar password pass
default login foo password pass
""")
with os_helper.EnvironmentVarGuard() as environ:
environ.set('HOME', d)
os.chmod(fn, 0o600)
nrc = netrc.netrc()
self.assertEqual(nrc.hosts['foo.domain.com'],
('bar', '', 'pass'))
os.chmod(fn, 0o622)
self.assertRaises(netrc.NetrcParseError, netrc.netrc)
with open(fn, 'wt') as f:
f.write("""\
machine foo.domain.com login anonymous password pass
default login foo password pass
""")
with os_helper.EnvironmentVarGuard() as environ:
environ.set('HOME', d)
os.chmod(fn, 0o600)
nrc = netrc.netrc()
self.assertEqual(nrc.hosts['foo.domain.com'],
('anonymous', '', 'pass'))
os.chmod(fn, 0o622)
self.assertEqual(nrc.hosts['foo.domain.com'],
('anonymous', '', 'pass'))
def test_main():
return run_unittest(NetrcTestCase)
if __name__ == "__main__":
test_main()
Reference in a new issue View git blame Copy permalink