Move stdin writing to a background thread in _communicate_streams_windows
to avoid blocking indefinitely when writing large input to a pipeline
where the subprocess doesn't consume stdin quickly.
This mirrors the fix made to Popen._communicate() for Windows in
commit 5b1862b (gh-87512).
Add test_pipeline_timeout_large_input to verify that TimeoutExpired
is raised promptly when run_pipeline() is called with large input
and a timeout, even when the first process is slow to consume stdin.
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
Extract the core selector-based I/O loop into a new _communicate_io_posix()
function that is shared by both _communicate_streams_posix() (used by
run_pipeline) and Popen._communicate() (used by Popen.communicate).
The new function:
- Takes a pre-configured selector and output buffers
- Supports resume via input_offset parameter (for Popen timeout retry)
- Returns (new_offset, completed) instead of raising TimeoutExpired
- Does not close streams (caller decides based on use case)
This reduces code duplication and ensures both APIs use the same
well-tested I/O multiplexing logic.
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
The comment suggested rewriting Popen._communicate() to use
non-blocking I/O on file objects now that Python 3's io module
is used instead of C stdio.
This is unnecessary - the current approach using select() to
detect ready fds followed by os.read()/os.write() is correct
and efficient. The selector already solves "when is data ready?"
so non-blocking mode would add complexity with no benefit.
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
Update the test to write 64KB to stderr from each process (128KB total)
instead of just small status messages. This better tests that the
multiplexed I/O handles concurrent large data on both stdout and stderr
without deadlocking.
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
Remove support for raw file descriptors in _communicate_streams(),
requiring all streams to be file objects. This simplifies both the
Windows and POSIX implementations by removing isinstance() checks
and fd-wrapping logic.
The run_pipeline() function now wraps the stderr pipe's read end
with os.fdopen() immediately after creation.
This change makes _communicate_streams() more compatible with
Popen.communicate() which already uses file objects, enabling
potential future refactoring to share the multiplexed I/O logic.
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
Add three tests that verify the multiplexed I/O implementation
properly handles large data volumes that would otherwise cause
pipe buffer deadlocks:
- test_pipeline_large_data_no_deadlock: 256KB through 2-stage pipeline
- test_pipeline_large_data_three_stages: 128KB through 3-stage pipeline
- test_pipeline_large_data_with_stderr: 64KB with concurrent stderr
These tests would timeout or deadlock without proper multiplexing.
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
Add _communicate_streams() helper function that properly multiplexes
read/write operations to prevent pipe buffer deadlocks. The helper
uses selectors on POSIX and threads on Windows, similar to
Popen.communicate().
This fixes potential deadlocks when large amounts of data flow through
the pipeline and significantly improves performance.
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
Document the new run_pipeline() function, PipelineResult class, and
PipelineError exception in the subprocess module documentation.
Includes:
- Function signature with stdin, stdout, stderr, capture_output, etc.
- Note about shared stderr pipe and text mode caveat for interleaved
multi-byte character sequences
- Note that universal_newlines is not supported (use text=True)
- Explanation that stdin connects to first process, stdout to last
- Usage examples showing basic pipelines, multi-command pipelines,
input handling, and error handling with check=True
- PipelineResult attributes: commands, returncodes, returncode,
stdout, stderr, and check_returncodes() method
- PipelineError attributes: commands, returncodes, stdout, stderr,
and failed list
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
Add a new run_pipeline() function to the subprocess module that enables
running multiple commands connected via pipes, similar to shell pipelines.
New API:
- run_pipeline(*commands, ...) - Run a pipeline of commands
- PipelineResult - Return type with commands, returncodes, stdout, stderr
- PipelineError - Raised when check=True and any command fails
Features:
- Supports arbitrary number of commands (minimum 2)
- capture_output, input, timeout, and check parameters like run()
- stdin= connects to first process, stdout= connects to last process
- Text mode support via text=True, encoding, errors
- All processes share a single stderr pipe for simplicity
- "pipefail" semantics: check=True fails if any command fails
Unlike run(), this function does not accept universal_newlines.
Use text=True instead.
Example:
result = subprocess.run_pipeline(
['cat', 'file.txt'],
['grep', 'pattern'],
['wc', '-l'],
capture_output=True, text=True
)
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
On Windows, Popen._communicate() previously wrote to stdin synchronously, which could block indefinitely if the subprocess didn't consume input= quickly and the pipe buffer filled up. The timeout= parameter was only checked when joining the reader threads, not during the stdin write.
This change moves the Windows stdin writing to a background thread (similar to how stdout/stderr are read in threads), allowing the timeout to be properly enforced. If timeout expires, TimeoutExpired is raised promptly and the writer thread continues in the background. Subsequent calls to communicate() will join the existing writer thread.
Adds test_communicate_timeout_large_input to verify that TimeoutExpired is raised promptly when communicate() is called with large input and a timeout, even when the subprocess doesn't consume stdin quickly.
This test already passed on POSIX (where select() is used) but failed on Windows where the stdin write blocks without checking the timeout.
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
Fix inconsistent subprocess.Popen.communicate() behavior between Windows
and POSIX when using memoryview objects with non-byte elements as input.
On POSIX systems, the code was incorrectly comparing bytes written against
element count instead of byte count, causing data truncation for large
inputs with non-byte element types.
Changes:
- Cast memoryview inputs to byte view when input is already a memoryview
- Fix progress tracking to use len(input_view) instead of len(self._input)
- Add comprehensive test coverage for memoryview inputs
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
* old-man-yells-at-ReST
* Update 2025-05-30-18-37-44.gh-issue-134453.kxkA-o.rst
* assertIsNone review feedback
* fix memoryview_nonbytes test to fail without our fix on main, and have a nicer error.
Thanks to Peter Bierma @ZeroIntensity for the code review.
* gh-141473: Fix subprocess.Popen.communicate to send input to stdin
* Docs: Clarify that `input` is one time only on `communicate()`
* NEWS entry
* Add a regression test.
---------
Co-authored-by: Gregory P. Smith <greg@krypto.org>
Added atomic operations to `scanner_begin()` and `scanner_end()` to prevent
race conditions on the `executing` flag in free-threaded builds. Also added
tests for concurrent usage of the `re` module.
Without the atomic operations, `test_scanner_concurrent_access()` triggers
`assert(self->executing)` failures, or a thread sanitizer run emits errors.
This simplifies the Lexical Analysis section on Names (but keeps it technically correct) by putting all the info about non-ASCII characters in a separate (and very technical) section.
It uses a mental model where the parser doesn't handle Unicode complexity “immediately”, but:
- parses any non-ASCII character (outside strings/comments) as part of a name, since these can't (yet) be e.g. operators
- normalizes the name
- validates the name, using the xid_start/xid_continue sets
Co-authored-by: Stan Ulbrych <89152624+StanFromIreland@users.noreply.github.com>
Co-authored-by: Blaise Pabon <blaise@gmail.com>
Co-authored-by: Micha Albert <info@micha.zone>
Co-authored-by: KeithTheEE <kmurrayis@gmail.com>
* gh-140550: PEP 793 reference documentation
Since the PEP calls for soft-deprecation of the existing initialization
function, this reorganizes the relevant docs to put the new way of
doing things first, and de-emphasize the old.
Some bits, like the tutorial, are left out of this patch. (See the
issue for a list.)
* Make Py_{SIZE,IS_TYPE,SET_SIZE} regular functions in stable ABI
Group them together with Py_TYPE & Py_SET_TYPE to cut down
on repetitive preprocessor macros.
Format repetitive definitions in object.c more concisely.
Py_SET_TYPE is still left out of the Limited API.
