classification
Title: subprocess: more general (non-buffering) communication
Type: enhancement Stage: resolved
Components: Library (Lib) Versions: Python 3.5
process
Status: closed Resolution: wont fix
Dependencies: Superseder:
Assigned To: gregory.p.smith Nosy List: astrand, cvrebert, georg.brandl, giampaolo.rodola, gregory.p.smith, ianbicking, martin.panter, rosslagerwall
Priority: normal Keywords: patch

Created on 2005-08-15 19:15 by ianbicking, last changed 2020-10-20 01:58 by gregory.p.smith. This issue is now closed.

Files
File name Uploaded Description Edit
subprocess_communicate_alternative.py ianbicking, 2005-08-15 19:15
Messages (8)
msg54590 - (view) Author: Ian Bicking (ianbicking) * Date: 2005-08-15 19:15
Right now you can use subprocess.Popen.communicate() to
make communication with subprocesses much easier and
less likely to block than communicating directly with
.stdin, .stdout, etc.  However, that requires
completely buffering the input and output.

The functionality of communicate() (which is somewhat
complex because of platform issues) could be made more
general fairly easily.  The current functionality of
communicate could then be implemented in terms of that
new method.

I attached a function I'm using which does that for the
posix systems (basically turning Popen's posix
communicate into a function with some modifications). 
Replace "proc" with "self" (and give the function a
better name) and you'd have a method.

If patch 1175984 was accepted, then this wouldn't be
that much of an issue:
http://sourceforge.net/tracker/index.php?func=detail&aid=1175984&group_id=5470&atid=305470

msg54591 - (view) Author: Josiah Carlson (josiahcarlson) * (Python triager) Date: 2005-08-22 19:02
Logged In: YES 
user_id=341410

Would an asynchronous subprocess (which you would poll
manually) be better/sufficient?

http://python.org/sf/1191964
msg54592 - (view) Author: Josiah Carlson (josiahcarlson) * (Python triager) Date: 2005-08-22 19:03
Logged In: YES 
user_id=341410

Also, what you post is not a 'bug', it is a 'feature request'.
msg54593 - (view) Author: Georg Brandl (georg.brandl) * (Python committer) Date: 2005-08-22 19:38
Logged In: YES 
user_id=1188172

Reclassifying as RFE. Assigning to Peter.
msg161338 - (view) Author: Ross Lagerwall (rosslagerwall) (Python committer) Date: 2012-05-22 08:45
Closed issue14872 as a duplicate of this.
msg222926 - (view) Author: Mark Lawrence (BreamoreBoy) * Date: 2014-07-13 13:58
Please note the work being performed on #1191964 which was first referenced in msg54591.
msg245159 - (view) Author: Martin Panter (martin.panter) * (Python committer) Date: 2015-06-11 08:18
The non-blocking read and write features proposed in Issue 1191964 wouldn’t be sufficient. Perhaps they might be useful for implementing this feature on Windows though; I’m not sure. And a non-blocking write on Posix may be slightly more efficient as well, removing the limitation of copying in chunks of PIPE_BUF bytes.

Ian’s original patch posted here proposes a version of communicate() which accepts a file reader for the subprocess’s input, and file writers for the subprocess’s outputs. Another option could be something like my SubprocessWriter class <https://github.com/vadmium/pacman-tools/blob/9ffdd88/makeaur#L179>. It provides a file writer interface for the caller to write to the subprocess’s input, while copying data from the subprocess’s output behind the scenes. I used it to stream a tar file, as it is written by the “tarfile” module, into a GPG subprocess to create a digital signature, while writing the output of GPG into a BytesIO buffer. Using Unix command pipeline pseudocode, it is used a bit like this:

tarfile.open(mode="w|") | Popen("gpg") | BytesIO()
msg379076 - (view) Author: Gregory P. Smith (gregory.p.smith) * (Python committer) Date: 2020-10-20 01:58
since the time this was filed, subprocess has evolved a lot and third party options for child process have appeared as well as modern things like:

https://docs.python.org/3/library/asyncio-subprocess.html (stdlib)

https://trio.readthedocs.io/en/stable/reference-io.html?highlight=subprocess#options-for-starting-subprocesses (trio, easier async framework to use than asyncio)

plumbing stdin/stdout/stderr data to python file-like objects from the stdlib subprocess APIs themselves is better left to third party solutions building on top of the subprocess module today.
History
Date User Action Args
2020-10-20 01:58:46gregory.p.smithsetstatus: open -> closed

assignee: gregory.p.smith

nosy: + gregory.p.smith
messages: + msg379076
resolution: wont fix
stage: test needed -> resolved
2019-05-02 04:04:48josiahcarlsonsetnosy: - josiahcarlson
2019-03-15 21:59:48BreamoreBoysetnosy: - BreamoreBoy
2016-09-25 13:19:04christian.heimesunlinkissue1175984 dependencies
2015-06-11 08:18:50martin.pantersetnosy: + martin.panter
messages: + msg245159
2014-07-13 13:58:56BreamoreBoysetnosy: + BreamoreBoy

messages: + msg222926
versions: + Python 3.5, - Python 3.2
2012-05-22 21:04:31cvrebertsetnosy: + cvrebert
2012-05-22 09:08:14giampaolo.rodolasetnosy: + giampaolo.rodola
2012-05-22 08:45:43rosslagerwallsetassignee: astrand -> (no value)

messages: + msg161338
nosy: + rosslagerwall
2012-05-22 08:45:11rosslagerwalllinkissue14872 superseder
2010-08-21 12:51:31BreamoreBoysetversions: - Python 2.7
2009-05-12 17:48:05ajaksu2setkeywords: + patch
stage: test needed
versions: + Python 2.7, Python 3.2, - Python 2.6
2009-02-15 22:15:42ajaksu2linkissue1175984 dependencies
2008-01-05 19:32:38christian.heimessetversions: + Python 2.6
2005-08-15 19:15:12ianbickingcreate