It would be terribly nice if the Popen class in the
subprocess module would allow a programmer to easily
say "send some data right now (if I have some to send)
and receive whatever information is available right
now".  Essentially the equivalent of
asyncore.loop(count=1), only that it returns the data
received, instead of placing it in a buffer.

Why would this functionality be useful?  Currently,
when using the subprocess module with pipes, the
interaction with a pipe is limited to "send data if
desired, close the subprocess' stdin, read all output
from the subprocess' stdout, ...". 

Certainly one can pull the pipes out of the Popen
instance, and perform the necessary functions on posix
systems (with select or poll), but the additional magic
on WIndows is a little less obvious (but still possible).

There is a post by Paul Du Bois with an example using
win32pipe.PeekNamedPipe:
http://groups-beta.google.com/group/comp.lang.python/msg/115e9332cc1ca09d?hl=en

And a message from Neil Hodgeson stating that
PeekNamedPipe works on anonymous pipes:
http://mail.python.org/pipermail/python-dev/2000-May/004200.html


With this modification, creating Expect-like modules
for any platform, as well as embedded shells inside any
GUI with a text input widget, becomes easy.  Heck, even
Idle could use it rather than a socket for its
interactive interpreter.

Logged In: YES 
user_id=473331

More control of sub-processes would be great.  Several times
in the past few years I've done stuff with os.system,
popenN, or spawn* and it would be useful to have even more
cross-platform consistency and support in this area. 
Anyway, this is just a vote to encurage other developers.

Logged In: YES 
user_id=341410

I've got a version of subprocess that has this functionality
with pywin32.  Making it work on *nix systems with usable
select is trivial.

About the only question is whether the functionality is
desireable, and if so, what kind of API is reasonable.

Perhaps adding an optional argument 'wait_for_completion' to
the communicate method, which defaults to True for executing
what is currently in the body of communicate.

If the 'wait_for_completion' argument is untrue, then it
does non-waiting asynchronous IO, returning either a 2 or
3-tuple on completion.  If input is None, it is a 2-tuple of
(stdout, stderr).  If input is a string, it is a 3-tuple of
(bytes_sent, stdout, stderr).

How does that sound?

Logged In: YES 
user_id=341410

I suppose I should mention one side-effect of all this.  It
requires three more functions from pywin32 be included in
the _subprocess driver; win32file.ReadFile,
win32file.WriteFile, and win32pipe.PeekNamedPipe .  Read and
Peek are for reading data from stdout and stderr, and Write
is for the support for partial writes to stdin.

Logged In: YES 
user_id=341410

How about if subprocesses have 3 new methods, send(input),
recv(maxlen), and recv_stderr(maxlen).

send(input) would perform like socket.send(), sending as
much as it currently can, returning the number of bytes sent.
recv(maxlen) and recv_stderr(maxlen) would recieve up to the
provided number of bytes from the stdout or stderr pipes
respectively.

I currently have an implementation of the above on Windows
and posix.  I include the context diff against revision 1.20
of subprocess.py in current CVS.

Logged In: YES 
user_id=341410

I've implemented this as a subclass of subprocess.Popen in
the Python cookbook, available here:
http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/440554

Essentially this is a request for inclusion in the standard
library for Python 2.5 .

Logged In: YES 
user_id=341410

I've implemented this as a subclass of subprocess.Popen in
the Python cookbook, available here:
http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/440554

I would also like to see this feature. I'm using Josiah Carlson's recipe for the time being. I'm agnostic about whether the asynchronicity is a "mode" or just a case of using different functions. However, if the former is chosen, then one should be able to switch modes at will, because sometimes I want blocking and sometimes I don't.

The way subprocess is currently written, if you were to use a blocking semantic, you would no longer be able to use an asynchronous semantic afterwards (it will read the result until there is nothing more to read).  Note that this is the case whether it is mode or method based.

If there were a blocking "read x bytes" type call, could you not do some non-blocking stuff afterwards? If you use the "read" method, with a byte limit, directly on the stdout member of Popen, that could return with your bytes, and then you do some non-blocking stuff afterwards. That's the external view of course, I suspect that you're saying there's some internal lower-level reason this is currently not possible.

Thanks for your interim class BTW, it is proving to be very useful.

Unless you use PeekNamedPipe on Windows, there is no guarantee that the pipe will return *any* data until the process ends, even if you say pipe.read(1) and there is 1k of data already sent.

Note that the two async reading methods that I provide (recv() and recv_err()) take a 'maximum bytes' argument.  Just like I have written a 'send_all()' utility function, I (or you) can easily write a 'recv_exact()' utility function that receives an exact number of bytes before returning.  That functionality would be more or less required for one of the expected use-cases I specify in the recipe, "writing a multi-platform 'expect' module".

Stick with async calls (with the utility recv_exact()) until you need to use the .communicate() method.

Josiah: can this be closed?

I don't believe this should be closed.  The functionality is still 
desired by me and others who have posted on and off since the patch was 
created.  This patch definitely needs some love (tests mostly).

Hello, I am working on this patch and some additional features for my
Google Summer of Code project (subdev.blogspot.com) and will eventually
attempt to get the code committed to Python 3.1 or 3.2 and Python 2.7. I
will have the unit tests completed shortly and will post a patch for
Python 2.7 and / or Python 3.1rc1.

Retargeting to 3.2 since 3.1 is now feature-frozen.

PEP 3145 has been written in response to this request.