It would be terribly nice if the Popen class in the
subprocess module would allow a programmer to easily
say "send some data right now (if I have some to send)
and receive whatever information is available right
now".  Essentially the equivalent of
asyncore.loop(count=1), only that it returns the data
received, instead of placing it in a buffer.

Why would this functionality be useful?  Currently,
when using the subprocess module with pipes, the
interaction with a pipe is limited to "send data if
desired, close the subprocess' stdin, read all output
from the subprocess' stdout, ...". 

Certainly one can pull the pipes out of the Popen
instance, and perform the necessary functions on posix
systems (with select or poll), but the additional magic
on WIndows is a little less obvious (but still possible).

There is a post by Paul Du Bois with an example using
win32pipe.PeekNamedPipe:
http://groups-beta.google.com/group/comp.lang.python/msg/115e9332cc1ca09d?hl=en

And a message from Neil Hodgeson stating that
PeekNamedPipe works on anonymous pipes:
http://mail.python.org/pipermail/python-dev/2000-May/004200.html


With this modification, creating Expect-like modules
for any platform, as well as embedded shells inside any
GUI with a text input widget, becomes easy.  Heck, even
Idle could use it rather than a socket for its
interactive interpreter.

Logged In: YES 
user_id=473331

More control of sub-processes would be great.  Several times
in the past few years I've done stuff with os.system,
popenN, or spawn* and it would be useful to have even more
cross-platform consistency and support in this area. 
Anyway, this is just a vote to encurage other developers.

Logged In: YES 
user_id=341410

I've got a version of subprocess that has this functionality
with pywin32.  Making it work on *nix systems with usable
select is trivial.

About the only question is whether the functionality is
desireable, and if so, what kind of API is reasonable.

Perhaps adding an optional argument 'wait_for_completion' to
the communicate method, which defaults to True for executing
what is currently in the body of communicate.

If the 'wait_for_completion' argument is untrue, then it
does non-waiting asynchronous IO, returning either a 2 or
3-tuple on completion.  If input is None, it is a 2-tuple of
(stdout, stderr).  If input is a string, it is a 3-tuple of
(bytes_sent, stdout, stderr).

How does that sound?

Logged In: YES 
user_id=341410

I suppose I should mention one side-effect of all this.  It
requires three more functions from pywin32 be included in
the _subprocess driver; win32file.ReadFile,
win32file.WriteFile, and win32pipe.PeekNamedPipe .  Read and
Peek are for reading data from stdout and stderr, and Write
is for the support for partial writes to stdin.

Logged In: YES 
user_id=341410

How about if subprocesses have 3 new methods, send(input),
recv(maxlen), and recv_stderr(maxlen).

send(input) would perform like socket.send(), sending as
much as it currently can, returning the number of bytes sent.
recv(maxlen) and recv_stderr(maxlen) would recieve up to the
provided number of bytes from the stdout or stderr pipes
respectively.

I currently have an implementation of the above on Windows
and posix.  I include the context diff against revision 1.20
of subprocess.py in current CVS.

Logged In: YES 
user_id=341410

I've implemented this as a subclass of subprocess.Popen in
the Python cookbook, available here:
http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/440554

Essentially this is a request for inclusion in the standard
library for Python 2.5 .

Logged In: YES 
user_id=341410

I've implemented this as a subclass of subprocess.Popen in
the Python cookbook, available here:
http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/440554

I would also like to see this feature. I'm using Josiah Carlson's recipe for the time being. I'm agnostic about whether the asynchronicity is a "mode" or just a case of using different functions. However, if the former is chosen, then one should be able to switch modes at will, because sometimes I want blocking and sometimes I don't.

The way subprocess is currently written, if you were to use a blocking semantic, you would no longer be able to use an asynchronous semantic afterwards (it will read the result until there is nothing more to read).  Note that this is the case whether it is mode or method based.

If there were a blocking "read x bytes" type call, could you not do some non-blocking stuff afterwards? If you use the "read" method, with a byte limit, directly on the stdout member of Popen, that could return with your bytes, and then you do some non-blocking stuff afterwards. That's the external view of course, I suspect that you're saying there's some internal lower-level reason this is currently not possible.

Thanks for your interim class BTW, it is proving to be very useful.

Unless you use PeekNamedPipe on Windows, there is no guarantee that the pipe will return *any* data until the process ends, even if you say pipe.read(1) and there is 1k of data already sent.

Note that the two async reading methods that I provide (recv() and recv_err()) take a 'maximum bytes' argument.  Just like I have written a 'send_all()' utility function, I (or you) can easily write a 'recv_exact()' utility function that receives an exact number of bytes before returning.  That functionality would be more or less required for one of the expected use-cases I specify in the recipe, "writing a multi-platform 'expect' module".

Stick with async calls (with the utility recv_exact()) until you need to use the .communicate() method.

Josiah: can this be closed?

I don't believe this should be closed.  The functionality is still 
desired by me and others who have posted on and off since the patch was 
created.  This patch definitely needs some love (tests mostly).

Hello, I am working on this patch and some additional features for my
Google Summer of Code project (subdev.blogspot.com) and will eventually
attempt to get the code committed to Python 3.1 or 3.2 and Python 2.7. I
will have the unit tests completed shortly and will post a patch for
Python 2.7 and / or Python 3.1rc1.

Retargeting to 3.2 since 3.1 is now feature-frozen.

PEP 3145 has been written in response to this request.

What is the status? What is required to get this into 3.3? See also issue14872.

Comments on Josiah's patch:

* It uses pywin32 for PeekNamedPipe -- this is now available from _winapi.

* I don't think send(), recv() and recv_exact() will work correctly if
  buffering is used -- an error should be raised in this case.

* I think send(), recv(), recv_exact() should raise errors if the
  associated stream is None instead of returning 0 or ''.


Additional comments on http://code.google.com/p/subprocdev

* In a few places the code does "raise Exception(...)" -- some other 
  exception class should be used.

* It is not clear to me why TextIOWrapper should try to implement seek()
  and tell().  (The file-like wrapper for a socket does not.)

* I don't like the hardwired timeouts for faking asynchronicity:
  Popen.asyncread(), Popen.asyncwrite() use 0.1 seconds, and
  TextIOWrapper.read() uses 1.25 seconds.

* There is no binary counterpart to TextIOWrapper.

* Using fcntl.fcntl() to switch back and forth between blocking and
  unblocking is unnecessary.  select() guarantees the read/write on a pipe
  will succeed without blocking.  (The same is not necessarily true with
  a socket.)

* __closecheck() should be renamed _closecheck().

* FileWrapper is just a wrapper for TextIOWrapper, except that it has an
  ignored newlines argument.

* I can't see any documentation.

Personally, I would factor out the code for Popen.communicate() in to a Communicator class which wraps a Popen object and has a method

    communicate(input, timeout=None) -> (bytes_written, output, error)

On Windows this would use threads, and on Unix, select.

There's documentation, but you have to switch to the Python 3k branch --
http://code.google.com/p/subprocdev/source/browse/?name=python3k#hg%2Fdoc.

As for the other criticisms, I'm sure there are plenty of things that need to
be improved upon; I was not a very experienced when I started the project.

> Personally, I would factor out the code for Popen.communicate() in to a > Communicator class which wraps a Popen object and has a method
>
>    communicate(input, timeout=None) -> (bytes_written, output, error)

How would this differ from the normal communicate()?

It seems like there are two different ideas for why people want an "asynchronous subprocess":

One is that they want to use communicate() but not be limited by memory issues.
I think a good API for this case is an asyncore style API or like the one from the patch in issue1260171.

Another use case is for an expect-type interface where you read and write based on a timeout or some kind of delimiter like a newline.

These should probably be addressed independently.

See also issue10482.

> How would this differ from the normal communicate()?

It would block until one of the following occurs:

* some data has been written to stdin,
* some data has been read from stdout or stderr, or
* timeout passes (if timeout is not None).

The normal communicate() could be implemented by running this in a loop.  The amount of data returned at once could be limited by an argument of the constructor.

Can anybody write a sum up on this?