classification
Title: subprocess.Popen unbuffered not work
Type: behavior Stage:
Components: Library (Lib) Versions: Python 3.1, Python 3.2
process
Status: closed Resolution: duplicate
Dependencies: Superseder:
Assigned To: Nosy List: gregory.p.smith, ocean-city, pitrou, r.david.murray, rosslagerwall
Priority: normal Keywords: patch

Created on 2010-09-23 18:15 by ocean-city, last changed 2011-03-27 08:59 by rosslagerwall. This issue is now closed.

Files
File name Uploaded Description Edit
py3k_fix_unbuffered_in_subprocess.patch ocean-city, 2010-09-23 18:15
py3k_fix_unbuffered_in_subprocess_v2.patch ocean-city, 2010-10-19 14:28 review
py3k_fix_unbuffered_in_subprocess_v3.patch ocean-city, 2010-10-19 14:34 review
Messages (11)
msg117213 - (view) Author: Hirokazu Yamamoto (ocean-city) * (Python committer) Date: 2010-09-23 18:15
Following script hangs on Python3.x.

from subprocess import *
import sys

p = Popen([sys.executable, "-c", "import sys; print(sys.stdin.read(1))"], stdin=PIPE)
p.stdin.write(b'x')
p.wait()

This is because unbuffered functionality of
subprocess.Popen is disabled. Is this still needed?

I confirmed test_subprocess passes with the attached
patch.
msg117214 - (view) Author: Hirokazu Yamamoto (ocean-city) * (Python committer) Date: 2010-09-23 18:23
Interestingly, following code also hangs. Maybe I'm
misunderstanding the meaning of *bufsize*....

from subprocess import *
import sys

p = Popen([sys.executable, "-c", "import sys; print(sys.stdin.read(1))"], stdin=PIPE)
p.stdin.write(b'xx') # size 2
p.wait()
msg117279 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2010-09-24 12:47
You are right that it doesn't work anymore (under Linux too).

Your patch is almost ok. It should raise a ValueError if bufsize=0 and universal_newlines is true (the two can't be satisfied together, since universal newlines implies buffering).

It would also be better to add a test in test_subprocess.

As a sidenote, bufsize=1 (line buffering) doesn't work anymore either, and this can't be fixed without breaking compatibility.
msg117280 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2010-09-24 12:49
As another sidenote, `bufsize` should only be relevant for stdin. For stdout and stderr, disabling buffering will only reduce performance, not add functionality. So I would suggest only setting buffering on stding.
msg117405 - (view) Author: Hirokazu Yamamoto (ocean-city) * (Python committer) Date: 2010-09-26 06:13
Antoine, can you create the patch? I'm not familiar with IO.
msg117461 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2010-09-27 18:53
But if line buffering doesn't work, disabling buffering on stdout/stderr does have a functional consequence: it allows process output to appear as generated instead of coming in chunks when the buffer is full.  Of course, I could be completely misunderstanding what you are talking about :)
msg117462 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2010-09-27 18:59
> But if line buffering doesn't work, disabling buffering on 
> stdout/stderr does have a functional consequence: it allows process 
> output to appear as generated instead of coming in chunks when the
> buffer is full

Yes, sorry, I had it backwards. It's buffering on stdin which doesn't make any functional difference (whether it's buffered or not, you always get data as soon as it arrives).
msg117501 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2010-09-28 02:07
> > But if line buffering doesn't work, disabling buffering on 
> > stdout/stderr does have a functional consequence: it allows process 
> > output to appear as generated instead of coming in chunks when the
> > buffer is full
> 
> Yes, sorry, I had it backwards. It's buffering on stdin which doesn't
> make any functional difference (whether it's buffered or not, you
> always get data as soon as it arrives).

Actually, I had it right (you confused me :-)). In the context of
subprocess, you write to the child's stdin pipe, and you read from the
child's stdout and stderr pipes. So, whether or not you buffer the reads
from stdout and stderr pipes doesn't make a difference (except in
performance): as soon as the child outputs a single byte, it becomes
available for the parent. But if you buffer the writes to stdin, the
child process will see data arrive only when the buffer is flushed.

Here is the relevant code in subprocess.py:

        if p2cwrite != -1:
            self.stdin = io.open(p2cwrite, 'wb', bufsize)
            if self.universal_newlines:
                self.stdin = io.TextIOWrapper(self.stdin)
        if c2pread != -1:
            self.stdout = io.open(c2pread, 'rb', bufsize)
            if universal_newlines:
                self.stdout = io.TextIOWrapper(self.stdout)
        if errread != -1:
            self.stderr = io.open(errread, 'rb', bufsize)
            if universal_newlines:
                self.stderr = io.TextIOWrapper(self.stderr)

Only stdin is opened in write mode.
msg119138 - (view) Author: Hirokazu Yamamoto (ocean-city) * (Python committer) Date: 2010-10-19 14:28
I implemented msg117279 with v2 patch. Can I commit it?

# If your are already working on this issue, please ignore
# my patch.
msg119139 - (view) Author: Hirokazu Yamamoto (ocean-city) * (Python committer) Date: 2010-10-19 14:34
Umm, v2 patch broke test_subprocess. I'll repost v3 patch
after removing "bufsize=0 && universal_newlines" check.

======================================================================
ERROR: test_universal_newlines (__main__.ProcessTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "e:\python-dev\py3k\lib\test\test_subprocess.py", line 422, in test_unive
rsal_newlines
    universal_newlines=1)
  File "e:\python-dev\py3k\lib\subprocess.py", line 621, in __init__
    raise ValueError("cannot use bufsize=0 with universal newlines")
ValueError: cannot use bufsize=0 with universal newlines

======================================================================
ERROR: test_universal_newlines_communicate (__main__.ProcessTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "e:\python-dev\py3k\lib\test\test_subprocess.py", line 442, in test_unive
rsal_newlines_communicate
    universal_newlines=1)
  File "e:\python-dev\py3k\lib\subprocess.py", line 621, in __init__
    raise ValueError("cannot use bufsize=0 with universal newlines")
ValueError: cannot use bufsize=0 with universal newlines
msg132295 - (view) Author: Ross Lagerwall (rosslagerwall) (Python committer) Date: 2011-03-27 08:59
Unbuffered subprocess was fixed in 1dc52ecb8949
Closing this as a duplicate of #11459.
History
Date User Action Args
2011-03-27 08:59:09rosslagerwallsetstatus: open -> closed

nosy: + rosslagerwall
messages: + msg132295

resolution: duplicate
2010-10-19 14:34:32ocean-citysetfiles: + py3k_fix_unbuffered_in_subprocess_v3.patch

messages: + msg119139
2010-10-19 14:28:35ocean-citysetfiles: + py3k_fix_unbuffered_in_subprocess_v2.patch

messages: + msg119138
2010-09-28 02:07:02pitrousetmessages: + msg117501
2010-09-27 18:59:03pitrousetmessages: + msg117462
2010-09-27 18:53:53r.david.murraysetnosy: + r.david.murray
messages: + msg117461
2010-09-26 06:13:48ocean-citysetmessages: + msg117405
2010-09-24 12:49:58pitrousetmessages: + msg117280
2010-09-24 12:47:46pitrousettitle: subprocess.Popen unbuffered not work (windows) -> subprocess.Popen unbuffered not work
nosy: + gregory.p.smith, pitrou

messages: + msg117279

components: + Library (Lib), - Windows
type: behavior
2010-09-23 18:23:20ocean-citysetmessages: + msg117214
2010-09-23 18:15:40ocean-citycreate