The subprocess module in Python 3 uses io.open(file_descriptor, mode, bufsize) to create its Popen stdout, stderr and stdin file objects.

In Python 2, it used the old os.fdopen which created an old-style python 2 file object that simply wraps libc's FILE* interface.

This results in a behavior difference between Python 2 and Python 3 subprocesses as the bufsize=0 on io.open()ed files results in a RawIOBase file object whos read() or write() methods map directly to a single underlying system call.  ie: In Python 3 if you Popen.read(30000) and there are only 12345 bytes in the pipe from the child, it will return 12345 bytes rather than blocking while it makes further read() syscalls until it gets enough data or EOF as it would with the libc backed file objects in Python 2.

This tripped up the imaplib module in Issue17443.  (since fixed by explicitly enabling buffered I/O).

This behavior difference will make porting code to work on both Python 2 and 3 a bit more painful as bufsize=non-zero must be specified by the user for consistent behavior.

I'd like to fix this by changing the default bufsize=0 to bufsize=io.DEFAULT_BUFFER_SIZE, but only if I can do that in 3.2 and 3.3 .  If I can't, it _does_ seem like an API change, I'll just make a documentation update to mention the issue and the best practice for 2.x and 3.x compatibility.

[note: this issue does not apply to my subprocess32 backport running on python 2.x because that uses python2's the old style file os.fdopen]

marking release blocker to ask for comments from the 3.2 and 3.3 release managers on if i can consider changing the default subprocess.Popen bufsize parameter value or not.

Considering that this behavior has been present in all of Python 3, it does not seem useful to make a backwards incompatible change in bugfix releases.

You could change it in 3.4 as a minor API change, but you've indicated that that wouldn't make sense to you.

The number of things we'll break by changing this errant behavior to be _correct_ is way less than the number of things that are already broken due to it.

If the bufsize=0 default is left in place the behavior differs between Windows and POSIX platforms and anyone porting code from Python 2 will be caught by surprise.  The behavior change introduced in 3.2 with the new subprocess module was 100% unintentional and does not match that of 2.7 or of the subprocess32 backport to 2.x that more people use _today_ than use 3.x at all.

I think being a purist about this being an API change here is going to cause problems.  3.x hasn't seen wide adoption yet, its use is ramping up and code is being ported from 2.  The bug this behavior causes is completely non-obvious and escapes most testing.  It can occur at random times, more likely on loaded systems than idle ones.

Grepping through the standard library, there are several more instances of uses of subprocess that are suspect given this behavior change.  "Fixing" them all instead of addressing the cause of the problem seems unwise.

(actually I'm not sure about the windows vs posix behavior difference, that may not be true; I don't have a windows system handy to test that on)

OK. That was a more passionate statement :)  It would actually be nice to know about Windows, but the way you describe it is that the default behavior now is quite useless.  Does bufsize=0 have any sensible use on Python 3?

Anyway, you've convinced me -- can you fix this quickly so that it gets into the rc's?

great!  fixing now. :)

New changeset 3ecf709dfe69 by Gregory P. Smith in branch '3.2':
Fixes issue #17488: Change the subprocess.Popen bufsize parameter default value
http://hg.python.org/cpython/rev/3ecf709dfe69

New changeset 4c2fc172afcc by Gregory P. Smith in branch '3.3':
Fixes issue #17488: Change the subprocess.Popen bufsize parameter default value
http://hg.python.org/cpython/rev/4c2fc172afcc

New changeset 3031d69f94ef by Gregory P. Smith in branch 'default':
Fixes issue #17488: Change the subprocess.Popen bufsize parameter default value
http://hg.python.org/cpython/rev/3031d69f94ef

For the record, this issue seemed to forget about the effect of buffering the pipe to the subprocess’s input stream. Buffering an input pipe means that data is hidden away until it is flushed, and the close() method can raise a broken pipe error. I have sometimes found that forcing “bufsize=0” is easier than handling the extra broken pipe error.

Anyway I think the damage has already been done. However I did raise Issue 19622 to clarify the documentation.