classification
Title: PIPE_BUF is invalid on AIX
Type: behavior Stage: resolved
Components: Library (Lib) Versions: Python 3.2, Python 2.7
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: r.david.murray Nosy List: amaury.forgeotdarc, r.david.murray, sable
Priority: normal Keywords: patch

Created on 2010-09-15 16:30 by sable, last changed 2010-10-18 09:24 by sable. This issue is now closed.

Files
File name Uploaded Description Edit
patch_broken_pipe_buf.diff sable, 2010-09-20 16:21 review
patch_broken_pipe_buf_updated.diff sable, 2010-09-21 09:05 review
Messages (7)
msg116467 - (view) Author: Sébastien Sablé (sable) Date: 2010-09-15 16:30
On AIX, the test test_communicate_pipe_buf in test_subprocess will hang forever (in py3k and py27):

test_communicate_pipe_buf (__main__.ProcessTestCase) ... 


  File "Lib/test/test_subprocess.py", line 386, in test_communicate_pipe_buf
    (stdout, stderr) = p.communicate(string_to_write)
  File "/san_cis/home/cis/data/bamboo-home-agent-runtime/xml-data/build-dir/RTAIX30-SUP/Python-2.7-svn/Lib/subprocess.py", line 740, in communicate
    return self._communicate(input)
  File "/san_cis/home/cis/data/bamboo-home-agent-runtime/xml-data/build-dir/RTAIX30-SUP/Python-2.7-svn/Lib/subprocess.py", line 1257, in _communicate
    stdout, stderr = self._communicate_with_poll(input)
  File "/san_cis/home/cis/data/bamboo-home-agent-runtime/xml-data/build-dir/RTAIX30-SUP/Python-2.7-svn/Lib/subprocess.py", line 1320, in _communicate_with_poll
    input_offset += os.write(fd, chunk)
KeyboardInterrupt

The comment in this test indicates:
  # communicate() with writes larger than pipe_buf
  # This test will probably deadlock rather than fail, if
  # communicate() does not work properly.

So I guess it means communicate() does not work properly on AIX.
msg116932 - (view) Author: Sébastien Sablé (sable) Date: 2010-09-20 13:35
The problem does not happen with Python 2.6.
The difference is that:

* in Python 2.6,  the subprocess module would try to write at most 512 bytes
cf L1221 in subprocess.py
chunk = input[input_offset : input_offset + 512]

* in Python 2.7 and py3k, the subprocess module will try to write at most _PIPE_BUF bytes
cf L1319 in subprocess.py
chunk = input[input_offset : input_offset + _PIPE_BUF]

When forcing PIPE_BUF to 512 in selectmodule.c, the test will pass.
PIPE_BUF seems to be broken on AIX.
msg116952 - (view) Author: Sébastien Sablé (sable) Date: 2010-09-20 16:21
PIPE_BUF in unistd.h is defined to 32768.

The test works with PIPE_BUF changed up to 6144 but won't work with 7168.

We could probably use 4096 as a safe value for faster result if needed.
I just do not define the value and leave the default of 512 be used.

Here is my patch which has been verified to work OK and should be applied to py3k and py27.
msg117031 - (view) Author: Amaury Forgeot d'Arc (amaury.forgeotdarc) * (Python committer) Date: 2010-09-21 06:33
Removing select.PIPE_BUF does not seem a good idea to me because this breaks compatibility.  I suggest to simply set it to 512 on AIX.

(An ideal solution would be to really determine the actual buffer size in ./configure; this is probably overkill)
msg117040 - (view) Author: Sébastien Sablé (sable) Date: 2010-09-21 09:05
OK for me.
Here is a new patch that defines PIPE_BUF to 512 instead of removing select.PIPE_BUF.
msg118849 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2010-10-16 00:49
Committed to py3k in r85554, 2.7 in r85556.  Sébastien, from what you say it sounds like this does not apply to 3.1, so I blocked it there.  If this is incorrect let me know and I'll backport it.
msg118998 - (view) Author: Sébastien Sablé (sable) Date: 2010-10-18 09:24
Thanks R. David.

I checked in 3.1 and PIPE_BUF is not defined in the select module, so the default value of 512 is used in subprocess.
So no correction is needed for that version.
History
Date User Action Args
2010-10-18 09:24:10sablesetmessages: + msg118998
2010-10-16 00:49:07r.david.murraysetstatus: open -> closed
resolution: fixed
messages: + msg118849

stage: commit review -> resolved
2010-10-15 23:15:43r.david.murraysetassignee: r.david.murray

type: behavior
nosy: + r.david.murray
stage: commit review
2010-09-21 09:05:53sablesetfiles: + patch_broken_pipe_buf_updated.diff

messages: + msg117040
2010-09-21 06:33:13amaury.forgeotdarcsetnosy: + amaury.forgeotdarc
messages: + msg117031
2010-09-20 16:24:49sablesettitle: test_subprocess hangs on AIX -> PIPE_BUF is invalid on AIX
2010-09-20 16:21:26sablesetfiles: + patch_broken_pipe_buf.diff
keywords: + patch
messages: + msg116952
2010-09-20 13:35:57sablesetmessages: + msg116932
2010-09-15 16:30:33sablecreate