This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: python3.2 AFTER b2 release has subprocess.Popen broken under colinux/windows
Type: behavior Stage:
Components: Interpreter Core, IO Versions: Python 3.2
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: gregory.p.smith Nosy List: georg.brandl, gregory.p.smith, kaizhu, loewis
Priority: release blocker Keywords:

Created on 2011-01-02 00:07 by kaizhu, last changed 2022-04-11 14:57 by admin. This issue is now closed.

Files
File name Uploaded Description Edit
trace2.txt kaizhu, 2011-01-02 14:08 strace of pipe2 system call
Messages (13)
msg125020 - (view) Author: kai zhu (kaizhu) Date: 2011-01-02 00:07
i have 2 debian i386 unstable distros.
1) python3.2 (latest hg) running under vps @ linode.com seems ok
2) python3.2 (latest hg) running under colinux (in windows xp) breaks

*NOTE
3) python3.2 (release b2) works fine under colinux

public@colinux: python3.2
Python 3.2b2+ (py3k, Jan  1 2011, 17:42:23)
[GCC 4.4.5] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import subprocess
>>> subprocess.Popen('ls')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/public/i486-pc-linux-gnu/lib/python3.2/subprocess.py", line 708, in __init__
    restore_signals, start_new_session)
  File "/home/public/i486-pc-linux-gnu/lib/python3.2/subprocess.py", line 1136, in _execute_child
    errpipe_read, errpipe_write = _create_pipe()
OSError: [Errno 38] Function not implemented
>>>
msg125028 - (view) Author: Gregory P. Smith (gregory.p.smith) * (Python committer) Date: 2011-01-02 02:58
The only thing that could cause this error is if the Modules/_posixsubprocess.c subprocess_cloexec_pipe function fails either in the pipe2() call or if HAVE_PIPE2 is not defined one of the fcntl() calls.

Can you look to see if HAVE_PIPE2 is defined?

I suspect it is but the colinux/windows environment should apperently not define it.  configure.in magic will be needed to make sure it does not get defined there.

Is your bugreport accurate?  This function was included in 3.2b2 so the failure should be the same in both versions.  As far as I can see nothing else has changed that should impact that.  Did you rerun configure properly on your colinux install?
msg125036 - (view) Author: kai zhu (kaizhu) Date: 2011-01-02 08:59
tested w/ following debug code.  looks like greg is correct - HAVE_PIPE2 should NOT b defined under colinux.

diff -r 6fa1e3b94d8f Modules/_posixsubprocess.c
--- a/Modules/_posixsubprocess.c        Sat Jan 01 22:18:46 2011 +0100
+++ b/Modules/_posixsubprocess.c        Sun Jan 02 03:48:47 2011 -0500
@@ -412,10 +412,12 @@
     int fds[2];
     int res;
 #ifdef HAVE_PIPE2
+    PyErr_Format(PyExc_RuntimeError, "HAVE_PIPE2 = %i, O_CLOEXEC = %i", HAVE_PIPE2, O_CLOEXEC); return NULL;
     Py_BEGIN_ALLOW_THREADS
     res = pipe2(fds, O_CLOEXEC);
     Py_END_ALLOW_THREADS
 #else
+    PyErr_Format(PyExc_RuntimeError, "HAVE_PIPE2 not defined, O_CLOEXEC = %i", O_CLOEXEC); return NULL;
     /* We hold the GIL which offers some protection from other code calling
      * fork() before the CLOEXEC flags have been set but we can't guarantee
      * anything without pipe2(). */



b2 release:
./python -c 'import _posixsubprocess; _posixsubprocess.cloexec_pipe()'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
RuntimeError: HAVE_PIPE2 not defined, O_CLOEXEC = 524288



latest hg:
./python -c 'import _posixsubprocess; _posixsubprocess.cloexec_pipe()'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
RuntimeError: HAVE_PIPE2 = 1, O_CLOEXEC = 524288
msg125037 - (view) Author: kai zhu (kaizhu) Date: 2011-01-02 09:04
i used the same almost vanilla configure for both:

$ ./configure --prefix=$USERPATH; make
msg125039 - (view) Author: Martin v. Löwis (loewis) * (Python committer) Date: 2011-01-02 09:56
Can you please run the script under strace, and report what system call is not implemented? I.e. put

  import subprocess
  subprocess.Popen('ls')

into a file (foo.py), then run

  strace -o trace.txt python foo.py

Please attach the output in case you cannot identify the problem.
msg125048 - (view) Author: kai zhu (kaizhu) Date: 2011-01-02 13:31
the culprit was my colinux kernel (2.6.26.8-co-0.7.7.1) did not have pipe2 support (which libc erronenously assumed). updating the kernel fixed the problem.

the libc issue is partially discussed @ http://www.0x61.com/forum/linux-kernel-f109/popen2-popen-call-t1229012.html.  according to manpage pipe2 was not added to the kernel til 2.6.27

my guess is this affects all unstable debian systems which are running kernels 2.6.26 or older (u decide whether or not using old kernels for debian unstable is an edge case ;)
msg125051 - (view) Author: kai zhu (kaizhu) Date: 2011-01-02 14:08
hi martin, did an strace & the 'not implemented' system call was pipe2()

pipe2 exists in libc (checked w/ ctypes), but is broken for old linux kernels as mentioned previously.
msg125052 - (view) Author: Georg Brandl (georg.brandl) * (Python committer) Date: 2011-01-02 14:13
Does not seem to be a Python problem then.  Thanks for diagnosing!
msg125089 - (view) Author: Martin v. Löwis (loewis) * (Python committer) Date: 2011-01-02 20:13
If more people report this, there is still something Python could do:

- the configure test could verify that the running kernel actually implements the system call, and undefine HAVE_PIPE2 if that's not the case. Of course this would only help if the resulting binary only ever runs on the same system, and if the kernel is only ever upgraded.
- the test could be deferred to run-time, having subprocess_cloexec_pipe fall back to the pipe()/fcntl() branch if the system call fails with ENOSYS.
msg125092 - (view) Author: Gregory P. Smith (gregory.p.smith) * (Python committer) Date: 2011-01-02 20:17
This bug has brought up a broader issue.  the pipe2 syscall on Linux is very new.  It is perfectly reasonable to expect a Python binary will be compiled against a C library that has a pipe2() function but run on a system with an older (pre 2.6.27) linux kernel.  I have many systems like that at work.

I'm going to make subprocess_cloexec_pipe check for a not implemented error from the pipe2 call and use the fallback pipe+fcntl code in that case instead.
msg125093 - (view) Author: Gregory P. Smith (gregory.p.smith) * (Python committer) Date: 2011-01-02 20:17
This bug has brought up a broader issue.  the pipe2 syscall on Linux is very new.  It is perfectly reasonable to expect a Python binary will be compiled against a C library that has a pipe2() function but run on a system with an older (pre 2.6.27) linux kernel.  I have many systems like that at work.

I'm going to make subprocess_cloexec_pipe check for a not implemented error from the pipe2 call and use the fallback pipe+fcntl code in that case instead.
msg125097 - (view) Author: Gregory P. Smith (gregory.p.smith) * (Python committer) Date: 2011-01-02 20:54
fixed r87651.

kaizhu if you still have the ability to run your colinux install with an old 2.6.26 kernel i'd appreciate it if you could explicitly test that the change does the right thing.
msg125110 - (view) Author: kai zhu (kaizhu) Date: 2011-01-02 21:54
re-tested under old 2.6.26 kernel using previous foo.py example:

1. unpatched python3.2 broken as expected
2. patched   python3.2 now works :)

strace confirms ENOSYS being raised from pipe2() in both cases
History
Date User Action Args
2022-04-11 14:57:10adminsetgithub: 55011
2011-01-02 21:54:04kaizhusetnosy: loewis, georg.brandl, gregory.p.smith, kaizhu
messages: + msg125110
2011-01-02 20:54:21gregory.p.smithsetstatus: open -> closed

messages: + msg125097
resolution: fixed
nosy: loewis, georg.brandl, gregory.p.smith, kaizhu
2011-01-02 20:17:53gregory.p.smithsetstatus: closed -> open

messages: + msg125093
resolution: works for me -> (no value)
nosy: loewis, georg.brandl, gregory.p.smith, kaizhu
2011-01-02 20:17:10gregory.p.smithsetnosy: loewis, georg.brandl, gregory.p.smith, kaizhu
messages: + msg125092
2011-01-02 20:13:40loewissetnosy: loewis, georg.brandl, gregory.p.smith, kaizhu
messages: + msg125089
2011-01-02 14:13:34georg.brandlsetstatus: open -> closed

messages: + msg125052
resolution: works for me
nosy: loewis, georg.brandl, gregory.p.smith, kaizhu
2011-01-02 14:08:28kaizhusetfiles: + trace2.txt
nosy: loewis, georg.brandl, gregory.p.smith, kaizhu
messages: + msg125051
2011-01-02 13:31:22kaizhusetnosy: loewis, georg.brandl, gregory.p.smith, kaizhu
messages: + msg125048
2011-01-02 09:56:38loewissetnosy: loewis, georg.brandl, gregory.p.smith, kaizhu
messages: + msg125039
2011-01-02 09:04:17kaizhusetnosy: loewis, georg.brandl, gregory.p.smith, kaizhu
messages: + msg125037
2011-01-02 08:59:27kaizhusetnosy: loewis, georg.brandl, gregory.p.smith, kaizhu
messages: + msg125036
2011-01-02 08:25:47loewissetnosy: + loewis
2011-01-02 02:58:52gregory.p.smithsetnosy: georg.brandl, gregory.p.smith, kaizhu
messages: + msg125028
2011-01-02 00:20:02georg.brandlsetpriority: normal -> release blocker
nosy: + georg.brandl
2011-01-02 00:11:15pitrousetassignee: gregory.p.smith

nosy: + gregory.p.smith
2011-01-02 00:07:47kaizhucreate