classification
Title: subprocess.Popen inside thread locks the thread in some case (2.4)
Type: behavior Stage:
Components: Library (Lib) Versions: Python 2.4
process
Status: closed Resolution: wont fix
Dependencies: Superseder:
Assigned To: astrand Nosy List: astrand, bigorilla, darkk, gregory.p.smith, ragnark, titty, tovrstra
Priority: normal Keywords: easy, patch

Created on 2006-01-13 16:05 by tovrstra, last changed 2008-02-18 02:51 by gregory.p.smith. This issue is now closed.

Files
File name Uploaded Description Edit
my_thread.py tovrstra, 2006-01-13 16:05 my_thread.py
main.py tovrstra, 2006-01-13 16:07 main.py
Messages (8)
msg27293 - (view) Author: Toon Verstraelen (tovrstra) Date: 2006-01-13 16:05
The bug can be verified with the example program
attached to this bugs. Two files are attached:
my_thread.py and main.py

When main.py is executed the thread hangs on
subprocess.Popen, while my_thread.py, which is imported
by main.py runs without problems. In fact both should
do exactly the same since main.py doesn nothing but
importing my_thread.

My python version:

Python 2.4.2 (#1, Oct 17 2005, 09:05:20)
[GCC 3.3.6 (Gentoo 3.3.6, ssp-3.3.6-1.0, pie-8.7.8)] on
linux2

If any more info is required, 
msg27294 - (view) Author: Ralf Schmitt (titty) Date: 2006-01-13 16:30
Logged In: YES 
user_id=17929

I cannot reproduce this error on ubuntu linux using python
2.4.2.
However, I am able to reproduce it using python 2.4.2
running on FreeBSD 4.11.

Stopping the program with ctrl-c results in the following
output:

ralf@stronzo:~/tmp$ python main.py 
before Popen
^CException in thread Thread-1:
Traceback (most recent call last):
  File "/usr/local/lib/python2.4/threading.py", line 442, in
__bootstrap
    self.run()
  File "/home/ralf/tmp/my_thread.py", line 7, in run
    sp = Popen(["ls", "-al"], stdout=PIPE, stderr=STDOUT)
  File "/usr/local/lib/python2.4/subprocess.py", line 542,
in __init__
    errread, errwrite)
  File "/usr/local/lib/python2.4/subprocess.py", line 970,
in _execute_child
    data = os.read(errpipe_read, 1048576) # Exceptions
limited to 1 MB
OSError: [Errno 4] Interrupted system call

Error in atexit._run_exitfuncs:
Traceback (most recent call last):
  File "/usr/local/lib/python2.4/atexit.py", line 24, in
_run_exitfuncs
    func(*targs, **kargs)
  File "/usr/local/lib/python2.4/threading.py", line 632, in
__exitfunc
    t.join()
  File "/usr/local/lib/python2.4/threading.py", line 539, in
join
    self.__block.wait()
  File "/usr/local/lib/python2.4/threading.py", line 203, in
wait
    waiter.acquire()
KeyboardInterrupt
Error in sys.exitfunc:
Traceback (most recent call last):
  File "/usr/local/lib/python2.4/atexit.py", line 24, in
_run_exitfuncs
    func(*targs, **kargs)
  File "/usr/local/lib/python2.4/threading.py", line 632, in
__exitfunc
    t.join()
  File "/usr/local/lib/python2.4/threading.py", line 539, in
join
    self.__block.wait()
  File "/usr/local/lib/python2.4/threading.py", line 203, in
wait
    waiter.acquire()
KeyboardInterrupt
ralf@stronzo:~/tmp$
msg27295 - (view) Author: Toon Verstraelen (tovrstra) Date: 2006-01-13 16:58
Logged In: YES 
user_id=970944

What version of glibc do you have on your ubuntu system? I
have glibc-2.3.5-r2, where r2 stands for second patch
revision from the gentoo distribution. In my case I can not
interrupt 'python main.py'. I have to use ctrl-z and then
'kill %1'.
msg27296 - (view) Author: Ralf Schmitt (titty) Date: 2006-01-13 17:41
Logged In: YES 
user_id=17929

glibc version 2.3.5.
If I add a time.sleep(1) at the end of my_thread.py I can
also reproduce this error reliably on ubuntu.
msg27297 - (view) Author: Yair Chuchem (bigorilla) Date: 2006-05-09 12:30
Logged In: YES 
user_id=232571

this thread explains why this happens and how to solve it:
http://groups.google.com/group/comp.lang.python/browse_thread/thread/5fae8a453c95ae89/42f5c9f9215dbb1e#42f5c9f9215dbb1e
I implemented the fix locally at mine:
at os.py, _execvpe, first line:
remove the "from errno import ...",
add a global "import errno" at the start
and change stuff imported from errno from "X" to "errno.X"
msg27298 - (view) Author: Ragnar Kjørstad (ragnark) Date: 2006-05-19 10:37
Logged In: YES 
user_id=10550

I too have verified that the testprograms (with an
additional sleep) cause a deadlock, and that moving the
import statement out of os._execvp solves the problem.


However, there is a more generic problem. If fork() is
executed while any locks are held by other threads than the
one executing fork, it will deadlock the new process.

E.g, if you have a process with 2 threads. The first one is
doing work that requires locking. This could be imports, but
also anything else that uses other locks.

The second thread does a fork. If this is accidently done
while thread 1 held the lock, it will deadlock if the new
process needs the same lock.

Now, the most common use of fork is of course to do exec
right afterwards, in which case it doesn't really matter,
but it's not the only use of fork. 

See the rationalie in the manpage of pthread_atfork for
details.
(http://www.opengroup.org/onlinepubs/009695399/functions/pthread_atfork.html)

msg62482 - (view) Author: Leonid Evdokimov (darkk) Date: 2008-02-17 06:14
I confirm the bug.

Deadlock at Linux 2.6 (^C does not work):
Python 2.4.4 (#1, Jan  8 2008, 23:42:40) 
[GCC 4.1.1 (Gentoo 4.1.1-r3)] on linux2

Sometimes work and sometimes fails with fatal error at FreeBSD 6.3:
Python 2.4.4 (#2, Oct 31 2007, 05:20:42) 
[GCC 3.4.6 [FreeBSD] 20060305] on freebsd6

before Popen
before first line
FFatal error '_pq_insert_tail: Already in priority queue' at line 200 in
file /usr/src/lib/libpthread/thread/thr_priority_queue.c (errno = 0)
after last line          

And sometimes it deadlocks and ^C shows same traceback Ralf Schmitt
already posted.
msg62514 - (view) Author: Gregory P. Smith (gregory.p.smith) * (Python committer) Date: 2008-02-18 02:51
This appears to have been fixed in 2.5 and trunk.  2.4.x is old and in
security fixes only mode so I wouldn't expect to see this in any
official 2.4.x source tree released in the future unless the bdfl
changes his mind on that.  here's the patch to fix it (as described in
the email thread mentioned earlier):

Index: Lib/os.py
===================================================================
--- Lib/os.py   (revision 60877)
+++ Lib/os.py   (working copy)
@@ -351,8 +351,8 @@
 
 __all__.extend(["execl","execle","execlp","execlpe","execvp","execvpe"])
 
+import errno
 def _execvpe(file, args, env=None):
-    from errno import ENOENT, ENOTDIR
 
     if env is not None:
         func = execve
@@ -379,7 +379,7 @@
             func(fullname, *argrest)
         except error, e:
             tb = sys.exc_info()[2]
-            if (e.errno != ENOENT and e.errno != ENOTDIR
+            if (e.errno != errno.ENOENT and e.errno != errno.ENOTDIR
                 and saved_exc is None):
                 saved_exc = e
                 saved_tb = tb
History
Date User Action Args
2008-02-18 02:51:39gregory.p.smithsetstatus: open -> closed
type: behavior
components: + Library (Lib), - Interpreter Core
title: subprocess.Popen inside thread locks the thread in some case -> subprocess.Popen inside thread locks the thread in some case (2.4)
keywords: + patch, easy
nosy: gregory.p.smith, titty, astrand, tovrstra, bigorilla, ragnark, darkk
messages: + msg62514
resolution: wont fix
2008-02-17 06:14:49darkksetnosy: + darkk
messages: + msg62482
2008-01-17 01:48:44gregory.p.smithsetnosy: + gregory.p.smith
2006-01-13 16:05:31tovrstracreate