classification
Title: Deadlocks with fork() and multithreading
Type: Stage:
Components: Library (Lib) Versions: Python 2.4
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: christian.heimes, facundobatista, georg.brandl, vanco-buca (4)
Priority: normal Keywords:

Created on 2007-07-16 23:34 by vanco-buca, last changed 2008-01-07 16:49 by facundobatista.

Messages (6)
msg32521 - (view) Author: Vanco Buca (vanco-buca) Date: 2007-07-16 23:34
Encountered a deadlock while executing a subprocess.call("<something>") in a multithreaded environment. After some debugging, found the child process stuck on a lock immediately after the fork, inside the module os's _execvpe() function. 

A Google search found a detailed discussion on this topic in the Python-Dev thread (see http://mail.python.org/pipermail/python-dev/2006-May/064983.html). 

My symptoms are exactly the same as the ones found by Rotem Yaari, and exactly the same fix worked as well (moving the "from errno import ENOENT, ENOTDIR" line to the beginning of the os.py file). 

Some details follow: 
- my code has several threads running, and some of them executes shell commands with subprocess.call(). 
- I'm running on Python 2.4.4, with a Linux 2.6.20 kernel, on a dual core system (uname -a shows: Linux rack1 2.6.18-gentoo-r3 #1 SMP PREEMPT Tue Dec 5 16:40:19 PST 2006 x86_64 AMD Opteron(tm) Processor 246 AuthenticAMD GNU/Linux)
msg57027 - (view) Author: Facundo Batista (facundobatista) Date: 2007-11-01 19:06
I followed the link you provided. All the discussion there ends asking
for a realiable way to test the problem (otherwise, we could be making
more mistakes than solving the problem in the different platforms).

Please provide a test case, so we include it in the regression tests and
know that this won't be broken again.

You should provide the smallest code that causes a deadlock for you.
msg57773 - (view) Author: Christian Heimes (christian.heimes) Date: 2007-11-23 08:36
It sounds like the importer dead lock problem. These problems are almost
impossible to unit test because they are usually race conditions.

I don't see a problem in moving the import to the module name space.
errno is a built-in module and the import isn't costly.
msg57879 - (view) Author: Facundo Batista (facundobatista) Date: 2007-11-27 18:50
Fixed in the trunk, rev 59195.
msg57899 - (view) Author: Georg Brandl (georg.brandl) Date: 2007-11-27 23:56
Will it be backported?
msg59464 - (view) Author: Facundo Batista (facundobatista) Date: 2008-01-07 16:49
Backported, commited in r59823.
History
Date User Action Args
2008-01-07 16:49:46facundobatistasetstatus: open -> closed
resolution: fixed
messages: + msg59464
2007-11-27 23:56:10georg.brandlsetnosy: + georg.brandl
messages: + msg57899
2007-11-27 18:50:42facundobatistasetmessages: + msg57879
2007-11-23 08:36:52christian.heimessetnosy: + christian.heimes
messages: + msg57773
2007-11-01 19:06:01facundobatistasetassignee: facundobatista ->
messages: + msg57027
2007-10-23 20:24:45facundobatistasetassignee: facundobatista
nosy: + facundobatista
2007-07-16 23:34:28vanco-bucacreate