classification
Title: Python should invalidate all non-owned 'thread.lock' objects when forking
Type: enhancement Stage: resolved
Components: Library (Lib) Versions: Python 2.7, Python 2.6
process
Status: closed Resolution: duplicate
Dependencies: Superseder: Locks in the standard library should be sanitized on fork
View: 6721
Assigned To: Nosy List: lesha
Priority: normal Keywords:

Created on 2012-01-13 09:24 by lesha, last changed 2012-01-13 10:42 by neologix. This issue is now closed.

Messages (2)
msg151165 - (view) Author: (lesha) Date: 2012-01-13 09:24
Here is a great description of the issue:

http://docs.oracle.com/cd/E19683-01/806-6867/gen-1/index.html


This enhancement proposes a way to make Python more resistant to this kind of deadlock.


Consider this program:


import threading
import subprocess
import time

l = threading.Lock() 

def f():
    l.acquire()  
    time.sleep(1)
    l.release()
 
t = threading.Thread(target=f)
t.start()

def g(l):
    l.acquire()
    l.release() 
    print 'ohai'


subprocess.Popen(['ls'], preexec_fn=lambda: g(l))



g() gets called in the forked process, which means that it's waiting on a *copy* of the lock, which can never get released.


This, in turn, means that the main thread will forever wait for the Popen to finish.



The program above incorrectly assumes that a threading lock can be shared across fork() parent and child.

I suspect adding such sharing is impractical, requiring OS support or excessive complexity. If the sharing could be had cheaply, it would be great -- programs like this would work as intended, but no other programs would break. 

Crazy idea: free the locks. Sadly, that is not safe! The ones that are currently locked by other threads might be protecting some network resource, and allowing the fork child to access them would result in a logical error.

However, it is always a bad idea for a fork() child to access a lock that is held by a thread that is not its fork() parent. That lock was locked at the time of the fork(), and will stay locked, because the child process will not get updated by the lock-holding threads.

So, it is always invalid to access that type of lock. Currently, you are guaranteed a deadlock.

Proposal: trying to acquire such a lock should crash the forked child with a nice, detailed error message (including the offending lock), rather than hang the entire program.

Sample steps to implement:

1) Store the process ID on each lock instance.
2) Acquire/release should crash if the lock does not belong to the current thread AND has a different process ID from the current one.

There are other potential implementations, such as explicitly enumerating such locks at the time of fork, and invalidating them.

This crash cannot be an exception in the child, because lock methods must not throw. However, it can and should be an exception in the fork() parent.

I think this enhancement would make it much easier to debug this kind of problem. It's an easy mistake to make, because preexec_fn or fork docs do not warn you of the danger, and locks can be acquired quite implicitly by innocent-looking code.
msg151166 - (view) Author: (lesha) Date: 2012-01-13 09:33
Actually, I think it does not matter which thread owns the lock, it is still invalid to try to acquire a lock that was grabbed by the fork() parent. Why? Because the fork() parent cannot free the child's copy of the lock anyway, and it's guaranteed to be dead also.
History
Date User Action Args
2012-01-13 10:42:12neologixsetstatus: open -> closed
resolution: duplicate
superseder: Locks in the standard library should be sanitized on fork
stage: resolved
2012-01-13 09:33:04leshasetmessages: + msg151166
2012-01-13 09:24:57leshacreate