Importing the module with the following contents results in RuntimeError:

==================
import os
pid = os.fork()

if pid == 0:
    print "In the child"
else:
    print "In the parent"
print "Done\n"
==================

Running the same module as main works just fine, so it seems to be a purely import issue.

I looked into the 2.6.6rc1 source.  This is what I think happens in Python/import.c file:

1. After the fork() call, _PyImport_ReInitLock() is run. It sets import_lock_thread to -1 in the child, line 310.

2. In _PyImport_ReleaseLock() line 290 compares import_loc_thread to the current thread id, and if they are not the same then -1 is returned, which results in RuntimeError in PyImport_ImportModuleLevel (line 2186-2189)

So this is guaranteed to happen in the child, every time fork() is executed inside the module being imported.  If I change line 290 to be:

    if (import_lock_thread != me && import_lock_thread != -1)

then import proceeds fine, although I'm not sure this is a proper solution.

This happens on Linux, Darwin, and Cygwin, with python 2.6.5 and higher. I'd be happy to assist solving this, please let me know how I can help.

This may be a case of "don't do that".  Starting a new thread or process during import (ie: while the import lock is held) is dangerous.  Nosying Brett, since he'll know the real story.

Without looking closer at it, don't do that. =)

I guess I am missing something here.  In a complex program, everything will be executed in some module or another.  Consequently, the module that contains the fork() call will cause the interpreter to quit.

How can this be worked around, short of placing the fork() in the main module?

> How can this be worked around, short of placing the fork()
> in the main module?

Why wouldn't you place the fork() call in a function?

I can place it in a function.  But if I execute that function from anything other than main module, the fork() will be called while import lock is held, one way or another.  It will just happen in another module.  So what?

1. If fork should not be called during import, it could raise an exception when invoked from import.  But it does not.  Is that a bug then?  BTW, fork during import worked with python 2.4 just fine.

2. The whole issue7242 was devoted to work out import locks during forking. 5 months ago r78527 was committed to to just that (although it is not perfect). If this is not the proper use case then had it been done in error?

3. belopolsky: Thanks for the advice to use the mailing list.  I'd appreciate it if instead you refrained from publishing my email address on this page.

Added Greg to nosy list as the one that fixed issue 7242 with the current _PyImport_ReInitLock semantics.

Also kicking over to Barry regarding implications for 2.6.6 (this is a regression from 2.6.4 due to the resolution of 7242).

7242 was about forking from a *thread*. This is about forking as a side effect of import, which, just like spawning a thread as a side effect of import, can easily cause issues.

The RuntimeError noted by the OP isn't thrown by the fork() call - it is thrown at the end of the import when control is returned to the main module. Completely reinitialising the lock without accounting for the current depth of nesting is incorrect. Instead, I believe ReInitLock should look more like:

    if (import_lock != NULL)
        import_lock = PyThread_allocate_lock();
    if (import_lock_level > 1) {
        /* Forked as a side effect of import */
        long me = PyThread_get_thread_ident();
        import_lock_thread = me;
        import_lock_level--;
    } else {
        import_lock_thread = -1;
        import_lock_level = 0;
    }

(I haven't tested that yet, but will soon)

One slight tweak to that suggested change - the lock reinitialisation needs to acquire the new lock in the first branch.

Minimal patch attached (no niceties like NEWS or unit tests included yet)

Test script attached that demonstrates the underlying problem directly via imp.lock_held() (this could easily form the basis of a unit test)

I think it's too late to try to get this into 2.6.6.  rc2's already been released, I don't expect/want an rc3, and I'm not comfortable changing this at this point.  Unless you can convince me it's absolutely critical, I won't approve this for 2.6.6.

Are there any applications out there that actually rely on forking during import?

(someone discovered this bug... i'd like to know why.  i think its a disgusting thing to do but never explicitly disallowed it in the past)

It turns out my proposed patch is incorrect anyway - it will do the wrong thing if a thread *other* than the one doing the fork is in the middle of a nested import at the time the fork occurs.

With issue 7242 establishing that the current thread ID may not survive the forking process on all platforms, the only way I can see to get completely correct semantics for the fork-as-a-side-effect of import case is to give the forking thread another way to detect "did I own the import lock before the fork?". One way to do that would be to move the lock nesting count into thread local storage, although that would likely suffer cross-platform incompatibility fun and games as well.

Given that, I'm inclined to go with what Brett said: don't do that. Use a __name__ == "__main__" guard so the fork only happens when run as a script, not when imported.

gregory.p.smith: This is my use case: we had the following situation with the test scripts at work.  Each script is required to import TestApi module in order to run the tests.  That module in turn imported the module that forks, and in the parent waits for the child to exit, then kills all child's children processes.  That way tests don't leave any processes behind.

So any script that imported the cleanup module, whether directly or via another module, had this cleanup functionality "for free". One workaround for this issue would be to change all existing test scripts to call the cleanup function, instead of the cleanup module calling it at the module level.

I can see your reservations about forking/starting threads during import, but it seems like it either should work or it should be disallowed.  The thing is, the actual import is working fine with the fork() call, it's releasing the lock that is messed up, because it was not initialized correctly after the fork.  The patch attached by ncoghlan looks good though.

I already worked around this for my use case.

For the future, it would be nice if fork() raised an exception if called during the import, and if the documentation mentioned that forking while in import is not allowed.

Agreed on the explicit exception and documentation. :)

Will starting a thread while in import also be disallowed?  If so, issue 7242 will also become moot...

On further further reflection - I'm back to thinking my patch is correct. With the way fork is now implemented, the forking thread *always* holds the import lock, so the assumption in my patch regarding the meaning of the nesting level is correct.

It could use a comment in _PyImport_ReInitLock to better explain that, though.

Unless there are any objections, I'll apply the fix to 2.7, 3.1 and 3.2.

2.6 is out of luck though (as per Barry's comment). I'll do a doc change for that, but I can't promise I'll get to it before the binary releases go out.

Fixed for 3.2 in r86928.

The fix has not been backported to 3.1, since it depends on the fix for issue 7242 (r78527) which was itself never backported to 3.1 after being forward ported to the py3k branch as part of a bulk merge (r83318)

Backporting to 2.7 would also be a manual process (although much easier, since issue 7242 is already fixed in that branch)

Given the obscurity of the error, I'm going to close this as fixed without backporting it. Anyone that wants it fixed in the 2.7 and 3.1 maintenance branches is free to develop and post the requisite patches :)