If you import a module that creates a multiprocessing.Queue, puts a value, and then waits for to be received again from the queue, you run into a deadlock.

The issue is that Queue._feed does 'from .util import is_existing' - which needs the import lock, but is still being held by the main thread.

Attached a script that illustrates this.

Patch is a two line change, import is_exiting in line 49, remove the import inside the thread:

49c49
< from multiprocessing.util import debug, info, Finalize, register_after_fork
---
> from multiprocessing.util import debug, info, Finalize, register_after_fork, is_exiting
232d231
<         from .util import is_exiting

@Davin I believe that you're interested in multiprocessing issues.

Confirmed that the issue can be reproduced under 2.7.9 on OS X 10.10.

It is not possible to reproduce the issue with default (3.5) -- taking a look at what's different there, notably the import of is_exiting has moved to the top of the queues module and is no longer only imported at the time Queue._feed is invoked, which is just as Florian advocates doing for 2.7.

I should take a closer look to understand what else has changed.


@Mark:  Cool -- thanks.

Attaching a patch for 2.7 that applies Florian's fix and provides a test for it as well.

Although the issue is not triggered on 3.4 or default (3.5), there is the potential for regression there -- attaching a single patch that works for both 3.4 and 3.5 to provide a regression test (only a test, nothing to "fix").

These patches have been tested on OS X 10.10 and Ubuntu 12.04.5 64-bit for each of 2.7, 3.4, and default (3.5).

I'm not sure there is a need to fix this issue. Using multiprocessing Queue at import time looks as yet one way to shoot yourself in the foot. But the patch looks harmful, I'll commit it.

That's all we need, harmful patches being committed :)  Harmless possibly?

Oh, right. Harmless. Thanks for all the fish Mark. I meant the bug.

New changeset 069c13ca7a70 by Serhiy Storchaka in branch '2.7':
Issue #22853: Fixed a deadlock when use multiprocessing.Queue at import time.
https://hg.python.org/cpython/rev/069c13ca7a70

As for 3.x, underscored test does not run, and when remove the underscore it runs, but produce a warning (you should run regrtests with -vv to see detailed warnings):

$ ./python -m test.regrtest -vv -m '*no_import_lock_contention*' test_multiprocessing_spawn
...
Warning -- threading._dangling was modified by test_multiprocessing_spawn
  Before: <_weakrefset.WeakSet object at 0xb6a960ac>
  After:  <_weakrefset.WeakSet object at 0xb6c4cc0c> 
1 test altered the execution environment:
    test_multiprocessing_spawn

Corrected patch for 3.4 and default/3.5 -- newly introduced test is now turned on this time and the dangling weak references are properly addressed as well as the reference to Empty.  Nastiness.

Good save, Serhiy.

New changeset cf12856bde17 by Serhiy Storchaka in branch '3.4':
Issue #22853: Added regression test for using multiprocessing.Queue at import
https://hg.python.org/cpython/rev/cf12856bde17

New changeset dcd6d41f2c9a by Serhiy Storchaka in branch 'default':
Issue #22853: Added regression test for using multiprocessing.Queue at import
https://hg.python.org/cpython/rev/dcd6d41f2c9a

Thank you Florian and Davin for your contribution.