I'm writing a library on top of multiprocessing. As part of the test suite, I create and destroy dozens of processes repeatedly. About once in 50 runs, the tests complete successfully but the program crashes with:

Error in atexit._run_exitfuncs:
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/multiprocessing/util.py", line 286, in _exit_function
    _run_finalizers(0)
  File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/multiprocessing/util.py", line 246, in _run_finalizers
    items = [x for x in list(_finalizer_registry.items()) if f(x)]
RuntimeError: dictionary changed size during iteration

(this is on OS X 10.10.3, Python 3.4.3; I've been unable to replicate the behavior on Python 2.7.9, 3.5.0b2, or pypy despite hundreds of runs)

The problematic code is available at https://github.com/jherskovic/mpetl and the tests should be run with nosetests

This happens reliably between 1-5% of the time on my home Mac (a 4.0 GHz i7). My work Mac Pro, a lot slower, doesn't exhibit this behavior on the same 3.4.3 interpreter.

Could it be related to a concurrency issue in the interpreter?

We're attempting to replicate on other systems.

More data:

I've been unable to replicate this behavior on Ubuntu 15.04 on Python 3.4.3 over ~20 thousand test runs. (Same machine, running in a VM)

An overnight repeated run on the original machine on OS X, 3.4.3, official distribution gave an actual frequency of 0.1% (7 crashes in 6980 runs).

Approximately 3,300 runs on Python 2.7.9 on the same machine have not resulted in any crashes.

I now built 3.4.3 from source using the latest clang and am trying it again, on the same machine, instead of using the official distribution. So far, 0 crashes in approximately 3,300 runs so far.

At the moment, then, the problem seems localized to the official 3.4.3 binaries for OS X.

Can you show the version info from the Python 3.4 where you see the failures?

/usr/local/bin/python3.4 -c 'import sys;print(sys.version)'

Exactly how are you building the Python 3.4 that does not fail for you?  FWIW, I am able to reproduce the failure easily on a 10.10.3 system using python.org 64-bit/32-bit 3.4.3, using a current MacPorts source-built 3.4.3 (which is built with clang), and repeatedly using the current head of the 3.4 branch (post-3.4.3) built with clang, both with and without pydebug enabled.

The failing Python:
3.4.3 (v3.4.3:9b73f1c3e601, Feb 23 2015, 02:52:03)
[GCC 4.2.1 (Apple Inc. build 5666) (dot 3)]

The non-failing Python:
3.4.3 (default, Jun 23 2015, 06:33:02)
[GCC 4.2.1 Compatible Apple LLVM 6.1.0 (clang-602.0.53)]

The non-failing python, I just downloaded the tarball and ran ./configure --prefix=/opt/python343, make, sudo make install, and then created a virtualenv using that binary.

Given what you're telling me, I'll run some more loops with the "non-failing" python. I'm grateful you've been able to reproduce the problem, FWIW.

FWIW, building 3.4.3 from source as you described results in failures about 30% of the time.  With some of the earlier tests, it failed even more frequently.  This is running on a 2.2GHz I7.

20,000+ iterations of the test suite with my homebuilt 3.4.3 throw no exceptions. 

Weirder and weirder. 

Any suggestions? Without any actual knowledge ("unencumbered by the thought process"), a concurrency issue in the interpreter itself is my guess. Especially since I can't reproduce it at all on other versions. 

I'll see if I can come up with a shorter, self-contained test that can illustrate the issue too.

Finally able to repro on my old Mac Pro at work (Universal 64/32bit 3.4.3 from python.org, OS X 10.10.3). For some reason I can't quite fathom, there were two exceptions this time.

Error in atexit._run_exitfuncs:
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/multiprocessing/util.py", line 286, in _exit_function
    _run_finalizers(0)
  File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/multiprocessing/util.py", line 246, in _run_finalizers
    items = [x for x in list(_finalizer_registry.items()) if f(x)]
RuntimeError: dictionary changed size during iteration
Exception ignored in: <Finalize object, dead>
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/multiprocessing/util.py", line 185, in __call__
  File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/multiprocessing/queues.py", line 207, in _finalize_close
  File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/threading.py", line 342, in notify
TypeError: 'NoneType' object is not callable

Ok, sorry, one more important thing I omitted in the previous comment.

^^^ The second exception above seems to be related to a new test I added, one that includes a daemonic process which itself relies on daemonic threads. 

The cleanup of that seems to be FUBARed, and it makes the original exception much more likely to pop up as well.

Ok, I *think* I tracked it down to Connections getting cleaned up in a non-orderly fashion, sometimes while _run_finalizers is executing. The following patch to lib/python3.4/multiprocessing/connection.py cures the problem on my machine.

@@ -124,10 +124,10 @@
         self._handle = handle
         self._readable = readable
         self._writable = writable
-
+        self._shutdown = util.Finalize(self, self._finalizer)
     # XXX should we use util.Finalize instead of a __del__?
-
-    def __del__(self):
+
+    def _finalizer(self):
         if self._handle is not None:
             self._close()

The 'crash' type is reserved for core dump type situations; 'behavior' is more appropriate for the misbehavior your describe.

I can confirm, I have managed to accidentally reproduce this issue on a custom build of 3.6.0 on RHEL.
sys.version:
'3.6.0 (default, Jan 18 2017, 11:23:11)
[GCC 4.4.7 20120313 (Red Hat 4.4.7-16)]'

The issue popped out of nowhere, on a code that was working w/out any hiccups on multiple systems; after an insignificant change was made to an unrelated part of code.

I managed to "patch it up" by forcingly closing and joining all Queues before letting the main process to stop.

Unfortunately, I cannot share the repro code (it's a) too large, b) closed), but I can confirm the issue was still open in 3.6.0.

The simple answer here is _run_finalizers() is not thread-safe, but finalizers can be registered or cancelled from any thread.

I could not write a simple synthetic script to reproduce the issue, but it should nevertheless be fixable.

Marcin, Jorge, Ned, could one of you apply the following simple patch and see if that seems to solve the issue?

Ok, I got to write a reproducer, which fails reliably here.

New changeset 1eb6c0074d17f4fd425cacfdda893d65f5f77f0a by Antoine Pitrou in branch 'master':
bpo-24484: Avoid race condition in multiprocessing cleanup (#2159)
https://github.com/python/cpython/commit/1eb6c0074d17f4fd425cacfdda893d65f5f77f0a

Python 2.7 doesn't suffer from this bug, so no actual fix is needed. Still, I'll backport the additional test.

New changeset 6fd03459957ee53941183212457bba19f977679f by Antoine Pitrou in branch '3.6':
[3.6] bpo-24484: Avoid race condition in multiprocessing cleanup (GH-2159) (#2166)
https://github.com/python/cpython/commit/6fd03459957ee53941183212457bba19f977679f

New changeset d09f1674d70dd84c5be37b5fe11bca4913146c6b by Antoine Pitrou in branch '2.7':
[2.7] bpo-24484: Avoid race condition in multiprocessing cleanup (GH-2159) (#2168)
https://github.com/python/cpython/commit/d09f1674d70dd84c5be37b5fe11bca4913146c6b

New changeset a0ecaab006849ade08fbcf8413fe1e92d9c8d71c by Antoine Pitrou in branch '3.5':
[3.5] bpo-24484: Avoid race condition in multiprocessing cleanup (GH-2159) (#2167)
https://github.com/python/cpython/commit/a0ecaab006849ade08fbcf8413fe1e92d9c8d71c

This should be fixed now.  Please reopen if not.