Here's a report from Ismail Donmez (cartman), extracted from the
issue 3419 discussion in an effort to avoid putting multiple
problems under one tracker issue.

[cartman]

With trunk when running test_multiprocessing in a tight loop I saw 
another problem:

test_multiprocessing
Process Process-61:
Traceback (most recent call last):
  File "/Users/cartman/Sources/py3k/Lib/multiprocessing/process.py", 
line 229, in _bootstrap
Process Process-60:
Traceback (most recent call last):
  File "/Users/cartman/Sources/py3k/Lib/multiprocessing/process.py", 
line 229, in _bootstrap
Process Process-62:
Traceback (most recent call last):
  File "/Users/cartman/Sources/py3k/Lib/multiprocessing/process.py", 
line 229, in _bootstrap
    util._run_after_forkers()
  File "/Users/cartman/Sources/py3k/Lib/multiprocessing/util.py", line 
138, in _run_after_forkers
    util._run_after_forkers()
  File "/Users/cartman/Sources/py3k/Lib/multiprocessing/util.py", line 
138, in _run_after_forkers
    util._run_after_forkers()
  File "/Users/cartman/Sources/py3k/Lib/multiprocessing/util.py", line 
138, in _run_after_forkers
    items = list(_afterfork_registry.items())
    items = list(_afterfork_registry.items())
  File "/Users/cartman/Sources/py3k/Lib/weakref.py", line 103, in items
  File "/Users/cartman/Sources/py3k/Lib/weakref.py", line 103, in items
    items = list(_afterfork_registry.items())
  File "/Users/cartman/Sources/py3k/Lib/weakref.py", line 103, in items
    for key, wr in self.data.items():
RuntimeError: dictionary changed size during iteration
    for key, wr in self.data.items():
RuntimeError: dictionary changed size during iteration
    for key, wr in self.data.items():
RuntimeError: dictionary changed size during iteration

I don't know if this is reproducible in 2.6 but I can reproduce on py3k 
branch, and also thanks for creating the bug!

Mmmh, the problem with the list(X.items()) idiom is that it's
thread-safe only if X.items() is implemented in C. Otherwise X can be
mutated if there is a thread-switch while executing bytecode in X.items().

In weakref.py (line 103), by replacing:
    for key, wr in self.data.items():
with:
    for key, wr in list(self.data.items()):

This particular error should disappear.

But this doesn't say why the dictionary is mutated at all. Does
multiprocessing (or at least that particular test) launch several
threads in a given process? Otherwise there may be something fishy going on.

py3k branch gives another error now, when running test_multiprocessing 
in a tight loop:

test test_multiprocessing failed -- Traceback (most recent call last):
  File "/Users/cartman/Sources/py3k/Lib/test/test_multiprocessing.py", 
line 1163, in test_remote
    queue = manager2.get_queue()
  File "/Users/cartman/Sources/py3k/Lib/multiprocessing/managers.py", 
line 641, in temp
    authkey=self._authkey, exposed=exp
  File "/Users/cartman/Sources/py3k/Lib/multiprocessing/managers.py", 
line 893, in AutoProxy
    incref=incref)
  File "/Users/cartman/Sources/py3k/Lib/multiprocessing/managers.py", 
line 702, in __init__
    self._incref()
  File "/Users/cartman/Sources/py3k/Lib/multiprocessing/managers.py", 
line 749, in _incref
    dispatch(conn, None, 'incref', (self._id,))
  File "/Users/cartman/Sources/py3k/Lib/multiprocessing/managers.py", 
line 85, in dispatch
    raise convert_to_error(kind, result)
multiprocessing.managers.RemoteError: 
------------------------------------------------------------------------
---
Traceback (most recent call last):
  File "/Users/cartman/Sources/py3k/Lib/multiprocessing/managers.py", 
line 187, in handle_request
    result = func(c, *args, **kwds)
  File "/Users/cartman/Sources/py3k/Lib/multiprocessing/managers.py", 
line 403, in incref
    self.id_to_refcount[ident] += 1
KeyError: '7389d0'

Ismail, that's the incref bug in issue 3419

Ah cool, we might be at the end of multiprocessing problems then I guess 
:-)

Le lundi 18 août 2008 à 19:42 +0000, Ismail Donmez a écrit :
> Ismail Donmez <ismail@namtrac.org> added the comment:
> 
> Ah cool, we might be at the end of multiprocessing problems then I guess 
> :-)

Well, not really, it should be diagnosed why the dictionary mutates in
the first place. This dictionary is apparently processed in a child
process just after a fork() occurred, so logically there shouldn't be
several threads running.
Unless processing the dictionary itself can create some new threads?