Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

'dictionary changed size' error in test_multiprocessing #47828

Closed
mdickinson opened this issue Aug 17, 2008 · 8 comments
Closed

'dictionary changed size' error in test_multiprocessing #47828

mdickinson opened this issue Aug 17, 2008 · 8 comments
Labels
extension-modules C modules in the Modules dir

Comments

@mdickinson
Copy link
Member

BPO 3578
Nosy @mdickinson, @pitrou
Superseder
  • bpo-7105: weak dict iterators are fragile because of unpredictable GC runs
    		•

    Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = None
closed_at = <Date 2009-10-24.12:55:46.973>
created_at = <Date 2008-08-17.13:42:07.177>
labels = ['extension-modules']
title = "'dictionary changed size' error in test_multiprocessing"
updated_at = <Date 2009-10-24.12:55:46.972>
user = 'https://github.com/mdickinson'

    bugs.python.org fields:

    activity = <Date 2009-10-24.12:55:46.972>
actor = 'pitrou'
assignee = 'jnoller'
closed = True
closed_date = <Date 2009-10-24.12:55:46.973>
closer = 'pitrou'
components = ['Extension Modules']
creation = <Date 2008-08-17.13:42:07.177>
creator = 'mark.dickinson'
dependencies = []
files = []
hgrepos = []
issue_num = 3578
keywords = []
message_count = 8.0
messages = ['71274', '71343', '71346', '71366', '71367', '71368', '71370', '94411']
nosy_count = 4.0
nosy_names = ['mark.dickinson', 'pitrou', 'donmez', 'jnoller']
pr_nums = []
priority = 'normal'
resolution = 'duplicate'
stage = None
status = 'closed'
superseder = '7105'
type = None
url = 'https://bugs.python.org/issue3578'
versions = ['Python 3.0']
    @mdickinson
    Copy link
    Member Author

    Here's a report from Ismail Donmez (cartman), extracted from the
    bpo-3419 discussion in an effort to avoid putting multiple
    problems under one tracker issue.

    [cartman]

    With trunk when running test_multiprocessing in a tight loop I saw
    another problem:

    test_multiprocessing
Process Process-61:
Traceback (most recent call last):
  File "/Users/cartman/Sources/py3k/Lib/multiprocessing/process.py", 
line 229, in _bootstrap
Process Process-60:
Traceback (most recent call last):
  File "/Users/cartman/Sources/py3k/Lib/multiprocessing/process.py", 
line 229, in _bootstrap
Process Process-62:
Traceback (most recent call last):
  File "/Users/cartman/Sources/py3k/Lib/multiprocessing/process.py", 
line 229, in _bootstrap
    util._run_after_forkers()
  File "/Users/cartman/Sources/py3k/Lib/multiprocessing/util.py", line 
138, in _run_after_forkers
    util._run_after_forkers()
  File "/Users/cartman/Sources/py3k/Lib/multiprocessing/util.py", line 
138, in _run_after_forkers
    util._run_after_forkers()
  File "/Users/cartman/Sources/py3k/Lib/multiprocessing/util.py", line 
138, in _run_after_forkers
    items = list(_afterfork_registry.items())
    items = list(_afterfork_registry.items())
  File "/Users/cartman/Sources/py3k/Lib/weakref.py", line 103, in items
  File "/Users/cartman/Sources/py3k/Lib/weakref.py", line 103, in items
    items = list(_afterfork_registry.items())
  File "/Users/cartman/Sources/py3k/Lib/weakref.py", line 103, in items
    for key, wr in self.data.items():
RuntimeError: dictionary changed size during iteration
    for key, wr in self.data.items():
RuntimeError: dictionary changed size during iteration
    for key, wr in self.data.items():
RuntimeError: dictionary changed size during iteration

    @mdickinson mdickinson added the extension-modules C modules in the Modules dir label Aug 17, 2008
    @donmez
    Copy link
    Mannequin

    donmez mannequin commented Aug 18, 2008

    I don't know if this is reproducible in 2.6 but I can reproduce on py3k
    branch, and also thanks for creating the bug!

    @pitrou
    Copy link
    Member

    pitrou commented Aug 18, 2008

    Mmmh, the problem with the list(X.items()) idiom is that it's
    thread-safe only if X.items() is implemented in C. Otherwise X can be
    mutated if there is a thread-switch while executing bytecode in X.items().

    In weakref.py (line 103), by replacing:
    for key, wr in self.data.items():
    with:
    for key, wr in list(self.data.items()):

    This particular error should disappear.

    But this doesn't say why the dictionary is mutated at all. Does
    multiprocessing (or at least that particular test) launch several
    threads in a given process? Otherwise there may be something fishy going on.

    @donmez
    Copy link
    Mannequin

    donmez mannequin commented Aug 18, 2008

    py3k branch gives another error now, when running test_multiprocessing
    in a tight loop:

    test test_multiprocessing failed -- Traceback (most recent call last):
  File "/Users/cartman/Sources/py3k/Lib/test/test_multiprocessing.py", 
line 1163, in test_remote
    queue = manager2.get_queue()
  File "/Users/cartman/Sources/py3k/Lib/multiprocessing/managers.py", 
line 641, in temp
    authkey=self._authkey, exposed=exp
  File "/Users/cartman/Sources/py3k/Lib/multiprocessing/managers.py", 
line 893, in AutoProxy
    incref=incref)
  File "/Users/cartman/Sources/py3k/Lib/multiprocessing/managers.py", 
line 702, in __init__
    self._incref()
  File "/Users/cartman/Sources/py3k/Lib/multiprocessing/managers.py", 
line 749, in _incref
    dispatch(conn, None, 'incref', (self._id,))
  File "/Users/cartman/Sources/py3k/Lib/multiprocessing/managers.py", 
line 85, in dispatch
    raise convert_to_error(kind, result)
multiprocessing.managers.RemoteError: 
------------------------------------------------------------------------
---
Traceback (most recent call last):
  File "/Users/cartman/Sources/py3k/Lib/multiprocessing/managers.py", 
line 187, in handle_request
    result = func(c, *args, **kwds)
  File "/Users/cartman/Sources/py3k/Lib/multiprocessing/managers.py", 
line 403, in incref
    self.id_to_refcount[ident] += 1
KeyError: '7389d0'

    @jnoller
    Copy link
    Mannequin

    jnoller mannequin commented Aug 18, 2008

    Ismail, that's the incref bug in bpo-3419

    @donmez
    Copy link
    Mannequin

    donmez mannequin commented Aug 18, 2008

    Ah cool, we might be at the end of multiprocessing problems then I guess
    :-)

    @pitrou
    Copy link
    Member

    pitrou commented Aug 18, 2008

    Le lundi 18 août 2008 à 19:42 +0000, Ismail Donmez a écrit :

    Ismail Donmez <ismail@namtrac.org> added the comment:

    Ah cool, we might be at the end of multiprocessing problems then I guess
    :-)

    Well, not really, it should be diagnosed why the dictionary mutates in
    the first place. This dictionary is apparently processed in a child
    process just after a fork() occurred, so logically there shouldn't be
    several threads running.
    Unless processing the dictionary itself can create some new threads?

    @pitrou
    Copy link
    Member

    pitrou commented Oct 24, 2009

    Actually, it's probably a duplicate of bpo-7105.

    @pitrou pitrou closed this as completed Oct 24, 2009
    @ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
    Assignees
    No one assigned
    Labels
    extension-modules C modules in the Modules dir
    Projects
    None yet
    Milestone
    No milestone
    Development

    No branches or pull requests
    2 participants
    @mdickinson @pitrou