Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

RuntimeError in pickle.whichmodule when sys.modules if mutated #66104

Closed
ogrisel mannequin opened this issue Jul 2, 2014 · 6 comments
Closed

RuntimeError in pickle.whichmodule when sys.modules if mutated #66104

ogrisel mannequin opened this issue Jul 2, 2014 · 6 comments
Labels
stdlib Python modules in the Lib dir type-bug An unexpected behavior, bug, or error

Comments

@ogrisel
Copy link
Mannequin

ogrisel mannequin commented Jul 2, 2014

BPO 21905
Nosy @pitrou, @avassalotti, @ogrisel
Files
  • pickle_whichmodule.patch
  • pickle_whichmodule_20140703.patch
    		•

    Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = None
closed_at = <Date 2014-10-04.20:18:24.133>
created_at = <Date 2014-07-02.13:03:12.776>
labels = ['type-bug', 'library']
title = 'RuntimeError in pickle.whichmodule  when sys.modules if mutated'
updated_at = <Date 2014-10-06.07:25:56.245>
user = 'https://github.com/ogrisel'

    bugs.python.org fields:

    activity = <Date 2014-10-06.07:25:56.245>
actor = 'Olivier.Grisel'
assignee = 'none'
closed = True
closed_date = <Date 2014-10-04.20:18:24.133>
closer = 'pitrou'
components = ['Library (Lib)']
creation = <Date 2014-07-02.13:03:12.776>
creator = 'Olivier.Grisel'
dependencies = []
files = ['35830', '35841']
hgrepos = []
issue_num = 21905
keywords = ['patch']
message_count = 6.0
messages = ['222099', '222188', '222193', '228488', '228489', '228646']
nosy_count = 4.0
nosy_names = ['pitrou', 'alexandre.vassalotti', 'python-dev', 'Olivier.Grisel']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue21905'
versions = ['Python 3.4', 'Python 3.5']
    @ogrisel
    Copy link
    Mannequin Author

    ogrisel mannequin commented Jul 2, 2014

    pickle.whichmodule performs an iteration over sys.modules and tries to perform getattr calls on those modules. Unfortunately some modules such as those from the six.moves dynamic module can trigger imports when calling getattr on them, hence mutating the sys.modules dict and causing a RuntimeError: dictionary changed size during iteration.

    This would also render pickle.whichmodule more thread-safe and cause concurrent thread perform new module imports and whichmodule calls.

    The attach patch protect the iterator by copying the dict items into a fixed list.

    I could write a tests involving dynamic module definitions as done in six.moves but it sounds very complicated for such a trivial fix.

    @ogrisel ogrisel mannequin added type-crash A hard crash of the interpreter, possibly with a core dump stdlib Python modules in the Lib dir labels Jul 2, 2014
    @pitrou
    Copy link
    Member

    pitrou commented Jul 3, 2014

    Sounds fine, but you should at least add a comment explaining the list() call.

    @pitrou pitrou added type-bug An unexpected behavior, bug, or error and removed type-crash A hard crash of the interpreter, possibly with a core dump labels Jul 3, 2014
    @ogrisel
    Copy link
    Mannequin Author

    ogrisel mannequin commented Jul 3, 2014

    New version of the patch to add an inline comment.

    @python-dev
    Copy link
    Mannequin

    python-dev mannequin commented Oct 4, 2014

    New changeset 86ba3bdfac15 by Antoine Pitrou in branch '3.4':
    Issue bpo-21905: Avoid RuntimeError in pickle.whichmodule() when sys.modules is mutated while iterating.
    https://hg.python.org/cpython/rev/86ba3bdfac15

    New changeset d748a3503ad5 by Antoine Pitrou in branch 'default':
    Issue bpo-21905: Avoid RuntimeError in pickle.whichmodule() when sys.modules is mutated while iterating.
    https://hg.python.org/cpython/rev/d748a3503ad5

    @pitrou
    Copy link
    Member

    pitrou commented Oct 4, 2014

    Sorry for the delay, Olivier. Your patch is now pushed. Thank you!

    @pitrou pitrou closed this as completed Oct 4, 2014
    @ogrisel
    Copy link
    Mannequin Author

    ogrisel mannequin commented Oct 6, 2014

    No problem. Thanks Antoine for the review!

    @ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
    Assignees
    No one assigned
    Labels
    stdlib Python modules in the Lib dir type-bug An unexpected behavior, bug, or error
    Projects
    None yet
    Milestone
    No milestone
    Development

    No branches or pull requests
    1 participant
    @pitrou