Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Importing multiprocessing breaks pickle.whichmodule #86572

Closed
renatolfc mannequin opened this issue Nov 19, 2020 · 7 comments
Closed

Importing multiprocessing breaks pickle.whichmodule #86572

renatolfc mannequin opened this issue Nov 19, 2020 · 7 comments
Assignees
@gpshead
Labels
3.8 only security fixes 3.9 only security fixes 3.10 only security fixes stdlib Python modules in the Lib dir type-bug An unexpected behavior, bug, or error

Comments

@renatolfc
Copy link
Mannequin

renatolfc mannequin commented Nov 19, 2020

BPO 42406
Nosy @gpshead, @pitrou, @renatolfc, @applio, @miss-islington
PRs
  • bpo-42406: Fix whichmodule() with multiprocessing #23403
  • [3.9] bpo-42406: Fix whichmodule() with multiprocessing (GH-23403) #23560
  • [3.8] bpo-42406: Fix whichmodule() with multiprocessing (GH-23403) #23561
  • [3.7] bpo-42406: Fix whichmodule() with multiprocessing (pythonGH-23403) #28489
    		•

    Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = 'https://github.com/gpshead'
closed_at = <Date 2020-11-29.23:48:33.708>
created_at = <Date 2020-11-19.13:19:31.708>
labels = ['3.8', 'type-bug', 'library', '3.9', '3.10']
title = 'Importing multiprocessing breaks pickle.whichmodule'
updated_at = <Date 2021-09-21.11:17:37.801>
user = 'https://github.com/renatolfc'

    bugs.python.org fields:

    activity = <Date 2021-09-21.11:17:37.801>
actor = 'python-dev'
assignee = 'gregory.p.smith'
closed = True
closed_date = <Date 2020-11-29.23:48:33.708>
closer = 'gregory.p.smith'
components = ['Library (Lib)']
creation = <Date 2020-11-19.13:19:31.708>
creator = 'renatolfc'
dependencies = []
files = []
hgrepos = []
issue_num = 42406
keywords = ['patch']
message_count = 7.0
messages = ['381410', '382024', '382085', '382086', '382088', '382097', '382160']
nosy_count = 6.0
nosy_names = ['gregory.p.smith', 'pitrou', 'renatolfc', 'python-dev', 'davin', 'miss-islington']
pr_nums = ['23403', '23560', '23561', '28489']
priority = 'normal'
resolution = 'fixed'
stage = 'commit review'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue42406'
versions = ['Python 3.8', 'Python 3.9', 'Python 3.10']
    @renatolfc
    Copy link
    Mannequin Author

    renatolfc mannequin commented Nov 19, 2020

    Importing multiprocessing prior to other modules that define ufuncs breaks pickle.whichmodule for those functions.

    Example:

        Python 3.8.6 (default, Sep 30 2020, 04:00:38)
    [GCC 10.2.0] on linux
    Type "help", "copyright", "credits" or "license" for more information.
    >>> import multiprocessing
    >>> from scipy.special import gdtrix
    >>> from pickle import whichmodule
    >>> whichmodule(gdtrix, gdtrix.__name__)
    '__mp_main__'

    Now, if we change the order of imports, whichmodule works fine:

        Python 3.8.6 (default, Sep 30 2020, 04:00:38)
    [GCC 10.2.0] on linux
    Type "help", "copyright", "credits" or "license" for more information.
    >>> from scipy.special import gdtrix
    >>> from pickle import whichmodule
    >>> import multiprocessing
    >>> whichmodule(gdtrix, gdtrix.__name__)
    'scipy.special._ufuncs'

    This happens because multiprocessing creates an alias to __main__ when imported:

    [Lib/multiprocessing/__init__.py](https://github.com/python/cpython/blob/main/Lib/multiprocessing/__init__.py)
37:    sys.modules['__mp_main__'] = sys.modules['__main__']

    But doing so changes the ordering of sys.modules, and whichmodule returns __mp_main__ instead of the correct scipy.special._ufuncs.

    This bug has arisen in the context of using dill (which imports multiprocessing) for serialization and is further documented at uqfoundation/dill#392.

    I intend to submit a patch/PR with a fix soon.

    @renatolfc renatolfc mannequin added interpreter-core (Objects, Python, Grammar, and Parser dirs) 3.7 (EOL) end of life 3.10 only security fixes 3.8 only security fixes 3.9 only security fixes type-bug An unexpected behavior, bug, or error stdlib Python modules in the Lib dir and removed interpreter-core (Objects, Python, Grammar, and Parser dirs) labels Nov 19, 2020
    @E-Paine E-Paine mannequin removed the 3.7 (EOL) end of life label Nov 28, 2020
    @gpshead
    Copy link
    Member

    gpshead commented Nov 28, 2020

    gross / nice find :)

    @gpshead gpshead self-assigned this Nov 28, 2020
    @gpshead
    Copy link
    Member

    gpshead commented Nov 29, 2020

    New changeset 8668431 by Renato Cunha in branch 'master':
    bpo-42406: Fix whichmodule() with multiprocessing (GH-23403)
    8668431

    @miss-islington
    Copy link
    Contributor

    New changeset b1c48e5 by Miss Islington (bot) in branch '3.8':
    bpo-42406: Fix whichmodule() with multiprocessing (GH-23403)
    b1c48e5

    @miss-islington
    Copy link
    Contributor

    New changeset fcf7391 by Miss Islington (bot) in branch '3.9':
    bpo-42406: Fix whichmodule() with multiprocessing (GH-23403)
    fcf7391

    @gpshead
    Copy link
    Member

    gpshead commented Nov 29, 2020

    thanks Renato!

    @gpshead gpshead closed this as completed Nov 29, 2020
    @renatolfc
    Copy link
    Mannequin Author

    renatolfc mannequin commented Nov 30, 2020

    No worries, Gregory. Glad I could help. :-)

    @ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
    Assignees

    @gpshead gpshead

    Labels
    3.8 only security fixes 3.9 only security fixes 3.10 only security fixes stdlib Python modules in the Lib dir type-bug An unexpected behavior, bug, or error
    Projects
    None yet
    Milestone
    No milestone
    Development

    No branches or pull requests
    2 participants
    @gpshead @miss-islington