classification
Title: Importing multiprocessing breaks pickle.whichmodule
Type: behavior Stage: commit review
Components: Library (Lib) Versions: Python 3.10, Python 3.9, Python 3.8
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: gregory.p.smith Nosy List: davin, gregory.p.smith, miss-islington, pitrou, python-dev, renatolfc
Priority: normal Keywords: patch

Created on 2020-11-19 13:19 by renatolfc, last changed 2021-09-21 11:17 by python-dev. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 23403 merged renatolfc, 2020-11-19 14:11
PR 23560 merged miss-islington, 2020-11-29 18:23
PR 23561 merged miss-islington, 2020-11-29 18:23
PR 28489 closed python-dev, 2021-09-21 11:17
Messages (7)
msg381410 - (view) Author: Renato Cunha (renatolfc) * Date: 2020-11-19 13:19
Importing multiprocessing prior to other modules that define `ufunc`s breaks pickle.whichmodule for those functions.

Example:

    Python 3.8.6 (default, Sep 30 2020, 04:00:38)
    [GCC 10.2.0] on linux
    Type "help", "copyright", "credits" or "license" for more information.
    >>> import multiprocessing
    >>> from scipy.special import gdtrix
    >>> from pickle import whichmodule
    >>> whichmodule(gdtrix, gdtrix.__name__)
    '__mp_main__'

Now, if we change the order of imports, whichmodule works fine:

    Python 3.8.6 (default, Sep 30 2020, 04:00:38)
    [GCC 10.2.0] on linux
    Type "help", "copyright", "credits" or "license" for more information.
    >>> from scipy.special import gdtrix
    >>> from pickle import whichmodule
    >>> import multiprocessing
    >>> whichmodule(gdtrix, gdtrix.__name__)
    'scipy.special._ufuncs'

This happens because multiprocessing creates an alias to `__main__` when imported:

    Lib/multiprocessing/__init__.py
    37:    sys.modules['__mp_main__'] = sys.modules['__main__']

But doing so changes the ordering of `sys.modules`, and `whichmodule` returns `__mp_main__` instead of the correct `scipy.special._ufuncs`.

This bug has arisen in the context of using `dill` (which imports multiprocessing) for serialization and is further documented at https://github.com/uqfoundation/dill/issues/392.

I intend to submit a patch/PR with a fix soon.
msg382024 - (view) Author: Gregory P. Smith (gregory.p.smith) * (Python committer) Date: 2020-11-28 22:25
gross / nice find :)
msg382085 - (view) Author: Gregory P. Smith (gregory.p.smith) * (Python committer) Date: 2020-11-29 18:23
New changeset 86684319d3dad8e1a7b0559727a48e0bc50afb01 by Renato Cunha in branch 'master':
bpo-42406: Fix whichmodule() with multiprocessing (GH-23403)
https://github.com/python/cpython/commit/86684319d3dad8e1a7b0559727a48e0bc50afb01
msg382086 - (view) Author: miss-islington (miss-islington) Date: 2020-11-29 18:43
New changeset b1c48e513624641d6472262c33d09624170ea1f5 by Miss Islington (bot) in branch '3.8':
bpo-42406: Fix whichmodule() with multiprocessing (GH-23403)
https://github.com/python/cpython/commit/b1c48e513624641d6472262c33d09624170ea1f5
msg382088 - (view) Author: miss-islington (miss-islington) Date: 2020-11-29 18:47
New changeset fcf7391f598e027a083173662a3ecf7865251291 by Miss Islington (bot) in branch '3.9':
bpo-42406: Fix whichmodule() with multiprocessing (GH-23403)
https://github.com/python/cpython/commit/fcf7391f598e027a083173662a3ecf7865251291
msg382097 - (view) Author: Gregory P. Smith (gregory.p.smith) * (Python committer) Date: 2020-11-29 23:48
thanks Renato!
msg382160 - (view) Author: Renato Cunha (renatolfc) * Date: 2020-11-30 16:37
No worries, Gregory. Glad I could help. :-)
History
Date User Action Args
2021-09-21 11:17:37python-devsetnosy: + python-dev

pull_requests: + pull_request26885
2020-11-30 16:37:02renatolfcsetmessages: + msg382160
2020-11-29 23:48:33gregory.p.smithsetstatus: open -> closed
resolution: fixed
messages: + msg382097

stage: patch review -> commit review
2020-11-29 18:47:37miss-islingtonsetmessages: + msg382088
2020-11-29 18:43:51miss-islingtonsetmessages: + msg382086
2020-11-29 18:23:41miss-islingtonsetpull_requests: + pull_request22442
2020-11-29 18:23:33miss-islingtonsetnosy: + miss-islington
pull_requests: + pull_request22441
2020-11-29 18:23:25gregory.p.smithsetmessages: + msg382085
2020-11-28 22:25:11gregory.p.smithsetassignee: gregory.p.smith

messages: + msg382024
nosy: + gregory.p.smith
2020-11-28 20:13:33epainesetnosy: + pitrou, davin

versions: - Python 3.6, Python 3.7
2020-11-19 14:11:06renatolfcsetkeywords: + patch
stage: patch review
pull_requests: + pull_request22295
2020-11-19 14:04:57renatolfcsetcomponents: + Library (Lib), - Interpreter Core
2020-11-19 13:19:31renatolfccreate