classification
Title: Importing multiprocessing breaks pickle.whichmodule
Type: behavior Stage: patch review
Components: Library (Lib) Versions: Python 3.10, Python 3.9, Python 3.8, Python 3.7, Python 3.6
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: renatolfc
Priority: normal Keywords: patch

Created on 2020-11-19 13:19 by renatolfc, last changed 2020-11-19 14:11 by renatolfc.

Pull Requests
URL Status Linked Edit
PR 23403 open renatolfc, 2020-11-19 14:11
Messages (1)
msg381410 - (view) Author: Renato Cunha (renatolfc) * Date: 2020-11-19 13:19
Importing multiprocessing prior to other modules that define `ufunc`s breaks pickle.whichmodule for those functions.

Example:

    Python 3.8.6 (default, Sep 30 2020, 04:00:38)
    [GCC 10.2.0] on linux
    Type "help", "copyright", "credits" or "license" for more information.
    >>> import multiprocessing
    >>> from scipy.special import gdtrix
    >>> from pickle import whichmodule
    >>> whichmodule(gdtrix, gdtrix.__name__)
    '__mp_main__'

Now, if we change the order of imports, whichmodule works fine:

    Python 3.8.6 (default, Sep 30 2020, 04:00:38)
    [GCC 10.2.0] on linux
    Type "help", "copyright", "credits" or "license" for more information.
    >>> from scipy.special import gdtrix
    >>> from pickle import whichmodule
    >>> import multiprocessing
    >>> whichmodule(gdtrix, gdtrix.__name__)
    'scipy.special._ufuncs'

This happens because multiprocessing creates an alias to `__main__` when imported:

    Lib/multiprocessing/__init__.py
    37:    sys.modules['__mp_main__'] = sys.modules['__main__']

But doing so changes the ordering of `sys.modules`, and `whichmodule` returns `__mp_main__` instead of the correct `scipy.special._ufuncs`.

This bug has arisen in the context of using `dill` (which imports multiprocessing) for serialization and is further documented at https://github.com/uqfoundation/dill/issues/392.

I intend to submit a patch/PR with a fix soon.
History
Date User Action Args
2020-11-19 14:11:06renatolfcsetkeywords: + patch
stage: patch review
pull_requests: + pull_request22295
2020-11-19 14:04:57renatolfcsetcomponents: + Library (Lib), - Interpreter Core
2020-11-19 13:19:31renatolfccreate