This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: Multiprocesing Pool borken on macOS REPL
Type: behavior Stage:
Components: Versions:
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: eric.smith, mbussonn, xtreak
Priority: normal Keywords:

Created on 2020-06-19 21:00 by mbussonn, last changed 2022-04-11 14:59 by admin.

Messages (5)
msg371900 - (view) Author: Matthias Bussonnier (mbussonn) * Date: 2020-06-19 21:00
$ python
Python 3.8.2 | packaged by conda-forge | (default, Apr 24 2020, 07:56:27)
[Clang 9.0.1 ] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from multiprocessing import Pool
>>>
>>> def f(x):
...     return x*x
...
 
>>> with Pool(5) as p:
...     print(p.map(f, [1, 2, 3]))
Process SpawnPoolWorker-1:
Process SpawnPoolWorker-2:
Process SpawnPoolWorker-3:
Traceback (most recent call last):
  File "/Users/bussonniermatthias/miniconda3/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap
    self.run()
  File "/Users/bussonniermatthias/miniconda3/lib/python3.8/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/Users/bussonniermatthias/miniconda3/lib/python3.8/multiprocessing/pool.py", line 114, in worker
    task = get()
  File "/Users/bussonniermatthias/miniconda3/lib/python3.8/multiprocessing/queues.py", line 358, in get
    return _ForkingPickler.loads(res)
AttributeError: Can't get attribute 'f' on <module '__main__' (built-in)>
Traceback (most recent call last):
...

This is likely due to https://bugs.python.org/issue33725 (use spawn on MacOS), we we can't use `fork()`.
msg371906 - (view) Author: Eric V. Smith (eric.smith) * (Python committer) Date: 2020-06-19 22:26
To be unpickle-able, the code for f needs to be imported, which it can't be from the repl. Windows has this same issue, due to also not using fork().

From https://docs.python.org/3/library/pickle.html#what-can-be-pickled-and-unpickled: "Thus the defining module must be importable in the unpickling environment, and the module must contain the named object, otherwise an exception will be raised."

I'll wait a while to see who else chimes in, but I think this should be closed as "not a bug".
msg371916 - (view) Author: Matthias Bussonnier (mbussonn) * Date: 2020-06-20 03:24
Thanks.

That's an annoying side effect of having spawn by default on macOS then. 

Could it be possible in `pool.map()` to detect that some of the objects are from main and would fail and then raise an error message in the parent process before starting the children ?
msg371917 - (view) Author: Karthikeyan Singaravelan (xtreak) * (Python committer) Date: 2020-06-20 04:26
There is a documentation ticket already open : https://bugs.python.org/issue33553
msg371918 - (view) Author: Karthikeyan Singaravelan (xtreak) * (Python committer) Date: 2020-06-20 04:36
Some related note in documentation about repl with commit : https://github.com/python/cpython/commit/73dd030c8b71a7080648554652912982054b1177

> Note Functionality within this package requires that the __main__ module be importable by the children. This is covered in Programming guidelines however it is worth pointing out here. This means that some examples, such as the multiprocessing.pool.Pool examples will not work in the interactive interpreter.
History
Date User Action Args
2022-04-11 14:59:32adminsetgithub: 85213
2020-06-20 05:29:16xtreaksetnosy: - Digital India

type: security -> behavior
title: Digital India -> Multiprocesing Pool borken on macOS REPL
2020-06-20 05:24:18xtreaksetmessages: - msg371919
2020-06-20 05:04:47Digital Indiasettitle: Multiprocesing Pool borken on macOS REPL -> Digital India
nosy: + Digital India

messages: + msg371919

type: security
2020-06-20 04:36:03xtreaksetmessages: + msg371918
2020-06-20 04:26:32xtreaksetnosy: + xtreak
messages: + msg371917
2020-06-20 03:24:27mbussonnsetmessages: + msg371916
2020-06-19 22:26:39eric.smithsetnosy: + eric.smith
messages: + msg371906
2020-06-19 21:00:05mbussonncreate