This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: Very first multiprocessing example not working on Windows 11
Type: behavior Stage:
Components: Documentation, Windows Versions: Python 3.11
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: docs@python Nosy List: docs@python, eryksun, paul.moore, quattrozhou, rhettinger, steve.dower, terry.reedy, tim.golden, zach.ware
Priority: normal Keywords:

Created on 2021-11-28 04:37 by quattrozhou, last changed 2022-04-11 14:59 by admin.

Messages (4)
msg407186 - (view) Author: Chang Zhou (quattrozhou) Date: 2021-11-28 04:37
Very first multiprocessing example not working on Windows 11

https://docs.python.org/3/library/multiprocessing.html

from multiprocessing import Pool

def f(x):
    return x*x

if __name__ == '__main__':
    with Pool(5) as p:
        print(p.map(f, [1, 2, 3]))

Tried Python 3.7, 3.8, 3.9, 3.10
Also tried Ubuntu which works fine.
But on Windows (clean installed python):


>>> from multiprocessing import Pool
>>> def f(x):
...     return x*x
...
>>> with Pool(5) as p:
...   print(p.map(f, [1, 2, 3]))
...
Process SpawnPoolWorker-14:
Process SpawnPoolWorker-13:
Traceback (most recent call last):
Process SpawnPoolWorker-15:
  File "C:\Program Files\Python39\lib\multiprocessing\process.py", line 315, in _bootstrap
    self.run()
Traceback (most recent call last):
  File "C:\Program Files\Python39\lib\multiprocessing\process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "C:\Program Files\Python39\lib\multiprocessing\process.py", line 315, in _bootstrap
    self.run()
  File "C:\Program Files\Python39\lib\multiprocessing\pool.py", line 114, in worker
    task = get()
  File "C:\Program Files\Python39\lib\multiprocessing\process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "C:\Program Files\Python39\lib\multiprocessing\queues.py", line 368, in get
    return _ForkingPickler.loads(res)
  File "C:\Program Files\Python39\lib\multiprocessing\pool.py", line 114, in worker
    task = get()
AttributeError: Can't get attribute 'f' on <module '__main__' (built-in)>
  File "C:\Program Files\Python39\lib\multiprocessing\queues.py", line 368, in get
    return _ForkingPickler.loads(res)
AttributeError: Can't get attribute 'f' on <module '__main__' (built-in)>
Traceback (most recent call last):
  File "C:\Program Files\Python39\lib\multiprocessing\process.py", line 315, in _bootstrap
    self.run()
  File "C:\Program Files\Python39\lib\multiprocessing\process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "C:\Program Files\Python39\lib\multiprocessing\pool.py", line 114, in worker
    task = get()
  File "C:\Program Files\Python39\lib\multiprocessing\queues.py", line 368, in get
    return _ForkingPickler.loads(res)
AttributeError: Can't get attribute 'f' on <module '__main__' (built-in)>
msg407189 - (view) Author: Eryk Sun (eryksun) * (Python triager) Date: 2021-11-28 05:27
> AttributeError: Can't get attribute 'f' 
> on <module '__main__' (built-in)>

The Windows API only supports the spawn method of process creation. In POSIX (except macOS), the default is the fork method, for which the child inherits the interactive main module of the parent. If you switch to the spawn method in Linux via multiprocessing.set_start_method('spawn'), you'll see the same error. 

multiprocessing is one package where it's necessary in Windows to test examples using a script. This is implied in the guidelines when it says to "[m]ake sure that the main module can be safely imported by a new Python interpreter without causing unintended side effects (such a starting a new process)". It's the fact the main module has to be importable by child processes that matters in this case. The behavior is noted with an example at the end of the introduction:

    Note
    Functionality within this package requires that the __main__ module
    be importable by the children. This is covered in Programming 
    guidelines however it is worth pointing out here. This means that 
    some examples, such as the multiprocessing.pool.Pool examples will 
    not work in the interactive interpreter. For example: ...

If it were up to me this note would be at the beginning of the introduction, where everyone would certainly see it. As is, the reader is expected to at least scan over the entire introduction.
msg407638 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2021-12-04 05:26
I think something should be said right after the example.  But the 20 line  note seems a bit too much at this location.  Perhaps insert

(For reasons given _below_, this will raise error if enter interactively.)

with _below_, say, linked to the 20 line note.

It is not quite clear to me it this only applies on Windows or is general.
msg407646 - (view) Author: Eryk Sun (eryksun) * (Python triager) Date: 2021-12-04 07:37
> not quite clear to me it this only applies on Windows or is general.

macOS defaults to the spawn method (bpo-33725).
History
Date User Action Args
2022-04-11 14:59:52adminsetgithub: 90072
2021-12-04 07:37:31eryksunsetmessages: + msg407646
2021-12-04 05:26:55terry.reedysetversions: + Python 3.11, - Python 3.9, Python 3.10
nosy: + rhettinger, terry.reedy, docs@python

messages: + msg407638

assignee: docs@python
components: + Documentation
2021-11-28 07:55:59AlexWaygoodsettype: crash -> behavior
versions: - Python 3.7, Python 3.8
2021-11-28 05:27:57eryksunsetnosy: + eryksun
messages: + msg407189
2021-11-28 04:37:54quattrozhoucreate