This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: Global variables are not accessible from child processes (multiprocessing.Pool)
Type: behavior Stage: resolved
Components: Library (Lib), macOS Versions: Python 3.8
process
Status: closed Resolution: not a bug
Dependencies: Superseder:
Assigned To: Nosy List: agmt, davin, graingert, ned.deily, pitrou, ronaldoussoren, vstinner
Priority: normal Keywords:

Created on 2020-03-11 09:20 by agmt, last changed 2022-04-11 14:59 by admin. This issue is now closed.

Files
File name Uploaded Description Edit
test.py agmt, 2020-03-11 09:20
Messages (5)
msg363892 - (view) Author: agmt (agmt) Date: 2020-03-11 09:20
Attached test works correctly on linux (3.7, 3.8) and mac (only 3.7).
Mac python3.8 falls with exception:

multiprocessing.pool.RemoteTraceback:
"""
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/pool.py", line 125, in worker
    result = (True, func(*args, **kwds))
  File "test.py", line 8, in work
    print(F"Work={arg} args={args}")
NameError: name 'args' is not defined
"""
msg363901 - (view) Author: Thomas Grainger (graingert) * Date: 2020-03-11 10:25
yeah this is normal. on Python 3.8 mac multiprocessing switched to spawn

you have to use https://docs.python.org/3/library/multiprocessing.shared_memory.html to share content between processes or pass it to be pickled in the args of imap_unordered

https://bugs.python.org/issue33725
msg363902 - (view) Author: agmt (agmt) Date: 2020-03-11 10:55
Shared memory/pickle cannot send file descriptors
msg363903 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-03-11 11:00
> Shared memory/pickle cannot send file descriptors

FYI Python 3.9 got new socket.send_fds() and socket.recv_fds() functions :-)
https://docs.python.org/dev/library/socket.html#socket.socket.send_fds
msg380595 - (view) Author: Ronald Oussoren (ronaldoussoren) * (Python committer) Date: 2020-11-09 16:18
Can this issue be closed?  Multiprocessing seems to work as designed. There is a behaviour change between 3.7 and 3.8, but that's documented in What's New.
History
Date User Action Args
2022-04-11 14:59:28adminsetgithub: 84112
2020-11-11 06:27:20josh.rsetstatus: open -> closed
resolution: not a bug
stage: resolved
2020-11-09 16:18:47ronaldoussorensetmessages: + msg380595
2020-03-11 15:25:06ned.deilysetnosy: + pitrou, davin
2020-03-11 11:00:15vstinnersetnosy: + vstinner
messages: + msg363903
2020-03-11 10:55:01agmtsetmessages: + msg363902
2020-03-11 10:25:27graingertsetnosy: + graingert
messages: + msg363901
2020-03-11 09:20:52agmtcreate