Message365249
The fix applied for this problem actually broke multiprocessing on MacOS. The change to the new default 'spawn' from 'fork' causes program to crash in spawn.py with `FileNotFoundError: [Errno 2] No such file or directory`.
I've tested this on MacOS Catalina 10.15.3 and 10.15.4, with Python-3.8.2 and Python-3.7.7.
With Python-3.7.7 everything works as expected.
Here's the output:
{{{
$ python3.8 multi1.py
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/opt/local/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main
exitcode = _main(fd, parent_sentinel)
File "/opt/local/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/spawn.py", line 126, in _main
self = reduction.pickle.load(from_parent)
File "/opt/local/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/synchronize.py", line 110, in __setstate__
self._semlock = _multiprocessing.SemLock._rebuild(*state)
FileNotFoundError: [Errno 2] No such file or directory
}}}
Here's the program:
{{{
#!/usr/bin/env python3
#
# Test "multiprocessing" package included with Python-3.6+
#
# Usage:
# ./mylti1.py [nElements [nProcesses [tSleep]]]
#
# nElements - total number of integers to put in the queue
# default: 100
# nProcesses - total number of parallel processes/threads
# default: number of physical cores available
# tSleep - number of milliseconds for a thread to sleep
# after it retrieved an element from the queue
# default: 17
#
# Algorithm:
# 1. Creates a queue and adds nElements integers to it,
# 2. Creates nProcesses threads
# 3. Each thread extracts an element from the queue and sleeps for tSleep milliseconds
#
import sys, queue, time
import multiprocessing as mp
def getElements(q, tSleep, idx):
l = [] # list of pulled numbers
while True:
try:
l.append(q.get(True, .001))
time.sleep(tSleep)
except queue.Empty:
if q.empty():
print(f'worker {idx} done, got {len(l)} numbers')
return
if __name__ == '__main__':
nElements = int(sys.argv[1]) if len(sys.argv) > 1 else 100
nProcesses = int(sys.argv[2]) if len(sys.argv) > 2 else mp.cpu_count()
tSleep = float(sys.argv[3]) if len(sys.argv) > 3 else 17
# Uncomment the following line to make it working with Python-3.8+
#mp.set_start_method('fork')
# Fill the queue with numbers from 0 to nElements
q = mp.Queue()
for k in range(nElements):
q.put(k)
# Start worker processes
for m in range(nProcesses):
p = mp.Process(target=getElements, args=(q, tSleep / 1000, m))
p.start()
}}} |
|
Date |
User |
Action |
Args |
2020-03-29 10:42:25 | mouse07410 | set | recipients:
+ mouse07410, barry, gregory.p.smith, ronaldoussoren, pitrou, vstinner, ned.deily, lukasz.langa, josh.r, tdsmith, davin, Ryan May, pablogsal, miss-islington, kapilt |
2020-03-29 10:42:25 | mouse07410 | set | messageid: <1585478545.75.0.00410545402563.issue33725@roundup.psfhosted.org> |
2020-03-29 10:42:25 | mouse07410 | link | issue33725 messages |
2020-03-29 10:42:25 | mouse07410 | create | |
|