This simple program causes a hang / leaked processes (easiest to run in an interactive shell):
import multiprocessing
tuple(multiprocessing.Pool(4).imap(print, (1, 2, 3)))
$ python3.6
Python 3.6.7 (default, Oct 22 2018, 11:32:17)
[GCC 8.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import multiprocessing
>>> tuple(multiprocessing.Pool(4).imap(print, (1, 2, 3)))
1
2
3
<<<hang>>>
^CProcess ForkPoolWorker-1:
Traceback (most recent call last):
Process ForkPoolWorker-2:
Process ForkPoolWorker-3:
Process ForkPoolWorker-4:
File "/usr/lib/python3.6/multiprocessing/pool.py", line 746, in next
item = self._items.popleft()
IndexError: pop from an empty deque
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/lib/python3.6/multiprocessing/pool.py", line 750, in next
self._cond.wait(timeout)
File "/usr/lib/python3.6/threading.py", line 295, in wait
waiter.acquire()
KeyboardInterrupt
$ python3.7
Python 3.7.2 (default, Dec 25 2018, 03:50:46)
[GCC 7.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import multiprocessing
>>> tuple(multiprocessing.Pool(4).imap(print, (1, 2, 3)))
1
2
3
(None, None, None)
>>>
KeyboardInterrupt
Process ForkPoolWorker-3:
Process ForkPoolWorker-1:
Process ForkPoolWorker-2:
Process ForkPoolWorker-4:
>>> Traceback (most recent call last):
File "/usr/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap
self.run()
File "/usr/lib/python3.7/multiprocessing/process.py", line 99, in run
self._target(*self._args, **self._kwargs)
Traceback (most recent call last):
File "/usr/lib/python3.7/multiprocessing/pool.py", line 110, in worker
task = get()
File "/usr/lib/python3.7/multiprocessing/queues.py", line 351, in get
with self._rlock:
File "/usr/lib/python3.7/multiprocessing/synchronize.py", line 95, in __enter__
return self._semlock.__enter__()
KeyboardInterrupt
File "/usr/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap
self.run()
File "/usr/lib/python3.7/multiprocessing/process.py", line 99, in run
self._target(*self._args, **self._kwargs)
File "/usr/lib/python3.7/multiprocessing/pool.py", line 110, in worker
task = get()
File "/usr/lib/python3.7/multiprocessing/queues.py", line 351, in get
with self._rlock:
File "/usr/lib/python3.7/multiprocessing/synchronize.py", line 95, in __enter__
return self._semlock.__enter__()
KeyboardInterrupt
Traceback (most recent call last):
File "/usr/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap
self.run()
File "/usr/lib/python3.7/multiprocessing/process.py", line 99, in run
self._target(*self._args, **self._kwargs)
File "/usr/lib/python3.7/multiprocessing/pool.py", line 110, in worker
task = get()
File "/usr/lib/python3.7/multiprocessing/queues.py", line 351, in get
with self._rlock:
File "/usr/lib/python3.7/multiprocessing/synchronize.py", line 95, in __enter__
return self._semlock.__enter__()
KeyboardInterrupt
Traceback (most recent call last):
File "/usr/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap
self.run()
File "/usr/lib/python3.7/multiprocessing/process.py", line 99, in run
self._target(*self._args, **self._kwargs)
File "/usr/lib/python3.7/multiprocessing/pool.py", line 110, in worker
task = get()
File "/usr/lib/python3.7/multiprocessing/queues.py", line 352, in get
res = self._reader.recv_bytes()
File "/usr/lib/python3.7/multiprocessing/connection.py", line 216, in recv_bytes
buf = self._recv_bytes(maxlength)
File "/usr/lib/python3.7/multiprocessing/connection.py", line 407, in _recv_bytes
buf = self._recv(4)
File "/usr/lib/python3.7/multiprocessing/connection.py", line 379, in _recv
chunk = read(handle, remaining)
KeyboardInterrupt
(python3.8 shows the same behaviour as python3.7)
$ ./python --version --version
Python 3.8.0a0 (heads/master:ede0b6fae2, Dec 31 2018, 09:19:17)
[GCC 7.3.0]
python2.7 also has similar behaviour.
I'm told this more reliably hangs on windows, though I don't have windows on hand.
I've "fixed" my code to explicitly open / close the pool:
with contextlib.closing(multiprocessing.Pool(jobs)) as pool:
tuple(pool.imap(...))
I suspect a refcounting / gc bug |