classification
Title: hang and/or leaked processes with multiprocessing.Pool(...).imap(...)
Type: behavior Stage: resolved
Components: Library (Lib) Versions: Python 3.8, Python 3.7
process
Status: closed Resolution: duplicate
Dependencies: Superseder: multiprocessing.Pool.imaps iterators do not maintain alive the multiprocessing.Pool objects
View: 35378
Assigned To: Nosy List: Anthony Sottile, pablogsal, pitrou, remi.lapeyre, vstinner
Priority: normal Keywords:

Created on 2018-12-31 17:28 by Anthony Sottile, last changed 2019-01-04 23:02 by vstinner. This issue is now closed.

Messages (7)
msg332825 - (view) Author: Anthony Sottile (Anthony Sottile) * Date: 2018-12-31 17:28
This simple program causes a hang / leaked processes (easiest to run in an interactive shell):

import multiprocessing
tuple(multiprocessing.Pool(4).imap(print, (1, 2, 3)))

$ python3.6
Python 3.6.7 (default, Oct 22 2018, 11:32:17) 
[GCC 8.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import multiprocessing
>>> tuple(multiprocessing.Pool(4).imap(print, (1, 2, 3)))
1
2
3
<<<hang>>>
^CProcess ForkPoolWorker-1:
Traceback (most recent call last):
Process ForkPoolWorker-2:
Process ForkPoolWorker-3:
Process ForkPoolWorker-4:
  File "/usr/lib/python3.6/multiprocessing/pool.py", line 746, in next
    item = self._items.popleft()
IndexError: pop from an empty deque

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python3.6/multiprocessing/pool.py", line 750, in next
    self._cond.wait(timeout)
  File "/usr/lib/python3.6/threading.py", line 295, in wait
    waiter.acquire()
KeyboardInterrupt


$ python3.7
Python 3.7.2 (default, Dec 25 2018, 03:50:46) 
[GCC 7.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import multiprocessing
>>> tuple(multiprocessing.Pool(4).imap(print, (1, 2, 3)))
1
2
3
(None, None, None)
>>> 
KeyboardInterrupt
Process ForkPoolWorker-3:
Process ForkPoolWorker-1:
Process ForkPoolWorker-2:
Process ForkPoolWorker-4:
>>> Traceback (most recent call last):
  File "/usr/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap
    self.run()
  File "/usr/lib/python3.7/multiprocessing/process.py", line 99, in run
    self._target(*self._args, **self._kwargs)
Traceback (most recent call last):
  File "/usr/lib/python3.7/multiprocessing/pool.py", line 110, in worker
    task = get()
  File "/usr/lib/python3.7/multiprocessing/queues.py", line 351, in get
    with self._rlock:
  File "/usr/lib/python3.7/multiprocessing/synchronize.py", line 95, in __enter__
    return self._semlock.__enter__()
KeyboardInterrupt
  File "/usr/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap
    self.run()
  File "/usr/lib/python3.7/multiprocessing/process.py", line 99, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/lib/python3.7/multiprocessing/pool.py", line 110, in worker
    task = get()
  File "/usr/lib/python3.7/multiprocessing/queues.py", line 351, in get
    with self._rlock:
  File "/usr/lib/python3.7/multiprocessing/synchronize.py", line 95, in __enter__
    return self._semlock.__enter__()
KeyboardInterrupt
Traceback (most recent call last):
  File "/usr/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap
    self.run()
  File "/usr/lib/python3.7/multiprocessing/process.py", line 99, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/lib/python3.7/multiprocessing/pool.py", line 110, in worker
    task = get()
  File "/usr/lib/python3.7/multiprocessing/queues.py", line 351, in get
    with self._rlock:
  File "/usr/lib/python3.7/multiprocessing/synchronize.py", line 95, in __enter__
    return self._semlock.__enter__()
KeyboardInterrupt
Traceback (most recent call last):
  File "/usr/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap
    self.run()
  File "/usr/lib/python3.7/multiprocessing/process.py", line 99, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/lib/python3.7/multiprocessing/pool.py", line 110, in worker
    task = get()
  File "/usr/lib/python3.7/multiprocessing/queues.py", line 352, in get
    res = self._reader.recv_bytes()
  File "/usr/lib/python3.7/multiprocessing/connection.py", line 216, in recv_bytes
    buf = self._recv_bytes(maxlength)
  File "/usr/lib/python3.7/multiprocessing/connection.py", line 407, in _recv_bytes
    buf = self._recv(4)
  File "/usr/lib/python3.7/multiprocessing/connection.py", line 379, in _recv
    chunk = read(handle, remaining)
KeyboardInterrupt


(python3.8 shows the same behaviour as python3.7)
$ ./python --version --version
Python 3.8.0a0 (heads/master:ede0b6fae2, Dec 31 2018, 09:19:17) 
[GCC 7.3.0]


python2.7 also has similar behaviour.

I'm told this more reliably hangs on windows, though I don't have windows on hand.

I've "fixed" my code to explicitly open / close the pool:

with contextlib.closing(multiprocessing.Pool(jobs)) as pool:
    tuple(pool.imap(...))


I suspect a refcounting / gc bug
msg332841 - (view) Author: Rémi Lapeyre (remi.lapeyre) * Date: 2019-01-01 00:13
Weirdly enough, it works with iPython:

    $ ipython3
    Python 3.7.1 (default, Nov  6 2018, 18:49:54)
    Type 'copyright', 'credits' or 'license' for more information
    IPython 6.4.0 -- An enhanced Interactive Python. Type '?' for help.

    In [1]: import multiprocessing

    In [2]: tuple(multiprocessing.Pool(4).imap(print, (1, 2, 3)))
    ...:
    1
    2
    3
    Out[2]: (None, None, None)

    In [3]:
msg332899 - (view) Author: Rémi Lapeyre (remi.lapeyre) * Date: 2019-01-02 23:36
I believe that this is similar to https://bugs.python.org/issue35378 on which @pablogsal is working.

You were right, the issue steems from a refcount bug. Until the resolution you can avoid the issue by explictly keeping a reference on the pool:

>>> import multiprocessing
>>> d = multiprocessing.Pool(4)
>>> tuple(d.imap(print, (1, 2, 3)))
1
2
3
(None, None, None)
>>>
msg332995 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2019-01-04 19:44
Indeed, looks like a duplicate.
msg333013 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-01-04 22:40
I suggest you to write:

with multiprocessing.Pool(4) as pool: result = tuple(pool.imap(print, (1, 2, 3)))

On Python 3.8, your example will now log a resource warning since you don't close/terminate explicitly the pool.
msg333014 - (view) Author: Anthony Sottile (Anthony Sottile) * Date: 2019-01-04 22:42
If you see the bottom of my issue, I've suggested (nearly) the same thing -- though I require python2.x compatibility so I'm using `contextlib.closing`
msg333019 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-01-04 23:02
> I'm using `contextlib.closing`

Oh, I missed that: good!
History
Date User Action Args
2019-01-04 23:02:09vstinnersetmessages: + msg333019
2019-01-04 22:42:42Anthony Sottilesetmessages: + msg333014
2019-01-04 22:40:33vstinnersetnosy: + vstinner
messages: + msg333013
2019-01-04 19:44:32pitrousetstatus: open -> closed
superseder: multiprocessing.Pool.imaps iterators do not maintain alive the multiprocessing.Pool objects
messages: + msg332995

resolution: duplicate
stage: resolved
2019-01-04 19:34:10terry.reedysetnosy: + pitrou

versions: - Python 3.6
2019-01-02 23:37:38remi.lapeyresetnosy: + pablogsal
2019-01-02 23:36:59remi.lapeyresetmessages: + msg332899
2019-01-01 00:13:42remi.lapeyresetnosy: + remi.lapeyre
messages: + msg332841
2018-12-31 17:28:33Anthony Sottilecreate