Issue35629
This issue tracker has been migrated to GitHub,
and is currently read-only.
For more information,
see the GitHub FAQs in the Python's Developer Guide.
Created on 2018-12-31 17:28 by Anthony Sottile, last changed 2022-04-11 14:59 by admin. This issue is now closed.
Messages (7) | |||
---|---|---|---|
msg332825 - (view) | Author: Anthony Sottile (Anthony Sottile) * | Date: 2018-12-31 17:28 | |
This simple program causes a hang / leaked processes (easiest to run in an interactive shell): import multiprocessing tuple(multiprocessing.Pool(4).imap(print, (1, 2, 3))) $ python3.6 Python 3.6.7 (default, Oct 22 2018, 11:32:17) [GCC 8.2.0] on linux Type "help", "copyright", "credits" or "license" for more information. >>> import multiprocessing >>> tuple(multiprocessing.Pool(4).imap(print, (1, 2, 3))) 1 2 3 <<<hang>>> ^CProcess ForkPoolWorker-1: Traceback (most recent call last): Process ForkPoolWorker-2: Process ForkPoolWorker-3: Process ForkPoolWorker-4: File "/usr/lib/python3.6/multiprocessing/pool.py", line 746, in next item = self._items.popleft() IndexError: pop from an empty deque During handling of the above exception, another exception occurred: Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/usr/lib/python3.6/multiprocessing/pool.py", line 750, in next self._cond.wait(timeout) File "/usr/lib/python3.6/threading.py", line 295, in wait waiter.acquire() KeyboardInterrupt $ python3.7 Python 3.7.2 (default, Dec 25 2018, 03:50:46) [GCC 7.3.0] on linux Type "help", "copyright", "credits" or "license" for more information. >>> import multiprocessing >>> tuple(multiprocessing.Pool(4).imap(print, (1, 2, 3))) 1 2 3 (None, None, None) >>> KeyboardInterrupt Process ForkPoolWorker-3: Process ForkPoolWorker-1: Process ForkPoolWorker-2: Process ForkPoolWorker-4: >>> Traceback (most recent call last): File "/usr/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap self.run() File "/usr/lib/python3.7/multiprocessing/process.py", line 99, in run self._target(*self._args, **self._kwargs) Traceback (most recent call last): File "/usr/lib/python3.7/multiprocessing/pool.py", line 110, in worker task = get() File "/usr/lib/python3.7/multiprocessing/queues.py", line 351, in get with self._rlock: File "/usr/lib/python3.7/multiprocessing/synchronize.py", line 95, in __enter__ return self._semlock.__enter__() KeyboardInterrupt File "/usr/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap self.run() File "/usr/lib/python3.7/multiprocessing/process.py", line 99, in run self._target(*self._args, **self._kwargs) File "/usr/lib/python3.7/multiprocessing/pool.py", line 110, in worker task = get() File "/usr/lib/python3.7/multiprocessing/queues.py", line 351, in get with self._rlock: File "/usr/lib/python3.7/multiprocessing/synchronize.py", line 95, in __enter__ return self._semlock.__enter__() KeyboardInterrupt Traceback (most recent call last): File "/usr/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap self.run() File "/usr/lib/python3.7/multiprocessing/process.py", line 99, in run self._target(*self._args, **self._kwargs) File "/usr/lib/python3.7/multiprocessing/pool.py", line 110, in worker task = get() File "/usr/lib/python3.7/multiprocessing/queues.py", line 351, in get with self._rlock: File "/usr/lib/python3.7/multiprocessing/synchronize.py", line 95, in __enter__ return self._semlock.__enter__() KeyboardInterrupt Traceback (most recent call last): File "/usr/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap self.run() File "/usr/lib/python3.7/multiprocessing/process.py", line 99, in run self._target(*self._args, **self._kwargs) File "/usr/lib/python3.7/multiprocessing/pool.py", line 110, in worker task = get() File "/usr/lib/python3.7/multiprocessing/queues.py", line 352, in get res = self._reader.recv_bytes() File "/usr/lib/python3.7/multiprocessing/connection.py", line 216, in recv_bytes buf = self._recv_bytes(maxlength) File "/usr/lib/python3.7/multiprocessing/connection.py", line 407, in _recv_bytes buf = self._recv(4) File "/usr/lib/python3.7/multiprocessing/connection.py", line 379, in _recv chunk = read(handle, remaining) KeyboardInterrupt (python3.8 shows the same behaviour as python3.7) $ ./python --version --version Python 3.8.0a0 (heads/master:ede0b6fae2, Dec 31 2018, 09:19:17) [GCC 7.3.0] python2.7 also has similar behaviour. I'm told this more reliably hangs on windows, though I don't have windows on hand. I've "fixed" my code to explicitly open / close the pool: with contextlib.closing(multiprocessing.Pool(jobs)) as pool: tuple(pool.imap(...)) I suspect a refcounting / gc bug |
|||
msg332841 - (view) | Author: Rémi Lapeyre (remi.lapeyre) * | Date: 2019-01-01 00:13 | |
Weirdly enough, it works with iPython: $ ipython3 Python 3.7.1 (default, Nov 6 2018, 18:49:54) Type 'copyright', 'credits' or 'license' for more information IPython 6.4.0 -- An enhanced Interactive Python. Type '?' for help. In [1]: import multiprocessing In [2]: tuple(multiprocessing.Pool(4).imap(print, (1, 2, 3))) ...: 1 2 3 Out[2]: (None, None, None) In [3]: |
|||
msg332899 - (view) | Author: Rémi Lapeyre (remi.lapeyre) * | Date: 2019-01-02 23:36 | |
I believe that this is similar to https://bugs.python.org/issue35378 on which @pablogsal is working. You were right, the issue steems from a refcount bug. Until the resolution you can avoid the issue by explictly keeping a reference on the pool: >>> import multiprocessing >>> d = multiprocessing.Pool(4) >>> tuple(d.imap(print, (1, 2, 3))) 1 2 3 (None, None, None) >>> |
|||
msg332995 - (view) | Author: Antoine Pitrou (pitrou) * | Date: 2019-01-04 19:44 | |
Indeed, looks like a duplicate. |
|||
msg333013 - (view) | Author: STINNER Victor (vstinner) * | Date: 2019-01-04 22:40 | |
I suggest you to write: with multiprocessing.Pool(4) as pool: result = tuple(pool.imap(print, (1, 2, 3))) On Python 3.8, your example will now log a resource warning since you don't close/terminate explicitly the pool. |
|||
msg333014 - (view) | Author: Anthony Sottile (Anthony Sottile) * | Date: 2019-01-04 22:42 | |
If you see the bottom of my issue, I've suggested (nearly) the same thing -- though I require python2.x compatibility so I'm using `contextlib.closing` |
|||
msg333019 - (view) | Author: STINNER Victor (vstinner) * | Date: 2019-01-04 23:02 | |
> I'm using `contextlib.closing` Oh, I missed that: good! |
History | |||
---|---|---|---|
Date | User | Action | Args |
2022-04-11 14:59:09 | admin | set | github: 79810 |
2019-01-04 23:02:09 | vstinner | set | messages: + msg333019 |
2019-01-04 22:42:42 | Anthony Sottile | set | messages: + msg333014 |
2019-01-04 22:40:33 | vstinner | set | nosy:
+ vstinner messages: + msg333013 |
2019-01-04 19:44:32 | pitrou | set | status: open -> closed superseder: multiprocessing.Pool.imaps iterators do not maintain alive the multiprocessing.Pool objects messages: + msg332995 resolution: duplicate stage: resolved |
2019-01-04 19:34:10 | terry.reedy | set | nosy:
+ pitrou versions: - Python 3.6 |
2019-01-02 23:37:38 | remi.lapeyre | set | nosy:
+ pablogsal |
2019-01-02 23:36:59 | remi.lapeyre | set | messages: + msg332899 |
2019-01-01 00:13:42 | remi.lapeyre | set | nosy:
+ remi.lapeyre messages: + msg332841 |
2018-12-31 17:28:33 | Anthony Sottile | create |