classification
Title: exception unpickling error causes `multiprocessing.Pool` to hang forever
Type: behavior Stage:
Components: Library (Lib) Versions: Python 3.8, Python 3.7
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: Anthony Sottile, alexandre.vassalotti
Priority: normal Keywords:

Created on 2018-11-25 22:02 by Anthony Sottile, last changed 2018-11-28 16:21 by SilentGhost.

Messages (2)
msg330397 - (view) Author: Anthony Sottile (Anthony Sottile) * Date: 2018-11-25 22:02
```
import multiprocessing

class E(Exception):
    def __init__(self, a1, a2):
        Exception.__init__(self, '{}{}'.format(a1, a2))

def f(_):
    raise E(1, 2)

multiprocessing.Pool(1).map(f, (1,))
```


Running this causes a hang:

```
$ python3.7 t2.py 
Exception in thread Thread-3:
Traceback (most recent call last):
  File "/usr/lib/python3.7/threading.py", line 917, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.7/threading.py", line 865, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/lib/python3.7/multiprocessing/pool.py", line 496, in _handle_results
    task = get()
  File "/usr/lib/python3.7/multiprocessing/contrnection.py", line 251, in recv
    return _ForkingPickler.loads(buf.getbuffer())
TypeError: __init__() missing 1 required positional argument: 'a2'

```

Upon eventual `^C`

```
^CTraceback (most recent call last):
Process ForkPoolWorker-1:
  File "t2.py", line 10, in <module>
    multiprocessing.Pool(1).map(f, (1,))
  File "/usr/lib/python3.7/multiprocessing/pool.py", line 290, in map
    return self._map_async(func, iterable, mapstar, chunksize).get()
  File "/usr/lib/python3.7/multiprocessing/pool.py", line 677, in get
    self.wait(timeout)
  File "/usr/lib/python3.7/multiprocessing/pool.py", line 674, in wait
Traceback (most recent call last):
  File "/usr/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap
    self.run()
  File "/usr/lib/python3.7/multiprocessing/process.py", line 99, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/lib/python3.7/multiprocessing/pool.py", line 110, in worker
    task = get()
  File "/usr/lib/python3.7/multiprocessing/queues.py", line 352, in get
    res = self._reader.recv_bytes()
  File "/usr/lib/python3.7/multiprocessing/connection.py", line 216, in recv_bytes
    buf = self._recv_bytes(maxlength)
  File "/usr/lib/python3.7/multiprocessing/connection.py", line 407, in _recv_bytes
    buf = self._recv(4)
  File "/usr/lib/python3.7/multiprocessing/connection.py", line 379, in _recv
    chunk = read(handle, remaining)
KeyboardInterrupt
    self._event.wait(timeout)
  File "/usr/lib/python3.7/threading.py", line 552, in wait
    signaled = self._cond.wait(timeout)
  File "/usr/lib/python3.7/threading.py", line 296, in wait
    waiter.acquire()
KeyboardInterrupt
Traceback (most recent call last):
  File "/usr/lib/python3.7/multiprocessing/util.py", line 265, in _run_finalizers
    finalizer()
  File "/usr/lib/python3.7/multiprocessing/util.py", line 189, in __call__
    res = self._callback(*self._args, **self._kwargs)
  File "/usr/lib/python3.7/multiprocessing/pool.py", line 611, in _terminate_pool
    "Cannot have cache with result_hander not alive")
AssertionError: Cannot have cache with result_hander not alive
```

(I've also tried this against 158695817d736df8b18682866033c87e46252309">master@158695817d736df8b18682866033c87e46252309)
msg330398 - (view) Author: Anthony Sottile (Anthony Sottile) * Date: 2018-11-25 22:07
`concurrent.futures` is affected as well:

```
import concurrent.futures

class E(Exception):
    def __init__(self, a1, a2):
        Exception.__init__(self, '{}{}'.format(a1, a2))

def f(_):
    raise E(1, 2)

with concurrent.futures.ProcessPoolExecutor(2) as exe:
    for _ in exe.map(f, (1,)):
        pass
```
History
Date User Action Args
2018-11-28 16:21:58SilentGhostsetnosy: + alexandre.vassalotti
type: behavior
2018-11-25 22:07:36Anthony Sottilesetmessages: + msg330398
2018-11-25 22:02:05Anthony Sottilecreate