classification
Title: multiprocessing.map skips finally blocks
Type: behavior Stage:
Components: Documentation, Library (Lib) Versions: Python 3.2, Python 3.3, Python 2.7
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: docs@python Nosy List: daniel.wagner-hall, docs@python, flox, jnoller, pitrou, terry.reedy
Priority: normal Keywords:

Created on 2011-10-09 02:05 by daniel.wagner-hall, last changed 2011-10-15 18:52 by jnoller.

Messages (6)
msg145201 - (view) Author: Daniel Wagner-Hall (daniel.wagner-hall) * Date: 2011-10-09 02:05
import random

from multiprocessing import Pool
from time import sleep

def Process(x):
  try:
    print x
    sleep(random.random())
    raise Exception('Exception: ' + x)
  finally:
    print 'Finally: ' + x

Pool(3).map(Process, ['1','2','3'])

Expect all three Finally blocks to be called (or at least, one Finally per x printed by line 8)

Actually, only one (occasionally two) are printed.

Same behaviour exhibited on dual-core Mac running OSX 10.6 with Python 2.7, and single core Ubuntu running Python 2.6.
msg145276 - (view) Author: Florent Xicluna (flox) * (Python committer) Date: 2011-10-09 20:58
Same behavior on Python 3.2 with this code:


from multiprocessing import Pool
from time import sleep

def Process(x):
    try:
        print(x)
        sleep(.6-x/10.)
        raise Exception('Exception: %d' % x)
    finally:
        print('Finally: %d' % x)

Pool(3).map(Process, [1, 2, 3])
msg145280 - (view) Author: Daniel Wagner-Hall (daniel.wagner-hall) * Date: 2011-10-09 21:37
Explanation of behaviour at http://stackoverflow.com/questions/7700929/python-multiprocessing-map-if-one-thread-raises-an-exception-why-arent-other

tl;dr SIGTERM kills subprocesses and finally blocks aren't called.

I still consider this a bug, though
msg145569 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2011-10-14 22:48
One might ask why an Exception in one process kills all in the Pool. Perhaps multiprocessing emulates threading too closely.
Perhaps it is assumed that if one is bad, all are. Jesse?
msg145592 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2011-10-15 14:31
As mentioned in http://docs.python.org/dev/library/multiprocessing#multiprocessing.pool.AsyncResult.get
“If the remote call raised an exception then that exception will be reraised by get().”

map() is just map_async() followed by a get() call on the result.
Also, worker processes are called in daemon mode, which explains that children get killed as soon as the parent exits.

If you rephrase your example as:

try:
    Pool(3).map(Process, ['1','2','3'])
finally:
    sleep(1)

then all the children's finally clauses get a chance to be executed before the parent exits.

I would therefore call it "not a bug", although you might add a sentence in the map() documentation stating that an exception is raised as soon as one of the worker fails.
msg145603 - (view) Author: Jesse Noller (jnoller) * (Python committer) Date: 2011-10-15 18:52
Antoine is correct, as he usually is. This is more of a documentation issue than bug.
History
Date User Action Args
2011-10-15 18:52:05jnollersetmessages: + msg145603
2011-10-15 14:31:29pitrousetnosy: + pitrou, docs@python
messages: + msg145592

assignee: docs@python
components: + Documentation
2011-10-14 22:48:17terry.reedysetnosy: + terry.reedy, jnoller

messages: + msg145569
versions: - Python 2.6
2011-10-09 21:37:38daniel.wagner-hallsetmessages: + msg145280
2011-10-09 20:58:32floxsetversions: + Python 3.2, Python 3.3
nosy: + flox

messages: + msg145276

components: + Library (Lib), - None
2011-10-09 02:05:46daniel.wagner-hallcreate