In pool.py, the worker function reads as follows:
http://svn.python.org/view/python/trunk/Lib/multiprocessing/pool.py?view=markup
"""
68 job, i, func, args, kwds = task
69 try:
70 result = (True, func(*args, **kwds))
71 except Exception, e:
72 result = (False, e)
...
488 if self._success:
489 return self._value
490 else:
491 raise self._value
"""
If an exception is raised in the function you sent to the pool, the exception you get has "raise self._value" as the last line; which is correct but useless for debugging.
mp_exception_bug.py reproduces this error. This is the output:
"""
Traceback (most recent call last):
File "mp_exception_bug.py", line 8, in <module>
print p.map(f, [1,2,3])
File "/usr/lib/python2.7/multiprocessing/pool.py", line 227, in map
return self.map_async(func, iterable, chunksize).get()
File "/usr/lib/python2.7/multiprocessing/pool.py", line 528, in get
raise self._value
NameError: global name 'y' is not defined
"""
As you can imagine, "NameError: global name 'y' is not defined" is not enough in complex projects.
If we apply some changes to the pool.py we could get something similar to this:
"""
Traceback (most recent call last):
File "/usr/lib/python2.7/multiprocessing/pool.py", line 98, in worker
result = (True, func(*args, **kwds))
File "/usr/lib/python2.7/multiprocessing/pool.py", line 67, in mapstar
return map(*args)
File "mp_exception_bug.py", line 4, in f
return x*y
NameError: global name 'y' is not defined
Traceback (most recent call last):
File "mp_exception_bug.py", line 8, in <module>
print p.map(f, [1,2,3])
File "/usr/lib/python2.7/multiprocessing/pool.py", line 231, in map
return self.map_async(func, iterable, chunksize).get()
File "/usr/lib/python2.7/multiprocessing/pool.py", line 535, in get
raise self._value[0]
NameError: global name 'y' is not defined
"""
The patch is simple but ugly:
"""
> import sys
> import traceback
72c100,102
< result = (False, e)
---
> exc_info = sys.exc_info()
> tb_string = traceback.format_exc(exc_info[2])
> result = (False, (e, tb_string))
491c532,535
< raise self._value
---
> # Do something with the exception here, the simplest (but ugliest)
> # thing to do is to simply print it to the console
> print self._value[1]
> raise self._value[0]
"""
Note that I tried to replace the "raise self._value[0]" with a raise with three parameters, being the last one the traceback we get using "exc_info = sys.exc_info()" but sadly it is not possible to pickle tracebacks.
I understand that printing is not the best thing to do here, but I wanted to get this discussion started and come to a real solution.
Thanks
|