Message260032
multiprocessing.imap will hang and not raise an error if an error occurs in the generator that is being mapped over. I'd expect the error to be raised and/or the process to fail.
For example, run the following code in python 2.7 or 3.4:
from multiprocessing import Pool
def add_one(v):
return v+1
pool = Pool(processes=2)
values = ["1", "2", "3", "4", "foo", "5", "6", "7", "8"]
value_iter = (int(v) for v in values)
for new_val in pool.imap(add_one, value_iter):
print(new_val)
And output should look something like this:
$ python demo_hanging.py
2
3
4
5
Exception in thread Thread-2:
Traceback (most recent call last):
File "/usr/lib/python3.4/threading.py", line 920, in _bootstrap_inner
self.run()
File "/usr/lib/python3.4/threading.py", line 868, in run
self._target(*self._args, **self._kwargs)
File "/usr/lib/python3.4/multiprocessing/pool.py", line 378, in _handle_tasks
for i, task in enumerate(taskseq):
File "/usr/lib/python3.4/multiprocessing/pool.py", line 286, in <genexpr>
self._taskqueue.put((((result._job, i, func, (x,), {})
File "demo_hanging.py", line 9, in <genexpr>
value_iter = (int(v) for v in values)
ValueError: invalid literal for int() with base 10: 'foo'
The script will then hang indefinitely. |
|
Date |
User |
Action |
Args |
2016-02-10 19:03:45 | Aaron Halfaker | set | recipients:
+ Aaron Halfaker |
2016-02-10 19:03:45 | Aaron Halfaker | set | messageid: <1455131025.64.0.640942257663.issue26333@psf.upfronthosting.co.za> |
2016-02-10 19:03:45 | Aaron Halfaker | link | issue26333 messages |
2016-02-10 19:03:45 | Aaron Halfaker | create | |
|