Issue17874
This issue tracker has been migrated to GitHub,
and is currently read-only.
For more information,
see the GitHub FAQs in the Python's Developer Guide.
Created on 2013-04-30 03:16 by Decade, last changed 2022-04-11 14:57 by admin. This issue is now closed.
Messages (7) | |||
---|---|---|---|
msg188122 - (view) | Author: (Decade) | Date: 2013-04-30 03:16 | |
ProcessPoolExecutor doesn't work in an interactive shell in Windows, such as IDLE or the command prompt. It does work in Unix, and it works if I use the ThreadPoolExecutor instead. For example, let's use the tutorial at http://eli.thegreenplace.net/2013/01/16/python-paralellizing-cpu-bound-tasks-with-concurrent-futures/ I get: >>> pool_factorizer_chunked([1,2,3,456,7,8],8) Exception in thread Thread-1: Traceback (most recent call last): File "C:\Python33\lib\threading.py", line 639, in _bootstrap_inner self.run() File "C:\Python33\lib\threading.py", line 596, in run self._target(*self._args, **self._kwargs) File "C:\Python33\lib\concurrent\futures\process.py", line 248, in _queue_management_worker shutdown_worker() File "C:\Python33\lib\concurrent\futures\process.py", line 208, in shutdown_worker call_queue.put_nowait(None) File "C:\Python33\lib\multiprocessing\queues.py", line 132, in put_nowait return self.put(obj, False) File "C:\Python33\lib\multiprocessing\queues.py", line 79, in put raise Full queue.Full Traceback (most recent call last): File "<pyshell#10>", line 1, in <module> pool_factorizer_chunked([1,2,3,456,7,8],8) File "<pyshell#5>", line 14, in pool_factorizer_chunked resultdict.update(f.result()) File "C:\Python33\lib\concurrent\futures\_base.py", line 392, in result return self.__get_result() File "C:\Python33\lib\concurrent\futures\_base.py", line 351, in __get_result raise self._exception concurrent.futures.process.BrokenProcessPool: A process in the process pool was terminated abruptly while the future was running or pending. >>> pool_factorizer_map([1,2,3,456,7,8],8) Exception in thread Thread-2: Traceback (most recent call last): File "C:\Python33\lib\threading.py", line 639, in _bootstrap_inner self.run() File "C:\Python33\lib\threading.py", line 596, in run self._target(*self._args, **self._kwargs) File "C:\Python33\lib\concurrent\futures\process.py", line 248, in _queue_management_worker shutdown_worker() File "C:\Python33\lib\concurrent\futures\process.py", line 208, in shutdown_worker call_queue.put_nowait(None) File "C:\Python33\lib\multiprocessing\queues.py", line 132, in put_nowait return self.put(obj, False) File "C:\Python33\lib\multiprocessing\queues.py", line 79, in put raise Full queue.Full Traceback (most recent call last): File "<pyshell#13>", line 1, in <module> pool_factorizer_map([1,2,3,456,7,8],8) File "<pyshell#12>", line 6, in pool_factorizer_map executor.map(factorize_naive, nums))} File "<pyshell#12>", line 4, in <dictcomp> return {num:factors for num, factors in File "C:\Python33\lib\concurrent\futures\_base.py", line 546, in result_iterator yield future.result() File "C:\Python33\lib\concurrent\futures\_base.py", line 399, in result return self.__get_result() File "C:\Python33\lib\concurrent\futures\_base.py", line 351, in __get_result raise self._exception concurrent.futures.process.BrokenProcessPool: A process in the process pool was terminated abruptly while the future was running or pending. >>> |
|||
msg188123 - (view) | Author: (Decade) | Date: 2013-04-30 03:34 | |
Oh, yeah, using Python 3.3.1 (v3.3.1:d9893d13c628, Apr 6 2013, 20:30:21) [MSC v.1600 64 bit (AMD64)] on win32 |
|||
msg188150 - (view) | Author: Richard Oudkerk (sbt) * | Date: 2013-04-30 10:43 | |
Some sort of error is expected. On Windows any functions or classes used in a task sent the executor must be picklable/unpicklable. This means that they must be defined in an importable module rather than being defined in the interactive shell. On Unix, you can use a definition made in the interactive shell, but *only* if it was defined before starting the executor. |
|||
msg188199 - (view) | Author: (Decade) | Date: 2013-05-01 01:49 | |
Ah. Then, a documentation error. The error message ("queue.Full"?) and the documentation are totally not clear about that. Does ProcessPoolExecutor just not require tasks to be picklable in Unix? Also, this raises questions about what exactly "picklable" means, and why it's not defined in the documentation. |
|||
msg188205 - (view) | Author: Richard Oudkerk (sbt) * | Date: 2013-05-01 10:19 | |
> Ah. Then, a documentation error. The error message ("queue.Full"?) and > the documentation are totally not clear about that. Once something goes wrong you are likely to get a cascade of errors, and the first one reported is not necessarily the original cause. > Does ProcessPoolExecutor just not require tasks to be picklable in Unix? On Unix the main process forks using os.fork() when the executor is created. The forked processes inherit all the definitions previously created in the main process. > Also, this raises questions about what exactly "picklable" means, and why > it's not defined in the documentation. Since concurrent.futures uses multiprocessing the following section applies http://docs.python.org/dev/library/multiprocessing.html#windows |
|||
msg201274 - (view) | Author: Brian Quinlan (bquinlan) * | Date: 2013-10-25 18:10 | |
Take a look at http://bugs.python.org/issue11161 and its fix. Do you think that just saying that ProcessPoolExecutor will not work from the interactive shell is sufficient? |
|||
msg202624 - (view) | Author: Richard Oudkerk (sbt) * | Date: 2013-11-11 12:58 | |
Fixed by #11161. |
History | |||
---|---|---|---|
Date | User | Action | Args |
2022-04-11 14:57:45 | admin | set | github: 62074 |
2013-11-11 12:58:20 | sbt | set | status: open -> closed superseder: futures.ProcessPoolExecutor hangs messages: + msg202624 resolution: fixed stage: resolved |
2013-10-25 18:10:40 | bquinlan | set | nosy:
+ bquinlan messages: + msg201274 |
2013-05-01 10:19:21 | sbt | set | messages: + msg188205 |
2013-05-01 01:49:50 | Decade | set | messages: + msg188199 |
2013-04-30 10:43:08 | sbt | set | type: crash -> behavior messages: + msg188150 nosy: + sbt |
2013-04-30 03:34:17 | Decade | set | messages: + msg188123 |
2013-04-30 03:16:37 | Decade | create |