classification
Title: ProcessPoolExecutor in interactive shell doesn't work in Windows
Type: behavior Stage: resolved
Components: Windows Versions: Python 3.3
process
Status: closed Resolution: fixed
Dependencies: Superseder: futures.ProcessPoolExecutor hangs
View: 11161
Assigned To: Nosy List: Decade, bquinlan, sbt
Priority: normal Keywords:

Created on 2013-04-30 03:16 by Decade, last changed 2013-11-11 12:58 by sbt. This issue is now closed.

Messages (7)
msg188122 - (view) Author: (Decade) Date: 2013-04-30 03:16
ProcessPoolExecutor doesn't work in an interactive shell in Windows, such as IDLE or the command prompt. It does work in Unix, and it works if I use the ThreadPoolExecutor instead.

For example, let's use the tutorial at http://eli.thegreenplace.net/2013/01/16/python-paralellizing-cpu-bound-tasks-with-concurrent-futures/

I get:
>>> pool_factorizer_chunked([1,2,3,456,7,8],8)
Exception in thread Thread-1:
Traceback (most recent call last):
  File "C:\Python33\lib\threading.py", line 639, in _bootstrap_inner
    self.run()
  File "C:\Python33\lib\threading.py", line 596, in run
    self._target(*self._args, **self._kwargs)
  File "C:\Python33\lib\concurrent\futures\process.py", line 248, in _queue_management_worker
    shutdown_worker()
  File "C:\Python33\lib\concurrent\futures\process.py", line 208, in shutdown_worker
    call_queue.put_nowait(None)
  File "C:\Python33\lib\multiprocessing\queues.py", line 132, in put_nowait
    return self.put(obj, False)
  File "C:\Python33\lib\multiprocessing\queues.py", line 79, in put
    raise Full
queue.Full

Traceback (most recent call last):
  File "<pyshell#10>", line 1, in <module>
    pool_factorizer_chunked([1,2,3,456,7,8],8)
  File "<pyshell#5>", line 14, in pool_factorizer_chunked
    resultdict.update(f.result())
  File "C:\Python33\lib\concurrent\futures\_base.py", line 392, in result
    return self.__get_result()
  File "C:\Python33\lib\concurrent\futures\_base.py", line 351, in __get_result
    raise self._exception
concurrent.futures.process.BrokenProcessPool: A process in the process pool was terminated abruptly while the future was running or pending.
>>> pool_factorizer_map([1,2,3,456,7,8],8)
Exception in thread Thread-2:
Traceback (most recent call last):
  File "C:\Python33\lib\threading.py", line 639, in _bootstrap_inner
    self.run()
  File "C:\Python33\lib\threading.py", line 596, in run
    self._target(*self._args, **self._kwargs)
  File "C:\Python33\lib\concurrent\futures\process.py", line 248, in _queue_management_worker
    shutdown_worker()
  File "C:\Python33\lib\concurrent\futures\process.py", line 208, in shutdown_worker
    call_queue.put_nowait(None)
  File "C:\Python33\lib\multiprocessing\queues.py", line 132, in put_nowait
    return self.put(obj, False)
  File "C:\Python33\lib\multiprocessing\queues.py", line 79, in put
    raise Full
queue.Full

Traceback (most recent call last):
  File "<pyshell#13>", line 1, in <module>
    pool_factorizer_map([1,2,3,456,7,8],8)
  File "<pyshell#12>", line 6, in pool_factorizer_map
    executor.map(factorize_naive, nums))}
  File "<pyshell#12>", line 4, in <dictcomp>
    return {num:factors for num, factors in
  File "C:\Python33\lib\concurrent\futures\_base.py", line 546, in result_iterator
    yield future.result()
  File "C:\Python33\lib\concurrent\futures\_base.py", line 399, in result
    return self.__get_result()
  File "C:\Python33\lib\concurrent\futures\_base.py", line 351, in __get_result
    raise self._exception
concurrent.futures.process.BrokenProcessPool: A process in the process pool was terminated abruptly while the future was running or pending.
>>>
msg188123 - (view) Author: (Decade) Date: 2013-04-30 03:34
Oh, yeah, using Python 3.3.1 (v3.3.1:d9893d13c628, Apr  6 2013, 20:30:21) [MSC v.1600 64 bit (AMD64)] on win32
msg188150 - (view) Author: Richard Oudkerk (sbt) * (Python committer) Date: 2013-04-30 10:43
Some sort of error is expected.

On Windows any functions or classes used in a task sent the executor must be picklable/unpicklable.  This means that they must be defined in an importable module rather than being defined in the interactive shell.

On Unix, you can use a definition made in the interactive shell, but *only* if it was defined before starting the executor.
msg188199 - (view) Author: (Decade) Date: 2013-05-01 01:49
Ah. Then, a documentation error. The error message ("queue.Full"?) and the documentation are totally not clear about that. Does ProcessPoolExecutor just not require tasks to be picklable in Unix?

Also, this raises questions about what exactly "picklable" means, and why it's not defined in the documentation.
msg188205 - (view) Author: Richard Oudkerk (sbt) * (Python committer) Date: 2013-05-01 10:19
> Ah. Then, a documentation error. The error message ("queue.Full"?) and 
> the documentation are totally not clear about that.

Once something goes wrong you are likely to get a cascade of errors, and the first one reported is not necessarily the original cause.

> Does ProcessPoolExecutor just not require tasks to be picklable in Unix?

On Unix the main process forks using os.fork() when the executor is created.  The forked processes inherit all the definitions previously created in the main process.

> Also, this raises questions about what exactly "picklable" means, and why 
> it's not defined in the documentation.

Since concurrent.futures uses multiprocessing the following section applies

    http://docs.python.org/dev/library/multiprocessing.html#windows
msg201274 - (view) Author: Brian Quinlan (bquinlan) * (Python committer) Date: 2013-10-25 18:10
Take a look at http://bugs.python.org/issue11161 and its fix. Do you think that just saying that ProcessPoolExecutor will not work from the interactive shell is sufficient?
msg202624 - (view) Author: Richard Oudkerk (sbt) * (Python committer) Date: 2013-11-11 12:58
Fixed by #11161.
History
Date User Action Args
2013-11-11 12:58:20sbtsetstatus: open -> closed
superseder: futures.ProcessPoolExecutor hangs
messages: + msg202624

resolution: fixed
stage: resolved
2013-10-25 18:10:40bquinlansetnosy: + bquinlan
messages: + msg201274
2013-05-01 10:19:21sbtsetmessages: + msg188205
2013-05-01 01:49:50Decadesetmessages: + msg188199
2013-04-30 10:43:08sbtsettype: crash -> behavior

messages: + msg188150
nosy: + sbt
2013-04-30 03:34:17Decadesetmessages: + msg188123
2013-04-30 03:16:37Decadecreate