This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: Hangs in concurrent.futures
Type: behavior Stage: resolved
Components: Library (Lib) Versions: Python 3.3
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: pitrou, python-dev, rosslagerwall
Priority: normal Keywords: patch

Created on 2011-07-01 05:25 by rosslagerwall, last changed 2022-04-11 14:57 by admin. This issue is now closed.

Files
File name Uploaded Description Edit
cfshutdown.patch pitrou, 2011-07-01 16:47 review
Messages (8)
msg139541 - (view) Author: Ross Lagerwall (rosslagerwall) (Python committer) Date: 2011-07-01 05:25
6d6099f7fe89 introduced a regression.

It causes the following code to hang fairly reliably for me:
"""
import concurrent.futures
import math

def is_prime(n):
    print(sqt(81))  <---- Note the type error!

def main():
    with concurrent.futures.ProcessPoolExecutor(2) as executor:
        executor.map(is_prime, [1, 2])

if __name__ == '__main__':
    main()
"""

From my limited knowledge of multiprocessing and concurrent futures, it seems that in shutdown_worker(), call_queue.put(None) hangs because the queue is full and there are no more workers consuming items in the queue (they exited early).

Increasing EXTRA_QUEUED_CALLS fixes the problem, though its probably not the correct solution.
msg139545 - (view) Author: Ross Lagerwall (rosslagerwall) (Python committer) Date: 2011-07-01 05:49
Further analysis seems to indicate that it is a race between the shutdown code and the processes finishing.

This code hangs:
"""
executor = concurrent.futures.ProcessPoolExecutor(2)
executor.map(is_prime, [1, 2])
executor.shutdown()
"""

while this doesn't:
"""
executor = concurrent.futures.ProcessPoolExecutor(2)
executor.map(is_prime, [1, 2])
time.sleep(1)
executor.shutdown()
"""
msg139587 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2011-07-01 16:47
This patch should fix the issue. Can you confirm?
msg139593 - (view) Author: Ross Lagerwall (rosslagerwall) (Python committer) Date: 2011-07-01 18:45
Yes, the patch does appear to fix the issue.

Thanks
msg139664 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2011-07-02 19:21
New changeset 51c1f2cedb96 by Antoine Pitrou in branch 'default':
Issue #12456: fix a possible hang on shutdown of a concurrent.futures.ProcessPoolExecutor.
http://hg.python.org/cpython/rev/51c1f2cedb96
msg139680 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2011-07-03 10:07
Looks like I broke the OS X buildbots:
http://www.python.org/dev/buildbot/all/builders/AMD64%20Snow%20Leopard%202%203.x/builds/609/steps/test/logs/stdio
msg139681 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2011-07-03 10:46
Ok, qsize() can raise NotImplementedError on OS X, which explains quite a bit:

Exception in thread Thread-2:
Traceback (most recent call last):
  File "/Users/buildbot/buildarea/custom.parc-snowleopard-1/build/Lib/threading.py", line 737, in _bootstrap_inner
    self.run()
  File "/Users/buildbot/buildarea/custom.parc-snowleopard-1/build/Lib/threading.py", line 690, in run
    self._target(*self._args, **self._kwargs)
  File "/Users/buildbot/buildarea/custom.parc-snowleopard-1/build/Lib/concurrent/futures/process.py", line 271, in _queue_management_worker
    if not pending_work_items and call_queue.qsize() == 0:
  File "/Users/buildbot/buildarea/custom.parc-snowleopard-1/build/Lib/multiprocessing/queues.py", line 139, in qsize
    return self._maxsize - self._sem._semlock._get_value()
NotImplementedError
msg139684 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2011-07-03 11:17
New changeset ce52310f61a0 by Antoine Pitrou in branch 'default':
Followup to 51c1f2cedb96 (and issue #12456):
http://hg.python.org/cpython/rev/ce52310f61a0
History
Date User Action Args
2022-04-11 14:57:19adminsetgithub: 56665
2011-07-03 12:46:29pitrousetstatus: open -> closed
2011-07-03 11:17:52python-devsetmessages: + msg139684
2011-07-03 10:46:17pitrousetmessages: + msg139681
2011-07-03 10:07:26pitrousetstatus: closed -> open

messages: + msg139680
2011-07-02 19:21:35pitrousetstatus: open -> closed
resolution: fixed
stage: patch review -> resolved
2011-07-02 19:21:15python-devsetnosy: + python-dev
messages: + msg139664
2011-07-01 18:45:29rosslagerwallsetmessages: + msg139593
2011-07-01 16:47:31pitrousetfiles: + cfshutdown.patch
keywords: + patch
messages: + msg139587

stage: patch review
2011-07-01 05:49:08rosslagerwallsetmessages: + msg139545
2011-07-01 05:25:22rosslagerwallcreate