classification
Title: concurrent.futures as_completed raise TimeoutError wrong
Type: behavior Stage:
Components: Versions: Python 3.6
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: jiangwanwei, josh.r
Priority: normal Keywords:

Created on 2017-03-06 11:23 by jiangwanwei, last changed 2017-03-06 15:48 by josh.r.

Messages (2)
msg289093 - (view) Author: jiangwanwei (jiangwanwei) Date: 2017-03-06 11:23
when I use as_completed function to wait my futures, if I sleep more than timeout seconds in each iteration , I found that futures has been set result, but raise TimeoutError. as my test example code shows:

from concurrent import futures
from multiprocessing import current_process
import time


def run(count):
    cp = current_process()
    print(cp.name, 'begin', count, 'at', time.time())
    time.sleep(count)
    print(cp.name, 'end', count, 'at', time.time())
    return count


if __name__ == '__main__':
    ppe = futures.ProcessPoolExecutor(max_workers=4)
    cp = current_process()

    fs = [ppe.submit(run, i) for i in range(4)]

    print('begin receive at', time.time())
    for f in futures.as_completed(fs, timeout=5):
        time.sleep(5)
        print(cp.name, 'receive', f.result(), 'at', time.time())
        print(cp.name, 'receive', [f.result() for f in fs], 'at', time.time())
    print('end receive at', time.time())


run above-mentioned example code, it will output :
begin receive at 1488799136.471536
Process-1 begin 0 at 1488799136.472969
Process-1 end 0 at 1488799136.473114
Process-3 begin 1 at 1488799136.473741
Process-2 begin 2 at 1488799136.474226
Process-4 begin 3 at 1488799136.474561
Process-3 end 1 at 1488799137.474495
Process-2 end 2 at 1488799138.475289
Process-4 end 3 at 1488799139.475696
MainProcess receive 0 at 1488799141.478663
MainProcess receive [0, 1, 2, 3] at 1488799141.478787
Traceback (most recent call last):
  File "test_futures.py", line 23, in <module>
    for f in futures.as_completed(fs, timeout=5):
  File "/Users/jiangwanwei/anaconda3/lib/python3.5/concurrent/futures/_base.py", line 213, in as_completed
    len(pending), len(fs)))
concurrent.futures._base.TimeoutError: 3 (of 4) futures unfinished
msg289116 - (view) Author: Josh Rosenberg (josh.r) * (Python triager) Date: 2017-03-06 15:48
The docs ( https://docs.python.org/3/library/concurrent.futures.html#concurrent.futures.as_completed ) do seem to indicate it shouldn't do so as long as results were available before the timeout expired:

"The returned iterator raises a concurrent.futures.TimeoutError if __next__() is called and the result isn’t available after timeout seconds from the original call to as_completed()."

My reading of that would be that it raises the error only when:

1. The timeout has expired
2. The call would block (or possibly, would have blocked after the timeout expired), indicating no result was available

Handling "would have blocked" is hard, but might it make sense to still allow a non-blocking wait on the event even if the timeout has expired, with the exception raised only if the non-blocking wait fails?

Side-note: Looks like this code is still using time.time, not time.monotonic, so it's vulnerable to system clock adjustments; NTP updates could cause a five second timeout to expire instantly, or take seconds or even minutes longer to expire.
History
Date User Action Args
2017-03-06 15:48:19josh.rsetnosy: + josh.r
messages: + msg289116
2017-03-06 11:23:33jiangwanweicreate