New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
multiprocessing.Pool.join() always takes at least 100 ms #79660
Comments
The join() method of multiprocessing.Pool calls self._worker_handler.join(): it's a thread running _handle_workers(). The core of this thread function is: while thread._state == RUN or (pool._cache and thread._state != TERMINATE):
pool._maintain_pool()
time.sleep(0.1) I understand that the delay of 100 ms is used to check regularly the stop condition changed. This sleep causes a mandatory delay of 100 ms on Pool.join(). |
Attached PR 11136 modify _worker_handler() loop to wait on threading.Event events, so Pool.join() completes as soon as possible. Example: import multiprocessing
import time
def the_test():
start_time = time.monotonic()
pool = multiprocessing.Pool(1)
res = pool.apply_async(int, ("1",))
pool.close()
#pool.terminate()
pool.join()
dt = time.monotonic() - start_time
print("%.3f sec" % dt)
the_test() Minimum timing with _handle_results() using:
Currently, join() minimum timing depends on _handle_results() sleep() duration (100 ms). With my PR, it completes as soon as possible: when state change and/or when a result is set. My PR still requires an hardcoded delay of 100 ms to workaround bpo-35478 bug: results are never set if the pool is terminated. |
My PR 11136 doesn't work: _maintain_pool() should be called frequently to check when a worker completed. Polling worker exit status seems inefficient :-( asyncio uses SIGCHLD signal to be notified when a child process completes. SafeChildWatcher calls os.waitpid(pid, os.WNOHANG) on each child process, whereas FastChildWatcher() uses os.waitpid(-1, os.WNOHANG). |
_worker_handler has two issues:
|
Nobody managed to find a solution in 3 years. I close the issue. |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
bugs.python.org fields:
The text was updated successfully, but these errors were encountered: