Author vstinner
Recipients davin, pablogsal, pitrou, vstinner
Date 2018-12-14.11:27:53
SpamBayes Score -1.0
Marked as misclassified Yes
Message-id <1544786874.12.0.788709270274.issue35493@psf.upfronthosting.co.za>
In-reply-to
Content
Currently, multiprocessing.Pool._worker_handler() checks every 100 ms if a worker exited using time.sleep(0.1). It causes a latency if worker exit frequently and the pool has to execute a large number of tasks.

Worst case:
---
import multiprocessing
import time
CONCURRENCY = 1
NTASK = 100
def noop():
    pass
with multiprocessing.Pool(CONCURRENCY, maxtasksperchild=1) as pool:
    start_time = time.monotonic()
    results = [pool.apply_async(noop, ()) for _ in range(NTASK)]
    for result in results:
        result.get()
    dt = time.monotonic() - start_time
    pool.terminate()
    pool.join()
print("Total: %.1f sec" % dt)
---

Output:
---
Total: 10.2 sec
---

The worst case is a pool of 1 process, each worker only executes a single task and the task does nothing (minimize task execution time): the latency is 100 ms per task, which means 10 seconds for 100 tasks.

Using SIGCHLD signal to be notified when a worker completes would allow to avoid polling: reduce the latency and reduce CPU usage (the thread doesn't have to be awaken every 100 ms anymore).
History
Date User Action Args
2018-12-14 11:27:54vstinnersetrecipients: + vstinner, pitrou, davin, pablogsal
2018-12-14 11:27:54vstinnersetmessageid: <1544786874.12.0.788709270274.issue35493@psf.upfronthosting.co.za>
2018-12-14 11:27:54vstinnerlinkissue35493 messages
2018-12-14 11:27:53vstinnercreate