classification
Title: Expose Queue maxsize parameter to multiprocessing.Pool class
Type: enhancement Stage:
Components: Library (Lib) Versions: Python 3.8
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: davin, noxdafox, pitrou, sbt
Priority: normal Keywords: patch

Created on 2013-10-05 17:30 by noxdafox, last changed 2018-04-13 14:15 by pitrou.

Files
File name Uploaded Description Edit
maxqueuesize.patch noxdafox, 2013-10-05 17:30 review
maxqueuesize.patch noxdafox, 2013-10-05 17:34 review
Messages (3)
msg198992 - (view) Author: Matteo Cafasso (noxdafox) Date: 2013-10-05 17:30
As a developer I want the multiprocessing Pool class to expose the internal queue size limit in order to better control the task flow in my application.

Consider the following scenarios:
1. The tasks I want to run into the pool require a considerably big amount of data as input (a big XML string for example). A limitless queue and a high number of tasks would cause my application to consume a huge amount of memory, this is unacceptable.
2. I want to always ensure a small amount of tasks queued into the pool so that, if my application crashes, I won't loose much data.

Patch includes code changes, tests and documentation.
msg198993 - (view) Author: Matteo Cafasso (noxdafox) Date: 2013-10-05 17:34
Please ignore the first provided patch, doc changes where wrong
msg315246 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2018-04-13 14:15
A similar issue for concurrent.futures is bpo-29595.
History
Date User Action Args
2018-04-13 14:15:39pitrousetmessages: + msg315246
versions: + Python 3.8, - Python 3.4
2018-04-12 19:35:34ned.deilysetnosy: + pitrou, davin
2013-10-05 20:54:31pitrousetnosy: + sbt
2013-10-05 17:34:55noxdafoxsetfiles: + maxqueuesize.patch

messages: + msg198993
2013-10-05 17:30:41noxdafoxcreate