This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

Author mhrivnak
Recipients Nam.Nguyen, Patrik Dufresne, Victor.Varvariuc, Vinay Anantharaman, Winterflower, bquinlan, mhrivnak, r.david.murray, thehesiod
Date 2017-11-08.15:47:47
SpamBayes Score -1.0
Marked as misclassified Yes
Message-id <1510156067.37.0.213398074469.issue14119@psf.upfronthosting.co.za>
In-reply-to
Content
My project also has a use case for this, very similar to the others described. Here's what we want:

with ThreadPoolExecutor(queue_size=500) as executor:
  for item in parse_a_long_list_of_work(somefile.xml):
    executor.submit(Job(item))

I do not want to parse the entire list of work items and load them into memory at once. It is preferable for the main thread running the above code to block on submit() when the queue size is above some threshold.

It's a classic case of the producer and consumer operating at different speeds. In the past, a Queue object has been the way to connect such a producer and consumer. The various Executor classes do not provide an easy way to consume from a provided Queue object, so giving them that capability would be a reasonable alternative to having the submit() method block.
History
Date User Action Args
2017-11-08 15:47:47mhrivnaksetrecipients: + mhrivnak, bquinlan, r.david.murray, Nam.Nguyen, Victor.Varvariuc, thehesiod, Winterflower, Patrik Dufresne, Vinay Anantharaman
2017-11-08 15:47:47mhrivnaksetmessageid: <1510156067.37.0.213398074469.issue14119@psf.upfronthosting.co.za>
2017-11-08 15:47:47mhrivnaklinkissue14119 messages
2017-11-08 15:47:47mhrivnakcreate