This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

Author tim.peters
Recipients DemGiran, tim.peters
Date 2018-07-20.18:18:21
SpamBayes Score -1.0
Marked as misclassified Yes
Message-id <>
If your `bucket` has 30 million items, then

    for element in bucket:
        executor.submit(kwargs['function']['name'], element, **kwargs)

is going to create 30 million Future objects (and all the under-the-covers objects needed to manage their concurrency) just as fast as the main thread can create them.  Nothing in your code waits for anything to finish until after they've _all_ been created and queued up under the covers.

So your producer is running vastly faster than your consumers can keep up with.  It's the huge backlog of pending work items that consume the RAM.  To slash RAM, you need to craft a way to interleave creating new work items with giving consumers time to deal with them.
Date User Action Args
2018-07-20 18:18:21tim.peterssetrecipients: + tim.peters, DemGiran
2018-07-20 18:18:21tim.peterssetmessageid: <>
2018-07-20 18:18:21tim.peterslinkissue34168 messages
2018-07-20 18:18:21tim.peterscreate