➜

This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

Author Windson Yang
Recipients Windson Yang, zach.ware
Date 2018-07-17.04:03:50
SpamBayes Score -1.0
Marked as misclassified Yes
Message-id <1531800230.72.0.56676864532.issue34134@psf.upfronthosting.co.za>
In-reply-to
Content 
I'm using macOX and I got huge memory usage when using generator with multiprocess. (see file) 

I think this is because (https://github.com/python/cpython/blob/master/Lib/multiprocessing/pool.py#L383)

    if not hasattr(iterable, '__len__'):
        iterable = list(iterable)

    if chunksize is None:
        chunksize, extra = divmod(len(iterable), len(self._pool) * 4)
        if extra:
            chunksize += 1

When we convert an iterable to list(iterable), we lost the advantage of using the generator. I'm not sure how to fix it, maybe we can set a default value for an object don't have '__len__' attr, any ideas?
History
Date User Action Args
2018-07-17 04:03:50Windson Yangsetrecipients: + Windson Yang, zach.ware
2018-07-17 04:03:50Windson Yangsetmessageid: <1531800230.72.0.56676864532.issue34134@psf.upfronthosting.co.za>
2018-07-17 04:03:50Windson Yanglinkissue34134 messages
2018-07-17 04:03:50Windson Yangcreate