Author carlorosati
Recipients carlorosati
Date 2018-08-16.01:36:43
SpamBayes Score -1.0
Marked as misclassified Yes
Message-id <1534383404.22.0.56676864532.issue34410@psf.upfronthosting.co.za>
In-reply-to
Content
I figured out that the problem is itertools.tee does not use a multiprocessing.Manager proxied object for shared state. I was able to create a workaround tee as follows.

def multiprocessing_tee(iterable, n=2):
    """Write a multiprocessing safe itertools.tee"""
    it = iter(iterable)
    m = multiprocessing.Manager()
    lists = [m.list() for i in range(n)]
    def gen(local_list):
        keep_m_alive = m
        while True:
            if not local_list:         # when the local list is empty
                newval = next(it)      # fetch a new value and
                for l in lists:        # load it to all the lists
                    l.append(newval)
            yield local_list.pop(-1)
    return tuple(gen(l) for l in lists)
History
Date User Action Args
2018-08-16 01:36:44carlorosatisetrecipients: + carlorosati
2018-08-16 01:36:44carlorosatisetmessageid: <1534383404.22.0.56676864532.issue34410@psf.upfronthosting.co.za>
2018-08-16 01:36:44carlorosatilinkissue34410 messages
2018-08-16 01:36:43carlorosaticreate