Message323587
I figured out that the problem is itertools.tee does not use a multiprocessing.Manager proxied object for shared state. I was able to create a workaround tee as follows.
def multiprocessing_tee(iterable, n=2):
"""Write a multiprocessing safe itertools.tee"""
it = iter(iterable)
m = multiprocessing.Manager()
lists = [m.list() for i in range(n)]
def gen(local_list):
keep_m_alive = m
while True:
if not local_list: # when the local list is empty
newval = next(it) # fetch a new value and
for l in lists: # load it to all the lists
l.append(newval)
yield local_list.pop(-1)
return tuple(gen(l) for l in lists) |
|
Date |
User |
Action |
Args |
2018-08-16 01:36:44 | carlorosati | set | recipients:
+ carlorosati |
2018-08-16 01:36:44 | carlorosati | set | messageid: <1534383404.22.0.56676864532.issue34410@psf.upfronthosting.co.za> |
2018-08-16 01:36:44 | carlorosati | link | issue34410 messages |
2018-08-16 01:36:43 | carlorosati | create | |
|