Message77806
Despite carefully matching my get() and task_done() statements I would
often trigger "raise ValueError('task_done() called too many times')" in
my multiprocessing.JoinableQueue (multiprocessing/queues.py)
Looking over the code (and a lot of debug logging), it appears that the
issue arises from JoinableQueue.put() not being protected with a locking
mechanism. A preemption after the first line allows other processes to
resume without releasing the _unfinished_tasks semaphore.
The simplest solution seems to be allowing task_done() to block while
waiting to acquire the _unfinished_tasks semaphore.
Replacing:
if not self._unfinished_tasks.acquire(False):
raise ValueError('task_done() called too many times')
With simply:
self._unfinished_tasks.acquire()
This would however remove the error checking provided (given the many
far more subtler error that can be made, I might argue it is of limited
value). Alternately the JoinableQueue.put() method could be better
protected. |
|
Date |
User |
Action |
Args |
2008-12-14 16:48:02 | merrellb | set | recipients:
+ merrellb |
2008-12-14 16:48:02 | merrellb | set | messageid: <1229273282.85.0.657676661464.issue4660@psf.upfronthosting.co.za> |
2008-12-14 16:48:02 | merrellb | link | issue4660 messages |
2008-12-14 16:48:01 | merrellb | create | |
|