This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

Author josh.r
Recipients josh.r, r.david.murray, serhiy.storchaka, spencera, terry.reedy
Date 2017-02-03.04:14:43
SpamBayes Score -1.0
Marked as misclassified Yes
Message-id <1486095284.0.0.734371929785.issue29423@psf.upfronthosting.co.za>
In-reply-to
Content
Looks like the problem here is that pickling self._consume implicitly pickles self, and after the first submission, self._futures contains Future objects. Those are probably what have the RLock in them (to synchronize between reading and populating threads), and it's correct not to pickle them; they're only meaningful in the parent process, and sending them to the workers would break, badly.

Given you don't seem to be using self for anything, you could either make _consume a staticmethod (if you need state from self, pass that specific state in as arguments) so it's not implicitly pickling and transmitting self, or failing that, define custom pickling functions that explicitly exclude self._futures from the set of attributes to pickle.

There is nothing to fix behaviorally on Python's side that wouldn't lead to confusion/breakage in other code. It would be nice if Python could give more detail on pickle chain that led to the unpicklable object (so you know it's self._consume->self->self._futures->some future object->future._rlock or whatever), but I'm not sure the pickling protocol design can preserve that sort of chain for error reporting...
History
Date User Action Args
2017-02-03 04:14:44josh.rsetrecipients: + josh.r, terry.reedy, r.david.murray, serhiy.storchaka, spencera
2017-02-03 04:14:44josh.rsetmessageid: <1486095284.0.0.734371929785.issue29423@psf.upfronthosting.co.za>
2017-02-03 04:14:43josh.rlinkissue29423 messages
2017-02-03 04:14:43josh.rcreate