This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: multiprocessing.Pool initialization seems to call methods it should not (and on badly formed objects)
Type: Stage:
Components: Library (Lib) Versions: Python 2.7
process
Status: closed Resolution: not a bug
Dependencies: Superseder:
Assigned To: Nosy List: Graham.Cummins
Priority: normal Keywords:

Created on 2011-05-31 17:30 by Graham.Cummins, last changed 2022-04-11 14:57 by admin. This issue is now closed.

Files
File name Uploaded Description Edit
bug.py Graham.Cummins, 2011-05-31 17:30 simplified test case and traceback.
Messages (2)
msg137373 - (view) Author: Graham Cummins (Graham.Cummins) Date: 2011-05-31 17:30
When attempting to pass instances of a custom class in the argument list
of a multiprocessing.Pool.map call, I encounter an inexplicable AttributeError. A simplified test script that creates the bug is included. It fails for both Python 2.7.1 and 3.2 on my 4-core 64 bit linux machine.

Briefly, the code creates a subclass of "dict" that checks for a "_ro" attribute before handling a "__setitem__". Empty instances of this class
can be passed to Pool.map fine, but if they have any content, I get
that multiprocessing/queues.py, line 378 ("recv()"), calls my custom __setitem__, which fails, due to a missing _ro attribute. 

This is very confusing. For one thing, much of the Traceback seems
missing. recv() is not directly a call to my customized __setitem__, 
and I can't thing why it should ever need to invoke such a call  

For another, the custom class _does_ set the _ro attribute, on __init__. Testing for this attribute, or calling __setitem__, succeeds in every other context I've tested, so it seems that the instance that is 
being used in the multiprocessing context is not the same underlying object that I pass in.
msg137384 - (view) Author: Graham Cummins (Graham.Cummins) Date: 2011-05-31 19:50
As far as I can tell, this is a version of the scope-dependent un-pickling issue mentioned in the documentation. I'm still rather confused, since in the original occurrence of the issue happens despite the fact that subprocesses should be able to import all relevant modules (unless they have a very odd environment), and the original class is pickleable. Still, until I can clearly distinguish my difficulty from the know issues presented in the programming guidelines, I see no reason to have it listed as a bug.
History
Date User Action Args
2022-04-11 14:57:18adminsetgithub: 56436
2011-05-31 19:50:28Graham.Cumminssetstatus: open -> closed
resolution: not a bug
messages: + msg137384
2011-05-31 17:30:31Graham.Cumminscreate