This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

Author macfreek
Recipients macfreek
Date 2010-08-13.19:02:45
SpamBayes Score 8.7056117e-07
Marked as misclassified No
Message-id <1281726168.11.0.648826556363.issue9592@psf.upfronthosting.co.za>
In-reply-to
Content
I came across three limitation in the multiprocessing module that were not handled correctly.

Attached is a file that reproduces the errors in minimal code. I tested them with Python 2.6.5 and 3.1.2.

Expected result:
multiprocessing.Pool's promises a map function where each result is returned transparently to the main process (despite that the calculation was done in a subprocess)

Actual result:
Not all values returned by a subprocess are returned transparently.
I expected multiprocessing to handle these cases gracefully by yielding an Exception in the Main process.

The cases I found are:

1) A multiprocessing worker can not return (return, not raise!) an Exception. 
If this is attempted, the result handler thread in the Pool calls the exception with no arguments, 
which might raise an error if multiple arguments are required:
TypeError: ('__init__() takes exactly 2 arguments (1 given)', <class '__main__.MyException'>, ())    


2) A multiprocessing worker can not return an hashlib Object.
If this is attempted, pickle returns an error:
PicklingError: Can't pickle <type '_hashlib.HASH'>: attribute lookup _hashlib.HASH failed


3) A multiprocessing worker can not return an Object which overrides __getattr__, and accesses a variable from self in __getattr__.
If this is attempted, Python 2.6 crashes with a bus error:
Program terminated by uncaught signal #10 after 1.56 seconds.
Python 3.1 yields the error:
RuntimeError: maximum recursion depth exceeded while calling a Python object
History
Date User Action Args
2010-08-13 19:02:48macfreeksetrecipients: + macfreek
2010-08-13 19:02:48macfreeksetmessageid: <1281726168.11.0.648826556363.issue9592@psf.upfronthosting.co.za>
2010-08-13 19:02:47macfreeklinkissue9592 messages
2010-08-13 19:02:46macfreekcreate