classification
Title: multiprocessing Pipe send of non-picklable objects doesn't raise error
Type: behavior Stage: resolved
Components: Windows Versions: Python 2.7
process
Status: closed Resolution: works for me
Dependencies: Superseder:
Assigned To: Nosy List: Ian.Bell, orsenthil, sbt
Priority: normal Keywords:

Created on 2012-06-27 03:25 by Ian.Bell, last changed 2013-06-20 15:45 by sbt. This issue is now closed.

Messages (7)
msg164118 - (view) Author: Ian Bell (Ian.Bell) Date: 2012-06-27 03:25
When a non-picklable object is sent through a multiprocessing.Pipe, no exception is raised, instead when trying to read the other end of the pipe, a TypeError is raised:

TypeError: Required argument 'handle' (pos 1) not found
msg164417 - (view) Author: Richard Oudkerk (sbt) * (Python committer) Date: 2012-06-30 19:25
What type of object did you try to send, and how can the problem be reproduced?

There are plenty of types which don't support pickling, and where pickling only succeeds in producing invalid data which cannot be successfully unpickled.  The exception raised when unpickling will depend on the type of the object.
msg164420 - (view) Author: Ian Bell (Ian.Bell) Date: 2012-06-30 19:39
I had issues with a class that I wrote myself.  It is a rather involved data structure with all kinds of interesting things going on.  Unfortunately I cannot put together a minimal working example that will cause a Python hang.
msg164457 - (view) Author: Richard Oudkerk (sbt) * (Python committer) Date: 2012-07-01 08:16
Then I doubt this is a bug in Python.

If your class does not override __reduce__, __reduce_ex__ or __getstate__/__setstate__, then it is probably one of the attributes of your instance which is causing the problem.  You could try to find out which one by trying

    cPickle.loads(cPickle.dumps(attribute, -1))

for each attribute of an instance of your class to see if it raises an error.
msg164459 - (view) Author: Senthil Kumaran (orsenthil) * (Python committer) Date: 2012-07-01 08:26
On Sun, Jul 1, 2012 at 1:16 AM, Richard Oudkerk <report@bugs.python.org> wrote:
> Then I doubt this is a bug in Python.

I guess, you meant, this is NOT a bug in Python.
msg164488 - (view) Author: Ian Bell (Ian.Bell) Date: 2012-07-01 17:22
I have repaired my class so that it pickles properly, but that does not resolve the issue that if you send a non-picklable object through a pipe, it should raise an error, rather than hang.
msg164491 - (view) Author: Richard Oudkerk (sbt) * (Python committer) Date: 2012-07-01 18:10
> I have repaired my class so that it pickles properly, but that does not 
> resolve the issue that if you send a non-picklable object through a pipe, 
> it should raise an error, rather than hang.

What do you mean by hang?  Do you mean that send() succeeds and recv() raises an error.  That might cause your *program* to hang, but send() itself should not hang.
History
Date User Action Args
2013-06-20 15:45:22sbtsetstatus: open -> closed
resolution: works for me
stage: resolved
2012-07-01 18:10:56sbtsetmessages: + msg164491
2012-07-01 17:22:03Ian.Bellsetmessages: + msg164488
2012-07-01 08:26:52orsenthilsetnosy: + orsenthil
messages: + msg164459
2012-07-01 08:16:29sbtsetmessages: + msg164457
2012-06-30 19:39:59Ian.Bellsetmessages: + msg164420
2012-06-30 19:25:53sbtsettype: crash -> behavior
messages: + msg164417
2012-06-27 11:17:13pitrousetnosy: + sbt
2012-06-27 03:26:17Ian.Bellsettype: crash
2012-06-27 03:25:48Ian.Bellcreate