classification
Title: buffer objects are picklable but result is not unpicklable
Type: behavior Stage: resolved
Components: Library (Lib) Versions: Python 2.7, Python 2.6
process
Status: closed Resolution: wont fix
Dependencies: Superseder:
Assigned To: jnoller Nosy List: flox, jnoller, jwilk, rschoon.old, sbt, serhiy.storchaka
Priority: normal Keywords:

Created on 2010-04-05 22:45 by rschoon.old, last changed 2015-11-12 22:18 by serhiy.storchaka. This issue is now closed.

Messages (5)
msg102423 - (view) Author: Robin Schoonover (rschoon.old) Date: 2010-04-05 22:45
The multiprocessing module's version of the Queue class, which causes objects to be pickled for process to process transfer, ignores pickle restrictions when objects are added to the queue.  Example code (buffer isn't pickleable):

    from multiprocessing import Queue

    q = Queue()
    q.put(buffer("this is a buffer"))
    print(q.get())

It results in an exception, which is expected, but the exception is confusing, after the problem has already occurred, and if I were actually using multiple processes, not in the process that tried to send an unsendable object:

Traceback (most recent call last):                                                           
  File "mppkbuffer.py", line 5, in <module>                                                
    print(q.get())
  File "/usr/lib/python2.6/multiprocessing/queues.py", line 91, in get
    res = self._recv()
TypeError: buffer() takes at least 1 argument (0 given)

Expected result would be a thrown exception when we attempt to put the object into the queue using .put(), NOT when we attempt pull it out of the queue using .get(), when it gets unpickled.  Basically, while pickle fails when it tries to dump, Queue succeeds the dump (somehow) and fails to load.

I have tested with python 2.6.4 and 2.7a4.  3.1 doesn't appear to have this bug (but it does have a different one which I will report later).
msg102424 - (view) Author: Robin Schoonover (rschoon.old) Date: 2010-04-05 22:52
Since these sort of buffer objects don't exist in 3.x (so far as I know), I came up with a different way to test in 3.x (basically, trying to pickle bound or unbound methods).

It turns out that using this method to test it in 2.6 seems to fail where it should (in .put()), so if there's some sort of object that isn't pickleable in the same way as buffer that exists in 3.x, it should be tested to see if this specific problem is actually confined to 2.6 or not.
msg143149 - (view) Author: Richard Oudkerk (sbt) * (Python committer) Date: 2011-08-29 15:26
Buffer objects *are* picklable with protocol 2 (but not with earlier protocols).  Unfortunately, the result is not unpicklable.  

This is not a problem with multiprocessing.  (buffer seems to inherit __reduce__ and __reduce_ex__ from object.)

Python 2.7.1+ (r271:86832, Apr 11 2011, 18:13:53) 
[GCC 4.5.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import cPickle
>>> cPickle.dumps(buffer("hello"), cPickle.HIGHEST_PROTOCOL)
'\x80\x02c__builtin__\nbuffer\nq\x01)\x81q\x02.'
>>> cPickle.loads(_)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: buffer() takes at least 1 argument (0 given)
msg161860 - (view) Author: Richard Oudkerk (sbt) * (Python committer) Date: 2012-05-29 08:56
It seems the real issue here is that buffer objects are picklable (depending on protocol) but the resulting string is not unpicklable.

There are probably lots of other examples where this happens: for instance Exception subclasses which do not set self.args correctly (see Issue #1692335).

buffer is only used in Python 2.x.  Python 2.7 is in bugfix mode, so this issue will not change now.

I will close this issue as "won't fix".
msg254563 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2015-11-12 22:18
While buffer is only used in Python 2.x, there is a similar issue in 3.x with memoryview.

The patch for issue22995 fixes this issue.
History
Date User Action Args
2015-11-12 22:18:35serhiy.storchakasetnosy: + serhiy.storchaka
messages: + msg254563
2012-08-18 14:50:01floxsetnosy: + flox
2012-05-29 08:56:00sbtsetstatus: open -> closed
resolution: wont fix
messages: + msg161860

stage: resolved
2011-08-29 15:26:22sbtsetnosy: + sbt

messages: + msg143149
title: multiprocessing.Queue ignores pickle restrictions in .put() -> buffer objects are picklable but result is not unpicklable
2011-04-04 13:28:53jwilksetnosy: + jwilk
2010-04-05 22:53:58benjamin.petersonsetassignee: jnoller

nosy: + jnoller
2010-04-05 22:52:06rschoon.oldsetmessages: + msg102424
2010-04-05 22:45:17rschoon.oldcreate