classification
Title: Difference between pickle implementations for function objects
Type: behavior Stage:
Components: Library (Lib) Versions: Python 2.7
process
Status: pending Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: BreamoreBoy, alexandre.vassalotti, pitrou, sbt, serhiy.storchaka
Priority: normal Keywords:

Created on 2012-03-16 19:03 by sbt, last changed 2017-02-19 18:43 by serhiy.storchaka.

Messages (3)
msg156069 - (view) Author: Richard Oudkerk (sbt) * (Python committer) Date: 2012-03-16 19:03
When pickling a function object, if it cannot be saved as a global the C implementation falls back to using copyreg/__reduce__/__reduce_ex__.

The comment for the changeset which added this fallback claims that it is for compatibility with the Python implementation.  See

   http://hg.python.org/cpython/rev/c6753db9c6af

However, the current python implementations do not have any such fallback.

This affects both 2.x and 3.x.

For example

Python 2.7.2 (default, Jun 12 2011, 15:08:59) [MSC v.1500 32 bit (Intel)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import pickle, cPickle, copy_reg
>>> def f():
...   pass
...
>>> _f = f
>>> del f
>>> copy_reg.pickle(type(_f), lambda obj: (str, ("FALLBACK",)))
>>> cPickle.dumps(_f)
"c__builtin__\nstr\np1\n(S'FALLBACK'\np2\ntp3\nRp4\n."
>>> pickle.dumps(_f)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "c:\Python27\lib\pickle.py", line 1374, in dumps
    Pickler(file, protocol).dump(obj)
  File "c:\Python27\lib\pickle.py", line 224, in dump
    self.save(obj)
  File "c:\Python27\lib\pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "c:\Python27\lib\pickle.py", line 748, in save_global
    (obj, module, name))
pickle.PicklingError: Can't pickle <function f at 0x0299A470>: it's not found as __main__.f

I don't know what should be done.  I would be tempted to always fall back to copyreg/__reduce__/__reduce_ex__ when save_global fails (not just for function objects) but that might make error messages less helpful.
msg222234 - (view) Author: Mark Lawrence (BreamoreBoy) * Date: 2014-07-03 22:38
I couldn't find the code referenced in the changeset in the new _pickle.c.  However I did find the comment that I've pasted below so I believe this issue only applies to 2.7.

/* The old cPickle had an optimization that used switch-case statement
   dispatching on the first letter of the type name.  This has was removed
   since benchmarks shown that this optimization was actually slowing
   things down. */
msg288152 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2017-02-19 18:43
In 3.x the fallback was removed by 6bd1f0a27e8e. If this is 2.7 only issue I prefer to close it as "won't fix".

See also issue26959.
History
Date User Action Args
2017-02-19 18:43:33serhiy.storchakasetstatus: open -> pending
nosy: + serhiy.storchaka
messages: + msg288152

2014-07-03 22:38:51BreamoreBoysetnosy: + BreamoreBoy

messages: + msg222234
versions: - Python 3.2, Python 3.3
2012-03-16 23:20:51eric.araujosetnosy: + pitrou, alexandre.vassalotti

versions: + Python 3.2
2012-03-16 19:03:38sbtcreate