New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
pickle can pickle the wrong function #47907
Comments
test_pickletools fails sporadically on at least two platforms I've seen. http://www.python.org/dev/buildbot/all/x86%20gentoo%20trunk/builds/4120/step-test/0 File 1 items had failures: |
The valgrind errors below are possibly related. Conditional jump or move depends on uninitialised value(s) Conditional jump or move depends on uninitialised value(s) Conditional jump or move depends on uninitialised value(s) Conditional jump or move depends on uninitialised value(s) Conditional jump or move depends on uninitialised value(s) |
Indeed. The problem was an incorrect conversion of str -> unicode, Hmmm, this fix is for 3.0 only, but the problem is happening in 2.6. Committed revision 66021. |
It seems that if the tests are run in this order: ./python -E -tt ./Lib/test/regrtest.py -u all test_xmlrpc test_ctypes The error will trigger consistently. That is in 2.6 with a debug build |
Neal, can you verify that this is still a problem now that bsddb has |
I can reproduce this on 2.6. |
The explanation is actually simple, do not blame bsddb :-) random.random is a built-in method, so its __module__ attribute is not I reproduced the problem with a simple file: then, start python and:
>>> import someModule, random, pickle
>>> pickle.dumps(random.random, 0)
'csomeModule\nrandom\np0\n.' You may have to change the name of "someModule", to be sure that it To correct this, one direction would be to search only built-in or |
I'm not sure that pickling random.random is a good idea; did you try to |
No thought went into picking random.random in the test -- it was just a >>> import pickletools
>>> dis(pickle.dumps(pickletools.dis, 0))
0: c GLOBAL 'pickletools dis'
17: p PUT 0
20: . STOP
highest protocol among opcodes = 0 |
But it still means pickling a function/method defined in a builtin |
Amaury, yes, it would be nice if pickle were more reliable here. But In any case, since pickle/cPickle have worked this way forever, and the |
BTW, note that the Title of this issue is misleading: if ... getattr(module, funcname, None) is func: ) to determine whether the given function object is supplied by a I expect this is why nobody has ever complained about it: unless you're It would still be nice if pickle could identify "the most natural" |
Well, it may have a consequence if pickle picks the "random" function |
I just committed Tim's suggested change in r68906. This seemed a no- (I hadn't noticed the pickletools failures until just after I committed a |
'works for me' contradicts 'open', so I unset that |
Revision 68903 was merged in py3k in r68908. It looks like a similar issue shows up in test_random: ====================================================================== Traceback (most recent call last):
File "/home/buildbot/slave/py-build/3.1.norwitz-amd64/build/Lib/test/test_random.py", line 107, in test_pickling
state = pickle.dumps(self.gen)
File "/home/buildbot/slave/py-build/3.1.norwitz-amd64/build/Lib/pickle.py", line 1358, in dumps
Pickler(f, protocol, fix_imports=fix_imports).dump(obj)
_pickle.PicklingError: Can't pickle <class 'random.Random'>: it's not the same object as random.Random |
> Traceback (most recent call last):
> File "/home/buildbot/slave/py-build/3.1.norwitz-amd64/build/Lib/test/test_random.py", line 107, in test_pickling
> state = pickle.dumps(self.gen)
> File "/home/buildbot/slave/py-build/3.1.norwitz-amd64/build/Lib/pickle.py", line 1358, in dumps
> Pickler(f, protocol, fix_imports=fix_imports).dump(obj)
> _pickle.PicklingError: Can't pickle <class 'random.Random'>: it's not the same object as random.Random Actually, this might have to do with the fix I committed to But at this point, I must admit I don't even understand the failure |
It should be noted that, contrary to Amaury's suggestion, pickling random.seed fails under 3.x: >>> pickle.dumps(random.seed)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/home/antoine/py3k/__svn__/Lib/pickle.py", line 1314, in dumps
Pickler(f, protocol, fix_imports=fix_imports).dump(obj)
_pickle.PicklingError: Can't pickle <class 'method'>: attribute lookup builtins.method failed Furthermore, the original problem can also be reproduced under 3.x, using Amaury's trick: >>> pickle.dumps(random.random)
b'\x80\x03crandom\nrandom\nq\x00.'
>>> list(sys.modules.values())[0].random = random.random
>>> pickle.dumps(random.random)
b'\x80\x03cheapq\nrandom\nq\x00.' I think a possible heuristic in whichmodule() would be, if __module__ is not found or None, to look for a __module__ attribute on __self__: >>> random.random.__module__
>>> random.random.__self__.__module__
'random' |
Antoine's fix in r82919 / r82920 fixes the test_random failure for me. (Before the fix, ./python.exe ./Lib/test/regrtest.py test___all__ test_threaded_import test_random was enough to produce the failure.) |
The randomly failing tests seem to have been the high priority issue. The remaining, eponymous issue seems to be of rather lower priority, so I'm setting it to normal. Although Tim wanted a separate issue for the pickling problem, I think there's too much useful info about the underlying problem in this issue for it to make sense to open a new one. |
This was fixed in 3.4 with the introduction of method pickling. I don't think it would be appropriate to backport this to 2.7. Thus, I am closing this as a won't fix for 2.x. |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
bugs.python.org fields:
The text was updated successfully, but these errors were encountered: