Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cannot pickle some objects that have a __getattr__() #64460

Closed
warsaw opened this issue Jan 14, 2014 · 7 comments
Closed

Cannot pickle some objects that have a __getattr__() #64460

warsaw opened this issue Jan 14, 2014 · 7 comments
Assignees
Labels
docs Documentation in the Doc dir release-blocker stdlib Python modules in the Lib dir

Comments

@warsaw
Copy link
Member

warsaw commented Jan 14, 2014

BPO 20261
Nosy @warsaw, @pitrou, @larryhastings, @avassalotti, @benjaminp, @ericsnowcurrently

Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

Show more details

GitHub fields:

assignee = 'https://github.com/larryhastings'
closed_at = <Date 2014-02-23.07:06:32.939>
created_at = <Date 2014-01-14.19:46:07.118>
labels = ['release-blocker', 'library', 'docs']
title = 'Cannot pickle some objects that have a __getattr__()'
updated_at = <Date 2014-02-23.07:06:32.938>
user = 'https://github.com/warsaw'

bugs.python.org fields:

activity = <Date 2014-02-23.07:06:32.938>
actor = 'larry'
assignee = 'larry'
closed = True
closed_date = <Date 2014-02-23.07:06:32.939>
closer = 'larry'
components = ['Documentation', 'Library (Lib)']
creation = <Date 2014-01-14.19:46:07.118>
creator = 'barry'
dependencies = []
files = []
hgrepos = []
issue_num = 20261
keywords = ['3.4regression']
message_count = 7.0
messages = ['208108', '208109', '208142', '208166', '211339', '211340', '211980']
nosy_count = 8.0
nosy_names = ['barry', 'pitrou', 'larry', 'alexandre.vassalotti', 'benjamin.peterson', 'Arfrever', 'docs@python', 'eric.snow']
pr_nums = []
priority = 'release blocker'
resolution = 'fixed'
stage = None
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue20261'
versions = ['Python 3.4']

@warsaw
Copy link
Member Author

warsaw commented Jan 14, 2014

I've been debugging a crash in nose 1.3.0, the root cause of which turned out to be an instance containing an attribute which itself was an instance of the following class (boiled down):

class Picky:
    def __getstate__(self):
        return {}
    def __getattr__(self, attr):
        return None

This crashes with a TypeError in Python 2.7 and Python 3 (albeit with slightly different tracebacks; and Python 3 is more difficult to debug because the TypeError doesn't include any useful information).

TypeError: 'NoneType' object is not callable

The culprit is __getattr__() returning None. In Python 3 for example, pickle tries to get the object's __reduce_ex__() function and then call it.

The problem is the (IMHO) bogus __getattr__() and I'm not sure why nose has this. But I wonder if the pickle documentation should warn against this kind of thing.

This isn't a bug in Python - the crash makes sense when you understand the implications, but perhaps a warning in the docs would have helped prevent this nose bug in the first place. I suppose I could also see improving _pickle.c to provide some additional feedback on the offending attribute, but that's probably more difficult.

It's okay to close this as won't fix if we can't think of appropriate wording. It's enough that there's a record of this issue for search engines now.

@warsaw warsaw added the docs Documentation in the Doc dir label Jan 14, 2014
@warsaw
Copy link
Member Author

warsaw commented Jan 14, 2014

Hmm, actually, this is a regression in Python 3.4. Let's amend the test class to include a __getnewargs__():

class Picky(object):
    """Options container that returns None for all options.
    """
    def __getstate__(self):
        return {}

    def __getnewargs__(self):
        return ()

    def __getattr__(self, attr):
        return None

This class is picklable in Python 2.7 - 3.3, but not in 3.4.

@warsaw warsaw added the stdlib Python modules in the Lib dir label Jan 14, 2014
@ericsnowcurrently
Copy link
Member

This is a duplicate of bpo-16251, no? Pickle looks up dunder ;) methods on instances rather than on classes, so __getattr__() gets triggered unexpectedly. I had to work around this in some code of mine by special-casing in __getattr__() names that start with '_'. I've said my peace over in bpo-16251.

@warsaw
Copy link
Member Author

warsaw commented Jan 15, 2014

I'll go ahead and dupe this to 16251, but will note the __getnewargs__() regression in 3.4.

@benjaminp
Copy link
Contributor

Fixed. Should be in 3.4.0.

I'll deal with bpo-16251 in 3.5.

@benjaminp
Copy link
Contributor

@larryhastings
Copy link
Contributor

My understanding is, this is fixed, and cherry-picked into 3.4. If that's in error please reopen.

@ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
docs Documentation in the Doc dir release-blocker stdlib Python modules in the Lib dir
Projects
None yet
Development

No branches or pull requests

4 participants