classification
Title: Cannot pickle some objects that have a __getattr__()
Type: Stage:
Components: Documentation, Library (Lib) Versions: Python 3.4
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: larry Nosy List: Arfrever, alexandre.vassalotti, barry, benjamin.peterson, docs@python, eric.snow, larry, pitrou
Priority: release blocker Keywords: 3.4regression

Created on 2014-01-14 19:46 by barry, last changed 2014-02-23 07:06 by larry. This issue is now closed.

Messages (7)
msg208108 - (view) Author: Barry A. Warsaw (barry) * (Python committer) Date: 2014-01-14 19:46
I've been debugging a crash in nose 1.3.0, the root cause of which turned out to be an instance containing an attribute which itself was an instance of the following class (boiled down):

class Picky:
    def __getstate__(self):
        return {}
    def __getattr__(self, attr):
        return None

This crashes with a TypeError in Python 2.7 and Python 3 (albeit with slightly different tracebacks; and Python 3 is more difficult to debug because the TypeError doesn't include any useful information).

TypeError: 'NoneType' object is not callable

The culprit is __getattr__() returning None.  In Python 3 for example, pickle tries to get the object's __reduce_ex__() function and then call it.

The problem is the (IMHO) bogus __getattr__() and I'm not sure why nose has this.  But I wonder if the pickle documentation should warn against this kind of thing.

This isn't a bug in Python - the crash makes sense when you understand the implications, but perhaps a warning in the docs would have helped prevent this nose bug in the first place.  I suppose I could also see improving _pickle.c to provide some additional feedback on the offending attribute, but that's probably more difficult.

It's okay to close this as won't fix if we can't think of appropriate wording.  It's enough that there's a record of this issue for search engines now.
msg208109 - (view) Author: Barry A. Warsaw (barry) * (Python committer) Date: 2014-01-14 19:54
Hmm, actually, this is a regression in Python 3.4.  Let's amend the test class to include a __getnewargs__():

class Picky(object):
    """Options container that returns None for all options.
    """
    def __getstate__(self):
        return {}

    def __getnewargs__(self):
        return ()

    def __getattr__(self, attr):
        return None

This class is picklable in Python 2.7 - 3.3, but not in 3.4.
msg208142 - (view) Author: Eric Snow (eric.snow) * (Python committer) Date: 2014-01-15 05:24
This is a duplicate of #16251, no?  Pickle looks up dunder ;) methods on instances rather than on classes, so __getattr__() gets triggered unexpectedly.  I had to work around this in some code of mine by special-casing in __getattr__() names that start with '_'.  I've said my peace over in #16251.
msg208166 - (view) Author: Barry A. Warsaw (barry) * (Python committer) Date: 2014-01-15 15:06
I'll go ahead and dupe this to 16251, but will note the __getnewargs__() regression in 3.4.
msg211339 - (view) Author: Benjamin Peterson (benjamin.peterson) * (Python committer) Date: 2014-02-16 18:50
Fixed. Should be in 3.4.0.

I'll deal with #16251 in 3.5.
msg211340 - (view) Author: Benjamin Peterson (benjamin.peterson) * (Python committer) Date: 2014-02-16 18:51
(See http://hg.python.org/cpython/rev/b328f8ccbccf)
msg211980 - (view) Author: Larry Hastings (larry) * (Python committer) Date: 2014-02-23 07:06
My understanding is, this is fixed, and cherry-picked into 3.4.  If that's in error please reopen.
History
Date User Action Args
2014-02-23 07:06:32larrysetstatus: open -> closed
resolution: fixed
messages: + msg211980
2014-02-16 18:51:51benjamin.petersonsetmessages: + msg211340
2014-02-16 18:50:22benjamin.petersonsetpriority: normal -> release blocker

superseder: pickle special methods are looked up on the instance rather than the type ->
assignee: docs@python -> larry
versions: - Python 3.1, Python 2.7, Python 3.2, Python 3.3
nosy: + benjamin.peterson, larry

messages: + msg211339
2014-01-15 15:06:37barrysetsuperseder: pickle special methods are looked up on the instance rather than the type
messages: + msg208166
2014-01-15 05:24:04eric.snowsetnosy: + eric.snow
messages: + msg208142
2014-01-14 23:06:49serhiy.storchakasetnosy: + pitrou, alexandre.vassalotti
2014-01-14 22:32:29Arfreversetnosy: + Arfrever
2014-01-14 19:54:44barrysetkeywords: + 3.4regression

messages: + msg208109
components: + Library (Lib)
2014-01-14 19:46:07barrycreate