classification
Title: _asdict breaks when inheriting from a namedtuple
Type: behavior Stage:
Components: Library (Lib) Versions: Python 3.6, Python 3.4, Python 3.5
process
Status: closed Resolution: not a bug
Dependencies: Superseder:
Assigned To: rhettinger Nosy List: Samuel Isaacson, _savage, doko, eric.snow, larry, nchammas, python-dev, rhettinger, veky
Priority: normal Keywords: 3.4regression

Created on 2015-08-24 23:49 by Samuel Isaacson, last changed 2015-12-27 04:29 by _savage. This issue is now closed.

Files
File name Uploaded Description Edit
nt_fix1.diff rhettinger, 2015-08-25 04:50 Rough draft patch review
Messages (14)
msg249082 - (view) Author: Samuel Isaacson (Samuel Isaacson) Date: 2015-08-24 23:49
When inheriting from namedtuples, _asdict and __dict__ return empty dictionaries:

    from collections import namedtuple

    class Point(namedtuple('_Point', ['x', 'y'])):
        pass

    a = Point(3, 4)
    print(a._asdict() == {})

gives False; it is True on Python 2.7.6
msg249093 - (view) Author: Samuel Isaacson (Samuel Isaacson) Date: 2015-08-25 00:56
Sorry; it returns True on Python 3.4, False on Python 2.7.6.
msg249100 - (view) Author: Raymond Hettinger (rhettinger) * (Python committer) Date: 2015-08-25 02:54
For __dict__, I'm not sure what the right behavior should by for subclasses that don't define __slots__.  In Python 3, the __dict__ is returning the dict for the subclass.  This might be the correct and most desirable behavior:

    >>> class Point(namedtuple('_Point', ['x', 'y'])):
            pass
    >>> a = Point(3, 4)
    >>> a.w = 5
    >>> a.__dict__
    {'w': 5}

If we leave the __dict__ behavior as is in Py3, then we still need to get _asdict() back to its documented behavior.  For that, we would need to disconnect it from __dict__ by restoring the Py2.7 code for _asdict():

    def _asdict(self):
        'Return a new OrderedDict which maps field names to their values'
        return OrderedDict(zip(self._fields, self))

All this needs to be thought-out carefully.  Putting in __dict__ support originally looked like a bugfix to get vars() working correctly, but it caused problems with pickling which then led to the addition of __getnewargs__.  It seems that defining __dict__ leads to problems no matter how you do it.

My inclination is to remove __dict__ and __getewargs__ from the namedtuple definition entirely and return to a simpler state of affairs that is easier to reason about and less likely to lead to unexpected behaviors like the one in this bug report.

One note:  using the Py2.7 namedtuple code in Python3 still doesn't restore the old behavior.  Something else in the language appears to have changed (causing the subclasses' __dict__ to take precedence over the inherited __dict__ property).
msg249358 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2015-08-30 16:17
New changeset fa3ac31cfa44 by Raymond Hettinger in branch '3.4':
Issue #24931:  Resolve __dict__ conflict in namedtuple subclasses.
https://hg.python.org/cpython/rev/fa3ac31cfa44
msg249403 - (view) Author: Eric Snow (eric.snow) * (Python committer) Date: 2015-08-31 15:09
Doesn't the fix mean that `vars(MyNamedtuple)` will no longer work?  While I personally don't mind (though I prefer that spelling) and appreciate the benefit of simpler code, isn't there a backward-compatibility issue here?

I do concede that fixing this bug without a compatibility break is complicated and probably not worth it.  However I want to make sure that the point is at least discussed briefly.
msg252796 - (view) Author: Matthias Klose (doko) * (Python committer) Date: 2015-10-11 10:11
reopening. the incompatible behavior is report in a Debian report as well.

"""
collections.namedtuple appears to not to create namedtuples which have the
__dict__ attribute any more. Given that the stdlib docs stated that using
vars() (and hence the existence of __dict__) was preferred, it seems odd that
it has disappeared.
""" 

https://bugs.debian.org/800370
msg252802 - (view) Author: Larry Hastings (larry) * (Python committer) Date: 2015-10-11 12:41
Why is this marked as a release blocker?  It doesn't strike me as all that major.
msg252804 - (view) Author: Matthias Klose (doko) * (Python committer) Date: 2015-10-11 12:45
now marked as regression, and lowered the priority.

but how should regressions on release branches be marked else?
msg252869 - (view) Author: Raymond Hettinger (rhettinger) * (Python committer) Date: 2015-10-12 16:18
> it seems odd that it has disappeared.

It disappeared because it was fundamentally broken in Python 3, so it had to be removed.  Providing __dict__ broke subclassing and produced odd behaviors.
msg256129 - (view) Author: Nicholas Chammas (nchammas) * Date: 2015-12-08 21:51
Should this change be called out in the 3.5.1 release docs? It makes some code that works on 3.5.0 break in 3.5.1.

See: http://stackoverflow.com/q/34166469/877069
msg256130 - (view) Author: Larry Hastings (larry) * (Python committer) Date: 2015-12-08 22:04
You're a little late; 3.5.1 was released two days ago.
msg256131 - (view) Author: Nicholas Chammas (nchammas) * Date: 2015-12-08 22:37
I know. I came across this issue after upgrading to the 3.5.1 release and seeing that vars(namedtuple) didn't work anymore.

I looked through the changelog [0] for an explanation of why that might be and couldn't find one, so I posted that question on Stack Overflow.

I'm guessing others will go through the same flow after they upgrade to 3.5.1 and wonder why their vars(namedtuple) code broke, so I posted here asking if we should amend the changelog to call this change out.

But I gather from your comment that the changelog cannot be updated after the release, so I guess there is nothing to do here. (Sorry about the distraction. I'm new to the Python dev community.)

[0] https://docs.python.org/3.5/whatsnew/changelog.html#python-3-5-1-final
msg256995 - (view) Author: Vedran Čačić (veky) * Date: 2015-12-25 15:56
I agree that namedtuples having __dict__ is probably more trouble than benefit. But in my view, that's no reason for _asdict to not work correctly. The whole point of separate function (even going through the pain of underscore-starting public API, since everything else is even bigger pain) is that we sidestep the question of vars() and other standard Python hooks, and provide our way of extracting a namedtuple's namespace, for ones who need it.

Of course, the fix/workaround is trivial, as Raymond says: just

    def _asdict(self):
        return collections.OrderedDict(zip(self._fields, self))

That way, it doesn't matter whether self has a `__dict__` or not.

If you think it shouldn't be done, the documentation really needs to be changed. Current wording is very misleading, showing that `_asdict` works on a class having `__slots__ = ()`, but containing no explanation of that attribute being _necessary_ for `_asdict` to work. (It just says "This helps keep memory requirements low by preventing the creation of instance dictionaries.")
msg257065 - (view) Author: Jens Troeger (_savage) * Date: 2015-12-27 04:29
With my update from Python 3.4.3 to Python 3.4.4 (default, Dec 25 2015, 06:14:41) I started experiencing crashes of my applications and I suspect this change is the culprit.

I have a class that inherits from namedtuple, and code calls vars() (i.e. retrieve __dict__) to iterate over an instance's attributes. Much like Raymond points out in http://bugs.python.org/msg249100

For example with 3.4.3:

>>> from collections import namedtuple
>>> Point = namedtuple('Point', ['x', 'y'])
>>> p = Point(1,2)
>>> p
Point(x=1, y=2)
>>> p.__dict__
OrderedDict([('x', 1), ('y', 2)])
>>> vars(p)
OrderedDict([('x', 1), ('y', 2)])

After the most recent update this breaks with 3.4.4:

>>> from collections import namedtuple
>>> Point = namedtuple('Point', ['x', 'y'])
>>> p = Point(1,2)
>>> p
Point(x=1, y=2)
>>> p.__dict__
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: 'Point' object has no attribute '__dict__'
>>> vars(p)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: vars() argument must have __dict__ attribute

I am not sure about the fix on my side. Should I use _asdict() instead of vars() although I would argue that vars() should remain functional across this change.  Calling _asdict() seems messy to me, but it works:

>>> p._asdict()
OrderedDict([('x', 1), ('y', 2)])

Why not keep the __dict__ property in tact?

  @property
  def __dict__(self):
      return self._asdict()

Thanks!
History
Date User Action Args
2015-12-27 04:29:13_savagesetnosy: + _savage
messages: + msg257065
2015-12-25 15:56:27vekysetnosy: + veky
messages: + msg256995
2015-12-08 22:37:58nchammassetmessages: + msg256131
2015-12-08 22:04:27larrysetmessages: + msg256130
2015-12-08 21:51:00nchammassetnosy: + nchammas
messages: + msg256129
2015-11-02 05:53:44rhettingersetstatus: open -> closed
resolution: not a bug
2015-10-12 16:18:22rhettingersetkeywords: - 3.5regression
priority: high -> normal
messages: + msg252869
2015-10-11 12:45:29dokosetkeywords: + 3.4regression, 3.5regression, - patch
priority: release blocker -> high
messages: + msg252804
2015-10-11 12:41:14larrysetmessages: + msg252802
2015-10-11 10:11:50dokosetstatus: closed -> open
priority: high -> release blocker

nosy: + larry, doko
messages: + msg252796

resolution: fixed -> (no value)
2015-08-31 15:09:55eric.snowsetnosy: + eric.snow
messages: + msg249403
2015-08-30 17:44:11rhettingersetstatus: open -> closed
resolution: fixed
2015-08-30 16:17:11python-devsetnosy: + python-dev
messages: + msg249358
2015-08-25 04:50:26rhettingersetfiles: + nt_fix1.diff
keywords: + patch
2015-08-25 02:54:32rhettingersetpriority: normal -> high

messages: + msg249100
versions: + Python 3.5, Python 3.6
2015-08-25 02:19:53rhettingersetassignee: rhettinger
2015-08-25 00:56:30Samuel Isaacsonsetmessages: + msg249093
2015-08-25 00:12:52yselivanovsetnosy: + rhettinger
2015-08-24 23:49:48Samuel Isaacsoncreate