classification
Title: Update module attribute on namedtuple methods for introspection.
Type: enhancement Stage: resolved
Components: Library (Lib) Versions: Python 3.8, Python 3.7
process
Status: closed Resolution: works for me
Dependencies: Superseder:
Assigned To: rhettinger Nosy List: a-feld, pmpp, rhettinger, serhiy.storchaka
Priority: normal Keywords: patch

Created on 2018-04-28 22:40 by a-feld, last changed 2018-05-02 01:43 by pmpp. This issue is now closed.

Files
File name Uploaded Description Edit
0001-bpo-33380-Update-module-attribute-on-namedtuple-meth.patch a-feld, 2018-04-28 23:05
Messages (7)
msg315869 - (view) Author: Allan Feldman (a-feld) * Date: 2018-04-28 22:40
Python 3.7 made several performance improvements to the namedtuple class as part of https://bugs.python.org/issue28638

Prior to the implementation of bpo-28638, the __module__ attribute for a namedtuple's methods (e.g. _asdict) would return the value 'namedtuple_%s' % typename (e.g. namedtuple_Point).

Due to the optimizations made, the __module__ attribute for a namedtuple's methods now returns 'collections'.

The proposed change as part of this issue is to report the more accurate derived module name for the namedtuple methods. Updating the __module__ attribute should help debug and introspection tools more accurately report the details of executing calls (in profilers for example).

Example from Python 3.6:
>>> from collections import namedtuple
>>> Point = namedtuple('Point', ('x', 'y'))
>>> p1 = Point(1,2)
>>> p1._asdict.__module__
'namedtuple_Point'

Example from Python 3.7.0b3:
>>> from collections import namedtuple
>>> Point = namedtuple('Point', ('x', 'y'))
>>> p1 = Point(1,2)
>>> p1._asdict.__module__
'collections'

Desired behavior:
>>> from collections import namedtuple
>>> Point = namedtuple('Point', ('x', 'y'))
>>> p1 = Point(1,2)
>>> p1._asdict.__module__
'__main__'
msg315870 - (view) Author: Allan Feldman (a-feld) * Date: 2018-04-28 23:05
Attached is a proposed change.
msg315926 - (view) Author: Raymond Hettinger (rhettinger) * (Python committer) Date: 2018-04-30 06:20
Am not sure I can see any value in pushing the __module__ attribute down into the methods and think it is reasonable for them to report their origin as being in the collections module.
msg315927 - (view) Author: pmp-p (pmpp) * Date: 2018-04-30 06:33
I see that as a good fix, obviously Point definition belongs to __main__ in the sample, like would any other subclass defined there eg if "some" class is defined in awe.py module :

>>> import awe
>>> class my(awe.some):pass
... 
>>> my.__module__
'__main__'
msg315977 - (view) Author: Raymond Hettinger (rhettinger) * (Python committer) Date: 2018-05-01 01:53
ISTM collections is the correct module reference because that is where the code is actually defined.  A debugging tool should look there instead of in the calling code.  This is the way other tools work in Python as well:

    >>> from collections.abc import Set
    >>> class S(Set):
            def __init__(self): pass
            def __iter__(self): yield 0
            def __len__(self): return 0
            def __contains__(self, key): return False
            
    >>> S.__or__.__module__
    'collections.abc'

    >>> from dataclasses import make_dataclass
    >>> P = make_dataclass('P', ['x', 'y'])
    >>> P.__repr__.__module__
    'dataclasses'

Likewise, the various tools that use closures used to report the module where the closure was defined rather than the module of the caller's code (see functools.cmp_to_key or functools.lru_cache).  I say "used to" because the pure Python code is now supplanted by C equivalents where the __module__ attribute__ is appropriately set to None:

    # Running from PyPy
    >>>> from functools import cmp_to_key
    >>>> c = cmp_to_key(lambda x, y: 0)
    >>>> c.__init__.__module__
    '_functools'

The Data Model section of the Language Reference defines __module__ as follows, "__module__: The name of the module the function was defined in, or None if unavailable."  In the prior version of namedtuple(), the function was defined in a virtual module (an execed string).  Now, that the methods are defined in a real module, the __module__ attribute should name that real module.
msg315978 - (view) Author: Allan Feldman (a-feld) * Date: 2018-05-01 02:23
That explanation makes sense to me. Thanks for taking the time to look into this!
msg316033 - (view) Author: pmp-p (pmpp) * Date: 2018-05-02 01:43
Indeed thanks for the deep explanation. It seems that not finding im_self anymore (not even in the dunder clutter), i've mistaken '.__self__.__module__' with '.__module__'. How joyfull to learn anew to trace a caller id, and sorry for the noise (again).
History
Date User Action Args
2018-05-02 01:43:41pmppsetmessages: + msg316033
2018-05-01 02:23:55a-feldsetstatus: open -> closed
resolution: works for me
messages: + msg315978

stage: resolved
2018-05-01 01:53:28rhettingersetmessages: + msg315977
2018-04-30 06:33:16pmppsetnosy: + pmpp
messages: + msg315927
2018-04-30 06:20:20rhettingersetmessages: + msg315926
2018-04-29 03:41:09serhiy.storchakasetassignee: rhettinger

nosy: + serhiy.storchaka
2018-04-28 23:05:54a-feldsetfiles: + 0001-bpo-33380-Update-module-attribute-on-namedtuple-meth.patch
keywords: + patch
messages: + msg315870
2018-04-28 22:40:12a-feldcreate