classification
Title: Dynamically generate the _source attribute of namedtuple to save memory)
Type: resource usage Stage:
Components: Versions: Python 3.5
process
Status: closed Resolution: rejected
Dependencies: Superseder:
Assigned To: rhettinger Nosy List: christian.heimes, eric.araujo, eric.smith, eric.snow, giampaolo.rodola, rhettinger, vstinner
Priority: low Keywords: patch

Created on 2013-11-18 09:42 by vstinner, last changed 2017-07-17 13:31 by giampaolo.rodola. This issue is now closed.

Files
File name Uploaded Description Edit
namedtuple_source.patch vstinner, 2014-03-18 08:57 review
Messages (13)
msg203270 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2013-11-18 09:42
The definition of a new nametuple creates a large Python script to create the new type. The code stores the code in a private attribute:

    namespace = dict(__name__='namedtuple_%s' % typename)
    exec(class_definition, namespace)
    result = namespace[typename]
    result._source = class_definition

This attribute wastes memory, I don't understand the purpose of the attribute. It was not discussed in an issue, so I guess that there is no real use case:

changeset:   68879:bffdd7e9265c
user:        Raymond Hettinger <python@rcn.com>
date:        Wed Mar 23 12:52:23 2011 -0700
files:       Doc/library/collections.rst Lib/collections/__init__.py Lib/test/test_collections.py
description:
Expose the namedtuple source with a _source attribute.

Can we just drop this attribute to reduce the Python memory footprint?
msg203271 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2013-11-18 09:43
I found this issue while using my tracemalloc module to analyze the memory consumption of Python. On the Python test suite, the _source attribute is the 5th line allocating the memory memory:

/usr/lib/python3.4/collections/__init__.py: 676.2 kB
msg203304 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2013-11-18 15:51
>  the 5th line allocating the memory memory

oops, the 5th line allocating the *most* memory
msg203707 - (view) Author: Eric Snow (eric.snow) * (Python committer) Date: 2013-11-22 00:42
As an alternative, how about turning _source into a property?
msg203870 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2013-11-22 20:55
In a first version namedtuple had an argument (named echo or verbose) that would cause the source code to be printed out, for use at the interactive prompt.  Raymond later changed it to a _source attribute, more easy to work with than printed output.

About the other question you asked on the ML (why isn’t there a base NamedTuple class to inherit): this has been discussed on python-ideas IIRC, and people have written ActiveState recipes for that idea.  It should be easy to find the ML archive links from the ActiveState posts.
msg203878 - (view) Author: Eric Snow (eric.snow) * (Python committer) Date: 2013-11-22 21:06
A while back, because of those python-ideas discussions, Raymond added a link at the bottom of the namedtuple section of the docs at http://docs.python.org/3.4/library/collections.html#namedtuple-factory-function-for-tuples-with-named-fields.  The link points to a nice recipe by Jan Kaliszewski.
msg213904 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2014-03-17 22:01
> As an alternative, how about turning _source into a property?

A class or an instance property? A class property requires a metaclass. I guess that each namedtuple type requires its own metaclass, right?
msg213946 - (view) Author: Eric Snow (eric.snow) * (Python committer) Date: 2014-03-18 07:33
It does not necessarily require a metaclass.  You can accomplish it using a custom descriptor:

class classattr:
    def __init__(self, getter):
        self.getter = getter
    def __get__(self, obj, cls):
        return self.getter(cls)

FWIW, this is a descriptor that may be worth adding somewhere regardless.
msg213949 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2014-03-18 08:57
namedtuple_source.patch: Replace _source attribute wasting memory with a property generating the source on demand. The patch adds also unit test for the verbose attribute (which is public and documented, even it is said to be "outdated").

The patch removes also repr_fmt and num_fields parameters of the class definition template, compute these values using the list of fields.

I suggested to change Python 3.4.1 and 3.5.

Test script:
---
import email
import http.client
import pickle
import test.regrtest
import test.test_os
import tracemalloc
import xmlrpc.server

snap = tracemalloc.take_snapshot()
with open("dump.pickle", "wb") as fp:
    pickle.dump(snap, fp, 2)
---

With the patch, the memory footprint is reduced by 176 kB.
msg214003 - (view) Author: Eric Snow (eric.snow) * (Python committer) Date: 2014-03-18 18:40
Also be sure the have Raymond's sign-off before committing anything for this. :)
msg214212 - (view) Author: Raymond Hettinger (rhettinger) * (Python committer) Date: 2014-03-20 12:32
FWIW, the "verbose" option is mentioned as outdated because the "_source" attribute was added.

Also, there are real use cases, people are using the _source as writing it to a .py file so that the dynamic namedtuple generation step can be skipped on subsequent imports.  This is useful when people want to avoid the use of eval or want to run cython on the code.

The attribute can be "dropped". It is part of the API.

Sorry, the memory use bugs you.  It is bigger than typical docstrings but is not a significant memory consumer in most applications.

I like the idea of dynamically generating the source upon lookup, but want to think about whether there are any unintended consequences to that space saving hack.
msg214219 - (view) Author: Raymond Hettinger (rhettinger) * (Python committer) Date: 2014-03-20 13:02
The size of the _source attribute is about 2k per namedtuple class:

>>> from collections import namedtuple
>>> Response = namedtuple('Response', ['code', 'msg', 'compressed', 'written'])
>>> len(Response._source)
2174
msg215398 - (view) Author: Raymond Hettinger (rhettinger) * (Python committer) Date: 2014-04-02 20:28
Victor, I don't think the added complexity is worth 2k per named tuple class.  Every time I've gone down the path of lazy evaluation, I've paid an unexpected price for it down the road.  If the savings were huge, it might be worth it, but that isn't the case here.  This isn't really different than proposing that all docstring be in a separate module to be lazily loaded only when people look at help.
History
Date User Action Args
2017-07-17 13:31:32giampaolo.rodolasetnosy: + giampaolo.rodola
2014-04-02 20:28:04rhettingersetstatus: open -> closed
resolution: rejected
messages: + msg215398
2014-03-20 13:02:05rhettingersetmessages: + msg214219
2014-03-20 12:33:18rhettingersetpriority: normal -> low
title: Drop _source attribute of namedtuple (waste memory) -> Dynamically generate the _source attribute of namedtuple to save memory)
2014-03-20 12:32:37rhettingersetmessages: + msg214212
2014-03-20 12:22:47rhettingersetversions: - Python 3.4
2014-03-20 12:19:56rhettingersetassignee: rhettinger
2014-03-18 18:40:21eric.snowsetmessages: + msg214003
2014-03-18 08:57:54vstinnersetfiles: + namedtuple_source.patch
versions: + Python 3.5
title: Drop _source attribute of namedtuple -> Drop _source attribute of namedtuple (waste memory)
messages: + msg213949

keywords: + patch
type: resource usage
2014-03-18 07:33:20eric.snowsetmessages: + msg213946
2014-03-17 22:01:32vstinnersetmessages: + msg213904
2013-11-22 21:06:08eric.snowsetmessages: + msg203878
2013-11-22 20:55:34eric.araujosetmessages: + msg203870
2013-11-22 20:52:51eric.araujosetnosy: + eric.araujo
2013-11-22 00:42:28eric.snowsetnosy: + eric.snow
messages: + msg203707
2013-11-18 15:51:48vstinnersetmessages: + msg203304
2013-11-18 10:23:16eric.smithsetnosy: + eric.smith
2013-11-18 09:55:18christian.heimessetnosy: + christian.heimes
2013-11-18 09:43:38vstinnersetmessages: + msg203271
2013-11-18 09:42:10vstinnercreate