classification
Title: itertools.tee does not release resources during iteration?
Type: resource usage Stage: resolved
Components: Library (Lib) Versions: Python 3.9
process
Status: closed Resolution: not a bug
Dependencies: Superseder:
Assigned To: rhettinger Nosy List: pwuertz, rhettinger
Priority: normal Keywords:

Created on 2020-03-23 14:34 by pwuertz, last changed 2020-03-23 19:44 by rhettinger. This issue is now closed.

Messages (3)
msg364849 - (view) Author: Peter W├╝rtz (pwuertz) Date: 2020-03-23 14:34
Itertools `tee` does not seem to de-reference yielded items, even after consumption of all items from all tee-iterators.

According to the documentation (to my understanding), there shouldn't be any extra memory requirement as long as the tee-iterators are consumed in a balanced way. I.e. after an item was pulled from all iterators there shouldn't be any residual reference to it.

This is true for the example-implementation mentioned in the documentation, but `itertools.tee` doesn't de-reference items until the tee-iterator itself is deleted:

https://pastebin.com/r3JUkH41

Is this a bug or am I missing something?
msg364886 - (view) Author: Raymond Hettinger (rhettinger) * (Python committer) Date: 2020-03-23 19:01
Whats going on is that the implementation tracks blocks of 56 references at a time, freeing entire blocks at time.   So the memory release does occur periodically, but not as aggressively as it could. This code shows what is going on:

    # Show the timing of when objects are decreffed

    from itertools import tee

    class Notify:
        def __init__(self, x):
                self.x = x
        def __del__(self):
                print('<Clearing: %r>' % self.x)
        def __repr__(self):
                return 'Notify(%r)' % self.x

    it = map(Notify, range(100))
    p, q = tee(it)
    for i in range(100):
        next(p)
        next(q)
        print(i)
    print('Deleting it')
    del it
    print('Deleting p')
    del p
    print('Deleting q')
    del q

I'll look to see if the objects can be freed sooner without doing major brain surgery to the existing code.
msg364891 - (view) Author: Raymond Hettinger (rhettinger) * (Python committer) Date: 2020-03-23 19:44
I've just reviewed the code.  The observed behavior is an intrinsic part of the design and was an expected behavior (noted in the comments).  I don't see a non-invasive way of making the memory releases more aggressive.
History
Date User Action Args
2020-03-23 19:44:20rhettingersetstatus: open -> closed
versions: + Python 3.9, - Python 3.7
messages: + msg364891

resolution: not a bug
stage: resolved
2020-03-23 19:01:55rhettingersetassignee: rhettinger

messages: + msg364886
nosy: + rhettinger
2020-03-23 14:34:33pwuertzcreate