classification
Title: can't run close through itertools.chain on inner generator
Type: enhancement Stage:
Components: Extension Modules Versions: Python 3.1, Python 2.7
process
Status: closed Resolution: rejected
Dependencies: Superseder:
Assigned To: rhettinger Nosy List: amaury.forgeotdarc, dangyogi, rhettinger
Priority: normal Keywords:

Created on 2008-09-11 19:03 by dangyogi, last changed 2008-09-12 21:31 by rhettinger. This issue is now closed.

Messages (4)
msg73052 - (view) Author: Bruce Frederiksen (dangyogi) Date: 2008-09-11 19:03
There is no way to get generators to clean up (run their 'finally'
clause) when used as an inner iterable to chain:

>>> def gen(n):
...     try:
...         # do stuff yielding values
...     finally:
...         # clean up
>>> c = chain.from_iterable(map(gen, (1,2,3)))
>>> next(c)
0
>>> # done with c, but can't clean up inner gen!

Could you add a 'close' method to itertools.chain that would call close
(if present) on both the inner and other iterable?  Then clean up could
be done as:

>>> with closing(chain.from_iterable(map(gen, (1,2,3)))) as c:
...    next(c)
>>> # generator finalized by "with closing"!
msg73079 - (view) Author: Amaury Forgeot d'Arc (amaury.forgeotdarc) * (Python committer) Date: 2008-09-12 08:43
You somehow must tell the iterator that you are done with it.
This is best done by a "del c" in your first snippet, since c is the
last reference to the iterator. 
To do this in a function (or a context manager), the trick is to wrap
the map() iterator inside a generator function.

def wrapiterator(it):
    for x in it:
        yield x

Then your sample becomes:
>>> with closing(wrapiterator(chain.from_iterable(
...                  map(gen, (1,2,3))))) as c:
...    next(c)


Which of course can be rewritten as:

def closingiterator(it):
    def wrapper():
        for x in it:
            yield x
    return closing(wrapper())

>>> with closingiterator(chain.from_iterable(map(gen, (1,2,3))))) as c:
...    next(c)


This works because the only reference to the map() iterator is held by
the wrapper generator function. Calling its close() method will
terminate the function, delete its locals, ultimately call the
deallocator of the gen() iterator, which will fire the "finally" block.

Your proposal implies that all c-based iterators need to grow a "close"
method. This is not practical, and is best simulated with this wrapper
generator: close()ing the wrapper will (recursively) destroy the iterators.
If this also works for you, I suggest to close this issue.
msg73118 - (view) Author: Bruce Frederiksen (dangyogi) Date: 2008-09-12 18:52
What you propose is not portable code.  It won't work on jython or 
ironpython or pypy because they don't have reference counting garbage 
collectors that immediately reclaim objects.

Also I'm not asking that all c-based iterators grow a 'close' method 
(though this would be nice! :-) ).

For all other c-based iterators, you can do:

with closing(gen()) as argument:
    for x in map(fn, argument): ...

which is portable.

So I am only asking that 'close' be added to itertools.chain because 
there is no portable solution there.

-bruce

Amaury Forgeot d'Arc wrote:
> Amaury Forgeot d'Arc <amauryfa@gmail.com> added the comment:
>
> You somehow must tell the iterator that you are done with it.
> This is best done by a "del c" in your first snippet, since c is the
> last reference to the iterator. 
> To do this in a function (or a context manager), the trick is to wrap
> the map() iterator inside a generator function.
>
> def wrapiterator(it):
>     for x in it:
>         yield x
>
> Then your sample becomes:
>   
>>>> with closing(wrapiterator(chain.from_iterable(
>>>>         
> ...                  map(gen, (1,2,3))))) as c:
> ...    next(c)
>
>
> Which of course can be rewritten as:
>
> def closingiterator(it):
>     def wrapper():
>         for x in it:
>             yield x
>     return closing(wrapper())
>
>   
>>>> with closingiterator(chain.from_iterable(map(gen, (1,2,3))))) as c:
>>>>         
> ...    next(c)
>
>
> This works because the only reference to the map() iterator is held by
> the wrapper generator function. Calling its close() method will
> terminate the function, delete its locals, ultimately call the
> deallocator of the gen() iterator, which will fire the "finally" block.
>
> Your proposal implies that all c-based iterators need to grow a "close"
> method. This is not practical, and is best simulated with this wrapper
> generator: close()ing the wrapper will (recursively) destroy the iterators.
> If this also works for you, I suggest to close this issue.
>
msg73141 - (view) Author: Raymond Hettinger (rhettinger) * (Python committer) Date: 2008-09-12 21:31
Sorry, am going to reject this.   The use cases are somewhat uncommon
and I don't want to clutter-up tools that need to remain as simple as
possible.  The pure python code for chain() is so simple that it's not
hard to roll your own version as needed.
History
Date User Action Args
2008-09-12 21:31:01rhettingersetstatus: pending -> closed
resolution: works for me -> rejected
messages: + msg73141
2008-09-12 18:52:20dangyogisetmessages: + msg73118
2008-09-12 08:43:20amaury.forgeotdarcsetstatus: open -> pending
resolution: works for me
messages: + msg73079
nosy: + amaury.forgeotdarc
2008-09-11 21:09:31benjamin.petersonsetassignee: rhettinger
nosy: + rhettinger
versions: + Python 3.1, Python 2.7, - Python 3.0
2008-09-11 19:03:14dangyogicreate