classification
Title: Add the close method for ElementTree.iterparse() object
Type: resource usage Stage: needs patch
Components: Library (Lib) Versions: Python 3.8
process
Status: open Resolution:
Dependencies: 25638 Superseder:
Assigned To: serhiy.storchaka Nosy List: abarry, furkanonder, scoder, serhiy.storchaka
Priority: normal Keywords:

Created on 2015-11-23 13:57 by serhiy.storchaka, last changed 2020-05-06 22:39 by furkanonder.

Messages (8)
msg255159 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2015-11-23 13:57
If ElementTree.iterparse() is called with file names, it opens a file. When resulting iterator is not exhausted, the file lefts not closed.

>>> import xml.etree.ElementTree as ET
>>> import gc
>>> ET.iterparse('/dev/null')
<xml.etree.ElementTree._IterParseIterator object at 0xb6f9e38c>
>>> gc.collect()
__main__:1: ResourceWarning: unclosed file <_io.BufferedReader name='/dev/null'>
34

Martin Panter proposed in issue25688 to add an explicit way to clean it up, like a generator.close() method.
msg255164 - (view) Author: Anilyka Barry (abarry) * (Python triager) Date: 2015-11-23 14:17
I am unable to reproduce the issue on Windows 7 with 3.5.0; I have tried opening a small (non-empty) text. Here's the result:


>>> import xml.etree.ElementTree as ET
>>> import gc
>>> ET.iterparse("E:/New.txt")
<xml.etree.ElementTree._IterParseIterator object at 0x0023ABB0>
>>> gc.collect()
59
msg255171 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2015-11-23 14:56
You have to enable deprecation warnings. Run the interpreter with the -Wa option.
msg255173 - (view) Author: Anilyka Barry (abarry) * (Python triager) Date: 2015-11-23 15:08
Oh, my bad. Ignore my last message, behaviour is identical then. Thanks for clearing that up.
msg341003 - (view) Author: Stefan Behnel (scoder) * (Python committer) Date: 2019-04-27 17:32
I don't think there is a need for a close() method. Instead, the iterator should close the file first thing when it's done with it, but only if it owns it. Therefore, the fix in issue 25688 seems correct.

Closing can also be done explicitly in a finaliser of the iterator, if implicit closing via decref is too lax.
msg341253 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2019-05-02 08:01
Implicit closing an exhausted iterator helps only the iterator is iterated to the end. If the iteration has been stopped before the end, we get a leak of the file descriptor. Closing the file descriptor in the finalizer can be deferred to undefined term, especially in implementations without reference counting. Since file descriptors are limited resource, this can cause troubles in real programs.

Reasons for close() in iterparse objects are the same as for close in files and generators.

Maybe we will need to implement the full generator protocol (send() and throw()) in the iterparse objects, but currently I do not know use cases for this.
msg341258 - (view) Author: Stefan Behnel (scoder) * (Python committer) Date: 2019-05-02 08:32
Ok, I think it's reasonable to make the resource management explicit for the specific case of letting iterparse() open the file. That suggests that there should also be context manager support, given that safe usages would often involve a try-finally.

Since it might not always be obvious for users when they need to close the iterator or not, I would also suggest to not let it raise an error on a double-close, i.e. if .close() was already called or the iterator was already exhausted (and the file closed automatically), calling .close() should just do nothing.
msg368308 - (view) Author: Furkan Onder (furkanonder) * Date: 2020-05-06 22:39
Python 3.8.2 (default, Apr  8 2020, 14:31:25) 
[GCC 9.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import xml.etree.ElementTree as ET
>>> import gc
>>> ET.iterparse('/dev/null')
<xml.etree.ElementTree.iterparse.<locals>.IterParseIterator object at 0x7fb96f679d00>
>>> gc.collect()
34

The warning(__main__:1: ResourceWarning: unclosed file <_io.BufferedReader name='/dev/null'>) is no longer available in python3.8.2
History
Date User Action Args
2020-05-06 22:39:43furkanondersetnosy: + furkanonder
messages: + msg368308
2019-05-02 08:32:19scodersetmessages: + msg341258
2019-05-02 08:01:27serhiy.storchakasetmessages: + msg341253
2019-04-27 17:32:50scodersetnosy: + scoder

messages: + msg341003
versions: + Python 3.8, - Python 3.6
2015-11-26 09:40:50serhiy.storchakasetdependencies: + Verify the etree_parse and etree_iterparse benchmarks are working appropriately
2015-11-23 15:08:12abarrysetmessages: + msg255173
2015-11-23 14:56:23serhiy.storchakasetmessages: + msg255171
2015-11-23 14:17:12abarrysetnosy: + abarry
messages: + msg255164
2015-11-23 13:57:25serhiy.storchakacreate