This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: xml.ElementTree iterparse filehandle left open
Type: resource usage Stage: resolved
Components: Library (Lib), XML Versions: Python 3.11, Python 3.10, Python 3.9
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: anthonypjshaw, dtom9424, jacobtylerwalls, martin.panter, miss-islington, serhiy.storchaka, vkisforever
Priority: normal Keywords: patch

Created on 2021-02-22 04:11 by vkisforever, last changed 2022-04-11 14:59 by admin. This issue is now closed.

Files
File name Uploaded Description Edit
ETiterparse.py vkisforever, 2021-02-22 04:11 iterparse issue recreator
Pull Requests
URL Status Linked Edit
PR 31696 merged jacobtylerwalls, 2022-03-05 15:26
PR 31720 merged miss-islington, 2022-03-07 09:32
PR 31721 merged miss-islington, 2022-03-07 09:32
Messages (10)
msg387494 - (view) Author: Viktor Kis (vkisforever) * Date: 2021-02-22 04:11
Standard library xml.ElementTree - iterparse does not close filehandle properly when breaking out of a for loop iterating with iterparse. The issue only occurs when the user passes a "string" as a source object instead of a filehandle (that is because a user's filehandle can safely be closed in a context manager as shown in the attachment).
msg387495 - (view) Author: anthony shaw (anthonypjshaw) * (Python triager) Date: 2021-02-22 04:21
which version of Python were you doing this on?

The function in question is https://github.com/python/cpython/blob/master/Lib/xml/etree/ElementTree.py#L1233-L1278
msg387496 - (view) Author: Viktor Kis (vkisforever) * Date: 2021-02-22 04:22
ran python 3.9.2 on windows 10

1) What did you do?
Tried to remove folder that contains a xml file that was worked on by ElementTree iterparse with argument source='filename' (string type input)
2. What happened?
python was not able to remove the folder due to file lock still held by iterparse.
3. What did you expect to happen
iterparse was supposed to close the filehandle such that the user could then write/move/delete the xml file worked on, however python gave a PermissionError:

Traceback (most recent call last):
  File "C:\Users\vkisf\Desktop\00_MYSTUFF\01_SOFTWARE\01_PYTHON\01_GITHUB\pycascades2021\ETiterparse.py", line 43, in <module>
    shutil.rmtree('subdir2')
  File "C:\Users\vkisf\Desktop\00_MYSTUFF\01_SOFTWARE\01_PYTHON\01_GITHUB\pycascades2021\cpython\lib\shutil.py", line 740, in rmtree
    return _rmtree_unsafe(path, onerror)
  File "C:\Users\vkisf\Desktop\00_MYSTUFF\01_SOFTWARE\01_PYTHON\01_GITHUB\pycascades2021\cpython\lib\shutil.py", line 618, in _rmtree_unsafe
    onerror(os.unlink, fullname, sys.exc_info())
  File "C:\Users\vkisf\Desktop\00_MYSTUFF\01_SOFTWARE\01_PYTHON\01_GITHUB\pycascades2021\cpython\lib\shutil.py", line 616, in _rmtree_unsafe
    os.unlink(fullname)
PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'subdir2\\app.xml'
msg387497 - (view) Author: anthony shaw (anthonypjshaw) * (Python triager) Date: 2021-02-22 04:32
Example script attached works perfectly on macOS 

> python3.9 test.py
[('default', 'http://schemas.openxmlformats.org/officeDocument/2006/extended-properties'), ('vt', 'http://schemas.openxmlformats.org/officeDocument/2006/docPropsVTypes')]
[('default', 'http://schemas.openxmlformats.org/officeDocument/2006/extended-properties'), ('vt', 'http://schemas.openxmlformats.org/officeDocument/2006/docPropsVTypes')]
msg387498 - (view) Author: Viktor Kis (vkisforever) * Date: 2021-02-22 04:37
Confirmed that no apps were using the created xml file or folder. Windows Explorer was closed for all folders only the terminal shell was open. Issue still persists on windows 10 OS.
msg387571 - (view) Author: Tom Dougherty (dtom9424) Date: 2021-02-23 14:18
"erase all files"
msg388457 - (view) Author: Martin Panter (martin.panter) * (Python committer) Date: 2021-03-10 20:14
Perhaps this can be handled with Issue 25707, which is open for adding an API to close the file, similar to how "os.scandir" iterator implements a context manager and "close" method.
msg414647 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2022-03-07 09:32
New changeset 496c428de3318c9c5770937491b71dc3d3f18a6a by Jacob Walls in branch 'main':
bpo-43292: Fix file leak in `ET.iterparse()` when not exhausted (GH-31696)
https://github.com/python/cpython/commit/496c428de3318c9c5770937491b71dc3d3f18a6a
msg414653 - (view) Author: miss-islington (miss-islington) Date: 2022-03-07 10:32
New changeset 8acbb93c0763fa53b5959fe05d86ba275c9e8a5b by Miss Islington (bot) in branch '3.10':
bpo-43292: Fix file leak in `ET.iterparse()` when not exhausted (GH-31696)
https://github.com/python/cpython/commit/8acbb93c0763fa53b5959fe05d86ba275c9e8a5b
msg414658 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2022-03-07 11:49
New changeset 852d9b77abefcad2bb8d203e3ab9f2ca49ab305f by Miss Islington (bot) in branch '3.9':
[3.9] bpo-43292: Fix file leak in `ET.iterparse()` when not exhausted (GH-31696) (GH-31720)
https://github.com/python/cpython/commit/852d9b77abefcad2bb8d203e3ab9f2ca49ab305f
History
Date User Action Args
2022-04-11 14:59:41adminsetgithub: 87458
2022-03-07 11:50:57serhiy.storchakasetstatus: open -> closed
versions: + Python 3.10, Python 3.11
resolution: fixed
components: + Library (Lib)
superseder: Add the close method for ElementTree.iterparse() object ->
stage: patch review -> resolved
2022-03-07 11:49:10serhiy.storchakasetmessages: + msg414658
2022-03-07 10:32:06miss-islingtonsetmessages: + msg414653
2022-03-07 09:32:30miss-islingtonsetpull_requests: + pull_request29836
2022-03-07 09:32:25serhiy.storchakasetnosy: + serhiy.storchaka
messages: + msg414647
2022-03-07 09:32:25miss-islingtonsetnosy: + miss-islington
pull_requests: + pull_request29835
2022-03-05 15:26:50jacobtylerwallssetkeywords: + patch
nosy: + jacobtylerwalls

pull_requests: + pull_request29815
stage: patch review
2021-03-10 20:15:23martin.pantersettype: security -> resource usage
2021-03-10 20:14:54martin.pantersetsuperseder: Add the close method for ElementTree.iterparse() object

messages: + msg388457
nosy: + martin.panter
2021-02-23 14:18:26dtom9424setnosy: + dtom9424
type: crash -> security
messages: + msg387571
2021-02-22 04:37:36vkisforeversetmessages: + msg387498
2021-02-22 04:32:11anthonypjshawsetnosy: + anthonypjshaw
messages: + msg387497
2021-02-22 04:22:11vkisforeversetnosy: - anthonypjshaw
messages: + msg387496
2021-02-22 04:21:13anthonypjshawsetnosy: + anthonypjshaw
messages: + msg387495
2021-02-22 04:11:29vkisforevercreate