classification
Title: More reliable use of scandir in Path.glob()
Type: resource usage Stage: resolved
Components: Library (Lib) Versions: Python 3.9, Python 3.8, Python 3.7
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: miss-islington, serhiy.storchaka
Priority: normal Keywords: patch

Created on 2020-03-09 16:54 by serhiy.storchaka, last changed 2020-03-11 17:07 by serhiy.storchaka. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 18880 merged serhiy.storchaka, 2020-03-09 16:58
PR 18934 merged miss-islington, 2020-03-11 16:42
PR 18935 merged miss-islington, 2020-03-11 16:42
Messages (4)
msg363750 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2020-03-09 16:54
Path.glob() uses os.scandir() in the following code.

    entries = list(scandir(parent_path))

It properly closes the internal file descriptor opened by scandir() if success because it is automatically closed closed when the iterator is exhausted. But if it was interrupted (by KeyboardInterrupt, MemoryError or OSError), the file descriptor will be closed only when the iterator be collected by the garbage collector. It is unreliable on implementations like PyPy and emits a ResourceWarning.

The proposed code uses more reliable code

    with scandir(parent_path) as scandir_it:
        entries = list(scandir_it)

which is used in other sites (in the shutil module). I have no idea why I did not write it in this form at first place.
msg363934 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2020-03-11 16:42
New changeset 704e2065f8b8021a4a6999470fb6ed3453f7679e by Serhiy Storchaka in branch 'master':
bpo-39916: Use os.scandir() as context manager in Path.glob(). (GH-18880)
https://github.com/python/cpython/commit/704e2065f8b8021a4a6999470fb6ed3453f7679e
msg363942 - (view) Author: miss-islington (miss-islington) Date: 2020-03-11 17:00
New changeset b1b1d5ff11108620f5bc97976bb889f5cbb24373 by Miss Islington (bot) in branch '3.7':
bpo-39916: Use os.scandir() as context manager in Path.glob(). (GH-18880)
https://github.com/python/cpython/commit/b1b1d5ff11108620f5bc97976bb889f5cbb24373
msg363944 - (view) Author: miss-islington (miss-islington) Date: 2020-03-11 17:07
New changeset c22879914b03ff2da768e557b5c00e9c8c62c695 by Miss Islington (bot) in branch '3.8':
bpo-39916: Use os.scandir() as context manager in Path.glob(). (GH-18880)
https://github.com/python/cpython/commit/c22879914b03ff2da768e557b5c00e9c8c62c695
History
Date User Action Args
2020-03-11 17:07:50serhiy.storchakasetstatus: open -> closed
resolution: fixed
stage: patch review -> resolved
2020-03-11 17:07:11miss-islingtonsetmessages: + msg363944
2020-03-11 17:00:16miss-islingtonsetmessages: + msg363942
2020-03-11 16:42:31miss-islingtonsetpull_requests: + pull_request18287
2020-03-11 16:42:24miss-islingtonsetnosy: + miss-islington
pull_requests: + pull_request18286
2020-03-11 16:42:07serhiy.storchakasetmessages: + msg363934
2020-03-09 16:58:44serhiy.storchakasetkeywords: + patch
stage: patch review
pull_requests: + pull_request18237
2020-03-09 16:54:33serhiy.storchakacreate