This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: DirEntry.stat method should release GIL
Type: performance Stage: resolved
Components: Library (Lib) Versions: Python 3.11
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: lukasz.langa, uosiu
Priority: normal Keywords: patch

Created on 2021-08-26 11:53 by uosiu, last changed 2022-04-11 14:59 by admin. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 28085 merged uosiu, 2021-08-31 11:06
Messages (5)
msg400337 - (view) Author: uosiu (uosiu) * Date: 2021-08-26 11:53
We have an application that crawls filesystem using `os.scandir`. It uses multiple threads for various things. Application is used on variety of filesystems, some of them might be slow or occasionally unresponsive.

We have found out that sometimes whole crawling process is stuck and no thread makes any progress, even threads that are really simple, makes no IO and do not hold any locks. After running py-spy on process that was stuck we saw that one of the threads has entered `dentry.stat(follow_symlinks=False)` line and still holds the GIL. Other threads are stuck, because they are waiting for the GIL. This situation can take a long time.

I think that `DirEntry` should release GIL when stat cache is empty and syscall is performed.

This bug has already been fixed in `scandir` module. See: https://github.com/benhoyt/scandir/issues/131
msg401317 - (view) Author: Łukasz Langa (lukasz.langa) * (Python committer) Date: 2021-09-07 17:55
New changeset 9dc363ee7cf2eb6ff374fbf7bbeb0b333f4afb8f by Stanisław Skonieczny in branch 'main':
bpo-45012: Release GIL around stat in os.scandir (GH-28085)
https://github.com/python/cpython/commit/9dc363ee7cf2eb6ff374fbf7bbeb0b333f4afb8f
msg401318 - (view) Author: Łukasz Langa (lukasz.langa) * (Python committer) Date: 2021-09-07 17:57
Dzięki za zgłoszenie i poprawkę, Stanisław! ✨ 🍰 ✨
msg401351 - (view) Author: uosiu (uosiu) * Date: 2021-09-08 07:41
Bardzo proszę :)

This fix is quite important for us. We would like to start using this fix in our product. Is there something I could do to backport it to 3.9?
msg401390 - (view) Author: Łukasz Langa (lukasz.langa) * (Python committer) Date: 2021-09-08 15:13
Sadly, we can't backport this to 3.9 as it's only accepting bugfixes and this is a performance improvement. 3.10 is out of scope for this too as 3.10.0rc2 shipped last night.
History
Date User Action Args
2022-04-11 14:59:49adminsetgithub: 89175
2021-09-08 15:13:01lukasz.langasetmessages: + msg401390
2021-09-08 07:41:18uosiusetmessages: + msg401351
2021-09-07 17:57:48lukasz.langasetstatus: open -> closed
versions: - Python 3.6, Python 3.7, Python 3.8, Python 3.9, Python 3.10
type: behavior -> performance
messages: + msg401318

resolution: fixed
stage: patch review -> resolved
2021-09-07 17:55:28lukasz.langasetnosy: + lukasz.langa
messages: + msg401317
2021-08-31 11:06:08uosiusetkeywords: + patch
stage: patch review
pull_requests: + pull_request26528
2021-08-26 11:53:33uosiucreate