Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

DirEntry.stat method should release GIL #89175

Closed
skonieczny mannequin opened this issue Aug 26, 2021 · 5 comments
Closed

DirEntry.stat method should release GIL #89175

skonieczny mannequin opened this issue Aug 26, 2021 · 5 comments
Labels
3.11 bug and security fixes performance Performance or resource usage stdlib Python modules in the Lib dir

Comments

@skonieczny
Copy link
Mannequin

skonieczny mannequin commented Aug 26, 2021

BPO 45012
Nosy @ambv, @skonieczny
PRs
  • bpo-45012 Release GIL around stat in os.scandir #28085
  • Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = None
    closed_at = <Date 2021-09-07.17:57:48.210>
    created_at = <Date 2021-08-26.11:53:33.816>
    labels = ['3.11', 'library', 'performance']
    title = 'DirEntry.stat method should release GIL'
    updated_at = <Date 2021-09-08.15:13:01.715>
    user = 'https://github.com/skonieczny'

    bugs.python.org fields:

    activity = <Date 2021-09-08.15:13:01.715>
    actor = 'lukasz.langa'
    assignee = 'none'
    closed = True
    closed_date = <Date 2021-09-07.17:57:48.210>
    closer = 'lukasz.langa'
    components = ['Library (Lib)']
    creation = <Date 2021-08-26.11:53:33.816>
    creator = 'uosiu'
    dependencies = []
    files = []
    hgrepos = []
    issue_num = 45012
    keywords = ['patch']
    message_count = 5.0
    messages = ['400337', '401317', '401318', '401351', '401390']
    nosy_count = 2.0
    nosy_names = ['lukasz.langa', 'uosiu']
    pr_nums = ['28085']
    priority = 'normal'
    resolution = 'fixed'
    stage = 'resolved'
    status = 'closed'
    superseder = None
    type = 'performance'
    url = 'https://bugs.python.org/issue45012'
    versions = ['Python 3.11']

    @skonieczny
    Copy link
    Mannequin Author

    skonieczny mannequin commented Aug 26, 2021

    We have an application that crawls filesystem using os.scandir. It uses multiple threads for various things. Application is used on variety of filesystems, some of them might be slow or occasionally unresponsive.

    We have found out that sometimes whole crawling process is stuck and no thread makes any progress, even threads that are really simple, makes no IO and do not hold any locks. After running py-spy on process that was stuck we saw that one of the threads has entered dentry.stat(follow_symlinks=False) line and still holds the GIL. Other threads are stuck, because they are waiting for the GIL. This situation can take a long time.

    I think that DirEntry should release GIL when stat cache is empty and syscall is performed.

    This bug has already been fixed in scandir module. See: benhoyt/scandir#131

    @skonieczny skonieczny mannequin added type-bug An unexpected behavior, bug, or error 3.7 (EOL) end of life 3.8 only security fixes 3.9 only security fixes 3.10 only security fixes 3.11 bug and security fixes stdlib Python modules in the Lib dir labels Aug 26, 2021
    @ambv
    Copy link
    Contributor

    ambv commented Sep 7, 2021

    New changeset 9dc363e by Stanisław Skonieczny in branch 'main':
    bpo-45012: Release GIL around stat in os.scandir (GH-28085)
    9dc363e

    @ambv
    Copy link
    Contributor

    ambv commented Sep 7, 2021

    Dzięki za zgłoszenie i poprawkę, Stanisław! ✨ 🍰 ✨

    @ambv ambv removed 3.7 (EOL) end of life 3.8 only security fixes 3.9 only security fixes 3.10 only security fixes labels Sep 7, 2021
    @ambv ambv closed this as completed Sep 7, 2021
    @ambv ambv added performance Performance or resource usage and removed type-bug An unexpected behavior, bug, or error 3.7 (EOL) end of life 3.8 only security fixes 3.9 only security fixes 3.10 only security fixes labels Sep 7, 2021
    @ambv ambv closed this as completed Sep 7, 2021
    @ambv ambv removed the type-bug An unexpected behavior, bug, or error label Sep 7, 2021
    @ambv ambv added the performance Performance or resource usage label Sep 7, 2021
    @skonieczny
    Copy link
    Mannequin Author

    skonieczny mannequin commented Sep 8, 2021

    Bardzo proszę :)

    This fix is quite important for us. We would like to start using this fix in our product. Is there something I could do to backport it to 3.9?

    @ambv
    Copy link
    Contributor

    ambv commented Sep 8, 2021

    Sadly, we can't backport this to 3.9 as it's only accepting bugfixes and this is a performance improvement. 3.10 is out of scope for this too as 3.10.0rc2 shipped last night.

    @ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
    Labels
    3.11 bug and security fixes performance Performance or resource usage stdlib Python modules in the Lib dir
    Projects
    None yet
    Development

    No branches or pull requests

    1 participant