Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

os.fwalk() silently skips remaining directories when error occurs #70047

Closed
SamsonLee mannequin opened this issue Dec 14, 2015 · 4 comments
Closed

os.fwalk() silently skips remaining directories when error occurs #70047

SamsonLee mannequin opened this issue Dec 14, 2015 · 4 comments
Assignees
Labels
stdlib Python modules in the Lib dir type-bug An unexpected behavior, bug, or error

Comments

@SamsonLee
Copy link
Mannequin

SamsonLee mannequin commented Dec 14, 2015

BPO 25860
Nosy @vstinner, @benhoyt, @hynek, @serhiy-storchaka
Files
  • fwalk-silently-skips-dirs-when-error-occurs.patch
  • fwalk-silently-skips-dirs-when-error-occurs-with-tests.patch
  • Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = 'https://github.com/serhiy-storchaka'
    closed_at = <Date 2015-12-23.09:31:10.785>
    created_at = <Date 2015-12-14.10:30:55.596>
    labels = ['type-bug', 'library']
    title = 'os.fwalk() silently skips remaining directories when error occurs'
    updated_at = <Date 2015-12-23.09:31:10.785>
    user = 'https://bugs.python.org/SamsonLee'

    bugs.python.org fields:

    activity = <Date 2015-12-23.09:31:10.785>
    actor = 'serhiy.storchaka'
    assignee = 'serhiy.storchaka'
    closed = True
    closed_date = <Date 2015-12-23.09:31:10.785>
    closer = 'serhiy.storchaka'
    components = ['Library (Lib)']
    creation = <Date 2015-12-14.10:30:55.596>
    creator = 'Samson Lee'
    dependencies = []
    files = ['41304', '41357']
    hgrepos = []
    issue_num = 25860
    keywords = ['patch']
    message_count = 4.0
    messages = ['256377', '256699', '256871', '256876']
    nosy_count = 7.0
    nosy_names = ['vstinner', 'benhoyt', 'neologix', 'python-dev', 'hynek', 'serhiy.storchaka', 'Samson Lee']
    pr_nums = []
    priority = 'normal'
    resolution = 'fixed'
    stage = 'resolved'
    status = 'closed'
    superseder = None
    type = 'behavior'
    url = 'https://bugs.python.org/issue25860'
    versions = ['Python 3.5', 'Python 3.6']

    @SamsonLee
    Copy link
    Mannequin Author

    SamsonLee mannequin commented Dec 14, 2015

    I noticed os.fwalk() produced different results as os.walk(). It turned out
    that os.fwalk() skips all remaining directories when OSError occurs (e.g. due
    to PermissionError).

    To reproduce the bug, first create a test directory structure:

        $ mkdir 1; touch 1/a
        $ mkdir 2; touch 2/b
        $ mkdir 3; touch 3/c
        $ mkdir 4; touch 4/c

    At this stage, everything is okay, both os.fwalk() os.walk() produce the same
    results:

        >>> import os
        >>> for root, dirs, files in os.walk('.'):
        ...     dirs.sort()
        ...     print(root, dirs, files)
        . ['1', '2', '3', '4'] []
        ./1 [] ['a']
        ./2 [] ['b']
        ./3 [] ['c']
        ./4 [] ['d']
        >>> for root, dirs, files, fd in os.fwalk('.'):
        ...     dirs.sort()
        ...     print(root, dirs, files)
        . ['1', '2', '3', '4'] []
        ./1 [] ['a']
        ./2 [] ['b']
        ./3 [] ['c']
        ./4 [] ['d']

    To introduce an error, force a PermissionError on one of the directories:

        $ sudo chown root:root 2
        $ sudo chmod 700 2

    Now, the os.fwalk() results are different (trust me, os.walk() is still okay):

        >>> for root, dirs, files, fd in os.fwalk('.'):
        ...     dirs.sort()
        ...     print(root, dirs, files)
        . ['1', '2', '3', '4'] []
        ./1 [] ['a']

    So it seems that os.fwalk skips remaining directories after an error occurs.

    The cause of the problem is in this part of os.py:

        for name in dirs:
            try:
                orig_st = stat(name, dir_fd=topfd, follow_symlinks=follow_symlinks)
                dirfd = open(name, O_RDONLY, dir_fd=topfd)
            except OSError as err:
                if onerror is not None:
                    onerror(err)
                break

    To fix it, simply replace break with continue. Patch attached.

    Cheers

    @SamsonLee SamsonLee mannequin added stdlib Python modules in the Lib dir type-bug An unexpected behavior, bug, or error labels Dec 14, 2015
    @serhiy-storchaka
    Copy link
    Member

    LGTM. Here is a patch with tests.

    @serhiy-storchaka serhiy-storchaka self-assigned this Dec 18, 2015
    @python-dev
    Copy link
    Mannequin

    python-dev mannequin commented Dec 22, 2015

    New changeset 767262c149ca by Serhiy Storchaka in branch '3.5':
    Issue bpo-25860: os.fwalk() no longer skips remaining directories when error occurs.
    https://hg.python.org/cpython/rev/767262c149ca

    New changeset a85675dabb8f by Serhiy Storchaka in branch 'default':
    Issue bpo-25860: os.fwalk() no longer skips remaining directories when error occurs.
    https://hg.python.org/cpython/rev/a85675dabb8f

    @python-dev
    Copy link
    Mannequin

    python-dev mannequin commented Dec 22, 2015

    New changeset 7995a81236b6 by Serhiy Storchaka in branch '3.5':
    Issue bpo-25860: Fixed test failure caused by inconsistency of os.walk() and
    https://hg.python.org/cpython/rev/7995a81236b6

    New changeset dcf9e9ae5393 by Serhiy Storchaka in branch 'default':
    Issue bpo-25860: Fixed test failure caused by inconsistency of os.walk() and
    https://hg.python.org/cpython/rev/dcf9e9ae5393

    @ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
    Labels
    stdlib Python modules in the Lib dir type-bug An unexpected behavior, bug, or error
    Projects
    None yet
    Development

    No branches or pull requests

    1 participant