classification
Title: os.fwalk() silently skips remaining directories when error occurs
Type: behavior Stage: resolved
Components: Library (Lib) Versions: Python 3.6, Python 3.5
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: serhiy.storchaka Nosy List: Samson Lee, benhoyt, hynek, neologix, python-dev, serhiy.storchaka, vstinner
Priority: normal Keywords: patch

Created on 2015-12-14 10:30 by Samson Lee, last changed 2015-12-23 09:31 by serhiy.storchaka. This issue is now closed.

Files
File name Uploaded Description Edit
fwalk-silently-skips-dirs-when-error-occurs.patch Samson Lee, 2015-12-14 10:30 review
fwalk-silently-skips-dirs-when-error-occurs-with-tests.patch serhiy.storchaka, 2015-12-18 18:32 review
Messages (4)
msg256377 - (view) Author: Samson Lee (Samson Lee) Date: 2015-12-14 10:30
I noticed os.fwalk() produced different results as os.walk(). It turned out
that os.fwalk() skips all remaining directories when OSError occurs (e.g. due
to PermissionError).

To reproduce the bug, first create a test directory structure:

    $ mkdir 1; touch 1/a
    $ mkdir 2; touch 2/b
    $ mkdir 3; touch 3/c
    $ mkdir 4; touch 4/c

At this stage, everything is okay, both os.fwalk() os.walk() produce the same
results:

    >>> import os
    >>> for root, dirs, files in os.walk('.'):
    ...     dirs.sort()
    ...     print(root, dirs, files)
    . ['1', '2', '3', '4'] []
    ./1 [] ['a']
    ./2 [] ['b']
    ./3 [] ['c']
    ./4 [] ['d']
    >>> for root, dirs, files, fd in os.fwalk('.'):
    ...     dirs.sort()
    ...     print(root, dirs, files)
    . ['1', '2', '3', '4'] []
    ./1 [] ['a']
    ./2 [] ['b']
    ./3 [] ['c']
    ./4 [] ['d']

To introduce an error, force a PermissionError on one of the directories:

    $ sudo chown root:root 2
    $ sudo chmod 700 2

Now, the os.fwalk() results are different (trust me, os.walk() is still okay):

    >>> for root, dirs, files, fd in os.fwalk('.'):
    ...     dirs.sort()
    ...     print(root, dirs, files)
    . ['1', '2', '3', '4'] []
    ./1 [] ['a']

So it seems that os.fwalk skips remaining directories after an error occurs.

The cause of the problem is in this part of os.py:

    for name in dirs:
        try:
            orig_st = stat(name, dir_fd=topfd, follow_symlinks=follow_symlinks)
            dirfd = open(name, O_RDONLY, dir_fd=topfd)
        except OSError as err:
            if onerror is not None:
                onerror(err)
            break

To fix it, simply replace break with continue. Patch attached.

Cheers
msg256699 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2015-12-18 18:32
LGTM. Here is a patch with tests.
msg256871 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2015-12-22 22:09
New changeset 767262c149ca by Serhiy Storchaka in branch '3.5':
Issue #25860: os.fwalk() no longer skips remaining directories when error occurs.
https://hg.python.org/cpython/rev/767262c149ca

New changeset a85675dabb8f by Serhiy Storchaka in branch 'default':
Issue #25860: os.fwalk() no longer skips remaining directories when error occurs.
https://hg.python.org/cpython/rev/a85675dabb8f
msg256876 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2015-12-22 22:38
New changeset 7995a81236b6 by Serhiy Storchaka in branch '3.5':
Issue #25860: Fixed test failure caused by inconsistency of os.walk() and
https://hg.python.org/cpython/rev/7995a81236b6

New changeset dcf9e9ae5393 by Serhiy Storchaka in branch 'default':
Issue #25860: Fixed test failure caused by inconsistency of os.walk() and
https://hg.python.org/cpython/rev/dcf9e9ae5393
History
Date User Action Args
2015-12-23 09:31:10serhiy.storchakasetstatus: open -> closed
resolution: fixed
stage: patch review -> resolved
2015-12-22 22:38:13python-devsetmessages: + msg256876
2015-12-22 22:09:25python-devsetnosy: + python-dev
messages: + msg256871
2015-12-18 18:32:27serhiy.storchakasetfiles: + fwalk-silently-skips-dirs-when-error-occurs-with-tests.patch

assignee: serhiy.storchaka
versions: + Python 3.6, - Python 3.4
nosy: + serhiy.storchaka

messages: + msg256699
stage: patch review
2015-12-14 10:57:36Samson Leesetnosy: + vstinner, benhoyt
2015-12-14 10:41:58Samson Leesetnosy: + neologix, hynek
2015-12-14 10:30:55Samson Leecreate