os.fwalk() unhandled exception when error occurs accessing symbolic link target #72539

SamsonLee · 2016-10-04T07:06:56Z

BPO	28353
Nosy	@loewis, @hynek, @serhiy-storchaka, @zhangyangyu, @Mariatta
PRs	[Do Not Merge] Convert `Misc/NEWS` so that it is managed by towncrier #552
Files	fwalk_oserror.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/serhiy-storchaka'
closed_at = <Date 2016-10-28.06:26:12.146>
created_at = <Date 2016-10-04.07:06:55.953>
labels = ['3.7', 'type-bug', 'library']
title = 'os.fwalk() unhandled exception when error occurs accessing symbolic link target'
updated_at = <Date 2017-03-31.16:36:09.654>
user = 'https://bugs.python.org/SamsonLee'

bugs.python.org fields:

activity = <Date 2017-03-31.16:36:09.654>
actor = 'dstufft'
assignee = 'serhiy.storchaka'
closed = True
closed_date = <Date 2016-10-28.06:26:12.146>
closer = 'xiang.zhang'
components = ['Library (Lib)']
creation = <Date 2016-10-04.07:06:55.953>
creator = 'Samson Lee'
dependencies = []
files = ['44982']
hgrepos = []
issue_num = 28353
keywords = ['patch']
message_count = 16.0
messages = ['278016', '278017', '278018', '278024', '278044', '278118', '278178', '279393', '279418', '279422', '279581', '279582', '279583', '279584', '279585', '279586']
nosy_count = 8.0
nosy_names = ['loewis', 'neologix', 'python-dev', 'hynek', 'serhiy.storchaka', 'xiang.zhang', 'Samson Lee', 'Mariatta']
pr_nums = ['552']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue28353'
versions = ['Python 3.5', 'Python 3.6', 'Python 3.7']

The text was updated successfully, but these errors were encountered:

SamsonLee · 2016-10-04T07:06:55Z

The bug is os.fwalk() crashes with unhandled exception when there is an error
accessing symbolic link targets.

To reproduce the bug, create a symbolic link that targets a file that you do not
have permission to access:

    $ touch handsoff
    $ sudo chown root:root handsoff
    $ sudo chmod 700 handsoff
    $ ln -s handsoff blah

Now, os.fwalk() fails:

    >>> for root, dirs, files, fd in os.fwalk('.'):
    ...     print(root, dirs, files)
    ...
    Traceback (most recent call last):
      File "test_fwalk_permission_error.py", line 3, in <module>
        for root, dirs, files, fd in os.fwalk('.'):
      File "/usr/lib64/python3.5/os.py", line 520, in fwalk
        yield from _fwalk(topfd, top, topdown, onerror, follow_symlinks)
      File "/usr/lib64/python3.5/os.py", line 537, in _fwalk
        if st.S_ISDIR(stat(name, dir_fd=topfd).st_mode):
    PermissionError: [Errno 13] Permission denied: 'blah'

The cause of the problem is in this part of os.py:

for name in names:
    try:
        # Here, we don't use AT_SYMLINK_NOFOLLOW to be consistent with
        # walk() which reports symlinks to directories as directories.
        # We do however check for symlinks before recursing into
        # a subdirectory.
        if st.S_ISDIR(stat(name, dir_fd=topfd).st_mode):
            dirs.append(name)
        else:
            nondirs.append(name)
    except FileNotFoundError:
        try:
            # Add dangling symlinks, ignore disappeared files
            if st.S_ISLNK(stat(name, dir_fd=topfd, follow_symlinks=False)
                        .st_mode):
                nondirs.append(name)
        except FileNotFoundError:
            continue

To fix it, simply replace FileNotFoundError with more general OSError.

Cheers

SamsonLee · 2016-10-04T07:08:56Z

Similar to http://bugs.python.org/issue25860

serhiy-storchaka · 2016-10-04T07:45:40Z

I can't reproduce the issue.

$ mkdir testdir
$ touch testdir/handsoff
$ sudo chown root:root testdir/handsoff
$ sudo chmod 700 testdir/handsoff
$ ln -s handsoff testdir/blah
$ python3.5
>>> import os
>>> list(os.walk('testdir'))
[('testdir', [], ['handsoff', 'blah'])]
>>> list(os.fwalk('testdir'))
[('testdir', [], ['handsoff', 'blah'], 3)]

Ubuntu 16.04, Linux 4.4, tested on ext4 and tmpfs filesystems.

SamsonLee · 2016-10-04T09:07:53Z

Sorry, the target file needs to be in a directory you don't have permission to access:

    $ mkdir handsoff
    $ sudo chown root:root handsoff
    $ sudo chmod 700 handsoff
    $ ln -s handsoff/anything blah

serhiy-storchaka · 2016-10-04T14:45:17Z

Yes, there is a difference between os.walk() and os.fwalk() in some cases. I don't know whether it was introduced deliberately or by accident. os.fwalk() was added in bpo-13734, FileNotFoundError is used since bpo-14773.

SamsonLee · 2016-10-05T10:49:36Z

This is definitely a bug because there is no way an unhandled exception like this is acceptable behaviour. The behaviour is that the fwalk iteration will stop and there is no way to recover / continue. Also fwalk takes an onerror callback function that won't be called with this error.

serhiy-storchaka · 2016-10-06T07:15:05Z

Here is a patch with tests. Needed to test it on Windows.

python-dev · 2016-10-25T11:35:36Z

New changeset fe1fea4ded04 by Serhiy Storchaka in branch '3.5':
Issue bpo-28353: os.fwalk() no longer fails on broken links.
https://hg.python.org/cpython/rev/fe1fea4ded04

New changeset e99ec3c77a63 by Serhiy Storchaka in branch '3.6':
Issue bpo-28353: os.fwalk() no longer fails on broken links.
https://hg.python.org/cpython/rev/e99ec3c77a63

New changeset 33a1a3dd0051 by Serhiy Storchaka in branch 'default':
Issue bpo-28353: os.fwalk() no longer fails on broken links.
https://hg.python.org/cpython/rev/33a1a3dd0051

python-dev · 2016-10-25T16:03:55Z

New changeset ec12e16ea6a1 by Serhiy Storchaka in branch '3.6':
Issue bpo-28353: Try to fix tests.
https://hg.python.org/cpython/rev/ec12e16ea6a1

New changeset a0913dbadea6 by Serhiy Storchaka in branch 'default':
Issue bpo-28353: Try to fix tests.
https://hg.python.org/cpython/rev/a0913dbadea6

python-dev · 2016-10-25T17:20:53Z

New changeset 470224ec16b6 by Serhiy Storchaka in branch '3.5':
Issue bpo-28353: Fixed tests of os.fwalk() with broken links.
https://hg.python.org/cpython/rev/470224ec16b6

zhangyangyu · 2016-10-28T04:54:53Z

Serhiy, after your commits, test_os requires root privileges or it'll fail. This is not the case before.

[cpython]$ ./python -m test test_os
Run tests sequentially
0:00:00 [1/1] test_os
test test_os crashed -- Traceback (most recent call last):
  File "/home/angwer/cpython/Lib/test/libregrtest/runtest.py", line 164, in runtest_inner
    test_runner()
  File "/home/angwer/cpython/Lib/test/libregrtest/runtest.py", line 163, in test_runner
    support.run_unittest(tests)
  File "/home/angwer/cpython/Lib/test/support/__init__.py", line 1849, in run_unittest
    _run_suite(suite)
  File "/home/angwer/cpython/Lib/test/support/__init__.py", line 1824, in _run_suite
    raise TestFailed(err)
test.support.TestFailed: multiple errors occurred; run in verbose mode for details

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/angwer/cpython/Lib/test/libregrtest/runtest.py", line 167, in runtest_inner
    test_time = time.time() - start_time
  File "/home/angwer/cpython/Lib/test/libregrtest/save_env.py", line 278, in __exit__
    restore(original)
  File "/home/angwer/cpython/Lib/test/libregrtest/save_env.py", line 235, in restore_files
    support.rmtree(fn)
  File "/home/angwer/cpython/Lib/test/support/__init__.py", line 377, in rmtree
    _rmtree(path)
  File "/home/angwer/cpython/Lib/shutil.py", line 474, in rmtree
    _rmtree_safe_fd(fd, path, onerror)
  File "/home/angwer/cpython/Lib/shutil.py", line 412, in _rmtree_safe_fd
    _rmtree_safe_fd(dirfd, fullname, onerror)
  File "/home/angwer/cpython/Lib/shutil.py", line 412, in _rmtree_safe_fd
    _rmtree_safe_fd(dirfd, fullname, onerror)
  File "/home/angwer/cpython/Lib/shutil.py", line 408, in _rmtree_safe_fd
    onerror(os.open, fullname, sys.exc_info())
  File "/home/angwer/cpython/Lib/shutil.py", line 406, in _rmtree_safe_fd
    dirfd = os.open(name, os.O_RDONLY, dir_fd=topfd)
PermissionError: [Errno 13] Permission denied: 'SUB21'

'test_os' left behind directory '@test_23400_tmp' and it couldn't be removed: [Errno 13] Permission denied: 'SUB21'
test_os failed

1 test failed:
test_os

Total duration: 292 ms
Tests result: FAILURE
Traceback (most recent call last):
  File "/home/angwer/cpython/Lib/test/support/__init__.py", line 914, in temp_dir
    yield path
  File "/home/angwer/cpython/Lib/test/support/__init__.py", line 963, in temp_cwd
    yield cwd_dir
  File "/home/angwer/cpython/Lib/test/libregrtest/main.py", line 468, in main
    self._main(tests, kwargs)
  File "/home/angwer/cpython/Lib/test/libregrtest/main.py", line 497, in _main
    sys.exit(len(self.bad) > 0 or self.interrupted)
SystemExit: True

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/angwer/cpython/Lib/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/home/angwer/cpython/Lib/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/home/angwer/cpython/Lib/test/__main__.py", line 2, in <module>
    main()
  File "/home/angwer/cpython/Lib/test/libregrtest/main.py", line 532, in main
    Regrtest().main(tests=tests, **kwargs)
  File "/home/angwer/cpython/Lib/test/libregrtest/main.py", line 468, in main
    self._main(tests, kwargs)
  File "/home/angwer/cpython/Lib/contextlib.py", line 100, in __exit__
    self.gen.throw(type, value, traceback)
  File "/home/angwer/cpython/Lib/test/support/__init__.py", line 963, in temp_cwd
    yield cwd_dir
  File "/home/angwer/cpython/Lib/contextlib.py", line 100, in __exit__
    self.gen.throw(type, value, traceback)
  File "/home/angwer/cpython/Lib/test/support/__init__.py", line 917, in temp_dir
    rmtree(path)
  File "/home/angwer/cpython/Lib/test/support/__init__.py", line 377, in rmtree
    _rmtree(path)
  File "/home/angwer/cpython/Lib/shutil.py", line 474, in rmtree
    _rmtree_safe_fd(fd, path, onerror)
  File "/home/angwer/cpython/Lib/shutil.py", line 412, in _rmtree_safe_fd
    _rmtree_safe_fd(dirfd, fullname, onerror)
  File "/home/angwer/cpython/Lib/shutil.py", line 412, in _rmtree_safe_fd
    _rmtree_safe_fd(dirfd, fullname, onerror)
  File "/home/angwer/cpython/Lib/shutil.py", line 412, in _rmtree_safe_fd
    _rmtree_safe_fd(dirfd, fullname, onerror)
  File "/home/angwer/cpython/Lib/shutil.py", line 408, in _rmtree_safe_fd
    onerror(os.open, fullname, sys.exc_info())
  File "/home/angwer/cpython/Lib/shutil.py", line 406, in _rmtree_safe_fd
    dirfd = os.open(name, os.O_RDONLY, dir_fd=topfd)
PermissionError: [Errno 13] Permission denied: 'SUB21'

serhiy-storchaka · 2016-10-28T06:00:30Z

Hmm, tests were passed when I ran them before committing. Seems this is heisenbug.

zhangyangyu · 2016-10-28T06:14:27Z

Seems like the dir you add breaks other cases. I change addCleanUp to os.chmod and get:

======================================================================
FAIL: test_file_like_path (test.test_os.BytesWalkTests)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/home/angwer/cpython/Lib/test/test_os.py", line 950, in test_file_like_path
    self.test_walk_prune(_PathLike(self.walk_path))
  File "/home/angwer/cpython/Lib/test/test_os.py", line 941, in test_walk_prune
    self.assertEqual(len(all), 2)
AssertionError: 3 != 2

======================================================================
FAIL: test_walk_bottom_up (test.test_os.BytesWalkTests)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/home/angwer/cpython/Lib/test/test_os.py", line 956, in test_walk_bottom_up
    self.assertEqual(len(all), 4, all)
AssertionError: 5 != 4 : [('@test_24702_tmp/TEST1/SUB2/SUB21', [], ['tmp3']), ('@test_24702_tmp/TEST1/SUB2', ['link', 'SUB21'], ['tmp3', 'broken_link3', 'broken_link', 'broken_link2']), ('@test_24702_tmp/TEST1/SUB1/SUB11', [], []), ('@test_24702_tmp/TEST1/SUB1', ['SUB11'], ['tmp2']), ('@test_24702_tmp/TEST1', ['SUB2', 'SUB1'], ['tmp1'])]

======================================================================
FAIL: test_walk_prune (test.test_os.BytesWalkTests)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/home/angwer/cpython/Lib/test/test_os.py", line 941, in test_walk_prune
    self.assertEqual(len(all), 2)
AssertionError: 3 != 2

======================================================================
FAIL: test_walk_topdown (test.test_os.BytesWalkTests)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/home/angwer/cpython/Lib/test/test_os.py", line 916, in test_walk_topdown
    self.assertEqual(len(all), 4)
AssertionError: 5 != 4

======================================================================
FAIL: test_file_like_path (test.test_os.FwalkTests)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/home/angwer/cpython/Lib/test/test_os.py", line 950, in test_file_like_path
    self.test_walk_prune(_PathLike(self.walk_path))
  File "/home/angwer/cpython/Lib/test/test_os.py", line 941, in test_walk_prune
    self.assertEqual(len(all), 2)
AssertionError: 3 != 2

======================================================================
FAIL: test_walk_bottom_up (test.test_os.FwalkTests)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/home/angwer/cpython/Lib/test/test_os.py", line 956, in test_walk_bottom_up
    self.assertEqual(len(all), 4, all)
AssertionError: 5 != 4 : [('@test_24702_tmp/TEST1/SUB2/SUB21', [], ['tmp3']), ('@test_24702_tmp/TEST1/SUB2', ['link', 'SUB21'], ['tmp3', 'broken_link3', 'broken_link', 'broken_link2']), ('@test_24702_tmp/TEST1/SUB1/SUB11', [], []), ('@test_24702_tmp/TEST1/SUB1', ['SUB11'], ['tmp2']), ('@test_24702_tmp/TEST1', ['SUB2', 'SUB1'], ['tmp1'])]

======================================================================
FAIL: test_walk_prune (test.test_os.FwalkTests)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/home/angwer/cpython/Lib/test/test_os.py", line 941, in test_walk_prune
    self.assertEqual(len(all), 2)
AssertionError: 3 != 2

======================================================================
FAIL: test_walk_topdown (test.test_os.FwalkTests)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/home/angwer/cpython/Lib/test/test_os.py", line 916, in test_walk_topdown
    self.assertEqual(len(all), 4)
AssertionError: 5 != 4

======================================================================
FAIL: test_file_like_path (test.test_os.WalkTests)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/home/angwer/cpython/Lib/test/test_os.py", line 950, in test_file_like_path
    self.test_walk_prune(_PathLike(self.walk_path))
  File "/home/angwer/cpython/Lib/test/test_os.py", line 941, in test_walk_prune
    self.assertEqual(len(all), 2)
AssertionError: 3 != 2

======================================================================
FAIL: test_walk_bottom_up (test.test_os.WalkTests)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/home/angwer/cpython/Lib/test/test_os.py", line 956, in test_walk_bottom_up
    self.assertEqual(len(all), 4, all)
AssertionError: 5 != 4 : [('@test_24702_tmp/TEST1/SUB2/SUB21', [], ['tmp3']), ('@test_24702_tmp/TEST1/SUB2', ['link', 'SUB21'], ['tmp3', 'broken_link3', 'broken_link', 'broken_link2']), ('@test_24702_tmp/TEST1/SUB1/SUB11', [], []), ('@test_24702_tmp/TEST1/SUB1', ['SUB11'], ['tmp2']), ('@test_24702_tmp/TEST1', ['SUB2', 'SUB1'], ['tmp1'])]

======================================================================
FAIL: test_walk_prune (test.test_os.WalkTests)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/home/angwer/cpython/Lib/test/test_os.py", line 941, in test_walk_prune
    self.assertEqual(len(all), 2)
AssertionError: 3 != 2

======================================================================
FAIL: test_walk_topdown (test.test_os.WalkTests)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/home/angwer/cpython/Lib/test/test_os.py", line 916, in test_walk_topdown
    self.assertEqual(len(all), 4)
AssertionError: 5 != 4

python-dev · 2016-10-28T06:19:47Z

New changeset 655510dd46fd by Serhiy Storchaka in branch '3.5':
Issue bpo-28353: Make test_os.WalkTests.test_walk_bad_dir stable.
https://hg.python.org/cpython/rev/655510dd46fd

New changeset df28e536f19d by Serhiy Storchaka in branch '3.6':
Issue bpo-28353: Make test_os.WalkTests.test_walk_bad_dir stable.
https://hg.python.org/cpython/rev/df28e536f19d

New changeset aa891d13e1cf by Serhiy Storchaka in branch 'default':
Issue bpo-28353: Make test_os.WalkTests.test_walk_bad_dir stable.
https://hg.python.org/cpython/rev/aa891d13e1cf

serhiy-storchaka · 2016-10-28T06:23:37Z

test_walk_bad_dir renamed one of subdirs and tearDown failed to clean up changed tree. An error happened depending on the order of listed subdirectories: ['SUB1', 'SUB2'] -- passed, ['SUB2', 'SUB1'] -- failed.

zhangyangyu · 2016-10-28T06:26:12Z

It works. Close. :-)

SamsonLee mannequin added the stdlib Python modules in the Lib dir label Oct 4, 2016

serhiy-storchaka added 3.7 (EOL) end of life type-bug An unexpected behavior, bug, or error labels Oct 4, 2016

serhiy-storchaka self-assigned this Oct 25, 2016

serhiy-storchaka closed this as completed Oct 26, 2016

zhangyangyu reopened this Oct 28, 2016

zhangyangyu closed this as completed Oct 28, 2016

ezio-melotti transferred this issue from another repository Apr 10, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

os.fwalk() unhandled exception when error occurs accessing symbolic link target #72539

os.fwalk() unhandled exception when error occurs accessing symbolic link target #72539

SamsonLee mannequin commented Oct 4, 2016

SamsonLee mannequin commented Oct 4, 2016

SamsonLee mannequin commented Oct 4, 2016

serhiy-storchaka commented Oct 4, 2016

SamsonLee mannequin commented Oct 4, 2016

serhiy-storchaka commented Oct 4, 2016

SamsonLee mannequin commented Oct 5, 2016

serhiy-storchaka commented Oct 6, 2016

python-dev mannequin commented Oct 25, 2016

python-dev mannequin commented Oct 25, 2016

python-dev mannequin commented Oct 25, 2016

zhangyangyu commented Oct 28, 2016

serhiy-storchaka commented Oct 28, 2016

zhangyangyu commented Oct 28, 2016

python-dev mannequin commented Oct 28, 2016

serhiy-storchaka commented Oct 28, 2016

zhangyangyu commented Oct 28, 2016

os.fwalk() unhandled exception when error occurs accessing symbolic link target #72539

os.fwalk() unhandled exception when error occurs accessing symbolic link target #72539

Comments

SamsonLee mannequin commented Oct 4, 2016

SamsonLee mannequin commented Oct 4, 2016

SamsonLee mannequin commented Oct 4, 2016

serhiy-storchaka commented Oct 4, 2016

SamsonLee mannequin commented Oct 4, 2016

serhiy-storchaka commented Oct 4, 2016

SamsonLee mannequin commented Oct 5, 2016

serhiy-storchaka commented Oct 6, 2016

python-dev mannequin commented Oct 25, 2016

python-dev mannequin commented Oct 25, 2016

python-dev mannequin commented Oct 25, 2016

zhangyangyu commented Oct 28, 2016

serhiy-storchaka commented Oct 28, 2016

zhangyangyu commented Oct 28, 2016

python-dev mannequin commented Oct 28, 2016

serhiy-storchaka commented Oct 28, 2016

zhangyangyu commented Oct 28, 2016