Issue28353
This issue tracker has been migrated to GitHub,
and is currently read-only.
For more information,
see the GitHub FAQs in the Python's Developer Guide.
Created on 2016-10-04 07:06 by Samson Lee, last changed 2022-04-11 14:58 by admin. This issue is now closed.
Files | ||||
---|---|---|---|---|
File name | Uploaded | Description | Edit | |
fwalk_oserror.patch | serhiy.storchaka, 2016-10-06 07:15 | review |
Pull Requests | |||
---|---|---|---|
URL | Status | Linked | Edit |
PR 552 | closed | dstufft, 2017-03-31 16:36 |
Messages (16) | |||
---|---|---|---|
msg278016 - (view) | Author: Samson Lee (Samson Lee) | Date: 2016-10-04 07:06 | |
The bug is os.fwalk() crashes with unhandled exception when there is an error accessing symbolic link targets. To reproduce the bug, create a symbolic link that targets a file that you do not have permission to access: $ touch handsoff $ sudo chown root:root handsoff $ sudo chmod 700 handsoff $ ln -s handsoff blah Now, os.fwalk() fails: >>> for root, dirs, files, fd in os.fwalk('.'): ... print(root, dirs, files) ... Traceback (most recent call last): File "test_fwalk_permission_error.py", line 3, in <module> for root, dirs, files, fd in os.fwalk('.'): File "/usr/lib64/python3.5/os.py", line 520, in fwalk yield from _fwalk(topfd, top, topdown, onerror, follow_symlinks) File "/usr/lib64/python3.5/os.py", line 537, in _fwalk if st.S_ISDIR(stat(name, dir_fd=topfd).st_mode): PermissionError: [Errno 13] Permission denied: 'blah' The cause of the problem is in this part of os.py: for name in names: try: # Here, we don't use AT_SYMLINK_NOFOLLOW to be consistent with # walk() which reports symlinks to directories as directories. # We do however check for symlinks before recursing into # a subdirectory. if st.S_ISDIR(stat(name, dir_fd=topfd).st_mode): dirs.append(name) else: nondirs.append(name) except FileNotFoundError: try: # Add dangling symlinks, ignore disappeared files if st.S_ISLNK(stat(name, dir_fd=topfd, follow_symlinks=False) .st_mode): nondirs.append(name) except FileNotFoundError: continue To fix it, simply replace FileNotFoundError with more general OSError. Cheers |
|||
msg278017 - (view) | Author: Samson Lee (Samson Lee) | Date: 2016-10-04 07:08 | |
Similar to http://bugs.python.org/issue25860 |
|||
msg278018 - (view) | Author: Serhiy Storchaka (serhiy.storchaka) * | Date: 2016-10-04 07:45 | |
I can't reproduce the issue. $ mkdir testdir $ touch testdir/handsoff $ sudo chown root:root testdir/handsoff $ sudo chmod 700 testdir/handsoff $ ln -s handsoff testdir/blah $ python3.5 >>> import os >>> list(os.walk('testdir')) [('testdir', [], ['handsoff', 'blah'])] >>> list(os.fwalk('testdir')) [('testdir', [], ['handsoff', 'blah'], 3)] Ubuntu 16.04, Linux 4.4, tested on ext4 and tmpfs filesystems. |
|||
msg278024 - (view) | Author: Samson Lee (Samson Lee) | Date: 2016-10-04 09:07 | |
Sorry, the target file needs to be in a directory you don't have permission to access: $ mkdir handsoff $ sudo chown root:root handsoff $ sudo chmod 700 handsoff $ ln -s handsoff/anything blah |
|||
msg278044 - (view) | Author: Serhiy Storchaka (serhiy.storchaka) * | Date: 2016-10-04 14:45 | |
Yes, there is a difference between os.walk() and os.fwalk() in some cases. I don't know whether it was introduced deliberately or by accident. os.fwalk() was added in issue13734, FileNotFoundError is used since issue14773. |
|||
msg278118 - (view) | Author: Samson Lee (Samson Lee) | Date: 2016-10-05 10:49 | |
This is definitely a bug because there is no way an unhandled exception like this is acceptable behaviour. The behaviour is that the fwalk iteration will stop and there is no way to recover / continue. Also fwalk takes an onerror callback function that won't be called with this error. |
|||
msg278178 - (view) | Author: Serhiy Storchaka (serhiy.storchaka) * | Date: 2016-10-06 07:15 | |
Here is a patch with tests. Needed to test it on Windows. |
|||
msg279393 - (view) | Author: Roundup Robot (python-dev) | Date: 2016-10-25 11:35 | |
New changeset fe1fea4ded04 by Serhiy Storchaka in branch '3.5': Issue #28353: os.fwalk() no longer fails on broken links. https://hg.python.org/cpython/rev/fe1fea4ded04 New changeset e99ec3c77a63 by Serhiy Storchaka in branch '3.6': Issue #28353: os.fwalk() no longer fails on broken links. https://hg.python.org/cpython/rev/e99ec3c77a63 New changeset 33a1a3dd0051 by Serhiy Storchaka in branch 'default': Issue #28353: os.fwalk() no longer fails on broken links. https://hg.python.org/cpython/rev/33a1a3dd0051 |
|||
msg279418 - (view) | Author: Roundup Robot (python-dev) | Date: 2016-10-25 16:03 | |
New changeset ec12e16ea6a1 by Serhiy Storchaka in branch '3.6': Issue #28353: Try to fix tests. https://hg.python.org/cpython/rev/ec12e16ea6a1 New changeset a0913dbadea6 by Serhiy Storchaka in branch 'default': Issue #28353: Try to fix tests. https://hg.python.org/cpython/rev/a0913dbadea6 |
|||
msg279422 - (view) | Author: Roundup Robot (python-dev) | Date: 2016-10-25 17:20 | |
New changeset 470224ec16b6 by Serhiy Storchaka in branch '3.5': Issue #28353: Fixed tests of os.fwalk() with broken links. https://hg.python.org/cpython/rev/470224ec16b6 |
|||
msg279581 - (view) | Author: Xiang Zhang (xiang.zhang) * | Date: 2016-10-28 04:54 | |
Serhiy, after your commits, test_os requires root privileges or it'll fail. This is not the case before. [cpython]$ ./python -m test test_os Run tests sequentially 0:00:00 [1/1] test_os test test_os crashed -- Traceback (most recent call last): File "/home/angwer/cpython/Lib/test/libregrtest/runtest.py", line 164, in runtest_inner test_runner() File "/home/angwer/cpython/Lib/test/libregrtest/runtest.py", line 163, in test_runner support.run_unittest(tests) File "/home/angwer/cpython/Lib/test/support/__init__.py", line 1849, in run_unittest _run_suite(suite) File "/home/angwer/cpython/Lib/test/support/__init__.py", line 1824, in _run_suite raise TestFailed(err) test.support.TestFailed: multiple errors occurred; run in verbose mode for details During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/angwer/cpython/Lib/test/libregrtest/runtest.py", line 167, in runtest_inner test_time = time.time() - start_time File "/home/angwer/cpython/Lib/test/libregrtest/save_env.py", line 278, in __exit__ restore(original) File "/home/angwer/cpython/Lib/test/libregrtest/save_env.py", line 235, in restore_files support.rmtree(fn) File "/home/angwer/cpython/Lib/test/support/__init__.py", line 377, in rmtree _rmtree(path) File "/home/angwer/cpython/Lib/shutil.py", line 474, in rmtree _rmtree_safe_fd(fd, path, onerror) File "/home/angwer/cpython/Lib/shutil.py", line 412, in _rmtree_safe_fd _rmtree_safe_fd(dirfd, fullname, onerror) File "/home/angwer/cpython/Lib/shutil.py", line 412, in _rmtree_safe_fd _rmtree_safe_fd(dirfd, fullname, onerror) File "/home/angwer/cpython/Lib/shutil.py", line 408, in _rmtree_safe_fd onerror(os.open, fullname, sys.exc_info()) File "/home/angwer/cpython/Lib/shutil.py", line 406, in _rmtree_safe_fd dirfd = os.open(name, os.O_RDONLY, dir_fd=topfd) PermissionError: [Errno 13] Permission denied: 'SUB21' 'test_os' left behind directory '@test_23400_tmp' and it couldn't be removed: [Errno 13] Permission denied: 'SUB21' test_os failed 1 test failed: test_os Total duration: 292 ms Tests result: FAILURE Traceback (most recent call last): File "/home/angwer/cpython/Lib/test/support/__init__.py", line 914, in temp_dir yield path File "/home/angwer/cpython/Lib/test/support/__init__.py", line 963, in temp_cwd yield cwd_dir File "/home/angwer/cpython/Lib/test/libregrtest/main.py", line 468, in main self._main(tests, kwargs) File "/home/angwer/cpython/Lib/test/libregrtest/main.py", line 497, in _main sys.exit(len(self.bad) > 0 or self.interrupted) SystemExit: True During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/angwer/cpython/Lib/runpy.py", line 193, in _run_module_as_main "__main__", mod_spec) File "/home/angwer/cpython/Lib/runpy.py", line 85, in _run_code exec(code, run_globals) File "/home/angwer/cpython/Lib/test/__main__.py", line 2, in <module> main() File "/home/angwer/cpython/Lib/test/libregrtest/main.py", line 532, in main Regrtest().main(tests=tests, **kwargs) File "/home/angwer/cpython/Lib/test/libregrtest/main.py", line 468, in main self._main(tests, kwargs) File "/home/angwer/cpython/Lib/contextlib.py", line 100, in __exit__ self.gen.throw(type, value, traceback) File "/home/angwer/cpython/Lib/test/support/__init__.py", line 963, in temp_cwd yield cwd_dir File "/home/angwer/cpython/Lib/contextlib.py", line 100, in __exit__ self.gen.throw(type, value, traceback) File "/home/angwer/cpython/Lib/test/support/__init__.py", line 917, in temp_dir rmtree(path) File "/home/angwer/cpython/Lib/test/support/__init__.py", line 377, in rmtree _rmtree(path) File "/home/angwer/cpython/Lib/shutil.py", line 474, in rmtree _rmtree_safe_fd(fd, path, onerror) File "/home/angwer/cpython/Lib/shutil.py", line 412, in _rmtree_safe_fd _rmtree_safe_fd(dirfd, fullname, onerror) File "/home/angwer/cpython/Lib/shutil.py", line 412, in _rmtree_safe_fd _rmtree_safe_fd(dirfd, fullname, onerror) File "/home/angwer/cpython/Lib/shutil.py", line 412, in _rmtree_safe_fd _rmtree_safe_fd(dirfd, fullname, onerror) File "/home/angwer/cpython/Lib/shutil.py", line 408, in _rmtree_safe_fd onerror(os.open, fullname, sys.exc_info()) File "/home/angwer/cpython/Lib/shutil.py", line 406, in _rmtree_safe_fd dirfd = os.open(name, os.O_RDONLY, dir_fd=topfd) PermissionError: [Errno 13] Permission denied: 'SUB21' |
|||
msg279582 - (view) | Author: Serhiy Storchaka (serhiy.storchaka) * | Date: 2016-10-28 06:00 | |
Hmm, tests were passed when I ran them before committing. Seems this is heisenbug. |
|||
msg279583 - (view) | Author: Xiang Zhang (xiang.zhang) * | Date: 2016-10-28 06:14 | |
Seems like the dir you add breaks other cases. I change addCleanUp to os.chmod and get: ====================================================================== FAIL: test_file_like_path (test.test_os.BytesWalkTests) ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/angwer/cpython/Lib/test/test_os.py", line 950, in test_file_like_path self.test_walk_prune(_PathLike(self.walk_path)) File "/home/angwer/cpython/Lib/test/test_os.py", line 941, in test_walk_prune self.assertEqual(len(all), 2) AssertionError: 3 != 2 ====================================================================== FAIL: test_walk_bottom_up (test.test_os.BytesWalkTests) ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/angwer/cpython/Lib/test/test_os.py", line 956, in test_walk_bottom_up self.assertEqual(len(all), 4, all) AssertionError: 5 != 4 : [('@test_24702_tmp/TEST1/SUB2/SUB21', [], ['tmp3']), ('@test_24702_tmp/TEST1/SUB2', ['link', 'SUB21'], ['tmp3', 'broken_link3', 'broken_link', 'broken_link2']), ('@test_24702_tmp/TEST1/SUB1/SUB11', [], []), ('@test_24702_tmp/TEST1/SUB1', ['SUB11'], ['tmp2']), ('@test_24702_tmp/TEST1', ['SUB2', 'SUB1'], ['tmp1'])] ====================================================================== FAIL: test_walk_prune (test.test_os.BytesWalkTests) ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/angwer/cpython/Lib/test/test_os.py", line 941, in test_walk_prune self.assertEqual(len(all), 2) AssertionError: 3 != 2 ====================================================================== FAIL: test_walk_topdown (test.test_os.BytesWalkTests) ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/angwer/cpython/Lib/test/test_os.py", line 916, in test_walk_topdown self.assertEqual(len(all), 4) AssertionError: 5 != 4 ====================================================================== FAIL: test_file_like_path (test.test_os.FwalkTests) ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/angwer/cpython/Lib/test/test_os.py", line 950, in test_file_like_path self.test_walk_prune(_PathLike(self.walk_path)) File "/home/angwer/cpython/Lib/test/test_os.py", line 941, in test_walk_prune self.assertEqual(len(all), 2) AssertionError: 3 != 2 ====================================================================== FAIL: test_walk_bottom_up (test.test_os.FwalkTests) ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/angwer/cpython/Lib/test/test_os.py", line 956, in test_walk_bottom_up self.assertEqual(len(all), 4, all) AssertionError: 5 != 4 : [('@test_24702_tmp/TEST1/SUB2/SUB21', [], ['tmp3']), ('@test_24702_tmp/TEST1/SUB2', ['link', 'SUB21'], ['tmp3', 'broken_link3', 'broken_link', 'broken_link2']), ('@test_24702_tmp/TEST1/SUB1/SUB11', [], []), ('@test_24702_tmp/TEST1/SUB1', ['SUB11'], ['tmp2']), ('@test_24702_tmp/TEST1', ['SUB2', 'SUB1'], ['tmp1'])] ====================================================================== FAIL: test_walk_prune (test.test_os.FwalkTests) ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/angwer/cpython/Lib/test/test_os.py", line 941, in test_walk_prune self.assertEqual(len(all), 2) AssertionError: 3 != 2 ====================================================================== FAIL: test_walk_topdown (test.test_os.FwalkTests) ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/angwer/cpython/Lib/test/test_os.py", line 916, in test_walk_topdown self.assertEqual(len(all), 4) AssertionError: 5 != 4 ====================================================================== FAIL: test_file_like_path (test.test_os.WalkTests) ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/angwer/cpython/Lib/test/test_os.py", line 950, in test_file_like_path self.test_walk_prune(_PathLike(self.walk_path)) File "/home/angwer/cpython/Lib/test/test_os.py", line 941, in test_walk_prune self.assertEqual(len(all), 2) AssertionError: 3 != 2 ====================================================================== FAIL: test_walk_bottom_up (test.test_os.WalkTests) ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/angwer/cpython/Lib/test/test_os.py", line 956, in test_walk_bottom_up self.assertEqual(len(all), 4, all) AssertionError: 5 != 4 : [('@test_24702_tmp/TEST1/SUB2/SUB21', [], ['tmp3']), ('@test_24702_tmp/TEST1/SUB2', ['link', 'SUB21'], ['tmp3', 'broken_link3', 'broken_link', 'broken_link2']), ('@test_24702_tmp/TEST1/SUB1/SUB11', [], []), ('@test_24702_tmp/TEST1/SUB1', ['SUB11'], ['tmp2']), ('@test_24702_tmp/TEST1', ['SUB2', 'SUB1'], ['tmp1'])] ====================================================================== FAIL: test_walk_prune (test.test_os.WalkTests) ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/angwer/cpython/Lib/test/test_os.py", line 941, in test_walk_prune self.assertEqual(len(all), 2) AssertionError: 3 != 2 ====================================================================== FAIL: test_walk_topdown (test.test_os.WalkTests) ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/angwer/cpython/Lib/test/test_os.py", line 916, in test_walk_topdown self.assertEqual(len(all), 4) AssertionError: 5 != 4 |
|||
msg279584 - (view) | Author: Roundup Robot (python-dev) | Date: 2016-10-28 06:19 | |
New changeset 655510dd46fd by Serhiy Storchaka in branch '3.5': Issue #28353: Make test_os.WalkTests.test_walk_bad_dir stable. https://hg.python.org/cpython/rev/655510dd46fd New changeset df28e536f19d by Serhiy Storchaka in branch '3.6': Issue #28353: Make test_os.WalkTests.test_walk_bad_dir stable. https://hg.python.org/cpython/rev/df28e536f19d New changeset aa891d13e1cf by Serhiy Storchaka in branch 'default': Issue #28353: Make test_os.WalkTests.test_walk_bad_dir stable. https://hg.python.org/cpython/rev/aa891d13e1cf |
|||
msg279585 - (view) | Author: Serhiy Storchaka (serhiy.storchaka) * | Date: 2016-10-28 06:23 | |
test_walk_bad_dir renamed one of subdirs and tearDown failed to clean up changed tree. An error happened depending on the order of listed subdirectories: ['SUB1', 'SUB2'] -- passed, ['SUB2', 'SUB1'] -- failed. |
|||
msg279586 - (view) | Author: Xiang Zhang (xiang.zhang) * | Date: 2016-10-28 06:26 | |
It works. Close. :-) |
History | |||
---|---|---|---|
Date | User | Action | Args |
2022-04-11 14:58:37 | admin | set | github: 72539 |
2017-03-31 16:36:09 | dstufft | set | pull_requests: + pull_request851 |
2016-10-28 06:26:12 | xiang.zhang | set | status: open -> closed messages: + msg279586 |
2016-10-28 06:23:36 | serhiy.storchaka | set | messages: + msg279585 |
2016-10-28 06:19:47 | python-dev | set | messages: + msg279584 |
2016-10-28 06:14:27 | xiang.zhang | set | messages: + msg279583 |
2016-10-28 06:00:30 | serhiy.storchaka | set | messages: + msg279582 |
2016-10-28 04:54:54 | xiang.zhang | set | status: closed -> open nosy: + xiang.zhang messages: + msg279581 |
2016-10-26 08:18:07 | serhiy.storchaka | set | status: open -> closed resolution: fixed stage: patch review -> resolved |
2016-10-25 17:20:52 | python-dev | set | messages: + msg279422 |
2016-10-25 16:03:55 | python-dev | set | messages: + msg279418 |
2016-10-25 11:35:35 | python-dev | set | nosy:
+ python-dev messages: + msg279393 |
2016-10-25 10:50:34 | serhiy.storchaka | set | assignee: serhiy.storchaka |
2016-10-06 07:15:33 | serhiy.storchaka | set | files:
+ fwalk_oserror.patch keywords: + patch |
2016-10-06 07:15:05 | serhiy.storchaka | set | messages:
+ msg278178 stage: patch review |
2016-10-05 10:49:35 | Samson Lee | set | messages: + msg278118 |
2016-10-05 04:50:20 | Mariatta | set | nosy:
+ Mariatta |
2016-10-04 14:45:16 | serhiy.storchaka | set | versions:
+ Python 3.7 nosy: + loewis, neologix, hynek messages: + msg278044 type: behavior |
2016-10-04 09:07:52 | Samson Lee | set | messages: + msg278024 |
2016-10-04 07:45:40 | serhiy.storchaka | set | messages: + msg278018 |
2016-10-04 07:08:56 | Samson Lee | set | nosy:
+ serhiy.storchaka messages: + msg278017 |
2016-10-04 07:06:56 | Samson Lee | create |