Issue26012
This issue tracker has been migrated to GitHub,
and is currently read-only.
For more information,
see the GitHub FAQs in the Python's Developer Guide.
Created on 2016-01-05 02:13 by gvanrossum, last changed 2022-04-11 14:58 by admin. This issue is now closed.
Messages (14) | |||
---|---|---|---|
msg257511 - (view) | Author: Guido van Rossum (gvanrossum) * ![]() |
Date: 2016-01-05 02:13 | |
I created a symlink loop as follows: mkdir tmp cd tmp ln -s ../tmp baz cd .. Then I tried to list it recursively using rglob(): >>> list(pathlib.Path('tmp').rglob('**/*') This caused an infinite regress: Traceback (most recent call last): File "<string>", line 1, in <module> File "/Users/guido/src/cpython36/Lib/pathlib.py", line 1065, in rglob for p in selector.select_from(self): File "/Users/guido/src/cpython36/Lib/pathlib.py", line 549, in _select_from for p in successor_select(starting_point, is_dir, exists, listdir): File "/Users/guido/src/cpython36/Lib/pathlib.py", line 548, in _select_from for starting_point in self._iterate_directories(parent_path, is_dir, listdir): File "/Users/guido/src/cpython36/Lib/pathlib.py", line 538, in _iterate_directories for p in self._iterate_directories(path, is_dir, listdir): [...] File "/Users/guido/src/cpython36/Lib/pathlib.py", line 538, in _iterate_directories for p in self._iterate_directories(path, is_dir, listdir): File "/Users/guido/src/cpython36/Lib/pathlib.py", line 537, in _iterate_directories if is_dir(path): File "/Users/guido/src/cpython36/Lib/pathlib.py", line 1303, in is_dir return S_ISDIR(self.stat().st_mode) File "/Users/guido/src/cpython36/Lib/pathlib.py", line 1111, in stat return self._accessor.stat(self) File "/Users/guido/src/cpython36/Lib/pathlib.py", line 371, in wrapped return strfunc(str(pathobj), *args) OSError: [Errno 62] Too many levels of symbolic links: 'tmp/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz' |
|||
msg257516 - (view) | Author: Josh Rosenberg (josh.r) * ![]() |
Date: 2016-01-05 05:25 | |
So would the goal of a fix be something like dev/inode memoization to prevent traversing the same link twice? To provide an argument to prevent following symlinks at all (as in stuff like os.walk), possibly defaulting to followlinks=False? It looks like languages like Ruby never follow symlinks when performing recursive globs to avoid this issue ( https://stackoverflow.com/questions/357754/can-i-traverse-symlinked-directories-in-ruby-with-a-glob ). It's probably easiest to just block recursing through symlinks (at least by default); memoizing risks unbounded memory overhead for large directory trees. |
|||
msg257519 - (view) | Author: Serhiy Storchaka (serhiy.storchaka) * ![]() |
Date: 2016-01-05 06:21 | |
glob.glob() just stops too deep recursion. >>> import glob >>> glob.glob('tmp/**/*') ['tmp/baz/baz'] >>> glob.glob('tmp/**/*', recursive=True) ['tmp/baz', 'tmp/baz/baz', 'tmp/baz/baz/baz', 'tmp/baz/baz/baz/baz', 'tmp/baz/baz/baz/baz/baz', 'tmp/baz/baz/baz/baz/baz/baz', 'tmp/baz/baz/baz/baz/baz/baz/baz', 'tmp/baz/baz/baz/baz/baz/baz/baz/baz', 'tmp/baz/baz/baz/baz/baz/baz/baz/baz/baz', 'tmp/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz', 'tmp/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz', 'tmp/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz', 'tmp/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz', 'tmp/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz', 'tmp/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz', 'tmp/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz', 'tmp/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz', 'tmp/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz', 'tmp/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz', 'tmp/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz', 'tmp/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz', 'tmp/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz', 'tmp/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz', 'tmp/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz', 'tmp/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz', 'tmp/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz', 'tmp/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz', 'tmp/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz', 'tmp/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz', 'tmp/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz', 'tmp/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz', 'tmp/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz', 'tmp/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz', 'tmp/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz', 'tmp/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz', 'tmp/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz', 'tmp/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz', 'tmp/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz', 'tmp/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz', 'tmp/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz', 'tmp/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz/baz'] |
|||
msg257522 - (view) | Author: Guido van Rossum (gvanrossum) * ![]() |
Date: 2016-01-05 07:03 | |
I agree it's easiest just not to traverse symlinks matching **. glob.py avoids the errors (but not the senseless recursion) by simply ignoring OSError coming out of listdir(). That might be a good idea anyways (it might even be a simpler way to avoid the PermissionError reported in #24120). |
|||
msg257620 - (view) | Author: Guido van Rossum (gvanrossum) * ![]() |
Date: 2016-01-06 17:45 | |
I'm going to fix this by skipping all symlinks in _RecursiveWildcardSelector. |
|||
msg257628 - (view) | Author: Roundup Robot (python-dev) ![]() |
Date: 2016-01-06 18:36 | |
New changeset 18f5b125a863 by Guido van Rossum in branch '3.4': Issue #26012: Don't traverse into symlinks for ** pattern in pathlib.Path.[r]glob(). https://hg.python.org/cpython/rev/18f5b125a863 New changeset 9826dbad1252 by Guido van Rossum in branch '3.5': Issue #26012: Don't traverse into symlinks for ** pattern in pathlib.Path.[r]glob(). (Merge 3.4->3.5) https://hg.python.org/cpython/rev/9826dbad1252 New changeset 36864abbfe02 by Guido van Rossum in branch 'default': Issue #26012: Don't traverse into symlinks for ** pattern in pathlib.Path.[r]glob(). (Merge 3.5->3.6) https://hg.python.org/cpython/rev/36864abbfe02 |
|||
msg257683 - (view) | Author: Martin Panter (martin.panter) * ![]() |
Date: 2016-01-07 11:43 | |
This seems to have broken test_pathlib, both on my computer and some Linux and BSD buildbots: ====================================================================== FAIL: test_rglob_symlink_loop (test.test_pathlib.PathTest) ---------------------------------------------------------------------- Traceback (most recent call last): File "/media/disk/home/proj/python/cpython/Lib/test/test_pathlib.py", line 1455, in test_rglob_symlink_loop self.assertEqual(given, {p / x for x in expect}) AssertionError: Items in the second set but not the first: PosixPath('/media/disk/home/proj/python/cpython/build/test_python_27756/@test_27756_tmp/dirA/linkC') PosixPath('/media/disk/home/proj/python/cpython/build/test_python_27756/@test_27756_tmp/dirB/fileB') PosixPath('/media/disk/home/proj/python/cpython/build/test_python_27756/@test_27756_tmp/dirB/linkD') ====================================================================== FAIL: test_rglob_symlink_loop (test.test_pathlib.PosixPathTest) ---------------------------------------------------------------------- Traceback (most recent call last): File "/media/disk/home/proj/python/cpython/Lib/test/test_pathlib.py", line 1455, in test_rglob_symlink_loop self.assertEqual(given, {p / x for x in expect}) AssertionError: Items in the second set but not the first: PosixPath('/media/disk/home/proj/python/cpython/build/test_python_27756/@test_27756_tmp/dirA/linkC') PosixPath('/media/disk/home/proj/python/cpython/build/test_python_27756/@test_27756_tmp/dirB/fileB') PosixPath('/media/disk/home/proj/python/cpython/build/test_python_27756/@test_27756_tmp/dirB/linkD') |
|||
msg257699 - (view) | Author: Guido van Rossum (gvanrossum) * ![]() |
Date: 2016-01-07 16:44 | |
Oops, sorry about that. I will see if I can figure out how to repro this -- on my own Mac it succeeds. In the mean time if you could have a look yourself (since you can repro it on your own computer) I'd be grateful. Initial hunches: maybe a case-insensitivity crept into the test? |
|||
msg257708 - (view) | Author: Guido van Rossum (gvanrossum) * ![]() |
Date: 2016-01-07 18:44 | |
Looks like the root cause of the breakage is actually that issue #24120 isn't quite fixed yet. I'm now able to repro on a Linux VM. |
|||
msg257711 - (view) | Author: Roundup Robot (python-dev) ![]() |
Date: 2016-01-07 18:58 | |
New changeset 4043e08e6e52 by Guido van Rossum in branch '3.4': Add another try/except PermissionError to avoid depending on listdir order. Fix issues #24120 and #26012. https://hg.python.org/cpython/rev/4043e08e6e52 New changeset 8a3b0c1fb3d3 by Guido van Rossum in branch '3.5': Add another try/except PermissionError to avoid depending on listdir order. Fix issues #24120 and #26012. (Merge 3.4->3.5) https://hg.python.org/cpython/rev/8a3b0c1fb3d3 New changeset 398cb8c183da by Guido van Rossum in branch 'default': Add another try/except PermissionError to avoid depending on listdir order. Fix issues #24120 and #26012. (Merge 3.5->3.6) https://hg.python.org/cpython/rev/398cb8c183da |
|||
msg257712 - (view) | Author: Guido van Rossum (gvanrossum) * ![]() |
Date: 2016-01-07 18:59 | |
Should be fixed for real now. |
|||
msg257717 - (view) | Author: Martin Panter (martin.panter) * ![]() |
Date: 2016-01-07 21:15 | |
Thanks, your latest change seems to have fixed it on my Linux computer, and most of the buildbots. However now there is a failure on <http://buildbot.python.org/all/builders/AMD64%20Windows7%20SP1%203.x>: ====================================================================== FAIL: test_rglob_common (test.test_pathlib.PathTest) ---------------------------------------------------------------------- Traceback (most recent call last): File "C:\buildbot.python.org\3.x.kloth-win64\build\lib\test\test_pathlib.py", line 1460, in test_rglob_common "linkB/fileB", "dirA/linkC/fileB"]) File "C:\buildbot.python.org\3.x.kloth-win64\build\lib\test\test_pathlib.py", line 1451, in _check self.assertEqual(set(glob), { P(BASE, q) for q in expected }) AssertionError: Items in the second set but not the first: WindowsPath('C:/buildbot.python.org/3.x.kloth-win64/build/build/test_python_1700/@test_1700_tmp/linkB/fileB') WindowsPath('C:/buildbot.python.org/3.x.kloth-win64/build/build/test_python_1700/@test_1700_tmp/dirB/linkD/fileB') WindowsPath('C:/buildbot.python.org/3.x.kloth-win64/build/build/test_python_1700/@test_1700_tmp/dirA/linkC/fileB') ====================================================================== FAIL: test_rglob_common (test.test_pathlib.WindowsPathTest) ---------------------------------------------------------------------- Traceback (most recent call last): File "C:\buildbot.python.org\3.x.kloth-win64\build\lib\test\test_pathlib.py", line 1460, in test_rglob_common "linkB/fileB", "dirA/linkC/fileB"]) File "C:\buildbot.python.org\3.x.kloth-win64\build\lib\test\test_pathlib.py", line 1451, in _check self.assertEqual(set(glob), { P(BASE, q) for q in expected }) AssertionError: Items in the second set but not the first: WindowsPath('C:/buildbot.python.org/3.x.kloth-win64/build/build/test_python_1700/@test_1700_tmp/linkB/fileB') WindowsPath('C:/buildbot.python.org/3.x.kloth-win64/build/build/test_python_1700/@test_1700_tmp/dirB/linkD/fileB') WindowsPath('C:/buildbot.python.org/3.x.kloth-win64/build/build/test_python_1700/@test_1700_tmp/dirA/linkC/fileB') I don’t have a Windows setup so I can’t really help much with this one. |
|||
msg257718 - (view) | Author: Guido van Rossum (gvanrossum) * ![]() |
Date: 2016-01-07 21:30 | |
I think I understand the Windows failure. I uncommented some tests that were previously broken due to the symlink loop and/or PermissionError, but one of these has a different expected outcome if symlinks don't work. I've pushed a hopeful fix for that (no Windows box here either, but the buildbots are good for this :-). |
|||
msg257726 - (view) | Author: Guido van Rossum (gvanrossum) * ![]() |
Date: 2016-01-07 23:11 | |
Looks like that Windows buildbot is now green. Of the stable bots only OpenIndiana is red, and it's unrelated. |
History | |||
---|---|---|---|
Date | User | Action | Args |
2022-04-11 14:58:25 | admin | set | github: 70200 |
2016-01-07 23:11:09 | gvanrossum | set | messages: + msg257726 |
2016-01-07 21:30:15 | gvanrossum | set | messages: + msg257718 |
2016-01-07 21:15:54 | martin.panter | set | messages: + msg257717 |
2016-01-07 18:59:52 | gvanrossum | set | status: open -> closed messages: + msg257712 |
2016-01-07 18:58:59 | python-dev | set | messages: + msg257711 |
2016-01-07 18:44:49 | gvanrossum | set | messages: + msg257708 |
2016-01-07 16:44:14 | gvanrossum | set | messages: + msg257699 |
2016-01-07 11:43:31 | martin.panter | set | status: closed -> open nosy: + martin.panter messages: + msg257683 |
2016-01-06 18:37:14 | gvanrossum | set | status: open -> closed type: behavior resolution: fixed stage: resolved |
2016-01-06 18:36:40 | python-dev | set | nosy:
+ python-dev messages: + msg257628 |
2016-01-06 17:45:20 | gvanrossum | set | assignee: gvanrossum messages: + msg257620 |
2016-01-05 07:03:20 | gvanrossum | set | messages: + msg257522 |
2016-01-05 06:21:55 | serhiy.storchaka | set | nosy:
+ serhiy.storchaka messages: + msg257519 |
2016-01-05 05:25:17 | josh.r | set | nosy:
+ josh.r messages: + msg257516 |
2016-01-05 02:39:44 | ned.deily | set | nosy:
+ pitrou |
2016-01-05 02:14:14 | gvanrossum | set | components: + Library (Lib) |
2016-01-05 02:13:51 | gvanrossum | create |