classification
Title: os.walk is limited by python's recursion limit
Type: Stage:
Components: Documentation Versions: Python 3.5
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: docs@python Nosy List: Elena.Oat, Sergey Chvalyuk, Thomas.Waldmann, docs@python, rhettinger
Priority: low Keywords:

Created on 2016-03-12 00:59 by Thomas.Waldmann, last changed 2016-06-10 11:56 by Sergey Chvalyuk.

Messages (5)
msg261626 - (view) Author: Thomas Waldmann (Thomas.Waldmann) Date: 2016-03-12 00:58
os.walk calls itself recursively and that limits the directory depth it can "walk" into.

On Linux, one can create directory hierarchies deeper than that.

For some more details see there:

https://github.com/borgbackup/borg/issues/380
msg261627 - (view) Author: Thomas Waldmann (Thomas.Waldmann) Date: 2016-03-12 01:07
Note: similar issues can be seen in other stdlib recursive filesystem-related functions also.
msg261681 - (view) Author: Raymond Hettinger (rhettinger) * (Python committer) Date: 2016-03-13 07:22
I think the code should be left as-is (we've used some variation of recursion for walking a very long time with no reported real-world problems).  Perhaps a documentation note can be added to the effect that the there is a recursion limit and that it can be changed by the user if needed.
msg262064 - (view) Author: Elena Oat (Elena.Oat) * Date: 2016-03-20 10:07
I've actually tried to test the recursion limit on my Mac OS X. It seems that I cannot create any subdirectories after around 500 subdir depth. I guess it's related to the allowed path length (<1024 chars?).
msg268116 - (view) Author: Sergey Chvalyuk (Sergey Chvalyuk) Date: 2016-06-10 11:56
There is another limitation with os.walk
it also cannot go very deep inside tree because of scandir can throw 
OSError(36, 'File name too long')

I have wrote tests for both cases:

https://gist.github.com/grubberr/3367f1d605e292103b576a17a46ef3c3

tested on linux only

on MacOS it throw 'File name too long' for 1-st test too
set sys.settrecursionlimit to lower values
History
Date User Action Args
2016-06-10 11:56:12Sergey Chvalyuksetnosy: + Sergey Chvalyuk
messages: + msg268116
2016-03-20 10:07:38Elena.Oatsetnosy: + Elena.Oat
messages: + msg262064
2016-03-13 07:22:35rhettingersetpriority: normal -> low

nosy: + rhettinger, docs@python
messages: + msg261681

assignee: docs@python
components: + Documentation, - Library (Lib)
2016-03-12 01:07:20Thomas.Waldmannsetmessages: + msg261627
2016-03-12 00:59:00Thomas.Waldmanncreate