Title: [doc] os.walk is limited by python's recursion limit
Type: Stage:
Components: Documentation Versions: Python 3.11, Python 3.10, Python 3.9
Status: open Resolution:
Dependencies: Superseder:
Assigned To: docs@python Nosy List: Elena.Oat, Sergey Chvalyuk, Thomas.Waldmann, docs@python, rhettinger, rmast
Priority: low Keywords: easy

Created on 2016-03-12 00:59 by Thomas.Waldmann, last changed 2021-08-18 06:30 by rmast.

Messages (5)
msg261626 - (view) Author: Thomas Waldmann (Thomas.Waldmann) Date: 2016-03-12 00:58
os.walk calls itself recursively and that limits the directory depth it can "walk" into.

On Linux, one can create directory hierarchies deeper than that.

For some more details see there:
msg261627 - (view) Author: Thomas Waldmann (Thomas.Waldmann) Date: 2016-03-12 01:07
Note: similar issues can be seen in other stdlib recursive filesystem-related functions also.
msg261681 - (view) Author: Raymond Hettinger (rhettinger) * (Python committer) Date: 2016-03-13 07:22
I think the code should be left as-is (we've used some variation of recursion for walking a very long time with no reported real-world problems).  Perhaps a documentation note can be added to the effect that the there is a recursion limit and that it can be changed by the user if needed.
msg262064 - (view) Author: Elena Oat (Elena.Oat) * Date: 2016-03-20 10:07
I've actually tried to test the recursion limit on my Mac OS X. It seems that I cannot create any subdirectories after around 500 subdir depth. I guess it's related to the allowed path length (<1024 chars?).
msg268116 - (view) Author: Sergey Chvalyuk (Sergey Chvalyuk) Date: 2016-06-10 11:56
There is another limitation with os.walk
it also cannot go very deep inside tree because of scandir can throw 
OSError(36, 'File name too long')

I have wrote tests for both cases:

tested on linux only

on MacOS it throw 'File name too long' for 1-st test too
set sys.settrecursionlimit to lower values
Date User Action Args
2021-08-18 06:30:22rmastsetnosy: + rmast
2021-06-22 13:42:59iritkatrielsetkeywords: + easy
title: os.walk is limited by python's recursion limit -> [doc] os.walk is limited by python's recursion limit
versions: + Python 3.9, Python 3.10, Python 3.11, - Python 3.5
2016-06-10 11:56:12Sergey Chvalyuksetnosy: + Sergey Chvalyuk
messages: + msg268116
2016-03-20 10:07:38Elena.Oatsetnosy: + Elena.Oat
messages: + msg262064
2016-03-13 07:22:35rhettingersetpriority: normal -> low

nosy: + rhettinger, docs@python
messages: + msg261681

assignee: docs@python
components: + Documentation, - Library (Lib)
2016-03-12 01:07:20Thomas.Waldmannsetmessages: + msg261627
2016-03-12 00:59:00Thomas.Waldmanncreate