This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

Author serhiy.storchaka
Recipients Shai, brett.cannon, gregory.p.smith, hongweipeng, pitrou, serhiy.storchaka
Date 2019-09-11.14:25:36
SpamBayes Score -1.0
Marked as misclassified Yes
Message-id <1568211937.05.0.211996807985.issue37935@roundup.psfhosted.org>
In-reply-to
Content
Yes, I am the author of the code that uses os.scandir() in os.walk(), os.fwalk(), glob.iglob() and Path.glob() (see issue23605, issue25996, issue25596, issue26032). And it was always in mind to not keep many file descriptors open when traverse directories recursively. It was also a reason of rejecting my earlier patch for speeding up os.walk() by using os.fwalk() (see issue15200).

It is safe to iterated over os.scandir() without turning it into a list if we do not do this recursively.

Unfortunately there were no special tests for this. PR 15956 adds them. You can easily break tests for pathlib if remove any of list(). It is harder to break tests for glob, because fnmatch.filter() consumes the iterator and implicitly closes the scandir iterator. You need to replace it with a generator and fnmatch.fnmatch() called in a loop. Breaking the tests for os.walk() is difficult. The code of os.walk() is so complex because it needs to return lists of files and subdirectories, and therefore it consumes the scandir iterator and closes it. But with some tricks it is possible to break tests for os.walk(). It is unlikely somebody will do this unintentionally.
History
Date User Action Args
2019-09-11 14:25:37serhiy.storchakasetrecipients: + serhiy.storchaka, brett.cannon, gregory.p.smith, pitrou, hongweipeng, Shai
2019-09-11 14:25:37serhiy.storchakasetmessageid: <1568211937.05.0.211996807985.issue37935@roundup.psfhosted.org>
2019-09-11 14:25:37serhiy.storchakalinkissue37935 messages
2019-09-11 14:25:36serhiy.storchakacreate