This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: pathlib.Path.iterdir doesn't throw an exception until you start iterating
Type: enhancement Stage: patch review
Components: Library (Lib) Versions: Python 3.8
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: Paul Pinterits, pitrou, prudvinit
Priority: normal Keywords: patch

Created on 2018-08-29 08:20 by Paul Pinterits, last changed 2022-04-11 14:59 by admin.

Pull Requests
URL Status Linked Edit
PR 8996 closed prudvinit, 2018-08-29 16:49
PR 8999 open prudvinit, 2018-08-29 17:05
Messages (3)
msg324307 - (view) Author: Paul Pinterits (Paul Pinterits) Date: 2018-08-29 08:20
The fact that `Path.iterdir()` only throws exceptions once you start iterating over it makes it very difficult to write correct code.

Let's look at an example: We'll iterate over all children of a directory and print their file size.

If we try to do it like this, the `try...except` has no effect whatsoever:

try:
    children = path_that_doesnt_exist.iterdir()
except FileNotFoundError:
    print("directory doesn't exist")
for path in children:
    print(path.stat().st_size)

If we explicitly check whether the path exists (and is a directory), we end up with a race condition:

if path_that_doesnt_exist.is_dir():
    print("directory doesn't exist")
else:
    for path in children:
        print(path.stat().st_size)

We can wrap the whole loop in a `try...except`, but then we might end up catching exceptions we didn't intend to catch. (For example, the exception that's thrown when we try to get the size of a broken symlink.)

try:
    for path in path_that_doesnt_exist.iterdir():
        print(path.stat().st_size)  # this can also throw FileNotFoundError
except FileNotFoundError:
    print("directory doesn't exist")

We can manually call `next` on the iterator inside of a `try..except` block, but that's awfully verbose and requires knowledge about iterators and the `next` function:

children = iter(path_that_doesnt_exist.iterdir())
while True:
    try:
        path = next(children)
    except FileNotFoundError:
        print("directory doesn't exist")
        break
    print(path.stat().st_size)

Or we can turn the iterator into a list inside of a `try...except`, which seems to be the best option, but completely defeats the point of having an iterator:

try:
    children = list(path_that_doesnt_exist.iterdir())
except FileNotFoundError:
    print("directory doesn't exist")
else:
    for path in children:
        print(path.stat().st_size)

As you can see, writing correct (and good) code with `iterdir` is more difficult than it has any right to be. Please change this behavior so that exceptions are thrown immediately when `iterdir` is called.
msg324332 - (view) Author: Paul Pinterits (Paul Pinterits) Date: 2018-08-29 17:06
As an afterthought, I'd like to suggest an alternative solution: If changing the `iterdir` behavior is not possible or not desirable for some reason, please add a `Path.listdir` method that returns a list instead of an iterator. Lazy file system operations can often introduce unnecessary race conditions in the code, so I believe it would be very useful if every lazy method had an eager equivalent.
msg324334 - (view) Author: Prudvi RajKumar Maddala (prudvinit) * Date: 2018-08-29 17:07
Made changes, pathlib.Path('.').iterdir() now throws a FileNotFoundError if the path is not valid.
History
Date User Action Args
2022-04-11 14:59:05adminsetgithub: 78722
2019-05-23 22:47:36cheryl.sabellasetnosy: + pitrou

type: behavior -> enhancement
versions: + Python 3.8, - Python 3.4, Python 3.5, Python 3.6, Python 3.7
2018-08-29 17:07:03prudvinitsetnosy: + prudvinit
messages: + msg324334
2018-08-29 17:06:39Paul Pinteritssetmessages: + msg324332
2018-08-29 17:05:12prudvinitsetpull_requests: + pull_request8470
2018-08-29 16:49:03prudvinitsetkeywords: + patch
stage: patch review
pull_requests: + pull_request8467
2018-08-29 08:20:21Paul Pinteritscreate