classification
Title: Improper NotADirectoryError when opening a file in a fake directory
Type: behavior Stage: resolved
Components: IO Versions: Python 3.8
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: andrei.avk, danny87105, eryksun, iritkatriel, lukasz.langa, miss-islington
Priority: normal Keywords: patch

Created on 2020-09-07 15:36 by danny87105, last changed 2021-08-03 21:12 by lukasz.langa. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 22818 closed iritkatriel, 2020-10-20 12:46
PR 27471 merged andrei.avk, 2021-07-30 03:23
PR 27576 merged miss-islington, 2021-08-03 11:28
PR 27577 merged miss-islington, 2021-08-03 11:28
Messages (26)
msg376505 - (view) Author: Danny Lin (danny87105) Date: 2020-09-07 15:36
On Linux (tested on Ubuntu 16.04), if "/path/to/file" is an existing file, the code:

    open('/path/to/file/somename.txt')

raises NotADirectoryError: [Errno 20] Not a directory: '/path/to/file/somename.txt'

On Windows, similar code:

    open(r'C:\path\to\file\somename.txt')

raises FileNotFoundError: [Errno 2] No such file or directory: 'C:\\path\\chrome\\to\\file\\somename.txt'

I think the behavior on Linux is not correct. The user probably cares about the existence of the file to be opened, rather than whether its ancestor directories are valid.

OTOH, if NotADirectoryError should be raised, it should mention '/path/to/file' rather then '/path/to/file/somename.txt'. But what if '/path/to' or '/path' is actually a file? Should it be '/path/to' or '/path' instead for the same reason?
msg376529 - (view) Author: Eryk Sun (eryksun) * (Python triager) Date: 2020-09-07 21:14
> if NotADirectoryError should be raised, it should mention '/path/to/file' 
> rather then '/path/to/file/somename.txt'.

POSIX specifies that C open() should set errno to ENOTDIR when an existing path prefix component is neither a directory nor a symlink to a directory [1]. What you propose isn't possible to implement reliably unless the filesystem is locked or readonly, so it should be handled by the application instead of by the system or standard libraries.

> FileNotFoundError: [Errno 2] No such file or directory: 
> 'C:\\path\\chrome\\to\\file\\somename.txt'

Windows specifies that this case should fail as follows: "[i]f Link.File.FileType is not DirectoryFile, the operation MUST be failed with STATUS_OBJECT_PATH_NOT_FOUND" [2] (see Phase 6 in the linked pseudocode). 

The Windows API maps this status code to ERROR_PATH_NOT_FOUND (3), which is distinct from ERROR_FILE_NOT_FOUND (2). However, the C runtime maps both of these system error codes to POSIX ENOENT. All isn't lost, however, because it also saves the OS error in _doserrno. io.FileIO could preset _doserrno to 0, and if it's non-zero after calling _wopen, use its value with PyErr_SetExcFromWindowsErrWithFilenameObject instead of calling PyErr_SetFromErrnoWithFilenameObject.

---

[1] https://pubs.opengroup.org/onlinepubs/9699919799/functions/open.html
[2] https://docs.microsoft.com/en-us/openspecs/windows_protocols/ms-fsa/8ada5fbe-db4e-49fd-aef6-20d54b748e40
msg376562 - (view) Author: Danny Lin (danny87105) Date: 2020-09-08 10:24
I'm not so familiar about the spec. If such behavior is confirmed due to implementation difference across OSes, and it's also not desirable to change the mapping of the OS error to Python exception, we can simplify left it as-is.

However, this behavior difference can potentially cause cross-platform compatibility. For example, the code:

    try:
        open('/path/to/file/somename.txt')
    except FileNotFoundError:
        """do something"""

would work on Windows but break on Linux (or POSIX?).

The current (3.8) description for exception NotADirectoryError is:

    Raised when a directory operation (such as os.listdir()) is requested on something which is not a directory. Corresponds to errno ENOTDIR.

According to this, a user probably won't expect to receive a NotADirectoryError for open('/path/to/file/somename.txt'), as this doesn't seem like a directory operation at all, unless he is expert enough to know about the implication of errno ENOTDIR.

I think a note should at least be added to the documentation if we are not going to change the behavior.
msg377288 - (view) Author: Irit Katriel (iritkatriel) * (Python committer) Date: 2020-09-21 22:02
The documentation for open says "If the file cannot be opened, an OSError is raised."

NotADirectoryError and FileNotFoundError are both OSErrors.

So the correct way to write Danny's code snippet, according to the documentation, is:

    try:
        open('/path/to/file/somename.txt')
    except OSError:
        """do something"""
msg379125 - (view) Author: Danny Lin (danny87105) Date: 2020-10-20 13:54
By writing "except FileNotFoundError:", the intention is to catch an error when the file being opened is not found, and don't catch an error for other cases, such as an existing file without adequate permission. Writing "except OSError:" catches just too much cases including unwanted ones.

As they are not equivalent, you cannot say that the latter is the "correct" way for the former.

And you totally omitted the argument for the inadequate documentation for NotADirectoryError. It says NotADirectoryError is raises when a DIRECTORY operation is requested, and does not cover the case of opening a file.
msg379128 - (view) Author: Irit Katriel (iritkatriel) * (Python committer) Date: 2020-10-20 14:23
Hi Danny, 

I'm not saying that OSError and FileNotFoundError are equivalent. I'm saying that the open() API, as documented, raises OSError when opening the file fails.

The way to check whether a file exists is to use os.path.exists(path) or os.path.isfile(path). 

I don't quite follow your last point - opening a file is indeed a file operation, but it contains within it a directory operation (finding the file).
msg379131 - (view) Author: Danny Lin (danny87105) Date: 2020-10-20 14:36
I don't think a general developer would expect that open('/path/to/file/somename.txt') implies a directory operation, and it also doesn't on Windows.

I suggest that a further notice be added to NotADirectoryError, such as:

    Raised when a directory operation (such as os.listdir()) is requested on something which is not a directory. Corresponds to errno ENOTDIR. In some filesystem such as POSIX, NotADirectoryError (ENOTDIR) is raised when attempting to open a path whose ancestor is not a directory.
msg379136 - (view) Author: Irit Katriel (iritkatriel) * (Python committer) Date: 2020-10-20 14:53
> I don't think a general developer would expect that open('/path/to/file/somename.txt') implies a directory operation, and it also doesn't on Windows.


Really? It's not obvious that finding a file would involve directory operations?

In what sense does it even matter whether you expect a directory operation to happen? The contract is that if the open() fails you get an OSError. The documentation doesn't say which OSError, and that is in fact a platform-specific implementation detail.
msg379141 - (view) Author: Danny Lin (danny87105) Date: 2020-10-20 15:26
> Really? It's not obvious that finding a file would involve directory operations?

Not in some senses, and that's possibly why Windows does not raise NotADirectoryError in such case.

I agree that it's a platform-specific implementation detail. However, there are lots of platform-specific implementation details written in documentation elsewhere already, especially OS related modules such as os and os.path. I don't think mentioning platform-specific implementation details for subclasses of OSError would be any less reasonable than that.

Adding such notice for NotADirectoryError helps people who want a EAFP style code for nonexistent file by preventing them getting trapped writing a cross-platform code.
msg379150 - (view) Author: Eryk Sun (eryksun) * (Python triager) Date: 2020-10-20 18:18
Regarding documentation, builtin open() and the io and os modules generally do not provide information about platform-specific errors. But documenting the behavior for the NotADirectoryError exception itself may be useful considering it applies to many POSIX functions that access filepaths, such as stat, open, mkdir, rmdir, unlink, and rename.

Regarding behavior, I don't see anything reasonable that can or should be done for POSIX. In Windows, it should be possible to know whether FileNotFoundError is due to a bad path prefix (ERROR_PATH_NOT_FOUND, 3) or a missing file (ERROR_FILE_NOT_FOUND, 2), but io.FileIO currently only uses standard C errno, which maps both of these cases to ENOENT. So one behavior that *can* be fixed in this situation is to get the actual Windows error code from MSVC _doserrno, such that the raised FileNotFoundError would have the relevant Windows error code in its winerror attribute.
msg398526 - (view) Author: Andrei Kulakov (andrei.avk) * (Python triager) Date: 2021-07-30 03:24
I've put up a PR that expands the docs for NotADirectoryError here: 

https://github.com/python/cpython/pull/27471/files
msg398527 - (view) Author: Danny Lin (danny87105) Date: 2021-07-30 07:59
> I've put up a PR that expands the docs for NotADirectoryError here: 
> https://github.com/python/cpython/pull/27471/files

Thank you.

Wouldn't it be more clear if an example is provided? Like:

(e.g. `/path/to` does not exist when running `open('/path/to/file')`)
msg398544 - (view) Author: Andrei Kulakov (andrei.avk) * (Python triager) Date: 2021-07-30 12:36
Danny: that would be inconsistent with the rest of the doc, it's written as a reference list for exceptions and none of them have examples currently.. It also seems fairly clear without an example.
msg398545 - (view) Author: Andrei Kulakov (andrei.avk) * (Python triager) Date: 2021-07-30 12:43
(in fact, there is one example (for f-string exception) but the overall pattern is to just have reference type descriptions)
msg398562 - (view) Author: Danny Lin (danny87105) Date: 2021-07-30 15:53
@Andrei Kulakov: I was commenting on the previous version. The revised version (f2ae30b0de3c4ba1f16fc2a430cf22b447c062ed) seems ok to me.

Another question: would it be better if we add "on some platforms" for the part that the error may raise on a file operation?
msg398566 - (view) Author: Andrei Kulakov (andrei.avk) * (Python triager) Date: 2021-07-30 16:03
Danny: then it would be probably more useful to say "On POSIX systems, it will be raised when ...".

I'm ambivalent on whether this is needed.

Eryk: wdyt?
msg398567 - (view) Author: Danny Lin (danny87105) Date: 2021-07-30 16:14
@Andrei Kulakov: That statement is mainly for illustration that such behavior may vary across platforms. I use a rather vague statement as I'm not totally sure it applies if (and only if) running on POSIX. A more accurate statement is welcome as long as it's proven true.
msg398591 - (view) Author: Eryk Sun (eryksun) * (Python triager) Date: 2021-07-30 17:49
I'd prefer a generic wording regarding the platform, and overall simpler phrasing without examples: "On some platforms, it may also be raised if an operation attempts to open or traverse a non-directory file as if it were a directory." 

The latter doesn't apply to Windows generally. Attempting to traverse a non-directory fails with ERROR_PATH_NOT_FOUND (mapped to C errno ENOENT), and attempting to open a non-directory with a path that has a trailing slash fails with ERROR_INVALID_NAME (mapped to C errno EINVAL).
msg398593 - (view) Author: Andrei Kulakov (andrei.avk) * (Python triager) Date: 2021-07-30 17:56
Eryk: sounds good, updated the PR.
msg398596 - (view) Author: Danny Lin (danny87105) Date: 2021-07-30 18:13
It should be emphasized that it may happen on a **file operation**... So at least be something like: "On some platforms, it may also be raised if a file operation involves an attempt to open or traverse a non-directory file as if it were a directory."
msg398602 - (view) Author: Eryk Sun (eryksun) * (Python triager) Date: 2021-07-30 18:53
> It should be emphasized that it may happen on a **file operation**

I think it's adequately covered by "attempts to open or traverse a non-directory file". The reader should know that opening "/path/to/file/somename.txt" requires traversing the components in the path. So if "file" isn't a directory, raising NotADirectoryError should be expected in POSIX.

---

If someone can verify the behavior on common non-Linux POSIX systems such as macOS, FreeBSD, and OpenBSD, then the wording could be narrowed down to "on most POSIX platforms" instead of "on some platforms". 

For example, given "spam" is a regular file in the current directory, check os.open('spam', os.O_DIRECTORY); os.open('spam/', 0); and os.open('spam/eggs', 0).
msg398603 - (view) Author: Andrei Kulakov (andrei.avk) * (Python triager) Date: 2021-07-30 19:04
Eryk: On MacOS, I get Not a directory error on all 3:

NotADirectoryError: [Errno 20] Not a directory: 'spam'

NotADirectoryError: [Errno 20] Not a directory: 'spam/'

NotADirectoryError: [Errno 20] Not a directory: 'spam/eggs'
msg398610 - (view) Author: Eryk Sun (eryksun) * (Python triager) Date: 2021-07-30 21:55
> On MacOS, I get Not a directory error on all 3

Confirmation on Linux and macOS suffices for "most POSIX systems", based on market share for desktops and servers. It would be nice to also confirm this for FreeBSD, AIX, HP-UX, and Solaris, but I suppose it isn't necessary. A system that strictly conforms to POSIX will behave the same for those operations.
msg398820 - (view) Author: Łukasz Langa (lukasz.langa) * (Python committer) Date: 2021-08-03 11:28
New changeset f7c23a99cd4f8179b6ba2cffaeb78b852c0f6488 by andrei kulakov in branch 'main':
bpo-41737: expand doc for NotADirectoryError (GH-27471)
https://github.com/python/cpython/commit/f7c23a99cd4f8179b6ba2cffaeb78b852c0f6488
msg398822 - (view) Author: Łukasz Langa (lukasz.langa) * (Python committer) Date: 2021-08-03 12:03
New changeset b5f026112768eb0a06622263bdea86d7d85981c5 by Miss Islington (bot) in branch '3.9':
bpo-41737: expand doc for NotADirectoryError (GH-27471) (GH-27577)
https://github.com/python/cpython/commit/b5f026112768eb0a06622263bdea86d7d85981c5
msg398823 - (view) Author: miss-islington (miss-islington) Date: 2021-08-03 12:05
New changeset 84494db41902774ea6ac72e5b308b429850bbf71 by Miss Islington (bot) in branch '3.10':
bpo-41737: expand doc for NotADirectoryError (GH-27471)
https://github.com/python/cpython/commit/84494db41902774ea6ac72e5b308b429850bbf71
History
Date User Action Args
2021-08-03 21:12:10lukasz.langasetstatus: open -> closed
resolution: fixed
stage: patch review -> resolved
2021-08-03 12:05:08miss-islingtonsetmessages: + msg398823
2021-08-03 12:03:44lukasz.langasetmessages: + msg398822
2021-08-03 11:28:33miss-islingtonsetpull_requests: + pull_request26083
2021-08-03 11:28:27miss-islingtonsetnosy: + miss-islington
pull_requests: + pull_request26082
2021-08-03 11:28:23lukasz.langasetnosy: + lukasz.langa
messages: + msg398820
2021-07-30 21:55:13eryksunsetmessages: + msg398610
2021-07-30 19:04:20andrei.avksetmessages: + msg398603
2021-07-30 18:53:49eryksunsetmessages: + msg398602
2021-07-30 18:13:20danny87105setmessages: + msg398596
2021-07-30 17:56:45andrei.avksetmessages: + msg398593
2021-07-30 17:49:12eryksunsetmessages: + msg398591
2021-07-30 16:14:54danny87105setmessages: + msg398567
2021-07-30 16:03:53andrei.avksetmessages: + msg398566
2021-07-30 15:53:39danny87105setmessages: + msg398562
2021-07-30 12:43:06andrei.avksetmessages: + msg398545
2021-07-30 12:36:56andrei.avksetmessages: + msg398544
2021-07-30 07:59:04danny87105setmessages: + msg398527
2021-07-30 03:24:47andrei.avksetmessages: + msg398526
2021-07-30 03:23:35andrei.avksetnosy: + andrei.avk
pull_requests: + pull_request25990
2020-10-20 18:18:14eryksunsetmessages: + msg379150
2020-10-20 15:26:54danny87105setmessages: + msg379141
2020-10-20 14:53:27iritkatrielsetmessages: + msg379136
2020-10-20 14:36:13danny87105setmessages: + msg379131
2020-10-20 14:23:09iritkatrielsetmessages: + msg379128
2020-10-20 13:54:41danny87105setmessages: + msg379125
2020-10-20 12:46:47iritkatrielsetkeywords: + patch
stage: patch review
pull_requests: + pull_request21774
2020-09-21 22:02:39iritkatrielsetnosy: + iritkatriel
messages: + msg377288
components: + IO
2020-09-11 23:23:46terry.reedysettitle: Improper NotADirectoryError when opening a file under a fake directory -> Improper NotADirectoryError when opening a file in a fake directory
2020-09-08 10:24:50danny87105setmessages: + msg376562
2020-09-07 21:14:39eryksunsetnosy: + eryksun
messages: + msg376529
2020-09-07 15:36:37danny87105create