classification
Title: Inconsistent exceptions thrown by pathlib.Path.mkdir on different OSes
Type: Stage:
Components: Library (Lib), Windows Versions: Python 3.9
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: Hong Xu, eryksun, iritkatriel, paul.moore, serhiy.storchaka, steve.dower, tim.golden, zach.ware
Priority: normal Keywords:

Created on 2021-01-09 08:15 by Hong Xu, last changed 2021-01-10 02:29 by Hong Xu.

Messages (5)
msg384715 - (view) Author: Hong Xu (Hong Xu) * Date: 2021-01-09 08:15
Consider the following code:

-------------------------

import pathlib

def main():
    pathlib.Path('tmp').touch()
    pathlib.Path('tmp/tmp_sub').mkdir(parents=True)

main()

------------------------

Run the code above in an empty directory.

On Linux, it throws a `NotADirectory` exception:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "main.py", line 5, in main
    pathlib.Path('tmp/tmp_sub').mkdir(parents=True)
  File "/usr/lib/python3.8/pathlib.py", line 1287, in mkdir
    self._accessor.mkdir(self, mode)
NotADirectoryError: [Errno 20] Not a directory: 'tmp/tmp_sub'

-----------------------------

On Windows, it throws a FileExistsError exception:

Traceback (most recent call last):
  File "C:\Users\hong\anaconda3\lib\pathlib.py", line 1284, in mkdir
    self._accessor.mkdir(self, mode)
FileNotFoundError: [WinError 3] The system cannot find the path specified: 'tmp\\tmp_sub'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "main.py", line 7, in <module>
    main()
  File "main.py", line 5, in main
    pathlib.Path('tmp/tmp_sub').mkdir(parents=True)
  File "C:\Users\hong\anaconda3\lib\pathlib.py", line 1288, in mkdir
    self.parent.mkdir(parents=True, exist_ok=True)
  File "C:\Users\hong\anaconda3\lib\pathlib.py", line 1284, in mkdir
    self._accessor.mkdir(self, mode)
FileExistsError: [WinError 183] Cannot create a file when that file already exists: 'tmp
msg384723 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2021-01-09 12:37
Yes, some operations can raise instances of different OSError subclasses on different platforms, because the corresponding C function sets different errno. It is so, and we cannot do anything with this. It is just a part of the difference between plaforms.
msg384727 - (view) Author: Eryk Sun (eryksun) * (Python triager) Date: 2021-01-09 13:06
NT filesystems are specified to fail with STATUS_OBJECT_PATH_NOT_FOUND (0xC000003A) if a parent component in a path either does not exist or is not a directory. In the Windows API, this translates to ERROR_PATH_NOT_FOUND (3), which in the C runtime translates to ENOENT (2), which in Python is raised as FileNotFoundError.

The design of Path.mkdir() assumes POSIX behavior, which splits the above two cases respectively into ENOENT and ENOTDIR. Thus it assumes that FileNotFoundError means a parent component does not exist. This eliminates the race condition of having to test whether the parent path exists. In Windows, it's not possible to differentiate the two cases by error code alone, so unfortunately Path.mkdir() tries to create the parent path when it contains a non-directory component, and thus a nested FileExistsError exception is raised.

The design of os.makedirs() instead checks whether the parent exists before trying to create it, and ignores FileExitsError if creating the parent tree fails due to a race condition. This design behaves a bit more consistently across platforms, but it still fails with a different exception on Windows vs POSIX, i.e. FileNotFoundError vs NotADirectoryError.
msg384746 - (view) Author: Irit Katriel (iritkatriel) * (Python triager) Date: 2021-01-10 00:09
See also issue41737.
msg384751 - (view) Author: Hong Xu (Hong Xu) * Date: 2021-01-10 02:29
Should we update the document at least? The document doesn't mention NotADirectoryError or its super classes at all.
History
Date User Action Args
2021-01-10 02:29:29Hong Xusetmessages: + msg384751
2021-01-10 00:09:46iritkatrielsetnosy: + iritkatriel
messages: + msg384746
2021-01-09 13:06:23eryksunsetnosy: + eryksun
messages: + msg384727
2021-01-09 12:37:30serhiy.storchakasetnosy: + serhiy.storchaka
messages: + msg384723
2021-01-09 08:17:38Hong Xusettitle: Inconsistent exceptions thrown by mkdir on different OSes -> Inconsistent exceptions thrown by pathlib.Path.mkdir on different OSes
2021-01-09 08:17:22Hong Xusettitle: Inconsistent exception thrown by mkdir on different OSes -> Inconsistent exceptions thrown by mkdir on different OSes
2021-01-09 08:15:26Hong Xucreate