classification
Title: shutil.copy raises IsADirectoryError when the directory does not actually exist
Type: behavior Stage: resolved
Components: Library (Lib) Versions: Python 3.11, Python 3.10, Python 3.9
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: gregory.p.smith Nosy List: Alex Grund, andrei.avk, eryksun, gregory.p.smith, jerpint, kulikjak, lukasz.langa, miss-islington
Priority: normal Keywords: patch

Created on 2021-02-14 05:16 by jerpint, last changed 2021-09-21 22:21 by lukasz.langa. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 27049 merged andrei.avk, 2021-07-07 02:29
PR 27081 merged miss-islington, 2021-07-10 03:47
PR 27082 merged miss-islington, 2021-07-10 03:48
PR 27257 merged kulikjak, 2021-07-20 10:18
PR 27267 merged miss-islington, 2021-07-20 18:16
PR 27268 merged miss-islington, 2021-07-20 18:16
PR 28507 merged miss-islington, 2021-09-21 21:53
PR 28508 merged miss-islington, 2021-09-21 21:53
Messages (19)
msg386932 - (view) Author: Jeremy Pinto (jerpint) Date: 2021-02-14 05:16
Issue: If you try to copy a file to a directory that doesn't exist using shutil.copy, a IsADirectory error is raised saying the directory exists. 

This issue is actually caused when `open(not_a_dir, 'wb') is called on a non-existing dir.

Expected behaviour: Should instead raise NotADirectoryError
-----------------------------

Steps to reproduce:

[nav] In [1]: import os
         ...: from pathlib import Path
         ...: from shutil import copy
         ...:
         ...: tmp_file = '/tmp/some_file.txt'
         ...: Path(tmp_file).touch()
         ...: nonexistent_dir = 'not_a_dir/'
         ...: assert not os.path.exists(nonexistent_dir)
         ...: copy(tmp_file, nonexistent_dir)
---------------------------------------------------------------------------
IsADirectoryError                         Traceback (most recent call last)
<ipython-input-2-b0e0ec4f4875> in <module>
      7 nonexistent_dir = 'not_a_dir/'
      8 assert not os.path.exists(nonexistent_dir)
----> 9 copy(tmp_file, nonexistent_dir)

~/miniconda3/lib/python3.7/shutil.py in copy(src, dst, follow_symlinks)
    243     if os.path.isdir(dst):
    244         dst = os.path.join(dst, os.path.basename(src))
--> 245     copyfile(src, dst, follow_symlinks=follow_symlinks)
    246     copymode(src, dst, follow_symlinks=follow_symlinks)
    247     return dst

~/miniconda3/lib/python3.7/shutil.py in copyfile(src, dst, follow_symlinks)
    119     else:
    120         with open(src, 'rb') as fsrc:
--> 121             with open(dst, 'wb') as fdst:
    122                 copyfileobj(fsrc, fdst)
    123     return dst

IsADirectoryError: [Errno 21] Is a directory: 'not_a_dir/'
msg386933 - (view) Author: Jeremy Pinto (jerpint) Date: 2021-02-14 05:28
In fact, the issue seems to be coming from open() itself when opening a non-existent directory in write mode:

[nav] In [1]: import os
         ...: nonexixstent_dir = 'not_a_dir/'
         ...: assert not os.path.exists(nonexixstent_dir)
         ...: with open(nonexixstent_dir, 'wb') as fdst:
         ...:     pass
---------------------------------------------------------------------------
IsADirectoryError                         Traceback (most recent call last)
<ipython-input-1-73d4010d6f34> in <module>
      2 dir_path = 'not_a_dir/'
      3 assert not os.path.exists(nonexixstent_dir)
----> 4 with open(nonexixstent_dir, 'wb') as fdst:
      5     pass

IsADirectoryError: [Errno 21] Is a directory: 'not_a_dir/'
msg386935 - (view) Author: Eryk Sun (eryksun) * (Python triager) Date: 2021-02-14 06:49
> IsADirectoryError: [Errno 21] Is a directory: 'not_a_dir/'

The trailing slash forces the OS to handle "not_a_dir" as a directory [1]. 

    A pathname that contains at least one non- <slash> character and that 
    ends with one or more trailing <slash> characters shall not be resolved
    successfully unless the last pathname component before the trailing 
    <slash> characters names an existing directory or a directory entry 
    that is to be created for a directory immediately after the pathname is
    resolved. 

Mode "w" corresponds to low-level POSIX open() flags O_CREAT | O_TRUNC | O_WRONLY. If write access is requested for a directory, the open() system call must fail with EISDIR [2].

    [EISDIR]
        The named file is a directory and oflag includes O_WRONLY or O_RDWR,
        or includes O_CREAT without O_DIRECTORY.

In most cases, opening a directory with O_CREAT also fails with E_ISDIR. POSIX does permit an implementation to create a directory with O_CREAT | O_DIRECTORY. In Linux, however, O_CREAT always creates a regular file, regardless of O_DIRECTORY, so open(pathname, O_CREAT | flags) always fails with EISDIR when pathname is an existing directory or names a directory by way of a trailing slash.

---
[1] https://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1_chap04.html#tag_04_13
[2] https://pubs.opengroup.org/onlinepubs/9699919799/functions/open.html
msg387134 - (view) Author: Eryk Sun (eryksun) * (Python triager) Date: 2021-02-17 02:02
I left this open in case someone wants to modify shutil.copy() and shutil.copy2() to raise a less misleading exception when `dst` names a non-existing directory such as 'not_a_dir/'. Failing with IsADirectoryError (errno EISDIR) is confusing since shutil.copy() and shutil.copy2() do support a destination directory.

Note that in Windows this case fails with EINVAL, which is at least less misleading than EISDIR. The EINVAL error is based on WinAPI ERROR_INVALID_NAME (123). CreateFileW() can create a directory if passed particular parameters, in which case a trailing slash is allowed. Otherwise it fails with ERROR_INVALID_NAME (123), unless it's an open-existing disposition, in which case it fails with ERROR_FILE_NOT_FOUND (2). This is specified in [MS-FSA] 2.1.5.1 [1]:

    Phase 6 -- Location of file (final path component):

        Search ParentFile.DirectoryList for a Link where Link.Name or
        Link.ShortName matches FileNameToOpen. If such a link is found:

            Set File = Link.File.
            Set Open.File to File.
            Set Open.Link to Link.

        Else:

            If (CreateDisposition == FILE_OPEN || CreateDisposition ==
            FILE_OVERWRITE), the operation MUST be failed with
            STATUS_OBJECT_NAME_NOT_FOUND.

    Phase 7 -- Type of stream to open:

        If PathName contains a trailing backslash:

            If StreamTypeToOpen is DataStream or
            CreateOptions.FILE_NON_DIRECTORY_FILE is TRUE, the operation
            MUST be failed with STATUS_OBJECT_NAME_INVALID.

NTAPI STATUS_OBJECT_NAME_NOT_FOUND and STATUS_OBJECT_NAME_INVALID map to WinAPI ERROR_FILE_NOT_FOUND and ERROR_INVALID_NAME.

---
[1] https://docs.microsoft.com/en-us/openspecs/windows_protocols/ms-fsa/8ada5fbe-db4e-49fd-aef6-20d54b748e40
msg397060 - (view) Author: Andrei Kulakov (andrei.avk) * (Python triager) Date: 2021-07-07 03:23
I've opened the PR here:
https://github.com/python/cpython/pull/27049/files

.. but can someone trigger builds for other OSes like Aix, FreeBSD, etc? I'm not sure what errors they would use for this case so the unit test may have to be updated to skip some of them.
msg397061 - (view) Author: Andrei Kulakov (andrei.avk) * (Python triager) Date: 2021-07-07 03:26
I'm not sure if shutil module docs should be updated or not, as this change applies to a narrow corner case.
msg397119 - (view) Author: Andrei Kulakov (andrei.avk) * (Python triager) Date: 2021-07-07 21:18
Note that someone else ran into this confusion here: https://bugs.python.org/issue43153 .
msg397235 - (view) Author: Gregory P. Smith (gregory.p.smith) * (Python committer) Date: 2021-07-10 03:47
New changeset 248173cc0483a9ad9261353302f1234cf9eb2ebe by andrei kulakov in branch 'main':
bpo-43219: shutil.copyfile, raise a less confusing exception instead of IsADirectoryError (GH-27049)
https://github.com/python/cpython/commit/248173cc0483a9ad9261353302f1234cf9eb2ebe
msg397236 - (view) Author: miss-islington (miss-islington) Date: 2021-07-10 04:07
New changeset 1577259cc51d0d46ad676798ce0a130039acf956 by Miss Islington (bot) in branch '3.10':
bpo-43219: shutil.copyfile, raise a less confusing exception instead of IsADirectoryError (GH-27049)
https://github.com/python/cpython/commit/1577259cc51d0d46ad676798ce0a130039acf956
msg397237 - (view) Author: miss-islington (miss-islington) Date: 2021-07-10 04:14
New changeset c89f0b2587eb0b16175a0bbb12d0b86314ff9320 by Miss Islington (bot) in branch '3.9':
[3.9] bpo-43219: shutil.copyfile, raise a less confusing exception instead of IsADirectoryError (GH-27049) (GH-27082)
https://github.com/python/cpython/commit/c89f0b2587eb0b16175a0bbb12d0b86314ff9320
msg397865 - (view) Author: Jakub Kulik (kulikjak) * Date: 2021-07-20 10:17
On Solaris (I checked this on Oracle and SmartOS), the error is:

  NotADirectoryError: [Errno 20] Not a directory: 'not_a_dir/'

which I think belongs to the 'errors are not confusing' category with Windows and macOS.
msg397893 - (view) Author: Gregory P. Smith (gregory.p.smith) * (Python committer) Date: 2021-07-20 18:16
New changeset 6564656495d456a1bcc1aaa06abfc696209f37b2 by Jakub Kulík in branch 'main':
bpo-43219: skip Solaris in the test as well (GH-27257)
https://github.com/python/cpython/commit/6564656495d456a1bcc1aaa06abfc696209f37b2
msg397897 - (view) Author: miss-islington (miss-islington) Date: 2021-07-20 18:45
New changeset dae4928dd07109db69e090b1c8193a023ce695cd by Miss Islington (bot) in branch '3.9':
[3.9] bpo-43219: skip Solaris in the test as well (GH-27257) (GH-27267)
https://github.com/python/cpython/commit/dae4928dd07109db69e090b1c8193a023ce695cd
msg397898 - (view) Author: miss-islington (miss-islington) Date: 2021-07-20 18:53
New changeset 574da4633b44b4048f74c93da496ed2a3ead99dd by Miss Islington (bot) in branch '3.10':
[3.10] bpo-43219: skip Solaris in the test as well (GH-27257) (GH-27268)
https://github.com/python/cpython/commit/574da4633b44b4048f74c93da496ed2a3ead99dd
msg397901 - (view) Author: Andrei Kulakov (andrei.avk) * (Python triager) Date: 2021-07-20 19:03
Thanks for reporting Jakub, and for patching Gregory!
msg402022 - (view) Author: Alex Grund (Alex Grund) Date: 2021-09-17 09:28
The changelog wrongfully links to https://bugs.python.org/issue41928 instead of this issue.

Also the fix introduced a regression: Trying to copy a directory now raises a FileNotFoundError
msg402373 - (view) Author: Łukasz Langa (lukasz.langa) * (Python committer) Date: 2021-09-21 21:53
New changeset b7eac52b466f697d3e89f47508e0df0196a98970 by andrei kulakov in branch 'main':
bpo-45234: Fix FileNotFound exception raised instead of IsADirectoryError in shutil.copyfile() (GH-28421)
https://github.com/python/cpython/commit/b7eac52b466f697d3e89f47508e0df0196a98970
msg402386 - (view) Author: Łukasz Langa (lukasz.langa) * (Python committer) Date: 2021-09-21 22:14
New changeset 41d48bc038b254cc4a78a2d840097196b9545a84 by Miss Islington (bot) in branch '3.10':
bpo-45234: Fix FileNotFound exception raised instead of IsADirectoryError in shutil.copyfile() (GH-28421) (GH-28508)
https://github.com/python/cpython/commit/41d48bc038b254cc4a78a2d840097196b9545a84
msg402394 - (view) Author: Łukasz Langa (lukasz.langa) * (Python committer) Date: 2021-09-21 22:21
New changeset 09390c837a0bf73e213db2fbde34d756fa77a837 by Miss Islington (bot) in branch '3.9':
bpo-45234: Fix FileNotFound exception raised instead of IsADirectoryError in shutil.copyfile() (GH-28421) (GH-28507)
https://github.com/python/cpython/commit/09390c837a0bf73e213db2fbde34d756fa77a837
History
Date User Action Args
2021-09-21 22:21:18lukasz.langasetmessages: + msg402394
2021-09-21 22:14:48lukasz.langasetmessages: + msg402386
2021-09-21 21:53:47miss-islingtonsetpull_requests: + pull_request26905
2021-09-21 21:53:43miss-islingtonsetpull_requests: + pull_request26903
2021-09-21 21:53:11lukasz.langasetnosy: + lukasz.langa
messages: + msg402373
2021-09-17 09:28:37Alex Grundsetnosy: + Alex Grund
messages: + msg402022
2021-07-20 19:03:44andrei.avksetmessages: + msg397901
2021-07-20 18:53:47miss-islingtonsetmessages: + msg397898
2021-07-20 18:45:11miss-islingtonsetmessages: + msg397897
2021-07-20 18:16:59miss-islingtonsetpull_requests: + pull_request25812
2021-07-20 18:16:35miss-islingtonsetpull_requests: + pull_request25811
2021-07-20 18:16:28gregory.p.smithsetmessages: + msg397893
2021-07-20 10:18:58kulikjaksetpull_requests: + pull_request25802
2021-07-20 10:17:56kulikjaksetnosy: + kulikjak
messages: + msg397865
2021-07-10 15:03:04gregory.p.smithsetstatus: open -> closed
resolution: fixed
stage: patch review -> resolved
2021-07-10 04:14:07miss-islingtonsetmessages: + msg397237
2021-07-10 04:07:41miss-islingtonsetmessages: + msg397236
2021-07-10 03:48:03miss-islingtonsetpull_requests: + pull_request25631
2021-07-10 03:47:59gregory.p.smithsetmessages: + msg397235
2021-07-10 03:47:58miss-islingtonsetnosy: + miss-islington
pull_requests: + pull_request25630
2021-07-09 05:04:39gregory.p.smithsetassignee: gregory.p.smith

nosy: + gregory.p.smith
versions: + Python 3.11, - Python 3.8
2021-07-07 21:18:58andrei.avksetmessages: + msg397119
2021-07-07 03:26:04andrei.avksetmessages: + msg397061
2021-07-07 03:23:12andrei.avksetmessages: + msg397060
2021-07-07 02:29:32andrei.avksetkeywords: + patch
nosy: + andrei.avk

pull_requests: + pull_request25605
stage: patch review
2021-03-17 07:30:11eryksunlinkissue35216 superseder
2021-02-24 18:35:02eryksunlinkissue24977 superseder
2021-02-17 02:02:47eryksunsettype: behavior
messages: + msg387134
components: + Library (Lib)
versions: + Python 3.8, Python 3.9, Python 3.10, - Python 3.7
2021-02-14 06:49:04eryksunsetnosy: + eryksun
messages: + msg386935
2021-02-14 05:28:45jerpintsetmessages: + msg386933
2021-02-14 05:16:28jerpintcreate