classification
Title: Different behavior of os.path.realpath('nul') in 3.7 and 3.8
Type: behavior Stage: resolved
Components: Library (Lib), Windows Versions: Python 3.9, Python 3.8
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: eryksun, iamsav, miss-islington, paul.moore, steve.dower, tim.golden, zach.ware
Priority: normal Keywords: patch

Created on 2019-09-10 06:30 by iamsav, last changed 2019-09-17 16:04 by eryksun. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 15899 merged steve.dower, 2019-09-11 09:24
PR 15903 merged miss-islington, 2019-09-11 09:49
PR 16156 merged steve.dower, 2019-09-15 08:43
PR 16187 merged miss-islington, 2019-09-16 14:25
Messages (14)
msg351577 - (view) Author: Александр Семенов (iamsav) Date: 2019-09-10 06:30
Windows 10:
```
C:\Users\User\Downloads>py -3.7 -c "import os.path;os.path.realpath('nul')"

C:\Users\User\Downloads>py -3.8 -c "import os.path;os.path.realpath('nul')"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "C:\Python38\lib\ntpath.py", line 592, in realpath
    path = _getfinalpathname_nonstrict(path)
  File "C:\Python38\lib\ntpath.py", line 566, in _getfinalpathname_nonstrict
    path = _readlink_deep(path, seen)
  File "C:\Python38\lib\ntpath.py", line 536, in _readlink_deep
    path = _nt_readlink(path)
OSError: [WinError 1] Неверная функция: 'nul'
```
I think it's a bug. pip uses this code, so 'pip install pandas' won't work in 3.8
msg351578 - (view) Author: Александр Семенов (iamsav) Date: 2019-09-10 06:37
1) Python 3.8.0b4 (tags/v3.8.0b4:d93605d, Aug 29 2019, 23:21:28) [MSC v.1916 64 bit (AMD64)] on win32
2) Looks like identical with https://bugs.python.org/issue1311
msg351583 - (view) Author: Александр Семенов (iamsav) Date: 2019-09-10 07:29
It breaks setuptools.sandbox.DirectorySandbox.__init__() with default param 'exceptions' which includes os.devnull and calls os.path.realpath() on it.
So, many distributions crashes.
msg351725 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2019-09-10 16:28
What does pip use it for?

Applying the below change avoids the exception, but produces \\.\nul as the result, which may or may not be any better.

diff --git a/Lib/ntpath.py b/Lib/ntpath.py
index 1d22d5f1dc..becfa20a83 100644
--- a/Lib/ntpath.py
+++ b/Lib/ntpath.py
@@ -537,7 +537,7 @@ else:
             except OSError as ex:
                 # Stop on file (2) or directory (3) not found, or
                 # paths that are not reparse points (4390)
-                if ex.winerror in (2, 3, 4390):
+                if ex.winerror in (1, 2, 3, 4390):
                     break
                 raise
             except ValueError:
@@ -555,7 +555,7 @@ else:

         # Allow file (2) or directory (3) not found, invalid syntax (123),
         # and symlinks that cannot be followed (1921)
-        allowed_winerror = 2, 3, 123, 1921
+        allowed_winerror = 2, 3, 87, 123, 1921

         # Non-strict algorithm is to find as much of the target directory
         # as we can and join the rest.
msg351764 - (view) Author: Александр Семенов (iamsav) Date: 2019-09-11 05:59
setuptools/sandbox.py:
class DirectorySandbox(AbstractSandbox):
    """Restrict operations to a single subdirectory - pseudo-chroot"""

When running user scripts it uses os.path.realpath(os.devnull) to include 'normalized' devnull to the allowed list of files in pseudo-chroot.

Yes, suggested patch returns realpath behavior from 3.7 and packages installs normally.

C:\Users\User\Downloads>py -3.7 -c "import os.path;print(os.path.realpath('nul'))"
\\.\nul

C:\Users\User\Downloads>py -3.8 -c "import os.path;print(os.path.realpath('nul'))"
\\.\nul

I think it must be included in 3.8 or windows users will get installation problems.
msg351785 - (view) Author: Zachary Ware (zach.ware) * (Python committer) Date: 2019-09-11 09:48
New changeset 92521fea5d0d4aeb9b6a3c3fdd4654af700ad5c8 by Zachary Ware (Steve Dower) in branch 'master':
bpo-38081: Fixes ntpath.realpath('NUL') (GH-15899)
https://github.com/python/cpython/commit/92521fea5d0d4aeb9b6a3c3fdd4654af700ad5c8
msg351787 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2019-09-11 09:50
Thanks for the report!
msg351807 - (view) Author: miss-islington (miss-islington) Date: 2019-09-11 10:43
New changeset 57491de7c33c5886c4cae2f468b474d9b4e6fed2 by Miss Islington (bot) in branch '3.8':
bpo-38081: Fixes ntpath.realpath('NUL') (GH-15899)
https://github.com/python/cpython/commit/57491de7c33c5886c4cae2f468b474d9b4e6fed2
msg351930 - (view) Author: Eryk Sun (eryksun) * (Python triager) Date: 2019-09-11 15:14
We should allow ERROR_INVALID_FUNCTION (1), ERROR_INVALID_PARAMETER (87), and ERROR_NOT_SUPPORTED (50) for readlink and _getfinalpathname, which can indicate a device that does not implement or is not mounted by a file system. We should also allow ERROR_BAD_NET_NAME (67, "the network name cannot be found"), which indicates that a server or share isn't found when opening a UNC path.

I don't know whether ERROR_INVALID_NAME (123) should be allowed. Also, it hasn't been added already, but I'd be equally unsure about adding ERROR_BAD_PATHNAME (161). These aren't like a missing file, path, or server, or an unsupported device. I know Python's realpath() is supposed to be permissive, but that's going too far I think.

Returning r"\\.\nul" is fine. I'd prefer to change os.devnull to match it. Scripts should be able to handle this since already abspath(os.devnull) is r"\\.\nul".
msg352044 - (view) Author: Eryk Sun (eryksun) * (Python triager) Date: 2019-09-12 03:18
In addition to ERROR_INVALID_FUNCTION (1), ERROR_INVALID_PARAMETER (87),  and ERROR_NOT_SUPPORTED (50) for an unsupported device, and ERROR_BAD_NET_NAME (67) for a missing server or share, it should also allow common permission errors:

    ERROR_ACCESS_DENIED (5)
        C:/Users/someone_else
        C:/Temp/deleted_but_still_linked_file
        E:/locked_volume

    ERROR_NOT_READY (21)
        D:/no_media

    ERROR_SHARING_VIOLATION (32)
        C:/pagefile.sys
msg352468 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2019-09-15 08:47
I added another PR with the additional error codes listed by Eryk Sun.

Theoretically we should be able to test most of them, but I haven't written those tests, and I'm not sure they'd prove enough to be worth the extra code. ntpath.realpath is a "best effort" function (realpath in all cases is, I guess), so given the choice between failing and returning a best-effort path, it's obviously better to go with the best-effort path.
msg352548 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2019-09-16 14:25
New changeset 89b8933bb537179f81003928786c5cc6183af591 by Steve Dower in branch 'master':
bpo-38081: Add more non-fatal error codes for ntpath.realpath (GH-16156)
https://github.com/python/cpython/commit/89b8933bb537179f81003928786c5cc6183af591
msg352550 - (view) Author: miss-islington (miss-islington) Date: 2019-09-16 14:43
New changeset 4924d558478c9bd7f7ee7cd9c00c72c0f281f1a5 by Miss Islington (bot) in branch '3.8':
bpo-38081: Add more non-fatal error codes for ntpath.realpath (GH-16156)
https://github.com/python/cpython/commit/4924d558478c9bd7f7ee7cd9c00c72c0f281f1a5
msg352643 - (view) Author: Eryk Sun (eryksun) * (Python triager) Date: 2019-09-17 16:04
Sorry, I mistakenly left out ERROR_BAD_NETPATH (53). It's at least used with mapped drives. For example, I have drive "M:" mapped to WebDAV "//live.sysinternals.com/tools", and I see this error if I disconnect the network:

    >>> try: nt._getfinalpathname('M:/')
    ... except OSError as e: print(e.winerror)
    ...
    53

whereas if I access the underlying UNC path directly the error in this case is ERROR_BAD_NET_NAME (67):

    >>> try: nt._getfinalpathname('//live.sysinternals.com/tools')
    ... except OSError as e: print(e.winerror)
    ...
    67

(Not that this case would normally succeed. A WebDAV share fails the internal request to get a normalized name, as expected for most network providers, but with an unexpected error code that's not handled by the API. It would succeed if we changed _getfinalpathname to fall back on getting the final name as opened, which skips expanding short component names.)

---
Discussion

The Multiple UNC Provider device (i.e. "\??\UNC" -> "\Device\Mup") resolves a UNC path prefix to a network provider (e.g. "Microsoft Windows Network" for SMB) by checking all providers in a registered order until one claims to handle the path. Typically a provider claims the server/share prefix, but the claimed prefix can be just the server, or a variable path length. Typically, if the "server" component isn't found, a provider returns STATUS_BAD_NETWORK_PATH. If the "share" component isn't found, it returns STATUS_BAD_NETWORK_NAME. However, since the request is to MUP, the final status is whatever MUP returns. As far as I can tell, post-Vista MUP prefix resolution returns STATUS_BAD_NETWORK_NAME even if all providers return STATUS_BAD_NETWORK_PATH. 

That said, MUP prefix resolution isn't used for mapped drives. A mapped drive sends the request directly to the provider that created the drive. Prior to Vista, this used to be a top-level named device such as "\Device\LanmanRedirector" (SMB). Since Vista, all redirected create/open requests are routed through MUP, but it doesn't use prefix resolution in this case. It has a sneaky way of implementing this. The provider's device name nowadays is an object SymbolicLink that targets MUP, but with a reserved component that indicates the redirector to use, e.g. "\Device\LanmanRedirector" -> "\Device\Mup\;LanmanRedirector". (A valid server name cannot begin with a semicolon, so this syntax is reserved by MUP. It also supports an optional second reserved component, with the drive name and logon session ID, such as ";Z:0000000000001234". These reserved components are removed from the parsed path, i.e. they are not included in the final path.) Nothing stops us from using this undocumented feature manually in a UNC path, as demonstrated by the examples below. 

The following shows that MUP parses ";LanmanRedirector" as the redirector name, not the "server" component.

    >>> os.path.samefile('//localhost/C$', '//;LanmanRedirector/localhost/C$')
    True

The following shows that an explicit redirector path does not use prefix resolution. This open fails because there's no WebDAV server on localhost.

    >>> try: os.stat('//;WebDavRedirector/localhost/C$')
    ... except OSError as e: print(e.winerror)
    ...
    53

The following shows that MUP fails an open with STATUS_OBJECT_PATH_INVALID (i.e. ERROR_BAD_PATHNAME, 161) if the redirector name is unknown:

    >>> try: os.stat('//;Lanman/localhost/C$')
    ... except OSError as e: print(e.winerror)
    ...
    161

When we misspell the server name as "localhos", we see that the error for an explicit redirector path, as is used in a mapped drive, is ERROR_BAD_NETPATH (53):

    >>> try: os.stat('//;LanmanRedirector/localhos/C$')
    ... except OSError as e: print(e.winerror)
    ...
    53

If we omit the explicit redirector name, then MUP tries prefix resolution, and the error is instead ERROR_BAD_NET_NAME (67):

    >>> try: os.stat('//localhos/C$')
    ... except OSError as e: print(e.winerror)
    ...
    67
History
Date User Action Args
2019-09-17 16:04:28eryksunsetmessages: + msg352643
2019-09-16 14:43:40miss-islingtonsetmessages: + msg352550
2019-09-16 14:26:39steve.dowersetstatus: open -> closed
stage: patch review -> resolved
2019-09-16 14:25:54miss-islingtonsetpull_requests: + pull_request15795
2019-09-16 14:25:15steve.dowersetmessages: + msg352548
2019-09-15 08:47:30steve.dowersetmessages: + msg352468
2019-09-15 08:43:40steve.dowersetstage: needs patch -> patch review
pull_requests: + pull_request15766
2019-09-12 03:18:54eryksunsetmessages: + msg352044
2019-09-11 15:22:49steve.dowersetassignee: steve.dower ->
stage: resolved -> needs patch
versions: + Python 3.9
2019-09-11 15:17:36eryksunsetstatus: closed -> open
2019-09-11 15:14:03eryksunsetmessages: + msg351930
2019-09-11 10:43:33miss-islingtonsetnosy: + miss-islington
messages: + msg351807
2019-09-11 09:50:49steve.dowersetstatus: open -> closed
resolution: fixed
messages: + msg351787

stage: patch review -> resolved
2019-09-11 09:49:57miss-islingtonsetpull_requests: + pull_request15544
2019-09-11 09:48:39zach.waresetmessages: + msg351785
2019-09-11 09:24:48steve.dowersetkeywords: + patch
stage: patch review
pull_requests: + pull_request15540
2019-09-11 05:59:43iamsavsetmessages: + msg351764
2019-09-10 16:28:16steve.dowersetnosy: + eryksun
2019-09-10 16:28:03steve.dowersetassignee: steve.dower
messages: + msg351725
2019-09-10 07:29:46iamsavsetmessages: + msg351583
2019-09-10 06:42:15iamsavsettitle: Different behavior of in 3.7 and 3.8 os.path.realpath('nul') -> Different behavior of os.path.realpath('nul') in 3.7 and 3.8
2019-09-10 06:37:36iamsavsetmessages: + msg351578
2019-09-10 06:34:22xtreaksetnosy: + paul.moore, tim.golden, zach.ware, steve.dower
components: + Windows
2019-09-10 06:30:37iamsavcreate