This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: open() fails to read app exec links
Type: behavior Stage:
Components: Windows Versions: Python 3.8
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: eryksun, paul.moore, saschanaz, steve.dower, tim.golden, zach.ware
Priority: normal Keywords:

Created on 2020-06-20 15:14 by saschanaz, last changed 2022-04-11 14:59 by admin.

Messages (4)
msg371939 - (view) Author: Kagami Sascha Rosylight (saschanaz) Date: 2020-06-20 15:14
After installing Python from Microsoft Store, this fails:

```
>>> open('C:\\Users\\Kagami\\AppData\\Local\\Microsoft\\WindowsApps\\PythonSoftwareFoundation.Python.3.8_qbz5n2kfra8p0\\python.exe')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
OSError: [Errno 22] Invalid argument: 'C:\\Users\\Kagami\\AppData\\Local\\Microsoft\\WindowsApps\\PythonSoftwareFoundation.Python.3.8_qbz5n2kfra8p0\\python.exe'
```

This causes virtualenv to fail on it:

```
INFO: Traceback (most recent call last):
INFO:   File "C:/Users/Kagami/.cargo/git/checkouts/mozjs-fa11ffc7d4f1cc2d/9a6d8fc/mozjs\third_party\python\virtualenv\virtualenv.py", line 2349, in <module>
INFO:     main()
INFO:   File "C:/Users/Kagami/.cargo/git/checkouts/mozjs-fa11ffc7d4f1cc2d/9a6d8fc/mozjs\third_party\python\virtualenv\virtualenv.py", line 703, in main
INFO:     create_environment(home_dir,
INFO:   File "C:/Users/Kagami/.cargo/git/checkouts/mozjs-fa11ffc7d4f1cc2d/9a6d8fc/mozjs\third_party\python\virtualenv\virtualenv.py", line 925, in create_environment
INFO:     py_executable = os.path.abspath(install_python(
INFO:   File "C:/Users/Kagami/.cargo/git/checkouts/mozjs-fa11ffc7d4f1cc2d/9a6d8fc/mozjs\third_party\python\virtualenv\virtualenv.py", line 1239, in install_python
INFO:     shutil.copyfile(executable, py_executable)
INFO:   File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.8_3.8.1008.0_x64__qbz5n2kfra8p0\lib\shutil.py", line 261, in copyfile
INFO:     with open(src, 'rb') as fsrc, open(dst, 'wb') as fdst:
INFO: OSError: [Errno 22] Invalid argument: 'C:\\Users\\Kagami\\AppData\\Local\\Microsoft\\WindowsApps\\PythonSoftwareFoundation.Python.3.8_qbz5n2kfra8p0\\python.exe'
```
msg371963 - (view) Author: Eryk Sun (eryksun) * (Python triager) Date: 2020-06-20 21:02
By design, appexec links (i.e. app execution aliases) cannot be followed automatically. There is no handler for them in the kernel. WinAPI CreateFileW fails with ERROR_CANT_ACCESS_FILE (1920), and the underlying NT status value is STATUS_IO_REPARSE_TAG_NOT_HANDLED (0xC0000279). 

Since 3.8, os.stat handles ERROR_CANT_ACCESS_FILE in all cases by trying to return the result for the reparse point instead. This at least allows getting the st_file_attributes and st_reparse_tag values. For example:

    >>> s = os.stat(sys.executable)
    >>> s.st_file_attributes & stat.FILE_ATTRIBUTE_REPARSE_POINT
    1024
    >>> s.st_reparse_tag == stat.IO_REPARSE_TAG_APPEXECLINK
    True 

CreateProcessW follows app-exec links manually by reading the reparse point. But it's not that simple. The link target under "%ProgramFiles%\WindowsApps" isn't unconditionally executable by standard users. In other words, unless a particular condition is met, trying to execute the target file fails with access denied. Execute access depends on a conditional access-control entry (conditional ACEs are supported in the kernel since Windows 8) that grants access if the user's access token contains a "WIN://SYSAPPID" attribute that identifies the package. Here's the SDDL definition of this ACE for the app distribution of Python 3.9:

    (XA;ID;0x1200a9;;;BU;(WIN://SYSAPPID Contains "PYTHONSOFTWAREFOUNDATION.PYTHON.3.9_QBZ5N2KFRA8P0")

        "XA" is an access-allowed callback (conditional) ACE
        "ID" means the ACE is inherited from the parent directory
        "BU" is the security principal BUILTIN\Users (local group)
        Access Mask 0x1200a9:
            FILE_GENERIC_READ | FILE_GENERIC_EXECUTE:
                SYNCHRONIZE
                READ_CONTROL
                FILE_READ_ATTRIBUTES
                FILE_EXECUTE
                FILE_READ_EA
                FILE_READ_DATA

If the app is installed for the user, CreateProcessW handles the access denied result by creating and impersonating a custom access token to execute the app, which includes the required WIN://SYSAPPID security attribute. You can attach a debugger to see the security attributes added to the app token:

    0:003> !token

    [...]

    Security Attributes Information:
     00 Attribute Name: WIN://SYSAPPID
        Value Type  : TOKEN_SECURITY_ATTRIBUTE_TYPE_STRING
        Value[0]    : PythonSoftwareFoundation.Python.3.9_3.9.179.0_x64__qbz5n2kfra8p0
        Value[1]    : Python
        Value[2]    : PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0

    [...]
msg371969 - (view) Author: Kagami Sascha Rosylight (saschanaz) Date: 2020-06-20 22:35
It seems libuv and pwsh decided to detect and read them just as symlinks:

https://github.com/libuv/libuv/pull/2812
https://github.com/PowerShell/PowerShell/pull/10331

Could Python do the same?
msg372028 - (view) Author: Eryk Sun (eryksun) * (Python triager) Date: 2020-06-22 02:27
Where the POSIX specification uses the term "symbolic link" [1], it means one and only one type of symlink, not multiple types of symlink with divergent behavior depending on the context. To be consistent, only one type of Windows reparse point [2] is classified as a POSIX symlink, the one that's designed to behave like a POSIX symlink in the kernel and API: IO_REPARSE_TAG_SYMLINK. 

It's particularly important that given os.path.islink is true, then os.readlink and os.symlink can create an equivalent copy. This is only implemented for IO_REPARSE_TAG_SYMLINK.

That said, Windows has a variety of filesystem link types, which it calls name-surrogate reparse points. Of this set, in most systems you're only likely to encounter IO_REPARSE_TAG_SYMLINK and IO_REPARSE_TAG_MOUNT_POINT. But Microsoft has a growing list of name-surrogate types, including: IO_REPARSE_TAG_IIS_CACHE, IO_REPARSE_TAG_GLOBAL_REPARSE (NPFS named-pipe symlink from server silo into host silo), IO_REPARSE_TAG_WCI_LINK and IO_REPARSE_TAG_WCI_TOMBSTONE (Windows container isolation), IO_REPARSE_TAG_PROJFS_TOMBSTONE (Projected filesystem tombstone, such as in VFS for Git), IO_REPARSE_TAG_LX_SYMLINK (WSL symlink created on a drvfs volume). In some cases these are used transparently behind the scenes (e.g. tombstones that mark deleted files), or there may be no handler for Windows callers (e.g. WSL symlinks are meaningless in Windows).

Note that the latter list does not include IO_REPARSE_TAG_APPEXECLINK. Even though "LINK" is in the name, this reparse point type is not any kind of filesystem link in practice since it is not handled by the I/O manager or a filter driver in the kernel. As discussed in my first message, all of the intended behavior of an app-exec link is implemented instead by user-mode API functions such as CreateProcessW. In that respect, an app-exec link is more like a shell link (i.e. a LNK file), which gets handled by ShellExecuteExW. I wouldn't expect the standard library to handle LNK files as symlinks.

---

The vast majority of registered reparse-point types are not link types (e.g. cloud files are dehydrated placeholder reparse points). The base Windows API has no special handling for the non-link cases. For example, MoveFileExW opens a non-link reparse point as a regular file. If it operated on the reparse point itself, like it does for a symbolic link, it would ignore the handler and potentially break something.

This is why we can't even rename an app-exec link, since there's no handler for the reparse tag:

    >>> src = os.path.join(os.path.dirname(sys.executable), 'idle3.exe')
    >>> dst = os.path.join(os.path.dirname(sys.executable), 'spam3.exe')
    >>> try: os.rename(src, dst)
    ... except OSError as e: print('winerror:', e.winerror)
    ...
    winerror: 1920

---

The standard library is not limited to just IO_REPARSE_TAG_SYMLINK links. It supports the broader category of Windows name-surrogate links in certain cases. For example, os.lstat doesn't follow them; os.readlink supports symlinks and mountpoints [*]; os.unlink operates on symlinks and mountpoints; and shutil.rmtree doesn't traverse mountpoints (unlike the POSIX implementation).

Caveat emptor regarding os.readlink, however. There are significant differences between symlinks and mountpoints. They're designed to behave like Unix symlinks and bind mountpoints, which have similar behavioral differences. Also, mountpoints are always evaluated on the server, so a remote mountpoint *must* be treated as opaque data [1]:

    The following reparse tags, with the exception of
    IO_REPARSE_TAG_SYMLINK, are processed on the server 
    and are not processed by a client after transmission 
    over the wire. Clients SHOULD treat associated 
    reparse data as opaque data. 

---

[1] https://pubs.opengroup.org/onlinepubs/9699919799/xrat/V4_xbd_chap03.html#tag_21_03_00_75

[2] https://docs.microsoft.com/en-us/openspecs/windows_protocols/ms-fscc/c8e77b37-3909-4fe6-a4ea-2b9d423b1ee4
History
Date User Action Args
2022-04-11 14:59:32adminsetgithub: 85225
2020-06-22 02:27:27eryksunsetmessages: + msg372028
2020-06-20 22:35:00saschanazsetmessages: + msg371969
2020-06-20 21:02:31eryksunsetnosy: + eryksun
messages: + msg371963
2020-06-20 15:14:53saschanazcreate