classification
Title: Startup failure if executable is a \\?\ path on Windows
Type: crash Stage: resolved
Components: Interpreter Core, Windows Versions: Python 3.7, Python 3.6
process
Status: closed Resolution: fixed
Dependencies: Superseder: Azure Pipelines: appx tests fail: init_fs_encoding: failed to get the Python codec of the filesystem encoding
View: 38322
Assigned To: Nosy List: eryksun, paul.moore, shubha_bloodhound, steve.dower, tim.golden, vstinner, zach.ware
Priority: normal Keywords:

Created on 2017-07-29 06:36 by eryksun, last changed 2019-10-22 23:18 by vstinner. This issue is now closed.

Messages (4)
msg299459 - (view) Author: Eryk Sun (eryksun) * (Python triager) Date: 2017-07-29 06:35
search_for_prefix in PC/getpathp.c sets the wrong path when Python is started with a \\?\ path on Windows, which results in the following crash:

    >>> subprocess.call(r'"\\?\C:\Program Files\Python36\python.exe"')
    Fatal Python error: Py_Initialize: unable to load the file system codec
    ModuleNotFoundError: No module named 'encodings'

    Current thread 0x00000b40 (most recent call first):
    3221226505

The problem is due to the implementation of join(), which calls PathCchCombineEx, which strips the \\?\ prefix from the joined path. Consequently in gotlandmark(), setting `prefix[n] = '\0'` nulls the wrong index because the value of n includes the original \\?\ prefix. 

    Breakpoint 0 hit
    python36_d!gotlandmark:
    00000000`6cc1a920 48894c2408      mov     qword ptr [rsp+8],rcx
            ss:000000c2`9a9ee920={python36_d!prefix (00000000`6cf697d0)}
    1:005> du python36_d!prefix
    00000000`6cf697d0  "\\?\C:\Program Files\Python36"

    1:005> pc
    python36_d!gotlandmark+0x15:
    00000000`6cc1a935 e8e60a0000      call    python36_d!wcsnlen_s (00000000`6cc1b420)
    1:005> pc
    python36_d!gotlandmark+0x2b:
    00000000`6cc1a94b e890010000      call    python36_d!join (00000000`6cc1aae0)
    1:005> p
    python36_d!gotlandmark+0x30:
    00000000`6cc1a950 33d2            xor     edx,edx
    1:005> du python36_d!prefix
    00000000`6cf697d0  "C:\Program Files\Python36\lib\os"
    00000000`6cf69810  ".py"

    1:005> pt
    python36_d!gotlandmark+0x7b:
    00000000`6cc1a99b c3              ret
    1:005> du python36_d!prefix
    00000000`6cf697d0  "C:\Program Files\Python36\lib"

I think the simplest solution is to remove the \\?\ prefix from the executable path that's returned by GetModuleFileNameW in get_progpath(). AFAIK, no version of Windows can reliably run programs from long paths, and the current MAXPATHLEN (256) doesn't allow it anyway. The only reason to use \\?\ for the executable path would be to avoid the normal DOS/Windows path normalization rules -- e.g. removing trailing dots and spaces or using DOS device names such as "con.exe". That's not really a practical concern.
msg299467 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2017-07-29 13:49
Sounds like the right fix. We should increase the maximum path length though, since 3.6 and later can work with much longer paths on Win10 through normal APIs and without prefix.
msg299487 - (view) Author: Shubha Ramani (shubha_bloodhound) Date: 2017-07-29 21:44
May I take resolve this issue and submit a patch ?
msg355177 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-10-22 23:18
I got the same bug recently and I fixed it:

commit dec39716ca93ee2e8d9b94915ece33014eb58e9e
Author: Victor Stinner <vstinner@python.org>
Date:   Mon Sep 30 14:49:34 2019 +0200

    bpo-38322: Fix gotlandmark() of PC/getpathp.c (GH-16489)
    
    Write the filename into a temporary buffer instead of reusing prefix.
    The problem is that join() modifies prefix inplace. If prefix is not
    normalized, join() can make prefix shorter and so gotlandmark()
    does modify prefix instead of returning it unmodified.
History
Date User Action Args
2019-10-22 23:18:35vstinnersetstatus: open -> closed

superseder: Azure Pipelines: appx tests fail: init_fs_encoding: failed to get the Python codec of the filesystem encoding

nosy: + vstinner
messages: + msg355177
resolution: fixed
stage: test needed -> resolved
2017-07-29 21:44:26shubha_bloodhoundsetnosy: + shubha_bloodhound
messages: + msg299487
2017-07-29 13:49:27steve.dowersetmessages: + msg299467
2017-07-29 06:36:00eryksuncreate