classification
Title: EnvBuilder and venv symlinks do not work on Windows on 3.7.2
Type: Stage: resolved
Components: Windows Versions: Python 3.8, Python 3.7
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: steve.dower Nosy List: eryksun, miss-islington, ned.deily, paul.moore, steve.dower, tim.golden, vinay.sajip, zach.ware
Priority: normal Keywords: 3.7regression, patch, patch, patch

Created on 2019-01-29 23:13 by steve.dower, last changed 2019-03-12 15:53 by steve.dower. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 11700 merged steve.dower, 2019-01-29 23:47
PR 11700 merged steve.dower, 2019-01-29 23:47
PR 11700 merged steve.dower, 2019-01-29 23:47
PR 11707 merged miss-islington, 2019-01-30 21:49
PR 11707 merged miss-islington, 2019-01-30 21:49
Messages (10)
msg334537 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2019-01-29 23:13
The change to pull the redirector executables as part of scripts was... perhaps too clever. There exists code out there that uses EnvBuilder to create environments _without_ copying scripts, which now results in an environment that does not include python.exe.

It also undid symlink support, as scripts are never symlinked.

I'll restore both.
msg334538 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2019-01-29 23:27
Actually, it seems like venv symlinks are so far from working that they haven't worked (on Windows) since 3.5 or possibly earlier.

I'm just going to make that option ignored and mark it as such in the docs.
msg334545 - (view) Author: Eryk Sun (eryksun) * (Python triager) Date: 2019-01-30 05:48
> Actually, it seems like venv symlinks are so far from working 
> that they haven't worked (on Windows) since 3.5 or possibly 
> earlier.

I use venv with --symlinks prior to 3.7.2. It works fine as far as I can tell. Perhaps you could elaborate. I assumed you removed it because it's not compatible with the Microsoft Store release for some reason. 

The loader doesn't resolve links, so it should work as long as the python.exe, python3x.dll, python3.dll, and vcruntime140.dll files are linked together in the Scripts directory. 

    >>> hPython3 = ctypes.WinDLL('python3')._handle
    >>> hVcruntime140 = ctypes.WinDLL('vcruntime140')._handle
    >>> for h in (0, sys.dllhandle, hPython3, hVcruntime140):
    ...     print(_winapi.GetModuleFileName(h), '->')
    ...     print('\t', os.readlink(_winapi.GetModuleFileName(h)))
    ...
    C:\Temp\env36\scripts\python.exe ->
             C:\Program Files\Python36\python.exe
    C:\Temp\env36\scripts\python36.dll ->
             C:\Program Files\Python36\python36.dll
    C:\Temp\env36\scripts\python3.dll ->
             C:\Program Files\Python36\python3.dll
    C:\Temp\env36\scripts\VCRUNTIME140.dll ->
             C:\Program Files\Python36\vcruntime140.dll

    >>> print(sys.executable)
    C:\Temp\env36\scripts\python.exe

    >>> print(*sys.path, sep='\n')

    C:\Temp\env36\scripts\python36.zip
    C:\Program Files\Python36\DLLs
    C:\Program Files\Python36\lib
    C:\Program Files\Python36
    C:\Temp\env36
    C:\Temp\env36\lib\site-packages
msg334561 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2019-01-30 14:27
No, removing it was accidental. But when I tried it myself with a few regular installs of 3.5 and later, none of them updated sys.path correctly, and all of them resolved the links in the loader (or perhaps the shell?).

I hadn't activated the environments, as I generally don't, but if that's required for symlinks to work here then I consider them broken.
msg334574 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2019-01-30 17:19
Okay, testing more thoroughly on 3.5, symlinks are fine from the console but not via Explorer.

Personally, I dislike that double-clicking python.exe is different from running it from the command line, but so be it. I'll add a note to the docs that this doesn't work.
msg334596 - (view) Author: Eryk Sun (eryksun) * (Python triager) Date: 2019-01-30 21:38
> Okay, testing more thoroughly on 3.5, symlinks are fine from the 
> console but not via Explorer.

I gave up on using EXE symlinks with Explorer and ShellExecute[Ex]. The shell handles symlinks like shortcuts instead of leaving it up to the file system. In Windows 7, I recall it was thoroughly broken. Symlinks wouldn't even execute. In Windows 10 it's broken in many cases because of the shortcut-like behavior.

I think I know why they're doing this, but I think the fix belongs at a lower level in the system runtime library and loader. It's to accommodate the importance of the application directory. People expect a symlink to an executable to work like a shortcut. In Unix this often just works because most libraries are installed to the system, and an application's private shared libraries can use "$ORIGIN" in the binary's RPATH (or RUNPATH), which refers to the resolved (final) executable directory. In contrast, Windows doesn't use the resolved executable path for the application directory. IMO, they could use a pair of application directories at the start of the search path -- the link's directory and then the resolved executable's directory (or the DLL directory when loading a DLL). Some resources do get resolved like this already. For example, when searching for a "<local name>\<exename>.mui" language resource, in Process Monitor we see that it will first try the unresolved application directory (e.g. "C:\Temp\en-US\notepad.exe.mui") and then the resolved path (e.g. "C:\Windows\en-US\notepad.exe.mui").
msg334597 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2019-01-30 21:49
New changeset a1f9a3332bd4767e47013ea787022f06b6dbcbbd by Steve Dower in branch 'master':
bpo-35854: Fix EnvBuilder and --symlinks in venv on Windows (GH-11700)
https://github.com/python/cpython/commit/a1f9a3332bd4767e47013ea787022f06b6dbcbbd
msg334598 - (view) Author: miss-islington (miss-islington) Date: 2019-01-30 22:14
New changeset 03082a836b707528f885080bda9732d89849d4e3 by Miss Islington (bot) in branch '3.7':
bpo-35854: Fix EnvBuilder and --symlinks in venv on Windows (GH-11700)
https://github.com/python/cpython/commit/03082a836b707528f885080bda9732d89849d4e3
msg337721 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2019-03-12 04:56
Can we close this issue now?
msg337762 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2019-03-12 15:53
Yep. Done
History
Date User Action Args
2019-03-12 15:53:40steve.dowersetstatus: open -> closed
messages: + msg337762

keywords: patch, patch, patch, 3.7regression
resolution: fixed
stage: patch review -> resolved
2019-03-12 04:56:04ned.deilysetkeywords: patch, patch, patch, 3.7regression
nosy: + ned.deily
messages: + msg337721

2019-01-30 22:14:42miss-islingtonsetnosy: + miss-islington
messages: + msg334598
2019-01-30 21:49:29miss-islingtonsetpull_requests: + pull_request11563
2019-01-30 21:49:27miss-islingtonsetpull_requests: + pull_request11562
2019-01-30 21:49:16steve.dowersetmessages: + msg334597
2019-01-30 21:38:25eryksunsetkeywords: patch, patch, patch, 3.7regression

messages: + msg334596
2019-01-30 17:19:54steve.dowersetkeywords: patch, patch, patch, 3.7regression

messages: + msg334574
2019-01-30 14:27:47steve.dowersetkeywords: patch, patch, patch, 3.7regression

messages: + msg334561
2019-01-30 05:48:36eryksunsetkeywords: patch, patch, patch, 3.7regression

messages: + msg334545
2019-01-29 23:48:01steve.dowersetkeywords: + patch
stage: patch review
pull_requests: + pull_request11550
2019-01-29 23:47:47steve.dowersetkeywords: + patch
stage: (no value)
pull_requests: + pull_request11549
2019-01-29 23:47:36steve.dowersetkeywords: + patch
stage: (no value)
pull_requests: + pull_request11548
2019-01-29 23:27:51steve.dowersetmessages: + msg334538
2019-01-29 23:13:14steve.dowercreate