Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

EnvBuilder and venv symlinks do not work on Windows on 3.7.2 #80035

Closed
zooba opened this issue Jan 29, 2019 · 10 comments
Closed

EnvBuilder and venv symlinks do not work on Windows on 3.7.2 #80035

zooba opened this issue Jan 29, 2019 · 10 comments
Assignees
Labels
3.7 (EOL) end of life 3.8 only security fixes OS-windows

Comments

@zooba
Copy link
Member

zooba commented Jan 29, 2019

BPO 35854
Nosy @pfmoore, @vsajip, @tjguk, @ned-deily, @zware, @eryksun, @zooba, @miss-islington
PRs
  • bpo-35854: Fix EnvBuilder and ignore --symlinks in venv on Windows #11700
  • bpo-35854: Fix EnvBuilder and ignore --symlinks in venv on Windows #11700
  • bpo-35854: Fix EnvBuilder and ignore --symlinks in venv on Windows #11700
  • [3.7] bpo-35854: Fix EnvBuilder and --symlinks in venv on Windows (GH-11700) #11707
  • [3.7] bpo-35854: Fix EnvBuilder and --symlinks in venv on Windows (GH-11700) #11707
  • Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = 'https://github.com/zooba'
    closed_at = <Date 2019-03-12.15:53:40.821>
    created_at = <Date 2019-01-29.23:13:14.466>
    labels = ['3.7', '3.8', 'OS-windows']
    title = 'EnvBuilder and venv symlinks do not work on Windows on 3.7.2'
    updated_at = <Date 2019-03-12.15:53:40.821>
    user = 'https://github.com/zooba'

    bugs.python.org fields:

    activity = <Date 2019-03-12.15:53:40.821>
    actor = 'steve.dower'
    assignee = 'steve.dower'
    closed = True
    closed_date = <Date 2019-03-12.15:53:40.821>
    closer = 'steve.dower'
    components = ['Windows']
    creation = <Date 2019-01-29.23:13:14.466>
    creator = 'steve.dower'
    dependencies = []
    files = []
    hgrepos = []
    issue_num = 35854
    keywords = ['patch', 'patch', 'patch', '3.7regression']
    message_count = 10.0
    messages = ['334537', '334538', '334545', '334561', '334574', '334596', '334597', '334598', '337721', '337762']
    nosy_count = 8.0
    nosy_names = ['paul.moore', 'vinay.sajip', 'tim.golden', 'ned.deily', 'zach.ware', 'eryksun', 'steve.dower', 'miss-islington']
    pr_nums = ['11700', '11700', '11700', '11707', '11707']
    priority = 'normal'
    resolution = 'fixed'
    stage = 'resolved'
    status = 'closed'
    superseder = None
    type = None
    url = 'https://bugs.python.org/issue35854'
    versions = ['Python 3.7', 'Python 3.8']

    @zooba
    Copy link
    Member Author

    zooba commented Jan 29, 2019

    The change to pull the redirector executables as part of scripts was... perhaps too clever. There exists code out there that uses EnvBuilder to create environments _without_ copying scripts, which now results in an environment that does not include python.exe.

    It also undid symlink support, as scripts are never symlinked.

    I'll restore both.

    @zooba zooba added 3.7 (EOL) end of life 3.8 only security fixes labels Jan 29, 2019
    @zooba zooba self-assigned this Jan 29, 2019
    @zooba
    Copy link
    Member Author

    zooba commented Jan 29, 2019

    Actually, it seems like venv symlinks are so far from working that they haven't worked (on Windows) since 3.5 or possibly earlier.

    I'm just going to make that option ignored and mark it as such in the docs.

    @eryksun
    Copy link
    Contributor

    eryksun commented Jan 30, 2019

    Actually, it seems like venv symlinks are so far from working
    that they haven't worked (on Windows) since 3.5 or possibly
    earlier.

    I use venv with --symlinks prior to 3.7.2. It works fine as far as I can tell. Perhaps you could elaborate. I assumed you removed it because it's not compatible with the Microsoft Store release for some reason.

    The loader doesn't resolve links, so it should work as long as the python.exe, python3x.dll, python3.dll, and vcruntime140.dll files are linked together in the Scripts directory.

        >>> hPython3 = ctypes.WinDLL('python3')._handle
        >>> hVcruntime140 = ctypes.WinDLL('vcruntime140')._handle
        >>> for h in (0, sys.dllhandle, hPython3, hVcruntime140):
        ...     print(_winapi.GetModuleFileName(h), '->')
        ...     print('\t', os.readlink(_winapi.GetModuleFileName(h)))
        ...
        C:\Temp\env36\scripts\python.exe ->
                 C:\Program Files\Python36\python.exe
        C:\Temp\env36\scripts\python36.dll ->
                 C:\Program Files\Python36\python36.dll
        C:\Temp\env36\scripts\python3.dll ->
                 C:\Program Files\Python36\python3.dll
        C:\Temp\env36\scripts\VCRUNTIME140.dll ->
                 C:\Program Files\Python36\vcruntime140.dll
    
        >>> print(sys.executable)
        C:\Temp\env36\scripts\python.exe
    >>> print(*sys.path, sep='\n')
    
    C:\Temp\env36\scripts\python36.zip
    C:\Program Files\Python36\DLLs
    C:\Program Files\Python36\lib
    C:\Program Files\Python36
    C:\Temp\env36
    C:\Temp\env36\lib\site-packages
    

    @zooba
    Copy link
    Member Author

    zooba commented Jan 30, 2019

    No, removing it was accidental. But when I tried it myself with a few regular installs of 3.5 and later, none of them updated sys.path correctly, and all of them resolved the links in the loader (or perhaps the shell?).

    I hadn't activated the environments, as I generally don't, but if that's required for symlinks to work here then I consider them broken.

    @zooba
    Copy link
    Member Author

    zooba commented Jan 30, 2019

    Okay, testing more thoroughly on 3.5, symlinks are fine from the console but not via Explorer.

    Personally, I dislike that double-clicking python.exe is different from running it from the command line, but so be it. I'll add a note to the docs that this doesn't work.

    @eryksun
    Copy link
    Contributor

    eryksun commented Jan 30, 2019

    Okay, testing more thoroughly on 3.5, symlinks are fine from the
    console but not via Explorer.

    I gave up on using EXE symlinks with Explorer and ShellExecute[Ex]. The shell handles symlinks like shortcuts instead of leaving it up to the file system. In Windows 7, I recall it was thoroughly broken. Symlinks wouldn't even execute. In Windows 10 it's broken in many cases because of the shortcut-like behavior.

    I think I know why they're doing this, but I think the fix belongs at a lower level in the system runtime library and loader. It's to accommodate the importance of the application directory. People expect a symlink to an executable to work like a shortcut. In Unix this often just works because most libraries are installed to the system, and an application's private shared libraries can use "$ORIGIN" in the binary's RPATH (or RUNPATH), which refers to the resolved (final) executable directory. In contrast, Windows doesn't use the resolved executable path for the application directory. IMO, they could use a pair of application directories at the start of the search path -- the link's directory and then the resolved executable's directory (or the DLL directory when loading a DLL). Some resources do get resolved like this already. For example, when searching for a "<local name>\<exename>.mui" language resource, in Process Monitor we see that it will first try the unresolved application directory (e.g. "C:\Temp\en-US\notepad.exe.mui") and then the resolved path (e.g. "C:\Windows\en-US\notepad.exe.mui").

    @zooba
    Copy link
    Member Author

    zooba commented Jan 30, 2019

    New changeset a1f9a33 by Steve Dower in branch 'master':
    bpo-35854: Fix EnvBuilder and --symlinks in venv on Windows (GH-11700)
    a1f9a33

    @miss-islington
    Copy link
    Contributor

    New changeset 03082a8 by Miss Islington (bot) in branch '3.7':
    bpo-35854: Fix EnvBuilder and --symlinks in venv on Windows (GH-11700)
    03082a8

    @ned-deily
    Copy link
    Member

    Can we close this issue now?

    @zooba
    Copy link
    Member Author

    zooba commented Mar 12, 2019

    Yep. Done

    @zooba zooba closed this as completed Mar 12, 2019
    @ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
    Labels
    3.7 (EOL) end of life 3.8 only security fixes OS-windows
    Projects
    None yet
    Development

    No branches or pull requests

    4 participants