classification
Title: Creating venv from venv no longer works in 3.7.2
Type: behavior Stage: resolved
Components: Library (Lib), Windows Versions: Python 3.7
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: eryksun, paul.moore, schlamar, steve.dower, tim.golden, vinay.sajip, zach.ware
Priority: normal Keywords: patch, patch, patch

Created on 2019-02-01 08:49 by schlamar, last changed 2019-03-26 15:17 by paul.moore. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 11745 merged steve.dower, 2019-02-03 00:00
PR 11745 merged steve.dower, 2019-02-03 00:00
PR 11745 merged steve.dower, 2019-02-03 00:00
PR 11753 merged steve.dower, 2019-02-04 07:22
PR 11753 merged steve.dower, 2019-02-04 07:22
PR 11753 merged steve.dower, 2019-02-04 07:22
Messages (12)
msg334658 - (view) Author: Marc Schlaich (schlamar) * Date: 2019-02-01 08:49
Creating a venv from the python.exe from another venv does not work since 3.7.2 on Windows. This is probably related to the change

    bpo-34977: venv on Windows will now use a python.exe redirector rather than copying the actual binaries from the base environment.

For example running 

    c:\users\$USER\.local\pipx\venvs\pipx-app\scripts\python.exe -m venv C:\Users\$USER\.local\pipx\venvs\tox
    C:\Users\$USER\.local\pipx\venvs\tox\Scripts\python.exe -m pip install --upgrade pip

results in pip installing to venvs\pipx-app and not in venvs\tox. 

Downstream bugreport in pipx is https://github.com/pipxproject/pipx-app/issues/81.
msg334674 - (view) Author: Eryk Sun (eryksun) * (Python triager) Date: 2019-02-01 13:17
The __PYVENV_LAUNCHER__ variable gets set by each launcher instance. We get one launcher process for every level of nesting, and the last launcher to run sets the final value that will be seen by the real python.exe. Possibly the launcher could resolve the home values until it reaches a solid python.exe (i.e. no pyvenv.cfg file), and thus avoid the nested chain of launcher processes and ensure that the real python.exe sees the correct value of __PYVENV_LAUNCHER__.
msg334773 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2019-02-02 22:44
This scenario should work, as running the other venv's redirector will update the variable.

The nearly identical report in issue35873 is apparently launching "default" Python, but presumably without clearing the variable. Since I can't tell what's going wrong with the report in this bug, I'm going to focus on the other one.
msg334776 - (view) Author: Eryk Sun (eryksun) * (Python triager) Date: 2019-02-02 23:04
> This scenario should work, as running the other venv's redirector 
> will update the variable.

The order gets reversed. In the simple case where we have two launchers, the launcher for the nested virtual environment executes the launcher for the outer (creator) virtual environment, which executes the real python.exe. So Python sees the wrong value for __PYVENV_LAUNCHER__. For example, if I create env37_2 from env37_1, here's the result in env37_2:

    C:\>C:\Temp\test\env37_2\scripts\python.exe
    Python 3.7.2 (tags/v3.7.2:9a3ffc0492, Dec 23 2018, 23:09:28)
    [MSC v.1916 64 bit (AMD64)] on win32
    Type "help", "copyright", "credits" or "license" for more information.
    >>> import sys, os
    >>> print(sys.executable)
    C:\Temp\Test\env37_1\scripts\python.exe

    >>> print(*sys.path, sep='\n')

    C:\Program Files\Python37\python37.zip
    C:\Program Files\Python37\DLLs
    C:\Program Files\Python37\lib
    C:\Program Files\Python37
    C:\Temp\Test\env37_1
    C:\Temp\Test\env37_1\lib\site-packages

    >>> print(os.environ['__PYVENV_LAUNCHER__'])
    C:\Temp\Test\env37_1\scripts\python.exe
msg334777 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2019-02-02 23:07
You can't actually nest virtual environments, at least with venv - it ought to be configuring it all against the original environment.

What is in your pyvenv.cfg files here?
msg334778 - (view) Author: Eryk Sun (eryksun) * (Python triager) Date: 2019-02-02 23:37
> You can't actually nest virtual environments, at least with venv

Yes, they can't be nested in terms of inheriting site-packages. But the "home" value in this case references the creating environment, which hasn't changed from previous versions. It's the same in Linux. In my first message I suggested having the launcher resolve pyvenv.cfg files until it finds the real python.exe to execute (the first one with no pyvenv.cfg file), thus bypassing the intervening launchers.

> What is in your pyvenv.cfg files here?

They're as expected:

    C:\>type C:\Temp\test\env37_1\pyvenv.cfg
    home = C:\Program Files\Python37
    include-system-site-packages = false
    version = 3.7.2

    C:\>type C:\Temp\test\env37_2\pyvenv.cfg
    home = C:\Temp\Test\env37_1\scripts
    include-system-site-packages = false
    version = 3.7.2
msg334779 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2019-02-02 23:44
Is there any reason to not resolve the base executable on creation and make it relative to that? So the second pyvenv.cfg would have the same home directory as the first?

My proposed fix for issue35873 (PR soon) is going to make this trivial.
msg334782 - (view) Author: Eryk Sun (eryksun) * (Python triager) Date: 2019-02-03 02:16
> Is there any reason to not resolve the base executable on creation and 
> make it relative to that? So the second pyvenv.cfg would have the same 
> home directory as the first?

I was trying to avoid changing the existing behavior of `home` from how it works in 3.6. Your solution will introduce a (probably minor) change, which may be for the best anyway. The computed argv0_path gets added to sys.path, which in the normal case is the installation directory, but it gets overridden by `home`. In 3.6, in the nested-create case, the argv0_path is the creating environment's Scripts directory. For example: 

    C:\>type C:\Temp\test\env36_2\pyvenv.cfg
    home = C:\Temp\Test\env36_1\scripts
    include-system-site-packages = false
    version = 3.6.4

    C:\>C:\Temp\test\env36_2\Scripts\python.exe -S
    Python 3.6.4 (v3.6.4:d48eceb, Dec 19 2017, 06:54:40)
    [MSC v.1900 64 bit (AMD64)] on win32
    >>> import sys
    >>> print(*sys.path, sep='\n')

    C:\Temp\test\env36_2\Scripts\python36.zip
    C:\Program Files\Python36\Lib\
    C:\Program Files\Python36\DLLs\
    C:\Temp\Test\env36_1\scripts

I don't know why we add argv0_path to sys.path. If it's meant to be the directory of the (not necessarily real) executable, then the last path in the above example should be "C:\Temp\Test\env36_2\Scripts" instead of "env36_1\Scripts". If the argv0_path is meant to be the installation directory, as part of the standard library, then it should be "C:\Program Files\Python36". PR 11745 will make it consistently the latter for all virtual environments.
msg334786 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2019-02-03 04:29
> I don't know why we add argv0_path to sys.path.

My guess is that the idea was to be able to load the .pyd files we used to put alongside it, though we found out they weren't being loaded anyway.

One potential side effect is that manually created .pth files may no longer have an effect, as those likely chain together just fine. But I don't think that's a likely scenario.
msg334812 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2019-02-04 07:19
New changeset a8474d025cab794257d2fd0bea67840779b9351f by Steve Dower in branch 'master':
bpo-35872 and bpo-35873: Clears __PYVENV_LAUNCHER__ variable (GH-11745)
https://github.com/python/cpython/commit/a8474d025cab794257d2fd0bea67840779b9351f
msg334828 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2019-02-04 15:20
New changeset 44467e8ea4cea390b0718702291b4cfe8ddd67ed by Steve Dower in branch '3.7':
bpo-35872 and bpo-35873: Clears __PYVENV_LAUNCHER__ variable (GH-11745)
https://github.com/python/cpython/commit/44467e8ea4cea390b0718702291b4cfe8ddd67ed
msg338892 - (view) Author: Paul Moore (paul.moore) * (Python committer) Date: 2019-03-26 15:17
See https://github.com/pypa/virtualenv/issues/1339 - it's possible that something in this area is still affecting virtualenv.
History
Date User Action Args
2019-03-26 15:17:30paul.mooresetkeywords: patch, patch, patch

messages: + msg338892
2019-02-24 10:39:07eryksunlinkissue36080 superseder
2019-02-04 15:23:21steve.dowersetkeywords: patch, patch, patch
status: open -> closed
resolution: fixed
stage: patch review -> resolved
2019-02-04 15:20:23steve.dowersetmessages: + msg334828
2019-02-04 07:22:57steve.dowersetpull_requests: + pull_request11691
2019-02-04 07:22:45steve.dowersetpull_requests: + pull_request11690
2019-02-04 07:22:33steve.dowersetpull_requests: + pull_request11689
2019-02-04 07:19:40steve.dowersetmessages: + msg334812
2019-02-03 04:29:59steve.dowersetkeywords: patch, patch, patch

messages: + msg334786
2019-02-03 02:16:00eryksunsetkeywords: patch, patch, patch

messages: + msg334782
2019-02-03 00:01:11steve.dowersetkeywords: + patch
stage: patch review
pull_requests: + pull_request11669
2019-02-03 00:00:57steve.dowersetkeywords: + patch
stage: (no value)
pull_requests: + pull_request11668
2019-02-03 00:00:44steve.dowersetkeywords: + patch
stage: (no value)
pull_requests: + pull_request11667
2019-02-02 23:44:14steve.dowersetnosy: + vinay.sajip
messages: + msg334779
2019-02-02 23:37:56eryksunsetmessages: + msg334778
2019-02-02 23:07:23steve.dowersetmessages: + msg334777
2019-02-02 23:04:11eryksunsetmessages: + msg334776
2019-02-02 22:44:09steve.dowersetmessages: + msg334773
2019-02-01 13:17:23eryksunsetnosy: + eryksun
messages: + msg334674
2019-02-01 08:49:13schlamarcreate