classification
Title: Controlling venv from venv no longer works in 3.7.2
Type: behavior Stage: resolved
Components: Library (Lib), Windows Versions: Python 3.7
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: paul.moore, schlamar, steve.dower, tim.golden, zach.ware
Priority: normal Keywords: patch, patch

Created on 2019-02-01 09:03 by schlamar, last changed 2019-02-04 15:23 by steve.dower. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 11745 merged steve.dower, 2019-02-03 00:00
PR 11745 merged steve.dower, 2019-02-03 00:01
PR 11753 merged steve.dower, 2019-02-04 07:22
Messages (4)
msg334659 - (view) Author: Marc Schlaich (schlamar) * Date: 2019-02-01 09:03
Controlling a venv from the python.exe from another venv does not work since 3.7.2 on Windows. This is probably related to the change

    bpo-34977: venv on Windows will now use a python.exe redirector rather than copying the actual binaries from the base environment.

This is obviously related to bpo-35872, but this could be a different bug.

When a Python script in a venv wants to control another venv by running commands like `another-venv\python.exe -m pip` with subprocess, python.exe is not referencing the other venv. It is referencing to the venv the script currently running from. This is probably because os.environ contains a '__PYVENV_LAUNCHER__' which is pointing to the venv from the script.

This can be reproduced with pipx (https://github.com/pipxproject/pipx-app) by running

    pipx install --python "C:\Program Files (x86)\Python37-32\python.exe" --verbose tox

This results in pip installing to venvs\pipx-app and not in venvs\tox. 


I assume a simpler reproduction might be (but I cannot check this anymore as I'm back on 3.7.1 right now):

    C:\Program Files (x86)\Python37-32\python.exe -m venv C:\Users\$USER\.local\pipx\venvs\tox
    c:\users\$USER\.local\pipx\venvs\pipx-app\scripts\python.exe -c "import subprocess; subprocess.run(['C:\Users\$USER\.local\pipx\venvs\tox\Scripts\python.exe', '-m', 'pip', 'install', 'tox'])"


Downstream bugreport in pipx is https://github.com/pipxproject/pipx-app/issues/81.
msg334774 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2019-02-02 22:49
So with the fix for multiprocessing, we currently rely on __PYVENV_LAUNCHER__ remaining set throughout the process.

However, it may be better to add a "sys.base_executable" property instead and clear the __PYVENV_LAUNCHER__ variable once we've read it. Then we can set it again in multiprocessing and launch the base executable, and otherwise default to launching the redirector.
msg334813 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2019-02-04 07:19
New changeset a8474d025cab794257d2fd0bea67840779b9351f by Steve Dower in branch 'master':
bpo-35872 and bpo-35873: Clears __PYVENV_LAUNCHER__ variable (GH-11745)
https://github.com/python/cpython/commit/a8474d025cab794257d2fd0bea67840779b9351f
msg334829 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2019-02-04 15:20
New changeset 44467e8ea4cea390b0718702291b4cfe8ddd67ed by Steve Dower in branch '3.7':
bpo-35872 and bpo-35873: Clears __PYVENV_LAUNCHER__ variable (GH-11745)
https://github.com/python/cpython/commit/44467e8ea4cea390b0718702291b4cfe8ddd67ed
History
Date User Action Args
2019-02-04 15:23:13steve.dowersetkeywords: patch, patch
status: open -> closed
resolution: fixed
stage: patch review -> resolved
2019-02-04 15:20:38steve.dowersetmessages: + msg334829
2019-02-04 07:22:45steve.dowersetpull_requests: + pull_request11692
2019-02-04 07:19:56steve.dowersetmessages: + msg334813
2019-02-03 00:01:07steve.dowersetkeywords: + patch
stage: patch review
pull_requests: + pull_request11671
2019-02-03 00:00:57steve.dowersetkeywords: + patch
stage: (no value)
pull_requests: + pull_request11670
2019-02-02 22:49:23steve.dowersetmessages: + msg334774
2019-02-01 09:03:04schlamarcreate