Author eryksun
Recipients eryksun, paul.moore, steve.dower, tim.golden, vinay.sajip, zach.ware
Date 2019-01-24.00:48:15
SpamBayes Score -1.0
Marked as misclassified Yes
Message-id <1548290896.4.0.108661661457.issue35811@roundup.psfhosted.org>
In-reply-to
Content
In 3.7.2 on Windows, venv now uses a redirecting launcher 'script' for python[w].exe. This replaces (on Windows only) the setup requirement to copy or symlink the interpreter binaries (i.e. python[w].exe, python37.dll, and vcruntime140.dll). Apparently this is required to be able to install Python as a Windows Store app. I haven't experimented with it yet, so I'll just accept that as a given. I'm curious to find out whether using symlinks would still work, and even if it doesn't work for the app installation, whether we could still support that option for desktop installations. I've used `--symlinks` for a while now, since I grant the required privilege to the authenticated users group and thus don't have to elevate to create symlinks. (I know it's not so easy for many Windows users.)

The new launcher reads pyvenv.cfg to locate and execute the real python.exe. Since the process image is no longer in the virtual environment's "Scripts" directory (which has various consequences), we need a way to communicate the launcher's path to make Python use the virtual environment. A -X command-line option could work, but then all packages and tools that spawn worker processes, such as multiprocessing, would need to be updated to look for this option in sys._xoptions and propagate it. Instead the launcher sets a special "__PYVENV_LAUNCHER__" environment variable. This is reasonable because processes are usually created with a copy of the parent's environment. Some environment variables may be added or modified, but it's rare for a child process to get a completely new environment. (One example of the latter would be creating a process that runs as a different user.)

An oversight in the current ecosystem is that py.exe and the distlib entry-point launchers do not unset "__PYVENV_LAUNCHER__". Thus, when executing a script from a virtual environment (e.g. either directly via py.exe or via the .py file association), it will mistakenly be pinned into the virtual environment if it runs in Python 3.7. Similarly, pip.exe for an installed Python 3.7 will mistakenly install into the virtual environment. However, the latter is out of scope here since the entry-point launchers are in distlib. 

It's also a problem if we run the fully-qualified path for an installed Python 3.7, e.g. from shutil.which('python'). We can't automatically address this since it's exactly the reason "__PYVENV_LAUNCHER__" exists. We have to know to manually unset "__PYVENV_LAUNCHER__" in the environment that's passed to the child. This should be documented somewhere -- maybe in the venv docs.

It's not a problem if a script runs unqualified "python.exe" since the final result is usually the same, but for different reasons. With the launcher, it's locked down by inheriting "__PYVENV_LAUNCHER__". With the previous design, the application directory was the virtual environment's "Scripts" directory. Unqualified "python.exe" was thus pinned for CreateProcessW, which checks the application directory first. It's also not a problem if we run sys.executable, since previously that was the virtual environment's executable. (In 3.7.2, sys.executable gets set to the virtual environment's launcher, but that breaks multiprocessing. See issue 35797.)
History
Date User Action Args
2019-01-24 00:48:18eryksunsetrecipients: + eryksun, paul.moore, vinay.sajip, tim.golden, zach.ware, steve.dower
2019-01-24 00:48:16eryksunsetmessageid: <1548290896.4.0.108661661457.issue35811@roundup.psfhosted.org>
2019-01-24 00:48:16eryksunlinkissue35811 messages
2019-01-24 00:48:15eryksuncreate