classification
Title: venv on Windows with symlinks is broken if invoked with -I
Type: Stage:
Components: Windows Versions: Python 3.10, Python 3.9
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: eryksun, gaborjbernat, paul.moore, steve.dower, tim.golden, zach.ware
Priority: normal Keywords:

Created on 2020-10-12 10:55 by gaborjbernat, last changed 2020-10-12 22:39 by eryksun.

Messages (12)
msg378485 - (view) Author: Bernat Gabor (gaborjbernat) * Date: 2020-10-12 10:55
Here's a small reproducible, run it on a Windows OS that has symlinks enabled:

import shutil
import venv
import subprocess

shutil.rmtree("venv", ignore_errors=True)
venv.EnvBuilder(with_pip=False, symlinks=True).create("venv")

# works
subprocess.check_call(["venv\\Scripts\\python.exe", "-c", "import sys; print(sys.executable)"])

# fails with No module named 'encodings'
subprocess.check_call(["venv\\Scripts\\python.exe", "-Ic", "import sys; print(sys.executable)"])
msg378487 - (view) Author: Eryk Sun (eryksun) * (Python triager) Date: 2020-10-12 12:26
I can't reproduce the issue with the normal 3.9.0 distribution from python.org. For example:

    >>> venv.EnvBuilder(with_pip=False, symlinks=True).create("venv")
    >>> subprocess.check_call(["venv\\Scripts\\python.exe", "-Ic", "import sys; print(sys.executable)"])
    C:\Temp\env\venv\Scripts\python.exe
    0

Which Python version(s) and distribution(s) did you test? Do you have the PYTHONHOME and/or PYTHONPATH environment variables set?
msg378488 - (view) Author: Bernat Gabor (gaborjbernat) * Date: 2020-10-12 12:33
❯ py -c 'import sys; print(sys.version)'
3.9.0 (tags/v3.9.0:9cf6752, Oct  5 2020, 15:34:40) [MSC v.1927 64 bit (AMD64)]
msg378490 - (view) Author: Filipe Laíns (FFY00) * Date: 2020-10-12 12:41
This error most likely happens because sys.path isn't being set properly and the standard library isn't being included.

We need more information about the specific environment this is triggering.

Which CPython build are you using? The one from Github Actions, right?
We also need the environment variables.
msg378491 - (view) Author: Filipe Laíns (FFY00) * Date: 2020-10-12 12:43
Well, actually the environment variables /should/ not matter as -I implies -E.
msg378492 - (view) Author: Bernat Gabor (gaborjbernat) * Date: 2020-10-12 12:45
Ok, the missing link is that the python you run into needs to be also a symlink venv:

❯ py -m venv env --without-pip --clear --symlinks
❯ .\env\Scripts\python.exe -c "import venv, subprocess; venv.EnvBuilder(with_pip=False, symlinks=True).create('venv'); subprocess.check_call(['venv\\Scripts\\python.exe', '-Ic', 'import sys; print(sys.executable)'])"

Fatal Python error: init_fs_encoding: failed to get the Python codec of the filesystem encoding
Python runtime state: core initialized
ModuleNotFoundError: No module named 'encodings'
msg378493 - (view) Author: Bernat Gabor (gaborjbernat) * Date: 2020-10-12 12:48
> We need more information about the specific environment this is triggering.

Both Gitub Actions Windows CPython3.9 or installer as downloaded from (on Windows 10) https://www.python.org/downloads/release/python-390/.

> PYTHONHOME should not be set all. PYTHONPATH needs to be set carefully. It should never include standard-library path

The issue manifests independent of those environment variables, netiher of them are set.
msg378494 - (view) Author: Eryk Sun (eryksun) * (Python triager) Date: 2020-10-12 12:50
> Well, actually the environment variables /should/ not matter as -I 
> implies -E.

The operative word there is "should". I was grasping for anything that might explain why I couldn't reproduce the issue.

> Ok, the missing link is that the python you run into needs to 
> be also a symlink venv

Thank you, that reproduces the problem for me.
msg378499 - (view) Author: Eryk Sun (eryksun) * (Python triager) Date: 2020-10-12 14:14
This issue is partly due to bpo-8901, which changed the behavior of the -E and -I command-line options to make them ignore the default PythonPath value in the registry key "Software\Python\PythonCore\X.Y\PythonPath". The change itself is not wrong. It's just exposing an underlying problem.

The `home` path in pyvenv.cfg is from sys._base_executable. In a launcher-based environment that's created from the base installation, sys._base_executable is the real base executable. OTOH, in a symlink-based virtual environment, sys._base_executable is the same as sys.executable. Consequently, if a virtual environment is created from a symlink-based virtual environment, the `home` path in pyvenv.cfg refers to the creating environment instead of the base installation. In this case, with the current implementation, the standard library can only be found by falling back on the default PythonPath in the registry.
msg378507 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2020-10-12 16:15
Thanks for figuring that out, Eryk.

Probably we should just update venv to do a realpath(sys._base_executable) to handle the venv-from-symlinked-venv scenario.

Though I'd also be quite happy to just disallow that entirely (as we used to?). If you enable system site packages thinking you're chaining venvs together, you may be surprised, so it might just be better to error out here. (On the *other* hand, if you're programmatically invoking venv, maybe you know what you're doing and we should allow it?)
msg378512 - (view) Author: Paul Moore (paul.moore) * (Python committer) Date: 2020-10-12 18:04
I'm inclined to think that creating a venv from within another venv should be allowed. Tools like pipx can result in *other* tools being run from within a virtual environment, and I don't think it's good to disallow that usage - as an example, I have virtualenv, tox and nox installed via pipx, and all of them create virtual environments.

So I'm +1 on fixing this by calling realpath.

I don't think we need to do anything special for --system-site-packages. The docs say "Give the virtual environment access to the system site-packages dir". If people are trying to chain venvs using it, they are misreading that comment - and the best they could hope for is to request that it's made clearer.
msg378532 - (view) Author: Eryk Sun (eryksun) * (Python triager) Date: 2020-10-12 22:39
> So I'm +1 on fixing this by calling realpath.

In POSIX, calculate_path() in Modules/getpath.c calls calculate_argv0_path() before it calls calculate_read_pyenv(), and calculate_argv0_path() in turn calls resolve_symlinks(&calculate->argv0_path). Thus "pyvenv.cfg" isn't found and isn't used to find the standard library in POSIX when a virtual environment uses symlinks. Later on, "pyvenv.cfg" gets read by the site module, if importing the latter isn't skipped via -S. The site module sets sys._home to the `home` value, but sys._home only appears to be used by sysconfig when running from the build directory.

In Windows, calculate_path() in PC/getpathp.c could get the final (real) path for argv0_path before calling calculate_pyvenv_file(). Then, just like in POSIX, the environment's "pyvenv.cfg" file won't be found and won't be used to find the standard library when a virtual environment uses symlinks.
History
Date User Action Args
2020-10-12 22:39:02eryksunsetmessages: + msg378532
2020-10-12 18:04:31paul.mooresetmessages: + msg378512
2020-10-12 16:15:01steve.dowersetmessages: + msg378507
2020-10-12 14:14:48eryksunsetmessages: + msg378499
2020-10-12 12:50:37eryksunsetmessages: - msg378489
2020-10-12 12:50:22eryksunsetmessages: + msg378494
2020-10-12 12:48:03gaborjbernatsetmessages: + msg378493
2020-10-12 12:45:04gaborjbernatsetnosy: - FFY00
messages: + msg378492
2020-10-12 12:43:58FFY00setmessages: + msg378491
2020-10-12 12:41:42FFY00setnosy: + FFY00
messages: + msg378490
2020-10-12 12:38:39eryksunsetmessages: + msg378489
2020-10-12 12:33:24gaborjbernatsetmessages: + msg378488
2020-10-12 12:26:42eryksunsetnosy: + eryksun
messages: + msg378487
2020-10-12 10:58:43gaborjbernatsetnosy: + paul.moore, tim.golden, zach.ware, steve.dower
components: + Windows
2020-10-12 10:55:28gaborjbernatcreate