Title: Add install scheme for virtual environments
Type: enhancement Stage:
Components: Versions: Python 3.11, Python 3.10
Status: open Resolution:
Dependencies: Superseder:
Assigned To: FFY00 Nosy List: FFY00, frenzy, gaborjbernat, hroncok, jaraco, petr.viktorin
Priority: normal Keywords:

Created on 2021-10-08 17:50 by FFY00, last changed 2021-10-18 10:15 by frenzy.

Messages (4)
msg403485 - (view) Author: Filipe Laíns (FFY00) * (Python triager) Date: 2021-10-08 17:50
Python 3.10 introduced sysconfig._get_preferred_schemes[1], a mechanism to allow downstream distributors to overwrite the default install scheme.
This is done to support downstream modifications where distributors change the installation layout (eg. different site-packages directory). So, distributors will change the default scheme to one that correctly represents their layout.

This presents an issue for projects/people that need to bootstrap virtual environments, like virtualenv (see [2]). As distributors might now be customizing the default install scheme, there is no guarantee that the information returned by sysconfig.get_default_scheme/get_paths is correct for the virtual environment, the only guarantee we have is that it correct for the *current* environment. When bootstrapping a virtual environment, we need to know its layout, so that we can place the files in the correct locations.

The usual solution in situations like this would be to invoke the interpreter we are targeting to get the correct information (eg. path/to/python -m sysconfig), but that is not possible here as the environment does not exist yet -- we are the ones trying to create it.

To solve this issue, I propose the addition of a "virtual" or "venv" install scheme, for virtual environments. The idea is that virtual environments (defined by the presence of pyvenv.cfg, see [3][4]) will always use this scheme, they will use the paths specified by this scheme and sysconfig.get_default_scheme will always return it (_get_preferred_schemes would have no effect here!).
This makes it possible to know the virtual environment layout for another interpreter, via sysconfig.get_paths(scheme='virtual').

I am not cure if this should be adopted in 3.10 or 3.11, as it effectively makes it impossible to reliably construct virtual environments, and requires workarounds such as [5], that pretty much implements this proposal with non-standardized downstream patches.

msg403486 - (view) Author: Miro Hrončok (hroncok) * Date: 2021-10-08 18:01
The existing install schemes contain values for all different kinds of OSes, somehow named according to them.

If we introduce a single "virtual"/"venv" scheme, it would need to have different contents on different OSes (e.g. Windows vs POSIX). I don't think that would cause any actual trouble, but it would be somewhat different than all the other schemes.

If we introduce multiple ones (e.g. "posix_venv" and "nt_venv") we would need an additional layer to get the one appropriate for the current platform.

Hence, I think having a single one is more pragmatic.
msg403513 - (view) Author: Filipe Laíns (FFY00) * (Python triager) Date: 2021-10-09 00:30
Yes, we could have several schemes, but I think having only one is more sensible.

The implementation would be fairly easy. We would just copy the "nt" scheme if on Windows, otherwise "posix_prefix".
msg403720 - (view) Author: Petr Viktorin (petr.viktorin) * (Python committer) Date: 2021-10-12 10:22
Starting out with just "venv" doesn't mean we can't add "posix_venv"/"nt_venv" later (if e.g. someone needs to install into a filesystem for another platform).
Date User Action Args
2021-10-18 10:15:46frenzysetnosy: + frenzy
2021-10-12 10:22:49petr.viktorinsetnosy: + petr.viktorin
messages: + msg403720
2021-10-09 00:30:45FFY00setmessages: + msg403513
2021-10-08 18:01:46hroncoksetmessages: + msg403486
2021-10-08 17:50:08FFY00settitle: Add install scheme for virtual environment -> Add install scheme for virtual environments
2021-10-08 17:50:02FFY00create