While trying to migrate pip’s internal usages of distutils to sysconfig,1 I noticed there isn’t a way for pip to select a scheme for sysconfig.get_paths() for pip install --target and pip install --user. I tried to implement some logic to "guess" a scheme ("posix_home" for --home, "nt_user" for --user when os.name is "nt", etc.), but eventually hit a wall trying to support alternative implementations.

PyPy, for example, adds additional schemes "pypy" and "pypy_nt", and it’s not clear whether pip should use then for --home or not. @mattip helped clear this up for PyPy (which also prompts bpo-43307), but we are worried that other implementations may introduce even more special rules that causes problems, and it’s also not a good idea for pip to implement special logic for every implementation.

I would propose two changes to sysconfig:

1. Make sysconfig._get_default_scheme() a public function. This function will be documented for implementations to return a default scheme to use when none is given to sysconfig.get_paths().
2. Add a new function sysconfig.get_preferred_schemes() for implementations to return preferred schemes for prefix, home, and user installations. This function should return a dict[str, str] with three keys "prefix", "home", and "user", and their values the scheme names to use.

I would be happy to work on a PR and iterate on the design if this sounds like a reasonable idea. For CPython, the implementation would be something like (to match distutils’s behaviour):

def get_preferred_schemes():
    if os.name == "nt":
        return {
            "prefix": "nt",
            "home": "posix_home",
            "user": "nt_user",
        }
    return {
        "prefix": "posix_prefix",
        "home": "posix_home",
        "user": "posix_user",
    }

Adding Christian to the nosy list since IIRC you said something related to this a while ago.

I wonder if the distro maintainers might be able to use this to reduce the patching they do for their schema?

Do you need all three items or would "get_preferred_scheme(name: str) -> str" be sufficient?

That would work as well (raising KeyError or ValueError if name is not valid?)

This seems like a good change to me. Making sure that the implementation is easily patchable sounds like a good plan, though I don't have any specific suggestions on what would be best.

How do we move the discussion forward? I would really want this to be included in 3.10. Assuming distutils is going to be removed in 3.12, pip would be left with an extremely short window if this has to wait for another year.

see also pypa/pip#9617

The Debian/Ubuntu packages have a local patch for distutils/setuptools introducing an --install-layout option. Maybe have the same for pip?

The intention with renaming/moving site-packages to /usr/lib/python3/dist-packages is to avoid users damaging their system installation, like

sudo python3 setup.py install
sudo pip install

overriding packages installed with the distro package manager. Just making this schema known, and making it the default would again allow pip acting on packages managed by the distro package manager.

Otoh, there are use cases, where you want to use the Python provided by the Linux distro and run into issues with the custom dist-packages.

• Creating a venv using the system Python probably should use
  the versioned site-packages.

• Building on top of a docker image for e.g. a Python App, you
  probably want to install into the dist-packages provided by
  the system Python.

So the problem to solve is

• let a "sudo pip install" fail by default on the real system
• let the same install succeed in a docker environment, or any
  other "image".
• behave transparently on venv and virtualenv installations.

Gentle ping again :) I’ve also created a PR for this.

The Debian/Ubuntu packages have a local patch for distutils/setuptools introducing an --install-layout option. Maybe have the same for pip?

Pip already has a similar mechanism. The default layout is prefix and you can change the installation prefix with --prefix; home and user layouts can be specified with --home={base} and --user.

So the problem to solve is

• let a "sudo pip install" fail by default on the real system
• let the same install succeed in a docker environment, or any other "image".

These need to be done in pip, so we’ll have a separate discussion on them elsewhere.

• behave transparently on venv and virtualenv installations.

This is what this issue tries to address. A distribution can overwrite sysconfig.get_default_scheme() and sysconfig.get_preferred_scheme(variant) to return the correct scheme based on whether the current Python is in a virtual environment (by detecting sys.prefix and sys.base_prefix, I think).

When in a virtual environment, it can return the same scheme as the upstream. Outside of a virtual environment, it can do whatever the platform sees fit, and pip (or whatever calls sysconfig) will install things into wherever it’s told to by the two functions.

New changeset d925133 by Tzu-ping Chung in branch 'master':
bpo-43312: Functions returning default and preferred sysconfig schemes (GH-24644)
d925133