While trying to migrate pip’s internal usages of distutils to sysconfig,[1] I noticed there isn’t a way for pip to select a scheme for sysconfig.get_paths() for `pip install --target` and `pip install --user`. I tried to implement some logic to "guess" a scheme ("posix_home" for `--home`, "nt_user" for `--user` when os.name is "nt", etc.), but eventually hit a wall trying to support alternative implementations.

PyPy, for example, adds additional schemes "pypy" and "pypy_nt", and it’s not clear whether pip should use then for `--home` or not. @mattip helped clear this up for PyPy (which also prompts bpo-43307), but we are worried that other implementations may introduce even more special rules that causes problems, and it’s also not a good idea for pip to implement special logic for every implementation.

I would propose two changes to sysconfig:

1. Make sysconfig._get_default_scheme() a public function. This function will be documented for implementations to return a default scheme to use when none is given to sysconfig.get_paths().
2. Add a new function sysconfig.get_preferred_schemes() for implementations to return preferred schemes for prefix, home, and user installations. This function should return a dict[str, str] with three keys "prefix", "home", and "user", and their values the scheme names to use.

I would be happy to work on a PR and iterate on the design if this sounds like a reasonable idea. For CPython, the implementation would be something like (to match distutils’s behaviour):

def get_preferred_schemes():
    if os.name == "nt":
        return {
            "prefix": "nt",
            "home": "posix_home",
            "user": "nt_user",
        }
    return {
        "prefix": "posix_prefix",
        "home": "posix_home",
        "user": "posix_user",
    }


[1]: https://github.com/pypa/pip/pull/9626

Adding Christian to the nosy list since IIRC you said something related to this a while ago.

I wonder if the distro maintainers might be able to use this to reduce the patching they do for their schema?

Do you need all three items or would "get_preferred_scheme(name: str) -> str" be sufficient?

That would work as well (raising KeyError or ValueError if `name` is not valid?)

This seems like a good change to me. Making sure that the implementation is easily patchable sounds like a good plan, though I don't have any specific suggestions on what would be best.

How do we move the discussion forward? I would really want this to be included in 3.10. Assuming distutils is going to be removed in 3.12, pip would be left with an extremely short window if this has to wait for another year.

see also https://github.com/pypa/pip/issues/9617

The Debian/Ubuntu packages have a local patch for distutils/setuptools introducing an --install-layout option.  Maybe have the same for pip?

The intention with renaming/moving site-packages to /usr/lib/python3/dist-packages is to avoid users damaging their system installation, like

  sudo python3 setup.py install
  sudo pip install

overriding packages installed with the distro package manager. Just making this schema known, and making it the default would again allow pip acting on packages managed by the distro package manager.

Otoh, there are use cases, where you want to use the Python provided by the Linux distro and run into issues with the custom dist-packages.

 - Creating a venv using the system Python probably should use
   the versioned site-packages.

 - Building on top of a docker image for e.g. a Python App, you
   probably want to install into the dist-packages provided by
   the system Python.

So the problem to solve is

 - let a "sudo pip install" fail by default on the real system
 - let the same install succeed in a docker environment, or any
   other "image".
 - behave transparently on venv and virtualenv installations.

Gentle ping again :) I’ve also created a PR for this.

> The Debian/Ubuntu packages have a local patch for distutils/setuptools introducing an --install-layout option.  Maybe have the same for pip?

Pip already has a similar mechanism. The default layout is `prefix` and you can change the installation prefix with `--prefix`; `home` and `user` layouts can be specified with `--home={base}` and `--user`.

> So the problem to solve is
>
>  - let a "sudo pip install" fail by default on the real system
>  - let the same install succeed in a docker environment, or any other "image".

These need to be done in pip, so we’ll have a separate discussion on them elsewhere.

>  - behave transparently on venv and virtualenv installations.

This is what this issue tries to address. A distribution can overwrite `sysconfig.get_default_scheme()` and `sysconfig.get_preferred_scheme(variant)` to return the correct scheme based on whether the current Python is in a virtual environment (by detecting `sys.prefix` and `sys.base_prefix`, I think).

When in a virtual environment, it can return the same scheme as the upstream. Outside of a virtual environment, it can do whatever the platform sees fit, and pip (or whatever calls sysconfig) will install things into wherever it’s told to by the two functions.

New changeset d92513390a1a0da781bb08c284136f4d7abea36d by Tzu-ping Chung in branch 'master':
bpo-43312: Functions returning default and preferred sysconfig schemes (GH-24644)
https://github.com/python/cpython/commit/d92513390a1a0da781bb08c284136f4d7abea36d