Title: Interface to select preferred "user" or "home" sysconfig scheme for an environment
Type: enhancement Stage: resolved
Components: Library (Lib) Versions: Python 3.10
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: christian.heimes, doko, mattip, paul.moore, piotr.dobrogost, stefanor, steve.dower, uranusjr
Priority: normal Keywords: patch

Created on 2021-02-24 09:26 by uranusjr, last changed 2021-04-27 10:33 by ncoghlan. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 24644 merged uranusjr, 2021-02-25 06:46
Messages (11)
msg387611 - (view) Author: Tzu-ping Chung (uranusjr) * Date: 2021-02-24 09:26
While trying to migrate pip’s internal usages of distutils to sysconfig,[1] I noticed there isn’t a way for pip to select a scheme for sysconfig.get_paths() for `pip install --target` and `pip install --user`. I tried to implement some logic to "guess" a scheme ("posix_home" for `--home`, "nt_user" for `--user` when is "nt", etc.), but eventually hit a wall trying to support alternative implementations.

PyPy, for example, adds additional schemes "pypy" and "pypy_nt", and it’s not clear whether pip should use then for `--home` or not. @mattip helped clear this up for PyPy (which also prompts bpo-43307), but we are worried that other implementations may introduce even more special rules that causes problems, and it’s also not a good idea for pip to implement special logic for every implementation.

I would propose two changes to sysconfig:

1. Make sysconfig._get_default_scheme() a public function. This function will be documented for implementations to return a default scheme to use when none is given to sysconfig.get_paths().
2. Add a new function sysconfig.get_preferred_schemes() for implementations to return preferred schemes for prefix, home, and user installations. This function should return a dict[str, str] with three keys "prefix", "home", and "user", and their values the scheme names to use.

I would be happy to work on a PR and iterate on the design if this sounds like a reasonable idea. For CPython, the implementation would be something like (to match distutils’s behaviour):

def get_preferred_schemes():
    if == "nt":
        return {
            "prefix": "nt",
            "home": "posix_home",
            "user": "nt_user",
    return {
        "prefix": "posix_prefix",
        "home": "posix_home",
        "user": "posix_user",

msg387612 - (view) Author: Tzu-ping Chung (uranusjr) * Date: 2021-02-24 09:28
Adding Christian to the nosy list since IIRC you said something related to this a while ago.
msg387613 - (view) Author: mattip (mattip) * Date: 2021-02-24 09:40
I wonder if the distro maintainers might be able to use this to reduce the patching they do for their schema?
msg387615 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2021-02-24 10:46
Do you need all three items or would "get_preferred_scheme(name: str) -> str" be sufficient?
msg387617 - (view) Author: Tzu-ping Chung (uranusjr) * Date: 2021-02-24 10:53
That would work as well (raising KeyError or ValueError if `name` is not valid?)
msg387646 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2021-02-25 00:21
This seems like a good change to me. Making sure that the implementation is easily patchable sounds like a good plan, though I don't have any specific suggestions on what would be best.
msg387804 - (view) Author: Tzu-ping Chung (uranusjr) * Date: 2021-02-28 09:14
How do we move the discussion forward? I would really want this to be included in 3.10. Assuming distutils is going to be removed in 3.12, pip would be left with an extremely short window if this has to wait for another year.
msg388360 - (view) Author: Matthias Klose (doko) * (Python committer) Date: 2021-03-09 12:21
see also
msg388361 - (view) Author: Matthias Klose (doko) * (Python committer) Date: 2021-03-09 12:41
The Debian/Ubuntu packages have a local patch for distutils/setuptools introducing an --install-layout option.  Maybe have the same for pip?

The intention with renaming/moving site-packages to /usr/lib/python3/dist-packages is to avoid users damaging their system installation, like

  sudo python3 install
  sudo pip install

overriding packages installed with the distro package manager. Just making this schema known, and making it the default would again allow pip acting on packages managed by the distro package manager.

Otoh, there are use cases, where you want to use the Python provided by the Linux distro and run into issues with the custom dist-packages.

 - Creating a venv using the system Python probably should use
   the versioned site-packages.

 - Building on top of a docker image for e.g. a Python App, you
   probably want to install into the dist-packages provided by
   the system Python.

So the problem to solve is

 - let a "sudo pip install" fail by default on the real system
 - let the same install succeed in a docker environment, or any
   other "image".
 - behave transparently on venv and virtualenv installations.
msg389745 - (view) Author: Tzu-ping Chung (uranusjr) * Date: 2021-03-29 20:39
Gentle ping again :) I’ve also created a PR for this.

> The Debian/Ubuntu packages have a local patch for distutils/setuptools introducing an --install-layout option.  Maybe have the same for pip?

Pip already has a similar mechanism. The default layout is `prefix` and you can change the installation prefix with `--prefix`; `home` and `user` layouts can be specified with `--home={base}` and `--user`.

> So the problem to solve is
>  - let a "sudo pip install" fail by default on the real system
>  - let the same install succeed in a docker environment, or any other "image".

These need to be done in pip, so we’ll have a separate discussion on them elsewhere.

>  - behave transparently on venv and virtualenv installations.

This is what this issue tries to address. A distribution can overwrite `sysconfig.get_default_scheme()` and `sysconfig.get_preferred_scheme(variant)` to return the correct scheme based on whether the current Python is in a virtual environment (by detecting `sys.prefix` and `sys.base_prefix`, I think).

When in a virtual environment, it can return the same scheme as the upstream. Outside of a virtual environment, it can do whatever the platform sees fit, and pip (or whatever calls sysconfig) will install things into wherever it’s told to by the two functions.
msg392047 - (view) Author: Paul Moore (paul.moore) * (Python committer) Date: 2021-04-27 08:46
New changeset d92513390a1a0da781bb08c284136f4d7abea36d by Tzu-ping Chung in branch 'master':
bpo-43312: Functions returning default and preferred sysconfig schemes (GH-24644)
Date User Action Args
2021-04-27 10:33:24ncoghlansetstatus: open -> closed
type: enhancement
resolution: fixed
stage: patch review -> resolved
2021-04-27 08:46:04paul.mooresetnosy: + paul.moore
messages: + msg392047
2021-04-06 18:20:59stefanorsetnosy: + stefanor
2021-03-29 20:39:55uranusjrsetmessages: + msg389745
2021-03-11 15:52:52piotr.dobrogostsetnosy: + piotr.dobrogost
2021-03-09 12:41:48dokosetmessages: + msg388361
2021-03-09 12:21:44dokosetnosy: + doko
messages: + msg388360
2021-02-28 09:14:16uranusjrsetmessages: + msg387804
2021-02-25 06:46:21uranusjrsetkeywords: + patch
stage: patch review
pull_requests: + pull_request23429
2021-02-25 00:21:53steve.dowersetnosy: + steve.dower
messages: + msg387646
2021-02-24 10:53:55uranusjrsetmessages: + msg387617
2021-02-24 10:46:45christian.heimessetmessages: + msg387615
2021-02-24 09:40:55mattipsetnosy: + mattip
messages: + msg387613
2021-02-24 09:28:34uranusjrsetnosy: + christian.heimes
messages: + msg387612
2021-02-24 09:26:40uranusjrcreate