On POSIX, the user scheme has a different 'platlib' location between distutils and sysconfig, dispite the comment claiming they should be the same.

This can be reproduced on Fedora 34's stock Python 3.9:

    $ docker run -it --rm -h=p fedora:34 bash
    ...
    [root@p /]# yum install python3 -y
    ...
    [root@p /]# type python3
    python3 is hashed (/usr/bin/python3)
    [root@p /]# python3 -V
    Python 3.9.6
    [root@p /]# python3.9 -q
    >>> from distutils.command.install import install
    >>> from distutils.dist import Distribution
    >>> c = install(Distribution())
    >>> c.user = True
    >>> c.finalize_options()
    >>> c.install_platlib
    '/root/.local/lib/python3.9/site-packages'
    >>> import sysconfig
    >>> sysconfig.get_path('platlib', 'posix_user')
    '/root/.local/lib64/python3.9/site-packages'

This issue was introduced by the sys.platlibdir value, and its usage in distutils and sysconfig. sysconfig sets posix_user's lib paths like this:

    'purelib': '{userbase}/lib/python{py_version_short}/site-packages',
    'platlib': '{userbase}/{platlibdir}/python{py_version_short}/site-packages',

https://github.com/python/cpython/blob/a40675c659cd8c0699f85ee9ac31660f93f8c2f5/Lib/sysconfig.py#L100-L108

But distutils naively sets both to the same value that does not account for platlibdir:

    'purelib': '$usersite',
    'platlib': '$usersite',

https://github.com/python/cpython/blob/a40675c659cd8c0699f85ee9ac31660f93f8c2f5/Lib/distutils/command/install.py#L68-L87

causing the mismatch, dispite the comment above clearly indicating the values are supposed to be the same.

This was introduced in bpo-1294959 which changed the platlib template to depend on sys.platlibdir, so a mismatch happens when the value of sys.platlibdir is not 'lib'.

(Adding frenzy and vstinner to the nosy list since you introduced the comment in distutils and the sys.platlibdir change, respectively.)

sys.platlibdir was introduced to install libraries in /usr/lib64 rather than /usr/lib. I'm not sure if it should be used to install libraries in $HOME/.local/lib64 rather than $HOME/.local/lib. Previously, Fedora already used $HOME/.local/lib and $HOME/.local/lib64 is not in the sys.path.

Does the site module add $HOME/.local/lib64 to sys.path if it exists?

> I'm not sure if it should be used to install libraries in $HOME/.local/lib64 rather than $HOME/.local/lib. Previously, Fedora already used $HOME/.local/lib and $HOME/.local/lib64 is not in the sys.path.

This was also briefly discussed in bpo-1294959, but did not go through since “changing posix_user should have no impact on end users”.

> Does the site module add $HOME/.local/lib64 to sys.path if it exists?

It does not, only lib is checked right now.

https://github.com/python/cpython/blob/c7ea1e3dcea6fbc9842463ce2b785b43501b1eaa/Lib/site.py#L288-L298

----

There are two possible solutions from what I can tell. We could just make posix_user match posix_prefix and always respect sys.platlibdir. This could be confusing to existing Python 3.9 users however since many of them already pip-installed things into ~/.local/lib and this would make their user-site packages split in two locations. The other would be to restore the pre-3.9 behaviour in sysconfig to use lib instead of depending on sys.platlibdir. I don’t know who uses sysconfig right now and can’t say what would break, but for pip this would be less disruptive since it currently installs things into ~/.local/ib (provided by distutils).

Installing to ~/.local/lib works, installing to ~/.local/lib64 breaks things, as it is not on sys.path. I agree that restoring the pre-3.9 behavior in sysconfig to use lib instead of depending on sys.platlibdir is a better fix, at least for 3.9 and 3.10. We can redesign things in 3.11 (but I wouldn't).

I’ve updated the linked PR to change sysconfig instead to not use sys.platlibdir when generating the posix_user scheme. This means posix_user would behave the same in 3.9+ as 3.8 and prior.

Marking as *potential* release blocker for 3.10.

Pablo, without this change the newest pip (with [PR 10358]) will not work on Python 3.10 built --with-platlibdir.
This configure option was added in 3.9 for distros that separate `/usr/lib` and `/usr/lib64`, which used downstream patches before.
Things will be broken if:
* this bug is not fixed in 3.10
* AND `pip` switches to _USE_SYSCONFIG in 3.10 (with merged [PR 10358])
* AND distros don't patch again.

So, if this isn't merged in 3.10, either pip or downstreams will need to implement a workaround. (If pip doesn't hold off, downstreams that build  --with-platlibdir will likely carry this exact patch.)
On the other hand, we're very late in the release cycle.

Note that there's a similar bug in bpo-45035

[PR 10358]: https://github.com/pypa/pip/pull/10358

IIUC we need to backport PR27655 to 3.10 no? Or do we need something else?

Possibly together with PR28011.

New changeset 608a6292366ebba20f33d93d8b52cbb928429e47 by Miss Islington (bot) in branch '3.10':
bpo-44860: Make sysconfig posix_user not depend on platlibdir (GH-27655) (GH-28197)
https://github.com/python/cpython/commit/608a6292366ebba20f33d93d8b52cbb928429e47

Petr, is something left to do for this release blocker? I am planning to start the release if everything is OK

I believe everything is in order now.

See also the email on python-dev:
https://mail.python.org/archives/list/python-dev@python.org/thread/5UU6V2B3KBS4Z7OG5T7D6YQZASFNSBJM/

Let's continue the discussion for the unix_home scheme in bpo-45035.

There seem to be a regression in test_user_similar: https://bugs.python.org/issue45136

New changeset 49acac00c08838d8080ce00d02c05284b94f8fb2 by Victor Stinner in branch 'main':
bpo-44860: Update test_sysconfig for posix_user platlib (GH-28235)
https://github.com/python/cpython/commit/49acac00c08838d8080ce00d02c05284b94f8fb2

New changeset 11103eb1f2199cacd8c2e29e3db0d19199885b45 by Miss Islington (bot) in branch '3.10':
bpo-44860: Update test_sysconfig for posix_user platlib (GH-28235) (GH-28251)
https://github.com/python/cpython/commit/11103eb1f2199cacd8c2e29e3db0d19199885b45

I marked bpo-45136 as a duplicate of this issue.

New changeset 01be51833db23414b5dc766f9c92953d838d82c3 by Pablo Galindo (Miss Islington (bot)) in branch '3.10':
bpo-44860: Update test_sysconfig for posix_user platlib (GH-28235) (GH-28251)
https://github.com/python/cpython/commit/01be51833db23414b5dc766f9c92953d838d82c3