classification
Title: sysconfig's posix_user scheme has different platlib value to distutils's unix_user
Type: Stage: resolved
Components: Distutils Versions: Python 3.11, Python 3.10, Python 3.9
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: dstufft, eric.araujo, frenzy, hroncok, lukasz.langa, miss-islington, pablogsal, petr.viktorin, uranusjr, vstinner
Priority: release blocker Keywords: patch

Created on 2021-08-07 14:29 by uranusjr, last changed 2021-10-04 19:18 by pablogsal. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 27655 merged uranusjr, 2021-08-07 14:56
PR 28197 merged miss-islington, 2021-09-07 10:29
PR 28235 merged vstinner, 2021-09-08 11:02
PR 28251 merged miss-islington, 2021-09-09 09:05
Messages (18)
msg399186 - (view) Author: Tzu-ping Chung (uranusjr) * Date: 2021-08-07 14:29
On POSIX, the user scheme has a different 'platlib' location between distutils and sysconfig, dispite the comment claiming they should be the same.

This can be reproduced on Fedora 34's stock Python 3.9:

    $ docker run -it --rm -h=p fedora:34 bash
    ...
    [root@p /]# yum install python3 -y
    ...
    [root@p /]# type python3
    python3 is hashed (/usr/bin/python3)
    [root@p /]# python3 -V
    Python 3.9.6
    [root@p /]# python3.9 -q
    >>> from distutils.command.install import install
    >>> from distutils.dist import Distribution
    >>> c = install(Distribution())
    >>> c.user = True
    >>> c.finalize_options()
    >>> c.install_platlib
    '/root/.local/lib/python3.9/site-packages'
    >>> import sysconfig
    >>> sysconfig.get_path('platlib', 'posix_user')
    '/root/.local/lib64/python3.9/site-packages'

This issue was introduced by the sys.platlibdir value, and its usage in distutils and sysconfig. sysconfig sets posix_user's lib paths like this:

    'purelib': '{userbase}/lib/python{py_version_short}/site-packages',
    'platlib': '{userbase}/{platlibdir}/python{py_version_short}/site-packages',

https://github.com/python/cpython/blob/a40675c659cd8c0699f85ee9ac31660f93f8c2f5/Lib/sysconfig.py#L100-L108

But distutils naively sets both to the same value that does not account for platlibdir:

    'purelib': '$usersite',
    'platlib': '$usersite',

https://github.com/python/cpython/blob/a40675c659cd8c0699f85ee9ac31660f93f8c2f5/Lib/distutils/command/install.py#L68-L87

causing the mismatch, dispite the comment above clearly indicating the values are supposed to be the same.

This was introduced in bpo-1294959 which changed the platlib template to depend on sys.platlibdir, so a mismatch happens when the value of sys.platlibdir is not 'lib'.

(Adding frenzy and vstinner to the nosy list since you introduced the comment in distutils and the sys.platlibdir change, respectively.)
msg399315 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2021-08-10 11:11
sys.platlibdir was introduced to install libraries in /usr/lib64 rather than /usr/lib. I'm not sure if it should be used to install libraries in $HOME/.local/lib64 rather than $HOME/.local/lib. Previously, Fedora already used $HOME/.local/lib and $HOME/.local/lib64 is not in the sys.path.

Does the site module add $HOME/.local/lib64 to sys.path if it exists?
msg399316 - (view) Author: Tzu-ping Chung (uranusjr) * Date: 2021-08-10 12:10
> I'm not sure if it should be used to install libraries in $HOME/.local/lib64 rather than $HOME/.local/lib. Previously, Fedora already used $HOME/.local/lib and $HOME/.local/lib64 is not in the sys.path.

This was also briefly discussed in bpo-1294959, but did not go through since “changing posix_user should have no impact on end users”.

> Does the site module add $HOME/.local/lib64 to sys.path if it exists?

It does not, only lib is checked right now.

https://github.com/python/cpython/blob/c7ea1e3dcea6fbc9842463ce2b785b43501b1eaa/Lib/site.py#L288-L298

----

There are two possible solutions from what I can tell. We could just make posix_user match posix_prefix and always respect sys.platlibdir. This could be confusing to existing Python 3.9 users however since many of them already pip-installed things into ~/.local/lib and this would make their user-site packages split in two locations. The other would be to restore the pre-3.9 behaviour in sysconfig to use lib instead of depending on sys.platlibdir. I don’t know who uses sysconfig right now and can’t say what would break, but for pip this would be less disruptive since it currently installs things into ~/.local/ib (provided by distutils).
msg399318 - (view) Author: Miro Hrončok (hroncok) * Date: 2021-08-10 12:47
Installing to ~/.local/lib works, installing to ~/.local/lib64 breaks things, as it is not on sys.path. I agree that restoring the pre-3.9 behavior in sysconfig to use lib instead of depending on sys.platlibdir is a better fix, at least for 3.9 and 3.10. We can redesign things in 3.11 (but I wouldn't).
msg399326 - (view) Author: Tzu-ping Chung (uranusjr) * Date: 2021-08-10 13:51
I’ve updated the linked PR to change sysconfig instead to not use sys.platlibdir when generating the posix_user scheme. This means posix_user would behave the same in 3.9+ as 3.8 and prior.
msg401217 - (view) Author: Petr Viktorin (petr.viktorin) * (Python committer) Date: 2021-09-07 09:36
Marking as *potential* release blocker for 3.10.

Pablo, without this change the newest pip (with [PR 10358]) will not work on Python 3.10 built --with-platlibdir.
This configure option was added in 3.9 for distros that separate `/usr/lib` and `/usr/lib64`, which used downstream patches before.
Things will be broken if:
* this bug is not fixed in 3.10
* AND `pip` switches to _USE_SYSCONFIG in 3.10 (with merged [PR 10358])
* AND distros don't patch again.

So, if this isn't merged in 3.10, either pip or downstreams will need to implement a workaround. (If pip doesn't hold off, downstreams that build  --with-platlibdir will likely carry this exact patch.)
On the other hand, we're very late in the release cycle.

Note that there's a similar bug in bpo-45035

[PR 10358]: https://github.com/pypa/pip/pull/10358
msg401218 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2021-09-07 10:03
IIUC we need to backport PR27655 to 3.10 no? Or do we need something else?
msg401221 - (view) Author: Petr Viktorin (petr.viktorin) * (Python committer) Date: 2021-09-07 10:19
Possibly together with PR28011.
msg401241 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2021-09-07 11:20
New changeset 608a6292366ebba20f33d93d8b52cbb928429e47 by Miss Islington (bot) in branch '3.10':
bpo-44860: Make sysconfig posix_user not depend on platlibdir (GH-27655) (GH-28197)
https://github.com/python/cpython/commit/608a6292366ebba20f33d93d8b52cbb928429e47
msg401242 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2021-09-07 11:21
Petr, is something left to do for this release blocker? I am planning to start the release if everything is OK
msg401250 - (view) Author: Petr Viktorin (petr.viktorin) * (Python committer) Date: 2021-09-07 11:50
I believe everything is in order now.
msg401264 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2021-09-07 13:20
See also the email on python-dev:
https://mail.python.org/archives/list/python-dev@python.org/thread/5UU6V2B3KBS4Z7OG5T7D6YQZASFNSBJM/
msg401266 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2021-09-07 13:23
Let's continue the discussion for the unix_home scheme in bpo-45035.
msg401373 - (view) Author: Miro Hrončok (hroncok) * Date: 2021-09-08 10:47
There seem to be a regression in test_user_similar: https://bugs.python.org/issue45136
msg401448 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2021-09-09 09:02
New changeset 49acac00c08838d8080ce00d02c05284b94f8fb2 by Victor Stinner in branch 'main':
bpo-44860: Update test_sysconfig for posix_user platlib (GH-28235)
https://github.com/python/cpython/commit/49acac00c08838d8080ce00d02c05284b94f8fb2
msg401452 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2021-09-09 09:36
New changeset 11103eb1f2199cacd8c2e29e3db0d19199885b45 by Miss Islington (bot) in branch '3.10':
bpo-44860: Update test_sysconfig for posix_user platlib (GH-28235) (GH-28251)
https://github.com/python/cpython/commit/11103eb1f2199cacd8c2e29e3db0d19199885b45
msg401455 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2021-09-09 09:54
I marked bpo-45136 as a duplicate of this issue.
msg403172 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2021-10-04 19:18
New changeset 01be51833db23414b5dc766f9c92953d838d82c3 by Pablo Galindo (Miss Islington (bot)) in branch '3.10':
bpo-44860: Update test_sysconfig for posix_user platlib (GH-28235) (GH-28251)
https://github.com/python/cpython/commit/01be51833db23414b5dc766f9c92953d838d82c3
History
Date User Action Args
2021-10-04 19:18:43pablogsalsetmessages: + msg403172
2021-09-09 09:54:22vstinnersetmessages: + msg401455
2021-09-09 09:54:05vstinnerlinkissue45136 superseder
2021-09-09 09:36:01vstinnersetmessages: + msg401452
2021-09-09 09:05:40miss-islingtonsetpull_requests: + pull_request26672
2021-09-09 09:02:58vstinnersetmessages: + msg401448
2021-09-08 11:02:22vstinnersetpull_requests: + pull_request26655
2021-09-08 10:47:32hroncoksetmessages: + msg401373
2021-09-07 13:23:15vstinnersetmessages: + msg401266
2021-09-07 13:20:03vstinnersetmessages: + msg401264
2021-09-07 11:50:00petr.viktorinsetmessages: + msg401250
2021-09-07 11:46:12pablogsalsetstatus: open -> closed
resolution: fixed
stage: patch review -> resolved
2021-09-07 11:21:45pablogsalsetmessages: + msg401242
2021-09-07 11:20:35pablogsalsetmessages: + msg401241
2021-09-07 10:29:40miss-islingtonsetnosy: + miss-islington
pull_requests: + pull_request26622
2021-09-07 10:19:26petr.viktorinsetmessages: + msg401221
2021-09-07 10:03:53pablogsalsetmessages: + msg401218
2021-09-07 09:36:50petr.viktorinsetpriority: normal -> release blocker
nosy: + petr.viktorin, lukasz.langa, pablogsal
messages: + msg401217

2021-08-10 13:51:04uranusjrsetmessages: + msg399326
2021-08-10 12:47:46hroncoksetmessages: + msg399318
2021-08-10 12:10:15uranusjrsetmessages: + msg399316
2021-08-10 11:28:26hroncoksetnosy: + hroncok
2021-08-10 11:11:00vstinnersetmessages: + msg399315
2021-08-07 14:56:08uranusjrsetkeywords: + patch
stage: patch review
pull_requests: + pull_request26149
2021-08-07 14:29:18uranusjrcreate