classification
Title: ensurepip: add configure --with-wheel-pkg-dir=PATH to get wheel packages from a system directory
Type: Stage: resolved
Components: Library (Lib) Versions: Python 3.10
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: christian.heimes, dstufft, hroncok, mcepl, ncoghlan, pradyunsg, vstinner
Priority: normal Keywords: patch

Created on 2021-01-07 11:21 by vstinner, last changed 2021-01-21 09:50 by vstinner. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 24151 closed vstinner, 2021-01-07 11:54
PR 24210 merged vstinner, 2021-01-13 14:43
Messages (7)
msg384577 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2021-01-07 11:21
The Fedora packaging policy recommends to "unbundle" bundled dependencies.

"Fedora packages SHOULD make every effort to avoid having multiple, separate, upstream projects bundled together in a single package."
https://docs.fedoraproject.org/en-US/packaging-guidelines/#bundling

The main motivation is to ease updates when fix serious vulnerabilities (less packaging work).

The ensurepip package contains bundled wheel packages of setuptools and pip:

$ ls Lib/ensurepip/_bundled/
pip-20.2.3-py2.py3-none-any.whl
setuptools-47.1.0-py3-none-any.whl

The Fedora python3 package doesn't contain the ensurepip._bundled package:

$ python3
Python 3.9.1 (default, Dec  8 2020, 00:00:00) 
>>> import ensurepip._bundled
ModuleNotFoundError: No module named 'ensurepip._bundled'

Instead, a separated RPM package python-pip-wheel provides wheel packages in /usr/share/python-wheels/ directory:

$ ls /usr/share/python-wheels/
pip-20.2.2-py2.py3-none-any.whl*
setuptools-49.1.3-py3-none-any.whl*
wheel-0.34.2-py2.py3-none-any.whl*

Fedora has a downstream patch on ensurepip (written by Miro Hrončok) to always use /usr/share/python-wheels/:

https://src.fedoraproject.org/rpms/python3.10/blob/master/f/00189-use-rpm-wheels.patch

Fedora has packages of 9 CPython versions: 2.6, 2.7, 3.4, 3.5, 3.6, 3.7, 3.8, 3.9, 3.10.

https://developer.fedoraproject.org/tech/languages/python/multiple-pythons.html

Having a separated package for wheel packages allows us to upgrade a single package (python-pip-wheel) for setuptools/pip bugfix or security vulnerability.

--

I propose to add a new --with-wheel-pkg-dir=PATH option to the ./configure script. If used, ensurepip will only use wheel packages from this directory. Otherwise, the existing code is unchanged. In short, the behavior is unchanged, unless the option is used explicitly.

If a directory is specified but wheel packages are missing, ensurepip fails.

If the directory contains multiple wheel packages of different versions, the most recent version is used of each package.

Note: In practice, the Fedora package only provides a single wheel package of each Python module. But I propose to make the Python upstream code as generic as possible.

I'm working on a pull request to implement this.

Downstream Fedora issue: https://bugzilla.redhat.com/show_bug.cgi?id=1874803

--

Fedora (Miro) already contributed to ensurepip to make ensurepip less dependent on pip internals:

commit 88f82b2b9ea3514359cb6e3218121f75334063ac
Author: Miro Hrončok <miro@hroncok.cz>
Date:   Tue Mar 10 22:16:28 2020 +0100

    bpo-38662: ensurepip invokes pip via runpy (GH-18901)
    
    The ensurepip module now invokes pip via the runpy module.
    Hence it is no longer tightly coupled with the internal API of the bundled
    pip version, allowing easier updates to a newer pip version both
    internally and for distributors.
    
    This way, any changes to the internal pip API won't mean ensurepip needs to be
    changed as well. Also, distributors can update their pip wheels independent on
    CPython release schedule.
    
    Co-Authored-By: Pradyun Gedam <pradyunsg@gmail.com>
    Co-Authored-By: Miro Hrončok <miro@hroncok.cz>

This change was already related to Fedora downstream change to get wheel packages from a different directory. Fedora can use a different pip version (older or more recent) than ensurepip._bundled.
msg384578 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2021-01-07 11:26
> If used, ensurepip will only use wheel packages from this directory.

An alternative is to find packages in all directories and pick the most recent version.

Example with a specified directory *and* ensurepip._bundled is available:

* wheel package directory: 

  * pip 20.2.2
  * setuptools 49.1.3
  * wheel 0.34.2

* ensurepip._bundled

  * pip 20.2.3
  * setuptools 47.1.0

Most recent versions:

* pip 20.2.3 from ensurepip._bundled
* setuptools 49.1.3 from the wheel package directory

Problem: I'm not sure that pip is fully compatible with any setuptools version.
msg384579 - (view) Author: Miro Hrončok (hroncok) * Date: 2021-01-07 11:33
For simplicity, I would avoid mixing wheels from 2 different directories.
msg385027 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2021-01-13 14:50
> For simplicity, I would avoid mixing wheels from 2 different directories.

Right. I wrote PR 24210 which is simpler. It either uses bundled wheels, or wheels from the directory.
msg385031 - (view) Author: Matej Cepl (mcepl) * Date: 2021-01-13 14:58
We (SUSE) have updated versions of the wheels as special Sources, and then this in the %prep stage of our SPEC file:

    # Replace bundled wheels with the updates ones
    rm -v Lib/ensurepip/_bundled/*.whl
    cp -v %{SOURCE20} %{SOURCE21} Lib/ensurepip/_bundled/
    STVER=$(basename %{SOURCE20}|cut -d- -f2)
    PIPVER=$(basename %{SOURCE21}|cut -d- -f2)
    sed -E -i -e "s/^(\s*_SETUPTOOLS_VERSION\s+=\s+)\"[0-9.]+\"/\1\"${STVER}\"/" \
              -e "s/^(\s*_PIP_VERSION\s+=\s+)\"[0-9.]+\"/\1\"${PIPVER}\"/" \
        Lib/ensurepip/__init__.py

A bit of manual work required, but it doesn't lead to so incredible convoluted constructs as I see in Fedora (nothing against it, but our build system is already convoluted enough).
msg385032 - (view) Author: Miro Hrončok (hroncok) * Date: 2021-01-13 15:00
In Fedora, we update the wheels independently without rebuilding Python.

What incredible convoluted constructs do you have in mind in particular?
msg385353 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2021-01-20 16:07
New changeset 75e59a97f5d1fddb0c30ed9747b1b8cb84420a62 by Victor Stinner in branch 'master':
bpo-42856: Add --with-wheel-pkg-dir=PATH configure option (GH-24210)
https://github.com/python/cpython/commit/75e59a97f5d1fddb0c30ed9747b1b8cb84420a62
History
Date User Action Args
2021-01-21 09:50:21vstinnersetstatus: open -> closed
resolution: fixed
stage: patch review -> resolved
2021-01-20 16:07:28vstinnersetmessages: + msg385353
2021-01-13 15:00:44hroncoksetmessages: + msg385032
2021-01-13 14:58:19mceplsetmessages: + msg385031
2021-01-13 14:51:56mceplsetnosy: + mcepl
2021-01-13 14:50:41vstinnersetmessages: + msg385027
2021-01-13 14:43:25vstinnersetpull_requests: + pull_request23037
2021-01-07 11:54:30vstinnersetkeywords: + patch
stage: patch review
pull_requests: + pull_request22979
2021-01-07 11:52:55christian.heimessetnosy: + ncoghlan, christian.heimes, dstufft, pradyunsg
2021-01-07 11:33:04hroncoksetnosy: + hroncok
messages: + msg384579
2021-01-07 11:26:59vstinnersetmessages: + msg384578
2021-01-07 11:21:06vstinnercreate