classification
Title: ensurepip --upgrade doesn't change the version of pip used by venv
Type: enhancement Stage: needs patch
Components: Library (Lib) Versions: Python 3.9
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: barry, cjw296, doko, dstufft, ncoghlan, ned.deily, paul.moore, petr.viktorin, steve.dower
Priority: normal Keywords:

Created on 2019-05-31 06:14 by cjw296, last changed 2019-06-10 17:23 by steve.dower.

Messages (6)
msg344029 - (view) Author: Chris Withers (cjw296) * (Python committer) Date: 2019-05-31 06:14
$ python3.7 -m ensurepip --upgrade
Looking in links: /var/folders/m6/tsd59qsj7pd_lldh4mhwh6kh0000gn/T/tmpqk_vncev
Requirement already up-to-date: setuptools in /Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages (39.0.1)
Requirement already up-to-date: pip in /Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages (18.1)

But:

$ python3.7 -m venv /tmp/test_venv
$ /tmp/test_venv/bin/pip --version
pip 10.0.1 from /private/tmp/test_venv/lib/python3.7/site-packages/pip (python 3.7)
msg345069 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2019-06-09 04:58
(Added packaging, Linux distro, and Windows and macOS installer folks to the cc list)

Chris and I were discussing this behaviour, and it turns out even I had forgotten how we had specified this feature in PEP 453: `ensurepip --upgrade` ensures that an older pip is brought up to date with the bundled version, but it does *not* check PyPI for the latest version the way that `python3 -m pip install --upgrade pip` does.

We both expected the ensurepip option to behave the same way as the pip option, since they share a name.

If I had the time machine keys, I'd use a more verbose name for the ensurepip flag (e.g. `--upgrade-to-match-bundle`) to help make it clearer that it does something different from the corresponding pip flag.

As it is though, for Python 3.9, I think we should change the behaviour of `--upgrade` to imply `python -m pip install --upgrade pip`, and then add a separate `--network-upgrade`/`--no-network-upgrade` option that allows folks to opt out of the PyPI part of the version check.

The make file would presumably be updated to pass the `--no-network-upgrade` flag, and I guess the macOS and Windows installers would as well (I'm not sure what the platform policies are around installers making random additional requests to external network services)
msg345070 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2019-06-09 05:10
Addressing the other part of Chris's initial post: there's also no `--upgrade-pip` option on `venv` itself.

Instead, there's only an `--upgrade` option that is intended for *Python* version upgrades, and restructures the internal layout of the venv to switch the Python major version number.

Unless you're using a Linux distro Python that has been patched to inject the external pip installation with rewheel or dirtbike, getting a venv that uses the externally updated version of pip requires running `python3 - m venv --system-site-packages --without-pip ...`.

So my suggestion there would be to:

- rename "venv --upgrade" to "venv --set-interpreter" (keeping `--upgrade` as a legacy alias)
- default to running `ensurepip --upgrade` with its new behaviour
- add `--network-upgrade/--no-network-upgrade` options which get passed straight through to the updated ensurepip
msg345082 - (view) Author: Chris Withers (cjw296) * (Python committer) Date: 2019-06-09 13:32
I don't suppose there's any chance we can treat the misnaming of these options as the bugs they feel like, so so fix them for 3.7+, rather than having people battle on with the confusion for another 3+ years until 3.9 is mainstream?
msg345113 - (view) Author: Petr Viktorin (petr.viktorin) * (Python committer) Date: 2019-06-10 09:44
Please don't forget that it is possible to use venv without PyPI.

With my Fedora hat on: as a distro, we don't trust PyPI as a package delivery infrastructure. Lots of users do, and we allow the users to easily but explicitly opt in to trusting it by running "pip install". (In the same way, we don't trust rubygems, npm, CPAN, hackage, etc. -- there are too many, and PyPI is not special.)

Changing venv's existing `--upgrade` option to start installing from PyPI is problematic. We'd probably need to patch it out, creating an inconsistency between the distro and upstream: it would not do what `python3 -m pip install --upgrade pip` does.

ISTM, the proposed semantics aren't consistent: "venv --upgrade" would not match what "pip --upgrade" does: it would do what "pip install --upgrade pip" does. The latter needs an extra argument, explicitly saying *what* to upgrade. Making "pip" implicit makes sense for "ensurepip", but not for "venv".

Also, in my view, "network" should not be used as a synonym for PyPI (outside pip).

Could we instead make `venv --upgrade` print out a warning, saying that it upgrades Python, and to upgrade pip you should `python -m pip install --upgrade pip` instead?

---

> Unless you're using a Linux distro Python that has been patched
> to inject the external pip installation with rewheel or dirtbike,
> getting a venv that uses the externally updated version of pip
> requires running
> `python3 - m venv --system-site-packages --without-pip ...`.

FWIW, "rewheel" is no more: Fedora now distributes the wheels themselves, and patches system ensurepip to look in /usr/share/python-wheels/.
msg345136 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2019-06-10 17:23
I'm with Petr here. ensurepip/venv should only go as far as we support, which is the version of the wheels included in our bundle. We should continue having the stdlib pretend that PyPI doesn't exist.

External network access would justify the longer option, in my opinion: "--upgrade-pip-from-pypi"
History
Date User Action Args
2019-06-10 17:23:32steve.dowersetmessages: + msg345136
2019-06-10 09:44:19petr.viktorinsetmessages: + msg345113
2019-06-09 13:32:30cjw296setmessages: + msg345082
2019-06-09 05:10:16ncoghlansetmessages: + msg345070
2019-06-09 04:58:58ncoghlansetversions: + Python 3.9, - Python 3.7
nosy: + ned.deily, barry, paul.moore, ncoghlan, petr.viktorin, doko, dstufft, steve.dower

messages: + msg345069

type: enhancement
stage: needs patch
2019-05-31 06:14:55cjw296create