Title: PEP 453: add the ensurepip module
Type: enhancement Stage: resolved
Components: Library (Lib) Versions: Python 3.4
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: ncoghlan Nosy List: dstufft, larry, loewis, ncoghlan, ned.deily, python-dev, tshepang
Priority: release blocker Keywords: patch

Created on 2013-10-26 12:42 by ncoghlan, last changed 2013-11-11 12:23 by ncoghlan. This issue is now closed.

File name Uploaded Description Edit
draft3.diff dstufft, 2013-10-26 14:13 Draft #3, includes a script to check if pip needs updated review
pep453_ensurepip_docs.diff ncoghlan, 2013-10-27 07:35 ReST documentation for ensurepip review
pep453_ensurepip_docs_v2.diff ncoghlan, 2013-10-31 12:28 Updated draft incorporating --user clarifications review
ensurepip.diff dstufft, 2013-10-31 13:27 Patch #4, fixes --root and adds pip3 and pip3.4 instead of pip2 and pip2.7 review
combined.diff dstufft, 2013-11-01 02:18 Combined Docs + ensurepip patch review
combined2.diff dstufft, 2013-11-02 00:56 Combined Docs + ensure pip Take #2 review
ensurepip-combined-altinstall.diff dstufft, 2013-11-10 19:41 Combined Docs + ensurepip with --altinstall and --default-install review
Messages (32)
msg201342 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2013-10-26 12:42
Adding the initial ensurepip implementation and module docs

(the "Installing Python Modules" updates will be handled in a separate issue)
msg201349 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2013-10-26 12:48
Issue 19347 tracks overall PEP 453 implementation progress
msg201350 - (view) Author: Donald Stufft (dstufft) * (Python committer) Date: 2013-10-26 12:52
Attached is an initial rough draft of the ensurepip module. There are some issues still, but they largely need resolved in pip.

1. Setuptools' use of dependency_links causes pip to still reach out to the internet.
2. Need to remove the --pre flag from the pip invocation once pip 1.5 is finalized
3. Need to fix the --root option when it's used on Wheels (

If you prefer to view it on github my branch for working on it is available at
msg201352 - (view) Author: Donald Stufft (dstufft) * (Python committer) Date: 2013-10-26 13:10
Added a second draft that handles the case when the stdlib isn't directly browseable (e.g. it's zipped up or something).
msg201357 - (view) Author: Donald Stufft (dstufft) * (Python committer) Date: 2013-10-26 14:13
Added a third draft, this one adds the script to check if pip needs updated. I've removed the first two drafts to make it simpler.
msg201435 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2013-10-27 07:35
Draft docs patch attached. This splits out a new "Software Packaging and Distribution" section in the main docs index, with distutils, ensurepip and venv as the specific entries.

There's a couple of questions I need answered regarding the behaviour of pip in order to document --user properly:

* how does pip install react when --user is passed while a virtual environment is active?
* how does pip install react when both --root and --user are passed?
msg201462 - (view) Author: Donald Stufft (dstufft) * (Python committer) Date: 2013-10-27 14:51
You cannot use --user in a virtual environment (well a venv, no idea about a pyvenv - but it should get a similar error message IMO if it doesn't).

If you use --root and --user together it will install to the --root location, using the user layout, so instead of installing to /home/dstufft/.local/lib/python3.4/site-packages/ it will install to /value/of/root/home/dstufft/.local/lib/python3.4/site-packages/
msg201806 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2013-10-31 12:28
Added most of the nosy list from the tracking issue 19347.

Updated draft docs. I believe this version of the module docs should be complete (since the installation guide updates are covered by issue 19407).

While Donald's attached patch is incomplete, the version on GitHub looks basically ready to go (aside from --root not working when running pip from a wheel file):

Shall we proceed with the commit based on the currently available pip, so Martin and Ned can get started on the installer updates?
msg201807 - (view) Author: Donald Stufft (dstufft) * (Python committer) Date: 2013-10-31 12:31
For what it's worth I can get --root ready to go shortly, I have a patch against pip for it ( I just need to write some tests to ensure it keeps working. Let me go off and do that right now.
msg201808 - (view) Author: Donald Stufft (dstufft) * (Python committer) Date: 2013-10-31 12:35
I also need to update the bundled Wheel to one that was created with Python 3.4 instead of 2.7 (which matters until the fix for which is lands). That fixes the issue where the wheel will have pip2, and pip2.7 baked into it.
msg201815 - (view) Author: Donald Stufft (dstufft) * (Python committer) Date: 2013-10-31 13:27
Ok, merged in the --root fix to pip and created a Wheel using Python 3.4 (which I installed the Wheel distribution using an ensurepip installed pip ;) ).

Updated on github and a patch added, all outstanding issues that affect this patch exist on the pip side of things. Primarily which, since I created the new pip wheel using 3.4, only affects this patch in that setuptools is still version specific. This'll be fixed soon in pip.
msg201872 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2013-11-01 02:07
Donald, could you upload an updated patch here that includes both your changes and my docs updates?

We'll give people a day to look it over, and then go ahead and commit it and create the tracker issues to update pyvenv/venv and the Mac OS X and Windows installers.
msg201873 - (view) Author: Donald Stufft (dstufft) * (Python committer) Date: 2013-11-01 02:23
There you go Nick.
msg201951 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2013-11-01 23:52
Here are my review comments (sorry, no Rietveld):


Why have the separate _run_pip function?  It would be simpler and less confusing to just merge run_pip into bootstrap and eliminate the extra for loop.  Both depend on the _PROJECTS list anyway. --  Ah, I see that the reason for the separation is for mocking purposes in test_ensurepip.  How about constructing the list of paths to insert in bootstrap and passing that as the new second argument to _run_pip.  Then the knowledge about the wheel names is only in one place.


It would be useful if returned an non-zero exit status in the case any projects are out-of-date.

Need to add ensurepip and ensurepip/_bundled to LIBSUBDIRS so that they are installed.


The actual wheel files are not in the patch.  For other reviewers, Donald pointed me to their source here:

Using those wheels, which, as Donald notes, are preliminary (especially the setuptools one), I did some installs using a non-standard --prefix for python.  Here are the contents of the bin directory (what would be installed into /usr/local/bin by default) after "make altinstall".

$ ls -l bin
total 18760
-rwxr-xr-x  1 nad  pyd      110 Nov  1 15:08 2to3-3.4*
-rwxr-xr-x  1 nad  pyd      108 Nov  1 15:08 idle3.4*
-rwxr-xr-x  1 nad  pyd       93 Nov  1 15:08 pydoc3.4*
-rwxr-xr-x  2 nad  pyd  4791296 Nov  1 15:08 python3.4*
-rwxr-xr-x  2 nad  pyd  4791296 Nov  1 15:08 python3.4dm*
-rwxr-xr-x  1 nad  pyd     2041 Nov  1 15:08 python3.4dm-config*
-rwxr-xr-x  1 nad  pyd      245 Nov  1 15:08 pyvenv-3.4*

Now after python3.4 --ensurepip is run:

-rwxr-xr-x  1 nad  pyd        110 Nov  1 15:08 2to3-3.4*
-rw-r-----  1 nad  pyd        687 Nov  1 15:19 easy_install
-rw-r-----  1 nad  staff      355 Nov  1 15:19 easy_install-3.3.pya
-rw-r-----  1 nad  pyd        687 Nov  1 15:19 easy_install-3.4
-rw-r-----  1 nad  staff      347 Nov  1 15:19 easy_install.pya
-rwxr-xr-x  1 nad  pyd        108 Nov  1 15:08 idle3.4*
-rw-r-----  1 nad  pyd        659 Nov  1 15:19 pip
-rw-r-----  1 nad  pyd        659 Nov  1 15:19 pip3
-rw-r-----  1 nad  pyd        659 Nov  1 15:19 pip3.4
-rwxr-xr-x  1 nad  pyd         93 Nov  1 15:08 pydoc3.4*
-rwxr-xr-x  2 nad  pyd    4791296 Nov  1 15:08 python3.4*
-rwxr-xr-x  2 nad  pyd    4791296 Nov  1 15:08 python3.4dm*
-rwxr-xr-x  1 nad  pyd       2041 Nov  1 15:08 python3.4dm-config*
-rwxr-xr-x  1 nad  pyd        245 Nov  1 15:08 pyvenv-3.4*

Now after "make install" is run:

lrwxr-x---  1 nad  pyd          8 Nov  1 15:21 2to3@ -> 2to3-3.4
-rwxr-xr-x  1 nad  pyd        110 Nov  1 15:21 2to3-3.4*
-rw-r-----  1 nad  pyd        687 Nov  1 15:19 easy_install
-rw-r-----  1 nad  staff      355 Nov  1 15:19 easy_install-3.3.pya
-rw-r-----  1 nad  pyd        687 Nov  1 15:19 easy_install-3.4
-rw-r-----  1 nad  staff      347 Nov  1 15:19 easy_install.pya
lrwxr-x---  1 nad  pyd          7 Nov  1 15:21 idle3@ -> idle3.4
-rwxr-xr-x  1 nad  pyd        108 Nov  1 15:21 idle3.4*
-rw-r-----  1 nad  pyd        659 Nov  1 15:19 pip
-rw-r-----  1 nad  pyd        659 Nov  1 15:19 pip3
-rw-r-----  1 nad  pyd        659 Nov  1 15:19 pip3.4
lrwxr-x---  1 nad  pyd          8 Nov  1 15:21 pydoc3@ -> pydoc3.4
-rwxr-xr-x  1 nad  pyd         93 Nov  1 15:21 pydoc3.4*
lrwxr-x---  1 nad  pyd          9 Nov  1 15:21 python3@ -> python3.4
lrwxr-x---  1 nad  pyd         16 Nov  1 15:21 python3-config@ -> python3.4-config
-rwxr-xr-x  2 nad  pyd    4791296 Nov  1 15:20 python3.4*
lrwxr-x---  1 nad  pyd         18 Nov  1 15:21 python3.4-config@ -> python3.4dm-config
-rwxr-xr-x  2 nad  pyd    4791296 Nov  1 15:20 python3.4dm*
-rwxr-xr-x  1 nad  pyd       2041 Nov  1 15:21 python3.4dm-config*
lrwxr-x---  1 nad  pyd         10 Nov  1 15:21 pyvenv@ -> pyvenv-3.4
-rwxr-xr-x  1 nad  pyd        245 Nov  1 15:21 pyvenv-3.4*

The problem I see here is that pip is unconditionally installing both a pip and a pip3 script along with the versioned script (pip3.4).  Setuptools also has the same issue (along with the wonky .pya scripts which I think should be Windows only).  For system installs, pip, setuptools, and ensurepip should support the equivalent of "altinstall" and "install" options, with the former only installing fully-versioned names and the latter adding "3" names and, to be consistent, neither should install the totally unversioned links for Python 3, so no "pip" or "easy_install".  Third-party packagers solve the conflict between Py2 and Py3 scripts in various ways, like providing some administrative command (e.g. debian "update-alternatives" or Macports "port select").  For virtual environments, the user has complete control over the bin directory and the current behavior of pip and easy_install is OK there (and would be introduce an incompatibility if the default were to change).  ensurepip should be conservative about this.  Perhaps the way to implement it is to introduce something like a non-default --alt (name TBD) option to both setuptools and pip (for backwards compatibility) and have ensurepip by default always install setuptools and pip with --alt unless installing into a virtual environment or with a "--makedefault" (name TBD) option?  It would be rather nasty for the default ensurepip to wipe out already installed versions of pip and easy_install for Python 2 (yes, you would need to have the appropriate privilege as well) in a shared bin directory, like /usr/local/bin or /usr/bin.  FTR, the OS X installer build script could work around the current behavior.
msg201952 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2013-11-02 00:02
I suppose the changes could be isolated to just ensurepip if it used a temporary dir for the scripts and then moved the appropriate ones itself to the standard scripts directory.
msg201953 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2013-11-02 00:12
Just for the record, here's what the installed ./lib/python3.4/site-packages looks like after ensurepip has run:

-rw-r--r--  1 nad  pyd       119 Nov  1 15:20 README
drwxr-x---  2 nad  pyd       204 Nov  1 15:21 __pycache__/
drwxr-x---  3 nad  pyd       170 Nov  1 15:21 _markerlib/
-rw-r-----  1 nad  staff     126 Nov  1 15:19
drwxr-x---  7 nad  pyd       782 Nov  1 15:21 pip/
drwxr-x---  2 nad  pyd       306 Nov  1 15:19 pip-1.5.dev1.dist-info/
-rw-r-----  1 nad  staff  101430 Nov  1 15:19
drwxr-x---  6 nad  pyd      1020 Nov  1 15:21 setuptools/
drwxr-x---  2 nad  pyd       374 Nov  1 15:19 setuptools-1.1.6.dist-info/

And, within pip/_vendor, the following additional projects are installed:

-rw-r-----  1 nad  staff    266 Nov  1 15:19
drwxr-x---  2 nad  pyd      272 Nov  1 15:21 __pycache__/
drwxr-x---  3 nad  pyd      306 Nov  1 15:21 colorama/
drwxr-x---  4 nad  pyd      714 Nov  1 15:21 distlib/
drwxr-x---  8 nad  pyd      544 Nov  1 15:21 html5lib/
-rw-r-----  1 nad  staff    773 Nov  1 15:19
drwxr-x---  4 nad  pyd      646 Nov  1 15:21 requests/
-rw-r-----  1 nad  staff  12415 Nov  1 15:19
msg201954 - (view) Author: Donald Stufft (dstufft) * (Python committer) Date: 2013-11-02 00:16
The .pya thing is an experimental extension type that setuptools added that just got missed during the new features to generate scripts during wheel install vs wheel build time. I opened a bug to remove that and it'll be gone before 1.5 is released.

I can fix the typos and add those to the

I agree the versioned scripts need some sort of resolution, I'm not entirely sure what that resolution is. This is coming from pip itself so it's not specific to ensurepip. So there are 4 cases that pip needs to handle.

1) Installation via the original bootstrapping methods into a pre Python 3.4 Python, including a 2.x Python.
2) Installation via ensurepip in Python 3.4+
3) Installation via easy_install
4) Installation via pip install --upgrade pip

So there's obviously a number of solutions that solve the problem in specifically the ensurepip case (adding a flag, temporary directory, etc). What I wonder is:

A) Is it considered reasonable that if someone installs pip with ensurepip, but then later upgrades it with ``pip install --upgrade pip`` that they'll get "typical" pip behavior of installing pip, pipX, and pipX.Y or is this believed to be something that fundamentally needs to change in pip?
B) What about in cases where there is *not* a third party distributor, is it reasonable that if there isn't already a ``pip`` command that pip would provide it, but if there is a pip command already pip *won't* ?
msg201955 - (view) Author: Donald Stufft (dstufft) * (Python committer) Date: 2013-11-02 00:39
Oh one thing, I can't move anything out of _run_pip because the part you're referring to is actually modifying the sys.path. If I move it then I can't prevent the tests from having side effects.
msg201956 - (view) Author: Donald Stufft (dstufft) * (Python committer) Date: 2013-11-02 00:40
Oh nevermind, I understand now. I misread the statement. I can do that.
msg201957 - (view) Author: Donald Stufft (dstufft) * (Python committer) Date: 2013-11-02 00:56
Attached is the second combined2 patch with Ned's feedback incorporated.

For anyone testing this the patch does not contain the binary files which can be found at
msg201958 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2013-11-02 01:38
The "install everything" approach is OK for Windows and virtual
environments. The challenge is the shared bin directories on *nix systems.

Perhaps pip itself could be updated such that it installs the bare "pip"
only if sys.executable matches shutil.which("python")? At the very least,
ensurepip should work that way, but setuptools actually has the same
problem with respect to easy_install.

Also, if we do the temp directory workaround then we need to copy and then
delete the original rather than move (otherwise the file context will be
wrong under SELinux)
msg201959 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2013-11-02 01:46
Without digging deeper, I'd be a little cautious about using a test involving sys.executable and shutil.which.  sys.executable in the past at least was not always what you might think in certain cases with OS X framework installs.
msg202137 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2013-11-04 13:53
I created for the script versioning issue on the pip side, as I'd prefer not to be doing the temporary directory dance if we don't need to.

How do we want to handle this for beta 1? Is it OK to do an initial commit that has the installation of non-versioned scripts as a known defect so we can make progress on the rest of the changes in the meantime?

Alternatively, we could perhaps tweak the embedded wheels as an interim fix until the problem is addressed upstream.
msg202138 - (view) Author: Donald Stufft (dstufft) * (Python committer) Date: 2013-11-04 13:58
Tweaking the Wheels won't work. The scripts are generated at install time.

We need to fix it in pip, I was waiting on answers to before coming up with a solution and making a PR request as that will influence the proposal/PR I make to pip. Basically we need to figure out what is considered reasonable during the initial ensurepip, and subsequent pip install --upgrade pip.
msg202140 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2013-11-04 14:14
I'm OK with an unqualified "pip install --upgrade pip" retaining pip's current behaviour of always writing the unqualified script versions.

However, running "pip3 install --upgrade pip" would ideally leave the "pip" and "easy_install" scripts alone.

For "make altinstall" compatibility, we definitely need a "--altinstall" flag that only installs the fully qualified versions of pip and easy_install.

Finally, there needs to be some way to explicitly request the "pip3" style behaviour when running via -m so we can use it from ensurepip.
msg202530 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2013-11-10 15:05
Donald, I know you've been busy with PyPI v2.0 the last few days, but I see the pull request to resolve has been merged.

If we can get an updated patch that sets ENSUREPIP_OPTIONS appropriately in the process environment, it should be possible to commit this one and let Ned and Martin get started on the installers.
msg202559 - (view) Author: Donald Stufft (dstufft) * (Python committer) Date: 2013-11-10 19:41
* Updated setuptools
* Updated pip to the latest development snapshot
* Installs default to installing easy_install-X.Y, pipX, and pipX.Y
* Added --altinstall which only installs easy_install-X.Y and pipX.Y
* Added --default-install which installs easy_install, easy_install-X.Y, pip, pipX, and pipX.Y
msg202608 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2013-11-11 10:26
Assigning to myself for final review and commit :)
msg202609 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2013-11-11 10:33
I'm also going to start creating the implementation issues for the installer and pyvenv updates.
msg202613 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2013-11-11 10:48
The inconsistency between altinstall and default_install bothered me, so I plan to change the spelling of the latter option to "default_pip" (since it directly controls whether or not the "pip" script gets installed).
msg202619 - (view) Author: Roundup Robot (python-dev) Date: 2013-11-11 12:15
New changeset 6a6b1ee306e3 by Nick Coghlan in branch 'default':
Close #19406: Initial implementation of ensurepip
msg202620 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2013-11-11 12:23
Aside from the default-install -> default-pip name change, the other fixes/changes between Donald's last patch and the committed version:

- added the missing docs for the new options
- updated What's New, ACKS, NEWS
- avoided repetition in the test code by using setUp and addCleanUp appropriately (this also prevented the earlier tests from altering os.environ)
- marked the .whl as binary in .hgeol
Date User Action Args
2013-11-11 12:26:00ncoghlanlinkissue19551 dependencies
2013-11-11 12:23:23ncoghlansettype: enhancement
messages: + msg202620
2013-11-11 12:15:25python-devsetstatus: open -> closed

nosy: + python-dev
messages: + msg202619

resolution: fixed
stage: resolved
2013-11-11 11:39:50ncoghlanlinkissue19553 dependencies
2013-11-11 10:48:14ncoghlansetmessages: + msg202613
2013-11-11 10:46:11ncoghlanlinkissue19552 dependencies
2013-11-11 10:33:27ncoghlanlinkissue19550 dependencies
2013-11-11 10:33:09ncoghlansetmessages: + msg202609
2013-11-11 10:26:53ncoghlansetassignee: dstufft -> ncoghlan
messages: + msg202608
2013-11-10 19:41:28dstufftsetfiles: + ensurepip-combined-altinstall.diff

messages: + msg202559
2013-11-10 15:05:08ncoghlansetmessages: + msg202530
2013-11-04 14:14:34ncoghlansetmessages: + msg202140
2013-11-04 13:58:22dstufftsetmessages: + msg202138
2013-11-04 13:53:30ncoghlansetmessages: + msg202137
2013-11-02 01:46:15ned.deilysetmessages: + msg201959
2013-11-02 01:38:01ncoghlansetmessages: + msg201958
2013-11-02 00:56:16dstufftsetfiles: + combined2.diff

messages: + msg201957
2013-11-02 00:40:30dstufftsetmessages: + msg201956
2013-11-02 00:39:54dstufftsetmessages: + msg201955
2013-11-02 00:16:30dstufftsetmessages: + msg201954
2013-11-02 00:12:19ned.deilysetmessages: + msg201953
2013-11-02 00:02:00ned.deilysetmessages: + msg201952
2013-11-01 23:52:34ned.deilysetmessages: + msg201951
2013-11-01 02:23:26dstufftsetmessages: + msg201873
2013-11-01 02:18:18dstufftsetfiles: + combined.diff
2013-11-01 02:07:11ncoghlansetmessages: + msg201872
2013-10-31 13:27:59dstufftsetfiles: + ensurepip.diff

messages: + msg201815
2013-10-31 12:35:07dstufftsetmessages: + msg201808
2013-10-31 12:31:21dstufftsetmessages: + msg201807
2013-10-31 12:28:52ncoghlansetfiles: + pep453_ensurepip_docs_v2.diff
nosy: + loewis, ned.deily, tshepang
messages: + msg201806

2013-10-27 14:51:46dstufftsetmessages: + msg201462
2013-10-27 07:35:31ncoghlansetfiles: + pep453_ensurepip_docs.diff

messages: + msg201435
2013-10-26 14:13:45dstufftsetfiles: - draft.diff
2013-10-26 14:13:35dstufftsetfiles: - draft2.diff
2013-10-26 14:13:10dstufftsetfiles: + draft3.diff

messages: + msg201357
2013-10-26 13:10:41dstufftsetfiles: + draft2.diff

messages: + msg201352
2013-10-26 12:52:02dstufftsetfiles: + draft.diff
keywords: + patch
messages: + msg201350
2013-10-26 12:49:13ncoghlansetmessages: - msg201347
2013-10-26 12:48:51ncoghlansetmessages: + msg201349
2013-10-26 12:48:22ncoghlansetmessages: + msg201347
2013-10-26 12:46:17ncoghlanlinkissue19347 dependencies
2013-10-26 12:45:27ncoghlanlinkissue19407 dependencies
2013-10-26 12:42:33ncoghlancreate