Adding the initial ensurepip implementation and module docs

(the "Installing Python Modules" updates will be handled in a separate issue)

Issue 19347 tracks overall PEP 453 implementation progress

Attached is an initial rough draft of the ensurepip module. There are some issues still, but they largely need resolved in pip.

1. Setuptools' use of dependency_links causes pip to still reach out to the internet.
2. Need to remove the --pre flag from the pip invocation once pip 1.5 is finalized
3. Need to fix the --root option when it's used on Wheels (https://github.com/pypa/pip/issues/925)


If you prefer to view it on github my branch for working on it is available at https://github.com/dstufft/cpython/compare/ensurepip

Added a second draft that handles the case when the stdlib isn't directly browseable (e.g. it's zipped up or something).

Added a third draft, this one adds the script to check if pip needs updated. I've removed the first two drafts to make it simpler.

Draft docs patch attached. This splits out a new "Software Packaging and Distribution" section in the main docs index, with distutils, ensurepip and venv as the specific entries.

There's a couple of questions I need answered regarding the behaviour of pip in order to document --user properly:

* how does pip install react when --user is passed while a virtual environment is active?
* how does pip install react when both --root and --user are passed?

You cannot use --user in a virtual environment (well a venv, no idea about a pyvenv - but it should get a similar error message IMO if it doesn't).

If you use --root and --user together it will install to the --root location, using the user layout, so instead of installing to /home/dstufft/.local/lib/python3.4/site-packages/ it will install to /value/of/root/home/dstufft/.local/lib/python3.4/site-packages/

Added most of the nosy list from the tracking issue 19347.

Updated draft docs. I believe this version of the module docs should be complete (since the installation guide updates are covered by issue 19407).

While Donald's attached patch is incomplete, the version on GitHub looks basically ready to go (aside from --root not working when running pip from a wheel file): https://github.com/dstufft/cpython/compare/ensurepip

Shall we proceed with the commit based on the currently available pip, so Martin and Ned can get started on the installer updates?

For what it's worth I can get --root ready to go shortly, I have a patch against pip for it (https://github.com/pypa/pip/pull/1272) I just need to write some tests to ensure it keeps working. Let me go off and do that right now.

I also need to update the bundled Wheel to one that was created with Python 3.4 instead of 2.7 (which matters until the fix for https://github.com/pypa/pip/issues/1067 which is https://github.com/pypa/pip/pull/1251 lands). That fixes the issue where the wheel will have pip2, and pip2.7 baked into it.

Ok, merged in the --root fix to pip and created a Wheel using Python 3.4 (which I installed the Wheel distribution using an ensurepip installed pip ;) ).

Updated on github and a patch added, all outstanding issues that affect this patch exist on the pip side of things. Primarily https://github.com/pypa/pip/issues/1067 which, since I created the new pip wheel using 3.4, only affects this patch in that setuptools is still version specific. This'll be fixed soon in pip.

Donald, could you upload an updated patch here that includes both your changes and my docs updates?

We'll give people a day to look it over, and then go ahead and commit it and create the tracker issues to update pyvenv/venv and the Mac OS X and Windows installers.

There you go Nick.

Here are my review comments (sorry, no Rietveld):

ensurepip/__init__.py
---------------------

Why have the separate _run_pip function?  It would be simpler and less confusing to just merge run_pip into bootstrap and eliminate the extra for loop.  Both depend on the _PROJECTS list anyway. --  Ah, I see that the reason for the separation is for mocking purposes in test_ensurepip.  How about constructing the list of paths to insert in bootstrap and passing that as the new second argument to _run_pip.  Then the knowledge about the wheel names is only in one place.


test_ensurepip.py
-----------------

s/bootsraping/bootsrapping/g


checkpip.py
-----------

It would be useful if checkpip.py returned an non-zero exit status in the case any projects are out-of-date.


Makefile.pre.in
---------------

Need to add ensurepip and ensurepip/_bundled to LIBSUBDIRS so that they are installed.


ensurepip/_bundeled
-------------------

The actual wheel files are not in the patch.  For other reviewers, Donald pointed me to their source here:
https://github.com/dstufft/cpython/blob/ensurepip/Lib/ensurepip/_bundled/


Using those wheels, which, as Donald notes, are preliminary (especially the setuptools one), I did some installs using a non-standard --prefix for python.  Here are the contents of the bin directory (what would be installed into /usr/local/bin by default) after "make altinstall".

$ ls -l bin
total 18760
-rwxr-xr-x  1 nad  pyd      110 Nov  1 15:08 2to3-3.4*
-rwxr-xr-x  1 nad  pyd      108 Nov  1 15:08 idle3.4*
-rwxr-xr-x  1 nad  pyd       93 Nov  1 15:08 pydoc3.4*
-rwxr-xr-x  2 nad  pyd  4791296 Nov  1 15:08 python3.4*
-rwxr-xr-x  2 nad  pyd  4791296 Nov  1 15:08 python3.4dm*
-rwxr-xr-x  1 nad  pyd     2041 Nov  1 15:08 python3.4dm-config*
-rwxr-xr-x  1 nad  pyd      245 Nov  1 15:08 pyvenv-3.4*

Now after python3.4 --ensurepip is run:

-rwxr-xr-x  1 nad  pyd        110 Nov  1 15:08 2to3-3.4*
-rw-r-----  1 nad  pyd        687 Nov  1 15:19 easy_install
-rw-r-----  1 nad  staff      355 Nov  1 15:19 easy_install-3.3.pya
-rw-r-----  1 nad  pyd        687 Nov  1 15:19 easy_install-3.4
-rw-r-----  1 nad  staff      347 Nov  1 15:19 easy_install.pya
-rwxr-xr-x  1 nad  pyd        108 Nov  1 15:08 idle3.4*
-rw-r-----  1 nad  pyd        659 Nov  1 15:19 pip
-rw-r-----  1 nad  pyd        659 Nov  1 15:19 pip3
-rw-r-----  1 nad  pyd        659 Nov  1 15:19 pip3.4
-rwxr-xr-x  1 nad  pyd         93 Nov  1 15:08 pydoc3.4*
-rwxr-xr-x  2 nad  pyd    4791296 Nov  1 15:08 python3.4*
-rwxr-xr-x  2 nad  pyd    4791296 Nov  1 15:08 python3.4dm*
-rwxr-xr-x  1 nad  pyd       2041 Nov  1 15:08 python3.4dm-config*
-rwxr-xr-x  1 nad  pyd        245 Nov  1 15:08 pyvenv-3.4*

Now after "make install" is run:

lrwxr-x---  1 nad  pyd          8 Nov  1 15:21 2to3@ -> 2to3-3.4
-rwxr-xr-x  1 nad  pyd        110 Nov  1 15:21 2to3-3.4*
-rw-r-----  1 nad  pyd        687 Nov  1 15:19 easy_install
-rw-r-----  1 nad  staff      355 Nov  1 15:19 easy_install-3.3.pya
-rw-r-----  1 nad  pyd        687 Nov  1 15:19 easy_install-3.4
-rw-r-----  1 nad  staff      347 Nov  1 15:19 easy_install.pya
lrwxr-x---  1 nad  pyd          7 Nov  1 15:21 idle3@ -> idle3.4
-rwxr-xr-x  1 nad  pyd        108 Nov  1 15:21 idle3.4*
-rw-r-----  1 nad  pyd        659 Nov  1 15:19 pip
-rw-r-----  1 nad  pyd        659 Nov  1 15:19 pip3
-rw-r-----  1 nad  pyd        659 Nov  1 15:19 pip3.4
lrwxr-x---  1 nad  pyd          8 Nov  1 15:21 pydoc3@ -> pydoc3.4
-rwxr-xr-x  1 nad  pyd         93 Nov  1 15:21 pydoc3.4*
lrwxr-x---  1 nad  pyd          9 Nov  1 15:21 python3@ -> python3.4
lrwxr-x---  1 nad  pyd         16 Nov  1 15:21 python3-config@ -> python3.4-config
-rwxr-xr-x  2 nad  pyd    4791296 Nov  1 15:20 python3.4*
lrwxr-x---  1 nad  pyd         18 Nov  1 15:21 python3.4-config@ -> python3.4dm-config
-rwxr-xr-x  2 nad  pyd    4791296 Nov  1 15:20 python3.4dm*
-rwxr-xr-x  1 nad  pyd       2041 Nov  1 15:21 python3.4dm-config*
lrwxr-x---  1 nad  pyd         10 Nov  1 15:21 pyvenv@ -> pyvenv-3.4
-rwxr-xr-x  1 nad  pyd        245 Nov  1 15:21 pyvenv-3.4*

The problem I see here is that pip is unconditionally installing both a pip and a pip3 script along with the versioned script (pip3.4).  Setuptools also has the same issue (along with the wonky .pya scripts which I think should be Windows only).  For system installs, pip, setuptools, and ensurepip should support the equivalent of "altinstall" and "install" options, with the former only installing fully-versioned names and the latter adding "3" names and, to be consistent, neither should install the totally unversioned links for Python 3, so no "pip" or "easy_install".  Third-party packagers solve the conflict between Py2 and Py3 scripts in various ways, like providing some administrative command (e.g. debian "update-alternatives" or Macports "port select").  For virtual environments, the user has complete control over the bin directory and the current behavior of pip and easy_install is OK there (and would be introduce an incompatibility if the default were to change).  ensurepip should be conservative about this.  Perhaps the way to implement it is to introduce something like a non-default --alt (name TBD) option to both setuptools and pip (for backwards compatibility) and have ensurepip by default always install setuptools and pip with --alt unless installing into a virtual environment or with a "--makedefault" (name TBD) option?  It would be rather nasty for the default ensurepip to wipe out already installed versions of pip and easy_install for Python 2 (yes, you would need to have the appropriate privilege as well) in a shared bin directory, like /usr/local/bin or /usr/bin.  FTR, the OS X installer build script could work around the current behavior.

I suppose the changes could be isolated to just ensurepip if it used a temporary dir for the scripts and then moved the appropriate ones itself to the standard scripts directory.

Just for the record, here's what the installed ./lib/python3.4/site-packages looks like after ensurepip has run:

-rw-r--r--  1 nad  pyd       119 Nov  1 15:20 README
drwxr-x---  2 nad  pyd       204 Nov  1 15:21 __pycache__/
drwxr-x---  3 nad  pyd       170 Nov  1 15:21 _markerlib/
-rw-r-----  1 nad  staff     126 Nov  1 15:19 easy_install.py
drwxr-x---  7 nad  pyd       782 Nov  1 15:21 pip/
drwxr-x---  2 nad  pyd       306 Nov  1 15:19 pip-1.5.dev1.dist-info/
-rw-r-----  1 nad  staff  101430 Nov  1 15:19 pkg_resources.py
drwxr-x---  6 nad  pyd      1020 Nov  1 15:21 setuptools/
drwxr-x---  2 nad  pyd       374 Nov  1 15:19 setuptools-1.1.6.dist-info/

And, within pip/_vendor, the following additional projects are installed:

-rw-r-----  1 nad  staff    266 Nov  1 15:19 __init__.py
drwxr-x---  2 nad  pyd      272 Nov  1 15:21 __pycache__/
drwxr-x---  3 nad  pyd      306 Nov  1 15:21 colorama/
drwxr-x---  4 nad  pyd      714 Nov  1 15:21 distlib/
drwxr-x---  8 nad  pyd      544 Nov  1 15:21 html5lib/
-rw-r-----  1 nad  staff    773 Nov  1 15:19 re-vendor.py
drwxr-x---  4 nad  pyd      646 Nov  1 15:21 requests/
-rw-r-----  1 nad  staff  12415 Nov  1 15:19 six.py

The .pya thing is an experimental extension type that setuptools added that just got missed during the new features to generate scripts during wheel install vs wheel build time. I opened a bug to remove that and it'll be gone before 1.5 is released.

I can fix the typos and add those to the Makefile.pre.in.

I agree the versioned scripts need some sort of resolution, I'm not entirely sure what that resolution is. This is coming from pip itself so it's not specific to ensurepip. So there are 4 cases that pip needs to handle.

1) Installation via the original bootstrapping methods into a pre Python 3.4 Python, including a 2.x Python.
2) Installation via ensurepip in Python 3.4+
3) Installation via easy_install
4) Installation via pip install --upgrade pip

So there's obviously a number of solutions that solve the problem in specifically the ensurepip case (adding a flag, temporary directory, etc). What I wonder is:

A) Is it considered reasonable that if someone installs pip with ensurepip, but then later upgrades it with ``pip install --upgrade pip`` that they'll get "typical" pip behavior of installing pip, pipX, and pipX.Y or is this believed to be something that fundamentally needs to change in pip?
B) What about in cases where there is *not* a third party distributor, is it reasonable that if there isn't already a ``pip`` command that pip would provide it, but if there is a pip command already pip *won't* ?

Oh one thing, I can't move anything out of _run_pip because the part you're referring to is actually modifying the sys.path. If I move it then I can't prevent the tests from having side effects.

Oh nevermind, I understand now. I misread the statement. I can do that.

Attached is the second combined2 patch with Ned's feedback incorporated.

For anyone testing this the patch does not contain the binary files which can be found at https://github.com/dstufft/cpython/blob/ensurepip/Lib/ensurepip/_bundled/.

The "install everything" approach is OK for Windows and virtual
environments. The challenge is the shared bin directories on *nix systems.

Perhaps pip itself could be updated such that it installs the bare "pip"
only if sys.executable matches shutil.which("python")? At the very least,
ensurepip should work that way, but setuptools actually has the same
problem with respect to easy_install.

Also, if we do the temp directory workaround then we need to copy and then
delete the original rather than move (otherwise the file context will be
wrong under SELinux)

Without digging deeper, I'd be a little cautious about using a test involving sys.executable and shutil.which.  sys.executable in the past at least was not always what you might think in certain cases with OS X framework installs.

I created https://github.com/pypa/pip/issues/1294 for the script versioning issue on the pip side, as I'd prefer not to be doing the temporary directory dance if we don't need to.

How do we want to handle this for beta 1? Is it OK to do an initial commit that has the installation of non-versioned scripts as a known defect so we can make progress on the rest of the changes in the meantime?

Alternatively, we could perhaps tweak the embedded wheels as an interim fix until the problem is addressed upstream.

Tweaking the Wheels won't work. The scripts are generated at install time.

We need to fix it in pip, I was waiting on answers to http://bugs.python.org/issue19406#msg201954 before coming up with a solution and making a PR request as that will influence the proposal/PR I make to pip. Basically we need to figure out what is considered reasonable during the initial ensurepip, and subsequent pip install --upgrade pip.

I'm OK with an unqualified "pip install --upgrade pip" retaining pip's current behaviour of always writing the unqualified script versions.

However, running "pip3 install --upgrade pip" would ideally leave the "pip" and "easy_install" scripts alone.

For "make altinstall" compatibility, we definitely need a "--altinstall" flag that only installs the fully qualified versions of pip and easy_install.

Finally, there needs to be some way to explicitly request the "pip3" style behaviour when running via -m so we can use it from ensurepip.

Donald, I know you've been busy with PyPI v2.0 the last few days, but I see the pull request to resolve https://github.com/pypa/pip/issues/1294 has been merged.

If we can get an updated patch that sets ENSUREPIP_OPTIONS appropriately in the process environment, it should be possible to commit this one and let Ned and Martin get started on the installers.

* Updated setuptools
* Updated pip to the latest development snapshot
* Installs default to installing easy_install-X.Y, pipX, and pipX.Y
* Added --altinstall which only installs easy_install-X.Y and pipX.Y
* Added --default-install which installs easy_install, easy_install-X.Y, pip, pipX, and pipX.Y

Assigning to myself for final review and commit :)

I'm also going to start creating the implementation issues for the installer and pyvenv updates.

The inconsistency between altinstall and default_install bothered me, so I plan to change the spelling of the latter option to "default_pip" (since it directly controls whether or not the "pip" script gets installed).

New changeset 6a6b1ee306e3 by Nick Coghlan in branch 'default':
Close #19406: Initial implementation of ensurepip
http://hg.python.org/cpython/rev/6a6b1ee306e3

Aside from the default-install -> default-pip name change, the other fixes/changes between Donald's last patch and the committed version:

- added the missing docs for the new options
- updated What's New, ACKS, NEWS
- avoided repetition in the test code by using setUp and addCleanUp appropriately (this also prevented the earlier tests from altering os.environ)
- marked the .whl as binary in .hgeol