classification
Title: latest setuptools breaks virtualenvs due to unbundling dependencies
Type: Stage:
Components: Distutils, Library (Lib) Versions: Python 3.7, Python 3.6
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: cstratak, dstufft, jason.coombs, merwok
Priority: normal Keywords: patch

Created on 2017-02-10 10:14 by cstratak, last changed 2017-02-16 13:52 by cstratak.

Files
File name Uploaded Description Edit
bundle-setuptools-dependencies.patch cstratak, 2017-02-10 10:14 review
bundle-setuptools-dependencies2.patch cstratak, 2017-02-10 19:20 review
Pull Requests
URL Status Linked Edit
PR 67 cstratak, 2017-02-16 13:52
Messages (9)
msg287504 - (view) Author: Charalampos Stratakis (cstratak) * Date: 2017-02-10 10:14
The latest versions of setuptools stopped bundling its dependencies and instead starting requiring them [0]. This seems to break virtualenvs as those dependencies are not bundled with python.

In order to reproduce it, replace the setuptools-28.8.0-py2.py3-none-any.whl with setuptools-34.1.1-py2.py3-none-any.whl at Lib/ensurepip/_bundled/ and change 

_SETUPTOOLS_VERSION = "28.8.0"

to 


_SETUPTOOLS_VERSION = "34.1.1"

Then configure, make and make install:

This message will appear first:
Could not find a version that satisfies the requirement packaging>=16.8 (from setuptools) (from versions: )
No matching distribution found for packaging>=16.8 (from setuptools)

Which is one of the new dependencies.

Then:
./python3.7 -m test test_venv -v

Where the EnsurePipTest will fail due to this assertion [1] at the do_test_with_pip test case with "No module named pip\n' != ''".

One way to fix this would be to bundle the other wheels with python, not sure if that would be the ideal approach though.

Attaching a patch for Lib/ensurepip/__init__.py. With this patch and adding the other wheels [2][3][4][5] at the _bundled directory fixes the issue.

[0] https://github.com/pypa/setuptools/blob/master/CHANGES.rst#v3400
[1] https://github.com/python/cpython/blob/master/Lib/test/test_venv.py#L384
[2] https://pypi.python.org/pypi/six/1.10.0
[3] https://pypi.python.org/pypi/appdirs/1.4.0
[4] https://pypi.python.org/pypi/packaging/16.8
[5] https://pypi.python.org/pypi/pyparsing/2.1.10
msg287535 - (view) Author: Jason R. Coombs (jason.coombs) * (Python committer) Date: 2017-02-10 13:58
This approach, bundling the necessary dependencies to install setuptools, seems like exactly the right thing to do, assuming that ensurepip should install setuptools at all, and I don't want to challenge that assumption here.

Looking at the patch and ensurepip, I wonder if the patch is necessary at all. It looks to me like the `find_links` declaration should be sufficient to allow a `pip install setuptools` to succeed and install its dependencies as long as they're bundled. @cstratak, can you test a less aggressive approach that merely bumps the dependency on setuptools and bundles the dependencies?
msg287536 - (view) Author: Jason R. Coombs (jason.coombs) * (Python committer) Date: 2017-02-10 14:01
For convenience, here's a link to the rendered changelog in the docs, which provides hyperlinks to the relevant issues [0].

[0] https://setuptools.readthedocs.io/en/latest/history.html#v34-0-0
msg287538 - (view) Author: Donald Stufft (dstufft) * (Python committer) Date: 2017-02-10 14:12
I haven't updated the bundled copy of ensurepip yet because of this. Most likely the way this will get resolved is that the PEP 518 PR will land in pip and we will no longer require setuptools to be pre-installed at all and ensurepip will go to only bundling pip itself.
msg287543 - (view) Author: Charalampos Stratakis (cstratak) * Date: 2017-02-10 14:49
@jason.coombs

Already tried to just bump the setuptools version and bundle the other wheels but the result is still the same
msg287546 - (view) Author: Jason R. Coombs (jason.coombs) * (Python committer) Date: 2017-02-10 15:10
Aah, yes. I see now. Ensure_pip uses _PROJECTS for three purposes: for defining additional_projects, extracting those packages to the tmpdir, and for declaring the installation of those projects. Only that last declaration is unnecessary.

This patch looks good to me.
msg287552 - (view) Author: Charalampos Stratakis (cstratak) * Date: 2017-02-10 16:59
Will send a pull request which includes the extra wheels.
msg287567 - (view) Author: Charalampos Stratakis (cstratak) * Date: 2017-02-10 19:20
Second version of the patch:

Adjusted the test_ensurepip test cases to account for the new modules
msg287692 - (view) Author: Charalampos Stratakis (cstratak) * Date: 2017-02-13 13:54
Pull Request has been sent: https://github.com/python/cpython/pull/67
History
Date User Action Args
2017-02-16 13:52:46cstrataksetpull_requests: + pull_request95
2017-02-13 13:54:31cstrataksetmessages: + msg287692
2017-02-10 19:20:34cstrataksetfiles: + bundle-setuptools-dependencies2.patch

messages: + msg287567
2017-02-10 16:59:17cstrataksetmessages: + msg287552
2017-02-10 15:10:32jason.coombssetmessages: + msg287546
2017-02-10 14:49:18cstrataksetmessages: + msg287543
2017-02-10 14:12:15dstufftsetmessages: + msg287538
2017-02-10 14:01:35jason.coombssetmessages: + msg287536
2017-02-10 13:58:45jason.coombssetmessages: + msg287535
2017-02-10 11:46:06berker.peksagsetnosy: + jason.coombs
2017-02-10 10:14:21cstratakcreate