classification
Title: ensurepip and pip install failures on macOS Sierra with non-system Python 2.7.x
Type: Stage: patch review
Components: macOS Versions: Python 3.7, Python 3.6, Python 2.7
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: ned.deily Nosy List: Marc.Culler, benjamin.peterson, ned.deily, python-dev, ronaldoussoren
Priority: Keywords: patch

Created on 2016-10-14 13:09 by Marc.Culler, last changed 2016-12-03 07:21 by ned.deily.

Files
File name Uploaded Description Edit
pipfails.patch Marc.Culler, 2016-10-14 13:08 patch
Messages (5)
msg278649 - (view) Author: Marc Culler (Marc.Culler) * Date: 2016-10-14 13:08
Changes made to /Library/Python on macOSX Sierra cause the --with-ensurepip compiler flag to fail, and lead to failures of pip after installing Python.

The new file that causes the problem on Sierra is:
/Library/Python/2.7/site-packages/Extras.pth
The current version of site.py reads that .pth file, which results in sys.path containing the path:
/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python
The latter directory (which is protected by SIP) contains many Python 2.7 packages, including easy_install, setuptools, six, py2app, numpy, pylab and pyOpenSSL.  The effect of including this SIP-protected path in sys.path is this:  installing or upgrade a package in /Library/Frameworks/Python.framework which is also installed as an "Extra" in the /System/Frameworks/Python.framework will cause pip to attempt to delete the "old" package from its SIP-protected directory, leading to a "Permission Denied" exception and a failed install.

Given that Apple has now tied /Library/Python to the system Python in this way, thereby making a separate Python framework in /Library/Frameworks unusable, the natural solution to this problem would be to stop including any /Library/Python paths in sys.path.

I am attaching a patch that removes the block of code in site.py which adds these paths.
msg278657 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2016-10-14 16:54
This also affects the -m ensurepip installation of pip itself: the pip install fails trying to upgrade the Apple-supplied version of setuptools.  The behavior of adding the system Python's site-packages directory to the search path of all framework-build Pythons was deliberately added as a result of Issue4865.  I think experience has shown this was not a good idea because of the coupling it introduced between separate Python installations and some third-party distributors of Python on OS X already patch this code out.  It's currently only an issue for 2.7.x since Apple has not yet shipped versions of Python 3.x but the code should be removed there, too.
msg279779 - (view) Author: Ronald Oussoren (ronaldoussoren) * (Python committer) Date: 2016-10-31 11:26
Another reason to remove this feature: installing python 2.7.12 using the installer on www.python.org breaks the system install of Python, likely because of this issue.

I'm a bit sad that this feature has to go, but modern Python use (in particular virtualenv) has reduced the need for this feature.
msg282272 - (view) Author: Roundup Robot (python-dev) Date: 2016-12-03 07:15
New changeset a8a342b3fbc7 by Ned Deily in branch '2.7':
Issue #28440: No longer add /Library/Python/site-packages, the Apple-supplied
https://hg.python.org/cpython/rev/a8a342b3fbc7
msg282275 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2016-12-03 07:21
I've pushed the change to revert Issue4865 for the 2.7 branch for release with 2.7.13.  I am leaving this open to push similar changes for 3.x branches in anticipation of Apple someday supplying system Python 3.x.  It would also be good to see if the post-install script that runs ensurepip can be made to fail more noisily.
History
Date User Action Args
2016-12-03 07:21:14ned.deilysetpriority: release blocker ->

messages: + msg282275
versions: + Python 3.6, Python 3.7
2016-12-03 07:15:15python-devsetnosy: + python-dev
messages: + msg282272
2016-10-31 11:26:02ronaldoussorensetmessages: + msg279779
2016-10-14 16:54:39ned.deilysetpriority: normal -> release blocker

type: compile error ->
assignee: ned.deily

title: pip failures on macOS Sierra -> ensurepip and pip install failures on macOS Sierra with non-system Python 2.7.x
nosy: + benjamin.peterson
versions: + Python 2.7
messages: + msg278657
stage: patch review
2016-10-14 15:07:42r.david.murraysetnosy: + ronaldoussoren, ned.deily
components: + macOS
2016-10-14 13:09:00Marc.Cullercreate