classification
Title: Using realpath for __PYVENV_LAUNCHER__ makes Homebrew installs fragile
Type: behavior Stage:
Components: macOS Versions: Python 3.5, Python 3.4
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: ronaldoussoren Nosy List: jason.coombs, ned.deily, ronaldoussoren, tdsmith, vinay.sajip
Priority: normal Keywords: patch

Created on 2014-09-25 05:30 by tdsmith, last changed 2016-12-31 11:46 by vinay.sajip.

Files
File name Uploaded Description Edit
dont-realpath-venv-dirname.diff tdsmith, 2014-09-25 05:30 review
dont-realpath-venv-dirname.diff-1 tdsmith, 2014-10-12 05:10 review
delete-venev-launcher.diff tdsmith, 2016-12-22 23:39 review
Messages (23)
msg227505 - (view) Author: Tim Smith (tdsmith) * Date: 2014-09-25 05:30
Homebrew, the OS X package manager, distributes python3 as a framework build. We like to be able to control the shebang that gets written to scripts installed with pip. [1]

The path we prefer for invoking the python3 interpreter is like /usr/local/opt/python3/bin/python3.4, which is symlinked to the framework stub launcher at /usr/local/Cellar/python3/3.4.1_1/Frameworks/Python.framework/Versions/3.4/bin/python3.4. For Python 2.x, we discovered that assigning "/usr/local/opt/python/bin/python2.7" to sys.executable in sitecustomize.py resulted in correct shebangs for scripts installed by pip. The same approach doesn't work with Python 3.

A very helpful conversation with Vinay Sajip [2] led us to consider how the python3 stub launcher sets __PYVENV_LAUNCHER__, which distlib uses in preference to sys.executable to discover the intended interpreter when pip writes shebangs.

Roughly, __PYVENV_LAUNCHER__ is set from argv[0], so it mimics the invocation of the stub, except that symlinks in the directory component of the path to the identified interpreter are resolved to a "real" path. For us, this means that __PYVENV_LAUNCHER__ (and therefore the shebangs of installed scripts) always points to the Cellar path, not the preferred opt path, even when python is invoked via the opt path.

Avoiding this symlink resolution would allow us to control pip's shebang (which sets the shebangs of all pip-installed scripts) by controlling the way we invoke python3 when we use ensurepip during installation.

Building python3 with the attached diff removes the symlink resolution.

[1]  This is important to Homebrew because packages are physically installed to versioned prefixes, like /usr/local/Cellar/python3/3.4.1_1/. References to these real paths are fragile and break when the version number changes or when the revision number ("_1") changes, when can happen when e.g. openssl is released and Python needs to be recompiled against the new library. To avoid this breakage, Homebrew maintains a version-independent symlink to each package, like /usr/local/opt/python3, which points to the .../Cellar/python3/3.4.1_1/ location.

[2] https://github.com/pypa/pip/issues/2031
msg227515 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2014-09-25 08:24
From the initial description of the problem, it's not clear to me that there is a problem here needing resolution in the stub launcher.  I've asked for clarification on the pip issue tracker.
msg227517 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2014-09-25 09:02
Also, the patch causes a test failure with a framework build:

======================================================================
FAIL: test_defaults (test.test_venv.BasicTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/py/dev/3x/root/fwd/Library/Frameworks/pytest_10.9.framework/Versions/3.5/lib/python3.5/test/test_venv.py", line 106, in test_defaults
    self.assertIn('home = %s' % path, data)
AssertionError: 'home = /py/dev/3x/root/fwd/./bin' not found in 'home = /py/dev/3x/root/fwd/bin\ninclude-system-site-packages = false\nversion = 3.5.0\n'
msg227561 - (view) Author: Ronald Oussoren (ronaldoussoren) * (Python committer) Date: 2014-09-25 17:58
I'm not convinced that this is a bug in python. __PYVENV_LAUNCHER__ is an implementation detail of CPython used in the implementation of the pyvenv functionality. 

I consider using this undocumented environment variable in distlib as a bug in distlib, which should be fixed there.  

It would be nice to know why distlib uses __PYVENV_LAUNCHER__ as that could lead us to a real bug or missing feature. 

BTW. I haven't read the discussion in [2] yet.
msg227577 - (view) Author: Ronald Oussoren (ronaldoussoren) * (Python committer) Date: 2014-09-25 21:10
> On 25 sep. 2014, at 19:58, Ronald Oussoren <report@bugs.python.org> wrote:
> 
> 
> Ronald Oussoren added the comment:
> 
> I consider using this undocumented environment variable in distlib as a bug in distlib, which should be fixed 

Speaking of which... That environment variable shouldn't leak into Python code in the first place, launching a regular Python interpreter from a venv could currently result in unwanted behavior.  I'll try to provide an example of that. 

Ronald
> 
> ----------
> 
> _______________________________________
> Python tracker <report@bugs.python.org>
> <http://bugs.python.org/issue22490>
> _______________________________________
msg227579 - (view) Author: Vinay Sajip (vinay.sajip) * (Python committer) Date: 2014-09-25 21:36
> I consider using this undocumented environment variable in distlib
> as a bug in distlib, which should be fixed there.  

Well, let me explain why it's used: when Python >= 3.3 starts up, it looks for a pyvenv.cfg file proximate to sys.executable. If found, that means we're in a venv, and sys.path is set up accordingly. On OS X, the stub launcher is copied/symlinked to the venv when a new venv is created - the real interpreter is not copied. So, shebangs written by distlib into scripts installed into a venv must be of the form #!/path/to/venv/bin/python, and this cannot be obtained from sys.executable because that is pointing to a framework executable. There would be no pyvenv.cfg anywhere near that location.

This is why the __PYVENV_LAUNCHER__ variable was created, and distlib uses it because it needs to conform to how venvs work. In this respect distlib is a bit like setuptools - it needs to understand some low-level details which other libraries don't need to worry about.

Scripts installed in venvs work as expected (AFAICT) when used with stock Python framework builds on OS X. With HomeBrew, the complication appears to be caused by two levels of symlink: the executable in /usr/local/ points to one in /usr/local/Cellar/..., which in turn points to the framework executable.

The failing test (test_defaults) could be fixed by looking for equivalence in the "home =" directories in the test, rather than a string-contains-value test as at present.
msg229114 - (view) Author: Tim Smith (tdsmith) * Date: 2014-10-12 05:10
I'm attaching an updated patch; it passes tests for me locally with a framework build.
msg229115 - (view) Author: Tim Smith (tdsmith) * Date: 2014-10-12 05:11
Er, because the test has been modified by taking Vinay's suggestion to test that the directories are physically identical instead of doing a string comparison.
msg229119 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2014-10-12 06:34
A comment on Vinay's comment: "and this cannot be obtained from sys.executable because that is pointing to a framework executable".  That *was* true prior to 3.3 and is still true at the moment for 2.7.x but it is not true for 3.3+.  Ronald's change (b79d276041a8) in Issue15307 for venv framework support changed the stub launcher to no longer do a realpath() on the executable name which means that sys.executable contains the path to the Python stub launcher whether or not in a venv:

$ ls -l /usr/local/bin/python2.7
lrwxr-xr-x  1 root  admin  71 Jul  3 01:21 /usr/local/bin/python2.7 -> ../../../Library/Frameworks/Python.framework/Versions/2.7/bin/python2.7
$ /usr/local/bin/python2.7 -c 'import sys;print(sys.executable)'
/Library/Frameworks/Python.framework/Versions/2.7/Resources/Python.app/Contents/MacOS/Python

$ ls -l /usr/local/bin/python3.4
lrwxr-xr-x  1 root  wheel  71 Oct  6 16:53 /usr/local/bin/python3.4 -> ../../../Library/Frameworks/Python.framework/Versions/3.4/bin/python3.4
$ /usr/local/bin/python3.4 -c 'import sys;print(sys.executable)'
/usr/local/bin/python3.4
msg229126 - (view) Author: Vinay Sajip (vinay.sajip) * (Python committer) Date: 2014-10-12 07:59
> Ronald's change ... changed the stub launcher to no longer do a realpath() on the executable name
> which means that sys.executable contains the path to the Python stub launcher whether or not in a venv

Ronald's change doesn't do a realpath() on the executable name itself, but it *does* do a realpath() on the executable's directory. This seems to be what's causing the problem in Homebrew builds: there are two levels of symlink at play.

/usr/local/bin/python3.4 -> /usr/local/opt/python3/bin/python3.4 -> /usr/local/Cellar/python3/3.4.1_1/Frameworks/Python.framework/Versions/3.4/bin/python3.4

Because of the realpath(), the "wrong" stub launcher (from Homebrew's perspective) gets pointed to by the environment variable. At least, if my understanding is correct.

https://github.com/pypa/pip/issues/2031#issuecomment-56294599
msg229129 - (view) Author: Tim Smith (tdsmith) * Date: 2014-10-12 08:14
We would like to refer to python3 as /usr/local/opt/python3/bin/python3, where /usr/local/opt/python3 is a symlink to ../Cellar/python3/3.4.2, and /usr/local/Cellar/python3/3.4.2/bin/python3 is a symlink to /usr/local/Cellar/python3/3.4.2/Frameworks/Python.framework/Versions/3.4/bin/python3, which is the stub launcher for /usr/local/Cellar/python3/3.4.2/Frameworks/Python.framework/Versions/3.4/Python.
msg229682 - (view) Author: Ronald Oussoren (ronaldoussoren) * (Python committer) Date: 2014-10-19 12:57
First of all, sorry about the slow response.

Vinay: I don't quite understand why you use __PYVENV_LAUNCHER__: When I create a pyvenv using python 3.3 (with a oldish build from the 3.3 branch) sys.executable point to a binary in the venv without mucking with environment variables:

$ pyvenv-3.3 testenv
$ testenv/bin/python
Python 3.3.5+ (3.3:a36d469f31c1, Aug 13 2014, 09:04:41) 
[GCC 4.2.1 Compatible Apple LLVM 5.1 (clang-503.0.40)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.executable
'/Users/ronald/Projects/pyobjc-hg/pyobjc/x/testenv/bin/python'


If older version of Python 3.x don't do this that's a bug in those versions and it would be acceptable to use the environment variable with those versions to work around this bug.

I still not convinced that __PYVENV_LAUNCHER__ needs to be changed. It's highly likely that any use of __PYVENV_LAUNCHER__ outside of CPython itself is an indication of a bug in CPython that needs to be fixed there.
msg230902 - (view) Author: Vinay Sajip (vinay.sajip) * (Python committer) Date: 2014-11-09 19:39
> I don't quite understand why you use __PYVENV_LAUNCHER__

When I first developed the venv functionality it was definitely needed, but it looks as if it is not needed now. Perhaps to fix this completely, the following needs to be done, on the assumption that __PYVENV_LAUNCHER__ is no longer needed:

1. Remove the reference to it in distlib, and use sys.executable instead. Once pip incorporates this fix, the Homebrew problem should go away. (I have already made the change in the distlib repo, but this needs to be released in order for pip to consider vendoring it, and then pip needs to be released before Python can incorporate it).

2. Remove references to the environment variable in Python itself, using sys.executable instead.

As the env var was an implementation detail, ISTM it could be removed in a 3.4 point release - would you agree?
msg230947 - (view) Author: Ronald Oussoren (ronaldoussoren) * (Python committer) Date: 2014-11-10 09:45
The environment variable itself cannot be removed from CPython, it is necessary to implement the correct behavior of pyvenv with framework builds of Python.

That said, I do think that the environment variable should be unset as soon as possible in the CPython startup code to avoid accidentally affecting other interpreters.
msg234301 - (view) Author: Tim Smith (tdsmith) * Date: 2015-01-19 05:57
Homebrew's interest in this ticket was resolved by the release of pip 6, which includes Vinay's change to distlib to use sys.executable instead of __PYVENV_LAUNCHER__. Many thanks!

I'm not marking this fixed in case it is useful to leave this open to discuss unsetting __PYVENV_LAUNCHER__.
msg258730 - (view) Author: Jason R. Coombs (jason.coombs) * (Python committer) Date: 2016-01-21 02:38
I believe this behavior (presence of the __PYVENV_LAUNCHER__ and inheritance in child processes) continues to cause problems. See https://github.com/pypa/virtualenv/issues/845 for a simple reproducing of the issue, even on pip 6+.
msg283859 - (view) Author: Tim Smith (tdsmith) * Date: 2016-12-22 20:00
I spoke prematurely; I recently rediscovered that the persistence of __PYVENV_LAUNCHER__ poisons the sys.executable of virtualenv interpreters launched as a subprocess of another Python interpreter:

$ virtualenv -p python3 test
$ test/bin/python3 -c 'import sys; print(sys.executable)'
/Users/tim/test/bin/python3

$ /usr/local/bin/python3 -c 'import subprocess; subprocess.call(["/Users/tim/test/bin/python3", "-c", "import sys; print(sys.executable)"])'
/usr/local/bin/python3

$ /usr/local/bin/python3 -c 'import subprocess, os; del os.environ["__PYVENV_LAUNCHER__"]; subprocess.call(["/Users/tim/test/bin/python3", "-c", "import sys; print(sys.executable)"])'
/Users/tim/test/bin/python3

If __PYVENV_LAUNCHER__ can be unset before script execution begins, that seems ideal.
msg283865 - (view) Author: Tim Smith (tdsmith) * Date: 2016-12-22 23:39
Since __PYVENV_LAUNCHER__ is consulted in site.py, it seems likely that the latest it can be deleted is in site.py. The attached patch does that.
msg284106 - (view) Author: Vinay Sajip (vinay.sajip) * (Python committer) Date: 2016-12-27 14:54
I'm not sure if msg230947 is still correct, and it seems neater to remove __PYVENV_LAUNCHER__ from where it's defined in the first place (the stub launcher C file) if it was just a shim needed around the time the functionality was developed (pre the 3.3 release).

It would be helpful if a Mac person could try removing it and seeing if the resulting framework builds of Python on OS X are adversely affected.
msg284116 - (view) Author: Jason R. Coombs (jason.coombs) * (Python committer) Date: 2016-12-27 17:01
On Python 3.6, I made the edit. I actually used the one-liner `   os.environ.pop('__PYVENV_LAUNCHER__', None)`. I then created a venv and installed a package and successfully ran a module in that package:

$ python -m venv ~/.envs/issue22490-test
$ ~/.envs/issue22490-test/bin/python -m pip install cherrypy
Collecting cherrypy
  Downloading CherryPy-8.5.0-py2.py3-none-any.whl (463kB)
    100% |████████████████████████████████| 471kB 2.2MB/s 
Collecting six (from cherrypy)
  Using cached six-1.10.0-py2.py3-none-any.whl
Installing collected packages: six, cherrypy
Successfully installed cherrypy-8.5.0 six-1.10.0
$ ~/.envs/issue22490-test/bin/python -m cherrypy            
...
[27/Dec/2016:11:08:52] ENGINE Serving on http://127.0.0.1:8080
[27/Dec/2016:11:08:52] ENGINE Bus STARTED

So pyvenv seems to be working.

Additionally, it's corrected the undesirable behavior that Tim demonstrated:

$ rm -R ~/.envs/issue22490-test 
$ python -m virtualenv ~/.envs/issue22490-test
Using base prefix '/Library/Frameworks/Python.framework/Versions/3.6'
New python executable in /Users/jaraco/.envs/issue22490-test/bin/python3
Also creating executable in /Users/jaraco/.envs/issue22490-test/bin/python
Installing setuptools, pip, wheel...done.
$ python -c 'import subprocess; subprocess.call(["/Users/jaraco/.envs/issue22490-test/bin/python", "-c", "import sys; print(sys.executable)"])'
/Users/jaraco/.envs/issue22490-test/bin/python


I tried making a Framework build of Python with the environment variable removed entirely, but I'm not experienced enough with builds on a Mac to know what I need to do to create a proper build. I build Python from source and tried to create a venv, but it crashed with a nondescript error:

$ ./python.exe -VV
Python 3.6.0+ (3.6:86a412584c02+, Dec 27 2016, 11:24:01) 
[GCC 4.2.1 Compatible Apple LLVM 8.0.0 (clang-800.0.42.1)]
$ ./python.exe -m venv ~/.envs/issue22490
Error: Command '['/Users/jaraco/.envs/issue22490/bin/python.exe', '-Im', 'ensurepip', '--upgrade', '--default-pip']' returned non-zero exit status 1.

Aah. The issue is probably no zlib.

$ curl https://bootstrap.pypa.io/get-pip.py -o - | ~/.envs/issue22490/bin/python
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 1558k  100 1558k    0     0  5221k      0 --:--:-- --:--:-- --:--:-- 5210k
Traceback (most recent call last):
  File "<stdin>", line 20061, in <module>
  File "<stdin>", line 194, in main
  File "<stdin>", line 82, in bootstrap
zipimport.ZipImportError: can't decompress data; zlib not available


After making sure I had zlib installed and available, by re-running xcode-select --install, I was able to build with zlib, create a venv, and install packages to it:

$ ./python.exe -m venv ~/.envs/issue22490 
$ ~/.envs/issue22490/bin/python -c "import os; os.environ['__PYVENV_LAUNCHER__']"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/Users/jaraco/Dropbox/code/public/cpython/Lib/os.py", line 669, in __getitem__
    raise KeyError(key) from None
KeyError: '__PYVENV_LAUNCHER__'
$ ~/.envs/issue22490/bin/pip install cherrypy
Collecting cherrypy
  Using cached CherryPy-8.5.0-py2.py3-none-any.whl
Collecting six (from cherrypy)
  Using cached six-1.10.0-py2.py3-none-any.whl
Installing collected packages: six, cherrypy
Successfully installed cherrypy-8.5.0 six-1.10.0
$ ~/.envs/issue22490/bin/python -m cherrypy
...
[27/Dec/2016:11:58:09] ENGINE Serving on http://127.0.0.1:8080
[27/Dec/2016:11:58:09] ENGINE Bus STARTED


What else would I need to test? What might one expect to fail due to the lack of that environment variable?
msg284117 - (view) Author: Vinay Sajip (vinay.sajip) * (Python committer) Date: 2016-12-27 17:35
Thanks. It's worth checking what shebang was written to a script that was installed into the venv - e.g. cherryd.
msg284124 - (view) Author: Jason R. Coombs (jason.coombs) * (Python committer) Date: 2016-12-27 18:46
That also looks good.

$ head -n1 ~/.envs/issue22490/bin/cherryd
#!/Users/jaraco/.envs/issue22490/bin/python
msg284385 - (view) Author: Vinay Sajip (vinay.sajip) * (Python committer) Date: 2016-12-31 11:46
> I tried making a Framework build of Python with the environment variable removed entirely, but I'm not experienced enough with builds on a Mac to know what I need to do to create a proper build.

Given Jason's comment above, it's probably best for one of the Mac-expert committers to remove the environment variable altogether from Mac/Tools/pythonw.c:main(), and confirm everything still builds and runs OK, and then for Tim to confirm that Homebrew isn't adversely impacted by the change.
History
Date User Action Args
2016-12-31 11:46:34vinay.sajipsetmessages: + msg284385
2016-12-27 18:46:37jason.coombssetmessages: + msg284124
2016-12-27 17:35:05vinay.sajipsetmessages: + msg284117
2016-12-27 17:01:34jason.coombssetmessages: + msg284116
2016-12-27 14:54:30vinay.sajipsetmessages: + msg284106
2016-12-22 23:39:18tdsmithsetfiles: + delete-venev-launcher.diff

messages: + msg283865
2016-12-22 20:00:38tdsmithsetmessages: + msg283859
2016-01-21 02:38:08jason.coombssetnosy: + jason.coombs
messages: + msg258730
2015-01-19 05:57:25tdsmithsetmessages: + msg234301
2014-11-10 09:45:31ronaldoussorensetmessages: + msg230947
2014-11-09 19:39:53vinay.sajipsetmessages: + msg230902
2014-10-19 12:57:21ronaldoussorensetmessages: + msg229682
2014-10-12 08:14:14tdsmithsetmessages: + msg229129
2014-10-12 07:59:05vinay.sajipsetmessages: + msg229126
2014-10-12 06:34:20ned.deilysetmessages: + msg229119
2014-10-12 05:11:26tdsmithsetmessages: + msg229115
2014-10-12 05:10:25tdsmithsetfiles: + dont-realpath-venv-dirname.diff-1

messages: + msg229114
2014-09-25 21:36:18vinay.sajipsetmessages: + msg227579
2014-09-25 21:10:02ronaldoussorensetmessages: + msg227577
2014-09-25 17:58:29ronaldoussorensetmessages: + msg227561
2014-09-25 09:02:41ned.deilysetmessages: + msg227517
2014-09-25 08:24:40ned.deilysetmessages: + msg227515
2014-09-25 05:30:45tdsmithcreate