Title: Using realpath for __PYVENV_LAUNCHER__ makes Homebrew installs fragile
Type: behavior Stage:
Components: macOS Versions: Python 3.5, Python 3.4
Status: open Resolution:
Dependencies: Superseder:
Assigned To: ronaldoussoren Nosy List: jason.coombs, ned.deily, ronaldoussoren, tdsmith, vinay.sajip
Priority: normal Keywords: patch

Created on 2014-09-25 05:30 by tdsmith, last changed 2016-01-21 02:38 by jason.coombs.

File name Uploaded Description Edit
dont-realpath-venv-dirname.diff tdsmith, 2014-09-25 05:30 review
dont-realpath-venv-dirname.diff-1 tdsmith, 2014-10-12 05:10 review
Messages (16)
msg227505 - (view) Author: Tim Smith (tdsmith) * Date: 2014-09-25 05:30
Homebrew, the OS X package manager, distributes python3 as a framework build. We like to be able to control the shebang that gets written to scripts installed with pip. [1]

The path we prefer for invoking the python3 interpreter is like /usr/local/opt/python3/bin/python3.4, which is symlinked to the framework stub launcher at /usr/local/Cellar/python3/3.4.1_1/Frameworks/Python.framework/Versions/3.4/bin/python3.4. For Python 2.x, we discovered that assigning "/usr/local/opt/python/bin/python2.7" to sys.executable in resulted in correct shebangs for scripts installed by pip. The same approach doesn't work with Python 3.

A very helpful conversation with Vinay Sajip [2] led us to consider how the python3 stub launcher sets __PYVENV_LAUNCHER__, which distlib uses in preference to sys.executable to discover the intended interpreter when pip writes shebangs.

Roughly, __PYVENV_LAUNCHER__ is set from argv[0], so it mimics the invocation of the stub, except that symlinks in the directory component of the path to the identified interpreter are resolved to a "real" path. For us, this means that __PYVENV_LAUNCHER__ (and therefore the shebangs of installed scripts) always points to the Cellar path, not the preferred opt path, even when python is invoked via the opt path.

Avoiding this symlink resolution would allow us to control pip's shebang (which sets the shebangs of all pip-installed scripts) by controlling the way we invoke python3 when we use ensurepip during installation.

Building python3 with the attached diff removes the symlink resolution.

[1]  This is important to Homebrew because packages are physically installed to versioned prefixes, like /usr/local/Cellar/python3/3.4.1_1/. References to these real paths are fragile and break when the version number changes or when the revision number ("_1") changes, when can happen when e.g. openssl is released and Python needs to be recompiled against the new library. To avoid this breakage, Homebrew maintains a version-independent symlink to each package, like /usr/local/opt/python3, which points to the .../Cellar/python3/3.4.1_1/ location.

msg227515 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2014-09-25 08:24
From the initial description of the problem, it's not clear to me that there is a problem here needing resolution in the stub launcher.  I've asked for clarification on the pip issue tracker.
msg227517 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2014-09-25 09:02
Also, the patch causes a test failure with a framework build:

FAIL: test_defaults (test.test_venv.BasicTest)
Traceback (most recent call last):
  File "/py/dev/3x/root/fwd/Library/Frameworks/pytest_10.9.framework/Versions/3.5/lib/python3.5/test/", line 106, in test_defaults
    self.assertIn('home = %s' % path, data)
AssertionError: 'home = /py/dev/3x/root/fwd/./bin' not found in 'home = /py/dev/3x/root/fwd/bin\ninclude-system-site-packages = false\nversion = 3.5.0\n'
msg227561 - (view) Author: Ronald Oussoren (ronaldoussoren) * (Python committer) Date: 2014-09-25 17:58
I'm not convinced that this is a bug in python. __PYVENV_LAUNCHER__ is an implementation detail of CPython used in the implementation of the pyvenv functionality. 

I consider using this undocumented environment variable in distlib as a bug in distlib, which should be fixed there.  

It would be nice to know why distlib uses __PYVENV_LAUNCHER__ as that could lead us to a real bug or missing feature. 

BTW. I haven't read the discussion in [2] yet.
msg227577 - (view) Author: Ronald Oussoren (ronaldoussoren) * (Python committer) Date: 2014-09-25 21:10
> On 25 sep. 2014, at 19:58, Ronald Oussoren <> wrote:
> Ronald Oussoren added the comment:
> I consider using this undocumented environment variable in distlib as a bug in distlib, which should be fixed 

Speaking of which... That environment variable shouldn't leak into Python code in the first place, launching a regular Python interpreter from a venv could currently result in unwanted behavior.  I'll try to provide an example of that. 

> ----------
> _______________________________________
> Python tracker <>
> <>
> _______________________________________
msg227579 - (view) Author: Vinay Sajip (vinay.sajip) * (Python committer) Date: 2014-09-25 21:36
> I consider using this undocumented environment variable in distlib
> as a bug in distlib, which should be fixed there.  

Well, let me explain why it's used: when Python >= 3.3 starts up, it looks for a pyvenv.cfg file proximate to sys.executable. If found, that means we're in a venv, and sys.path is set up accordingly. On OS X, the stub launcher is copied/symlinked to the venv when a new venv is created - the real interpreter is not copied. So, shebangs written by distlib into scripts installed into a venv must be of the form #!/path/to/venv/bin/python, and this cannot be obtained from sys.executable because that is pointing to a framework executable. There would be no pyvenv.cfg anywhere near that location.

This is why the __PYVENV_LAUNCHER__ variable was created, and distlib uses it because it needs to conform to how venvs work. In this respect distlib is a bit like setuptools - it needs to understand some low-level details which other libraries don't need to worry about.

Scripts installed in venvs work as expected (AFAICT) when used with stock Python framework builds on OS X. With HomeBrew, the complication appears to be caused by two levels of symlink: the executable in /usr/local/ points to one in /usr/local/Cellar/..., which in turn points to the framework executable.

The failing test (test_defaults) could be fixed by looking for equivalence in the "home =" directories in the test, rather than a string-contains-value test as at present.
msg229114 - (view) Author: Tim Smith (tdsmith) * Date: 2014-10-12 05:10
I'm attaching an updated patch; it passes tests for me locally with a framework build.
msg229115 - (view) Author: Tim Smith (tdsmith) * Date: 2014-10-12 05:11
Er, because the test has been modified by taking Vinay's suggestion to test that the directories are physically identical instead of doing a string comparison.
msg229119 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2014-10-12 06:34
A comment on Vinay's comment: "and this cannot be obtained from sys.executable because that is pointing to a framework executable".  That *was* true prior to 3.3 and is still true at the moment for 2.7.x but it is not true for 3.3+.  Ronald's change (b79d276041a8) in Issue15307 for venv framework support changed the stub launcher to no longer do a realpath() on the executable name which means that sys.executable contains the path to the Python stub launcher whether or not in a venv:

$ ls -l /usr/local/bin/python2.7
lrwxr-xr-x  1 root  admin  71 Jul  3 01:21 /usr/local/bin/python2.7 -> ../../../Library/Frameworks/Python.framework/Versions/2.7/bin/python2.7
$ /usr/local/bin/python2.7 -c 'import sys;print(sys.executable)'

$ ls -l /usr/local/bin/python3.4
lrwxr-xr-x  1 root  wheel  71 Oct  6 16:53 /usr/local/bin/python3.4 -> ../../../Library/Frameworks/Python.framework/Versions/3.4/bin/python3.4
$ /usr/local/bin/python3.4 -c 'import sys;print(sys.executable)'
msg229126 - (view) Author: Vinay Sajip (vinay.sajip) * (Python committer) Date: 2014-10-12 07:59
> Ronald's change ... changed the stub launcher to no longer do a realpath() on the executable name
> which means that sys.executable contains the path to the Python stub launcher whether or not in a venv

Ronald's change doesn't do a realpath() on the executable name itself, but it *does* do a realpath() on the executable's directory. This seems to be what's causing the problem in Homebrew builds: there are two levels of symlink at play.

/usr/local/bin/python3.4 -> /usr/local/opt/python3/bin/python3.4 -> /usr/local/Cellar/python3/3.4.1_1/Frameworks/Python.framework/Versions/3.4/bin/python3.4

Because of the realpath(), the "wrong" stub launcher (from Homebrew's perspective) gets pointed to by the environment variable. At least, if my understanding is correct.
msg229129 - (view) Author: Tim Smith (tdsmith) * Date: 2014-10-12 08:14
We would like to refer to python3 as /usr/local/opt/python3/bin/python3, where /usr/local/opt/python3 is a symlink to ../Cellar/python3/3.4.2, and /usr/local/Cellar/python3/3.4.2/bin/python3 is a symlink to /usr/local/Cellar/python3/3.4.2/Frameworks/Python.framework/Versions/3.4/bin/python3, which is the stub launcher for /usr/local/Cellar/python3/3.4.2/Frameworks/Python.framework/Versions/3.4/Python.
msg229682 - (view) Author: Ronald Oussoren (ronaldoussoren) * (Python committer) Date: 2014-10-19 12:57
First of all, sorry about the slow response.

Vinay: I don't quite understand why you use __PYVENV_LAUNCHER__: When I create a pyvenv using python 3.3 (with a oldish build from the 3.3 branch) sys.executable point to a binary in the venv without mucking with environment variables:

$ pyvenv-3.3 testenv
$ testenv/bin/python
Python 3.3.5+ (3.3:a36d469f31c1, Aug 13 2014, 09:04:41) 
[GCC 4.2.1 Compatible Apple LLVM 5.1 (clang-503.0.40)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.executable

If older version of Python 3.x don't do this that's a bug in those versions and it would be acceptable to use the environment variable with those versions to work around this bug.

I still not convinced that __PYVENV_LAUNCHER__ needs to be changed. It's highly likely that any use of __PYVENV_LAUNCHER__ outside of CPython itself is an indication of a bug in CPython that needs to be fixed there.
msg230902 - (view) Author: Vinay Sajip (vinay.sajip) * (Python committer) Date: 2014-11-09 19:39
> I don't quite understand why you use __PYVENV_LAUNCHER__

When I first developed the venv functionality it was definitely needed, but it looks as if it is not needed now. Perhaps to fix this completely, the following needs to be done, on the assumption that __PYVENV_LAUNCHER__ is no longer needed:

1. Remove the reference to it in distlib, and use sys.executable instead. Once pip incorporates this fix, the Homebrew problem should go away. (I have already made the change in the distlib repo, but this needs to be released in order for pip to consider vendoring it, and then pip needs to be released before Python can incorporate it).

2. Remove references to the environment variable in Python itself, using sys.executable instead.

As the env var was an implementation detail, ISTM it could be removed in a 3.4 point release - would you agree?
msg230947 - (view) Author: Ronald Oussoren (ronaldoussoren) * (Python committer) Date: 2014-11-10 09:45
The environment variable itself cannot be removed from CPython, it is necessary to implement the correct behavior of pyvenv with framework builds of Python.

That said, I do think that the environment variable should be unset as soon as possible in the CPython startup code to avoid accidentally affecting other interpreters.
msg234301 - (view) Author: Tim Smith (tdsmith) * Date: 2015-01-19 05:57
Homebrew's interest in this ticket was resolved by the release of pip 6, which includes Vinay's change to distlib to use sys.executable instead of __PYVENV_LAUNCHER__. Many thanks!

I'm not marking this fixed in case it is useful to leave this open to discuss unsetting __PYVENV_LAUNCHER__.
msg258730 - (view) Author: Jason R. Coombs (jason.coombs) * (Python committer) Date: 2016-01-21 02:38
I believe this behavior (presence of the __PYVENV_LAUNCHER__ and inheritance in child processes) continues to cause problems. See for a simple reproducing of the issue, even on pip 6+.
Date User Action Args
2016-01-21 02:38:08jason.coombssetnosy: + jason.coombs
messages: + msg258730
2015-01-19 05:57:25tdsmithsetmessages: + msg234301
2014-11-10 09:45:31ronaldoussorensetmessages: + msg230947
2014-11-09 19:39:53vinay.sajipsetmessages: + msg230902
2014-10-19 12:57:21ronaldoussorensetmessages: + msg229682
2014-10-12 08:14:14tdsmithsetmessages: + msg229129
2014-10-12 07:59:05vinay.sajipsetmessages: + msg229126
2014-10-12 06:34:20ned.deilysetmessages: + msg229119
2014-10-12 05:11:26tdsmithsetmessages: + msg229115
2014-10-12 05:10:25tdsmithsetfiles: + dont-realpath-venv-dirname.diff-1

messages: + msg229114
2014-09-25 21:36:18vinay.sajipsetmessages: + msg227579
2014-09-25 21:10:02ronaldoussorensetmessages: + msg227577
2014-09-25 17:58:29ronaldoussorensetmessages: + msg227561
2014-09-25 09:02:41ned.deilysetmessages: + msg227517
2014-09-25 08:24:40ned.deilysetmessages: + msg227515
2014-09-25 05:30:45tdsmithcreate