classification
Title: Python does not play well with 'stow'.
Type: enhancement Stage: resolved
Components: Interpreter Core Versions: Python 3.5
process
Status: closed Resolution: not a bug
Dependencies: Superseder:
Assigned To: Nosy List: eric.araujo, lemburg, mishikal, r.david.murray, vinay.sajip
Priority: normal Keywords:

Created on 2013-12-13 00:15 by mishikal, last changed 2020-09-13 20:47 by vinay.sajip. This issue is now closed.

Messages (7)
msg205993 - (view) Author: Quanah Gibson-Mount (mishikal) Date: 2013-12-13 00:15
I found that when trying to use Python with the "stow" utility, that it incorrectly hard codes the full DESTDIR path into python, instead of the relative portion of the DESTDIR path.  As a result, DESTDIR usage is fundamentally broken in relation to all other software packages I've ever built.  For example:

make install DESTDIR=/usr/local/stow/python-3.3.2

results in a sys.path that includes /usr/local/stow/python-3.3.2, when instead what is desired is just "/usr/local", so you can properly stow the package:

[quanah@git Python-3.3.2]$ python3
Python 3.3.2 (default, Dec 12 2013, 17:18:31)
[GCC 4.4.7 20120313 (Red Hat 4.4.7-3)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> print(sys.path)
['', '/usr/local/stow/python-3.3.2/lib/python33.zip', '/usr/local/stow/python-3.3.2/lib/python3.3', '/usr/local/stow/python-3.3.2/lib/python3.3/plat-linux', '/usr/local/stow/python-3.3.2/lib/python3.3/lib-dynload', '/usr/local/stow/python-3.3.2/lib/python3.3/site-packages']

Python itself was correctly built with --prefix=/usr/local, as that is the desired prefix.
msg205994 - (view) Author: Quanah Gibson-Mount (mishikal) Date: 2013-12-13 00:20
Or to summarize a bit differently --

the point of DESTDIR is to allow you to install your software in some location other than what was specified with --prefix, without losing the --prefix settings.
msg206012 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2013-12-13 02:06
sys.path is computed dynamically at run time.  Try moving the install directory around on your filesystem, and you'll see.

I'm not familiar with stow, and I don't know if anyone else on the team is either, so you may have to explain your issue in more detail.  If there is a real issue here, my guess would be that the solution will be based on the new venv support.
msg206135 - (view) Author: Quanah Gibson-Mount (mishikal) Date: 2013-12-13 18:30
Ok, so the general idea is to be able to install your software in a specific location and then symlink it back into another location (like /usr/local).  This allows quick and easy software version swapping.  I've used it to do things like test mariadb vs mysql, or various builds of openldap.  It works well with perl too.

The issue I'm hitting here with python is with site packages.  I wanted to be able to install the specific versions of various modules into their own location, and link then back via stow, but python can't find them since the site path is the literal location of the install instead of the symlink'd location.  I ended up just installing the modules into the default python location to get around it by now, but it means I can't trivially swap out different versions the python modules now.
msg206139 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2013-12-13 19:26
Since it appears to be a perl based, it is not too surprising it works well with perl :)

A little googling turns up someone suggesting that the logic for finding things be changed slightly, such that if the binary is itself a symbolic link, it will look in the home dir of the symbolic link to see if it looks like a "python installation" where it can find things.  This *might* be a reasonable idea, but I'd prefer if the people who have worked this stuff out in the context of virtual environments were to consider this use case and make suggestions, so I've added Vinay to nosy.

It isn't a build or even an install issue, though, it's a runtime issue.  I've changed the title to reflect the real issue: python not playing well with stow's symlink setup.

I've marked this as an enhancement for 3.5, since it seems to me like a significant change in behavior, but it could be that an argument could be made that this is actually a bug, given that it seems like most unix programs work fine under such a symlink setup.

You can work around it by using a .pth file to add the site-packages dir you want onto the path, by the way, though you'd have to add that to each install by yourself.
msg206144 - (view) Author: Quanah Gibson-Mount (mishikal) Date: 2013-12-13 19:52
Thank you very much for looking at it. :)  Since I can work around it easily enough, I'm set for now, but it would be great to see Python play nicely with stow in a future release.
msg376847 - (view) Author: Vinay Sajip (vinay.sajip) * (Python committer) Date: 2020-09-13 20:47
Closing, as the OP said "Since I can work around it easily enough, I'm set for now."
History
Date User Action Args
2020-09-13 20:47:07vinay.sajipsetstatus: open -> closed
resolution: not a bug
messages: + msg376847

stage: resolved
2013-12-13 19:52:54mishikalsetmessages: + msg206144
2013-12-13 19:26:58r.david.murraysettype: enhancement
title: Using DESTDIR breaks sys.path -> Python does not play well with 'stow'.
components: + Interpreter Core, - Build

nosy: + vinay.sajip
versions: + Python 3.5, - Python 3.3
messages: + msg206139
2013-12-13 18:30:16mishikalsetmessages: + msg206135
2013-12-13 17:53:16eric.araujosetnosy: + eric.araujo
2013-12-13 02:06:29r.david.murraysetnosy: + r.david.murray
messages: + msg206012
2013-12-13 01:16:25eric.snowsetnosy: + lemburg
2013-12-13 00:20:30mishikalsetmessages: + msg205994
2013-12-13 00:15:44mishikalcreate