Title: python installation is broken if prefix is overridden on an install target
Type: Stage: resolved
Components: Build, Installation Versions: Python 3.6, Python 3.5, Python 2.7
Status: closed Resolution: wont fix
Dependencies: Superseder:
Assigned To: Nosy List: cosmicduck, loewis, martin.panter, ned.deily, terry.reedy
Priority: normal Keywords: easy

Created on 2013-06-28 07:28 by cosmicduck, last changed 2016-03-16 00:42 by ned.deily. This issue is now closed.

Messages (10)
msg191977 - (view) Author: (cosmicduck) Date: 2013-06-28 07:28
Using Solaris 10 x86
Compiling actual 3.3.2 or 2.7.5 looks like work.
Installing as non-root with 
make altinstall prefix='${HOME}/scripts/tools/python/python-3.3.2' exec-prefix='${HOME}/scripts/tools/python/python-3.3.2' installs looks like correctly to the target.
Extends LD_LIBRARYPATH and PATH to ${HOME}/scripts/tools/python/python-3.3.2/lib and ${HOME}/scripts/tools/python/python-3.3.2/bin has been done.
Python is started correctly with python3.3.
If I start idle3.3 or pydoc3.3 I get error:

/usr/local/bin/python3.3: bad interpreter: No such file or directory.

So it looks like the she-bang is not correctly set for shell scripts to the path prefix and exec-prefix path.

Correcting that by manually setup to ${HOME}/scripts/tools/python/python-3.3.2/bin fix that problem. But I guess it's better to modify the installation process by make to have not future problems.
msg223276 - (view) Author: Mark Lawrence (BreamoreBoy) * Date: 2014-07-16 21:11
@Terry is this something you can comment on please?
msg223293 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2014-07-16 23:08
3.3 only gets security fixes.

This looks to be a linux altinstall installation problem rather than an idle problem. Idle and pydoc only start if Python starts first.  One way to start idle is to start Python and then enter 'import idlelib.idle'.

How did you start python that worked.
How did you try to start idle or pydoc that failed.
msg223306 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2014-07-17 00:35
The problem you see is being caused by trying to override "prefix" on the "altinstall" or "install" targets.  While Makefiles for many products will let you do that successfully, you really can't do that with Python's.  Unfortunately, the values for "prefix" and "exec_prefix" specified (or defaulted) on ./configure are used during the main make build targets.  This includes the customization of the shebangs lines for the scripts like "idle" and "pydoc".  If you change the "prefix" value for the "install" phase, most files do get installed at the changed location but there are various places where the original, configured values remain.  You can see this if you run "python -m sysconfig" and examine the prefix values.  One result is that any scripts installed via Distutils (including using pip) will get installed to the configured location, e.g. default /usr/local/bin.  This also affects ensurepip (new in 3.4): the pip and easy_install scripts are installed in the configured value of prefix, e.g. /usr/local/bin, rather than the overridden value.  That's not good.

Changing prefix on the install targets seems like a fairly esoteric use case and one with an easy workaround: don't do that, use ./configure --prefix=/path/to instead.  I don't think it's worth trying to support since it may require some major Makefile work to get all the right dependencies.  But, at a minimum, it would be worth adding a test to the various install targets that would check and abort if prefix is attempted to be changed so you don't end up with a silently broken installation.
msg223320 - (view) Author: Martin v. Löwis (loewis) * (Python committer) Date: 2014-07-17 08:13
-1 on adding such a test. Some people may rely on the current behaviour, and use that to put the installation into some intermediate location (although DESTDIR would be a better approach for that). Also, how many variables would you want to protect from being changed? People may come up with ideas of changing arbitrary other Makefile variables, and guarding all of them is just not feasible.

I propose to close this as "won't fix".
msg223357 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2014-07-17 19:18
"Also, how many variables would you want to protect from being changed? People may come up with ideas of changing arbitrary other Makefile variables, and guarding all of them is just not feasible."

I certainly wasn't contemplating guarding *all* of them.  I'm concerned about those that are documented and conventionally used in standard Makefiles, e.g.  Of those, the only ones that appear to be overridable in the Python top-level Makefile are "prefix", "exec_prefix", and "datarootdir" (the latter does not seem to be used directly in the top-level Makefile but is used in ctypes libffi Makefile).  And "prefix" is the one that causes the problems.  If you really think someone is currently using this productively somehow, even with producing a broken Python, we could just issue a warning.  I just think we should not allow people to shoot themselves in the foot so easily without warning: when I do "make install", I expect everything to be installed under prefix, not partially under another prefix, and I expect subsequent installations using Distutils to work properly.
msg223387 - (view) Author: (cosmicduck) Date: 2014-07-17 22:50

this is a problem for all users which have to want to use python on there own home directory.
I wonder me, how did some other projects could install python to different directories e.g. OpenCSW it did on /opt/csw/...

My most problem is, that I can do many but without root access, which is denied. In this case a installation/build to a different path is very helpfull.

The situation was, that python and there most modules are working well also post installed ones.

Only the problem with the shebang to the scripts was the major problems which I've fixed by a line change on all scripts.
But to build python on a different root path e.g. my home folder, this should be changed simply by make itself and not by hand as I did it.

I'm unsure that other problems with my steps exists, but only the shebang problem is found by me.

This problem could also exists on later releases than 3.3. So for me it's ok to close this issue for 3.3. 
But for later releases, e.g. 3.4 it should checked and fixed, please.
msg223389 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2014-07-17 22:56
cosmicduck, the recommended and supported way to do this is by specifying the installation prefix at the configure step, not at the "make install" step:

  ./configure --prefix=${HOME}/scripts/tools/python/python-3.3.2
  make install

If done this way, the installed scripts will have the correct shebang lines and Distutils will install properly.
msg261826 - (view) Author: Martin Panter (martin.panter) * (Python committer) Date: 2016-03-15 20:44
I agree with Martin v. Löwis that this is not a real bug. Cosmicduck doesn’t seem aware of the “configure --prefix=. . .” option, but I don’t think it is right to add hacks to the makefile to prevent overriding variables.
msg261837 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2016-03-16 00:42
Allowing prefix to be overridden after configure-time is certainly not the most critical shortcoming of the Python build system and it is possible someone is relying on that behavior.  Since no one, including me, seems very interested at this point in trying to change it, let's close this issue.
Date User Action Args
2016-03-16 00:42:50ned.deilysetstatus: open -> closed
versions: + Python 3.6, - Python 3.4
messages: + msg261837

resolution: wont fix
stage: needs patch -> resolved
2016-03-15 23:20:56BreamoreBoysetnosy: - BreamoreBoy
2016-03-15 20:44:48martin.pantersetnosy: + martin.panter
messages: + msg261826
2014-07-17 22:56:41ned.deilysetmessages: + msg223389
2014-07-17 22:50:59cosmicducksetmessages: + msg223387
2014-07-17 19:18:16ned.deilysetmessages: + msg223357
2014-07-17 08:13:03loewissetnosy: + loewis
messages: + msg223320
2014-07-17 00:35:04ned.deilysetcomponents: + Build
title: idle, pydoc search python on wrong path at make altinstall with different exec- and prefix e.g. $HOME -> python installation is broken if prefix is overridden on an install target
keywords: + easy
nosy: + ned.deily

messages: + msg223306
stage: needs patch
2014-07-16 23:08:23terry.reedysetmessages: + msg223293
components: + Installation, - IDLE
versions: + Python 3.4, Python 3.5, - Python 3.3
2014-07-16 21:11:05BreamoreBoysetnosy: + terry.reedy, BreamoreBoy
messages: + msg223276
2013-06-28 07:28:59cosmicduckcreate