Title: Inconsistent behavior of search_for_exec_prefix() results in startup failure in certain cases
Type: crash Stage: resolved
Components: Interpreter Core Versions: Python 3.2, Python 3.3, Python 3.4, Python 2.7
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: Muflone, foutrelis, loewis, ncoghlan, vstinner
Priority: normal Keywords:

Created on 2012-09-19 19:57 by foutrelis, last changed 2020-02-12 01:29 by vstinner. This issue is now closed.

Messages (3)
msg170760 - (view) Author: Evangelos Foutras (foutrelis) Date: 2012-09-19 19:57
On Arch Linux /lib is a symbolic link to /usr/lib. When the Python interpreter is provided with an argv[0] of e.g. '/python2.7' and the current working directory is /, it'll fail to start with the following error:

IOError: invalid Python installation: unable to open //include/python2.7/pyconfig.h (No such file or directory)

From what I understand, what is happening inside Modules/getpath.c is:

1) search_for_exec_prefix() is given an empty `argv0_path`
2) PYTHONHOME is not set, and we're not in a build directory, so step three is executed:

 * Step 3. Try to find prefix and exec_prefix relative to argv0_path,
 * backtracking up the path until it is exhausted.  This is the most common
 * step to succeed.  Note that if prefix and exec_prefix are different,
 * exec_prefix is more likely to be found; however if exec_prefix is a
 * subdirectory of prefix, both will be found.

3) copy_absolute() sets `exec_prefix` to '/'
4) 'lib/python2.7' gets appended to `exec_prefix` using joinpath()
5) 'lib-dynload' gets appended to `exec_prefix` using joinpath()
6) '/lib/python2.7/lib-dynload' exists and the function returns 1 (success)
7) control is returned to calculate_path() which later reduces `exec_prefix` to '/'

During further initialization, tries to open pyconfig.h, whose path is calculated as {exec_prefix}/include/python2.7/pyconfig.h; thus ending up with the nonexistent path //include/python2.7/pyconfig.h. The correct exec_prefix would be /usr.

Moreover, if argv[0] and/or the current working directory are one level deeper (or more), `exec_prefix` will not be reduced to '/' and search_for_exec_prefix() will proceed to step four:

 * Step 4. Search the directories pointed to by the preprocessor variables
 * PREFIX and EXEC_PREFIX.  These are supplied by the Makefile but can be
 * passed in as options to the configure script.

i.e.: If search_for_exec_prefix() is passed an `argv0_path` with the value '/mnt', step three will only check '/mnt' but not '/', because '/mnt' will be reduced to '' and the `while (exec_prefix[0])` condition will be false.

I see two problems with the behavior I describe above:

1) Step three will skip checking the root directory (/) if argv[0] or the current working directory are one or more levels below / (in other words, not directly under /). Its behavior in this regard is inconsistent.
2) When argv[0] is e.g. '/python2.7' and the current working directory is /, it'll use '/' as the exec_prefix and fail to start. The /lib -> /usr/lib symbolic link should get dereferenced and not used as is.

I'm not sure how this should be fixed, so I only tried to present the issue with as many details as I could. If something is unclear, let me know.

Lastly, search_for_prefix() has very similar code, so any fix will have to be applied there too.

(There is also a downstream bug report @
msg182342 - (view) Author: (Muflone) Date: 2013-02-18 21:03
Confirmed for Arch Linux x86_64 with python version 2.7.3
msg361858 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-02-12 01:29
This issue was reported on Python 2.7 eight years ago. getpath.c has been deeply reworked, especially by the PEP 587 (PyConfig) implementation. I consider that the issue is now fixed. If it's not the case, feel free to reopen it.
Date User Action Args
2020-02-12 01:29:27vstinnersetstatus: open -> closed

nosy: + vstinner
messages: + msg361858

resolution: fixed
stage: needs patch -> resolved
2013-02-19 14:53:24ezio.melottisetnosy: + loewis, ncoghlan
stage: needs patch

versions: + Python 3.4
2013-02-18 21:03:26Muflonesetnosy: + Muflone

messages: + msg182342
versions: + Python 3.3
2012-09-19 19:57:40foutreliscreate