classification
Title: sys.executable: wrong location if zeroth command argument is modified.
Type: behavior Stage: needs patch
Components: Interpreter Core Versions: Python 3.1, Python 3.2, Python 2.7, Python 2.6, Python 2.5
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: Arfrever, chris.jerdonek, ezio.melotti, flox, haypo, jnoller, pitrou, ronaldoussoren, schmir, tarek
Priority: normal Keywords: patch

Created on 2010-01-24 23:23 by tarek, last changed 2012-10-09 01:47 by chris.jerdonek. This issue is now closed.

Files
File name Uploaded Description Edit
issue7774_sysconfig.diff flox, 2010-03-11 02:29 Patch, apply to 2.x
getpath.patch haypo, 2010-03-11 11:30
Messages (24)
msg98257 - (view) Author: Tarek Ziadé (tarek) * (Python committer) Date: 2010-01-24 23:23
test_subprocess.test_executable now has a failure in the subprocess created (see http://www.python.org/dev/buildbot/builders/amd64%20gentoo%20trunk/builds/303/steps/test/logs/stdio)

the bbots don't get red because this is happening in the subprocess and the test just look for the return code.

This is hapenning since I've added the sysconfig module. This change revealed a bug in subprocess.

This call :

  subprocess.Popen(["somethingyoudonthave", "-c", "import sys; print sys.executable"], executable=sys.executable)   

will print a directory name for sys.executable, instead of the real value. That's fooling sysconfig, which is now called through site.py, to set a few variables.
msg98258 - (view) Author: Tarek Ziadé (tarek) * (Python committer) Date: 2010-01-24 23:24
Notice that this happen only under Linux and Solaris AFAIK. I couldn't reproduce it under Mac OS X
msg98259 - (view) Author: Florent Xicluna (flox) * (Python committer) Date: 2010-01-24 23:39
Confirmed on all Python versions.

Same behaviour without subprocess:
~ $ sh -c "exec -a missingfile python -c 'import sys; print sys.executable'"
/home/name/
msg98261 - (view) Author: Jesse Noller (jnoller) * (Python committer) Date: 2010-01-25 01:40
I'm not the subprocess owner Tarek :(
msg98282 - (view) Author: Tarek Ziadé (tarek) * (Python committer) Date: 2010-01-25 12:09
Ooops sorry Jesse, my brain has a hard link to your name when I see the word "process" ;)

It comes out this bug is more related to 'sys' anyways.
msg98283 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2010-01-25 12:57
How would you suggest to implement sys.executable if not by looking at sys.argv[0] (or the C equivalent, rather)?
msg98284 - (view) Author: Florent Xicluna (flox) * (Python committer) Date: 2010-01-25 13:06
It may help to find other methods:
http://stackoverflow.com/questions/1023306/finding-current-executables-path-without-proc-self-exe


By the way, it should not "absolutize" the path when sys.executable is irrelevant. IMHO, it should render an empty thing in such case (code below).


--- Modules/getpath.c   (revision 77750)
+++ Modules/getpath.c   (working copy)
@@ -441,7 +441,7 @@
        }
        else
                progpath[0] = '\0';
-       if (progpath[0] != SEP)
+       if (progpath[0] != SEP && progpath[0] != '\0')
                absolutize(progpath);
        strncpy(argv0_path, progpath, MAXPATHLEN);
        argv0_path[MAXPATHLEN] = '\0';
msg98285 - (view) Author: Ralf Schmitt (schmir) Date: 2010-01-25 13:06
readlink("/proc/self/exe") would work on linux. (there's a similar link on freebsd).
msg99203 - (view) Author: Florent Xicluna (flox) * (Python committer) Date: 2010-02-11 09:51
Similar failures with or without argv 0:

 ~ $ sh -c 'exec -a missing ./python -c "import sys; print sys.executable"'
/home/user/dev/python/trunk/

 ~ $ sh -c 'exec -a "" ./python -c "import sys; print sys.executable"'
Could not find platform independent libraries <prefix>
Could not find platform dependent libraries <exec_prefix>
Consider setting $PYTHONHOME to <prefix>[:<exec_prefix>]
'import site' failed; use -v for traceback
/usr/bin/python
msg100820 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2010-03-11 00:34
"Thanks to" my commit r78826 (fixing #3137), site errors are now fatal: Python exits with an error instead of just printing "import site failed; ...". It's the come back of #7774: test_executable_with_cwd of test_subprocess now always fail.

The test fails because of an error raise by the site module:
-----
Traceback (most recent call last):
  File "Lib/site.py", line 542, in <module>
    main()
  File "Lib/site.py", line 524, in main
    known_paths = addusersitepackages(known_paths)
  File "Lib/site.py", line 257, in addusersitepackages
    user_site = getusersitepackages()
  File "Lib/site.py", line 239, in getusersitepackages
    user_base = getuserbase() # this will also set USER_BASE
  File "Lib/site.py", line 229, in getuserbase
    USER_BASE = get_config_var('userbase')
  File "/home/SHARE/SVN/python-trunk/Lib/sysconfig.py", line 494, in get_config_var
    return get_config_vars().get(name)
  File "/home/SHARE/SVN/python-trunk/Lib/sysconfig.py", line 402, in get_config_vars
    _init_posix(_CONFIG_VARS)
  File "/home/SHARE/SVN/python-trunk/Lib/sysconfig.py", line 255, in _init_posix
    raise IOError(msg)
IOError: invalid Python installation: unable to open /usr/local/lib/python2.7/config/Makefile (No such file or directory)
-----
msg100822 - (view) Author: Florent Xicluna (flox) * (Python committer) Date: 2010-03-11 01:04
Fixed test failures related to incorrect sys.executable, which were visible after #3137 was fixed. Changesets r78830 and r78831.

Note: there's 2 comments related to #7774. To review when this issue is fixed:
 - Lib/sysconfig.py
 - Lib/test/test_subprocess.py
msg100827 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2010-03-11 02:01
This issue is not related to subprocess, subprocess is just used in examples to show the bug. The problem is that Python defines an invalid sys.executable if the process was created with a custom argv[0]. In Python trunk, the site module now uses sysconfig which depends on sys.executable. If sys.executable, sysconfig fails, site import fails, and finally Python exists (since my fix for #3137). I think that we have to split the problem in two parts:

 1) Python startup fails if sysconfig raises an error
 2) sys.executable is invalid is argv[0] is modified

(1) Functions using (indirectly) sysconfig in site module:

 - addbuilddir()
 - addusersitepackages()

I don't know if Python should exits with an error if these functions fail.

(2) Except of Python unit test, is it required in "real world" applications? I know that Twisted does modify argv[0] to display "twistd" in process list. It looks like there is trivial, portable and/or reliable solution for this problem.
msg100828 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2010-03-11 02:06
Oooh, I shouldn't post messages at 3:05.

"If sys.executable *is invalid*, sysconfig fails, site import fails, and finally Python *exits* (since my fix for #3137)."

"It looks like there is *no* trivial, portable and/or reliable solution *to fix* this problem."
msg100829 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2010-03-11 02:18
Twisted uses the following code (on UNIX-like OS):

def launchWithName(name):
    if name and name != sys.argv[0]:
        exe = os.path.realpath(sys.executable)
        log.msg('Changing process name to ' + name)
        os.execv(exe, [name, sys.argv[0], '--originalname'] + sys.argv[1:])

Twisted doesn't care if sys.executable is invalid when argv[0] has been changed. sys.executable is only used to change the process name (sys.executable is used before argv[0] is modified).
msg100830 - (view) Author: Florent Xicluna (flox) * (Python committer) Date: 2010-03-11 02:29
Actually there's 3 steps to fix this:
1) do not absolutize the sys.executable if it is empty (in Modules/getpath.c)
2) change sysconfig.py to deal with empty sys.executable (see this patch)
3) do a best effort to provide a valid sys.executable.
   It should be possible in most cases.

See message above for details on (3):
http://bugs.python.org/issue7774#msg98284

Patch attached for (2). (Fixes tests)
msg100838 - (view) Author: Ronald Oussoren (ronaldoussoren) * (Python committer) Date: 2010-03-11 06:58
This doesn't fail on OSX when using a framework build of Python because that uses a different way to find sys.prefix: in Python.framework the shared library with the interpreter (libpython.so on Linux) is always in a known location relative to sys.prefix.
msg100843 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2010-03-11 11:30
getpath.patch: another patch written by flox. If argv[0] is set to a non existent program name, sys.executable is set to an empty string instead of a directory name.

Example of the bug with Python 2.5:

$ sh -c "exec -a xxx python -c 'import sys, os; print repr(sys.executable), repr(os.getcwd())'"
'/home/haypo/' '/home/haypo'
msg100849 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2010-03-11 12:56
r78835 (trunk) fixes getpath and includes also a fix for sysconfig which should fix test_executable_with_cwd() of test_subprocess.

My fix on sysconfig is different than flox's patch: since I fixed getpath, sys.executable cannot be a directory name. At revision r78835, sys.executable can have 3 values:
 (a) absolute path to Python executable name (most common case)
 (b) empty string if argv[0] has been set to a non existent program name
 (c) absolute path to a different program if argv[0] has been set to a different program, with or without path (eg. "true" and "/bin/path")

(c) is the worst case: Python refuses to start. r78835 fixes the site module error for case (b).

There are different methods to get the real program name, but no one is portable. As flox wrote, we can "do a best effort to provide a valid sys.executable". Extract of stackoverflow link:

 * Mac OS X: _NSGetExecutablePath() (man 3 dyld)
 * Linux: readlink /proc/self/exe
 * Solaris: getexecname()
 * FreeBSD: sysctl CTL_KERN KERN_PROC KERN_PROC_PATHNAME -1
 * BSD with procfs: readlink /proc/curproc/file
 * Windows: GetModuleFileName() with hModule = NULL
msg100850 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2010-03-11 13:28
There is a 4th case about sys.executable: If argv[0] is an empty string, Python search the absolute path of "python" (hardcoded program name). The result is /usr/bin/python in my Linux box, whereas the real program name should be /opt/pythontrunk/bin/python. I think that this case is a bug and should be fixed.
msg100851 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2010-03-11 13:52
Oh well, it's really hard to test the tests in my working copy because it depends on a lot of factors: the current working directory, is Python installed or not, the OS, etc. The test introduced in r78835 was incorrect (only the test, the fix is correct): r78836 fixes the test for Windows (where sys.executable is correct even if argv[0] is set to a non existent program name, I don't understand how) and r78837 fixes the test for Ubuntu.

r78837 is a temporary fix to workaround #7774 bug: it's the same hack used in test_executable_with_cwd() from test_subprocess.
msg100934 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2010-03-12 14:21
> r78835 (trunk) fixes getpath and includes also a fix for sysconfig which should fix test_executable_with_cwd() of test_subprocess.

Backported as r78868 to the py3k branch.
msg101425 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2010-03-21 13:50
> r78835 (trunk) fixes getpath

Backport done: r79202 (2.6).
msg101472 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2010-03-22 00:33
The initial problem (test_subprocess failures) is solved thanks to different fixes. I consider that we can close this issue. Use non portable tricks to get the real program name if argv[0] is invalid doesn't sound like something importat. If someone changes argv[0], he have to assume that sys.executable is invalid or empty.
msg172434 - (view) Author: Chris Jerdonek (chris.jerdonek) * (Python committer) Date: 2012-10-09 01:47
I just created issue 16170 about a newly-added subprocess.Popen() test that succeeds on Windows and Mac but fails on Linux.  It seems closely related to the issue discussed here.

Perhaps it signals an underlying issue that needs to be fixed.
History
Date User Action Args
2012-10-09 01:47:04chris.jerdoneksetnosy: + chris.jerdonek
messages: + msg172434
2010-03-22 00:33:15hayposetstatus: open -> closed
resolution: fixed
messages: + msg101472
2010-03-21 13:50:02hayposetmessages: + msg101425
2010-03-14 14:01:11Arfreversetnosy: + Arfrever
2010-03-12 14:21:51hayposetmessages: + msg100934
2010-03-11 14:56:00adminsetmessages: + msg98284
2010-03-11 13:52:03hayposetmessages: + msg100851
2010-03-11 13:50:33hayposetmessages: - msg98284
2010-03-11 13:28:26hayposetmessages: + msg100850
2010-03-11 12:56:44hayposetmessages: + msg100849
2010-03-11 11:30:34hayposetfiles: + getpath.patch

messages: + msg100843
2010-03-11 06:58:11ronaldoussorensetnosy: + ronaldoussoren
messages: + msg100838
2010-03-11 02:29:17floxsetfiles: + issue7774_sysconfig.diff
keywords: + patch
messages: + msg100830
2010-03-11 02:18:46hayposetmessages: + msg100829
2010-03-11 02:06:52hayposetmessages: + msg100828
2010-03-11 02:01:43hayposetmessages: + msg100827
2010-03-11 01:04:52floxsetmessages: + msg100822
2010-03-11 00:34:40hayposetmessages: + msg100820
2010-03-05 15:38:00hayposetnosy: + haypo
2010-02-11 09:51:06floxsetmessages: + msg99203
2010-02-09 12:31:52floxsetnosy: pitrou, schmir, tarek, jnoller, ezio.melotti, flox
components: + Interpreter Core, - Library (Lib)
stage: test needed -> needs patch
2010-01-25 13:06:38schmirsetnosy: + schmir
messages: + msg98285
2010-01-25 13:06:03floxsetmessages: + msg98284
2010-01-25 12:57:04pitrousetnosy: + pitrou
messages: + msg98283
2010-01-25 12:09:57tareksetmessages: + msg98282
2010-01-25 01:40:10jnollersetassignee: jnoller ->
messages: + msg98261
2010-01-24 23:40:53floxsettitle: subprocess executable option wrong location -> sys.executable: wrong location if zeroth command argument is modified.
2010-01-24 23:39:51floxsetversions: + Python 2.6, Python 2.5, Python 3.1
2010-01-24 23:39:26floxsetnosy: + flox
messages: + msg98259
2010-01-24 23:28:39ezio.melottisetnosy: + ezio.melotti

stage: test needed
2010-01-24 23:24:08tareksetmessages: + msg98258
2010-01-24 23:23:20tarekcreate