test_subprocess.test_executable now has a failure in the subprocess created (see http://www.python.org/dev/buildbot/builders/amd64%20gentoo%20trunk/builds/303/steps/test/logs/stdio)

the bbots don't get red because this is happening in the subprocess and the test just look for the return code.

This is hapenning since I've added the sysconfig module. This change revealed a bug in subprocess.

This call :

  subprocess.Popen(["somethingyoudonthave", "-c", "import sys; print sys.executable"], executable=sys.executable)   

will print a directory name for sys.executable, instead of the real value. That's fooling sysconfig, which is now called through site.py, to set a few variables.

Notice that this happen only under Linux and Solaris AFAIK. I couldn't reproduce it under Mac OS X

Confirmed on all Python versions.

Same behaviour without subprocess:
~ $ sh -c "exec -a missingfile python -c 'import sys; print sys.executable'"
/home/name/

I'm not the subprocess owner Tarek :(

Ooops sorry Jesse, my brain has a hard link to your name when I see the word "process" ;)

It comes out this bug is more related to 'sys' anyways.

How would you suggest to implement sys.executable if not by looking at sys.argv[0] (or the C equivalent, rather)?

It may help to find other methods:
http://stackoverflow.com/questions/1023306/finding-current-executables-path-without-proc-self-exe


By the way, it should not "absolutize" the path when sys.executable is irrelevant. IMHO, it should render an empty thing in such case (code below).


--- Modules/getpath.c   (revision 77750)
+++ Modules/getpath.c   (working copy)
@@ -441,7 +441,7 @@
        }
        else
                progpath[0] = '\0';
-       if (progpath[0] != SEP)
+       if (progpath[0] != SEP && progpath[0] != '\0')
                absolutize(progpath);
        strncpy(argv0_path, progpath, MAXPATHLEN);
        argv0_path[MAXPATHLEN] = '\0';

readlink("/proc/self/exe") would work on linux. (there's a similar link on freebsd).

Similar failures with or without argv 0:

 ~ $ sh -c 'exec -a missing ./python -c "import sys; print sys.executable"'
/home/user/dev/python/trunk/

 ~ $ sh -c 'exec -a "" ./python -c "import sys; print sys.executable"'
Could not find platform independent libraries <prefix>
Could not find platform dependent libraries <exec_prefix>
Consider setting $PYTHONHOME to <prefix>[:<exec_prefix>]
'import site' failed; use -v for traceback
/usr/bin/python

"Thanks to" my commit r78826 (fixing #3137), site errors are now fatal: Python exits with an error instead of just printing "import site failed; ...". It's the come back of #7774: test_executable_with_cwd of test_subprocess now always fail.

The test fails because of an error raise by the site module:
-----
Traceback (most recent call last):
  File "Lib/site.py", line 542, in <module>
    main()
  File "Lib/site.py", line 524, in main
    known_paths = addusersitepackages(known_paths)
  File "Lib/site.py", line 257, in addusersitepackages
    user_site = getusersitepackages()
  File "Lib/site.py", line 239, in getusersitepackages
    user_base = getuserbase() # this will also set USER_BASE
  File "Lib/site.py", line 229, in getuserbase
    USER_BASE = get_config_var('userbase')
  File "/home/SHARE/SVN/python-trunk/Lib/sysconfig.py", line 494, in get_config_var
    return get_config_vars().get(name)
  File "/home/SHARE/SVN/python-trunk/Lib/sysconfig.py", line 402, in get_config_vars
    _init_posix(_CONFIG_VARS)
  File "/home/SHARE/SVN/python-trunk/Lib/sysconfig.py", line 255, in _init_posix
    raise IOError(msg)
IOError: invalid Python installation: unable to open /usr/local/lib/python2.7/config/Makefile (No such file or directory)
-----

Fixed test failures related to incorrect sys.executable, which were visible after #3137 was fixed. Changesets r78830 and r78831.

Note: there's 2 comments related to #7774. To review when this issue is fixed:
 - Lib/sysconfig.py
 - Lib/test/test_subprocess.py

This issue is not related to subprocess, subprocess is just used in examples to show the bug. The problem is that Python defines an invalid sys.executable if the process was created with a custom argv[0]. In Python trunk, the site module now uses sysconfig which depends on sys.executable. If sys.executable, sysconfig fails, site import fails, and finally Python exists (since my fix for #3137). I think that we have to split the problem in two parts:

 1) Python startup fails if sysconfig raises an error
 2) sys.executable is invalid is argv[0] is modified

(1) Functions using (indirectly) sysconfig in site module:

 - addbuilddir()
 - addusersitepackages()

I don't know if Python should exits with an error if these functions fail.

(2) Except of Python unit test, is it required in "real world" applications? I know that Twisted does modify argv[0] to display "twistd" in process list. It looks like there is trivial, portable and/or reliable solution for this problem.

Oooh, I shouldn't post messages at 3:05.

"If sys.executable *is invalid*, sysconfig fails, site import fails, and finally Python *exits* (since my fix for #3137)."

"It looks like there is *no* trivial, portable and/or reliable solution *to fix* this problem."

Twisted uses the following code (on UNIX-like OS):

def launchWithName(name):
    if name and name != sys.argv[0]:
        exe = os.path.realpath(sys.executable)
        log.msg('Changing process name to ' + name)
        os.execv(exe, [name, sys.argv[0], '--originalname'] + sys.argv[1:])

Twisted doesn't care if sys.executable is invalid when argv[0] has been changed. sys.executable is only used to change the process name (sys.executable is used before argv[0] is modified).

Actually there's 3 steps to fix this:
1) do not absolutize the sys.executable if it is empty (in Modules/getpath.c)
2) change sysconfig.py to deal with empty sys.executable (see this patch)
3) do a best effort to provide a valid sys.executable.
   It should be possible in most cases.

See message above for details on (3):
http://bugs.python.org/issue7774#msg98284

Patch attached for (2). (Fixes tests)

This doesn't fail on OSX when using a framework build of Python because that uses a different way to find sys.prefix: in Python.framework the shared library with the interpreter (libpython.so on Linux) is always in a known location relative to sys.prefix.

getpath.patch: another patch written by flox. If argv[0] is set to a non existent program name, sys.executable is set to an empty string instead of a directory name.

Example of the bug with Python 2.5:

$ sh -c "exec -a xxx python -c 'import sys, os; print repr(sys.executable), repr(os.getcwd())'"
'/home/haypo/' '/home/haypo'

r78835 (trunk) fixes getpath and includes also a fix for sysconfig which should fix test_executable_with_cwd() of test_subprocess.

My fix on sysconfig is different than flox's patch: since I fixed getpath, sys.executable cannot be a directory name. At revision r78835, sys.executable can have 3 values:
 (a) absolute path to Python executable name (most common case)
 (b) empty string if argv[0] has been set to a non existent program name
 (c) absolute path to a different program if argv[0] has been set to a different program, with or without path (eg. "true" and "/bin/path")

(c) is the worst case: Python refuses to start. r78835 fixes the site module error for case (b).

There are different methods to get the real program name, but no one is portable. As flox wrote, we can "do a best effort to provide a valid sys.executable". Extract of stackoverflow link:

 * Mac OS X: _NSGetExecutablePath() (man 3 dyld)
 * Linux: readlink /proc/self/exe
 * Solaris: getexecname()
 * FreeBSD: sysctl CTL_KERN KERN_PROC KERN_PROC_PATHNAME -1
 * BSD with procfs: readlink /proc/curproc/file
 * Windows: GetModuleFileName() with hModule = NULL

There is a 4th case about sys.executable: If argv[0] is an empty string, Python search the absolute path of "python" (hardcoded program name). The result is /usr/bin/python in my Linux box, whereas the real program name should be /opt/pythontrunk/bin/python. I think that this case is a bug and should be fixed.

Oh well, it's really hard to test the tests in my working copy because it depends on a lot of factors: the current working directory, is Python installed or not, the OS, etc. The test introduced in r78835 was incorrect (only the test, the fix is correct): r78836 fixes the test for Windows (where sys.executable is correct even if argv[0] is set to a non existent program name, I don't understand how) and r78837 fixes the test for Ubuntu.

r78837 is a temporary fix to workaround #7774 bug: it's the same hack used in test_executable_with_cwd() from test_subprocess.

> r78835 (trunk) fixes getpath and includes also a fix for sysconfig which should fix test_executable_with_cwd() of test_subprocess.

Backported as r78868 to the py3k branch.

> r78835 (trunk) fixes getpath

Backport done: r79202 (2.6).

The initial problem (test_subprocess failures) is solved thanks to different fixes. I consider that we can close this issue. Use non portable tricks to get the real program name if argv[0] is invalid doesn't sound like something importat. If someone changes argv[0], he have to assume that sys.executable is invalid or empty.

I just created issue 16170 about a newly-added subprocess.Popen() test that succeeds on Windows and Mac but fails on Linux.  It seems closely related to the issue discussed here.

Perhaps it signals an underlying issue that needs to be fixed.

New changeset 0ef8c157e9195df0115c54ba875a5efb92ac22fb by Victor Stinner in branch 'master':
bpo-28552: Fix distutils.sysconfig for empty sys.executable (GH-12875)
https://github.com/python/cpython/commit/0ef8c157e9195df0115c54ba875a5efb92ac22fb

New changeset 3076a3e0d1c54a2a6cc54c84521cd0f640d7cffb by Victor Stinner (Miss Islington (bot)) in branch '3.7':
bpo-28552: Fix distutils.sysconfig for empty sys.executable (GH-12875) (GH-12948)
https://github.com/python/cpython/commit/3076a3e0d1c54a2a6cc54c84521cd0f640d7cffb

New changeset f4edd39017a211d4544570a1e2ac2110ef8e51b4 by Victor Stinner in branch '2.7':
bpo-28552: Fix distutils.sysconfig for empty sys.executable (GH-12875) (GH-12949)
https://github.com/python/cpython/commit/f4edd39017a211d4544570a1e2ac2110ef8e51b4