New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
sys.executable: wrong location if zeroth command argument is modified. #52022
Comments
test_subprocess.test_executable now has a failure in the subprocess created (see http://www.python.org/dev/buildbot/builders/amd64%20gentoo%20trunk/builds/303/steps/test/logs/stdio) the bbots don't get red because this is happening in the subprocess and the test just look for the return code. This is hapenning since I've added the sysconfig module. This change revealed a bug in subprocess. This call : subprocess.Popen(["somethingyoudonthave", "-c", "import sys; print sys.executable"], executable=sys.executable) will print a directory name for sys.executable, instead of the real value. That's fooling sysconfig, which is now called through site.py, to set a few variables. |
Notice that this happen only under Linux and Solaris AFAIK. I couldn't reproduce it under Mac OS X |
Confirmed on all Python versions. Same behaviour without subprocess: |
I'm not the subprocess owner Tarek :( |
Ooops sorry Jesse, my brain has a hard link to your name when I see the word "process" ;) It comes out this bug is more related to 'sys' anyways. |
How would you suggest to implement sys.executable if not by looking at sys.argv[0] (or the C equivalent, rather)? |
It may help to find other methods: By the way, it should not "absolutize" the path when sys.executable is irrelevant. IMHO, it should render an empty thing in such case (code below). --- Modules/getpath.c (revision 77750)
+++ Modules/getpath.c (working copy)
@@ -441,7 +441,7 @@
}
else
progpath[0] = '\0';
- if (progpath[0] != SEP)
+ if (progpath[0] != SEP && progpath[0] != '\0')
absolutize(progpath);
strncpy(argv0_path, progpath, MAXPATHLEN);
argv0_path[MAXPATHLEN] = '\0'; |
readlink("/proc/self/exe") would work on linux. (there's a similar link on freebsd). |
Similar failures with or without argv 0: ~ $ sh -c 'exec -a missing ./python -c "import sys; print sys.executable"' ~ $ sh -c 'exec -a "" ./python -c "import sys; print sys.executable"' |
"Thanks to" my commit r78826 (fixing bpo-3137), site errors are now fatal: Python exits with an error instead of just printing "import site failed; ...". It's the come back of bpo-7774: test_executable_with_cwd of test_subprocess now always fail. The test fails because of an error raise by the site module: Traceback (most recent call last):
File "Lib/site.py", line 542, in <module>
main()
File "Lib/site.py", line 524, in main
known_paths = addusersitepackages(known_paths)
File "Lib/site.py", line 257, in addusersitepackages
user_site = getusersitepackages()
File "Lib/site.py", line 239, in getusersitepackages
user_base = getuserbase() # this will also set USER_BASE
File "Lib/site.py", line 229, in getuserbase
USER_BASE = get_config_var('userbase')
File "/home/SHARE/SVN/python-trunk/Lib/sysconfig.py", line 494, in get_config_var
return get_config_vars().get(name)
File "/home/SHARE/SVN/python-trunk/Lib/sysconfig.py", line 402, in get_config_vars
_init_posix(_CONFIG_VARS)
File "/home/SHARE/SVN/python-trunk/Lib/sysconfig.py", line 255, in _init_posix
raise IOError(msg)
IOError: invalid Python installation: unable to open /usr/local/lib/python2.7/config/Makefile (No such file or directory) |
This issue is not related to subprocess, subprocess is just used in examples to show the bug. The problem is that Python defines an invalid sys.executable if the process was created with a custom argv[0]. In Python trunk, the site module now uses sysconfig which depends on sys.executable. If sys.executable, sysconfig fails, site import fails, and finally Python exists (since my fix for bpo-3137). I think that we have to split the problem in two parts:
(1) Functions using (indirectly) sysconfig in site module:
I don't know if Python should exits with an error if these functions fail. (2) Except of Python unit test, is it required in "real world" applications? I know that Twisted does modify argv[0] to display "twistd" in process list. It looks like there is trivial, portable and/or reliable solution for this problem. |
Oooh, I shouldn't post messages at 3:05. "If sys.executable *is invalid*, sysconfig fails, site import fails, and finally Python *exits* (since my fix for bpo-3137)." "It looks like there is *no* trivial, portable and/or reliable solution *to fix* this problem." |
Twisted uses the following code (on UNIX-like OS): def launchWithName(name):
if name and name != sys.argv[0]:
exe = os.path.realpath(sys.executable)
log.msg('Changing process name to ' + name)
os.execv(exe, [name, sys.argv[0], '--originalname'] + sys.argv[1:]) Twisted doesn't care if sys.executable is invalid when argv[0] has been changed. sys.executable is only used to change the process name (sys.executable is used before argv[0] is modified). |
Actually there's 3 steps to fix this:
See message above for details on (3): Patch attached for (2). (Fixes tests) |
This doesn't fail on OSX when using a framework build of Python because that uses a different way to find sys.prefix: in Python.framework the shared library with the interpreter (libpython.so on Linux) is always in a known location relative to sys.prefix. |
getpath.patch: another patch written by flox. If argv[0] is set to a non existent program name, sys.executable is set to an empty string instead of a directory name. Example of the bug with Python 2.5: $ sh -c "exec -a xxx python -c 'import sys, os; print repr(sys.executable), repr(os.getcwd())'"
'/home/haypo/' '/home/haypo' |
r78835 (trunk) fixes getpath and includes also a fix for sysconfig which should fix test_executable_with_cwd() of test_subprocess. My fix on sysconfig is different than flox's patch: since I fixed getpath, sys.executable cannot be a directory name. At revision r78835, sys.executable can have 3 values: (c) is the worst case: Python refuses to start. r78835 fixes the site module error for case (b). There are different methods to get the real program name, but no one is portable. As flox wrote, we can "do a best effort to provide a valid sys.executable". Extract of stackoverflow link:
|
There is a 4th case about sys.executable: If argv[0] is an empty string, Python search the absolute path of "python" (hardcoded program name). The result is /usr/bin/python in my Linux box, whereas the real program name should be /opt/pythontrunk/bin/python. I think that this case is a bug and should be fixed. |
Oh well, it's really hard to test the tests in my working copy because it depends on a lot of factors: the current working directory, is Python installed or not, the OS, etc. The test introduced in r78835 was incorrect (only the test, the fix is correct): r78836 fixes the test for Windows (where sys.executable is correct even if argv[0] is set to a non existent program name, I don't understand how) and r78837 fixes the test for Ubuntu. r78837 is a temporary fix to workaround bpo-7774 bug: it's the same hack used in test_executable_with_cwd() from test_subprocess. |
Backported as r78868 to the py3k branch. |
Backport done: r79202 (2.6). |
The initial problem (test_subprocess failures) is solved thanks to different fixes. I consider that we can close this issue. Use non portable tricks to get the real program name if argv[0] is invalid doesn't sound like something importat. If someone changes argv[0], he have to assume that sys.executable is invalid or empty. |
I just created bpo-16170 about a newly-added subprocess.Popen() test that succeeds on Windows and Mac but fails on Linux. It seems closely related to the issue discussed here. Perhaps it signals an underlying issue that needs to be fixed. |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
bugs.python.org fields:
The text was updated successfully, but these errors were encountered: