This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: sys.executable can be not normalized
Type: behavior Stage:
Components: Interpreter Core, Tests Versions: Python 3.10, Python 3.9, Python 3.8
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: eryksun, serhiy.storchaka, terry.reedy, vstinner
Priority: normal Keywords:

Created on 2017-07-25 12:41 by serhiy.storchaka, last changed 2022-04-11 14:58 by admin.

Messages (8)
msg299065 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2017-07-25 12:41
test_executable in Lib/test/test_sys.py tests that sys.executable is absolute. But this test is too strong, it fails if sys.executable is not normalized.

$ Lib/../python -m test test_sys
Run tests sequentially
0:00:00 load avg: 0.39 [1/1] test_sys
test test_sys failed -- Traceback (most recent call last):
  File "/home/serhiy/py/cpython/Lib/test/test_sys.py", line 649, in test_executable
    self.assertEqual(os.path.abspath(sys.executable), sys.executable)
AssertionError: '/home/serhiy/py/cpython/python' != '/home/serhiy/py/cpython/Lib/../python'
- /home/serhiy/py/cpython/python
+ /home/serhiy/py/cpython/Lib/../python
?                         +++++++


test_sys failed

1 test failed:
    test_sys

Total duration: 2 sec
Tests result: FAILURE

I don't know what is wrong: the value of sys.executable or the test.
msg299067 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2017-07-25 12:44
test_venv fails for the same cause.

$ Lib/../python -m test -m test_defaults test_venv
Run tests sequentially
0:00:00 load avg: 0.31 [1/1] test_venv
test test_venv failed -- Traceback (most recent call last):
  File "/home/serhiy/py/cpython/Lib/test/test_venv.py", line 103, in test_defaults
    self.assertIn('home = %s' % path, data)
AssertionError: 'home = /home/serhiy/py/cpython/Lib/..' not found in 'home = /home/serhiy/py/cpython\ninclude-system-site-packages = false\nversion = 3.7.0\n'

test_venv failed

1 test failed:
    test_venv

Total duration: 166 ms
Tests result: FAILURE
msg299068 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-07-25 12:45
Technically, /home/serhiy/py/cpython/Lib/../python is absolute and is a valid path. I'm not sure that it's possible to normalize sys.executable early in the Python initialization.

If we modify the site module to normalize sys.executable, it would mean that sys.executable wouldn't be normalized when using the python3 -S command line option. I dislike having subtle differences between site and no site.

I suggest to normalize sys.executable in the unit test.
msg299444 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2017-07-29 00:31
On Windows, sys.executable is normalized before or during startup. C:\Programs\Python36>Lib\..\python -m test test_sys prints
 test_executable (__main__.SysModuleTest) ... ok

C:\Programs\Python36>Lib\..\python -c "import sys; print(sys.executable)"
C:\Programs\Python36\python.exe
msg299456 - (view) Author: Eryk Sun (eryksun) * (Python triager) Date: 2017-07-29 05:36
Terry, the Windows implementation calls GetModuleFileNameW to get the executable's fully-qualified path from the loader. However, depending on how Python is started, this could be a \\?\ path, which leads to a bug in the startup code that determines the prefix path based on the landmark "lib\os.py". Probably the easiest solution would be to normalize the executable path to remove the \\?\ or \\.\ prefix. This prefix isn't useful here anyway since Windows doesn't properly support running programs from long paths, not even in Windows 10. I'll open a new issue for this.
msg387775 - (view) Author: Eryk Sun (eryksun) * (Python triager) Date: 2021-02-27 10:33
In Python 3.10 in POSIX, it's still the case that executable, prefix, exec_prefix, base_prefix, and base_exec_prefix in the sys module do not get normalized. For example, in Linux:

    $ .local/bin/../bin/python3.10 -c "import sys; print(sys.executable)"
    /home/someone/.local/bin/../bin/python3.10

In test/test_sys.py, assertEqual(os.path.abspath(sys.executable), sys.executable) fails. In test/test_venv.py, assertIn('home = %s' % path, data) and assertEqual(out.strip(), expected.encode()) both fail, respectively in test_defaults and test_prefixes.

In Windows, prior to Python 3.6, this can be a problem if Python is run via CreateProcessW using a non-normalized path in lpApplicationName (rare, but possible) instead of letting the system search for the executable in lpCommandLine. For example:

    >>> cmd = 'python -c "import sys; print(sys.executable)"'
    >>> exe = r'C:\Program Files\Python35\..\Python35\python.exe'
    >>> subprocess.call(cmd, executable=exe)
    C:\Program Files\Python35\..\Python35\python.exe
    0

It's no longer an issue in Windows with Python 3.6 and above. When getting the program path while initializing, the GetModuleFileNameW(NULL, ..) result gets canonicalized (e.g. via PathCchCanonicalizeEx). For example:

    >>> exe = r'C:\Program Files\Python36\..\Python36\python.exe'
    >>> subprocess.call(cmd, executable=exe)
    C:\Program Files\Python36\python.exe
    0
msg387782 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2021-02-27 18:48
This issue was opened with 2 related facts of 2017:
a) sys.executable is not normalized
b) as a result, two tests failed
Serhiy then said "I don't know what is wrong: the value of sys.executable or the test."  If it was the tests, they seem to have been fixed elsewhere.

If it is the non-normalization of sys.executable, my concern for IDLE is that it starts the execution subprocess with subprocess.Popen([sys.executable, ...]).  Could this be a reason for mysterious IDLE failures?

Eryk, you stated above that on Windows a \\?\ path is possible but will not work for startup, and that you would open an issue for this.  Have you?

Do any of the non-normalized Linux (or *nix) executable paths fail for startup?
msg387787 - (view) Author: Eryk Sun (eryksun) * (Python triager) Date: 2021-02-27 20:06
> If it was the tests, they seem to have been fixed elsewhere.

The cited tests haven't been changed to work with non-normalized sys.executable, sys.prefix, etc. But it's not common to run the test suite with a non-normalized path such as "Lib/../build/python".

> you stated above that on Windows a \\?\ path is possible but 
> will not work for startup, and that you would open an issue 

Sorry, I don't remember the details of the bug that I mentioned. But it became a non-issue in 3.6+, which canonicalizes the executable path. For example:

    >>> exe = r'\\?\C:\Program Files\Python36\python.exe'
    >>> cmd = 'python -c "import sys; print(sys.executable)"'
    >>> subprocess.call(cmd, executable=exe)
    C:\Program Files\Python36\python.exe
    0

Compare the latter to the output of GetModuleFileNameW(NULL, ...) in this case, which returns a \\?\ extended (verbatim) path:

    >>> exe = r'\\?\C:\Program Files\Python36\python.exe'
    >>> cmd = 'python -c "import _winapi; print(_winapi.GetModuleFileName(0))"'
    >>> subprocess.call(cmd, executable=exe)
    \\?\C:\Program Files\Python36\python.exe
    0

> Do any of the non-normalized Linux (or *nix) executable paths fail for startup?

Not that I'm aware of. AFAIK, it's just about the particular tests failing in this case.
History
Date User Action Args
2022-04-11 14:58:49adminsetgithub: 75213
2021-02-27 20:06:55eryksunsetmessages: + msg387787
2021-02-27 18:48:27terry.reedysetmessages: + msg387782
2021-02-27 10:33:32eryksunsetmessages: + msg387775
versions: + Python 3.8, Python 3.9, Python 3.10, - Python 2.7, Python 3.6, Python 3.7
2017-07-29 05:36:54eryksunsetnosy: + eryksun
messages: + msg299456
2017-07-29 00:31:28terry.reedysetnosy: + terry.reedy
messages: + msg299444
2017-07-25 12:45:57vstinnersetnosy: + vstinner
messages: + msg299068
2017-07-25 12:44:46serhiy.storchakasetmessages: + msg299067
2017-07-25 12:41:50serhiy.storchakacreate