classification
Title: __code__. co_filename should always be an absolute path
Type: behavior Stage: patch review
Components: Versions: Python 3.5
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: ncoghlan, r.david.murray, vstinner, xtreak, yselivanov
Priority: normal Keywords: patch

Created on 2014-01-30 02:15 by yselivanov, last changed 2019-06-28 14:49 by vstinner.

Pull Requests
URL Status Linked Edit
PR 13527 closed BTaskaya, 2019-05-23 19:20
PR 14053 merged vstinner, 2019-06-13 11:42
PR 14446 merged vstinner, 2019-06-28 14:17
Messages (10)
msg209699 - (view) Author: Yury Selivanov (yselivanov) * (Python committer) Date: 2014-01-30 02:15
There are many issues on tracker related to the relative paths in co_filename. Most of them are about introspection functions in inspect module--which are usually broken after 'os.chdir'. 

Test case: create a file t.py:

   def foo(): pass
   print(foo.__code__.co_filename)

Execute it with '$ python t.py' -> it will print 't.py'.

Ideally, when executing a python file, interpreter should expand all relative paths for __file__ and __code__.co_filename attributes.
msg345501 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-06-13 11:43
PR 14053 is a different approach than PR 13527: compute the absolute path to the script filename in PyConfig_Read() just after parsing the command line.
msg345521 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-06-13 14:30
One of the side effect of my PR 14053 is that tracebacks are more verbose: the filename is now absolute rather than relative.

Currently:

$ python3 x.py 
Traceback (most recent call last):
  File "x.py", line 4, in <module>
    func()
  File "x.py", line 2, in func
    bug
NameError: name 'bug' is not defined

With my PR:

$ ./python x.py 
Traceback (most recent call last):
  File "/home/vstinner/prog/python/master/x.py", line 4, in <module>
    func()
  File "/home/vstinner/prog/python/master/x.py", line 2, in func
    bug
NameError: name 'bug' is not defined
msg345585 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-06-14 11:13
The site module tries to compute the absolute path of __file__ and __cached__ attributes of all modules in sys.modules:

def abs_paths():
    """Set all module __file__ and __cached__ attributes to an absolute path"""
    for m in set(sys.modules.values()):
        if (getattr(getattr(m, '__loader__', None), '__module__', None) not in
                ('_frozen_importlib', '_frozen_importlib_external')):
            continue   # don't mess with a PEP 302-supplied __file__
        try:
            m.__file__ = os.path.abspath(m.__file__)
        except (AttributeError, OSError, TypeError):
            pass
        try:
            m.__cached__ = os.path.abspath(m.__cached__)
        except (AttributeError, OSError, TypeError):
            pass

The __path__ attribute isn't updated.

Another approach would be to hack importlib to compute the absolute path before loading a module, rather than trying to fix it *afterwards*.

One pratical problem: posixpath and ntpath are not available when importlib is setup, these modules are implemented in pure Python and so must be imported.

Maybe importlib could use a naive implementation of os.path.abspath(). Maybe the C function _Py_abspath() that I implemented in PR 14053 should be exposed somehow to importlib as a private function using a builtin module like _imp, so it can be used directly.
msg346207 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2019-06-21 11:56
Perhaps `os._abspath_for_import`? If importlib finds it, then it can handle making early paths absolute itself, otherwise it will defer doing that until it has the full external import system bootstrapped. (importlib does try to make all paths absolute as it loads modules - it's just that some early modules aren't loaded that way)
msg346262 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-06-21 22:35
Effect of my PR 14053 using script.py:

import sys
print(f"{__file__=}")
print(f"{sys.argv=}")
print(f"{sys.path[0]=}")

Before:

$ ./python script.py 
__file__=script.py
sys.argv=['script.py']
sys.path[0]=/home/vstinner/prog/python/master

With the change:

$ ./python script.py 
__file__=/home/vstinner/prog/python/master/script.py
sys.argv=['/home/vstinner/prog/python/master/script.py']
sys.path[0]=/home/vstinner/prog/python/master
msg346524 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-06-25 13:03
New changeset 3939c321c90283b49eddde762656e4b1940e7150 by Victor Stinner in branch 'master':
bpo-20443: _PyConfig_Read() gets the absolute path of run_filename (GH-14053)
https://github.com/python/cpython/commit/3939c321c90283b49eddde762656e4b1940e7150
msg346789 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-06-28 00:18
Example of case where a module path is still relative:
---
import sys
import os
modname = 'relpath'
filename = modname + '.py'
sys.path.insert(0, os.curdir)
with open(filename, "w") as fp:
    print("import sys", file=fp)
    print("mod = sys.modules[__name__]", file=fp)
    print("print(f'{__file__=}')", file=fp)
    print("print(f'{mod.__file__=}')", file=fp)
    print("print(f'{mod.__cached__=}')", file=fp)
__import__(modname)
os.unlink(filename)
---

Output:
---
__file__='./relpath.py'
mod.__file__='./relpath.py'
mod.__cached__='./__pycache__/relpath.cpython-39.pyc'
---

__file__ and mod.__file__ are relative paths: not absolute paths.
msg346800 - (view) Author: Karthikeyan Singaravelan (xtreak) * (Python triager) Date: 2019-06-28 04:46
This change seems to produce a new warning on my Mac.

./Modules/getpath.c:764:23: warning: incompatible pointer types passing 'char [1025]' to parameter of type 'const wchar_t *' (aka 'const int *') [-Wincompatible-pointer-types]
            _Py_isabs(execpath))
                      ^~~~~~~~
./Include/fileutils.h:158:42: note: passing argument to parameter 'path' here
PyAPI_FUNC(int) _Py_isabs(const wchar_t *path);
                                         ^
1 warning generated.
msg346823 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-06-28 14:49
New changeset 3029035ef34c9bae0c8d965290cd9b273c8de1ea by Victor Stinner in branch 'master':
bpo-20443: Fix calculate_program_full_path() warning (GH-14446)
https://github.com/python/cpython/commit/3029035ef34c9bae0c8d965290cd9b273c8de1ea
History
Date User Action Args
2019-06-28 14:49:41vstinnersetmessages: + msg346823
2019-06-28 14:17:51vstinnersetpull_requests: + pull_request14262
2019-06-28 04:46:45xtreaksetnosy: + xtreak
messages: + msg346800
2019-06-28 00:18:23vstinnersetmessages: + msg346789
2019-06-25 13:03:08vstinnersetmessages: + msg346524
2019-06-21 22:35:17vstinnersetmessages: + msg346262
2019-06-21 11:56:15ncoghlansetmessages: + msg346207
2019-06-14 11:13:33vstinnersetmessages: + msg345585
2019-06-13 14:30:45vstinnersetmessages: + msg345521
2019-06-13 11:43:55vstinnersetnosy: + vstinner
messages: + msg345501
2019-06-13 11:42:55vstinnersetpull_requests: + pull_request13915
2019-05-23 19:20:53BTaskayasetkeywords: + patch
stage: needs patch -> patch review
pull_requests: + pull_request13441
2014-01-30 02:15:56yselivanovsetnosy: + ncoghlan
2014-01-30 02:15:15yselivanovcreate