When there is a file called "sys" in the local directory of a python script and warning.warn is called with an invalid stacklevel, python tries to import that file and throws an error like the following:

>>> import warnings
>>> warnings.warn("foo", Warning, stacklevel=2)
Traceback (most recent call last):
  File "/usr/lib/python3.5/tokenize.py", line 392, in find_cookie
    line_string = line.decode('utf-8')
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x80 in position 24: invalid start byte

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python3.5/warnings.py", line 18, in showwarning
    file.write(formatwarning(message, category, filename, lineno, line))
  File "/usr/lib/python3.5/warnings.py", line 26, in formatwarning
    line = linecache.getline(filename, lineno) if line is None else line
  File "/usr/lib/python3.5/linecache.py", line 16, in getline
    lines = getlines(filename, module_globals)
  File "/usr/lib/python3.5/linecache.py", line 47, in getlines
    return updatecache(filename, module_globals)
  File "/usr/lib/python3.5/linecache.py", line 136, in updatecache
    with tokenize.open(fullname) as fp:
  File "/usr/lib/python3.5/tokenize.py", line 456, in open
    encoding, lines = detect_encoding(buffer.readline)
  File "/usr/lib/python3.5/tokenize.py", line 433, in detect_encoding
    encoding = find_cookie(first)
  File "/usr/lib/python3.5/tokenize.py", line 397, in find_cookie
    raise SyntaxError(msg)
SyntaxError: invalid or missing encoding declaration for 'sys'

In this case "sys" is a binary that belongs to openafs (/usr/bin/sys) and of course it is no valid python.

"import sys" produces no error though, so typically python is able to distinguish between its sys module and this file.

A workaround is to run python with the "-I" parameter.

Expected output:

>>> import warnings
>>> warnings.warn("foo", Warning, stacklevel=2)
sys:1: Warning: foo

This bug was spotted in Gnome's pygobject bindings. Here is the corresponding bug report:
https://bugzilla.gnome.org/show_bug.cgi?id=757184

I'll have to dig into this to figure out what's going on, but since the sys module is built into Python it makes it so import won't accidentally import some file named sys. the real question is who thinks the 'sys' file should be considered.

The "sys" file is not imported. It is read by linecache.

warnings.warn() tries to determine the file name by looking at __file__ in module's globals. If it fails, it falls back to the module name. Definitely it fails also for builtin modules, binary extensions and zipimported modules.

Hmm.  I remember fixing problems with linecache and inspect.  I wonder if this is a variation on that?  I don't remember the issue number.

So I'm going to close this as "won't fix" because linecache is horribly over-engineered and I'm afraid trying to fix this will break code that actually worked previously. This is obviously an odd edge case that I think we can live with.

To add some info why we hit that bug:

https://bugs.python.org/issue24305 changed the warning stacklevel needed for import warnings from 8 to 2. As a result any code doing import warnings will likely hit that edge case when run with 3.5 and openafs installed. But I'm not sure if there is much code out there doing import warnings.. so ymmv.


For anyone with the same problem looking for a workaround:

I went through all Python versions and searched for the needed stacklevel.

def get_import_stacklevel(import_hook):
    """Returns the stacklevel value for warnings.warn() for when the warning
    gets emitted by an imported module, but the warning should point at the
    code doing the import.

    Pass import_hook=True if the warning gets generated by an import hook
    (warn() gets called in load_module(), see PEP302)
    """

    py_version = sys.version_info[:2]
    if py_version <= (3, 2):
        # 2.7 included
        return 4 if import_hook else 2
    elif py_version == (3, 3):
        return 8 if import_hook else 10
    elif py_version == (3, 4):
        return 10 if import_hook else 8
    else:
        # fixed again in 3.5+, see https://bugs.python.org/issue24305
        return 4 if import_hook else 2

It seems like there is already sufficient detection of invalid stack levels in warnings.warn, and one of the code paths does `module = "<string>"` and later another does `filename = module`, so `filename` can be intentionally junk data, which will be passed to `linecache`.

I expect this could be satisfactorily resolved by warn() setting filename = '<invalid stacklevel>', and `formatwarning` not invoking linecache when the filename is '<string>', '<invalid stacklevel>', etc., or at least ignoring the exception from linecache when the filename is <foo>.

Looking forward, why not let Python 3.6 warn() behave 'better' when the stacklevel is invalid.

e.g. it could raise ValueError (ouch, but 'correct'), or it could reset the stacklevel to 1 (a sensible fallback) and issue an auxillary SyntaxWarning to inform everyone that the stacklevel requested was incorrect.