On Feb 8, 2018 12:55, "Jeroen Demeyer" <report@bugs.python.org> wrote:

New submission from Jeroen Demeyer <jdemeyer@cage.ugent.be>:

Displaying the source code in tracebacks for Cython-compiled extension
modules in IPython no longer works due to PEP 302. Various fixes are
possible, the two most obvious ones are:

To be clear this is nothing to do with PEP 302, but has more to do with
details of the import system reimplementation of Python 3.3, which PEP 302
predates by quite a lot.

 linecache should continue searching for the source file even if
loader.get_source() returns None.

I don't necessarily agree here. For some modules there may not be a real
file associated with it in the first place, much less on sys.path. I'm not
exactly sure why that fallback is there in the first place, but if a module
does have a __loader__ that should have the say of where the module's
source code is found (if at all).

2. the method ExtensionFileLoader.get_source() should be removed (instead
of being implemented and returning None).

Why? What would that help with? PEP 302 says get_source() can return None
of no sources are found. That ExtensionFileLoader does this is not wrong
(though it might be nice if it had a way to show C sources). It certainly
doesn't know anything about Cython.

If anything, as you and I discussed, Cython should be providing its own
loader for Cython modules (and perhaps have a way to better distinguish
Cython modules from other extension modules).

Now why was this broken and how do the above fix that?

When IPython needs to display a traceback, it uses linecache.getlines() to
get the source code to display. For Cython extensions, the filename is a
correct *relative* filename (it must be a relative filename since Cython
does not know where the sources will be after installation).

Since the filename is relative, linecache does not immediately find it, so
it looks for a PEP 302 loader. For extension modules (Cython or not), it
finds an instance of ExtensionFileLoader. If the loader has a get_source()
method, then it uses that to get the sources. Since
ExtensionFileLoader.get_source()
returns None, linecache stops looking further for sources.

Instead, what should happen is that linecache continues looking for the
sources in sys.path where it has a chance of finding them (if they are
installed somewhere in sys.path).

The problem with this analysis is that the fact that this used to work at
all was relying on undocumented behavior. Also in the case of Sage, which
was using this to find Cython sources, it's because we were putting Cython
sources on a sys.path entry which is not a normal thing to do.

I don't think there's a bug in Python here, and that this is a problem that
needs to be solved on the Cython end.