>> IMO, debating whether a None return means there's absolutely no chance of
>> there being source is silly - the best the loader can say is that it can't
>> provide source.
> 
> I don't think it is silly: if "None" means that the loader cannot provide the
> source, then it makes sense to continue looking for the source. If "None"
> means that there is certainly no source, then it does not make sense to
> continue looking.

It's silly to think that the loader can *ever* say that there's "certainly" no
source, so debating whether None means that is silly. The only thing None could
*possibly* mean is that the loader can't give you the source. No more, no less.

>> On the other hand, what is Cython source code even doing on sys.path?
> 
> It's specifically installed for tracebacks (and other debugging). In a way,
> it's no different from installing both .py and .pyc files. Nobody questions
> that, even though the .pyc files are sufficient.

The key difference is that Python source only needs Python to compile it. We
have Python source on sys.path and Python automatically compiles it to bytecode
as needed. You can't put Cython source on sys.path and have the Python
interpreter compile it for you. Basically, sys.path is for code that the Python
interpreter can (compile and) run, and it's at least arguable that non-Python
source doesn't belong on there.

I'm not trying to *make* that argument, though. All *I* am trying to say is
that it's not exactly surprising if stdlib code doesn't expect to find
non-Python source on sys.path.

>> I sort of understand the benefit, but it does seem to be a practice peculiar
>> to one scenario
> 
> Sure, it's particular but it's a real existing scenario.

... that hasn't worked since Python 3.3, but no-one has noticed or complained
until now.

>> and I'm not sure the standard library should deal with it.
> 
> Depends what you mean with "deal with it". Basically, all that I'm asking for
> is at least one officially supported way to enable finding sources for
> extension modules. There are many possible ways to fix this, that's why I
> made this post to python-ideas.

But the standard library has no need to ever find source for extension modules
(unless I'm unaware of some detail of the Cython/IPython specific situation).
So there's no need for the stdlib to be involved - Cython could define a
protocol for finding source code independently, and IPython (and anything else
that wants to display the source of a Cython extension) could use it.

>> I'm struggling to see how a Python programmer would benefit from access to
>> the source code of a compiled extension anyway.
> 
> Cython code looks very much like Python. Very likely, a Python programmer
> would understand the Cython code. Anyway, this is besides the point: even if
> it just benefits the author of the Cython code, it's already useful.

Point taken.

>> it's not exactly an urgent issue.
> 
> I never proposed to make it a release blocker :-) It's not urgent but it
> should still be fixed.

Note that I haven't said it shouldn't be fixed, merely that I'm not as
convinced, having read this discussion, that having linecache do a path search
if the loader returns None is *necessarily* the best solution here. I do still
feel that if we don't make that change, then the linecache docs should be
clarified, but I wouldn't say the linecache docs on the whole module_globals
argument are very clear in the first place - I certainly wouldn't know how to
use that argument based on the current docs.

I'm going to leave the call on whether your proposed change to linecache is OK
to someone who understands (or has the time to review and confirm) the impact
on the 4 places in the stdlib where it's called with a module_global parameter
- asyncio.base_tasks, bdb (twice) and pdb. What I will say is that the code in
the PR looks fine to me as an implementation of the proposed change.

I'm strongly against introducing any extra complexity into the loader
get_source function signature. I think it's fine as it is, and trying to add
nuances to the meaning behind None will only make life harder for people
implementing loaders.

Ideally, of course, there would be a CythonExtensionLoader that handled this in
get_source. But it's hard to see how such a thing would work as "Cython
extensions" don't look any different from extensions built with C, or Rust, or
anything else.

>> Cython still needs to work around the issue for users of older Pythons.
> 
> No. Older versions already work because there is no ExtensionFileLoader. The
> changes in Python 3.x(?) actually broke this. Certainly in Python 2.7, those
> tracebacks work fine.

By "older Pythons" I meant 3.3+. ExtensionFileLoader was introduced in 3.3, so
presumably this issue exists in all versions of Python from 3.3 onwards. I
don't consider 2.7 relevant here, as the import machinery was completely
different then.

Of course, it's up to Cython whether they choose to care about handling this
for Python 3.x users earlier than 3.8 (or earlier, if it's agreed that this
should be backported). And I should note that I'd be less worried about the
potential impact on existing code if this were being proposed solely for 3.8+.
But the way you're talking here, I'm assuming that you're hoping for a
backport. Please correct me if that assumption is wrong.