As a completely minimal strawman design that's the closest fit to what SafeMath is already doing: what if there was a "linecache.fallback_source_path" attribute that linecache searched with importlib whenever a loader returned None to indicate that the module existed, but not in a format that provided source code?

Then instead of adding the source directory to sys.path (which was only working because the legacy import system never implemented PEP 302 properly), SageMath could instead add the source directory to that new linecache.fallback_source_path list.

(Given a separate linecache.fallback_metapath, a suitably clever Finder implementation could then even provide the URL source lookup feature)

There'd be some API design issues to work out around ensuring that directories added to linecache.fallback_source_path are added in the same relative order as directories on sys.path, but the end result would likely be more reliable and resilient than relying on "additional directory installed later in sys.path" as the fallback strategy for handling sourceless modules.