By providing a __main__.py submodule, a package is saying "I am directly executable". When runpy says it isn't (because running that file happened to raise ImportError), then runpy is wrong.

runpy can't make the file work (it's genuinely broken), but it can avoid reporting a misleading error message - that's the bug to be fixed.

As far as the testing goes, if you look at the existing runpy tests, the usual approach is to create a temporary directory, write out a package with the desired behaviour, and then run it from there. So, for this test, it's a matter of copying the general structure of the existing package execution test in test_runpy, but replacing the body of the __main__.py file with something like "raise ImportError('This should not be replaced')"