Yanking the implicit cache validation does seem to be the simplest solution here.  Perhaps we could come up with a complex way of making this work, but I don't think it's worth it.

What may be worth it is making the admonition of "use invalidate_caches()" easier to find.  I imagine the situation like this:

1. dynamically generate module
2. try to import it and get ImportError
3. scratch head and make sure it's actually valid code, reboot, etc.
4. still not working
5. search google or Python docs for an answer

and step 5 isn't working out well enough.  Removing the implicit cache invalidation would only make a more reliable step 5 even more important.

How to make that happen?  I'm not sure.  Perhaps a dedicated "Trouble-shooting Imports" FAQ with a links to it on relevant pages like importlib, imp, and the import statement.  Maybe we already have something like that and I've just missed it.