There's some lingering weirdness in runpy due to its reliance on pkgutil's import emulation (see #15272).

For 3.3, I'd like to explicitly switch runpy over to using importlib. This was supposed to happen automatically, but the relevant API is "importlib.find_loader" rather than the "imp.get_loader" that runpy is currently looking for.

Ooh, first "customer" of importlib.find_loader()! Obviously if the API doesn't work for you let me know since it's a 3.3 addition and can be tweaked to meet your needs.

OK, the way I plan to tackle this is to actually redirect the pkgutil APIs to importlib (rather than using the private import emulation).

The reason I'm going to tackle it like this is that there are some assumptions pkgutil deals with that importlib doesn't. Specifically, runpy expects find_loader() to take care of importing the parent package. In addition, importlib is prone to throwing AttributeError or TypeError for problems where pkgutil consistently throws ImportError. By using pkgutil as a compatibility shim, most code (including runpy itself) will be able to transition to importlib without actually needing any changes.

This approach also brings test_runpy and test_cmd_line_script more fully to bear on the importlib implementation, which is a nice bonus :)

Oh, special - importlib.find_loader() currently expects every module to have a __loader__ attribute, but there's at least one that doesn't: __main__.

Guess I'll be tweaking it to handle that case more gracefully after all :)

Ah, __main__, thou art such a special beast.

Here's my current plan:

1. __main__ will be initialised with __loader__ set to BuiltinImporter. This covers cases like the interactive prompt or execution of stdin where there is no source code available for __main__

2. When executing a script (either directly or via runpy), __main__.__loader__ will be *changed* at a later point to refer to the appropriate kind of loader object.

New changeset 3987667bf98f by Nick Coghlan in branch 'default':
Take the first step in resolving the messy pkgutil vs importlib edge cases by basing pkgutil explicitly on importlib, deprecating its internal import emulation and setting __main__.__loader__ correctly so that runpy still works (Affects #15343, #15314, #15357)
http://hg.python.org/cpython/rev/3987667bf98f

OK, I think runpy and __main__.__loader__ are all sorted now. The relevant runpy and command line script tests all check for the expected values, and I added a couple of new tests to cover the "-c" and "reading from stdin" cases.

A test fails on the XP buildbots:



======================================================================
FAIL: test_get_importer_avoids_emulation (test.test_pkgutil.ImportlibMigrationTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "D:\Buildslave\3.x.moore-windows\build\lib\test\test_pkgutil.py", line 289, in test_get_importer_avoids_emulation
    self.assertIsNotNone(pkgutil.get_importer(sys.path[0]))
AssertionError: unexpectedly None

New changeset 16e50e85d684 by Nick Coghlan in branch 'default':
Issue #15314: Tweak a pkgutil test to hopefully be more Windows friendly
http://hg.python.org/cpython/rev/16e50e85d684

OK, I just pushed a tweak to the test that will hopefully resolve the problem. If the tweak works, though, it does suggest we have a bit of weirdness in importlib: it means pkgutil.get_importer("") is returning an importer on POSIX systems, but None on Windows.

The test tweak to get OS X passing for #15230 on 3.2 (where it was already passing on trunk) suggests we may also have a case where 3.2 called os.path.realpath() before setting __file__, while 3.3 does not.

Buildbots are green (at least as far as this issue goes), so closing again.

I created #15384 for the surprising behaviour of FileFinder on Windows.

#15385 records my investigation into the odd symlink behaviour on 3.2 (it was just a discrepancy between the emulation in pkgutil and the real import system)