Title: Use importlib instead of pkgutil in runpy
Type: behavior Stage: resolved
Components: Versions: Python 3.3
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: ncoghlan Nosy List: Arfrever, brett.cannon, eric.araujo, georg.brandl, ncoghlan, pitrou, python-dev
Priority: release blocker Keywords:

Created on 2012-07-10 03:50 by ncoghlan, last changed 2012-07-18 11:15 by ncoghlan. This issue is now closed.

Messages (11)
msg165153 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2012-07-10 03:50
There's some lingering weirdness in runpy due to its reliance on pkgutil's import emulation (see #15272).

For 3.3, I'd like to explicitly switch runpy over to using importlib. This was supposed to happen automatically, but the relevant API is "importlib.find_loader" rather than the "imp.get_loader" that runpy is currently looking for.
msg165187 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2012-07-10 13:11
Ooh, first "customer" of importlib.find_loader()! Obviously if the API doesn't work for you let me know since it's a 3.3 addition and can be tweaked to meet your needs.
msg165504 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2012-07-15 04:03
OK, the way I plan to tackle this is to actually redirect the pkgutil APIs to importlib (rather than using the private import emulation).

The reason I'm going to tackle it like this is that there are some assumptions pkgutil deals with that importlib doesn't. Specifically, runpy expects find_loader() to take care of importing the parent package. In addition, importlib is prone to throwing AttributeError or TypeError for problems where pkgutil consistently throws ImportError. By using pkgutil as a compatibility shim, most code (including runpy itself) will be able to transition to importlib without actually needing any changes.

This approach also brings test_runpy and test_cmd_line_script more fully to bear on the importlib implementation, which is a nice bonus :)
msg165505 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2012-07-15 04:17
Oh, special - importlib.find_loader() currently expects every module to have a __loader__ attribute, but there's at least one that doesn't: __main__.

Guess I'll be tweaking it to handle that case more gracefully after all :)
msg165507 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2012-07-15 05:41
Ah, __main__, thou art such a special beast.

Here's my current plan:

1. __main__ will be initialised with __loader__ set to BuiltinImporter. This covers cases like the interactive prompt or execution of stdin where there is no source code available for __main__

2. When executing a script (either directly or via runpy), __main__.__loader__ will be *changed* at a later point to refer to the appropriate kind of loader object.
msg165513 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2012-07-15 08:10
New changeset 3987667bf98f by Nick Coghlan in branch 'default':
Take the first step in resolving the messy pkgutil vs importlib edge cases by basing pkgutil explicitly on importlib, deprecating its internal import emulation and setting __main__.__loader__ correctly so that runpy still works (Affects #15343, #15314, #15357)
msg165514 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2012-07-15 08:48
OK, I think runpy and __main__.__loader__ are all sorted now. The relevant runpy and command line script tests all check for the expected values, and I added a couple of new tests to cover the "-c" and "reading from stdin" cases.
msg165651 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2012-07-16 19:31
A test fails on the XP buildbots:

FAIL: test_get_importer_avoids_emulation (test.test_pkgutil.ImportlibMigrationTests)
Traceback (most recent call last):
  File "D:\Buildslave\3.x.moore-windows\build\lib\test\", line 289, in test_get_importer_avoids_emulation
AssertionError: unexpectedly None
msg165711 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2012-07-17 11:38
New changeset 16e50e85d684 by Nick Coghlan in branch 'default':
Issue #15314: Tweak a pkgutil test to hopefully be more Windows friendly
msg165713 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2012-07-17 11:43
OK, I just pushed a tweak to the test that will hopefully resolve the problem. If the tweak works, though, it does suggest we have a bit of weirdness in importlib: it means pkgutil.get_importer("") is returning an importer on POSIX systems, but None on Windows.

The test tweak to get OS X passing for #15230 on 3.2 (where it was already passing on trunk) suggests we may also have a case where 3.2 called os.path.realpath() before setting __file__, while 3.3 does not.
msg165756 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2012-07-18 11:15
Buildbots are green (at least as far as this issue goes), so closing again.

I created #15384 for the surprising behaviour of FileFinder on Windows.

#15385 records my investigation into the odd symlink behaviour on 3.2 (it was just a discrepancy between the emulation in pkgutil and the real import system)
Date User Action Args
2012-07-18 11:15:25ncoghlansetstatus: open -> closed

messages: + msg165756
2012-07-17 11:43:36ncoghlansetmessages: + msg165713
2012-07-17 11:38:58python-devsetmessages: + msg165711
2012-07-16 19:31:24pitrousetstatus: closed -> open
nosy: + pitrou
messages: + msg165651

2012-07-15 08:48:12ncoghlansetstatus: open -> closed
resolution: fixed
messages: + msg165514

stage: needs patch -> resolved
2012-07-15 08:10:06python-devsetnosy: + python-dev
messages: + msg165513
2012-07-15 05:41:24ncoghlansetpriority: high -> release blocker

messages: + msg165507
2012-07-15 04:17:33ncoghlansetmessages: + msg165505
2012-07-15 04:03:02ncoghlansetmessages: + msg165504
2012-07-14 12:43:10eric.araujosetnosy: + eric.araujo
2012-07-11 03:21:46Arfreversetnosy: + Arfrever
2012-07-10 13:11:14brett.cannonsetnosy: + brett.cannon
messages: + msg165187
2012-07-10 03:50:36ncoghlancreate