classification
Title: Replace __import__ w/ importlib.__import__
Type: behavior Stage: committed/rejected
Components: Interpreter Core Versions: Python 3.3
process
Status: closed Resolution: fixed
Dependencies: 1559549 14605 Superseder:
Assigned To: brett.cannon Nosy List: Arfrever, Trundle, alex, benjamin.peterson, brett.cannon, eric.araujo, eric.smith, eric.snow, haypo, ncoghlan, pitrou, python-dev, roger.serwy
Priority: release blocker Keywords: patch

Created on 2008-03-18 02:40 by brett.cannon, last changed 2012-05-04 17:44 by brett.cannon. This issue is now closed.

Files
File name Uploaded Description Edit
bench_startup.py haypo, 2012-02-08 00:10
c011ff345a78.diff brett.cannon, 2012-02-25 03:03 review
131d1d107f26.diff brett.cannon, 2012-03-01 22:50 review
0fdf350657e3.diff brett.cannon, 2012-04-02 23:48 review
update_idle.diff brett.cannon, 2012-04-14 23:22 review
Repositories containing patches
http://hg.python.org/sandbox/bcannon#bootstrap_importlib
Messages (58)
msg63851 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2008-03-18 02:40
Python/import.c should be replaced by the implementation under
development contained in the py3k-importlib branch.
msg139233 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2011-06-27 00:46
OK, so first step is to simply replace __import__ w/ importlib.__import__ using builtins.__import__ in order to make sure that all tests pass as expected. Can probably do this by doing the switch in Py_Initialize() somewhere.

Next step after that will be seeing if _io can be used by importlib w/ the import of os postponed until after importlib is bootstrapped.
msg147625 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2011-11-14 18:01
Just some notes on the branch:

The FAILING file contains tests known to fail. There is a comment explaining the failure and possible fixes. If something makes it here and stays for a while it probably means upstream changes are needed (eg. test_pydoc needs ImportError to grow an attribute as to what import failed).

Python/pythonrun.c contains XXX markers of what is left to be done (although currently the zipimport comment is out-of-date and there needs to be one for exposing some APIs created in importlib.__init__). If you are looking for something to do, just search for XXX and try to tackle one of the comments.
msg149518 - (view) Author: Eric Snow (eric.snow) * (Python committer) Date: 2011-12-15 08:24
AFAICT, #1559549 is the ImportError attribute ticket.
msg152631 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2012-02-04 16:13
OK, I'm down to a single bug to be solved to call this work a "success" (there are other test failures, but they are not overtly difficult to solve).

At this point the bootstrapping is failing in the face of sub-interpreters. Specifically, when I try to load the frozen importlib code in Py_NewInterpreter() it leads to an assertion failure in the GC code when handling references in a GC generation. I have zero experience with sub-interpreters and my GC experience is rusty, so if anyone can have a look at the code I would appreciate it to see if they can figure out why loading a frozen module (while in marshal) is leading to a GC assertion error.
msg152785 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2012-02-06 21:53
Thanks to Benjamin, the test_capi assert failure is fixed. At this point the only failures are listed in the FAILING file and are (probably) minor.
msg152788 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2012-02-06 22:06
Is there a benchmark for import? How slow is importlib? :)
msg152814 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2012-02-07 18:28
On Mon, Feb 6, 2012 at 17:06, STINNER Victor <report@bugs.python.org> wrote:

>
> STINNER Victor <victor.stinner@haypocalc.com> added the comment:
>
> Is there a benchmark for import? How slow is importlib? :)
>

importlib.test.benchmark
msg152822 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2012-02-08 00:10
bench_startup.py: short script to compute the best startup time (I wrote the original script for the hash collision issue, #13703). Result on my PC:
 - original: 22.2 ms
 - importlib: 27.9 ms

So importlib adds an overhead of 25.7% in startup time.
msg153510 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2012-02-16 23:16
Just a quick update. I have refactored importlib in the cpython repo to allow for implementing bits of importlib.__import__() and importlib._gcd_import() in C. This means that the built-in __import__() is now calling importlib underneath the covers. Eventually what is in importlib.__import__() and _gcd_import() directly will be translated into the equivalent C code. This will speed up accessing sys.modules. After that whatever is deemed on the critical path and worth rewriting in C can be done function by function. That, though, should wait for Python-level profiling and optimization before wasting one's time.
msg153511 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2012-02-16 23:17
Add Nick since the refactoring of importlib.__import__() into functions was his idea.
msg153957 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2012-02-22 15:23
So refactoring the Python code into C code has been done, but it's crashing. =) As usual, manual memory management sucks.

I also think that their is still too much C code as it makes the whole thing somewhat brittle to any refactoring of importlib. I am seriously thinking of tossing the C code I have written and writing in C only the bare minimum needed to get to the sys.modules check, and otherwise punting to importlib for everything else in a single call or two. So that would mean performing the sanity check on the arguments, calculating the absolute name of the module, grabbing the import lock, checking sys.modules, failing that going to importlib, and then figuring out what to return regardless of where the module came from thanks to fromlist. So about four functions calls. Compare that to the 8 I'm making now along with the need to muck with other things and you can see why starting from scratch where I only care about the sys.modules fast path starts to look mighty attractive.
msg153999 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2012-02-22 22:10
Makes sense to me (your new proposal sounds pretty close to the sketch I posted on the mailing list).

It's really only the performance of in-function imports that concerned me and those are almost always going to get hits in sys.modules.
msg154022 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2012-02-23 00:35
> I also think that [there] is still too much C code as it makes the whole thing somewhat brittle
> to any refactoring of importlib. I am seriously thinking of tossing the C code I have written and
> writing in C only the bare minimum needed to get to the sys.modules check, and otherwise punting
> to importlib for everything else in a single call or two.

A few thoughts from my outsider viewpoint:

1) I think the point of importlib is to have maintainable code (i.e. the anti-import.c <wink>), so writing only the minimum code in C makes sense.

2) Speed is a concern, especially with sysconfig in 3.3 slowing startup, but a secondary concern.

3) Maybe some parts can be written in C later, after profiling, and then override Python functions (à la “from _heapq import *”, if that’s possible at all for importlib).

4) When is PyPy going to be the reference implementation again?  Then we’ll have no speed issues—ha ha only serious.

5) importlib rocks hard.
msg154070 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2012-02-23 16:47
On Wed, Feb 22, 2012 at 19:35, Éric Araujo <report@bugs.python.org> wrote:

I have already done that and pushed the initial change. The ultimate goal
will be fetching from sys.modules when level == 0 and not bool(fromlist)
only touch C code. All other work will touch varying levels of Python code.
Someone else can worry about speeding up fromlist so that only C code is
touched when level == 0.

I also have some failing tests, but that all has to do with Antoine's
caching work and thus fails in the default branch as well when the tests
are run under importlib itself (i.e. tests that generate a file and then
immediately try to import it). But what's nice is that test_reprlib now
fails only in my bootstrap branch and not default because it is just fast
enough to beat the directory mtime granularity. =)

Ha! I wish. I have to get within reasonable striking distance of current
performance or else I will get blocked for including the work.

It's possible as long as it is exposed through imp.

> 4) When is PyPy going to be the reference implementation again?  Then
> we’ll have no speed issues—ha ha only serious.
>
> 5) importlib rocks hard.

=) Thanks
msg154072 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2012-02-23 16:58
That last comment from me looks odd because somewhere along the way my replies to Éric's comment got stripped out.

First two paragraphs are in response to 1)
Third paragraph is for 2)
Fourth paragraph is for 3)

Rest makes sense since the inline reply survived.
msg154103 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2012-02-24 01:06
OK, so I have now streamlined the case so that only C is called when a module is in sys.modules, level == 0 and not fromlist.

Using the normal_startup benchmark, bootstrapped importlib is 1.12x slower. A fast run of startup_nosite puts it at 1.2x slower.

Using importlib.test.benchmark, the results are:


Comparing new vs. old

sys.modules : 331,097 vs. 383,180 (86.407694%)
Built-in module : 14,906 vs. 33,071 (45.072722%)
Source writing bytecode: small : 1,244 vs. 2,128 (58.458647%)
Source w/o bytecode: small : 2,420 vs. 4,784 (50.585284%)
Source w/ bytecode: small : 2,757 vs. 5,221 (52.805976%)
Source writing bytecode: tabnanny : 129 vs. 133 (96.992481%)
Source w/o bytecode: tabnanny : 144 vs. 147 (97.959184%)
Source w/ bytecode: tabnanny : 1,004 vs. 1,120 (89.642857%)
Source writing bytecode: decimal : 9 vs. 9 (100.000000%)
Source w/o bytecode: decimal : 9 vs. 9 (100.000000%)
Source w/ bytecode: decimal : 96 vs. 98 (97.959184%)


Where does that leave us? Well, obviously on medium-sized modules and up, the I/O involved along with parsing and execution outweighs importlib's costs to the point of being negligible. sys.modules is also now fast enough (I don't care what people say; being able to import from sys.modules at 0.0026 ms vs. 0.003 ms is not important in the grand scheme of things).

Built-in modules and such could be faster, but (a) there are only so many and they end up in sys.modules quickly anyway, and (b) even at their current speed they are still fast. So the real hold-up is small modules and whether they matter. The tabnanny benchmark is the median size of the stdlib, so half of modules will see no slowdown while half will see something of some level.

The most critical thing is to not use _gcd_import() in situations where the import function is currently passed in but instead use builtins.__import__() so as to get any C speed-ups (which I measured to make sure there are in all cases). After that I think unrolling the mentioned functions would help speed things up, but otherwise it becomes time to profile the Python code to see where the inefficiencies lie.

Some more C unrolling could be done. The situation of not fromlist could be written in C. Both _calc___packages__() and _resolve_name() could probably be written in C if someone cared.
msg154106 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2012-02-24 02:18
So I finished rewriting _return_module() so that if fromlist is not defined then it's all in C (and subsequently naming the function _handle_fromlist()).

That leaves rewriting in C _calc___package__() and _resolve_name(). After that I don't think there is anything left to do in C for __import__() itself (beyond maybe some reorganizing to avoid stupid things).

After that the only thing I can think of writing in C would be BuiltinImporter (and FrozenImporter if someone truly cared).

And once all that is said and done, that only leaves optimizing the Python code to try to appease the performance gods.
msg154143 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2012-02-24 18:29
OK, I have now done as much C code as I'm going to do for the __import__() function. It has gotten bootstrapped importlib within 10% of normal_startup against default.

That leaves (possibly) rewriting BuiltinImporter in C and then just good old fashioned optimizations of the Python code.

I'm going to continue to use the normal_startup benchmark as my gold standard benchmark and importlib.test.benchmark just for iterative testing that a change made an impact (until there is another Python 3 benchmark that measures startup time using a real-world app). With that in mind and taking a PyPy style approach, that means I should focus on top-level, non-package imports first (28 of them), followed by builtins (14), submodules (4), packages (2), and then extensions (2). And if you don't want to do the math, there are 50 imports when starting Python, of which only 9 seem to follow (and include) site.
msg154145 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2012-02-24 18:32
Are you looking for reviews at this point?
msg154149 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2012-02-24 19:24
For the record, compilation fails here:

Python/import.c: In function ‘PyImport_ImportModuleLevelObject’:
Python/import.c:2956:9: erreur: ‘for’ loop initial declarations are only
allowed in C99 mode
Python/import.c:2956:9: note: use option -std=c99 or -std=gnu99 to
compile your code
make: *** [Python/import.o] Erreur 1
msg154152 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2012-02-24 20:05
Fixed the 'for' loop variable declaration. Surprised clang didn't warn me about that.

As for reviews, I'm totally happy to get them, but I also don't know if I have hit the performance point well enough for people to allow me to merge the code into default. I guess the real question is whether *you're* happy with the 10% slowdown in raw startup, Antoine, since you were the most vocal critic from a performance standpoint?
msg154157 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2012-02-24 20:36
> As for reviews, I'm totally happy to get them, but I also don't know
> if I have hit the performance point well enough for people to allow me
> to merge the code into default. I guess the real question is whether
> *you're* happy with the 10% slowdown in raw startup, Antoine, since
> you were the most vocal critic from a performance standpoint?

Well, it seems current performance is acceptable now :)
msg154179 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2012-02-25 03:08
Then if you want to give it a review that would be great!

I still need to solve the test_pydoc failure (either with Brian's patch to add a name attribute to ImportError or implement importlib.find_module()). Otherwise all other failures at the moment are because of mtime race conditions (as you know =) or exist in the default branch. And I still need to integrate rebuilding importlib.h into the Makefile, but that should be easy since it will probably be a separate command so you don't accidentally break import by busting importlib.
msg154716 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2012-03-01 22:49
I have replied to Antoine's review and so generated a new patch.

At this point my bootstrap_importlib branch is 5% slower in a standard build in the normal_startup benchmark (11% if you use a debug build).

This is still w/o profiling the Python code to look for inefficiencies (of which I'm sure there are some considering how long I have been banging away at this code).
msg154717 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2012-03-01 23:14
> At this point my bootstrap_importlib branch is 5% slower in a standard
> build in the normal_startup benchmark (11% if you use a debug build).

I think that's fine.
msg154912 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2012-03-04 21:27
I have taken a look at the latest code again, and I'm worried that import.c still has large chunks of code dealing with module finding and loading. These functionalities should be redirected to the corresponding importlib code instead.
msg154915 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2012-03-05 00:14
Yes, the import related C API should be turned into a mere wrapper around the appropriate importlib code wherever possible.

I don't think that needs to be a gating criteria for the initial commit, though.
msg154917 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2012-03-05 00:23
> I don't think that needs to be a gating criteria for the initial commit, though.

I think it should, otherwise we will end up in a situation worse than
currently (we will have two competing default implementations of import,
depending which API you invoke).
msg155108 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2012-03-07 19:55
The finding/loading code in import.c is purely because of the imp module's public API (e.g. imp.find_module()). I have been waiting to find out if importlib would get bootstrapped before making the current module _imp and then creating an imp.py which handles most of the high-level stuff (e.g. imp.py containing find_module() and have it use importlib). That would let all of that C code go away since I already tie into the low-level C API for C-level import function calls.
msg155268 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2012-03-09 23:07
Two things I realized yesterday that need to be implemented (and some double-checking and/or opinion would be nice) while I wait for a full patch review.

One is ``python -v`` support. sys.flags has a verbose attribute that can be used to properly guard printing imported modules. It might be tricky, though, if sys.stderr is not set up properly during very early imports. I guess at worst I expose some sprintf() in imp just for this case (ugh).

Two is getting __import__() for situations where another import is triggered (e.g. fromlist stuff). I think the proper semantics is ``globals['__builtins__']['__import__'] if '__builtins__' in globals else  builtins.__import__``. Now where this gets tricky is that doing this means importlib.__import__(), when used directly from the importlib module, would sometimes use its implementation, and in other cases use builtins.__import__(). So either importlib.__import__() gets forked from builtins.__import__() so that it always uses importlib internally or simply don't worry about it and just have importlib.__import__() use builtins.__import__() when the need to trigger another import comes up. What do people think should happen?
msg155269 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2012-03-09 23:11
> One is ``python -v`` support. sys.flags has a verbose attribute that
> can be used to properly guard printing imported modules. It might be
> tricky, though, if sys.stderr is not set up properly during very early
> imports.

Might or might not. You should try, there's a fallback stderr at
interpreter startup.

> Two is getting __import__() for situations where another import is
> triggered (e.g. fromlist stuff). I think the proper semantics is
> ``globals['__builtins__']['__import__'] if '__builtins__' in globals
> else  builtins.__import__``. Now where this gets tricky is that doing
> this means importlib.__import__(), when used directly from the
> importlib module, would sometimes use its implementation, and in other
> cases use builtins.__import__(). So either importlib.__import__() gets
> forked from builtins.__import__() so that it always uses importlib
> internally or simply don't worry about it and just have
> importlib.__import__() use builtins.__import__() when the need to
> trigger another import comes up. What do people think should happen?

I don't think I have understood anything :) It probably doesn't help,
but I think the __import__ signature is generally crazy, though.
msg155481 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2012-03-12 21:15
So after looking at import.c and how it handles stuff, I have decided how I already implemented __import__() triggering __import__() itself is fine. So Antoine can just stay confused by my comment as it is now a moot point. =)
msg155485 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2012-03-12 21:38
Looks like Windows as a PC/import_nt.c:_PyWin_FindRegisteredModule() function used to find modules in the registry. Probably should care about that at some point (I guess).
msg155852 - (view) Author: Benjamin Peterson (benjamin.peterson) * (Python committer) Date: 2012-03-15 04:24
I'm guessing the most important part to review is the binding of importlib into being the main import implementation? (It's been in the tree a while, so I assume the kinks of the actual import implementation have been mostly ironed out.)
msg155895 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2012-03-15 15:32
Yes, that's the important bit. Thomas has given it a once over at this point so you should be able to see his review comments as well to prevent duplicate work.
msg155963 - (view) Author: Benjamin Peterson (benjamin.peterson) * (Python committer) Date: 2012-03-15 21:53
Okay; I added review.
msg156049 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2012-03-16 15:00
Thanks to everyone for the reviews. I'm moving house tomorrow and I suspect Andrea wants me to take a little Python break so I might not get to the reviews before a2, but I will definitely get to them an merged by a3.
msg156099 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2012-03-16 23:11
There are also reference leaks:

$ ./python -m test -R 3:2 test_bz2
[1/1] test_bz2
beginning 5 repetitions
12345
.....
test_bz2 leaked [-1, -1] references, sum=-2

$ ./python -m test -R 3:2 test_hashlib
[1/1] test_hashlib
beginning 5 repetitions
12345
.....
test_hashlib leaked [-1, -1] references, sum=-2
1 test failed:
    test_hashlib

$ ./python -m test -R 3:2 test_capi
[1/1] test_capi
beginning 5 repetitions
12345
.....
test_capi leaked [78, 78] references, sum=156
1 test failed:
    test_capi
msg157392 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2012-04-03 01:08
OK, everyone's code review comments have been addressed. That means I have test_pydoc still failing (and that won't get fixed until ImportError grows a module_name attribute; issue #1559549) and test_trace (it doesn't like frozen modules). I also have not plugged the memory leaks that Antoine pointed out (but then again I'm not sure where they might be considering how many people have gone over this code and not spotted a leak that I have not fixed). I also have not dealt with python -v or Windows registry imports. But once everything but the memory leaks are done I will generate a massive patch and commit to default.
msg157675 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2012-04-06 18:48
OK, -v/PYTHONVERBOSE is as done as it is going to be by me. Next up is (attempting) Windows registry stuff. After that I will push to default with test_trace and test_pydoc skipped so others can help me with those.
msg157683 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2012-04-06 20:05
> OK, -v/PYTHONVERBOSE is as done as it is going to be by me. Next up is
> (attempting) Windows registry stuff. After that I will push to default
> with test_trace and test_pydoc skipped so others can help me with
> those.

Skipped? How so? I think it would be better if you tried to debug them.
I don't think we have anyone active knowledgeable on either trace or
pydoc.
msg157742 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2012-04-07 17:43
On Fri, Apr 6, 2012 at 16:05, Antoine Pitrou <report@bugs.python.org> wrote:

>
> Antoine Pitrou <pitrou@free.fr> added the comment:
>
> > OK, -v/PYTHONVERBOSE is as done as it is going to be by me. Next up is
> > (attempting) Windows registry stuff. After that I will push to default
> > with test_trace and test_pydoc skipped so others can help me with
> > those.
>
> Skipped? How so?

By raising unittest.SkipTest.

I already know how to fix pydoc, but I need to get module names attached to
ImportError and I don't want to bother with that until importlib is in
(else it will be weird having it added into default but not in
Python/import.c). As for trace, I have not looked at it, but I know what
the failure is caused by and it's a question of how best to deal with it.
msg158001 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2012-04-11 01:11
OK, I have fixed test_trace by tweaking the trace module in default. I have decided to follow Antoine's suggestion-by-question and get test_pydoc working before I merge by getting ImportError spruced up in default but pydoc changed in bootstrap_importlib. I am *not* going to try to get registry-based imports working because I don't have a Windows machine and could quite easily break the build. I would rather have a Windows expert add the support.

IOW:

* Issue #1559549
* Fix pydoc in bootstrap_importlib
* Big patch merged into default
* ???
* Profit!
msg158267 - (view) Author: Roundup Robot (python-dev) Date: 2012-04-14 18:11
New changeset 2dd046be2c88 by Brett Cannon in branch 'default':
Issue #2377: Make importlib the implementation of __import__().
http://hg.python.org/cpython/rev/2dd046be2c88
msg158268 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2012-04-14 18:27
While the code has been committed, I'm leaving the issue open until I have checked that the language spec is up-to-date and I have written the "What's New" entry. I am holding off on both, though, unti any tweaks I make to the import process is in for Python 3.3.
msg158270 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2012-04-14 18:40
Notes on what to mention:

importlib.invalidate_caches()
doctests and ImportError now spitting out the full module name
ImportError's new attributes
msg158271 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2012-04-14 18:41
More notes:

5% startup loss according to normal_startup; within realm of compiler optimizations.
msg158297 - (view) Author: Roger Serwy (roger.serwy) * (Python committer) Date: 2012-04-14 23:08
Brett, your latest commit breaks IDLE. Here's the error message:

Failed to import extension:  FormatParagraph
Failed to load extension 'FormatParagraph'
Traceback (most recent call last):
  File "./idlelib/EditorWindow.py", line 998, in load_standard_extensions
    self.load_extension(name)
  File "./idlelib/EditorWindow.py", line 1008, in load_extension
    mod = __import__(name, globals(), locals(), [])
  File "<frozen importlib._bootstrap>", line 974, in _find_and_load
ImportError: No module named 'FormatParagraph'

Same error occurs for ZoomHeight, ScriptBinding, CallTips, ParenMatch, and AutoComplete.

I reverted to e730da0cd489, recompiled, and these errors went away. Just to be sure, I updated to tip, recompiled, and the errors reappeared.
msg158298 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2012-04-14 23:15
Another thing to note:

index does not default to -1 anymore but to 0; bug that should have gone away in Python 2.7.
msg158300 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2012-04-14 23:22
So IDLE broke because it was relying on buggy behaviour accidentally left in Python 2.7 and carried forward (plus it was not updated to use best practices like importlib.import_module()). Roger, can you try the patch I have uploaded and see if that fixes things?
msg158301 - (view) Author: Roger Serwy (roger.serwy) * (Python committer) Date: 2012-04-14 23:38
I tested update_idle.diff and it corrects the issue. 

While IDLE's use of __import__ may be "buggy", I don't see anything in the documentation about deprecation or other warnings about this usage. This is a backwards-incompatible change.
msg158302 - (view) Author: Roger Serwy (roger.serwy) * (Python committer) Date: 2012-04-15 00:01
I caused a segmentation fault with the following (on Linux):

$ mkdir crash
$ touch crash/mod.py
$ echo "__import__('mod', globals(), locals(), [], 1)" > crash/__init__.py
$ ./python3 -m crash
msg158304 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2012-04-15 00:53
I committed the fix. Thanks for testing, Roger.

As for the change in semantics, I'm fully aware it is not backwards-compatible. Unfortunately the incorrect usage was not even discovered until I started my bootstrap work because the import statement does the right thing but __import__() itself was never updated so it is only noticeable if you never updated your code to use importlib.import_module() which has been the preferred way to programmatically import code since Python 2.7/3.1. Plus the correct semantics are documented in PEP 328 (http://python.org/dev/peps/pep-0328/) and referenced in the language spec (http://docs.python.org/py3k/reference/simple_stmts.html#the-import-statement) so I'm not going to change it back since this brings the function more in line with expectations. And since the fix is as simple as a try/except and two import calls then it falls within the realm of having to fix code for any new Python release.

And as for the crash, I will have a look.
msg158305 - (view) Author: Roger Serwy (roger.serwy) * (Python committer) Date: 2012-04-15 01:25
Brett, I see your point. 

The docs for __import__ should be updated to include your two-import fix as well as reference PEP328. 
http://docs.python.org/dev/library/functions.html?highlight=__import__#__import__
msg158307 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2012-04-15 01:52
OK, crasher is fixed (as is importlib as it failed on the test case as well thanks to the slicing of [:-0] returning the empty string instead of the entire string).

And I will update the docs to be a bit more clear about things (at least those docs have the right default value for level).
msg158573 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2012-04-17 23:06
Note:

__import__('sys', level=1) no longer works; raises KeyError instead.
msg158967 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2012-04-22 15:12
To note:

_frozen_importlib is a CPython implementation detail
History
Date User Action Args
2012-05-04 17:44:03brett.cannonsetstatus: open -> closed
resolution: fixed
2012-04-22 19:38:23brett.cannonsetdependencies: + Make import machinery explicit
2012-04-22 15:12:13brett.cannonsetmessages: + msg158967
2012-04-20 06:46:59Arfreversetnosy: + Arfrever
2012-04-17 23:06:22brett.cannonsetmessages: + msg158573
2012-04-15 01:52:37brett.cannonsetmessages: + msg158307
2012-04-15 01:25:08roger.serwysetmessages: + msg158305
2012-04-15 00:53:24brett.cannonsetresolution: fixed -> (no value)
messages: + msg158304
2012-04-15 00:01:47roger.serwysetmessages: + msg158302
2012-04-14 23:38:57roger.serwysetmessages: + msg158301
2012-04-14 23:22:44brett.cannonsetfiles: + update_idle.diff

messages: + msg158300
2012-04-14 23:15:29brett.cannonsetmessages: + msg158298
2012-04-14 23:08:39roger.serwysetnosy: + roger.serwy
messages: + msg158297
2012-04-14 18:41:35brett.cannonsetmessages: + msg158271
2012-04-14 18:40:11brett.cannonsetmessages: + msg158270
2012-04-14 18:28:00brett.cannonsetpriority: normal -> release blocker
2012-04-14 18:27:51brett.cannonsetresolution: fixed
messages: + msg158268
stage: needs patch -> committed/rejected
2012-04-14 18:11:10python-devsetnosy: + python-dev
messages: + msg158267
2012-04-11 01:13:07brett.cannonsetdependencies: + ImportError needs attributes for module and file name
2012-04-11 01:11:57brett.cannonsetmessages: + msg158001
2012-04-07 17:43:08brett.cannonsetmessages: + msg157742
2012-04-06 20:05:24pitrousetmessages: + msg157683
2012-04-06 18:48:27brett.cannonsetmessages: + msg157675
2012-04-03 01:08:52brett.cannonsetmessages: + msg157392
2012-04-02 23:48:09brett.cannonsetfiles: + 0fdf350657e3.diff
2012-03-16 23:11:08pitrousetmessages: + msg156099
2012-03-16 15:00:37brett.cannonsetmessages: + msg156049
2012-03-15 21:53:08benjamin.petersonsetmessages: + msg155963
2012-03-15 15:32:10brett.cannonsetmessages: + msg155895
2012-03-15 04:24:29benjamin.petersonsetmessages: + msg155852
2012-03-12 21:38:38brett.cannonsetmessages: + msg155485
2012-03-12 21:15:41brett.cannonsetmessages: + msg155481
2012-03-12 00:16:20eric.smithsetnosy: + eric.smith
2012-03-09 23:11:26pitrousetmessages: + msg155269
2012-03-09 23:07:43brett.cannonsetmessages: + msg155268
2012-03-07 19:55:28brett.cannonsetmessages: + msg155108
2012-03-05 00:23:03pitrousetmessages: + msg154917
2012-03-05 00:14:09ncoghlansetmessages: + msg154915
2012-03-04 21:27:03pitrousetmessages: + msg154912
2012-03-01 23:14:55pitrousetmessages: + msg154717
2012-03-01 22:51:15brett.cannonsetfiles: - f0b459af26fb.diff
2012-03-01 22:50:51brett.cannonsetfiles: + 131d1d107f26.diff
2012-03-01 22:49:35brett.cannonsetmessages: + msg154716
2012-02-25 03:08:02brett.cannonsetmessages: + msg154179
2012-02-25 03:04:14brett.cannonsetfiles: + c011ff345a78.diff
2012-02-24 20:36:47pitrousetmessages: + msg154157
2012-02-24 20:10:45alexsetnosy: + alex
2012-02-24 20:05:50brett.cannonsetmessages: + msg154152
2012-02-24 19:24:41pitrousetmessages: + msg154149
2012-02-24 18:32:39pitrousetnosy: + pitrou
messages: + msg154145
2012-02-24 18:29:26brett.cannonsetmessages: + msg154143
2012-02-24 02:18:34brett.cannonsetmessages: + msg154106
2012-02-24 01:06:38brett.cannonsetmessages: + msg154103
2012-02-23 16:58:04brett.cannonsetmessages: + msg154072
2012-02-23 16:47:45brett.cannonsetmessages: + msg154070
2012-02-23 00:35:05eric.araujosetmessages: + msg154022
2012-02-22 22:10:40ncoghlansetmessages: + msg153999
2012-02-22 15:23:05brett.cannonsetmessages: + msg153957
2012-02-16 23:17:43brett.cannonsetnosy: + ncoghlan
messages: + msg153511
2012-02-16 23:16:18brett.cannonsetmessages: + msg153510
2012-02-08 00:10:51hayposetfiles: + bench_startup.py

messages: + msg152822
2012-02-07 18:28:37brett.cannonsetmessages: + msg152814
2012-02-07 00:51:42brett.cannonlinkissue13959 dependencies
2012-02-06 22:06:40hayposetnosy: + haypo
messages: + msg152788
2012-02-06 21:53:33brett.cannonsetmessages: + msg152785
2012-02-04 16:14:33brett.cannonsetfiles: + f0b459af26fb.diff
keywords: + patch
2012-02-04 16:13:31brett.cannonsetmessages: + msg152631
2011-12-15 08:24:59eric.snowsetmessages: + msg149518
2011-11-24 15:18:32eric.araujosetnosy: + eric.araujo
2011-11-14 18:01:53brett.cannonsetmessages: + msg147625
2011-07-09 20:55:47eric.snowsetnosy: + eric.snow
2011-06-27 00:47:02Trundlesetnosy: + Trundle
2011-06-27 00:46:18brett.cannonsetmessages: + msg139233
2011-06-27 00:36:10brett.cannonlinkissue667770 dependencies
2011-06-27 00:30:41brett.cannonsettitle: Replace import.c with a pure Python implementation -> Replace __import__ w/ importlib.__import__
2011-06-27 00:29:53brett.cannonsethgrepos: + hgrepo34
2010-08-03 23:40:27brett.cannonsetstage: needs patch
versions: + Python 3.3, - Python 3.2
2010-08-02 08:54:14eric.araujosetversions: + Python 3.2, - Python 3.1
2008-07-12 20:20:35benjamin.petersonsetnosy: + benjamin.peterson
2008-07-12 19:50:12brett.cannonsetpriority: high -> normal
2008-07-12 19:50:02brett.cannonsetpriority: critical -> high
2008-07-12 19:49:44brett.cannonsetversions: + Python 3.1, - Python 3.0
2008-03-18 02:40:12brett.cannoncreate