Title: Re-implement parts of imp in pure Python
Type: Stage: needs patch
Components: Library (Lib) Versions: Python 3.3
Status: closed Resolution: fixed
Dependencies: 2377 14605 14657 15166 15167 Superseder:
Assigned To: brett.cannon Nosy List: Arfrever, benjamin.peterson, berker.peksag, brett.cannon, eric.araujo, eric.snow, georg.brandl, jpe, meador.inge, pitrou, python-dev, vstinner
Priority: release blocker Keywords: patch

Created on 2012-02-07 00:51 by brett.cannon, last changed 2017-12-06 16:25 by vstinner. This issue is now closed.

File name Uploaded Description Edit
imp.diff brett.cannon, 2012-03-14 20:52 review
imp.diff eric.snow, 2012-03-21 05:42 review
use_of_imp.txt eric.snow, 2012-03-21 05:43 stdlib modules that use imp (could be refactored to use importlib)
imp_find_module.diff eric.snow, 2012-04-17 05:57 sketch of find_module() implementation review
issue13959_reload.diff eric.snow, 2012-04-22 07:41 port of _imp.reload() to review
issue13959_magic_and_tag.diff eric.snow, 2012-04-22 10:40 one approach to moving the magic and tag code out of import.c review
explicit_meta_path.diff brett.cannon, 2012-04-23 04:22
explicit_path_hooks.diff brett.cannon, 2012-04-23 04:22
issue13959_magic.diff eric.snow, 2012-04-24 05:38 porting MAGIC to importlib review
issue13959_tag.diff eric.snow, 2012-04-24 05:40 porting TAG to importlib review jpe, 2012-06-05 18:49
issue13959_get_tag.diff eric.snow, 2012-06-20 01:47 new TAG implementation using sys.implementation review
Pull Requests
URL Status Linked Edit
PR 4736 merged vstinner, 2017-12-06 15:54
Messages (76)
msg152800 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2012-02-07 00:51
A bunch of code in Python/import.c exists purely for the imp module, but a large chunk of it does not need to be implemented in C but instead can be implemented in Python code.

Once importlib is bootstrapped in then an attempt to scale back the C code should be done by re-implementing parts of imp in pure Python (either in some _imp module or directly in _importlib itself).
msg155790 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2012-03-14 20:52
Attached is a start for re-implementing imp. Still need code for cache_from_source, source_from_cache, find_module, load_module, reload, load_compiled, load_source, and load_package.
msg156473 - (view) Author: Eric Snow (eric.snow) * (Python committer) Date: 2012-03-21 05:42
Here's an incomplete (though essentially test-passing) patch that flips more of imp into importlib.  It builds on Brett's patch.  The gist is that is a minimal wrapper around importlib, which uses _imp directly.

Ultimately the things implemented in Lib/importlib/ should be yanked out of Python/import.c (and Python/dynload_*.c).  I expect that this will wait for the bootstrap merge (#2377) to land.
msg156474 - (view) Author: Eric Snow (eric.snow) * (Python committer) Date: 2012-03-21 05:43
Here's a listing of the places where import-ish modules are used in CPython.  Once things all the import stuff currently on the table is wrapped up, I'll probably work on switching things over to using importlib directly.
msg156475 - (view) Author: Eric Snow (eric.snow) * (Python committer) Date: 2012-03-21 06:00
Of note for my patch is the addition of SUFFIXES and MODE to the 3 main "file loader" classes.  I did this to reverse the dependence on imp.get_suffixes().

As well, a couple of extra functions are added to Python/importlib/ for a similar reason, but should also be meaningful externally in their own right.

load_module_from_file(), is one of these, though the current incarnation relies on a mythical new method on loaders that may not pass muster.

(FWIW, I accidently included the use_of_imp.txt file in the patch and don't intend it to get included)
msg156527 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2012-03-22 01:04
> Once things all the import stuff currently on the table is wrapped up,
> I'll probably work on switching things over to using importlib directly.

Please leave distutils unchanged; we don’t clean it up or improve it in any way, it’s bugfixes only.  Changing packaging is fine; I’ll just keep using the imp module in the distutils2-python3 backport.
msg156530 - (view) Author: Eric Snow (eric.snow) * (Python committer) Date: 2012-03-22 01:08
Sounds good.  It will be a while before we get there, but at that point I was already planning on getting in touch with the maintainers of relevant modules before messing with them.
msg156964 - (view) Author: Eric Snow (eric.snow) * (Python committer) Date: 2012-03-28 03:44
(Éric, I'd reply in the review if reitveld recognized my current username -- see  FYI, I also did not get an email for your review, which is likely related to that issue.)

Regarding __all__, I didn't realize that about being a tuple.  I'll fix it. 

As to the "dm" in the SOABI value, that is definitely something we'll have to factor in.  There are actually a few such places in the current patch that rely on problematic data.   It's either hidden in C or not available during bootstrapping.  That's something we'll have to iron out before the solution for this issue gets committed.
msg158269 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2012-04-14 18:29
I think this should be a blocker for 3.3.
msg158362 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2012-04-15 19:57
Just because I was thinking about it, I wonder if necessarily all the frozen stuff really needs to stay in import.c. I mean a frozen module is really just an entry in an array of structs that has a name of an char*[]. I don't see why one couldn't simply have a get_frozen_bytes() method to convert that char*[] into a bytes object and use that to construct a module in pure Python code.
msg158364 - (view) Author: Roundup Robot (python-dev) Date: 2012-04-15 20:09
New changeset d777f854a66e by Brett Cannon in branch 'default':
Issue #13959: Rename imp to _imp and add Lib/ and begin
msg158365 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2012-04-15 20:12
OK, so I have started to check this stuff in, but I think it's best to do it piecemeal. Going forward I would like to commit in units of functions being replaced, and prioritize stuff that cuts out C code (e.g. the load_*() methods, find_module(), etc.). That way it's clear that progress is being made. Obviously the best way to tell if code is hanging on just because of imp is to comment out the interface code and see what your compiler complains about in terms of dead code (or at least clang is good at this).
msg158370 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2012-04-15 21:52
It looks like in order to get a clear sign of what it will take to remove various parts of import.c, imp.load_module() needs to go along with imp.load_package() (since they call each other in the C code). You also have to take care of imp.reload(), but I am simplifying the C code greatly and will take care of referencing other code in the module.
msg158371 - (view) Author: Roundup Robot (python-dev) Date: 2012-04-15 21:56
New changeset 4dce3afc392c by Brett Cannon in branch 'default':
Issue #13959: Simplify imp.reload() by relying on a module's
msg158377 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2012-04-15 22:34
I am seeing how this is going to go down. the load_dynamic, load_source, etc. family of functions are simply dispatched to by load_module(). So to keep some semblance of backwards-compatibility, each of those modules need to be implemented and then have load_module() simply dispatch to them based on the "type" of module it is.
msg158383 - (view) Author: Roundup Robot (python-dev) Date: 2012-04-16 00:25
New changeset 2df37938b8e1 by Brett Cannon in branch 'default':
Issue #13959: Re-implement imp.load_module() in
msg158398 - (view) Author: Roundup Robot (python-dev) Date: 2012-04-16 02:28
New changeset 4256df44023b by Brett Cannon in branch 'default':
Issue #13959: Re-implement imp.load_package() in
msg158474 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2012-04-16 15:48
From Eric Smith on python-dev:

> +    suffix, mode, type_ = details
> +    if mode and (not mode.startswith(('r', 'U'))) or '+' in mode:
> +        raise ValueError('invalid file open mode {!r}'.format(mode))

Should this be:
if mode and (not mode.startswith(('r', 'U')) or '+' in mode):

to match:

> -    if (*mode) {
> -        if (!(*mode == 'r' || *mode == 'U') || strchr(mode, '+')) {
> -            PyErr_Format(PyExc_ValueError,
> -                         "invalid file open mode %.200s", mode);
msg158522 - (view) Author: Roundup Robot (python-dev) Date: 2012-04-17 02:11
New changeset 3b5b4b4bb43c by Brett Cannon in branch 'default':
Issue #13959: Re-implement imp.load_source() in
msg158531 - (view) Author: Eric Snow (eric.snow) * (Python committer) Date: 2012-04-17 05:57
This is a mostly-working sketch of find_module() in  I've run out of time tonight, but wanted to get it up in case it's useful to anyone else (a.k.a Brett).  If nobody's touched it before then, I'll finish it up tomorrow.
msg158555 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2012-04-17 16:08
I doubt I will beat you to it, Eric, but I did want to say that your overall design was what I had in my head when I was thinking about how to re-implement the function, so keep at it!
msg158575 - (view) Author: Roundup Robot (python-dev) Date: 2012-04-17 23:14
New changeset 66bd85bcf916 by Brett Cannon in branch 'default':
Issue #13959: Re-implement imp.load_compiled() in
msg158593 - (view) Author: Eric Snow (eric.snow) * (Python committer) Date: 2012-04-18 05:51
2 "problems" with that last patch:

* the types of the loaders that get returned by _bootstrap._find_module() are not the classes in _bootstrap (e.g. _frozen_importlib._SourceFileLoader).  That doesn't smell right.
* tokenize.get_encoding? is saying that Lib/test/ has an encoding of utf-8, when test_find_module_encoding is expecting it to not.  So does PyTokenizer_FindEncodingFilename (which the old import code used) behaving differently than tokenize.get_encoding()?  FYI, tokenize.get_encoding() is what importlib uses...
msg158594 - (view) Author: Eric Snow (eric.snow) * (Python committer) Date: 2012-04-18 05:54
rather, tokenize.detect_encoding()
msg158622 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2012-04-18 14:25
You could change Lib/ to have ``import _frozen_importlib as _bootstrap`` if you want the same modules coming from imp, but I would argue against changing importlib itself being changed as that complicates development since if you screw up importlib._bootstrap when you compile it becomes a major pain to revert the importlib.h change, recompile, and continue to do that until you get it right. Plus you would only care about this if you are doing isinstance() checks on what kind of loader you have which you shouldn't care about since we have clearly defined ABCs to test against.

As for Lib/test/, it *does* have a source encoding of UTF-8 since it does not explicitly specify an encoding. Based on the name I'm assuming the file itself has bad UTF-8 characters, but that doesn't mean that the file is not supposed to be interpreted as UTF-8.
msg158628 - (view) Author: Eric Snow (eric.snow) * (Python committer) Date: 2012-04-18 14:49
On the _frozen_importlib point, I'm fine with that.  It was just counter-intuitive that the importlib._bootstrap functions were returning loaders whose classes weren't also defined there.

I'll have another look at that test tonight and run the patch through the full test suite, but otherwise I think find_module() is basically done.
msg158704 - (view) Author: Eric Snow (eric.snow) * (Python committer) Date: 2012-04-19 07:53
Looking it over, I'm confident that tokenizer.detect_encoding() does not raise a SyntaxError where PyTokenizer_FindEncodingFilename() does.  I've run out of time tonight, but I'll look at it more tomorrow.

Once find_module() is done, I'd like to move on to reload(), which I expect will be pretty straightforward at this point.  Then the feasibility of issue14618 should be clear.
msg158895 - (view) Author: Roundup Robot (python-dev) Date: 2012-04-20 22:04
New changeset c820aa9c0c00 by Brett Cannon in branch 'default':
Issue #13959: Keep imp.get_magic() in C code, but cache in importlib
msg158908 - (view) Author: Eric Snow (eric.snow) * (Python committer) Date: 2012-04-21 02:07
I'd still like to consider this a bit more.  When you're trying to understand imports, having one place to look (Lib/importlib/ is better than two, especially when the one is pure Python code.  So it still may be worth it to pull in the odds and ends that play into that.
msg158930 - (view) Author: Roundup Robot (python-dev) Date: 2012-04-21 22:53
New changeset b773a751c2e7 by Brett Cannon in branch 'default':
Issue #13959: Re-implement imp.cache_from_source() in Lib/

New changeset ea46ebba8a0f by Brett Cannon in branch 'default':
Issue #13959: Re-implement imp.source_from_cache() in Lib/
msg158931 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2012-04-21 23:02
Time for a recap!

I am personally not bothering with moving imp.get_tag() and imp.get_magic() to importlib._bootstrap as the amount of C code required to support the public C API is not worth it. If someone else once to do the work then by all means attach a patch.

I still need to port imp.find_module() (and the various constants) and will base it off of Eric's code.

NullImporter will get ported once issue #14605 (exposing the import machinery) lands.

get_suffixes() will also get ported, but that is a little bit more involved as I need to change how _DynLoadFiletab works by only storing the file extensions and not the other fluff (which is the same for all OSs).

After all of that is done then I will expose some API in importlib to replace find_module() and load_*() functions and then deprecate them (see issue #14551 for an ongoing discussion the possible API).
msg158939 - (view) Author: Eric Snow (eric.snow) * (Python committer) Date: 2012-04-22 00:58
Yeah, I'm hoping to keep pressing those odds and ends forward.  I have one lingering, oddball bug in find_module, but that patch is pretty much standing on its own.
msg158941 - (view) Author: Roundup Robot (python-dev) Date: 2012-04-22 01:15
New changeset 085cf1480cfe by Brett Cannon in branch 'default':
Issue #13959: Re-implement imp.find_module() in Lib/
msg158951 - (view) Author: Eric Snow (eric.snow) * (Python committer) Date: 2012-04-22 07:41
Ported _imp.reload() (Python/import.c) to Lib/  Included is the change to PyImport_ReloadModule() to make it simply a wrapper around the pure Python imp.reload().  There's a good chance I don't have this right or that I have some reference leak.  I haven't worked a ton on the C side of Python (sounds tropical).  This patch also removes 'modules_reloading' from the interpreter state, since it's no longer used anywhere (see issue14618).
msg158955 - (view) Author: Eric Snow (eric.snow) * (Python committer) Date: 2012-04-22 10:40
A patch for magic and tag.  It's not quite finished, but I wanted to see if the approach was palatable.  FYI, I'm also trying to push forward the sys.implementation stuff, which would help on the pyc tag.
msg158968 - (view) Author: Eric Snow (eric.snow) * (Python committer) Date: 2012-04-22 15:38
After thinking about it, is MAGIC an implementation detail?  It certainly reflects changes specific to the CPython interpreter.  I'm much more comfortable with leaving implementation details in Python/import.c.

On the other hand, there's already no small amount of rather CPython-specific stuff in Lib/importlib/, which belongs there.  The pyc magic bytes are tightly coupled with it.  Because of that, I realize, I'm still fine with the patch.

But it makes me wonder if it might be worth having a clear separation between the general and CPython-specific stuff in, for the sake of people who look at the code for the first (or tenth) time.  That's the same rationale I have for advocating moving as much over from import.c as relates to the importlib implementation.
msg158971 - (view) Author: Roundup Robot (python-dev) Date: 2012-04-22 17:02
New changeset 4e853913054c by Brett Cannon in branch 'default':
Issue #13959: Continue to try to accomodate altsep in importlib by not
msg158985 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2012-04-22 19:54
First off, you should separate the patches for get_magic() and get_tag(). Second, why is there _get_pyc_magic_int() when it is never called? Third, all of this would be greatly simplified if you just had a _RAW_MAGIC_NUMBER of 3220, did the bytes object creation for _MAGIC_NUMBER in-place (i.e. no separate function), and then in the C code just got _RAW_MAGIC_NUMBER and did the MAGIC macro work there.

As for what is CPython-specific and what isn't, only the other VMs can state that officially, so I'm not going to worry about that yet (but I will ask before Python 3.3 goes out so as to minimize backporting patches in the future). But importlib needs to stabilize more before that can happen.
msg158986 - (view) Author: Eric Snow (eric.snow) * (Python committer) Date: 2012-04-22 19:58
Good feedback.  The some of that code was the result of directly translating the C.  I'll get a new, simpler patch up probably tomorrow night.  Thanks.
msg158995 - (view) Author: Eric Snow (eric.snow) * (Python committer) Date: 2012-04-23 00:53
How consistent do the semantics of reload() need to remain?  (The C version does more type checking than the Python version probably needs to worry about.  reload() seems to be one of those bits that doesn't have much test coverage.)

Also, what's the best way to exercise the changes one makes to the C code?
msg158998 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2012-04-23 02:42
Loosening the type restrictions is fine.

As for testing, that's what the test suite is supposed to do. So if need be just write tests in Python that exercise the C code.
msg158999 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2012-04-23 04:22
To try and narrow down the issue, I now have separate patches for an explicit sys.meta_path and a sys.path_hooks. Both fail on tests, but for different reasons.

The explicit sys.meta_path is still failing on the __main__ issue on three of the tests and for threaded imports on the other.

The explicit sys.path_hooks patch fails on two tests: one on threading and  then test_cmd_line_script on apparently lacking an absolute file path thanks to issue #8202 (and why an explicit sys.path_hooks triggers that I don't know).

Both failures Nick might know about.
msg159118 - (view) Author: Eric Snow (eric.snow) * (Python committer) Date: 2012-04-24 05:38
updated patch for magic number support in imp/importlib
msg159119 - (view) Author: Eric Snow (eric.snow) * (Python committer) Date: 2012-04-24 05:40
updated patch for moving TAG to importlib/
msg159628 - (view) Author: Roundup Robot (python-dev) Date: 2012-04-29 16:50
New changeset eb5c5c23ca9b by Brett Cannon in branch 'default':
Issue #13959: Re-implement imp.NullImporter in Lib/
msg159630 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2012-04-29 17:20
Update time!

With NullImporter dealt with, that leaves get_magic(), get_tag(), reload(), and get_suffixes() as things to potentially move to Lib/ I would also like to re-implement PyImport_ExecCodeModuleObject() as it's keeping a lot of C code alive just for its personal use.
msg159631 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2012-04-29 17:23
1659 lines less than 3.2’s import.c so far!
msg159641 - (view) Author: Roundup Robot (python-dev) Date: 2012-04-29 18:40
New changeset eb68502731dd by Brett Cannon in branch 'default':
Issues #13959, 14647: Re-implement imp.reload() in Lib/
msg159644 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2012-04-29 19:38
So here is the deal with PyImport_ExecCodeModuleObject(): bootstrapping and Barry has made this a little annoying. =)

First off, PyImport_ImportFrozenModuleObject() uses PyImport_ExecCodeModuleObject(), so that precludes just directly importing imp to handle this function entirely. OK, so that means just trying to chop out the path manipulation stuff since that is duplicating code found in imp/importlib.

The problem, though, is that PyImport_ExecCodeModuleWithPathnames() will take its pathname argument and try to get a source path from it if it points to some .pyc file (PEP 3147 or sourceless .pyc, and if that new path exists then it is used instead of the given path. Unfortunately that API was introduced in Python 3.2, so there is a backwards-compatibility issue in that one can't just rip out the code w/o supporting it. But those semantics are the reason the equivalent of imp.source_from_cache() continues to exist in Python/import.c.

I see two options here. One is to simply leave the C code in, but that has the drawback of duplicated Python and C code. Two is to stick in a call to imp.source_from_cache() between PyImport_ExecCodeModuleWithPathnames() and PyImport_ExecCodeModuleObject() so the former retains the semantics and the latter doesn't pick up the bad habit before 3.3 is released.
msg159757 - (view) Author: Benjamin Peterson (benjamin.peterson) * (Python committer) Date: 2012-05-01 14:10
Windows is currently failing test_imp:
msg159758 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2012-05-01 14:32
That test is going to stay intermittent until issue #14657 gets resolved else the exact reason for the failure is going to be hard to debug remotely.
msg159960 - (view) Author: Roundup Robot (python-dev) Date: 2012-05-04 19:20
New changeset 257cbd2fac38 by Brett Cannon in branch 'default':
Issue #13959: Re-implement imp.get_suffixes() in Lib/
msg159962 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2012-05-04 19:23
OK, I'm waiting on issue #14657 (avoiding the _frozen_importlib/importlib dichotomy) is resolved before I document the new imp.extension_suffixes() as it would be good to know where I should pull in the source and bytecode suffixes.

I think this only leaves get_magic() and get_tag() as the only things to move + trying to figure out how to handle PyImport_ExecCodeModuleObject() and its grip on various bits of C code.
msg159969 - (view) Author: Roundup Robot (python-dev) Date: 2012-05-04 20:13
New changeset 22b0689346f9 by Brett Cannon in branch 'default':
Issue #13959: Move module type constants to Lib/
msg160424 - (view) Author: Roundup Robot (python-dev) Date: 2012-05-11 16:59
New changeset b81ddaf0db47 by Brett Cannon in branch 'default':
Issue #13959: Deprecate imp.get_suffixes() for new attributes on
msg160433 - (view) Author: Eric Snow (eric.snow) * (Python committer) Date: 2012-05-11 17:41
Question on this one:

@@ -126,7 +131,7 @@ def load_compiled(name, pathname, file=N
 # XXX deprecate
 def load_package(name, path):
     if os.path.isdir(path):
-        extensions = _bootstrap._SOURCE_SUFFIXES + [_bootstrap._BYTECODE_SUFFIX]
+        extensions = machinery.SOURCE_SUFFIXES[:] + [machinery.BYTECODE_SUFFIXES]
         for extension in extensions:
             path = os.path.join(path, '__init__'+extension)
             if os.path.exists(path):

Should that be the following?

  extensions = machinery.SOURCE_SUFFIXES[:] + machinery.BYTECODE_SUFFIXES[:]

Also, why the "[:]"?

Finally, in a couple spots you use the first element of the list (like the old case of "machinery.SOURCE_SUFFIXES[0]" in source_from_cache() and the new one in find_module()).  Will this be a problem where the source file has one of the other suffixes?  I'm not sure it's a big enough deal for the moment to worry about, but thought I'd ask.  :)
msg160434 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2012-05-11 17:47

And the [:] copies the list so I don't accidentally mutate in-place (did that once already, so now I'm just being paranoid; I doubt I need it hardly anywhere I don't do +=).

As for using SOURCE_SUFFIXES[0], I'm done following the ported code. I really don't care if .pyw isn't supported in that case. But in general people shouldn't assume just .py (or .py at all).
msg160439 - (view) Author: Roundup Robot (python-dev) Date: 2012-05-11 18:48
New changeset 626d5c6fbd95 by Brett Cannon in branch 'default':
Issue #13959: Have
msg160493 - (view) Author: Roundup Robot (python-dev) Date: 2012-05-12 21:43
New changeset 7bf8ac742d2f by Brett Cannon in branch 'default':
Issue #13959: Introduce importlib.find_loader().
msg160495 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2012-05-12 21:49
I have importlib.find_loader() coded up, but getting rid of find_module() is going to be a pain thanks to pkgutil, multiprocessing.forking, modulefinder, and idlelib.EditorWindow.
msg160524 - (view) Author: Roundup Robot (python-dev) Date: 2012-05-13 17:04
New changeset 59870239813c by Brett Cannon in branch 'default':
Issue #13959: Document imp.find_module/load_module as deprecated.
msg162276 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2012-06-04 17:13
Need to update the docstrings for imp.find_module() and load_module() to mention the functions are deprecated since there is no specific raised deprecation, only documented deprecation.
msg162356 - (view) Author: John Ehresman (jpe) * Date: 2012-06-05 16:30
This may be a known problem, but imp.load_module fails when trying to load a C extension.  This is with the 3.3a4 release on Windows, though I suspect the problem is cross-platform.
msg162357 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2012-06-05 17:08
Does it work in Python 3.2, John? If it does then it's just an oversight thanks to the lack of tests in test_imp and it shouldn't be too difficult to support.

But do realize I have deprecated the function. =)
msg162364 - (view) Author: John Ehresman (jpe) * Date: 2012-06-05 18:49
On 6/5/12 1:08 PM, Brett Cannon wrote:
> Brett Cannon<>  added the comment:
> Does it work in Python 3.2, John? If it does then it's just an oversight thanks to the lack of tests in test_imp and it shouldn't be too difficult to support.
> But do realize I have deprecated the function. =)

Attached is a short test file to demonstrate this.  It assumes the 
standard win32 layout, but shouldn't be hard to modify.  It does work in 
Python 3.2, but the else: clause in 3.3's imp.load_module function 
raises an ImportError.  I think the fix is to add an elif C_EXTENSION: 
clause that loads the .so / .pyd.

I've already rewritten my code to use importlib when running in Python 3.3.
msg162384 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2012-06-06 00:15
You're right about the solution, John. I (or someone else) just needs to code it up.
msg162940 - (view) Author: Roundup Robot (python-dev) Date: 2012-06-15 23:39
New changeset 034c814eb187 by Brett Cannon in branch 'default':
Issue #13959: Add to imp.find_module() and load_module's docstrings
msg163212 - (view) Author: Georg Brandl (georg.brandl) * (Python committer) Date: 2012-06-19 20:15
Which parts are still missing here?
msg163232 - (view) Author: Eric Snow (eric.snow) * (Python committer) Date: 2012-06-19 23:39
Here are the things I'm aware of:

* implement imp.get_tag() using sys.implementation
* *maybe* implement imp.get_magic() in pure Python (patch already attached here)
* (low priority) find a way to rip out the bulk of PyImport_ExecCodeModuleObject() from imp.c, if practical

From my discussions with Brett, there was a laundry list he was aiming to get in for 3.3 (for import stuff in general).  I know much of it got done, but not all of it.  Of the remainder, I'm not sure how much of that he still wants to see get in.  Some of it pertains to this issue, though not all.
msg163569 - (view) Author: Georg Brandl (georg.brandl) * (Python committer) Date: 2012-06-23 10:13
OK, sounds like none of it would block beta1.
msg163798 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2012-06-24 15:30
Sorry, been on vacation the past week.

Georg is right, nothing left is a b1 blocker, just stuff I want to get in before 3.3 ships.
msg164102 - (view) Author: Georg Brandl (georg.brandl) * (Python committer) Date: 2012-06-26 20:51
Moving back to blocker for beta2.
msg165116 - (view) Author: Roundup Robot (python-dev) Date: 2012-07-09 20:09
New changeset efb5e6ab10f4 by Brett Cannon in branch 'default':
Issue #15167 (as part of #13959): imp.get_magic() is no implemented in
msg165120 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2012-07-09 20:10
Since the final issue is just a nicety, I am considering this issue closed.
msg165148 - (view) Author: Eric Snow (eric.snow) * (Python committer) Date: 2012-07-10 02:48
msg307747 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-12-06 16:25
New changeset 672b6baa71010f236ee8c8ce912e98cb542385c6 by Victor Stinner in branch 'master':
bpo-32030: pass interp to _PyImport_Init() (#4736)
Date User Action Args
2017-12-06 16:25:57vstinnersetnosy: + vstinner
messages: + msg307747
2017-12-06 15:54:01vstinnersetpull_requests: + pull_request4640
2012-07-10 02:48:03eric.snowsetmessages: + msg165148
2012-07-09 20:23:48brett.cannonsetstatus: open -> closed
2012-07-09 20:10:51brett.cannonsetresolution: fixed
dependencies: - Deprecate imp.find_module()/load_module()
messages: + msg165120
2012-07-09 20:09:35python-devsetmessages: + msg165116
2012-06-26 20:51:41georg.brandlsetpriority: deferred blocker -> release blocker

messages: + msg164102
2012-06-24 19:51:41brett.cannonsetdependencies: + Re-implement imp.get_magic() in pure Python
2012-06-24 19:50:23brett.cannonsetdependencies: + Implement imp.get_tag() using sys.implementation
2012-06-24 15:30:27brett.cannonsetmessages: + msg163798
2012-06-23 10:13:43georg.brandlsetpriority: release blocker -> deferred blocker

messages: + msg163569
2012-06-20 01:47:15eric.snowsetfiles: + issue13959_get_tag.diff
2012-06-19 23:39:22eric.snowsetmessages: + msg163232
2012-06-19 20:15:46georg.brandlsetnosy: + georg.brandl
messages: + msg163212
2012-06-15 23:39:13python-devsetmessages: + msg162940
2012-06-06 00:15:10brett.cannonsetmessages: + msg162384
2012-06-05 18:49:10jpesetfiles: +

messages: + msg162364
2012-06-05 17:08:55brett.cannonsetmessages: + msg162357
2012-06-05 16:30:41jpesetnosy: + jpe
messages: + msg162356
2012-06-04 17:13:15brett.cannonsetmessages: + msg162276
2012-05-13 17:04:28python-devsetmessages: + msg160524
2012-05-13 16:32:23brett.cannonsetdependencies: + Deprecate imp.find_module()/load_module()
2012-05-12 21:49:57brett.cannonsetmessages: + msg160495
2012-05-12 21:43:33python-devsetmessages: + msg160493
2012-05-11 18:48:49python-devsetmessages: + msg160439
2012-05-11 17:47:49brett.cannonsetmessages: + msg160434
2012-05-11 17:41:23eric.snowsetmessages: + msg160433
2012-05-11 16:59:01python-devsetmessages: + msg160424
2012-05-04 20:13:36python-devsetmessages: + msg159969
2012-05-04 19:23:39brett.cannonsetdependencies: + Avoid two importlib copies
messages: + msg159962
2012-05-04 19:20:48python-devsetmessages: + msg159960
2012-05-01 14:32:39brett.cannonsetmessages: + msg159758
2012-05-01 14:10:23benjamin.petersonsetnosy: + benjamin.peterson
messages: + msg159757
2012-04-29 19:38:21brett.cannonsetmessages: + msg159644
2012-04-29 18:40:51python-devsetmessages: + msg159641
2012-04-29 17:23:15eric.araujosetmessages: + msg159631
2012-04-29 17:20:21brett.cannonsetmessages: + msg159630
2012-04-29 16:50:38python-devsetmessages: + msg159628
2012-04-24 05:40:10eric.snowsetfiles: + issue13959_tag.diff

messages: + msg159119
2012-04-24 05:38:29eric.snowsetfiles: + issue13959_magic.diff

messages: + msg159118
2012-04-23 04:23:02brett.cannonsetfiles: + explicit_path_hooks.diff
2012-04-23 04:22:32brett.cannonsetfiles: + explicit_meta_path.diff

messages: + msg158999
2012-04-23 02:42:28brett.cannonsetmessages: + msg158998
2012-04-23 00:53:25eric.snowsetmessages: + msg158995
2012-04-22 19:58:44eric.snowsetmessages: + msg158986
2012-04-22 19:54:14brett.cannonsetmessages: + msg158985
2012-04-22 17:02:38python-devsetmessages: + msg158971
2012-04-22 15:38:47eric.snowsetmessages: + msg158968
2012-04-22 10:40:23eric.snowsetfiles: + issue13959_magic_and_tag.diff

messages: + msg158955
2012-04-22 07:41:32eric.snowsetfiles: + issue13959_reload.diff

messages: + msg158951
2012-04-22 01:15:34python-devsetmessages: + msg158941
2012-04-22 00:58:15eric.snowsetmessages: + msg158939
2012-04-21 23:02:38brett.cannonsetassignee: brett.cannon
dependencies: + Make import machinery explicit
messages: + msg158931
stage: needs patch
2012-04-21 22:53:27python-devsetmessages: + msg158930
2012-04-21 02:07:05eric.snowsetmessages: + msg158908
2012-04-20 22:04:19python-devsetmessages: + msg158895
2012-04-19 07:53:22eric.snowsetmessages: + msg158704
2012-04-18 14:49:29eric.snowsetmessages: + msg158628
2012-04-18 14:25:40brett.cannonsetmessages: + msg158622
2012-04-18 05:54:42eric.snowsetmessages: + msg158594
2012-04-18 05:51:58eric.snowsetmessages: + msg158593
2012-04-17 23:14:33python-devsetmessages: + msg158575
2012-04-17 16:08:19brett.cannonsetmessages: + msg158555
2012-04-17 05:57:45eric.snowsetfiles: + imp_find_module.diff

messages: + msg158531
2012-04-17 02:11:32python-devsetmessages: + msg158522
2012-04-16 15:48:30brett.cannonsetmessages: + msg158474
2012-04-16 10:47:07berker.peksagsetnosy: + berker.peksag
2012-04-16 02:29:00python-devsetmessages: + msg158398
2012-04-16 00:25:29python-devsetmessages: + msg158383
2012-04-15 22:34:58brett.cannonsetmessages: + msg158377
2012-04-15 21:56:19python-devsetmessages: + msg158371
2012-04-15 21:52:18brett.cannonsetmessages: + msg158370
2012-04-15 20:12:37brett.cannonsetmessages: + msg158365
2012-04-15 20:09:11python-devsetnosy: + python-dev
messages: + msg158364
2012-04-15 19:57:26brett.cannonsetmessages: + msg158362
2012-04-14 18:29:43pitrousetpriority: normal -> release blocker
nosy: + pitrou
messages: + msg158269

2012-03-28 03:44:02eric.snowsetmessages: + msg156964
2012-03-22 01:08:53eric.snowsetmessages: + msg156530
2012-03-22 01:04:03eric.araujosetmessages: + msg156527
2012-03-21 06:00:29eric.snowsetmessages: + msg156475
2012-03-21 05:43:31eric.snowsetfiles: + use_of_imp.txt

messages: + msg156474
2012-03-21 05:42:40eric.snowsetfiles: + imp.diff

messages: + msg156473
2012-03-14 20:52:29brett.cannonsetfiles: + imp.diff
keywords: + patch
messages: + msg155790
2012-03-13 01:43:43brett.cannonsetnosy: + eric.snow
2012-02-16 05:01:41meador.ingesetnosy: + meador.inge
2012-02-11 04:24:44Arfreversetnosy: + Arfrever
2012-02-11 03:46:47eric.araujosetnosy: + eric.araujo
2012-02-07 00:51:42brett.cannonsetdependencies: + Replace __import__ w/ importlib.__import__
2012-02-07 00:51:29brett.cannoncreate