classification
Title: imp.load_module doesn't support C_EXTENSION type
Type: behavior Stage: commit review
Components: Library (Lib) Versions: Python 3.3
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: georg.brandl Nosy List: brett.cannon, georg.brandl, metolone, ncoghlan, paul.moore, python-dev, tim.golden
Priority: release blocker Keywords: 3.3regression, patch

Created on 2012-08-31 07:32 by metolone, last changed 2012-09-10 12:05 by paul.moore. This issue is now closed.

Files
File name Uploaded Description Edit
issue15828_handle_extension_modules.diff ncoghlan, 2012-08-31 12:59 Handle C extension modules in imp.load_module (and fix typo in error message) review
Messages (17)
msg169506 - (view) Author: Mark Tolonen (metolone) Date: 2012-08-31 07:32
I built a C extension using SWIG for both Python 3.2 and Python 3.3.  The Python file supplying class proxies uses imp.load_module and fails with the following traceback on 3.3.0rc1, but works on 3.2.3:

   Traceback (most recent call last):
     File "<stdin>", line 1, in <module>
     File ".\Widget.py", line 26, in <module>
       _Widget = swig_import_helper()
     File ".\Widget.py", line 22, in swig_import_helper
       _mod = imp.load_module('_Widget', fp, pathname, description)
     File "D:\dev\python33\lib\imp.py", line 171, in load_module
       raise ImportError(msg, name=name)
   ImportError: Don't know how to import _Widget (type code 3

It looks like imp.py is new in Python 3.3, and imp.load_module no longer supports the C_EXTENSION type.  It looks like a regression.

To reproduce, I was able to run the following lines with the .pyd file in the current directory:

   import imp
   fp, pathname, description = imp.find_module('_Widget')
   imp.load_module('_Widget', fp, pathname, description)

Nit: Also note the missing final ')' in the ImportError message.
msg169528 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2012-08-31 12:33
An even easier reproducer:

>>> import imp
[93559 refs]
>>> details = imp.find_module("_elementtree")
[93572 refs]
>>> mod = imp.load_module('_elementtree', *details)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/ncoghlan/devel/py3k/Lib/imp.py", line 171, in load_module
    raise ImportError(msg, name=name)
ImportError: Don't know how to import _elementtree (type code 3
[93572 refs]
msg169529 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2012-08-31 12:55
Patch adds test case and fix (there was simply a missing entry for C_EXTENSION/load_dynamic in the load_module if-elif chain).

Also moves an imp module test from test_import to test_imp (it's the test I used as a guide to check I wasn't missing anything obvious when writing the new one).
msg169530 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2012-08-31 12:59
Tweaked patch to also fix the typo in the error message
msg169531 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2012-08-31 13:07
I'm heading to bed, so whoever reviews this one, please feel free to add a NEWS entry and check it in :)
msg169533 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2012-08-31 13:08
LGTM
msg169537 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2012-08-31 14:12
Since I'm still up, I guess I'll check it in, then :)
msg169538 - (view) Author: Roundup Robot (python-dev) Date: 2012-08-31 14:13
New changeset 9ba57938f54d by Nick Coghlan in branch 'default':
Issue #15828: Restore support for C extension modules in imp.load_module()
http://hg.python.org/cpython/rev/9ba57938f54d
msg169539 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2012-08-31 14:15
Over to Georg to cherry pick this one into rc2
msg169543 - (view) Author: Roundup Robot (python-dev) Date: 2012-08-31 15:31
New changeset d54f047312a8 by Brett Cannon in branch 'default':
Issue #15828: Don't try to close a file if imp.find_module() doesn't
http://hg.python.org/cpython/rev/d54f047312a8
msg170031 - (view) Author: Georg Brandl (georg.brandl) * (Python committer) Date: 2012-09-08 05:52
Picked as 4f6f59303146 and a4a9c5717204.
msg170091 - (view) Author: Roundup Robot (python-dev) Date: 2012-09-09 09:18
New changeset 4f6f59303146 by Nick Coghlan in branch 'default':
Issue #15828: Restore support for C extension modules in imp.load_module()
http://hg.python.org/cpython/rev/4f6f59303146

New changeset a4a9c5717204 by Brett Cannon in branch 'default':
Issue #15828: Don't try to close a file if imp.find_module() doesn't
http://hg.python.org/cpython/rev/a4a9c5717204
msg170173 - (view) Author: Paul Moore (paul.moore) * (Python committer) Date: 2012-09-10 10:43
The applied fix appears to have a regression - the file argument is not allowed to be None. The pywin32 post-install script calls imp.load_module for a C extension with file=None, so presumably this worked in earlier versions.

The regression breaks the postinstall script for pywin32, meaning that the start menu shortcuts do not get installed. I do not know if it has any more serious consequences for pywin32.
msg170174 - (view) Author: Tim Golden (tim.golden) * (Python committer) Date: 2012-09-10 10:47
Paul, are you using the hg tip of pywin32? pywin32_postintall.py was
patched a couple of months ago to use imp.load_dynamic (essentially to
work around this issue).
msg170175 - (view) Author: Paul Moore (paul.moore) * (Python committer) Date: 2012-09-10 11:30
On 10 September 2012 11:47, Tim Golden <report@bugs.python.org> wrote:
>
> Tim Golden added the comment:
>
> Paul, are you using the hg tip of pywin32? pywin32_postintall.py was
> patched a couple of months ago to use imp.load_dynamic (essentially to
> work around this issue).

No, this is the 217 release. Looks like they may need a new release then...

Paul
msg170176 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2012-09-10 11:49
Paul, could you please report that issue/question separately? If it's a
regression, it's likely in the importlib migration in general and was
hidden by this bug rather than being introduced by the fix.
msg170179 - (view) Author: Paul Moore (paul.moore) * (Python committer) Date: 2012-09-10 12:05
Logged as #15902.
History
Date User Action Args
2012-09-10 12:05:33paul.mooresetmessages: + msg170179
2012-09-10 11:49:09ncoghlansetmessages: + msg170176
2012-09-10 11:30:47paul.mooresetmessages: + msg170175
2012-09-10 10:47:51tim.goldensetnosy: + tim.golden
messages: + msg170174
2012-09-10 10:43:44paul.mooresetnosy: + paul.moore
messages: + msg170173
2012-09-09 09:18:57python-devsetmessages: + msg170091
2012-09-08 05:52:24georg.brandlsetstatus: open -> closed
resolution: fixed
messages: + msg170031
2012-08-31 15:31:30python-devsetmessages: + msg169543
2012-08-31 14:15:01ncoghlansetassignee: georg.brandl
messages: + msg169539
2012-08-31 14:13:55python-devsetnosy: + python-dev
messages: + msg169538
2012-08-31 14:12:02ncoghlansetmessages: + msg169537
2012-08-31 13:08:11brett.cannonsetmessages: + msg169533
2012-08-31 13:07:08ncoghlansetmessages: + msg169531
2012-08-31 12:59:19ncoghlansetfiles: - issue15828_handle_extension_modules.diff
2012-08-31 12:59:08ncoghlansetfiles: + issue15828_handle_extension_modules.diff

messages: + msg169530
2012-08-31 12:57:40ncoghlansetnosy: + georg.brandl

stage: commit review
2012-08-31 12:55:39ncoghlansetfiles: + issue15828_handle_extension_modules.diff
keywords: + patch
messages: + msg169529
2012-08-31 12:33:38ncoghlansetkeywords: + 3.3regression

messages: + msg169528
2012-08-31 11:10:20pitrousetpriority: normal -> release blocker
nosy: + brett.cannon, ncoghlan
2012-08-31 07:32:28metolonecreate