classification
Title: Output .pyd name in error message of ImportError when DLL load fails
Type: enhancement Stage: resolved
Components: Extension Modules Versions: Python 3.3
process
Status: open Resolution: fixed
Dependencies: 1559549 Superseder:
Assigned To: brian.curtin Nosy List: amaury.forgeotdarc, brian.curtin, cgohlke, eric.snow, georg.brandl, kristjan.jonsson, loewis, ncoghlan, piotr.dobrogost, python-dev, techtonik
Priority: normal Keywords: patch

Created on 2011-01-07 15:48 by techtonik, last changed 2012-07-22 22:53 by kristjan.jonsson.

Files
File name Uploaded Description Edit
issue10854.diff brian.curtin, 2011-07-27 04:01 review
ImportError.patch kristjan.jonsson, 2012-07-22 11:28 review
Messages (23)
msg125649 - (view) Author: anatoly techtonik (techtonik) Date: 2011-01-07 15:48
When an extension could not be loaded, because it requires some DLL that is missing, Python shows the following error message:

  ImportError: DLL load failed: The specified module could not be found.

It will help tremendously in debugging extension problems if the missing DLL name is specified in error message.
msg125651 - (view) Author: Amaury Forgeot d'Arc (amaury.forgeotdarc) * (Python committer) Date: 2011-01-07 16:21
Is it even possible? Each time I tried, the only solutions involved an external program like Dependency Walker.
msg125652 - (view) Author: Brian Curtin (brian.curtin) * (Python committer) Date: 2011-01-07 16:22
I'm pretty sure we can't do this, if I understand your request.

Say you have techtonik.pyd as your extension and it depends on foobar.dll. If we try to load techtonik.pyd and this pyd can't find or successfully load foobar.dll, Python doesn't know about *this* DLL. It just knows that it can't load techtonik.pyd, so that's what we'd be able to output, which isn't that helpful.
msg125653 - (view) Author: Brian Curtin (brian.curtin) * (Python committer) Date: 2011-01-07 16:23
Agree with Amaury. depends has always been my solution to this type of problem.
msg125663 - (view) Author: anatoly techtonik (techtonik) Date: 2011-01-07 17:42
Can you cross reference the part of code where this error is catched?

Why Python can't get information about the reason .DLL is not loaded?

Is it at least possible to add a hook point at the exact time the import fails to insert component that is able to inquiry system about this information (like depends)?
msg125665 - (view) Author: Brian Curtin (brian.curtin) * (Python committer) Date: 2011-01-07 17:47
See _PyImport_GetDynLoadFunc in Python/dynload_win.c -- that's where this is happening.


> Why Python can't get information about the reason .DLL is not loaded?

Windows does not provide it in the case you are speaking of. If I call LoadLibraryEx for techtonik.pyd, either it succeeds because everything is there, or it fails because something is not there. We get no information about dependency hierarchies failing to load.
msg125668 - (view) Author: Amaury Forgeot d'Arc (amaury.forgeotdarc) * (Python committer) Date: 2011-01-07 17:53
The code is in Python/dynload_win.c

To load an extension module mymodule.pyd, Python calls LoadLibrary('/path/to/mymodule.pyd'); when it returns NULL, the code calls GetLastError() (which returns 126 in this case) then FormatMessage to get an error message.

To get the full path to the .pyd, you can use imp.find_module(), but there is nothing else we can do I fear.
msg125682 - (view) Author: anatoly techtonik (techtonik) Date: 2011-01-07 18:59
I see. But depends.exe dependency walker somehow finds the exact code that is failing, so there should be a way to do extra investigation in case of error.

py2exe has a custom .dll loader that bypasses LoadLibrary(Ex) calls from that I can see at  http://py2exe.svn.sourceforge.net/viewvc/py2exe/trunk/py2exe/source/

Even if this code can't be used as a replacement for LoadLibrary calls - it can be very useful for troubleshooting problems in case of missing dependencies. It will also help when embedding python or developing extensions.
msg125727 - (view) Author: Martin v. Löwis (loewis) * (Python committer) Date: 2011-01-07 23:28
Closing this as "won't fix". Python is not going to reimplement depends.exe.
msg125765 - (view) Author: anatoly techtonik (techtonik) Date: 2011-01-08 08:57
Martin, shouldn't there be at least majority to reach consensus on this matter, or you're self-proclaimed BSDL II? =)

At least failed .pyd module name should be present in error message.
msg125768 - (view) Author: Georg Brandl (georg.brandl) * (Python committer) Date: 2011-01-08 09:30
No idea what a BSDL is, but a developer is definitely entitled to close a bug without resorting to a "majority" decision (what is that, anyway? should we hold monthly parliament over bugs?).

You're of course free to reopen with a different request, as you did, and I also think it'd be useful to have the failed module name in the message.
msg126279 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2011-01-14 17:20
As Martin says, we aren't inclined to implement or maintain the equivalent of depends.exe in order to provide a slightly better error message.

If anyone wants to reopen this issue, provide a patch. Otherwise devs are just going to close it again.

Optionally, campaign for a fix on python-list and see if you can persuade someone *else* to provide a patch.

You could even go to the source of the problem and ask *Microsoft* to provide a better error message.
msg126281 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2011-01-14 17:24
Sorry, I missed that the request had changed to just the .pyd name. That at least is useful, since you then know where to start with depends.exe. It should be fairly straightforward to implement as well.

Adjusted issue title accordingly.
msg126283 - (view) Author: anatoly techtonik (techtonik) Date: 2011-01-14 17:45
Great. Thanks!
msg141205 - (view) Author: Brian Curtin (brian.curtin) * (Python committer) Date: 2011-07-27 04:01
How about something like this?

ImportError moves from being a "simple" exception to a "complex" one, then adds a "name" and "path" attribute. In dynload_win.c where we try (and fail) to load C extensions, the name and path are set on the ImportError. The test simply creates a file following the C extension name format and checks that the attributes end up being correct upon ImportError.

Jazzing up the ImportErrors elsewhere in the code might be useful after this, but I'll handle that in another issue if this gains traction.
msg141209 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2011-07-27 05:53
See also #1559549, which similarly adds attribute support to ImportError and covers some of the issues with using positional arguments to do so.

Extending that approach to a path keyword argument as well should work nicely.
msg147439 - (view) Author: Brian Curtin (brian.curtin) * (Python committer) Date: 2011-11-11 16:30
Marked #1559549 as a dependency. I combine the patch in this issue with the one over there.
msg158399 - (view) Author: Roundup Robot (python-dev) Date: 2012-04-16 05:10
New changeset f341b99bb370 by Brian Curtin in branch 'default':
Fix #10854. Make use of the new path and name attributes on ImportError
http://hg.python.org/cpython/rev/f341b99bb370
msg162376 - (view) Author: Kristján Valur Jónsson (kristjan.jonsson) * (Python committer) Date: 2012-06-05 22:05
repr() or str() of ImportError doesn't include the 'name' or 'path' members.  It would be useful to have them added to the repr if present.
msg162616 - (view) Author: Kristján Valur Jónsson (kristjan.jonsson) * (Python committer) Date: 2012-06-11 13:18
Brian, reopening this since the original issue isn't addressed:  The "path" and "module" attributes aren't part of the error "repr"
msg166124 - (view) Author: Kristján Valur Jónsson (kristjan.jonsson) * (Python committer) Date: 2012-07-22 11:28
An example patch that aims to put the "name" and "path" in str() and repr() representations of ImportError.

Output when _socket.pyd is not found:

>>> import socket
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "D:\pydev\hg\cpython3\lib\socket.py", line 47, in <module>
    import _socket
ImportError: No module named '_socket', _socket,
>>> sys.last_value
<ImportError "No module named '_socket'", '_socket', None>
>>>
msg166143 - (view) Author: Amaury Forgeot d'Arc (amaury.forgeotdarc) * (Python committer) Date: 2012-07-22 15:35
How it the new message better than "ImportError: No module named '_socket'"?
msg166179 - (view) Author: Kristján Valur Jónsson (kristjan.jonsson) * (Python committer) Date: 2012-07-22 22:53
Ok, the demo demo I added sucks, because I didn't demo the original problem (.pyd found but could not be loaded), but the point is, it displays the "name" and "path" attributes.  They can be set, for example, if the module is found but cannot be loaded for some reason.

That's the point of this issue.  The fix added a "name" and "path" attributes, but they were never actually _output_ when such an error occurred.
History
Date User Action Args
2012-07-22 22:53:15kristjan.jonssonsetmessages: + msg166179
2012-07-22 15:35:22amaury.forgeotdarcsetmessages: + msg166143
2012-07-22 11:28:31kristjan.jonssonsetfiles: + ImportError.patch

messages: + msg166124
2012-07-21 20:19:35piotr.dobrogostsetnosy: + piotr.dobrogost
2012-06-11 13:18:52kristjan.jonssonsetstatus: closed -> open

messages: + msg162616
2012-06-05 22:05:42kristjan.jonssonsetnosy: + kristjan.jonsson
messages: + msg162376
2012-04-16 05:12:12brian.curtinsetstatus: open -> closed
resolution: fixed
stage: needs patch -> resolved
2012-04-16 05:10:32python-devsetnosy: + python-dev
messages: + msg158399
2011-11-22 21:37:58eric.snowsetnosy: + eric.snow
2011-11-22 20:32:50brian.curtinsetassignee: brian.curtin
2011-11-11 16:30:13brian.curtinsetdependencies: + ImportError needs attributes for module and file name
messages: + msg147439
2011-07-27 05:53:18ncoghlansetmessages: + msg141209
2011-07-27 04:01:52brian.curtinsetfiles: + issue10854.diff
keywords: + patch
messages: + msg141205
2011-01-21 03:55:19cgohlkesetnosy: + cgohlke
2011-01-14 17:45:39techtoniksetnosy: loewis, georg.brandl, amaury.forgeotdarc, ncoghlan, techtonik, brian.curtin
messages: + msg126283
2011-01-14 17:24:56ncoghlansetstatus: closed -> open
title: Output DLL name in error message of ImportError when DLL is missing -> Output .pyd name in error message of ImportError when DLL load fails
nosy: loewis, georg.brandl, amaury.forgeotdarc, ncoghlan, techtonik, brian.curtin
messages: + msg126281

resolution: wont fix -> (no value)
2011-01-14 17:20:28ncoghlansetstatus: open -> closed
nosy: + ncoghlan
messages: + msg126279

2011-01-08 09:30:08georg.brandlsetnosy: + georg.brandl
messages: + msg125768
2011-01-08 08:57:22techtoniksetstatus: closed -> open
nosy: loewis, amaury.forgeotdarc, techtonik, brian.curtin
messages: + msg125765
2011-01-07 23:28:13loewissetstatus: open -> closed

nosy: + loewis
messages: + msg125727

resolution: wont fix
2011-01-07 19:02:17brian.curtinsetnosy: amaury.forgeotdarc, techtonik, brian.curtin
stage: needs patch
2011-01-07 18:59:28techtoniksetnosy: amaury.forgeotdarc, techtonik, brian.curtin
messages: + msg125682
2011-01-07 17:53:16amaury.forgeotdarcsetnosy: amaury.forgeotdarc, techtonik, brian.curtin
messages: + msg125668
2011-01-07 17:47:07brian.curtinsetnosy: amaury.forgeotdarc, techtonik, brian.curtin
messages: + msg125665
2011-01-07 17:42:29techtoniksetnosy: amaury.forgeotdarc, techtonik, brian.curtin
messages: + msg125663
2011-01-07 16:23:12brian.curtinsetnosy: amaury.forgeotdarc, techtonik, brian.curtin
messages: + msg125653
2011-01-07 16:22:20brian.curtinsetnosy: + brian.curtin
messages: + msg125652
2011-01-07 16:21:53amaury.forgeotdarcsetnosy: + amaury.forgeotdarc
messages: + msg125651
2011-01-07 15:59:25brian.curtinsettype: resource usage -> enhancement
versions: + Python 3.3, - Python 2.7
2011-01-07 15:48:43techtonikcreate