classification
Title: Change of behavior for importlib between 3.4 and 3.5 with DLL loading
Type: behavior Stage: resolved
Components: Extension Modules, Windows Versions: Python 3.6, Python 3.5
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: ncoghlan Nosy List: James Salter, brett.cannon, ebfortin, eric.snow, larry, ncoghlan, paul.moore, petr.viktorin, python-dev, steve.dower, tim.golden, zach.ware
Priority: deferred blocker Keywords: patch

Created on 2015-07-29 13:20 by ebfortin, last changed 2015-09-07 06:46 by ncoghlan. This issue is now closed.

Files
File name Uploaded Description Edit
pywintypes.py ebfortin, 2015-07-29 13:20 pywintypes.py from pywin32
issue24748.patch petr.viktorin, 2015-09-04 10:10 review
Messages (25)
msg247573 - (view) Author: Etienne Fortin (ebfortin) Date: 2015-07-29 13:20
The pywin32 package use imp.load_dynamic() to load a DLL with Windows specific type. On Python 3.4+ imp.load_dynamic() point to the following code which use the newer importlib module:

        import importlib.machinery
        loader = importlib.machinery.ExtensionFileLoader(name, path)
        return loader.load_module()

In pywin32 a mechanism is used to be able to have part of a module globals defined in python (pywintypes.py), and the reminder in a DLL (pywintypesXX.dll). The code in pywin32 between 3.4 and 3.5 is the same.

In Python 3.4, calling imp.load_dynamic(), which point to the code above, inside the python part of the module definition loads the types defined in the DLL.

In Python 3.5, calling imp.load_dynamic(), which also points to the code above, inside the python part of the module definition reloads the same python module, not the DLL. Even though a path to a DLL is given AND when doing introspection of the module it clearly points to the specified DLL. 

This is a change of behavior that breaks pywin32, but possibly other modules that rely on this behavior.
msg247578 - (view) Author: Eric Snow (eric.snow) * (Python committer) Date: 2015-07-29 14:22
This is most likely related to PEP 489, which changed extension module loading in what was meant to be a backward-compatible way.
msg247579 - (view) Author: Etienne Fortin (ebfortin) Date: 2015-07-29 15:28
It is also possible that the root cause is related to Microsoft Windows Update 2999226 and/or 3065987. The behavior was the same between 3.4 and 3.5 on a machine without these updates and is believed to have changed after the install of these updates.
msg247581 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2015-07-29 16:14
Did the behavior change for 3.4 after the updates? It seems unlikely.

Zach's right: we need a test to formally state the expectations here.
msg247582 - (view) Author: Etienne Fortin (ebfortin) Date: 2015-07-29 16:24
No the behavior only changed for 3.5. 3.4 works just fine.
msg247590 - (view) Author: Etienne Fortin (ebfortin) Date: 2015-07-29 18:05
I suggest the test should use pywin32. The test script could be only:

import pywintypes
dir(pywintypes)

Testing for an attribute that is defined in the DLL would make it pass or fail.
msg247593 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2015-07-29 18:23
I'd rather have our own test pyd that shows a minimal repro of the behavior. Requiring pywin32 for running one test is excessive, won't reveal the issue (if it exists) on other platforms, and is not necessarily going to help someone debug the issue.

I agree this is probably due to PEP 489, which means that Eric is almost certainly the best person to look into it. It may also be due to the list of suffixes changing for 3.5 (".cp35-win32.pyd" is now a valid suffix as well as just ".pyd", for example), but looking at pywintypes.py that seems unlikely.
msg247600 - (view) Author: Petr Viktorin (petr.viktorin) * (Python committer) Date: 2015-07-29 18:34
Eric or me.

I'm not sure I understand the description clearly. Etienne, would it be possible to write a smaller reproducer, that wouldn't be tied to Windows?
Or is this Windows-only behavior?
msg247602 - (view) Author: Etienne Fortin (ebfortin) Date: 2015-07-29 18:47
At this point I can't say if it's Windows only or if it affect all platform. I can't even guarantee that it will appear on all Windows platform. On my platform (see following), it doesn't work:

Windows 7 64 bits with updates 2999226 and 3065987 installed.

I can't confirm the update are related either. I just did a quick analysis and between the time 3.5 was working and wasn't anymore these updates were installed.

I will create a quick script to reproduce the behavior.
msg247606 - (view) Author: Etienne Fortin (ebfortin) Date: 2015-07-29 19:05
The only dll / pyd files I have are all in pywin32. I don't have a build environment for extensions. Can anyone provide me with a very simple extension DLL with at least one exported attribute?
msg247608 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2015-07-29 19:21
3.5 won't work without 2999226 at all, so ignore that. It's the C runtime.
msg247609 - (view) Author: Etienne Fortin (ebfortin) Date: 2015-07-29 19:23
Is it possible that the C runtime introduced with 2999226, which I believe is the "universal runtime" Microsoft is trying to introduce, modified something that makes importlib break on 3.5???
msg247613 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2015-07-29 19:46
It's possible, but that isn't solvable or testable by removing the update. To solve that we need a test and then fix the bug.
msg247670 - (view) Author: Etienne Fortin (ebfortin) Date: 2015-07-30 13:45
I replaced:
        import importlib.machinery
        loader = importlib.machinery.ExtensionFileLoader(name, path)
        return loader.load_module()

With:
    import importlib.machinery
    loader = importlib.machinery.ExtensionFileLoader(modname, filename)
    spec = importlib.machinery.ModuleSpec(
            name = modname,
            loader = loader,
            origin = filename,
            loader_state = 1234,
            is_package = False,
        )
    mod = loader.create_module(spec)
    loader.exec_module(mod)

And it now works as advertised. Since load_module() is flagged as Deprecated, I believe no correction is necessary as the preffered way to load a module, with exec_module(), is working. 

I will do some more tests to be sure it's the case.
msg249277 - (view) Author: James Salter (James Salter) * Date: 2015-08-28 13:19
This also affects py2exe, which dynamically generates stub .pyc loaders inside a ZIP which then load .pyd modules outside the ZIP with the same name.

Windows 8, x64, visual studio 2015 enterprise.

It is simple enough to work around by doing a del sys.modules[modname] prior to the load_dynamic call, but obviously this goes against the documented behaviour.
msg249312 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2015-08-29 03:57
Given that both pywin32 *and* py2exe have hit this, even if it can't be resolved for 3.5.0, it would be highly desirable to address it for 3.5.1.

It's certainly intended that load_module() keep working as it has previously, rather than forcing people to switch to exec_module() in order to support extensions modules properly on Windows. Aside from general backwards compatibility expectations, there are still some things that load_module() can do that the new PEP 451/489 mechanisms can't handle yet.
msg249636 - (view) Author: Petr Viktorin (petr.viktorin) * (Python committer) Date: 2015-09-03 13:14
Indeed. I don't have access to a Windows machine, but I will try reproducing the problem on another system.
msg249729 - (view) Author: Larry Hastings (larry) * (Python committer) Date: 2015-09-04 07:12
Anything happening with this?  We tag 3.5.0rc3 in about 36 hours.
msg249748 - (view) Author: Petr Viktorin (petr.viktorin) * (Python committer) Date: 2015-09-04 10:10
So, if I understand correctly, the problem is that the new imp.load_dynamic in 3.5.0b checks sys.modules, so if a module of the same name was loaded previously, it's only reloaded, skipping the create_module step.

This patch bypasses that check, so a new module is always loaded.

This brings this aspect of imp.load_dynamic to how it was in 3.4. However, I have no way to test if it fixes pywin32 or py2exe.
msg249808 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2015-09-04 18:38
Assuming I didn't screw this up -- I'm new to the whole Windows development thing -- the new test along with the rest of the test suite pass under Windows 10 on the default branch.
msg249895 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2015-09-05 05:37
Huh, for some reason the Rietveld diff missed the changes to _testmultiphase.c that are in the patch file.

The change looks good to me. The test adds a new "test.imp_dummy" module, imports that, and then ensures it can be replaced by using imp.load_dynamic to import "test.imp_dummy" from the _testmultiphase module instead.

The functional change itself is isolated to imp.load_dynamic, so no other code paths will be affected.
msg249903 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2015-09-05 11:13
3.5.0 PR for the change is at https://bitbucket.org/larry/cpython350/pull-requests/16

This *is* a regression (since it's an unintended behavioural change in imp.load_dynamic), but it's one with a relatively straightforward workaround, so it would also be reasonable to defer fixing it to 3.5.1.

On the other hand, the risk is low, since the only functional change is in imp.load_dynamic itself, rather than in the core import machinery (that calls the underlying loader methods directly rather than going through the imp module API).
msg249957 - (view) Author: Larry Hastings (larry) * (Python committer) Date: 2015-09-06 04:04
Pull request accepted.  I had to do it manually, as I got a Misc/NEWS merge conflict.  But from where I'm sitting it looks like Bitbucket understands the pull request was accepted and merged.

Please forward-merge.  Thanks!
msg250068 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2015-09-07 05:38
New changeset 087464c9f982 by Nick Coghlan in branch '3.5':
Close #24748: Restore imp.load_dynamic compatibility
https://hg.python.org/cpython/rev/087464c9f982
msg250082 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2015-09-07 06:46
Just noting explicitly that this has been forward merged to the default branch by Steve Dower after Larry merged in the latest 3.5.0 rc changes.
History
Date User Action Args
2015-09-07 06:46:39ncoghlansetmessages: + msg250082
2015-09-07 05:38:03python-devsetnosy: + python-dev
messages: + msg250068
2015-09-06 04:04:37larrysetstatus: open -> closed
assignee: ncoghlan
resolution: fixed
stage: commit review -> resolved
2015-09-06 04:04:15larrysetmessages: + msg249957
2015-09-05 11:13:52ncoghlansetmessages: + msg249903
2015-09-05 05:37:54ncoghlansetmessages: + msg249895
stage: test needed -> commit review
2015-09-04 18:38:05brett.cannonsetmessages: + msg249808
2015-09-04 10:10:09petr.viktorinsetfiles: + issue24748.patch
keywords: + patch
messages: + msg249748
2015-09-04 07:12:05larrysetmessages: + msg249729
2015-09-03 13:14:31petr.viktorinsetmessages: + msg249636
2015-08-29 03:57:30ncoghlansetpriority: normal -> deferred blocker
nosy: + larry
messages: + msg249312

2015-08-28 13:19:55James Saltersetnosy: + James Salter
messages: + msg249277
2015-07-30 13:45:35ebfortinsetmessages: + msg247670
2015-07-29 19:46:52steve.dowersetmessages: + msg247613
2015-07-29 19:23:16ebfortinsetmessages: + msg247609
2015-07-29 19:21:09steve.dowersetmessages: + msg247608
2015-07-29 19:05:40ebfortinsetmessages: + msg247606
2015-07-29 18:47:05ebfortinsetmessages: + msg247602
2015-07-29 18:34:14petr.viktorinsetmessages: + msg247600
2015-07-29 18:23:10steve.dowersetmessages: + msg247593
2015-07-29 18:05:06ebfortinsetmessages: + msg247590
2015-07-29 16:24:00ebfortinsetmessages: + msg247582
2015-07-29 16:14:36steve.dowersetmessages: + msg247581
2015-07-29 15:32:30zach.waresetstage: test needed
2015-07-29 15:28:10ebfortinsetmessages: + msg247579
2015-07-29 14:22:18eric.snowsetnosy: + petr.viktorin, eric.snow, brett.cannon, ncoghlan

messages: + msg247578
versions: + Python 3.6
2015-07-29 13:20:27ebfortincreate