The pywin32 package use imp.load_dynamic() to load a DLL with Windows specific type. On Python 3.4+ imp.load_dynamic() point to the following code which use the newer importlib module:

        import importlib.machinery
        loader = importlib.machinery.ExtensionFileLoader(name, path)
        return loader.load_module()

In pywin32 a mechanism is used to be able to have part of a module globals defined in python (pywintypes.py), and the reminder in a DLL (pywintypesXX.dll). The code in pywin32 between 3.4 and 3.5 is the same.

In Python 3.4, calling imp.load_dynamic(), which point to the code above, inside the python part of the module definition loads the types defined in the DLL.

In Python 3.5, calling imp.load_dynamic(), which also points to the code above, inside the python part of the module definition reloads the same python module, not the DLL. Even though a path to a DLL is given AND when doing introspection of the module it clearly points to the specified DLL.

This is a change of behavior that breaks pywin32, but possibly other modules that rely on this behavior.

This is most likely related to PEP-489, which changed extension module loading in what was meant to be a backward-compatible way.

It is also possible that the root cause is related to Microsoft Windows Update 2999226 and/or 3065987. The behavior was the same between 3.4 and 3.5 on a machine without these updates and is believed to have changed after the install of these updates.

Did the behavior change for 3.4 after the updates? It seems unlikely.

Zach's right: we need a test to formally state the expectations here.

No the behavior only changed for 3.5. 3.4 works just fine.

I suggest the test should use pywin32. The test script could be only:

import pywintypes
dir(pywintypes)

Testing for an attribute that is defined in the DLL would make it pass or fail.

I'd rather have our own test pyd that shows a minimal repro of the behavior. Requiring pywin32 for running one test is excessive, won't reveal the issue (if it exists) on other platforms, and is not necessarily going to help someone debug the issue.

I agree this is probably due to PEP-489, which means that Eric is almost certainly the best person to look into it. It may also be due to the list of suffixes changing for 3.5 (".cp35-win32.pyd" is now a valid suffix as well as just ".pyd", for example), but looking at pywintypes.py that seems unlikely.

Eric or me.

I'm not sure I understand the description clearly. Etienne, would it be possible to write a smaller reproducer, that wouldn't be tied to Windows?
Or is this Windows-only behavior?

At this point I can't say if it's Windows only or if it affect all platform. I can't even guarantee that it will appear on all Windows platform. On my platform (see following), it doesn't work:

Windows 7 64 bits with updates 2999226 and 3065987 installed.

I can't confirm the update are related either. I just did a quick analysis and between the time 3.5 was working and wasn't anymore these updates were installed.

I will create a quick script to reproduce the behavior.

The only dll / pyd files I have are all in pywin32. I don't have a build environment for extensions. Can anyone provide me with a very simple extension DLL with at least one exported attribute?

3.5 won't work without 2999226 at all, so ignore that. It's the C runtime.

Is it possible that the C runtime introduced with 2999226, which I believe is the "universal runtime" Microsoft is trying to introduce, modified something that makes importlib break on 3.5???

It's possible, but that isn't solvable or testable by removing the update. To solve that we need a test and then fix the bug.

I replaced:
import importlib.machinery
loader = importlib.machinery.ExtensionFileLoader(name, path)
return loader.load_module()

With:
import importlib.machinery
loader = importlib.machinery.ExtensionFileLoader(modname, filename)
spec = importlib.machinery.ModuleSpec(
name = modname,
loader = loader,
origin = filename,
loader_state = 1234,
is_package = False,
)
mod = loader.create_module(spec)
loader.exec_module(mod)

And it now works as advertised. Since load_module() is flagged as Deprecated, I believe no correction is necessary as the preffered way to load a module, with exec_module(), is working.

I will do some more tests to be sure it's the case.

This also affects py2exe, which dynamically generates stub .pyc loaders inside a ZIP which then load .pyd modules outside the ZIP with the same name.

Windows 8, x64, visual studio 2015 enterprise.

It is simple enough to work around by doing a del sys.modules[modname] prior to the load_dynamic call, but obviously this goes against the documented behaviour.

Given that both pywin32 *and* py2exe have hit this, even if it can't be resolved for 3.5.0, it would be highly desirable to address it for 3.5.1.

It's certainly intended that load_module() keep working as it has previously, rather than forcing people to switch to exec_module() in order to support extensions modules properly on Windows. Aside from general backwards compatibility expectations, there are still some things that load_module() can do that the new PEP 451/489 mechanisms can't handle yet.

Indeed. I don't have access to a Windows machine, but I will try reproducing the problem on another system.

Anything happening with this? We tag 3.5.0rc3 in about 36 hours.

So, if I understand correctly, the problem is that the new imp.load_dynamic in 3.5.0b checks sys.modules, so if a module of the same name was loaded previously, it's only reloaded, skipping the create_module step.

This patch bypasses that check, so a new module is always loaded.

This brings this aspect of imp.load_dynamic to how it was in 3.4. However, I have no way to test if it fixes pywin32 or py2exe.

Assuming I didn't screw this up -- I'm new to the whole Windows development thing -- the new test along with the rest of the test suite pass under Windows 10 on the default branch.

Huh, for some reason the Rietveld diff missed the changes to _testmultiphase.c that are in the patch file.

The change looks good to me. The test adds a new "test.imp_dummy" module, imports that, and then ensures it can be replaced by using imp.load_dynamic to import "test.imp_dummy" from the _testmultiphase module instead.

The functional change itself is isolated to imp.load_dynamic, so no other code paths will be affected.

3.5.0 PR for the change is at https://bitbucket.org/larry/cpython350/pull-requests/16

This *is* a regression (since it's an unintended behavioural change in imp.load_dynamic), but it's one with a relatively straightforward workaround, so it would also be reasonable to defer fixing it to 3.5.1.

On the other hand, the risk is low, since the only functional change is in imp.load_dynamic itself, rather than in the core import machinery (that calls the underlying loader methods directly rather than going through the imp module API).

Pull request accepted. I had to do it manually, as I got a Misc/NEWS merge conflict. But from where I'm sitting it looks like Bitbucket understands the pull request was accepted and merged.

Please forward-merge. Thanks!

New changeset 087464c9f982 by Nick Coghlan in branch '3.5':
Close bpo-24748: Restore imp.load_dynamic compatibility
https://hg.python.org/cpython/rev/087464c9f982

Just noting explicitly that this has been forward merged to the default branch by Steve Dower after Larry merged in the latest 3.5.0 rc changes.