classification
Title: ctypes loads wrong version of C runtime, leading to error message box from system
Type: behavior Stage: needs patch
Components: ctypes Versions: Python 3.2, Python 3.3, Python 2.7
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: Artfunkel, Igor.Skochinsky, Keno Fischer, amaury.forgeotdarc, asvetlov, belopolsky, cgohlke, dancol, eryksun, ezio.melotti, mark.dickinson, meador.inge, palm.kevin
Priority: normal Keywords:

Created on 2013-02-16 05:33 by dancol, last changed 2016-07-09 13:47 by Keno Fischer.

Messages (9)
msg182212 - (view) Author: Daniel Colascione (dancol) Date: 2013-02-16 05:33
Suppose we're running a Python program in an environment where PATH contains a directory that contains msvcr90.dll version A and that python27.exe is manifested to use msvcr90.dll version B, which is available in the SxS store but not on PATH.

Normally, python27.exe's side-by-side (SxS) manifest contains a description of *precisely* the correct version of msvcr90.dll to use, version B, so when python27.exe starts, the NT loader ignores msvcr90.dll version A and loads msvcr90.dll version B.

Everything works fine until somebody calls ctypes.CDLL("c"); uuid.py, which tries to find "uuid_generate_random" in libc on module load, is an example of a component that unexpectedly tries to load libc through ctypes.

Now, when someone tried to load "c", ctypes internally maps "c" to the C runtime library using ctypes.util.find_msvcrt, then calls _ctypes.LoadLibrary, which calls the Win32 API function LoadLibraryA. LoadLibraryA tries to find "msvcr90.dll", but WITHOUT CONSULTING THE SXS ACTIVATION CONTEXT, meaning that LoadLibrary finds msvcr90.dll version A, not version B. msvcr90.dll version A isn't loaded yet, so the NT loader does the usual loading and initialization; msvcr90.dll version A's DllMain runs, notices that it's not being loaded as part of an SxS manifest, and presents the user with an R6034 error message, "an application has made an attempt to laod the C runtime library incorrectly".

The overall result is that users of Python programs see error message popups from "Microsoft Visual C++ Runtime Library" that, in most cases, are completely benign. This problem doesn't occur if the correct version of msvcr90.dll happens to be in PATH.

One solution is to have _ctypes.LoadLibrary use the correct activation context; another is to use GetModuleHandleEx(GET_MODULE_HANDLE_EX_FLAG_FROM_ADDRESS, (LPCTSTR) &fopen, &module) to retrieve a handle to the current C runtime library without having to go through LoadLibrary at all.
msg182700 - (view) Author: Ezio Melotti (ezio.melotti) * (Python committer) Date: 2013-02-22 21:19
Thanks for the report.  Could you also provide a patch?
msg194562 - (view) Author: Igor Skochinsky (Igor.Skochinsky) Date: 2013-08-06 16:13
Just had this issue when using networkx (which imports uuid). One keyword that would help visibility is R6034 (the runtime error number). A couple of reports related to this:

https://projects.blender.org/tracker/index.php?func=detail&aid=27666
http://forums.boxee.tv/showthread.php?t=15425

The proper solution is to fix ctypes, but in the meantime we can at least fix uuid. The error seems to be triggered by this snippet:

    # The uuid_generate_* routines are provided by libuuid on at least
    # Linux and FreeBSD, and provided by libc on Mac OS X.
    for libname in ['uuid', 'c']:
        try:
            lib = ctypes.CDLL(ctypes.util.find_library(libname))
        except:
            continue
        if hasattr(lib, 'uuid_generate_random'):
            _uuid_generate_random = lib.uuid_generate_random
        if hasattr(lib, 'uuid_generate_time'):
            _uuid_generate_time = lib.uuid_generate_time

Since this code is useless on Windows, protecting it in "if os.name not in ['nt', 'ce']" does the trick.

BTW, instead of going all way with activation context etc., a simpler solution would be to add to ctypes something like this in CDLL.__init__:

if os.name in ['nt', 'ce'] and name == util.find_msvcrt():  # TODO: handle the extension
  self._handle = windll.kernel32.GetModuleHandleA(self._name)

i.e. use the already present runtime DLL instead of trying to load it again.
msg209307 - (view) Author: Tom Edwards (Artfunkel) Date: 2014-01-26 13:17
Still an issue in 3.3.3. Igor's fix still works, thankfully.
msg210095 - (view) Author: Palm Kevin (palm.kevin) Date: 2014-02-03 08:21
+1
msg210102 - (view) Author: Palm Kevin (palm.kevin) Date: 2014-02-03 09:11
Reproducible for Py 3.2.5
msg269992 - (view) Author: Keno Fischer (Keno Fischer) Date: 2016-07-08 15:52
pyzmq has a similar issue. I believe one solution would be to call
`_Py_ActivateActCtx` in ctypes' load_library, i.e. do
```
#if HAVE_SXS
        ULONG_PTR cookie = _Py_ActivateActCtx();
#endif
        hMod = LoadLibraryW(name);
#if HAVE_SXS
        _Py_DeactivateActCtx(cookie);
#endif
```
in that function. I don't know enough about python or ctypes to say whether this patch works as is (_Py_ActivateActCtx seems to be an internal function, not sure if ctypes has access to that).
msg270036 - (view) Author: Eryk Sun (eryksun) * (Python triager) Date: 2016-07-09 12:08
This is only a problem for ctypes when python27.dll is used in an application that isn't manifested to load the "Microsoft.VC90.CRT" assembly. ctypes doesn't have this problem for a regular script run via the 2.7 version of python.exe, since the loader uses the python.exe activation context. Daniel claimed otherwise, but his claim is demonstrably false. Maybe he was embedding Python.

Keno, I don't think it's a good idea to arbitrarily set the Python DLL's activation context every time ctypes calls LoadLibrary. I proposed a more generic solution in issue 24429, msg246754.

An alternative solution would be to implement a pure-Python activation context class that works as a Python context manager. For example, it could create and activate a context using the hModule and lpResourceName fields:

    with ctypes.ACTCTX(module_handle=sys.dllhandle, 
                       resource_name=2):
        libc = ctypes.CDLL('msvcr90')

I have a Stack Overflow answer that demonstrates using activation contexts with ctypes:

http://stackoverflow.com/a/27392347/205580
msg270041 - (view) Author: Keno Fischer (Keno Fischer) Date: 2016-07-09 13:47
Yes, you are correct about it being only an issue in the embedding context.
I agree that it might not be a good idea to do do this for every library, but I wanted to revive the discussion since this kind of thing seems like something that comes up frequently when people embed python. Thanks for the reference to the other thread. A solution that allows the activation context to be specified would be great.
History
Date User Action Args
2016-07-09 13:47:04Keno Fischersetmessages: + msg270041
2016-07-09 12:08:21eryksunsetnosy: + eryksun
messages: + msg270036
2016-07-08 15:52:10Keno Fischersetnosy: + Keno Fischer
messages: + msg269992
2014-04-17 07:54:01cgohlkesetnosy: + cgohlke
2014-02-03 09:11:05palm.kevinsetmessages: + msg210102
versions: + Python 3.2
2014-02-03 08:29:59mark.dickinsonsetnosy: + mark.dickinson
2014-02-03 08:21:57palm.kevinsetnosy: + palm.kevin
messages: + msg210095
2014-01-26 13:33:00Artfunkelsetversions: + Python 2.7
2014-01-26 13:17:02Artfunkelsetnosy: + Artfunkel

messages: + msg209307
versions: + Python 3.3, - Python 2.7
2013-08-23 18:11:46asvetlovsetnosy: + asvetlov
2013-08-06 16:13:42Igor.Skochinskysetnosy: + Igor.Skochinsky
messages: + msg194562
2013-02-22 21:19:19ezio.melottisetnosy: + ezio.melotti
messages: + msg182700
2013-02-22 17:36:42terry.reedysetnosy: + amaury.forgeotdarc, belopolsky, meador.inge
stage: needs patch

versions: + Python 2.7, - Python 3.5
2013-02-16 05:33:20dancolcreate