classification
Title: ctypes.CDLL returns singleton objects, resulting in usage conflicts
Type: behavior Stage:
Components: ctypes Versions: Python 2.7
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: Ivan.Pozdeev, amaury.forgeotdarc, belopolsky, eryksun, meador.inge, native_api, r.david.murray
Priority: normal Keywords: patch

Created on 2014-10-04 01:54 by Ivan.Pozdeev, last changed 2018-01-10 17:13 by Ivan.Pozdeev.

Files
File name Uploaded Description Edit
cdll_dont_cache.patch Ivan.Pozdeev, 2014-10-04 02:13
add-warnings.diff native_api, 2014-10-23 13:21
Messages (9)
msg228423 - (view) Author: Ivan Pozdeev (Ivan.Pozdeev) * Date: 2014-10-04 01:54
a LibraryLoader returns the same _FuncPtr object for a given function every time. This way, if two libraries set its attributes (most possibly, `argtypes') to incompatible values (both representing the same C-level entities), one of them will stop working.

I've just discovered such a problem with `pyreadline' and `colorama' which both utilize `windll.kernel32.GetConsoleScreenBufferInfo'.

One solution is to make LibraryLoader return a new object each time. Another (worse IMO) is to provide a clone function for _FuncPtr (it cannot be clones by `copy.copy()').

An example code:

>>> import pyreadline
>>> import pip._vendor.colorama
Readline internal error
Traceback (most recent call last):
  File "c:\python27\lib\site-packages\pyreadline\console\console.py", line 768,
in hook_wrapper_23
    res = ensure_str(readline_hook(prompt))
  File "c:\python27\lib\site-packages\pyreadline\rlmain.py", line 569, in readline
    self.readline_setup(prompt)
  File "c:\python27\lib\site-packages\pyreadline\rlmain.py", line 565, in readline_setup
    self._print_prompt()
  File "c:\python27\lib\site-packages\pyreadline\rlmain.py", line 466, in _print_prompt
    x, y = c.pos()
  File "c:\python27\lib\site-packages\pyreadline\console\console.py", line 261,
in pos
    self.GetConsoleScreenBufferInfo(self.hout, byref(info))
ArgumentError: argument 2: <type 'exceptions.TypeError'>: expected LP_CONSOLE_SCREEN_BUFFER_INFO instance instead of pointer to CONSOLE_SCREEN_BUFFER_INFO

(the same error is printed continuously, on Ctrl-C, the interpreter crashes)
msg228425 - (view) Author: Ivan Pozdeev (Ivan.Pozdeev) * Date: 2014-10-04 02:24
Another possible solution is to prohibit settings attributes of vanilla _FuncPtr objects, requiring a library to make a private copy to be able to do that.
msg228426 - (view) Author: Eryk Sun (eryksun) * Date: 2014-10-04 02:57
The ctypes global LibraryLoader instances are convenient for Windows scripts and applications. Actually what they cache is the library (a CDLL instance), which in turn caches function pointers. 

Python packages, on the other hand, shouldn't use these particular loaders. As you've experienced, doing so can lead to conflicting definitions for restype, argtypes, and errcheck. Instead create a private loader such as cdll = LibraryLoader(CDLL), or windll = LibraryLoader(WinDLL).

Doing this would be pretty much pointless on non-Windows platforms. To gain the benefit of the cache you'd have to use subscripting such as cdll['libc.so.6']. It's simpler to use libc = CDLL('libc.so.6').
msg228503 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2014-10-04 22:14
How does this relate to issue 14201?  That is, is the answer just "use getitem if you don't want caching"?  (And apply the doc patch from that issue :)
msg228522 - (view) Author: Ivan Pozdeev (Ivan.Pozdeev) * Date: 2014-10-05 01:33
@R. David Murray: haha, the reverse change that introduced this problem in the first place! issue 14201's problem is exactly why I was going to suggest to also make _FuncPtr's compare equal if they point to the same function.

@eryksun: Packages do this because it's the natural thing to do - there's no apparent way to clone a pointer, there's not even a notion they _need_ to be cloned. https://docs.python.org/2/library/ctypes.html#loading-shared-libraries only documents __getattr__() and behavior of a LibraryLoader, not of a CDLL . 

Bottom line: ctypes currently leaves a trap for users to fall into. While Python's paradigm is "make right things easy, make wrong things hard".

So, he optimal way seems to require to clone function pointers to set attributes. I'm going to add a guard attribute and a `clone' method to _FuncPtr. It's discussable whether to make cloned/altered pointers compare equal to the originals.
msg228525 - (view) Author: Eryk Sun (eryksun) * Date: 2014-10-05 01:50
> How does this relate to issue 14201?  That is, is the answer just 
> "use getitem if you don't want caching"? 

Unlike CDLL.__getitem__, LibraryLoader.__getitem__ does use the attribute cache. Why use a LibraryLoader if you don't want the cached libs and access to their cached function pointers? That's its main reason for existing.  But really only in Windows, since most DLL names work as a attributes and Windows LoadLibrary appends .DLL. If you don't want cached libs, don't use a LibraryLoader. Just use CDLL('msvcr100'), WinDLL('kernel32'), etc. If you want cached libs without polluting ctypes.cdll or ctypes.windll, just create your own instance such as windll = ctypes.LibraryLoader(ctypes.WinDLL).
msg228535 - (view) Author: Eryk Sun (eryksun) * Date: 2014-10-05 03:29
> Packages do this because it's the natural thing to do

I guess the tutorial is channeling projects toward using the cdll/windll LibraryLoader instances on Windows. It even shows using cdll.LoadLibrary('libc.so.6') on Linux. That's equivalent to CDLL('libc.so.6'); I don't know why one would bother with cdll.LoadLibrary.

> there's not even a notion they _need_ to be cloned. 

The ctypes reference has always explained how CDLL instances cache function pointers via __getattr__ and (formerly) __getitem__. 

The same section also documents that LibraryLoader.__getattr__ caches libraries. However, it's missing an explanation of LibraryLoader.__getitem__, which returns getattr(self, name), for use when the library name isn't a valid Python identifier.

> there's no apparent way to clone a pointer

You can use pointer casting or from_buffer_copy to create a new function pointer. It isn't a clone because it only uses the function pointer type, not the current value of restype, argtypes, and errcheck. But this may be all you need. For example:

    >>> from ctypes import *
    >>> libm = CDLL('libm.so.6')

cast:

    >>> sin = cast(libm.sin, CFUNCTYPE(c_double, c_double))
    >>> sin(3.14/2)
    0.9999996829318346

    >>> sin2 = cast(sin, type(sin))
    >>> sin2.argtypes
    (<class 'ctypes.c_double'>,)
    >>> sin2.restype
    <class 'ctypes.c_double'>

from_buffer_copy:

    >>> sin = CFUNCTYPE(c_double, c_double).from_buffer_copy(libm.sin)
    >>> sin(3.14/2)
    0.9999996829318346

https://docs.python.org/3/library/ctypes.html#ctypes.cast
https://docs.python.org/3/library/ctypes.html#function-prototypes
msg228538 - (view) Author: Ivan Pozdeev (Ivan.Pozdeev) * Date: 2014-10-05 04:17
> If you want cached libs without polluting ctypes.cdll or ctypes.windll, just create your own instance such as windll = ctypes.LibraryLoader(ctypes.WinDLL).

This one looks like the next best thing to the current state of affairs, requiring minimal change to existing code.

`cast' appears to be the "right way" when saving individual _FuncPtr's in a local namespace but it's far from being obvious.

If going this way, a prominent warning in https://docs.python.org/2.7/library/ctypes.html?highlight=ctypes#loading-dynamic-link-libraries with the way to go would probably suffice.
msg229872 - (view) Author: Ivan Pozdeev (native_api) * Date: 2014-10-23 13:21
Here's the warnings patch. No sure if the `copy.copy' recipe is officially supported.
History
Date User Action Args
2018-01-10 17:13:21Ivan.Pozdeevsettype: crash -> behavior
2014-10-23 13:21:24native_apisetfiles: + add-warnings.diff
nosy: + native_api
messages: + msg229872

2014-10-05 04:17:40Ivan.Pozdeevsetmessages: + msg228538
2014-10-05 03:29:15eryksunsetmessages: + msg228535
2014-10-05 01:50:25eryksunsetmessages: + msg228525
2014-10-05 01:33:11Ivan.Pozdeevsetmessages: + msg228522
2014-10-04 22:14:51r.david.murraysetnosy: + r.david.murray
messages: + msg228503
2014-10-04 06:08:54ned.deilysetnosy: + amaury.forgeotdarc, belopolsky, meador.inge
2014-10-04 02:57:37eryksunsetnosy: + eryksun
messages: + msg228426
2014-10-04 02:24:57Ivan.Pozdeevsetmessages: + msg228425
2014-10-04 02:19:58Ivan.Pozdeevsettitle: ctypes.LibraryLoader returns singleton objects, resulting in usage conflicts -> ctypes.CDLL returns singleton objects, resulting in usage conflicts
2014-10-04 02:13:27Ivan.Pozdeevsetfiles: + cdll_dont_cache.patch
keywords: + patch
2014-10-04 01:54:49Ivan.Pozdeevcreate