classification
Title: CDLL __init__ no longer supports name being passed as None when the handle is not None
Type: behavior Stage: test needed
Components: ctypes, Windows Versions: Python 3.9, Python 3.8
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: David Heffernan, Jonathan Hsu, eryksun, paul.moore, steve.dower, tim.golden, zach.ware
Priority: normal Keywords: 3.8regression, easy, newcomer friendly

Created on 2020-01-07 10:53 by David Heffernan, last changed 2020-06-07 06:19 by Jonathan Hsu.

Messages (7)
msg359501 - (view) Author: David Heffernan (David Heffernan) Date: 2020-01-07 10:53
When creating an instance of CDLL (or indeed WinDLL) for a DLL that is already loaded, you pass the HMODULE in the handle argument to the constructor.

In older versions of ctypes you could pass None as the name argument when doing so. However, the changes in https://github.com/python/cpython/commit/2438cdf0e932a341c7613bf4323d06b91ae9f1f1 now mean that such code fails with a NoneType is not iterable error.

The relevant change is in __init__ for CDLL. The code inside the if _os.name == "nt" block sets up mode, but this is pointless is handle is not None. Because the mode variable is never used, rightly so because the DLL is already loaded.

The issue could be resolved by changing

if _os.name == "nt":

to

if _os.name == "nt" and handle is not None:


The following program demonstrates the issue:

import ctypes

handle = ctypes.windll.kernel32._handle
print(handle)
lib = ctypes.WinDLL(name=None, handle=handle)
print(lib._handle)

This runs to completion in Python 3.7 and earlier, but fails in Python 3.8 and later:

Traceback (most recent call last):
  File "test.py", line 5, in <module>
    lib = ctypes.WinDLL(name=None, handle=handle)
  File "C:\Program Files (x86)\Python\38\lib\ctypes\__init__.py", line 359, in __init__
    if '/' in name or '\\' in name:
TypeError: argument of type 'NoneType' is not iterable
msg359536 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2020-01-07 20:29
Good catch.

We should probably make the line 351 check "if name and ('/' in name ..." like the others in the same function.
msg359539 - (view) Author: David Heffernan (David Heffernan) Date: 2020-01-07 20:40
Personally I'd hang this off whether handle has been specified. It seems pointless to set the mode if you are never going to use it.
msg359540 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2020-01-07 20:42
In that case, we should refactor the init method to check whether handle has been specified earlier, so that it's obvious that the two conditional blocks are never executed in that case.
msg359543 - (view) Author: David Heffernan (David Heffernan) Date: 2020-01-07 20:46
I would approve of that

On Tue, 7 Jan 2020, 20:43 Steve Dower, <report@bugs.python.org> wrote:

>
> Steve Dower <steve.dower@python.org> added the comment:
>
> In that case, we should refactor the init method to check whether handle
> has been specified earlier, so that it's obvious that the two conditional
> blocks are never executed in that case.
>
> ----------
>
> _______________________________________
> Python tracker <report@bugs.python.org>
> <https://bugs.python.org/issue39243>
> _______________________________________
>
msg359565 - (view) Author: Eryk Sun (eryksun) * (Python triager) Date: 2020-01-08 05:23
I'd like to match POSIX here by supporting path-like names. Also, I think it would be cleaner to split out the platform-specific work into a separate _load_library method, like how subprocess.Popen is designed, and to stop pretending that WINAPI LoadLibraryEx is POSIX dlopen. For example:

    if _os.name == "nt":
        import nt as _nt
        from _ctypes import LoadLibrary as _LoadLibrary
        from _ctypes import FUNCFLAG_STDCALL as _FUNCFLAG_STDCALL
        # ...
    else:
        from _ctypes import dlopen as _dlopen

    # ...

    class CDLL:
        # ...
        
        def __init__(self, name, mode=DEFAULT_MODE, handle=None,
                     use_errno=False, use_last_error=False, winmode=None):

            class _FuncPtr(_CFuncPtr):
                _restype_ = self._func_restype_
                _flags_ = self._func_flags_
                if use_errno:
                    _flags_ |= _FUNCFLAG_USE_ERRNO
                if use_last_error:
                    _flags_ |= _FUNCFLAG_USE_LASTERROR

            self._FuncPtr = _FuncPtr
            self._name = name
            if handle is None:
                self._handle = self._load_library(name, mode, winmode)
            else:
                self._handle = handle

        if _os.name == "nt":
            def _load_library(self, name, mode, winmode):
                if winmode is None:
                    winmode = _nt._LOAD_LIBRARY_SEARCH_DEFAULT_DIRS
                if name:
                    name = _os.fsdecode(name)
                    # WINAPI LoadLibrary searches for a DLL if the given name
                    # is not fully qualified with an explicit drive. For POSIX
                    # compatibility, and because the DLL search path no longer
                    # contains the working directory, begin by fully resolving
                    # any name that contains a path separator.
                    if '/' in name or '\\' in name:
                        name = _nt._getfullpathname(name)
                        # Given a fully-qualified DLL name, allow loading
                        # dependents from its directory.
                        winmode |= _nt._LOAD_LIBRARY_SEARCH_DLL_LOAD_DIR
                return _LoadLibrary(name, winmode)

        else:
            def _load_library(self, name, mode, winmode):
                if _sys.platform.startswith("aix"):
                    # When the name contains ".a(" and ends with ")", for example,
                    # "libFOO.a(libFOO.so)" - this is taken to be an 
                    # archive(member) syntax for dlopen(), and the mode is
                    # adjusted. Otherwise, name is presented to dlopen() as a
                    # file argument.
                    if name and name.endswith(")") and ".a(" in name:
                        mode |= _os.RTLD_MEMBER | _os.RTLD_NOW
                return _dlopen(name, mode)
msg359607 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2020-01-08 16:35
> I think it would be cleaner to split out the platform-specific work into a separate _load_library method

SGTM
History
Date User Action Args
2020-06-07 06:19:59Jonathan Hsusetnosy: + Jonathan Hsu
2020-01-08 16:35:27steve.dowersetmessages: + msg359607
2020-01-08 05:23:17eryksunsetnosy: + eryksun

messages: + msg359565
stage: test needed
2020-01-07 20:46:23David Heffernansetmessages: + msg359543
2020-01-07 20:42:59steve.dowersetmessages: + msg359540
2020-01-07 20:40:10David Heffernansetmessages: + msg359539
2020-01-07 20:29:17steve.dowersetkeywords: + easy, newcomer friendly
2020-01-07 20:29:07steve.dowersetkeywords: + 3.8regression

messages: + msg359536
2020-01-07 10:53:45David Heffernancreate