If I load a dll in ctypes, then delete that loaded DLL instance, the DLL is not unloaded until the script finishes and exits.

I'm trying to write some unit tests in python to exercise that DLL where each test case loads a DLL, does some work, then unloads the DLL.  Unfortunately the DLL only gets unloaded when the unit tests finish.

I've tried forcing the garbage collector to run to get the DLL to unload.  It did nothing...

# load the DLL
parser_dll = CDLL(dllpath)

# do some work here

# 'unload' the dll (or as close as I can get it to it)
if (parser_dll):
    del parser_dll

I should mention also, that this is mostly an issue for me on Win7 x64.  It
does behave 'slightly' better on WinXP x86.

(I have the 64-bit version of python installed on Win7 x64 & the 32-bit
version installed on WinXP)

thanks,
Pat.

On 16 April 2012 14:09, Pat Lynch <report@bugs.python.org> wrote:

>
> New submission from Pat Lynch <plynch76@gmail.com>:
>
> If I load a dll in ctypes, then delete that loaded DLL instance, the DLL
> is not unloaded until the script finishes and exits.
>
> I'm trying to write some unit tests in python to exercise that DLL where
> each test case loads a DLL, does some work, then unloads the DLL.
>  Unfortunately the DLL only gets unloaded when the unit tests finish.
>
> I've tried forcing the garbage collector to run to get the DLL to unload.
>  It did nothing...
>
> # load the DLL
> parser_dll = CDLL(dllpath)
>
> # do some work here
>
> # 'unload' the dll (or as close as I can get it to it)
> if (parser_dll):
>    del parser_dll
>
> ----------
> components: ctypes
> messages: 158433
> nosy: plynch76
> priority: normal
> severity: normal
> status: open
> title: Cannot unload dll in ctypes until script exits
> type: enhancement
> versions: Python 2.7
>
> _______________________________________
> Python tracker <report@bugs.python.org>
> <http://bugs.python.org/issue14597>
> _______________________________________
>

In general it is difficult to impossible to get Python2 to unload modules before the interpreter shuts down.  See issue 9072.  I'm not savvy enough with the C stuff to know if the fact that you loaded it via ctypes changes anything, but I doubt it.

Note that the implication of that issue is that if you could move to Python3 there might be a way to do it, but that would indeed be an enhancement as there is no direct support for it yet.

thanks for the very quick response.

Since LoadLibrary is called in the constructor, why can't FreeLibrary be
called in the destructor?  or at least expose a function to unload that
calls FreeLibrary?

http://msdn.microsoft.com/en-us/library/windows/desktop/ms683152%28v=vs.85%29.aspx

thanks again,
Pat.

On 16 April 2012 14:30, R. David Murray <report@bugs.python.org> wrote:

>
> R. David Murray <rdmurray@bitdance.com> added the comment:
>
> In general it is difficult to impossible to get Python2 to unload modules
> before the interpreter shuts down.  See issue 9072.  I'm not savvy enough
> with the C stuff to know if the fact that you loaded it via ctypes changes
> anything, but I doubt it.
>
> Note that the implication of that issue is that if you could move to
> Python3 there might be a way to do it, but that would indeed be an
> enhancement as there is no direct support for it yet.
>
> ----------
> nosy: +r.david.murray
> versions: +Python 3.3 -Python 2.7
>
> _______________________________________
> Python tracker <report@bugs.python.org>
> <http://bugs.python.org/issue14597>
> _______________________________________
>

In principle, it should be possible (but perhaps not desirable, see below) to call FreeLibrary in a CDLL's __del__. However, since this would be a new feature, it can't go into 2.7. Patches are welcome; make sure to support both FreeLIbrary and dlclose.

There is a general issue with closing/freeing DLLs: if they are still referenced somewhere (e.g. in an atexit function, a C++ virtual method table, or on the call stack of another thread), then a later access to the code will crash the interpreter. In a typical DLL today (including all Python extension modules), the likelihood of crashes is close to 100%. For that reason, it's probably not a good idea to have ctypes auto-close DLLs; instead, it should be an opt-in mechanism.

For most ctypes uses, closing is irrelevant, since people typically access system libraries that are independently loaded anyway, so closing them would not have any effect.

ok, that's fair enough if most usage of ctypes is from people accessing
system libraries :)

I wouldn't have thought my usage was that weird though (given the strength
of using python for unit testing).

In local tests, adding a function CDLL::ForceUnloadDll  (which just calls
FreeLibrary(self._handle)) seems to work well.  I haven't done intensive
testing though at this point.  I could be missing something though.

thanks,
Pat.

On 16 April 2012 14:45, Martin v. Löwis <report@bugs.python.org> wrote:

>
> Martin v. Löwis <martin@v.loewis.de> added the comment:
>
> In principle, it should be possible (but perhaps not desirable, see below)
> to call FreeLibrary in a CDLL's __del__. However, since this would be a new
> feature, it can't go into 2.7. Patches are welcome; make sure to support
> both FreeLIbrary and dlclose.
>
> There is a general issue with closing/freeing DLLs: if they are still
> referenced somewhere (e.g. in an atexit function, a C++ virtual method
> table, or on the call stack of another thread), then a later access to the
> code will crash the interpreter. In a typical DLL today (including all
> Python extension modules), the likelihood of crashes is close to 100%. For
> that reason, it's probably not a good idea to have ctypes auto-close DLLs;
> instead, it should be an opt-in mechanism.
>
> For most ctypes uses, closing is irrelevant, since people typically access
> system libraries that are independently loaded anyway, so closing them
> would not have any effect.
>
> ----------
> nosy: +loewis
>
> _______________________________________
> Python tracker <report@bugs.python.org>
> <http://bugs.python.org/issue14597>
> _______________________________________
>

Just to update:-

I've run this pretty extensively on multiple systems (XP x86 & Win7 64-bit)
and it appears to behave as expected (haven't checked it on Linux).  I have
that code being called in 100s of unit tests.

For python 3.1, would it make sense to add it as a ForceUnload function??
- for safety bail out if handle was not None when passed into the
constructor?  i.e. if somebody has accessed an independently loaded DLL,
they will pass in the handle when constructing the CDLL object.  Disallow
ForceUnload in that case.  ForceUnload will only be allowed in cases where
we created that type by passing in the path to the DLL.

I'll be using this code as a local patch, so no rush to put it into 3.1 etc.

thanks for all the info - much appreciated :)

Pat.

On 16 April 2012 15:04, Pat Lynch <report@bugs.python.org> wrote:

>
> Pat Lynch <plynch76@gmail.com> added the comment:
>
> ok, that's fair enough if most usage of ctypes is from people accessing
> system libraries :)
>
> I wouldn't have thought my usage was that weird though (given the strength
> of using python for unit testing).
>
> In local tests, adding a function CDLL::ForceUnloadDll  (which just calls
> FreeLibrary(self._handle)) seems to work well.  I haven't done intensive
> testing though at this point.  I could be missing something though.
>
> thanks,
> Pat.
>
> On 16 April 2012 14:45, Martin v. Löwis <report@bugs.python.org> wrote:
>
> >
> > Martin v. Löwis <martin@v.loewis.de> added the comment:
> >
> > In principle, it should be possible (but perhaps not desirable, see
> below)
> > to call FreeLibrary in a CDLL's __del__. However, since this would be a
> new
> > feature, it can't go into 2.7. Patches are welcome; make sure to support
> > both FreeLIbrary and dlclose.
> >
> > There is a general issue with closing/freeing DLLs: if they are still
> > referenced somewhere (e.g. in an atexit function, a C++ virtual method
> > table, or on the call stack of another thread), then a later access to
> the
> > code will crash the interpreter. In a typical DLL today (including all
> > Python extension modules), the likelihood of crashes is close to 100%.
> For
> > that reason, it's probably not a good idea to have ctypes auto-close
> DLLs;
> > instead, it should be an opt-in mechanism.
> >
> > For most ctypes uses, closing is irrelevant, since people typically
> access
> > system libraries that are independently loaded anyway, so closing them
> > would not have any effect.
> >
> > ----------
> > nosy: +loewis
> >
> > _______________________________________
> > Python tracker <report@bugs.python.org>
> > <http://bugs.python.org/issue14597>
> > _______________________________________
> >
>
> ----------
>
> _______________________________________
> Python tracker <report@bugs.python.org>
> <http://bugs.python.org/issue14597>
> _______________________________________
>

Current default will become 3.3.  3.1 has been out for a while :)

Your thought sounds reasonable, though Martin may have further input.

Would you are to propose a patch?  Otherwise most like nothing will happen with this issue.  3.3 Beta is scheduled for mid-June, so that would be the deadline for new features.

(PS: Could you please trim your replies when replying to tracker messages?  The quoted text makes the tracker issue hard to read.)

>>For most ctypes uses, closing is irrelevant, since people typically access system libraries that are independently loaded anyway, so closing them would not have any effect.

My use case is this: I test some things, then I need to recompile my .dll without closing the interpreter (as building stuff to begin tests is quiet expensive) then reload it. I can't recompile it because Windows blocks the .dll file and gcc can't overwrite it. Deleting the reference and running gc.collect() doesn't work.
It would be very nice to have a way to do this.

Did you consider creating a copy with a random suffix?

I think I have this issue even after interpreter exits. My setup.py creates a .dll then later loads the dll for testing using ctypes. Subsequent runs of setup.py that would force rebuilding the .dll attempt to delete the old dll first if it exists, but I get permission denied.

Oddly, if I put the load library call inside a function, then, after exiting the interpreter the dll can be deleted.

Windos 7 x64
Python 2.7.10

Sorry if this is by design, a Windows feature, unrelated or the wrong issue. I search SO and these bugs, and only found answers related to unloading dll during script.

> My setup.py creates a .dll then later loads the dll for testing 
> using ctypes. Subsequent runs of setup.py that would force 
> rebuilding the .dll attempt to delete the old dll first if it 
> exists, but I get permission denied.
>
> Oddly, if I put the load library call inside a function, then, 
> after exiting the interpreter the dll can be deleted.

I'm having trouble imagining what the problem could be here. Can you attach a minimal example script that reproduces the problem?

The _ctypes extension module provides dlclose on POSIX and FreeLibrary on Windows. For the reasons already stated in this issue, ctypes should not automatically call these functions, but maybe they should be documented and imported to the ctypes module instead of being buried in the private _ctypes extension module. You could also implement them with ctypes itself, but the existing functions in the _ctypes module already set a proper exception if the call fails and properly use pointers (an HMODULE is a pointer).

As to not being able to delete a loaded DLL on Windows, a workaround that may help in some instances is to rename it to a temporary name on the same volume. This is useful for upgrading in place. If you have admin privileges you can even flag the renamed DLL to be deleted when the system reboots:

    MoveFileExW(renamed_dll_path, NULL, MOVEFILE_DELAY_UNTIL_REBOOT)

Renaming a loaded DLL succeeds because the system opens the file with delete sharing, which allows renaming (i.e. relinking) an open file. 

To verify this, let's check the associated file object that's used by the mapping for python35.dll. We can see this in a local kernel debugging session. The following is in 64-bit Windows 7.

Attach to the Python process address space:

    lkd> .tlist python.exe
     0n2296 python.exe
    lkd> !process 0n2296 0
    Searching for Process with Cid == 8f8
    Cid handle table at fffff8a003506000 with 951 entries in use

    PROCESS fffffa800712fb10
        SessionId: 0  Cid: 08f8    Peb: 7fffffdf000  ParentCid: 02b8
        DirBase: 20780d000  ObjectTable: fffff8a0189fe1a0  HandleCount:  67.
        Image: python.exe

    lkd> .process /p /r 0xfffffa800712fb10 
    Implicit process is now fffffa80`0712fb10
    Loading User Symbols
    ...................................

Get the base address for python35.dll.

    lkd> lm m python35
    start             end                 module name
    00000000`66fb0000 00000000`67393000   python35   (deferred)

Get the associated Virtual Address Descriptor (VAD).

    lkd> !vad 66fb0000
    VAD             level  start  end    commit
    fffffa800855f230 (-1)  66fb0  67392  151 Mapped  Exe
                                         EXECUTE_WRITECOPY
                                         \Program Files\Python35\python35.dll

Get the File object reference from the associated control area.

    lkd> ?? ((nt!_MMVAD *)0xfffffa800855f230)->Subsection->ControlArea->FilePointer
    struct _EX_FAST_REF
       +0x000 Object           : 0xfffffa80`093f1503 Void
       +0x000 RefCnt           : 0y0011
       +0x000 Value            : 0xfffffa80`093f1503

The low nibble in the above address is information that needs to be masked out, so the address is actually 0xfffffa80`093f1500. Verify we have the correct file object by checking the filename.
       
    lkd> ?? ((nt!_FILE_OBJECT *)0xfffffa80`093f1500)->FileName
    struct _UNICODE_STRING
     "\Program Files\Python35\python35.dll"
       +0x000 Length           : 0x48
       +0x002 MaximumLength    : 0x78
       +0x008 Buffer           : 0xfffff8a0`04b23690
                                 "\Program Files\Python35\python35.dll"

Finally, check the sharing mode. We see below that the file for a DLL mapping is loaded with read and delete sharing, but not write sharing.

    lkd> ?? ((nt!_FILE_OBJECT *)0xfffffa80`093f1500)->SharedRead
    unsigned char 0x01 ''
    lkd> ?? ((nt!_FILE_OBJECT *)0xfffffa80`093f1500)->SharedDelete
    unsigned char 0x01 ''
    lkd> ?? ((nt!_FILE_OBJECT *)0xfffffa80`093f1500)->SharedWrite
    unsigned char 0x00 ''

Thus when deleting a DLL fails, it's not due to ERROR_SHARING_VIOLATION (32). The sharing mode would actually permit deleting the file. Instead the error is ERROR_ACCESS_DENIED (5), which results from the system call NtSetInformationFile returning STATUS_CANNOT_DELETE (0xC0000121), because the file currently has a mapped image or data section. But that doesn't stop us from renaming it to get it out of the way of an in-place upgrade.

Snippets of "proprietary" code

`setup.py`
----------
# run clean or build libraries if they don't exist
if 'clean' in sys.argv:
    try:
        os.remove(os.path.join(LIB_DIR, SOLPOSAM_LIB_FILE))
        os.remove(os.path.join(LIB_DIR, SPECTRL2_LIB_FILE))
    except OSError as err:
        sys.stderr.write('%s\n' % err)

`core.py`
---------
    # load the DLL
    solposAM_dll = ctypes.cdll.LoadLibrary(SOLPOSAMDLL)
    _solposAM = solposAM_dll.solposAM

After deleting files, setup.py tries to build, then runs test_cdlls.py a unittest that contains test_solposAM() that calls solposAM() and imports core.py that contains solposAM.

Case 1:
-------
I put the load library calls inside the function solposAM().
Function calls works as expected.
I am able to delete dlls after interpreter exits and restarts using os.remove()

Case 2:
-------
I put the load library calls at the module level.
Function calls works as expected.
If I try to delete dlls after interpreter exits and restarts using os.remove() I get access denied, but I can delete them from windows explorer.

Automatically decrementing the reference count of a shared library (e.g. via WinAPI FreeLibrary or POSIX dlclose) when a CDLL instance is finalized would require significant design changes to ensure that all ctypes pointers, scalars, and aggregates that reference memory in the DLL also always reference the CDLL instance in their _objects attribute. I don't think this feature request is worth the work that involves, or the risk of crashing if some case is overlooked in the implementation. Nothing prevents a script from implementing this manually on a case by case basis.