If I load a dll in ctypes, then delete that loaded DLL instance, the DLL is not unloaded until the script finishes and exits.

I'm trying to write some unit tests in python to exercise that DLL where each test case loads a DLL, does some work, then unloads the DLL.  Unfortunately the DLL only gets unloaded when the unit tests finish.

I've tried forcing the garbage collector to run to get the DLL to unload.  It did nothing...

# load the DLL
parser_dll = CDLL(dllpath)

# do some work here

# 'unload' the dll (or as close as I can get it to it)
if (parser_dll):
    del parser_dll

I should mention also, that this is mostly an issue for me on Win7 x64.  It
does behave 'slightly' better on WinXP x86.

(I have the 64-bit version of python installed on Win7 x64 & the 32-bit
version installed on WinXP)

thanks,
Pat.

On 16 April 2012 14:09, Pat Lynch <report@bugs.python.org> wrote:

>
> New submission from Pat Lynch <plynch76@gmail.com>:
>
> If I load a dll in ctypes, then delete that loaded DLL instance, the DLL
> is not unloaded until the script finishes and exits.
>
> I'm trying to write some unit tests in python to exercise that DLL where
> each test case loads a DLL, does some work, then unloads the DLL.
>  Unfortunately the DLL only gets unloaded when the unit tests finish.
>
> I've tried forcing the garbage collector to run to get the DLL to unload.
>  It did nothing...
>
> # load the DLL
> parser_dll = CDLL(dllpath)
>
> # do some work here
>
> # 'unload' the dll (or as close as I can get it to it)
> if (parser_dll):
>    del parser_dll
>
> ----------
> components: ctypes
> messages: 158433
> nosy: plynch76
> priority: normal
> severity: normal
> status: open
> title: Cannot unload dll in ctypes until script exits
> type: enhancement
> versions: Python 2.7
>
> _______________________________________
> Python tracker <report@bugs.python.org>
> <http://bugs.python.org/issue14597>
> _______________________________________
>

In general it is difficult to impossible to get Python2 to unload modules before the interpreter shuts down.  See issue 9072.  I'm not savvy enough with the C stuff to know if the fact that you loaded it via ctypes changes anything, but I doubt it.

Note that the implication of that issue is that if you could move to Python3 there might be a way to do it, but that would indeed be an enhancement as there is no direct support for it yet.

thanks for the very quick response.

Since LoadLibrary is called in the constructor, why can't FreeLibrary be
called in the destructor?  or at least expose a function to unload that
calls FreeLibrary?

http://msdn.microsoft.com/en-us/library/windows/desktop/ms683152%28v=vs.85%29.aspx

thanks again,
Pat.

On 16 April 2012 14:30, R. David Murray <report@bugs.python.org> wrote:

>
> R. David Murray <rdmurray@bitdance.com> added the comment:
>
> In general it is difficult to impossible to get Python2 to unload modules
> before the interpreter shuts down.  See issue 9072.  I'm not savvy enough
> with the C stuff to know if the fact that you loaded it via ctypes changes
> anything, but I doubt it.
>
> Note that the implication of that issue is that if you could move to
> Python3 there might be a way to do it, but that would indeed be an
> enhancement as there is no direct support for it yet.
>
> ----------
> nosy: +r.david.murray
> versions: +Python 3.3 -Python 2.7
>
> _______________________________________
> Python tracker <report@bugs.python.org>
> <http://bugs.python.org/issue14597>
> _______________________________________
>

In principle, it should be possible (but perhaps not desirable, see below) to call FreeLibrary in a CDLL's __del__. However, since this would be a new feature, it can't go into 2.7. Patches are welcome; make sure to support both FreeLIbrary and dlclose.

There is a general issue with closing/freeing DLLs: if they are still referenced somewhere (e.g. in an atexit function, a C++ virtual method table, or on the call stack of another thread), then a later access to the code will crash the interpreter. In a typical DLL today (including all Python extension modules), the likelihood of crashes is close to 100%. For that reason, it's probably not a good idea to have ctypes auto-close DLLs; instead, it should be an opt-in mechanism.

For most ctypes uses, closing is irrelevant, since people typically access system libraries that are independently loaded anyway, so closing them would not have any effect.

ok, that's fair enough if most usage of ctypes is from people accessing
system libraries :)

I wouldn't have thought my usage was that weird though (given the strength
of using python for unit testing).

In local tests, adding a function CDLL::ForceUnloadDll  (which just calls
FreeLibrary(self._handle)) seems to work well.  I haven't done intensive
testing though at this point.  I could be missing something though.

thanks,
Pat.

On 16 April 2012 14:45, Martin v. Löwis <report@bugs.python.org> wrote:

>
> Martin v. Löwis <martin@v.loewis.de> added the comment:
>
> In principle, it should be possible (but perhaps not desirable, see below)
> to call FreeLibrary in a CDLL's __del__. However, since this would be a new
> feature, it can't go into 2.7. Patches are welcome; make sure to support
> both FreeLIbrary and dlclose.
>
> There is a general issue with closing/freeing DLLs: if they are still
> referenced somewhere (e.g. in an atexit function, a C++ virtual method
> table, or on the call stack of another thread), then a later access to the
> code will crash the interpreter. In a typical DLL today (including all
> Python extension modules), the likelihood of crashes is close to 100%. For
> that reason, it's probably not a good idea to have ctypes auto-close DLLs;
> instead, it should be an opt-in mechanism.
>
> For most ctypes uses, closing is irrelevant, since people typically access
> system libraries that are independently loaded anyway, so closing them
> would not have any effect.
>
> ----------
> nosy: +loewis
>
> _______________________________________
> Python tracker <report@bugs.python.org>
> <http://bugs.python.org/issue14597>
> _______________________________________
>

Just to update:-

I've run this pretty extensively on multiple systems (XP x86 & Win7 64-bit)
and it appears to behave as expected (haven't checked it on Linux).  I have
that code being called in 100s of unit tests.

For python 3.1, would it make sense to add it as a ForceUnload function??
- for safety bail out if handle was not None when passed into the
constructor?  i.e. if somebody has accessed an independently loaded DLL,
they will pass in the handle when constructing the CDLL object.  Disallow
ForceUnload in that case.  ForceUnload will only be allowed in cases where
we created that type by passing in the path to the DLL.

I'll be using this code as a local patch, so no rush to put it into 3.1 etc.

thanks for all the info - much appreciated :)

Pat.

On 16 April 2012 15:04, Pat Lynch <report@bugs.python.org> wrote:

>
> Pat Lynch <plynch76@gmail.com> added the comment:
>
> ok, that's fair enough if most usage of ctypes is from people accessing
> system libraries :)
>
> I wouldn't have thought my usage was that weird though (given the strength
> of using python for unit testing).
>
> In local tests, adding a function CDLL::ForceUnloadDll  (which just calls
> FreeLibrary(self._handle)) seems to work well.  I haven't done intensive
> testing though at this point.  I could be missing something though.
>
> thanks,
> Pat.
>
> On 16 April 2012 14:45, Martin v. Löwis <report@bugs.python.org> wrote:
>
> >
> > Martin v. Löwis <martin@v.loewis.de> added the comment:
> >
> > In principle, it should be possible (but perhaps not desirable, see
> below)
> > to call FreeLibrary in a CDLL's __del__. However, since this would be a
> new
> > feature, it can't go into 2.7. Patches are welcome; make sure to support
> > both FreeLIbrary and dlclose.
> >
> > There is a general issue with closing/freeing DLLs: if they are still
> > referenced somewhere (e.g. in an atexit function, a C++ virtual method
> > table, or on the call stack of another thread), then a later access to
> the
> > code will crash the interpreter. In a typical DLL today (including all
> > Python extension modules), the likelihood of crashes is close to 100%.
> For
> > that reason, it's probably not a good idea to have ctypes auto-close
> DLLs;
> > instead, it should be an opt-in mechanism.
> >
> > For most ctypes uses, closing is irrelevant, since people typically
> access
> > system libraries that are independently loaded anyway, so closing them
> > would not have any effect.
> >
> > ----------
> > nosy: +loewis
> >
> > _______________________________________
> > Python tracker <report@bugs.python.org>
> > <http://bugs.python.org/issue14597>
> > _______________________________________
> >
>
> ----------
>
> _______________________________________
> Python tracker <report@bugs.python.org>
> <http://bugs.python.org/issue14597>
> _______________________________________
>

Current default will become 3.3.  3.1 has been out for a while :)

Your thought sounds reasonable, though Martin may have further input.

Would you are to propose a patch?  Otherwise most like nothing will happen with this issue.  3.3 Beta is scheduled for mid-June, so that would be the deadline for new features.

(PS: Could you please trim your replies when replying to tracker messages?  The quoted text makes the tracker issue hard to read.)