classification
Title: Reference cycle when importing ctypes
Type: resource usage Stage:
Components: ctypes Versions: Python 3.3
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: amaury.forgeotdarc, meador.inge, poq, python-dev, terry.reedy, vladris
Priority: normal Keywords: patch

Created on 2011-05-21 22:39 by poq, last changed 2012-06-28 14:53 by meador.inge.

Files
File name Uploaded Description Edit
ctypes-leak.patch poq, 2012-02-03 20:10 review
Messages (9)
msg136485 - (view) Author: (poq) Date: 2011-05-21 22:39
When importing ctypes after gc.set_debug(gc.DEBUG_LEAK), the garbage collector finds a 'c_int_Array_3' class and some related objects.

The class is created in ctypes/_endian.py:
_array_type = type(c_int * 3)

It seems that this could be avoided with:
_array_type = type(Array)

Of course, I realize this is not a bug because normally it will just get collected. It is just an extremely minor annoyance because this is currently the only thing still found by DEBUG_LEAK for my program ;)
msg137148 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2011-05-28 19:50
If you are able to rebuild Python, have you tried running the ctypes test after rebuilding with this change? And, does the test cover the internal uses of _array_type?
msg137371 - (view) Author: (poq) Date: 2011-05-31 17:07
Tests succeed with this change.

There is only one use of _array_type, which is in the same module. This use is presumably tested, because the test fails if I change the line to _array_type = type(Structure).

In fact, everything must behave exactly the same after this change, because the two values are identical:

>>> from ctypes import *
>>> type(c_int * 3) is type(Array)
True
msg137379 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2011-05-31 18:25
Thank you for the test and explanation. There currently is no specific cytpes maintainer. But from what you have said, I might feel comfortable enough applying this, if no one else does, when I have the necessary setup on a new machine.
msg140098 - (view) Author: Vlad Riscutia (vladris) Date: 2011-07-11 04:16
I ran full test suit after making the _array_type = type(Array) change and everything passes.

I also took a look at this and found additional leak. gc shows this as garbage:

[(<class '_ctypes._SimpleCData'>,), <class 'ctypes.c_longdouble'>, <attribute '_
_dict__' of 'c_longdouble' objects>, <attribute '__weakref__' of 'c_longdouble'
objects>, (<class 'ctypes.c_longdouble'>, <class '_ctypes._SimpleCData'>, <class
 '_ctypes._CData'>, <class 'object'>), {'__dict__': <attribute '__dict__' of 'c_
longdouble' objects>, '_type_': 'g', '__module__': 'ctypes', '__weakref__': <att
ribute '__weakref__' of 'c_longdouble' objects>, '__doc__': None}]

This is all caused by these lines in ctypes __init__.py:

class c_longdouble(_SimpleCData):
    _type_ = "g"
if sizeof(c_longdouble) == sizeof(c_double):
    c_longdouble = c_double

For me sizeof(c_longdouble) == sizeof(c_double) (I believe MS compiler always does this) but when we assign c_longdouble = c_double, there is a leak. I removed the alias lines:

if sizeof(c_longdouble) == sizeof(c_double):
    c_longdouble = c_double

And the leak was gone. Looks like changing c_longdouble after declaring it causes a leak. Below for similar aliasing of longlong types, we have this:

if _calcsize("l") == _calcsize("q"):
    # if long and long long have the same size, make c_longlong an alias for c_long
    c_longlong = c_long
    c_ulonglong = c_ulong
else:
    class c_longlong(_SimpleCData):
        _type_ = "q"
    _check_size(c_longlong)

    class c_ulonglong(_SimpleCData):
        _type_ = "Q"

This avoids declaring c_longlong and c_ulonglong as class if not needed to. The problem is _calcsize("g") causes an error because "g" is used as long double througout ctypes but _calcsize is function from _struct.c, where "g" (long double) is not defined. Not sure why it isn't...

So in short:
As far as I can tell _array_type = type(Array) doesn't break anything
Looks like we have another leak in ctypes (which isn't a big deal)
We have elegant fix for the leak once _struct.c will support long double
msg152552 - (view) Author: (poq) Date: 2012-02-03 20:10
I've attached a patch for the _array_type change.

The long double fix is probably dependent on PEP3118 (#3132).
msg152661 - (view) Author: Roundup Robot (python-dev) Date: 2012-02-05 06:29
New changeset 205da7a19a78 by Meador Inge in branch '3.2':
Issue #12142: Fixed reference cycle when importing ctypes
http://hg.python.org/cpython/rev/205da7a19a78

New changeset b228d9da8bd3 by Meador Inge in branch 'default':
Issue #12142: Fixed reference cycle when importing ctypes
http://hg.python.org/cpython/rev/b228d9da8bd3

New changeset 7cdbf627f958 by Meador Inge in branch '2.7':
Issue #12142: Fixed reference cycle when importing ctypes
http://hg.python.org/cpython/rev/7cdbf627f958
msg164258 - (view) Author: Amaury Forgeot d'Arc (amaury.forgeotdarc) * (Python committer) Date: 2012-06-28 14:28
Meador, can we close this issue?
msg164262 - (view) Author: Meador Inge (meador.inge) * (Python committer) Date: 2012-06-28 14:53
> Meador, can we close this issue?

I wanted to keep it open until the 'long double' problem is fixed as well.
History
Date User Action Args
2012-06-28 14:53:45meador.ingesetmessages: + msg164262
2012-06-28 14:28:43amaury.forgeotdarcsetnosy: + amaury.forgeotdarc
messages: + msg164258
2012-02-05 06:29:10python-devsetnosy: + python-dev
messages: + msg152661
2012-02-03 20:20:01pitrousetnosy: + meador.inge
2012-02-03 20:10:36poqsetfiles: + ctypes-leak.patch
keywords: + patch
messages: + msg152552
2011-07-11 04:16:13vladrissetnosy: + vladris
messages: + msg140098
2011-05-31 18:25:20terry.reedysetmessages: + msg137379
2011-05-31 17:07:51poqsetmessages: + msg137371
2011-05-28 19:50:49terry.reedysetnosy: + terry.reedy
messages: + msg137148
2011-05-21 22:41:11poqsettitle: eference cycle when importing ctypes -> Reference cycle when importing ctypes
2011-05-21 22:41:04poqsettitle: Circular reference when importing ctypes -> eference cycle when importing ctypes
2011-05-21 22:39:48poqcreate