classification
Title: ABC caches should use weak refs
Type: resource usage Stage: patch review
Components: Library (Lib) Versions: Python 2.7
process
Status: closed Resolution: fixed
Dependencies: 8268 Superseder:
Assigned To: Nosy List: BreamoreBoy, ajaksu2, amaury.forgeotdarc, benjamin.peterson, bluag, flox, jackdied, pitrou, stutzbach
Priority: normal Keywords: patch

Created on 2008-03-31 15:12 by amaury.forgeotdarc, last changed 2011-05-13 17:43 by amaury.forgeotdarc. This issue is now closed.

Files
File name Uploaded Description Edit
abc_leak.patch stutzbach, 2010-03-31 23:44 Patch to fix the bug
leak_test.patch stutzbach, 2010-03-31 23:45 Patch to add a test case
leak_test2.patch stutzbach, 2010-08-18 15:27 Patch to add a test case, version 2
Messages (28)
msg64784 - (view) Author: Amaury Forgeot d'Arc (amaury.forgeotdarc) * (Python committer) Date: 2008-03-31 15:12
The following function seems to 8 references each time it is run:

import io, gc
def f():
   class C: pass
   c=C()
   assert isinstance(c, io.StringIO) is False
   gc.collect();gc.collect();gc.collect()


This is because io.StringIO._abc_negative_cache contains a strong
reference to each "class C", as soon as isinstance() is called. These
are never released.

Python3.0 does use WeakSet for these caches, and does not leak.
This is the (deep) reason why test_io.IOTest.test_destructor() leaks in
the following report:
http://mail.python.org/pipermail/python-checkins/2008-March/067918.html
(The test derives from io.FileIO, and it is the call to the base class'
method which calls isinstance() and put the class in the cache)
msg87720 - (view) Author: Daniel Diniz (ajaksu2) Date: 2009-05-13 22:54
Confirmed on release26-maint and trunk.
msg101907 - (view) Author: Daniel Stutzbach (stutzbach) (Python committer) Date: 2010-03-29 22:21
Now that WeakSet has been backported to trunk, I've backported the fix for this reference-leak (patch with test case attached).

However, after making the patch, I discovered that old-style classes are not weak-referenceable.  Consequently, the patch is not suitable for commiting.  

If old-style classes can be made weak-referenceable, then the patch should work.  If not, then this bug is essentially impossible to solve in 2.x unless the ABC cache is done away with entirely.

Other notes:

Since abc.py is imported during bootstrapping before paths are set up, I needed to add _weakref as a built-in module.  (It's already a built-in module in the py3k branch.)

Since the patch tweaks Modules/Setup.dist, you need to "make distclean" and rerun ./configure after applying the patch.

Also, on unpatched trunk the example posted by Amaury no longer seems to trigger the issue.  However, the example I posted in Issue 8022 still triggers the leak.  I used that as the basis for the test case in the patch.
msg101960 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2010-03-30 23:35
Hmm, Benjamin pointed out that ABCs only support new-style classes, so old-style classes could be detected early instead of being added to the _abc_negative_cache.
msg101961 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2010-03-30 23:38
By the way, Daniel, your patch doesn't look right.
First, you shouldn't need all the sortedlist/sortedset hierarchy.
Second, len(gc.get_objects()) is a truly horrible way of checking the classes have been destroyed. Just take a weakref to the class before deleting it, and check that calling the weakref returns None.
(besides, we also have a refleak-detection run (regrtest -R) which can detect leaks even if you don't check them explicitly in your tests)
msg101962 - (view) Author: Daniel Stutzbach (stutzbach) (Python committer) Date: 2010-03-30 23:48
I hadn't realized that old style classes didn't support ABCs.  That certainly simplifies things!  I'm working on a new patch.
msg101963 - (view) Author: Daniel Stutzbach (stutzbach) (Python committer) Date: 2010-03-30 23:58
Are you sure the old-style classes don't support ABCs?

ABCTestCase.validate_isinstance in Lib/test/test_collection.py specifically tests that both new-style and old-style classes work, unless I'm reading it wrong.

(and those tests fail if I make ABCMeta.__instancecheck__ and ABCMeta.__subclasscheck__ always return False for old-style classes)
msg101964 - (view) Author: Benjamin Peterson (benjamin.peterson) * (Python committer) Date: 2010-03-31 00:02
In those cases, it's because __subclasscheck__ is overridden. You can't register a old-style class.
msg101965 - (view) Author: Florent Xicluna (flox) * (Python committer) Date: 2010-03-31 00:06
> ABCTestCase.validate_isinstance ... specifically tests that
> both new-style and old-style classes work...

I fixed some parts with issue #7624, and changeset r78800.
msg101967 - (view) Author: Daniel Stutzbach (stutzbach) (Python committer) Date: 2010-03-31 00:36
You can't register an old-style class, but many ABCs support duck-typing by implementing __subclasshook__.  ABCMeta caches those results and stores a reference to old-style classes, sometimes in _abc_cache and sometimes in _abc_negative_cache.  It can't simply return False.

I guess that leaves two options:

1) Make old-style classes weak-referenceable and cache the results of __subclasshook__.
2) Refuse to cache old-style classes and call __subclasshook__ every time for old-style classes.

Python 2.7a4+ (trunk:79493, Mar 30 2010, 19:19:13)
>>> class old_iterable_class:
...   def __iter__(self):
...     pass
...
>>> import collections
>>> issubclass(old_iterable_class, collections.Iterable)
True
msg102036 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2010-03-31 21:39
> Make old-style classes weak-referenceable

Now done (r79535).
msg102037 - (view) Author: Daniel Stutzbach (stutzbach) (Python committer) Date: 2010-03-31 21:41
Cool!  I will revise the patch based on your comments about my test case.
msg102047 - (view) Author: Daniel Stutzbach (stutzbach) (Python committer) Date: 2010-03-31 23:45
New patches uploaded.  I separated out the patch to add the test case, to make it easier to test before and after applying the fix.
msg104287 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2010-04-26 23:21
Someone with a better knowledge of ABCs than me should probably do a final review of this.
msg104290 - (view) Author: Daniel Stutzbach (stutzbach) (Python committer) Date: 2010-04-26 23:51
Someone with appropriate permissions want to update the "Stage"?
msg105931 - (view) Author: Daniel Stutzbach (stutzbach) (Python committer) Date: 2010-05-17 19:10
Antoine, do you have a suggestion for someone with with better knowledge of ABCs to do the final review, so that I may very politely pester them? ;-)
msg105934 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2010-05-17 20:07
> Antoine, do you have a suggestion for someone with with better 
> knowledge of ABCs to do the final review, so that I may very politely 
> pester them? ;-)

I guess Benjamin could.
msg110801 - (view) Author: Mark Lawrence (BreamoreBoy) * Date: 2010-07-19 19:22
Patched the unit test, then ran the test before applying the fix which failed, after applying the fix the test ran successfully.  Tested on Windows Vista 32 bit against 2.7 maintainance release.  The patches are short and sweet, I see no reason why they can't go forward.
msg112983 - (view) Author: Mark Lawrence (BreamoreBoy) * Date: 2010-08-05 14:21
Can a committer take a look at this please.
msg112991 - (view) Author: Jack Diederich (jackdied) * (Python committer) Date: 2010-08-05 15:51
This is a change in the codepath for instances that don't have __class__ defined.
         subclass = getattr(instance, '__class__', None)
-        if subclass in cls._abc_cache:
+        if subclass is not None and subclass in cls._abc_cache:

I think the same thing happens in either case (from visual inspection of the code) but I'd rather not change it if we don't need to.
msg113485 - (view) Author: Daniel Stutzbach (stutzbach) (Python committer) Date: 2010-08-09 22:02
Jack,

The change is necessary because "None in WeakSet()" would throw an exception.
msg114093 - (view) Author: Benjamin Peterson (benjamin.peterson) * (Python committer) Date: 2010-08-17 01:35
Some comments:
- The test should be in test_abc.py and should probably not use a collections class.
- Use test_support.gc_collect().
- What's the __len__() stuff for?
msg114226 - (view) Author: Daniel Stutzbach (stutzbach) (Python committer) Date: 2010-08-18 14:42
Benjamin,

Thanks for the feedback.  Since you only commented on the test case, may I assume that the fix itself looked good to you?

I will work on revising the test case based on your comments.  Since I ran into the bug while working with the ABCs in the collections module, that biased my thinking when writing the test.  :-)  

I needed to define__len__ because it's an abstract method in the ABC I used in the test (collections.Sized).  I found that overriding it again in a sub-sub-class and calling it were necessary to trigger all of the ABC machinery leading to the leak.
msg114233 - (view) Author: Daniel Stutzbach (stutzbach) (Python committer) Date: 2010-08-18 15:27
Attached is a new test case, based on Benjamin's comments.
msg114476 - (view) Author: Benjamin Peterson (benjamin.peterson) * (Python committer) Date: 2010-08-21 03:03
r84230
msg114477 - (view) Author: Daniel Stutzbach (stutzbach) (Python committer) Date: 2010-08-21 03:05
Thanks! :-)
msg135931 - (view) Author: H. (bluag) Date: 2011-05-13 17:15
ImportError: No module named _weakrefset
Here are some references while i was trying to install Pinax framework.

http://paste.pound-python.org/show/6536/

And i saw that the _weakrefset.py is not included in the package. So I have copied from Python's source with version : 3.1.* to my d:\sosyal\ folder. and everything works fine.
msg135935 - (view) Author: Amaury Forgeot d'Arc (amaury.forgeotdarc) * (Python committer) Date: 2011-05-13 17:43
bluag: the script you run contains a list of some modules required to start Python
(I found a copy here: https://github.com/pinax/pinax/raw/master/scripts/pinax-boot.py )
This script is obviously out of date wrt the new version of Python. Please report this to the pinax project!
History
Date User Action Args
2011-05-13 17:43:32amaury.forgeotdarcsetmessages: + msg135935
2011-05-13 17:15:50bluagsetnosy: + bluag
messages: + msg135931
2010-08-21 03:05:09stutzbachsetmessages: + msg114477
2010-08-21 03:03:50benjamin.petersonsetstatus: open -> closed
resolution: fixed
messages: + msg114476
2010-08-18 15:27:58stutzbachsetfiles: + leak_test2.patch

messages: + msg114233
2010-08-18 14:45:17stutzbachsetnosy: amaury.forgeotdarc, pitrou, jackdied, ajaksu2, benjamin.peterson, stutzbach, flox, BreamoreBoy
components: + Library (Lib), - Interpreter Core
versions: - Python 2.6
2010-08-18 14:42:01stutzbachsetmessages: + msg114226
2010-08-17 01:35:36benjamin.petersonsetkeywords: - 26backport

messages: + msg114093
2010-08-09 22:02:00stutzbachsetmessages: + msg113485
2010-08-05 15:51:07jackdiedsetnosy: + jackdied
messages: + msg112991
2010-08-05 14:21:30BreamoreBoysetmessages: + msg112983
2010-07-19 19:22:01BreamoreBoysetnosy: + BreamoreBoy
messages: + msg110801
2010-05-17 20:07:42pitrousetmessages: + msg105934
2010-05-17 19:10:16stutzbachsetmessages: + msg105931
2010-04-26 23:55:55michael.foordsetstage: test needed -> patch review
2010-04-26 23:51:51stutzbachsetmessages: + msg104290
versions: + Python 2.7
2010-04-26 23:21:24pitrousetmessages: + msg104287
2010-03-31 23:45:12stutzbachsetfiles: - abc_leak.patch
2010-03-31 23:45:04stutzbachsetfiles: + leak_test.patch

messages: + msg102047
2010-03-31 23:44:15stutzbachsetfiles: + abc_leak.patch
2010-03-31 21:41:43stutzbachsetmessages: + msg102037
2010-03-31 21:39:32pitrousetmessages: + msg102036
2010-03-31 00:36:34stutzbachsetmessages: + msg101967
2010-03-31 00:06:09floxsetmessages: + msg101965
2010-03-31 00:02:08benjamin.petersonsetnosy: + benjamin.peterson
messages: + msg101964
2010-03-30 23:58:07stutzbachsetmessages: + msg101963
2010-03-30 23:48:13stutzbachsetmessages: + msg101962
2010-03-30 23:38:19pitrousetmessages: + msg101961
2010-03-30 23:35:49pitrousetmessages: + msg101960
2010-03-30 22:58:36pitrousetdependencies: + Make old-style classes weak referenceable
2010-03-29 22:21:34stutzbachsetfiles: + abc_leak.patch

nosy: + stutzbach
messages: + msg101907

keywords: + patch
2010-02-26 10:15:48floxsetnosy: + flox
2010-02-26 02:46:08benjamin.petersonlinkissue8022 superseder
2009-05-13 22:54:08ajaksu2setpriority: normal

nosy: + ajaksu2
messages: + msg87720

type: resource usage
stage: test needed
2008-05-07 11:35:15pitrousetnosy: + pitrou
2008-03-31 15:12:29amaury.forgeotdarccreate