classification
Title: Weak*Dictionary KeyErrors during callbacks
Type: Stage:
Components: Library (Lib) Versions: Python 2.7
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: Nils.Bruin, vbraun
Priority: normal Keywords: patch

Created on 2013-04-22 16:41 by Nils.Bruin, last changed 2013-05-02 00:35 by Nils.Bruin.

Files
File name Uploaded Description Edit
17816_custom_clear.patch Nils.Bruin, 2013-05-02 00:35
Messages (5)
msg187570 - (view) Author: Nils Bruin (Nils.Bruin) Date: 2013-04-22 16:41
The following program is a little dependent on memory layout but will usually generate lots of

    Exception KeyError: (A(9996),) in <function remove at 0xa47050> ignored

messages in Python 2.7.

    import weakref

    class A(object):
        def __init__(self,n):
            self.n=n
        def __repr__(self):
            return "A(%d)"%self.n

    def mess(n):
        D=weakref.WeakValueDictionary()
        L=[A(i) for i in range(n)]
        for i in range(n-1):
            j=(i+10)%n
            D[L[i]]=L[j]
        return D

    D=mess(10000)
    D.clear()

The reason is that on D.clear() all entries are removed from D before actually deleting those entries. Once the entries are deleted one-by-one, sometimes the removal of a key will result in deallocation of that key, which may be a not-yet-deleted ex-value of the dictionary as well. The callback triggers on the weakref, but the dict itself was already emptied, so nothing is found.

I've checked and on Python 3.2.3 this problem does not seem to occur. I haven't checked the Python source to see how Python 3 behaves differently and whether that behaviour would be easy to backport to fix this bug in 2.7.
msg187572 - (view) Author: Volker Braun (vbraun) Date: 2013-04-22 17:06
This is http://bugs.python.org/issue7105. The patch from there could easily be backported, I think.
msg187578 - (view) Author: Nils Bruin (Nils.Bruin) Date: 2013-04-22 18:23
Have you tried if the fix at issue7105 solves the problem? I don't see the patch there introduce a `clear` method override for WeakValueDictionary or WeakKeyDictionary. The one for WeakSet still calls self.data.clear(), which for dictionaries would still result in the problem in this ticket (but not for WeakSet, because clearing a WeakSet shouldn't decref anything other than the weak references stored in the underlying set).
msg187603 - (view) Author: Nils Bruin (Nils.Bruin) Date: 2013-04-23 01:50
I think the difference in behaviour between Py3 and Py2 is coming from:

http://hg.python.org/cpython/file/a26df2d03989/Objects/dictobject.c#l1275

which first clears all values before removing any keys. For a WeakValueDictionary that means all the weakrefs are neutralized before the can be activated. I don't quite understand how Py3 manages to avoid problems for a WeakKeyDictionary, but apparently it does.
msg188249 - (view) Author: Nils Bruin (Nils.Bruin) Date: 2013-05-02 00:35
One solution is to patch both WeakValueDictionary and WeakKeyDictionary with their own clear methods where we first store the strong links (to keys, resp. values) in a list, then clear the underlying dictionaries (this will now trigger the deletion of the weakrefs, so all callbacks are neutralized), and then delete the list. It does use more storage that way, but it gets rid of the ignored key errors.

This is a different problem from issue7105, which deals with the (much more complicated) scenario of avoiding dictionary reshaping due to GC when iterators are still (potentially) active.
History
Date User Action Args
2013-05-02 00:35:53Nils.Bruinsetfiles: + 17816_custom_clear.patch
keywords: + patch
messages: + msg188249
2013-04-23 01:50:11Nils.Bruinsetmessages: + msg187603
2013-04-22 18:23:53Nils.Bruinsetmessages: + msg187578
2013-04-22 17:06:31vbraunsetnosy: + vbraun
messages: + msg187572
2013-04-22 16:41:04Nils.Bruincreate