classification
Title: dict.update() optimisation gives unexpected/invalid results when passed a subclass of DictType
Type: behavior Stage:
Components: Interpreter Core Versions: Python 2.6, Python 2.5
process
Status: closed Resolution: duplicate
Dependencies: Superseder:
Assigned To: rhettinger Nosy List: david, rhettinger
Priority: normal Keywords: patch

Created on 2008-07-28 16:24 by david, last changed 2008-07-29 00:17 by rhettinger. This issue is now closed.

Files
File name Uploaded Description Edit
dictionary.patch david, 2008-07-28 16:24 proposed patch against python 2.5.2
Messages (3)
msg70353 - (view) Author: David Farrar (david) Date: 2008-07-28 16:24
Using a python module that expected me to pass a dictionary as a
parameter to a function, I found that I was getting unexpected results
when using a class which inherits from types.DictType.

The module was passing the instance I had supplied as a parameter to
update() on a newly created dictionary. I wanted my class to present its
methods as items of the dictionary it was pretending to be. Even though
I could see that __getattribute__ had been called to get 'keys', keys()
was not called and I ended up with an empty dictionary.

Trying to find the cause of the problem, I downloaded the python source
package and found that in PyDict_Merge (in Objects/dictobject.c, line
1359), there is an optimisation that is run if PyDict_Check determines
that the parameter we passed is itself a dictionary. The optimisation
takes advantage of the fact that we know how data is stored in the
dictionary and accesses whatever it needs directly.

I had overridden keys() to provide the result I wanted but this
optimisation prevented the code from ever being run, leading to code
that seems like it should work failing with no apparent cause. The only
way I could find around this was to raise AttributeError in
__getattribute__ when getting 'keys', causing dict_update_common to call
PyDict_MergeFromSeq2 in place of PyDict_Merge.

Should the call to PyDict_Check in PyDict_Merge not be replaced with
PyDict_CheckExact ? This would keep the optimisation in place in most
cases, where we know that it is safe, still allowing keys() to be
overridden as you expect.
msg70363 - (view) Author: Raymond Hettinger (rhettinger) * (Python committer) Date: 2008-07-28 17:06
Will look back in my notes.  I believe this was previously discussed 
and rejected.
msg70373 - (view) Author: Raymond Hettinger (rhettinger) * (Python committer) Date: 2008-07-29 00:17
This is duplicate of http://bugs.python.org/issue1615701 .  The 
check_exact solution was attempted once but it broke code in a subtle 
and hard to find way so it had to be reverted.  For your use case, try 
UserDict instead of subclassing dict.
History
Date User Action Args
2008-07-29 00:17:50rhettingersetstatus: open -> closed
resolution: duplicate
messages: + msg70373
2008-07-28 17:06:17rhettingersetassignee: rhettinger
messages: + msg70363
nosy: + rhettinger
2008-07-28 16:24:26davidcreate