classification
Title: pprint.pprint raises TypeError on dictionaries with user-defined types as keys
Type: behavior Stage: resolved
Components: Library (Lib) Versions: Python 3.2, Python 3.3
process
Status: closed Resolution: fixed
Dependencies: Superseder: pprint._safe_key is not always safe enough
View: 14998
Assigned To: Nosy List: amaury.forgeotdarc, anton.barkovsky, arno, flox, giampaolo.rodola, python-dev, rbp, rhettinger
Priority: normal Keywords: easy, patch

Created on 2010-10-02 20:35 by arno, last changed 2012-07-21 09:27 by flox. This issue is now closed.

Files
File name Uploaded Description Edit
pprint_fix.patch anton.barkovsky, 2012-07-18 23:24 Fix sorting of class objects and add tests for the issue review
Messages (8)
msg117895 - (view) Author: Arnaud Delobelle (arno) Date: 2010-10-02 20:35
The pprint function in the python 3.1 pprint module fails when
printing a dictionary containing more than one item and with one item
being a user-defined type.  It seems pprint tries to sort the keys but
fails, (maybe because calling __lt__ on a user-defined type doesn't
bind its first argument to 'self'?  Looking into pprint.py would
probably yield the answer).

This seems related to issue 3976 but it was fixed in r76389 and
r76390.  My example below fails in r79147.  I'm not in a position to check more recent revisions right now.

In Python 2.6, the following works fine:

Python 2.6.5 (r265:79063, Apr 16 2010, 13:09:56) 
[GCC 4.4.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> from pprint import pprint
>>> class A(object): pass
... 
>>> pprint({A:1, 1:2})
{1: 2, <class __main__.A at 0xb77dc47c>: 1}


But in Python 3.1, it fails with the error below:

Python 3.1.2 (r312:79147, Apr 15 2010, 12:35:07) 
[GCC 4.4.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> from pprint import pprint
>>> class A: pass
... 
>>> pprint({A:1, 1:2})
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python3.1/pprint.py", line 55, in pprint
    printer.pprint(object)
  File "/usr/lib/python3.1/pprint.py", line 132, in pprint
    self._format(object, self._stream, 0, 0, {}, 0)
  File "/usr/lib/python3.1/pprint.py", line 155, in _format
    rep = self._repr(object, context, level - 1)
  File "/usr/lib/python3.1/pprint.py", line 242, in _repr
    self._depth, level)
  File "/usr/lib/python3.1/pprint.py", line 254, in format
    return _safe_repr(object, context, maxlevels, level)
  File "/usr/lib/python3.1/pprint.py", line 296, in _safe_repr
    items = sorted(object.items(), key=_safe_tuple)
  File "/usr/lib/python3.1/pprint.py", line 89, in __lt__
    rv = self.obj.__lt__(other.obj)
TypeError: expected 1 arguments, got 0
msg118209 - (view) Author: Rodrigo Bernardo Pimentel (rbp) (Python committer) Date: 2010-10-08 17:58
FWIW, the problem still occurs on the most recent release31-maint checkout (as of r85323), and does not happen on py3k (3.2a2).
msg118214 - (view) Author: Rodrigo Bernardo Pimentel (rbp) (Python committer) Date: 2010-10-08 18:51
If I'm understanding this correctly, this fails on 3.1 and not (although, actually, it does) on py3k/3.2 because:

* pprint._safe_key.__lt__ checks "rv = self.obj.__lt__(other.obj)" and falls back to id comparison if rv is NotImplemented

* If the object passed to _safe_key is a class, self.obj.__lt__ will expect *self* as well as the other object. Therefore the verification above fails with "TypeError: expected 1 arguments, got 0". You can see that pprint works with an instance:

>>> pprint.pprint({A(): 1, 1: 2})
{<__main__.A object at 0x8594d4c>: 1, 1: 2}

* Finally, this works on py3k *for your example* because, for some reason, on py3k the comparison is being based on the 1 key. That is, the comparison on _safe_key.__lt__ happens to be 1.__lt__(A), instead of A.__lt__(1). Perhaps hashing changed after the 3.1 release?

Anyway, py3k still fails when you force the comparison to happen on the class:

>>> class B(object): pass
... 
>>> pprint.pprint({A: 1, B: 2})
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/rbp/python/dev/py3k/Lib/pprint.py", line 55, in pprint
    printer.pprint(object)
  File "/home/rbp/python/dev/py3k/Lib/pprint.py", line 132, in pprint
    self._format(object, self._stream, 0, 0, {}, 0)
  File "/home/rbp/python/dev/py3k/Lib/pprint.py", line 155, in _format
    rep = self._repr(object, context, level - 1)
  File "/home/rbp/python/dev/py3k/Lib/pprint.py", line 245, in _repr
    self._depth, level)
  File "/home/rbp/python/dev/py3k/Lib/pprint.py", line 257, in format
    return _safe_repr(object, context, maxlevels, level)
  File "/home/rbp/python/dev/py3k/Lib/pprint.py", line 299, in _safe_repr
    items = sorted(object.items(), key=_safe_tuple)
  File "/home/rbp/python/dev/py3k/Lib/pprint.py", line 89, in __lt__
    rv = self.obj.__lt__(other.obj)
TypeError: expected 1 arguments, got 0
>>> 

So, basically, the fix on issue 3976 does't (always) work when there are classes as dict keys. I think switching from

rv = self.obj.__lt__(other.obj)
if rv is NotImplemented:

to something like

try:
    rv = self.obj < other.obj
except TypeError:
    rv = (str(type(self.obj)), id(self.obj)) < \
    (str(type(other.obj)), id(other.obj))

solves this. Or we can check first whether self.obj is a 'type', but I think it gets convoluted.

If anyone can confirm that that's the way to go, I can produce one (though it's a trivial one). Raymond?
msg118227 - (view) Author: Amaury Forgeot d'Arc (amaury.forgeotdarc) * (Python committer) Date: 2010-10-08 21:38
A simpler change would to replace:
    rv = self.obj.__lt__(other.obj)
with
    rv = type(self.obj).__lt__(self.obj, other.obj)
msg165790 - (view) Author: Florent Xicluna (flox) * (Python committer) Date: 2012-07-18 17:26
Confirmed on python 3.2 and 3.3.
msg165817 - (view) Author: Anton Barkovsky (anton.barkovsky) * Date: 2012-07-18 23:24
Here's a patch with fix and tests.

Note that class objects are not comparable and have the same type so they
fall back on sorting by id. Should we sort them by name as a special case instead?
msg165999 - (view) Author: Roundup Robot (python-dev) Date: 2012-07-21 09:23
New changeset 03cda5360dc6 by Florent Xicluna in branch '3.2':
Issues #10017 and #14998: Fix TypeError using pprint on dictionaries with unorderable key.
http://hg.python.org/cpython/rev/03cda5360dc6

New changeset 4d0dcfbdf45b by Florent Xicluna in branch 'default':
Issues #10017 and #14998: Fix TypeError using pprint on dictionaries with unorderable key.
http://hg.python.org/cpython/rev/4d0dcfbdf45b
msg166000 - (view) Author: Florent Xicluna (flox) * (Python committer) Date: 2012-07-21 09:27
Thanks for this patch.
I've reviewed the issue and merged the patch of issue #14998 which fixes both cases.
History
Date User Action Args
2012-07-21 09:27:04floxsetstatus: open -> closed
superseder: pprint._safe_key is not always safe enough
messages: + msg166000

resolution: fixed
stage: needs patch -> resolved
2012-07-21 09:23:13python-devsetnosy: + python-dev
messages: + msg165999
2012-07-18 23:24:53anton.barkovskysetfiles: + pprint_fix.patch
keywords: + patch
messages: + msg165817
2012-07-18 22:09:41anton.barkovskysetnosy: + anton.barkovsky
2012-07-18 17:26:47floxsettype: behavior
versions: + Python 3.2, Python 3.3, - Python 3.1
keywords: + easy
nosy: + flox

messages: + msg165790
stage: needs patch
2011-12-15 17:20:53giampaolo.rodolasetnosy: + giampaolo.rodola
2010-10-08 21:38:05amaury.forgeotdarcsetnosy: + amaury.forgeotdarc
messages: + msg118227
2010-10-08 18:51:38rbpsetmessages: + msg118214
2010-10-08 17:58:45rbpsetnosy: + rbp
messages: + msg118209
2010-10-02 23:04:46r.david.murraysetnosy: + rhettinger
2010-10-02 20:35:18arnocreate