This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: pprint._safe_repr is not general enough in one instance
Type: Stage:
Components: Library (Lib) Versions: Python 3.1, Python 3.2
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: rhettinger Nosy List: LambertDW, aronacher, belopolsky, erickt, georg.brandl, idadesub, pitrou, rbp, rhettinger, robert.kern
Priority: normal Keywords:

Created on 2008-09-26 16:14 by erickt, last changed 2022-04-11 14:56 by admin. This issue is now closed.

Messages (12)
msg73858 - (view) Author: Erick Tryzelaar (erickt) Date: 2008-09-26 16:14
I've run into a case where pprint isn't able to print out a particular 
data structure, and have distilled it down to a simple example:

import pprint

class A:
    pass

pprint.pprint({A(): 1, A(): 2})

Which throws this exception:

Traceback (most recent call last):
  File 
"/opt/local/Library/Frameworks/Python.framework/Versions/3.0/lib/python3
.0/pprint.py", line 272, in _safe_repr
    items = sorted(items)
TypeError: unorderable types: A() < A()

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "foo.py", line 6, in <module>
    pprint.pprint({A(): 1, A(): 2})
  File 
"/opt/local/Library/Frameworks/Python.framework/Versions/3.0/lib/python3
.0/pprint.py", line 55, in pprint
    printer.pprint(object)
  File 
"/opt/local/Library/Frameworks/Python.framework/Versions/3.0/lib/python3
.0/pprint.py", line 106, in pprint
    self._format(object, self._stream, 0, 0, {}, 0)
  File 
"/opt/local/Library/Frameworks/Python.framework/Versions/3.0/lib/python3
.0/pprint.py", line 129, in _format
    rep = self._repr(object, context, level - 1)
  File 
"/opt/local/Library/Frameworks/Python.framework/Versions/3.0/lib/python3
.0/pprint.py", line 216, in _repr
    self._depth, level)
  File 
"/opt/local/Library/Frameworks/Python.framework/Versions/3.0/lib/python3
.0/pprint.py", line 228, in format
    return _safe_repr(object, context, maxlevels, level)
  File 
"/opt/local/Library/Frameworks/Python.framework/Versions/3.0/lib/python3
.0/pprint.py", line 277, in _safe_repr
    items = sorted(items, key=sortkey)
TypeError: unorderable types: A() < A()


This is happening because of this block of code:

        try:
            items = sorted(items)
        except TypeError:
            def sortkey(item):
                key, value = item
                return str(type(key)), key, value
            items = sorted(items, key=sortkey)

The exception block is trying to sort the items again, but in this 
instance, it's still not orderable. Could we get _safe_repr to at least 
give up on sorting at this point? Or, we could try just falling back to 
sorting according to the class name, with:

        try:
            items = sorted(items)
        except TypeError:
            def sortkey(item):
                key, value = item
                return str(type(key)), key, value
            try:
                items = sorted(items, key=sortkey)
            except TypeError:
                def sortkey(item):
                    key, value = item
                    return str(type(key))

That would at least give some ordering to the output. Unfortunately, in 
this case it's a shame that we don't have the cmp function any more, 
because then we could just fall back to giving up on the ordering for 
just certain unorderable keys, but still have sorted output for 
orderable keys. I thought maybe we could test if the key and value have 
__lt__, but it looks like all classes now have that function, even if 
the user didn't implement it. In the long run though, I suppose the case 
where you have mixed types in a dict there's no sensible ordering 
anyway.
msg73860 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2008-09-26 16:27
Another solution would be to separate the dict items by key type, try to
sort each items list with separate fallback on onsorted. Then merge the
whole thing, ordered by key type name.
msg73864 - (view) Author: Erick Tryzelaar (idadesub) Date: 2008-09-26 16:45
fyi, I found another case where pprint needs a "safe sort", this is
when you have a list that contains a dictionary. Anyway, Here's a
simple patch that creates a _safe_sorted function that implements the
fallback:

--- /opt/local/lib/python3.0/pprint.py	2008-09-26 09:35:21.000000000 -0700
+++ /tmp/pprint.py	2008-09-26 09:35:13.000000000 -0700
@@ -145,7 +145,7 @@
                 if length:
                     context[objid] = 1
                     indent = indent + self._indent_per_level
-                    items  = sorted(object.items())
+                    items  = _safe_sorted(object.items())
                     key, ent = items[0]
                     rep = self._repr(key, context, level)
                     write(rep)
@@ -267,14 +267,7 @@
         append = components.append
         level += 1
         saferepr = _safe_repr
-        items = object.items()
-        try:
-            items = sorted(items)
-        except TypeError:
-            def sortkey(item):
-                key, value = item
-                return str(type(key)), key, value
-            items = sorted(items, key=sortkey)
+        items = _safe_sorted(object.items())
         for k, v in items:
             krepr, kreadable, krecur = saferepr(k, context, maxlevels, level)
             vrepr, vreadable, vrecur = saferepr(v, context, maxlevels, level)
@@ -321,6 +314,20 @@
     rep = repr(object)
     return rep, (rep and not rep.startswith('<')), False

+def _safe_sorted(items):
+    try:
+        return sorted(items)
+    except TypeError:
+        def sortkey(item):
+            key, value = item
+            return str(type(key)), key, value
+        try:
+            return sorted(items, key=sortkey)
+        except TypeError:
+            def sortkey(item):
+                key, value = item
+                return str(type(key))
+            return sorted(items, key=sortkey)

 def _recursion(object):
     return ("<Recursion on %s with id=%s>"

One other thing to note is that I'm also aware that the yaml project
also has this problem, and they've got their own "safe_sorted"
function. It might be worthwhile formalizing this in a public function
in the api somewhere.
msg78682 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2009-01-01 02:25
For this to be integrated, it should also add an unit test.
msg79278 - (view) Author: Alexander Belopolsky (belopolsky) * (Python committer) Date: 2009-01-06 19:12
The proposed patch appears to give up sorting by key,value altogether if 
there are a few incomparable items.  It would be better to group items by 
type and sort within each group.  For example, 
pprint({1:1,2:2,A():3,A():4}) should print int:int items in order.
msg92904 - (view) Author: Georg Brandl (georg.brandl) * (Python committer) Date: 2009-09-20 17:17
Also note that this patch will not sort sequences of mixed types where
some types are intercomparable "correctly" (in the way that Python 2
did), e.g. for

{1:2, 2:3, 'a':4, 1.5: 5}

the 1.5 key will not be placed between the 1 and 2 keys.

I'm not aware of a way to implement that behavior without support for a
general comparison function instead of a DSU key function.
msg92905 - (view) Author: Raymond Hettinger (rhettinger) * (Python committer) Date: 2009-09-20 17:32
I'm thinking that pprint should not try to sort unsortable items.

   try:
       items = sorted(items)
   except TypeError:
       items = list(items)
msg92906 - (view) Author: Georg Brandl (georg.brandl) * (Python committer) Date: 2009-09-20 17:35
OK, there *is* a way.  Consider this:

class safe_key(object):
    __slots__ = ('obj',)

    def __init__(self, obj):
        self.obj = obj

    def __eq__(self, other):
        return self.obj.__eq__(other.obj)

    def __lt__(self, other):
        try:
            return self.obj < other.obj
        except TypeError:
            return id(type(self.obj)) < id(type(other.obj))


ls = [2, 1, 1.0, 1.5, 'a', 'c', 'b']
print(sorted(ls, key=safe_key))
msg92907 - (view) Author: Armin Ronacher (aronacher) * (Python committer) Date: 2009-09-20 17:37
@Georg: Instead of catching a TypeError i would rather call __gt__ /
__lt__ directly and check for NotImplemented.  Python 2.x did not catch
TypeErrors either.
msg92908 - (view) Author: Armin Ronacher (aronacher) * (Python committer) Date: 2009-09-20 17:39
Eg, something like this:

class safe_key(object):

    __slots__ = ('obj',)

    def __init__(self, obj):
        self.obj = obj

    def __eq__(self, other):
        return self.obj.__eq__(other.obj)

    def __lt__(self, other):
        rv = self.obj.__lt__(other.obj)
        if rv is NotImplemented:
            rv = id(type(self.obj)) < id(type(other.obj))
        return rv


ls = [2, 1, 1.0, 1.5, 'a', 'c', 'b']
print(sorted(ls, key=safe_key))
msg95460 - (view) Author: Raymond Hettinger (rhettinger) * (Python committer) Date: 2009-11-19 01:07
Fixed.  See r76389 and r76390.
msg118217 - (view) Author: Rodrigo Bernardo Pimentel (rbp) (Python committer) Date: 2010-10-08 19:00
Armin: this has the problem that, if the object you're trying to compare is a class, self.obj.__lt__ expects a different number of parameters (i.e., it expects the instance). See issue 10017 . Testing with "<" works.
History
Date User Action Args
2022-04-11 14:56:39adminsetgithub: 48226
2010-10-08 19:00:39rbpsetnosy: + rbp
messages: + msg118217
2009-11-19 01:07:22rhettingersetstatus: open -> closed
resolution: fixed
messages: + msg95460
2009-11-18 16:21:58LambertDWsetnosy: + LambertDW
2009-09-20 17:39:50aronachersetmessages: + msg92908
2009-09-20 17:37:54aronachersetmessages: + msg92907
2009-09-20 17:35:07georg.brandlsetmessages: + msg92906
2009-09-20 17:32:39rhettingersetversions: + Python 3.1, Python 3.2, - Python 3.0
nosy: + rhettinger

messages: + msg92905

assignee: rhettinger
2009-09-20 17:17:52georg.brandlsetnosy: + georg.brandl
messages: + msg92904
2009-09-20 17:10:36aronachersetnosy: + aronacher
2009-02-04 10:41:45robert.kernsetnosy: + robert.kern
2009-01-06 19:12:24belopolskysetnosy: + belopolsky
messages: + msg79278
2009-01-01 02:25:30pitrousetmessages: + msg78682
2008-09-26 16:45:53idadesubsetnosy: + idadesub
messages: + msg73864
2008-09-26 16:27:20pitrousetnosy: + pitrou
messages: + msg73860
2008-09-26 16:14:59ericktcreate