classification
Title: pprint._safe_key is not always safe enough
Type: behavior Stage: resolved
Components: Library (Lib) Versions: Python 3.2, Python 3.3
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: Shawn.Brown, ezio.melotti, flox, python-dev, rhettinger
Priority: normal Keywords: patch

Created on 2012-06-04 03:32 by Shawn.Brown, last changed 2012-07-21 11:05 by flox. This issue is now closed.

Files
File name Uploaded Description Edit
pprint_safe_key.patch Shawn.Brown, 2012-06-08 03:36 Patch adds a try/catch to pprint._safe_key to handle certain unsortable types when used as dict keys (also adds relevant test case assertions). review
Messages (7)
msg162249 - (view) Author: Shawn Brown (Shawn.Brown) Date: 2012-06-04 03:32
This is related to resolved issue 3976 and, to a lesser extent, issue 10017.

I've run across another instance where pprint throws an exception (but works fine in 2.7 and earlier):

Python 3.2 (r32:88445, Mar 25 2011, 19:28:28) 
[GCC 4.5.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> from pprint import pprint
>>> pprint({(0,): 1, (None,): 2})
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python3.2/pprint.py", line 55, in pprint
    printer.pprint(object)
  File "/usr/lib/python3.2/pprint.py", line 132, in pprint
    self._format(object, self._stream, 0, 0, {}, 0)
  File "/usr/lib/python3.2/pprint.py", line 155, in _format
    rep = self._repr(object, context, level - 1)
  File "/usr/lib/python3.2/pprint.py", line 245, in _repr
    self._depth, level)
  File "/usr/lib/python3.2/pprint.py", line 257, in format
    return _safe_repr(object, context, maxlevels, level)
  File "/usr/lib/python3.2/pprint.py", line 299, in _safe_repr
    items = sorted(object.items(), key=_safe_tuple)
  File "/usr/lib/python3.2/pprint.py", line 89, in __lt__
    rv = self.obj.__lt__(other.obj)
TypeError: unorderable types: int() < NoneType()

The above example might seem contrived but I stumbled across the issue quite naturally. Honest!

In working with multiple lists and computing results using combinations of these lists' values.  I _could_ organize the results as a dictionary of dictionaries of dictionaries but that would get confusing very quickly.  Instead, I'm using a single dictionary with a composite key ("flat is better than nested"). So I've got code like this...

>>> combinations = itertools.product(lst_x, lst_y, lst_z)
>>> results = {(x,y,z): compute(x,y,z) for x,y,z in combinations}

... and it is not uncommon for one or more of the values to be None -- resulting in the above exception should anyone (including unittest) attempt to pprint the dictionary.
msg162250 - (view) Author: Shawn Brown (Shawn.Brown) Date: 2012-06-04 04:59
Currently, I'm monkey patching _safe_key (adding a try/except) as follows:

>>> import pprint
>>>
>>> class _safe_key(pprint._safe_key):
>>>     def __lt__(self, other):
>>>         try:
>>>             rv = self.obj.__lt__(other.obj)
>>>         except TypeError:   # Exception instead of TypeError?
>>>             rv = NotImplemented
>>> 
>>>         if rv is NotImplemented:
>>>             rv = (str(type(self.obj)), id(self.obj)) < \
>>>                  (str(type(other.obj)), id(other.obj))
>>>         return rv
>>>         
>>> pprint._safe_key = _safe_key
>>> 
>>> pprint.pprint({(0,): 1, (None,): 2})
{(None,): 2, (0,): 1}
msg162519 - (view) Author: Shawn Brown (Shawn.Brown) Date: 2012-06-08 03:36
Here's a patch for 3.3 -- as well as two new assertions in test_pprint.py

The added try/catch also fixes the issues mentioned in issue 10017 so I added a test for that case as well.
msg165998 - (view) Author: Roundup Robot (python-dev) Date: 2012-07-21 09:23
New changeset 03cda5360dc6 by Florent Xicluna in branch '3.2':
Issues #10017 and #14998: Fix TypeError using pprint on dictionaries with unorderable key.
http://hg.python.org/cpython/rev/03cda5360dc6

New changeset 4d0dcfbdf45b by Florent Xicluna in branch 'default':
Issues #10017 and #14998: Fix TypeError using pprint on dictionaries with unorderable key.
http://hg.python.org/cpython/rev/4d0dcfbdf45b
msg166001 - (view) Author: Florent Xicluna (flox) * (Python committer) Date: 2012-07-21 09:29
Thank you for this patch.
msg166004 - (view) Author: Florent Xicluna (flox) * (Python committer) Date: 2012-07-21 10:13
I just broke tests :(
msg166011 - (view) Author: Florent Xicluna (flox) * (Python committer) Date: 2012-07-21 11:05
Test test_pprint fixed with changesets 29642f82bbcc and 79d44f4920d9.
History
Date User Action Args
2012-07-21 11:05:24floxsetstatus: open -> closed

messages: + msg166011
2012-07-21 10:13:47floxsetstatus: closed -> open

messages: + msg166004
2012-07-21 09:29:03floxsetstatus: open -> closed
resolution: fixed
messages: + msg166001

stage: patch review -> resolved
2012-07-21 09:27:04floxlinkissue10017 superseder
2012-07-21 09:23:13python-devsetnosy: + python-dev
messages: + msg165998
2012-06-15 08:05:07ezio.melottisetnosy: + ezio.melotti

stage: needs patch -> patch review
2012-06-08 03:36:57Shawn.Brownsetfiles: + pprint_safe_key.patch
keywords: + patch
messages: + msg162519
2012-06-04 21:14:08r.david.murraysetnosy: + rhettinger
stage: needs patch
type: behavior

versions: + Python 3.3
2012-06-04 07:37:50floxsetnosy: + flox
2012-06-04 04:59:37Shawn.Brownsetmessages: + msg162250
2012-06-04 03:32:15Shawn.Browncreate