New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Use backslashreplace in pprint #63299
Comments
Currently pprint.pprint() fails on unencodable characters. $ LANG=en_US.utf8 ./python -c "import pprint; pprint.pprint('\u20ac')"
'€'
$ LANG= ./python -c "import pprint; pprint.pprint('\u20ac')"
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/home/serhiy/py/cpython/Lib/pprint.py", line 56, in pprint
printer.pprint(object)
File "/home/serhiy/py/cpython/Lib/pprint.py", line 137, in pprint
self._format(object, self._stream, 0, 0, {}, 0)
File "/home/serhiy/py/cpython/Lib/pprint.py", line 274, in _format
write(rep)
UnicodeEncodeError: 'ascii' codec can't encode character '\u20ac' in position 1: ordinal not in range(128) This is a regression from Python 2 in which repr() always returns ascii string. $ LANG= python2.7 -c "import pprint; pprint.pprint(u'\u20ac')"
u'\u20ac' Perhaps pprint() should use the backslashreplace error handler (as sys.displayhook()). With the proposed patch: $ LANG= ./python -c "import pprint; pprint.pprint('\u20ac')"
'\u20ac' |
Any review? |
In new patch wrapping stream is moved to PrettyPrinter constructor. |
This is not the fault of pprint. IMHO it doesn't make sense to fix anything here, at least not for pprint specifically. print() has the same "problem": $ LANG= ./python -c "print('\u20ac')"
Traceback (most recent call last):
File "<string>", line 1, in <module>
UnicodeEncodeError: 'ascii' codec can't encode character '\u20ac' in position 0: ordinal not in range(128) |
pprint is not print. >>> print('\u20ac')
€
>>> import pprint; pprint.pprint('\u20ac')
'€' Default sys.displayhook doesn't fail on unencodable output. $ LANG=C ./python
Python 3.4.0b1 (default:e961a166dc70+, Dec 11 2013, 13:57:17)
[GCC 4.6.3] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> '\u20ac'
'\u20ac' |
sys.displayhook doesn't fail, because it uses the backslashreplace error handler, and for sys.displayhook that's OK, because it's only used for screen output and there some output is better than no output. However print and pprint.pprint might be used for output that is consumed by other programs (via pipes etc.) and IMHO in this case "Errors should never pass silently." |
The purpose of pprint.pprint() is to produce human-readable output. In this case some output is better than nothing. It isn't designed to be parseable by other programs, because sometimes it is even less accurate than the result of repr() (pprint() truncates long reprs and losses information for dict subclasses). Also result of pprint() can be changed from version to version (e.g. bpo-17150). The main source of non-ASCII characters is string reprs and for them the backslashreplace error handler doesn't lose information. And pprint.pprint() is mainly used for screen output too. |
I agree with Serhiy that using a permissive error handler with pprint() is appropriate. What is the reasoning behind the DecodeWriter case, where the original stream has an interesting encoding, but “buffer” is None? Are there any real-world cases like that? Your mock test case sets encoding="latin1" with no buffer, but that class will also write non-latin1 strings, so there is no problem. Also I wonder if flushing the stream once or twice for each pprint() call is a wise move. Another way to tackle this might be a function that translates the non-Latin-1 or whatever characters, allowing the original write() or whatever method to still be used. Here is a Python 2 and 3 compatible attempt: <https://bitbucket.org/Gfy/pyrescene/src/560cafe/rescene/utility.py#cl-426\>. Python 3 only version: <vadmium/python-iview@68b0559\>. This function is originally used for printing descriptive comments to stdout (alongside other text where the “strict” error handler is appropriate). But I think it could be generally usable for pprint(), sys.displayhook(), etc as well. |
The linked code at vadmium/python-iview@68b0559 seems strange to me:
is the same as: return text.encode(encoding, errors).decode(encoding) because when there are no unencodable characters in text, the error handler will never be invoked. |
sys.stdout and sys.stderr in IDLE. |
Walter: the first line encoding with textio.errors is meant to handle the case where the output stream already has its own permissive error handler set. But anyway I was just trying to point out that it might be better to do the backslash escaping at the text level, and write the escaped text string to the original stream. Serhiy: thanks for pointing out IDLE’s stdout. It seems the encoding can be set to say ASCII by the locale, yet it still accepts non-ASCII text. But I guess that’s a separate issue. I haven’t tested the patch, but reading it, I think the there may be a couple of problems:
bpo-15216 is slightly related, and has a patch apparently allowing the encoding and error handler to be changed on a text stream. But I guess it is no good here because you need backwards compatibility with other non-TextIOWrapper streams. |
Try with LANG=en_US. And even UTF-8 can fail. |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
bugs.python.org fields:
The text was updated successfully, but these errors were encountered: