New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
pydoc 3.x raises UnicodeEncodeError on sqlite3 package #67563
Comments
I'm probably doing something wrong, but I've tried everything I can think of In Python 2.7, the pydoc command successfully displays help for the sqlite3 Here's a sample run: % PAGER=cat pydoc3.5 sqlite3
Traceback (most recent call last):
File "/Users/skip/local/bin/pydoc3.5", line 5, in <module>
pydoc.cli()
File "/Users/skip/local/lib/python3.5/pydoc.py", line 2591, in cli
help.help(arg)
File "/Users/skip/local/lib/python3.5/pydoc.py", line 1874, in help
elif request: doc(request, 'Help on %s:', output=self._output)
File "/Users/skip/local/lib/python3.5/pydoc.py", line 1612, in doc
pager(render_doc(thing, title, forceload))
File "/Users/skip/local/lib/python3.5/pydoc.py", line 1412, in pager
pager(text)
File "/Users/skip/local/lib/python3.5/pydoc.py", line 1428, in <lambda>
return lambda text: pipepager(text, os.environ['PAGER'])
File "/Users/skip/local/lib/python3.5/pydoc.py", line 1455, in pipepager
pipe.write(text)
UnicodeEncodeError: 'ascii' codec can't encode character '\xe4' in position 600: ordinal not in range(128) I understand the error, but I see no way to convince it to use any codec
This is on a Mac running Yosemite with pydoc invoked in Apple's Terminal The source it is attempting to display has a coding cookie, so it should |
What are sys.getfilesystemencoding(), locale.getpreferredencoding(False), os.popen('cat', 'w').encoding? |
Without setting any environment variables: >>> import sys
>>> sys.getfilesystemencoding()
'utf-8'
>>> import locale
>>> locale.getpreferredencoding(False)
'US-ASCII'
>>> import os
>>> os.popen('cat', 'w').encoding
'US-ASCII' If I set PYTHONIOENCODING=UTF-8: >>> import sys, locale, os
>>> sys.getfilesystemencoding()
'utf-8'
>>> locale.getpreferredencoding(False)
'US-ASCII'
>>> os.popen('cat', 'w').encoding
'US-ASCII' If I set LANG=en_US.utf8: >>> import sys, locale, os
>>> sys.getfilesystemencoding()
'utf-8'
>>> locale.getpreferredencoding(False)
'US-ASCII'
>>> os.popen('cat', 'w').encoding
'US-ASCII' It appears neither of these environment variables does much in my environment. I should point out that I just updated to Mac OS X 10.10.2 a couple |
Peter Otten posted a solution on c.l.py. The issue is that I didn't hgpython% LANG=en_US.UTF-8 python3.5 -c 'import locale; |
On Sun, Feb 1, 2015 at 2:19 PM, Skip Montanaro <report@bugs.python.org> wrote:
Actually, it's that the hyphen is required in "utf-8" or "UTF-8". |
Final note here. Peter also did a bit of digging. Here's his note about The pager is invoked by os.popen(), and after some digging I find that it I was also able to provoke this problem on an openSuSE 12.2 system with I don't know if Python should try to be accommodating here, but my |
Maybe because a pager sends its bytes more-or-less straight throught from input to output, the PYTHONIOENCODING (sys.stdout.encoding?) should be used for the TextIOWrapper to the pager’s input in this case. I’m not so sure this should be assumed in general though. |
There are few levels of this issue:
Here is a patch which solves first level -- makes pydoc using appropriate encoding with the backslashreplace error handler. |
Added a test. |
Patch looks sensible to me. This is another example of where bpo-15216 would be useful (a standard way to modify the encoding settings of a stream). |
In the case of this issue pydoc needs change not the encoding of stdout, but errors handler of stdout. There is similar issue with pprint (bpo-19100). |
New changeset e7b6b1f57268 by Serhiy Storchaka in branch '3.4': New changeset affe167a45f3 by Serhiy Storchaka in branch 'default': |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
bugs.python.org fields:
The text was updated successfully, but these errors were encountered: