Title: UnicodeEncodeError on pydoc's CLI
Type: behavior Stage: resolved
Components: Library (Lib) Versions: Python 2.7
Status: closed Resolution: out of date
Dependencies: Superseder:
Assigned To: Nosy List: akitada, berker.peksag, christoph, eric.araujo, flox, ggenellina, torsten, vstinner
Priority: normal Keywords: patch

Created on 2009-08-02 13:00 by christoph, last changed 2014-01-13 10:56 by torsten. This issue is now closed.

File name Uploaded Description Edit
unicode.patch christoph, 2009-08-02 13:00 Patch encoding output with system encoding
pydoc_unicode_testcase_notworking.diff christoph, 2009-08-05 11:29 Added test for unicode (defunc)
issue6625_pydoc.diff flox, 2010-02-22 12:53 Patch, apply to trunk
Messages (10)
msg91182 - (view) Author: Christoph Burgmer (christoph) Date: 2009-08-02 13:00
pydoc fails with a UnicodeEncodeError for properly specified Unicode
docstrings (u"""...""") on the command line interface.

See attached patch that encodes the output with the system's encoding.
msg91295 - (view) Author: Gabriel Genellina (ggenellina) Date: 2009-08-05 02:00
After applying your patch, are string (not unicode) docstrings still 
being handled properly?
Adding a test case (in Lib/test/ would be nice too.
msg91315 - (view) Author: Christoph Burgmer (christoph) Date: 2009-08-05 11:29
Here is a diff for test/ (against Python2.6) which though
doesn't trigger due to how Python handles output encoding. This test
here will pass, but pydoc will still fail:

$ pydoc test/ > /dev/null
Traceback (most recent call last):
  File "/usr/bin/pydoc", line 5, in <module>
  File "/usr/lib/python2.5/", line 2226, in cli
  File "/usr/lib/python2.5/", line 1691, in help
    else: doc(request, 'Help on %s:')
  File "/usr/lib/python2.5/", line 1482, in doc
    pager(title % desc + '\n\n' + text.document(object, name))
  File "/usr/lib/python2.5/", line 1300, in pager
  File "/usr/lib/python2.5/", line 1398, in plainpager
UnicodeEncodeError: 'ascii' codec can't encode character u'\xfc' in
position 936: ordinal not in range(128)
msg99716 - (view) Author: Torsten Landschoff (torsten) * Date: 2010-02-22 10:26
This problem persists with current Python 2.7 trunk (I used hg mirror):
52aa81c14f87 (trunk) tip

I also checked with the py3k branch and it does not show this limitation. Just as one would have expected...
msg99717 - (view) Author: Torsten Landschoff (torsten) * Date: 2010-02-22 10:35
As Subversion is still the master repo (correct me if I am wrong), the hg version tested maps to r78290 in subversion on /trunk.
msg99723 - (view) Author: Florent Xicluna (flox) * (Python committer) Date: 2010-02-22 11:36
This patch do not work for me.

See the patch attached on duplicate bug #7625, with more tests:
msg99728 - (view) Author: Florent Xicluna (flox) * (Python committer) Date: 2010-02-22 12:53
Patch with same tests as the previous one, but using better heuristic for output encoding (like Christoph patch).
Added the "replace" error handling, if the output encoding cannot encode all characters.
msg208006 - (view) Author: Akira Kitada (akitada) * Date: 2014-01-13 06:40
I suppose this is a duplicate of #1065986.
msg208013 - (view) Author: Berker Peksag (berker.peksag) * (Python committer) Date: 2014-01-13 08:40
> I suppose this is a duplicate of #1065986.

Yes, it is. I created a test file from the tests in issue6625_pydoc.diff and ran it on the current 2.7 branch.

def foo():
    u"""fooo bar
    return 42

def bar():
    u"f\xfcr Elise"
    return 11
msg208018 - (view) Author: Torsten Landschoff (torsten) * Date: 2014-01-13 10:56
I tested this as well and it seems to work now. :-) Thanks for fixing it!
Date User Action Args
2014-01-13 10:56:30torstensetmessages: + msg208018
2014-01-13 08:40:41berker.peksagsetstatus: open -> closed

nosy: + berker.peksag
messages: + msg208013

resolution: out of date
stage: patch review -> resolved
2014-01-13 06:40:19akitadasetnosy: + akitada
messages: + msg208006
2012-07-21 13:50:21floxsetstage: needs patch -> patch review
2011-04-27 08:52:44eric.araujosetnosy: + eric.araujo

versions: - Python 2.6
2010-11-02 20:36:51vstinnersetnosy: + vstinner
2010-08-01 19:42:07floxlinkissue8994 superseder
2010-02-22 12:53:07floxsetfiles: + issue6625_pydoc.diff

messages: + msg99728
2010-02-22 11:36:36floxsetpriority: normal

type: behavior
components: + Library (Lib), - Extension Modules
versions: + Python 2.7, - Python 2.5
nosy: + flox

messages: + msg99723
stage: needs patch
2010-02-22 11:06:55floxlinkissue7675 superseder
2010-02-22 10:35:37torstensetmessages: + msg99717
2010-02-22 10:26:18torstensetnosy: + torsten
messages: + msg99716
2009-08-05 11:29:49christophsetfiles: + pydoc_unicode_testcase_notworking.diff

messages: + msg91315
2009-08-05 02:00:31ggenellinasetnosy: + ggenellina
messages: + msg91295
2009-08-02 13:00:46christophcreate