LC_CTYPE=C: pydoc leaves terminal in an unusable state #65597

skrah · 2014-04-30T11:33:21Z

BPO	21398
Nosy	@vstinner, @bitdancer, @skrah, @matrixise
Files	pydoc_encoding.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2014-05-13.00:08:11.505>
created_at = <Date 2014-04-30.11:33:21.452>
labels = ['type-bug']
title = 'LC_CTYPE=C:  pydoc leaves terminal in an unusable state'
updated_at = <Date 2015-03-29.19:23:09.038>
user = 'https://github.com/skrah'

bugs.python.org fields:

activity = <Date 2015-03-29.19:23:09.038>
actor = 'r.david.murray'
assignee = 'none'
closed = True
closed_date = <Date 2014-05-13.00:08:11.505>
closer = 'vstinner'
components = []
creation = <Date 2014-04-30.11:33:21.452>
creator = 'skrah'
dependencies = []
files = ['35132']
hgrepos = []
issue_num = 21398
keywords = ['patch']
message_count = 17.0
messages = ['217604', '217607', '217608', '217609', '217611', '217612', '217613', '217614', '217615', '217618', '217779', '217780', '217784', '217803', '218388', '218389', '239513']
nosy_count = 5.0
nosy_names = ['vstinner', 'r.david.murray', 'skrah', 'python-dev', 'matrixise']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = None
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue21398'
versions = ['Python 3.4', 'Python 3.5']

skrah · 2014-04-30T11:33:21Z

$ ./python -m pydoc heapq

Traceback (most recent call last):
  File "/home/stefan/hg/cpython/Lib/runpy.py", line 170, in _run_module_as_main
    "__main__", mod_spec)
  File "/home/stefan/hg/cpython/Lib/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/home/stefan/hg/cpython/Lib/pydoc.py", line 2615, in <module>
    cli()
  File "/home/stefan/hg/cpython/Lib/pydoc.py", line 2580, in cli
    help.help(arg)
  File "/home/stefan/hg/cpython/Lib/pydoc.py", line 1862, in help
    elif request: doc(request, 'Help on %s:', output=self._output)
  File "/home/stefan/hg/cpython/Lib/pydoc.py", line 1600, in doc
    pager(render_doc(thing, title, forceload))
  File "/home/stefan/hg/cpython/Lib/pydoc.py", line 1408, in pager
    pager(text)
  File "/home/stefan/hg/cpython/Lib/pydoc.py", line 1422, in <lambda>
    return lambda text: pipepager(text, os.environ['PAGER'])
  File "/home/stefan/hg/cpython/Lib/pydoc.py", line 1449, in pipepager
    pipe.write(text)
UnicodeEncodeError: 'ascii' codec can't encode character '\xe7' in position 3574: ordinal not in range(128)

$ stty sane
$ locale      
LANG=en_US.UTF-8
LANGUAGE=
LC_CTYPE="C"
LC_NUMERIC="C"
LC_TIME="C"
LC_COLLATE="C"
LC_MONETARY="C"
LC_MESSAGES="C"
LC_PAPER="C"
LC_NAME="C"
LC_ADDRESS="C"
LC_TELEPHONE="C"
LC_MEASUREMENT="C"
LC_IDENTIFICATION="C"
LC_ALL=C

bitdancer · 2014-04-30T12:22:26Z

Works fine for me.

matrixise · 2014-04-30T12:28:36Z

the pydoc module works fine but when I use CTRL-C to quit it, I get this error in the terminal.

I think it's an other bug but the traceback is really similar.

Stephane

Traceback (most recent call last):
  File "/Users/stephane/src/projects/externals/cpython/Lib/runpy.py", line 170, in _run_module_as_main
    "__main__", mod_spec)
  File "/Users/stephane/src/projects/externals/cpython/Lib/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/Users/stephane/src/projects/externals/cpython/Lib/pydoc.py", line 2615, in <module>
    cli()
  File "/Users/stephane/src/projects/externals/cpython/Lib/pydoc.py", line 2580, in cli
    help.help(arg)
  File "/Users/stephane/src/projects/externals/cpython/Lib/pydoc.py", line 1862, in help
    elif request: doc(request, 'Help on %s:', output=self._output)
  File "/Users/stephane/src/projects/externals/cpython/Lib/pydoc.py", line 1600, in doc
    pager(render_doc(thing, title, forceload))
  File "/Users/stephane/src/projects/externals/cpython/Lib/pydoc.py", line 1408, in pager
    pager(text)
  File "/Users/stephane/src/projects/externals/cpython/Lib/pydoc.py", line 1428, in <lambda>
    return lambda text: pipepager(text, 'less')
  File "/Users/stephane/src/projects/externals/cpython/Lib/pydoc.py", line 1450, in pipepager
    pipe.close()
  File "/Users/stephane/src/projects/externals/cpython/Lib/os.py", line 957, in close
    returncode = self._proc.wait()
  File "/Users/stephane/src/projects/externals/cpython/Lib/subprocess.py", line 1581, in wait
    (pid, sts) = self._try_wait(0)
  File "/Users/stephane/src/projects/externals/cpython/Lib/subprocess.py", line 1529, in _try_wait
    (pid, sts) = _eintr_retry_call(os.waitpid, self.pid, wait_flags)
  File "/Users/stephane/src/projects/externals/cpython/Lib/subprocess.py", line 502, in _eintr_retry_call
    return func(*args)
KeyboardInterrupt

skrah · 2014-04-30T12:30:14Z

Did you use the same locale settings?

matrixise · 2014-04-30T12:32:55Z

locale
LANG="en_US.UTF-8"
LC_COLLATE="en_US.UTF-8"
LC_CTYPE="en_US.UTF-8"
LC_MESSAGES="en_US.UTF-8"
LC_MONETARY="en_US.UTF-8"
LC_NUMERIC="en_US.UTF-8"
LC_TIME="en_US.UTF-8"
LC_ALL="en_US.UTF-8"

skrah · 2014-04-30T12:37:47Z

Sorry, then I should have been more explicit: The failure only occurs
with LC_CTYPE=C.

bitdancer · 2014-04-30T12:40:00Z

Ah, yes, my lc_ctype was en_US.utf-8. I can reproduce it if I change that.

matrixise · 2014-04-30T12:43:22Z

I use OSX 10.9 on my laptop, Python 3.5 and I get this error in one case. If I use CTRL-C to quit the application and if LC_CTYPE=C.

with the 'q' key, I don't get this problem.

Just LC_CTYPE=C and CTRL-C and I have to reset my terminal.

skrah · 2014-04-30T12:53:54Z

I can also confirm the need to reset the terminal when using
a working locale and Ctrl-C.

So we have two issues then:

The UnicodeDecodeError should not happen.
pydoc behaves erratically after various exceptions.

In Python2.7 neither of the issues is present.

matrixise · 2014-04-30T12:59:59Z

In python 2.7, If I use my working locales (utf-8) and I use CTRL-C, pydoc does not quit but leave a message when the screen is cleaned.

same result with LC_CTYPE=C

bash-4.3$ python -m pydoc heapq
Traceback (most recent call last):
  File "/usr/local/Cellar/python/2.7.6_1/Frameworks/Python.framework/Versions/2.7/lib/python2.7/runpy.py", line 162, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/usr/local/Cellar/python/2.7.6_1/Frameworks/Python.framework/Versions/2.7/lib/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "/usr/local/Cellar/python/2.7.6_1/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pydoc.py", line 2359, in <module>
    if __name__ == '__main__': cli()
  File "/usr/local/Cellar/python/2.7.6_1/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pydoc.py", line 2328, in cli
    help.help(arg)
  File "/usr/local/Cellar/python/2.7.6_1/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pydoc.py", line 1793, in help
    elif request: doc(request, 'Help on %s:')
  File "/usr/local/Cellar/python/2.7.6_1/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pydoc.py", line 1532, in doc
    pager(render_doc(thing, title, forceload))
  File "/usr/local/Cellar/python/2.7.6_1/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pydoc.py", line 1337, in pager
    pager(text)
  File "/usr/local/Cellar/python/2.7.6_1/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pydoc.py", line 1357, in <lambda>
    return lambda text: pipepager(text, 'less')
  File "/usr/local/Cellar/python/2.7.6_1/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pydoc.py", line 1379, in pipepager
    pipe.close()
KeyboardInterrupt
bash-4.3$

vstinner · 2014-05-02T19:39:18Z

heapq documentation contains "François" which is not encodable to ASCII. When using LC_ALL=C, the locale encoding is ASCII (at least on Linux). It's not easy to specify a different error handler globally in pydoc, different functions are used and child processes are spawned. The subprocess module doesn't allow to specify an error handler different than strict (see bpo-6135).

I propose pydoc_encoding.patch which escapes manually non-encodable characters. I chose sys.getfilesystemencoding(), I'm not sure that it's the encoding used for all cases of getpager() on all platforms. For example, on Windows, sys.getfilesystemencoding() is the ANSI code page, whereas sys.stdout.encoding is the OEM code page.

vstinner · 2014-05-02T19:46:04Z

"LC_CTYPE=C: pydoc leaves terminal in an unusable state"

In the use case, pydoc doesn't touch the terminal, it's the pager: the program "less". I don't see how to ensure that the terminal state is restored, even on error.

skrah · 2014-05-02T20:10:48Z

STINNER Victor <report@bugs.python.org> wrote:

I don't see how to ensure that the terminal state is restored, even on error.

Python2 suppresses the exception until after normal exit (pressing 'q').
I think that behavior is better.

In Python3 you can also get the unusable terminal by pressing Ctrl-C,
i.e. without any UnicodeError.

skrah · 2014-05-02T21:59:48Z

The patch works well BTW. We can create another issue for the
general misbehavior of pydoc with other exceptions.

python-dev · 2014-05-13T00:06:43Z

New changeset 89a29e92416f by Victor Stinner in branch '3.4':
Issue bpo-21398: Fix an unicode error in the pydoc pager when the documentation
http://hg.python.org/cpython/rev/89a29e92416f

New changeset 3424d65ad5ce by Victor Stinner in branch 'default':
(Merge 3.4) Issue bpo-21398: Fix an unicode error in the pydoc pager when the
http://hg.python.org/cpython/rev/3424d65ad5ce

vstinner · 2014-05-13T00:08:11Z

I fixed the initial bug and so I close the issue.

Open a new issue if you have an idea to restore the terminal state when the pager breaks the terminal. I don't think that it's possible to save/restore the terminal state in a portable way.

bitdancer · 2015-03-29T19:23:09Z

The KeybaordInterrupt problem has been dealt with in bpo-23792.

skrah mannequin added the type-bug An unexpected behavior, bug, or error label Apr 30, 2014

skrah mannequin changed the title ~~pydoc heapq leaves terminal in an unusable state~~ LC_CTYPE=C: pydoc leaves terminal in an unusable state Apr 30, 2014

vstinner closed this as completed May 13, 2014

ezio-melotti transferred this issue from another repository Apr 10, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

LC_CTYPE=C: pydoc leaves terminal in an unusable state #65597

LC_CTYPE=C: pydoc leaves terminal in an unusable state #65597

skrah mannequin commented Apr 30, 2014

skrah mannequin commented Apr 30, 2014

bitdancer commented Apr 30, 2014

matrixise commented Apr 30, 2014

skrah mannequin commented Apr 30, 2014

matrixise commented Apr 30, 2014

skrah mannequin commented Apr 30, 2014

bitdancer commented Apr 30, 2014

matrixise commented Apr 30, 2014

skrah mannequin commented Apr 30, 2014

matrixise commented Apr 30, 2014

vstinner commented May 2, 2014

vstinner commented May 2, 2014

skrah mannequin commented May 2, 2014

skrah mannequin commented May 2, 2014

python-dev mannequin commented May 13, 2014

vstinner commented May 13, 2014

bitdancer commented Mar 29, 2015

LC_CTYPE=C: pydoc leaves terminal in an unusable state #65597

LC_CTYPE=C: pydoc leaves terminal in an unusable state #65597

Comments

skrah mannequin commented Apr 30, 2014

skrah mannequin commented Apr 30, 2014

bitdancer commented Apr 30, 2014

matrixise commented Apr 30, 2014

skrah mannequin commented Apr 30, 2014

matrixise commented Apr 30, 2014

skrah mannequin commented Apr 30, 2014

bitdancer commented Apr 30, 2014

matrixise commented Apr 30, 2014

skrah mannequin commented Apr 30, 2014

matrixise commented Apr 30, 2014

vstinner commented May 2, 2014

vstinner commented May 2, 2014

skrah mannequin commented May 2, 2014

skrah mannequin commented May 2, 2014

python-dev mannequin commented May 13, 2014

vstinner commented May 13, 2014

bitdancer commented Mar 29, 2015