classification
Title: Importing readline produces erroneous output
Type: Stage: patch review
Components: Extension Modules Versions: Python 3.5, Python 3.4, Python 2.7
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: Arfrever, David.Edelsohn, Edd.Barrett, Maxime Belanger, benjamin.peterson, berker.peksag, bkabrda, dmalcolm, geoffreyspear, haypo, ischwabacher, iyogeshjoshi, jcea, martin.panter, ned.deily, pitrou, python-dev, rpointel, vajrasky, vlee
Priority: high Keywords: patch

Created on 2013-12-04 12:06 by bkabrda, last changed 2016-06-18 09:31 by martin.panter.

Files
File name Uploaded Description Edit
readline_disable_meta_key.patch haypo, 2014-06-12 09:21
meta-osx.patch martin.panter, 2016-06-18 09:31
Messages (31)
msg205217 - (view) Author: Bohuslav "Slavek" Kabrda (bkabrda) * Date: 2013-12-04 12:06
A simple reproducer:

python -c 'import readline' | xxd
0000000: 1b5b 3f31 3033 3468                      .[?1034h

This was reported at [1] and originally at [2]. The readline maintainer suggests [3] using:

rl_variable_bind ("enable-meta-key", "off");

which was introduced in readline 6.1. Do you think it'd be safe to add the above line?

Thanks.

[1] https://bugzilla.redhat.com/show_bug.cgi?id=880393
[2] https://bugzilla.redhat.com/show_bug.cgi?id=593799
[3] http://lists.gnu.org/archive/html/bug-readline/2011-04/msg00009.html
msg205221 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2013-12-04 15:00
I can't reproduce under Ubuntu 13.10. Is this Red Hat-specific?

(according to Dave, """This is a readline bug; it looks like it should not emit those characters when stdout is not a tty""")
msg205228 - (view) Author: Vajrasky Kok (vajrasky) * Date: 2013-12-04 15:51
Reproducible under Fedora 18.

$ ./python -c "import readline" | hexdump -C
00000000  1b 5b 3f 31 30 33 34 68                           |.[?1034h|
00000008

$ TERM=dumb ./python -c "import readline" | hexdump -C
msg205293 - (view) Author: Bohuslav "Slavek" Kabrda (bkabrda) * Date: 2013-12-05 12:59
I can also reproduce it on Arch Linux. It seems that the bad characters are only output if env variable TERM starts with "xterm".
msg218275 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2014-05-11 15:17
We seem to have gotten bit by this in issue21425.

Dave, is it possible for you to validate the proposed fix? I can't test this under Ubuntu.

Setting priority to "high" as it seems to actually annoy many people:

http://stackoverflow.com/questions/15760712/python-readline-module-prints-escape-character-during-import
http://reinout.vanrees.org/weblog/2009/08/14/readline-invisible-character-hack.html
http://lists.gnu.org/archive/html/bug-readline/2007-08/msg00004.html
msg220336 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2014-06-12 09:21
Attached readline_disable_meta_key.patch: Implement the workaround suggested in (*), but only use the workaround if stdout is not a TTY (ex: output redirected), to limit the risk of regression.

(*) http://lists.gnu.org/archive/html/bug-readline/2011-04/msg00009.html

Extract of the patch:

+    if (!isatty(STDOUT_FILENO)) {
+        /* Issue #19884: Don't try to enable any meta modifier key the terminal
+           claims to support when it is called. On many terminals (ex:
+           xterm-256color), the meta key is used to send eight-bit characters
+           (ANSI sequence "\033[1034h"). */
+        rl_variable_bind ("enable-meta-key", "off");
+    }

This issue becomes very annoying on my Fedora 20. The output of any Mercurial command now starts with "\033.[?1034h" (Mercurial uses Python 2.7). Example:

haypo@smithers$ hg root|hexdump -C
00000000  1b 5b 3f 31 30 33 34 68  2f 68 6f 6d 65 2f 68 61  |.[?1034h/home/ha|
00000010  79 70 6f 2f 70 72 6f 67  2f 70 79 74 68 6f 6e 2f  |ypo/prog/python/|
00000020  64 65 66 61 75 6c 74 0a                           |default.|
00000028

Fedora 18 changed the default TERM environment variable to "xterm-256color":
http://fedoraproject.org/wiki/Features/256_Color_Terminals

Workaround in your application (to run on unpatched Python): set the TERM environment variable to "dummy", or unset this variable.
msg223823 - (view) Author: Roundup Robot (python-dev) Date: 2014-07-24 10:25
New changeset 0177d8a4e82a by Victor Stinner in branch '2.7':
Issue #19884: readline: Disable the meta modifier key if stdout is not a
http://hg.python.org/cpython/rev/0177d8a4e82a

New changeset 6303266beb80 by Victor Stinner in branch '3.4':
Issue #19884: readline: Disable the meta modifier key if stdout is not a
http://hg.python.org/cpython/rev/6303266beb80

New changeset f85a968f9e01 by Victor Stinner in branch 'default':
(Merge 3.4) Issue #19884: readline: Disable the meta modifier key if stdout is
http://hg.python.org/cpython/rev/f85a968f9e01
msg223824 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2014-07-24 10:25
I commited my patch.
msg223864 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2014-07-24 18:02
The test fails on AMD64 OpenIndiana 2.7:

http://buildbot.python.org/all/builders/AMD64%20OpenIndiana%202.7/builds/2354/steps/test/logs/stdio

test test_readline failed -- Traceback (most recent call last):
  File "/export/home/buildbot/64bits/2.7.cea-indiana-amd64/build/Lib/test/test_readline.py", line 52, in test_init
    self.assertEqual(stdout, b'')
AssertionError: '\x1b[?1034h' != ''
msg223865 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2014-07-24 18:24
The changes are also causing segfaults when readline is linked with BSD libedit (the default on OS X) rather than GNU readline:

======================================================================
FAIL: test_init (test.test_readline.TestReadline)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/py/dev/3x/source/Lib/test/test_readline.py", line 52, in test_init
    TERM='xterm-256color')
  File "/py/dev/3x/source/Lib/test/script_helper.py", line 69, in assert_python_ok
    return _assert_python(True, *args, **env_vars)
  File "/py/dev/3x/source/Lib/test/script_helper.py", line 55, in _assert_python
    "stderr follows:\n%s" % (rc, err.decode('ascii', 'ignore')))
AssertionError: Process return code is -11, stderr follows:
Fatal Python error: Segmentation fault

Current thread 0x00007fff75489310 (most recent call first):
  File "<frozen importlib._bootstrap>", line 321 in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 1664 in load_module
  File "<frozen importlib._bootstrap>", line 540 in _check_name_wrapper
  File "<frozen importlib._bootstrap>", line 1110 in _load_backward_compatible
  File "<frozen importlib._bootstrap>", line 1140 in _load_unlocked
  File "<frozen importlib._bootstrap>", line 2175 in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 2186 in _find_and_load
  File "<string>", line 1 in <module>
msg223873 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2014-07-24 19:07
> The changes are also causing segfaults when readline is linked with BSD libedit (the default on OS X) rather than GNU readline:

Oh wow. Do you have an idea of to fix the issue with libedit? Or make
the code condition, only use it with native readline?
msg223879 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2014-07-24 19:44
Currently, readline.c uses #ifdef __APPLE__ to guard libedit-specific code (there is another open issue to generalize libedit support to other platforms).
msg223889 - (view) Author: Roundup Robot (python-dev) Date: 2014-07-24 20:12
New changeset f0ab6f9f0603 by Victor Stinner in branch '2.7':
Issue #19884, readline: calling rl_variable_bind ("enable-meta-key", "off")
http://hg.python.org/cpython/rev/f0ab6f9f0603

New changeset 3f08c1156050 by Victor Stinner in branch '3.4':
Issue #19884, readline: calling rl_variable_bind ("enable-meta-key", "off")
http://hg.python.org/cpython/rev/3f08c1156050

New changeset 0ed1801bf4bd by Victor Stinner in branch 'default':
(Merge 3.4) Issue #19884, readline: calling rl_variable_bind
http://hg.python.org/cpython/rev/0ed1801bf4bd
msg224012 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2014-07-25 23:36
The test fails also on OpenBSD:

http://buildbot.python.org/all/builders/x86%20OpenBSD%205.5%203.x/builds/671/steps/test/logs/stdio

======================================================================
FAIL: test_init (test.test_readline.TestReadline)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/python-builds/3.x.borja-openbsd-x86/build/Lib/test/test_readline.py", line 53, in test_init
    self.assertEqual(stdout, b'')
AssertionError: b'\x1b[?1034h' != b''
msg224484 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2014-08-01 10:35
Oh, the test also fails on the new Red Hat 6 buildbot:

http://buildbot.python.org/all/builders/x86%20RHEL%206%203.x/builds/4358/steps/test/logs/stdio

======================================================================
FAIL: test_init (test.test_readline.TestReadline)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/buildbot/buildarea/3.x.coghlan-redhat/build/Lib/test/test_readline.py", line 53, in test_init
    self.assertEqual(stdout, b'')
AssertionError: b'\x1b[?1034h' != b''
msg225008 - (view) Author: Edd Barrett (Edd.Barrett) Date: 2014-08-07 12:59
This problem shows up on OpenBSD too. It breaks 'hg view' also.
msg225593 - (view) Author: Benjamin Peterson (benjamin.peterson) * (Python committer) Date: 2014-08-20 23:05
It seems this bug was fixed properly by readline in version 6.3; rl_initialize won't put meta codes on the screen. Frankly, I'm surprised distros like Fedora haven't upgraded or patched readline themselves to fix this. Aren't other programs affected?

Turning off "enable-meta-key" isn't great, since it doesn't work on older readline or libedit, and is probably technically a backwards compatibility problem.

I think the best solution would be not call rl_initialize during the initialization of the readline module, but I'm not sure if that could create compatibility problems.
msg226551 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2014-09-08 01:30
Should this be a release blocker for 3.4.2?
msg226558 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2014-09-08 05:57
The test fails on some platforms but it's not a regression.
msg230345 - (view) Author: David Edelsohn (David.Edelsohn) * Date: 2014-10-31 14:42
Issue22773 provides a patch to export Readline version and skip the test for earlier releases of libreadline where turning off enable-meta-key does not work.
msg230622 - (view) Author: Roundup Robot (python-dev) Date: 2014-11-04 13:53
New changeset c4b5a5d44254 by Antoine Pitrou in branch '3.4':
Issue #22773: fix failing test with old readline versions due to issue #19884.
https://hg.python.org/cpython/rev/c4b5a5d44254

New changeset be374b8c40c8 by Antoine Pitrou in branch 'default':
Issue #22773: fix failing test with old readline versions due to issue #19884.
https://hg.python.org/cpython/rev/be374b8c40c8
msg230624 - (view) Author: Roundup Robot (python-dev) Date: 2014-11-04 13:56
New changeset eba6e68e818c by Antoine Pitrou in branch '2.7':
Issue #22773: fix failing test with old readline versions due to issue #19884.
https://hg.python.org/cpython/rev/eba6e68e818c
msg230676 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2014-11-05 14:01
Arfever, Antoine: If buildbots are happy (green), you can close the issue. (I'm answering to your question on IRC ;-))
msg236724 - (view) Author: Maxime Belanger (Maxime Belanger) Date: 2015-02-27 02:03
I think the version check (`readline._READLINE_VERSION < 0x0600`) is incorrect, as the test still fails for me on Mac OS X 10.9 with readline version 6.2 (0x602). Upgrading to 6.3 (0x603) fixes it, though.
msg236734 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2015-02-27 08:59
Mac OS X uses libedit, not libreadline.
msg237154 - (view) Author: Isaac Schwabacher (ischwabacher) * Date: 2015-03-03 17:15
From the OP:

> This was reported at [1] and originally at [2]. The readline maintainer suggests [3] using:
> 
>     rl_variable_bind ("enable-meta-key", "off");
> 
> which was introduced in readline 6.1. Do you think it'd be safe to add the above line?

From 3.4.3 final:

>     @unittest.skipIf(readline._READLINE_VERSION < 0x0600
>                      and "libedit" not in readline.__doc__,
>                      "not supported in this library version")


The test currently fails on readline version 6.0.  The version to test on needs a bump to 0x0610.
msg237155 - (view) Author: Isaac Schwabacher (ischwabacher) * Date: 2015-03-03 17:25
Whoops, that's 0x0601.  Though Maxime gives evidence that the version should in fact be 0x0603.  (Note that while OS X ships with libedit over libreadline, anyone who wants to can install the real thing instead of that pale imitation; the test would have been skipped if Maxime were using libedit.)
msg251656 - (view) Author: Berker Peksag (berker.peksag) * (Python committer) Date: 2015-09-26 16:53
The version check doesn't work because 0ed1801bf4bd added #ifndef __APPLE__ to guard the code, so if you're using readline on OS X, that rl_variable_bind workaround won't work.

There are two alternatives:

1) convert ifndef to ifdef and add a check for the ``using_libedit_emulation`` variable.

    #ifdef __APPLE__
        if (using_libedit_emulation) {
            ...
    ...

looks like a common idiom in readline.c.


2) just skip the test if _READLINE_VERSION < 0x0603
msg252096 - (view) Author: Yogesh Joshi (iyogeshjoshi) Date: 2015-10-02 09:15
I tried applying this patch and its failing and throwing the following error:

patching file Lib/test/test_readline.py
Hunk #1 FAILED at 1.
Hunk #2 FAILED at 43.
2 out of 2 hunks FAILED -- saving rejects to file Lib/test/test_readline.py.rej
patching file Modules/readline.c
Hunk #1 FAILED at 1019.
1 out of 1 hunk FAILED -- saving rejects to file Modules/readline.c.rej

I'm using Mac Os El Captain
msg268780 - (view) Author: Martin Panter (martin.panter) * (Python committer) Date: 2016-06-18 09:20
Yogesh: Victor’s patch has already been applied. What is left to do is another patch that enables Victor’s code on OS X when Gnu Readline is being used, as opposed to the usual Apple Editline.

Also, I think it is valid to update the version check to 6.1. According to <https://cnswww.cns.cwru.edu/php/chet/readline/CHANGES>, enable-meta-key is in 6.1, but not 6.0. On 6.0 the code probably has no effect.
msg268781 - (view) Author: Martin Panter (martin.panter) * (Python committer) Date: 2016-06-18 09:31
Here is a patch to enable the workaround on OS X, and to adjust the test condition to 6.1. It would be nice if someone with OS X and Gnu Readline can confirm that this fixes the problem.
History
Date User Action Args
2016-06-18 09:31:33martin.pantersetfiles: + meta-osx.patch

messages: + msg268781
stage: needs patch -> patch review
2016-06-18 09:20:21martin.pantersetnosy: + martin.panter

messages: + msg268780
stage: needs patch
2015-10-02 09:15:11iyogeshjoshisetnosy: + iyogeshjoshi
messages: + msg252096
2015-09-26 16:53:56berker.peksagsetnosy: + berker.peksag
messages: + msg251656
2015-04-14 21:32:54r.david.murraylinkissue13164 superseder
2015-03-22 05:13:48r.david.murraylinkissue23736 superseder
2015-03-03 17:25:30ischwabachersetmessages: + msg237155
2015-03-03 17:15:30ischwabachersetnosy: + ischwabacher
messages: + msg237154
2015-02-27 08:59:04hayposetmessages: + msg236734
2015-02-27 02:03:31Maxime Belangersetnosy: + Maxime Belanger
messages: + msg236724
2015-02-06 07:47:02vleesetnosy: + vlee
2014-12-20 07:25:36berker.peksaglinkissue23092 superseder
2014-11-05 14:01:10hayposetmessages: + msg230676
2014-11-04 13:56:46python-devsetmessages: + msg230624
2014-11-04 13:53:59python-devsetmessages: + msg230622
2014-10-31 18:33:01Arfreversetnosy: + Arfrever
2014-10-31 14:42:33David.Edelsohnsetnosy: + David.Edelsohn
messages: + msg230345
2014-10-15 19:44:24ned.deilylinkissue22647 superseder
2014-10-14 15:40:09skrahsetnosy: - skrah
2014-10-10 16:56:40jceasetnosy: + jcea
2014-09-08 05:57:28hayposetmessages: + msg226558
2014-09-08 01:30:44ned.deilysetmessages: + msg226551
2014-08-20 23:05:49benjamin.petersonsetnosy: + benjamin.peterson
messages: + msg225593
2014-08-19 18:59:53geoffreyspearsetnosy: + geoffreyspear
2014-08-07 13:13:56rpointelsetnosy: + rpointel
2014-08-07 12:59:03Edd.Barrettsetnosy: + Edd.Barrett
messages: + msg225008
2014-08-01 10:35:39hayposetmessages: + msg224484
2014-07-25 23:36:33hayposetmessages: + msg224012
2014-07-24 20:12:55python-devsetmessages: + msg223889
2014-07-24 19:44:20ned.deilysetmessages: + msg223879
2014-07-24 19:07:13hayposetmessages: + msg223873
2014-07-24 18:24:19ned.deilysetnosy: + ned.deily
messages: + msg223865
2014-07-24 18:02:27hayposetstatus: closed -> open
resolution: fixed ->
messages: + msg223864
2014-07-24 10:25:35hayposetstatus: open -> closed
resolution: fixed
messages: + msg223824
2014-07-24 10:25:02python-devsetnosy: + python-dev
messages: + msg223823
2014-06-12 09:21:51hayposetfiles: + readline_disable_meta_key.patch

nosy: + haypo
messages: + msg220336

keywords: + patch
2014-05-11 15:17:05pitrousetpriority: normal -> high
versions: + Python 3.5, - Python 3.3
nosy: + skrah

messages: + msg218275
2013-12-05 12:59:32bkabrdasetmessages: + msg205293
2013-12-04 15:51:24vajraskysetnosy: + vajrasky
messages: + msg205228
2013-12-04 15:00:53pitrousetnosy: + dmalcolm, pitrou

messages: + msg205221
versions: + Python 3.4
2013-12-04 12:06:16bkabrdacreate