New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Importing readline produces erroneous output #64083
Comments
A simple reproducer: python -c 'import readline' | xxd This was reported at [1] and originally at [2]. The readline maintainer suggests [3] using: rl_variable_bind ("enable-meta-key", "off"); which was introduced in readline 6.1. Do you think it'd be safe to add the above line? Thanks. [1] https://bugzilla.redhat.com/show_bug.cgi?id=880393 |
I can't reproduce under Ubuntu 13.10. Is this Red Hat-specific? (according to Dave, """This is a readline bug; it looks like it should not emit those characters when stdout is not a tty""") |
Reproducible under Fedora 18. $ ./python -c "import readline" | hexdump -C
00000000 1b 5b 3f 31 30 33 34 68 |.[?1034h|
00000008
$ TERM=dumb ./python -c "import readline" | hexdump -C |
I can also reproduce it on Arch Linux. It seems that the bad characters are only output if env variable TERM starts with "xterm". |
We seem to have gotten bit by this in bpo-21425. Dave, is it possible for you to validate the proposed fix? I can't test this under Ubuntu. Setting priority to "high" as it seems to actually annoy many people: http://stackoverflow.com/questions/15760712/python-readline-module-prints-escape-character-during-import |
Attached readline_disable_meta_key.patch: Implement the workaround suggested in (*), but only use the workaround if stdout is not a TTY (ex: output redirected), to limit the risk of regression. (*) http://lists.gnu.org/archive/html/bug-readline/2011-04/msg00009.html Extract of the patch: + if (!isatty(STDOUT_FILENO)) { This issue becomes very annoying on my Fedora 20. The output of any Mercurial command now starts with "\033.[?1034h" (Mercurial uses Python 2.7). Example: haypo@smithers$ hg root|hexdump -C Fedora 18 changed the default TERM environment variable to "xterm-256color": Workaround in your application (to run on unpatched Python): set the TERM environment variable to "dummy", or unset this variable. |
New changeset 0177d8a4e82a by Victor Stinner in branch '2.7': New changeset 6303266beb80 by Victor Stinner in branch '3.4': New changeset f85a968f9e01 by Victor Stinner in branch 'default': |
I commited my patch. |
The test fails on AMD64 OpenIndiana 2.7: http://buildbot.python.org/all/builders/AMD64%20OpenIndiana%202.7/builds/2354/steps/test/logs/stdio test test_readline failed -- Traceback (most recent call last):
File "/export/home/buildbot/64bits/2.7.cea-indiana-amd64/build/Lib/test/test_readline.py", line 52, in test_init
self.assertEqual(stdout, b'')
AssertionError: '\x1b[?1034h' != '' |
The changes are also causing segfaults when readline is linked with BSD libedit (the default on OS X) rather than GNU readline: ====================================================================== Traceback (most recent call last):
File "/py/dev/3x/source/Lib/test/test_readline.py", line 52, in test_init
TERM='xterm-256color')
File "/py/dev/3x/source/Lib/test/script_helper.py", line 69, in assert_python_ok
return _assert_python(True, *args, **env_vars)
File "/py/dev/3x/source/Lib/test/script_helper.py", line 55, in _assert_python
"stderr follows:\n%s" % (rc, err.decode('ascii', 'ignore')))
AssertionError: Process return code is -11, stderr follows:
Fatal Python error: Segmentation fault Current thread 0x00007fff75489310 (most recent call first): |
Oh wow. Do you have an idea of to fix the issue with libedit? Or make |
Currently, readline.c uses #ifdef __APPLE__ to guard libedit-specific code (there is another open issue to generalize libedit support to other platforms). |
New changeset f0ab6f9f0603 by Victor Stinner in branch '2.7': New changeset 3f08c1156050 by Victor Stinner in branch '3.4': New changeset 0ed1801bf4bd by Victor Stinner in branch 'default': |
The test fails also on OpenBSD: http://buildbot.python.org/all/builders/x86%20OpenBSD%205.5%203.x/builds/671/steps/test/logs/stdio ====================================================================== Traceback (most recent call last):
File "/home/python-builds/3.x.borja-openbsd-x86/build/Lib/test/test_readline.py", line 53, in test_init
self.assertEqual(stdout, b'')
AssertionError: b'\x1b[?1034h' != b'' |
Oh, the test also fails on the new Red Hat 6 buildbot: http://buildbot.python.org/all/builders/x86%20RHEL%206%203.x/builds/4358/steps/test/logs/stdio ====================================================================== Traceback (most recent call last):
File "/home/buildbot/buildarea/3.x.coghlan-redhat/build/Lib/test/test_readline.py", line 53, in test_init
self.assertEqual(stdout, b'')
AssertionError: b'\x1b[?1034h' != b'' |
This problem shows up on OpenBSD too. It breaks 'hg view' also. |
It seems this bug was fixed properly by readline in version 6.3; rl_initialize won't put meta codes on the screen. Frankly, I'm surprised distros like Fedora haven't upgraded or patched readline themselves to fix this. Aren't other programs affected? Turning off "enable-meta-key" isn't great, since it doesn't work on older readline or libedit, and is probably technically a backwards compatibility problem. I think the best solution would be not call rl_initialize during the initialization of the readline module, but I'm not sure if that could create compatibility problems. |
Should this be a release blocker for 3.4.2? |
The test fails on some platforms but it's not a regression. |
bpo-22773 provides a patch to export Readline version and skip the test for earlier releases of libreadline where turning off enable-meta-key does not work. |
New changeset c4b5a5d44254 by Antoine Pitrou in branch '3.4': New changeset be374b8c40c8 by Antoine Pitrou in branch 'default': |
New changeset eba6e68e818c by Antoine Pitrou in branch '2.7': |
Arfever, Antoine: If buildbots are happy (green), you can close the issue. (I'm answering to your question on IRC ;-)) |
I think the version check ( |
Mac OS X uses libedit, not libreadline. |
From the OP:
From 3.4.3 final:
The test currently fails on readline version 6.0. The version to test on needs a bump to 0x0610. |
Whoops, that's 0x0601. Though Maxime gives evidence that the version should in fact be 0x0603. (Note that while OS X ships with libedit over libreadline, anyone who wants to can install the real thing instead of that pale imitation; the test would have been skipped if Maxime were using libedit.) |
The version check doesn't work because 0ed1801bf4bd added #ifndef __APPLE__ to guard the code, so if you're using readline on OS X, that rl_variable_bind workaround won't work. There are two alternatives:
#ifdef __APPLE__
if (using_libedit_emulation) {
...
... looks like a common idiom in readline.c.
|
I tried applying this patch and its failing and throwing the following error: patching file Lib/test/test_readline.py I'm using Mac Os El Captain |
Yogesh: Victor’s patch has already been applied. What is left to do is another patch that enables Victor’s code on OS X when Gnu Readline is being used, as opposed to the usual Apple Editline. Also, I think it is valid to update the version check to 6.1. According to <https://cnswww.cns.cwru.edu/php/chet/readline/CHANGES\>, enable-meta-key is in 6.1, but not 6.0. On 6.0 the code probably has no effect. |
Here is a patch to enable the workaround on OS X, and to adjust the test condition to 6.1. It would be nice if someone with OS X and Gnu Readline can confirm that this fixes the problem. |
I'm still seeing this test failure on EL6.8 with python 3.4.5: [268/391] test_readline Traceback (most recent call last):
File "/builddir/build/BUILD/Python-3.4.5/Lib/test/test_readline.py", line 57, in test_init
self.assertEqual(stdout, b'')
AssertionError: b'\x1b[?1034h' != b'' Ran 2 tests in 0.111s With readline-devel-6.0-4.el6.x86_64. |
Updating the version check to 6.1 as in the patch from Martin certainly avoids the failing test. |
Thanks, I can try to commit the version fix part of the patch when I get a chance. What is EL6.8, is that a Red Hat (Enterprise Linux) thing? |
New changeset 55ec5fdc3099 by Martin Panter in branch '2.7': New changeset 782d9b5d2e90 by Martin Panter in branch '3.5': New changeset 72e034afeb55 by Martin Panter in branch 'default': |
I committed my patch in full, so hopefully the Gnu Readline situation on OS X is also improved. Original fixes went into 3.4, but my patch only went into 3.5+. |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
bugs.python.org fields:
The text was updated successfully, but these errors were encountered: