-
-
Notifications
You must be signed in to change notification settings - Fork 29.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
readline: Wrong tab completion scope indices in Unicode terminals #60386
Comments
Tab completion in the readline module does not seem to work well with Unicode terminals. The get_line_buffer function converts the line buffer to the str type (which are Unicode strings in Python 3), but the indices returned by get_begidx and get_endidx are not adjusted with respect to possible wide characters in the buffer, and hence are not very useful. The documentation is a bit vague on the index functions, but I think they should be relative to code points, regardless of the encoding used by the C library. The suggested correction is attached. My second point of complaint is related to the use of PyUnicode_FromString in the module. The strings returned by the readline library use the current locale encoding, which is not necessarily UTF-8. I wonder if PyUnicode_DecodeLocale should be used instead for more portable code. |
@Kaarle please accept our apologies for the delay in getting back to you. Can one of our unicode gurus comment please. |
Yes, the readline module is broken in Python 3. Underlying C library operates C strings and use locale-depended C functions to split it on Unicode characters. The Python wrapper always uses the UTF-8 encoding for converting between Python strings and C strings. It works only on UTF-8 locales. get_begidx() and get_endidx() don't correctly work at all for non-ASCII data. We should use locale encoding for converting. Proposed patch makes the readline module to use locale depending coding functions instead of default UTF-8. It also corrects indices for get_begidx() and get_endidx(). |
I’m a bit worried about flex_complete() covering up errors. If I add calls to PyErr_WriteUnraisable(), I can see both the s1 and s2 decodes failing. Input the following line:
and then move the cursor back one place, so it is directly after the “x”, but not after the semicolon (;). Then press Tab. Both errors are: ValueError: embedded null byte It looks like the reason is that PyUnicode_DecodeLocaleAndSize() requires that str[len] is a null character (the error message is misleading). It seems the len parameter is mainly there to verify that there are no embedded null characters, i.e. you cannot use it to give a truncated string. It looks like Py_DecodeLocale() is used underneath; maybe it is simpler to call that directly. But it does not solve the string truncating problem. A test case (perhaps using a pseudo-terminal) might also help pick this kind of thing up, if we can’t report errors any other way. |
Attached patch adds a possible test, and fixes the truncation problem I mentioned above. I tried testing set_completer_delims() with a UTF-8 locale, but I suspect Gnu Readline does not support it. I called set_completer_delims("\xF6"), which encodes as C3 B6, but it seems to be breaking any UTF-8 sequence in half at a C3 byte. In other words, it is treating the delimiter list as a list of bytes, not code points. So I changed to an ASCII delimiter. |
V3 finishes what I started in v2:
I tried to test it with Editline on Linux (using my patch for bpo-13501). There seem to be many quirks with my version of Editline, some of which are not easy to work around:
I suspect Apple has patched their version of Editline, but if these quirks exist on Apple as well, it might be simplest to skip the test for Editline. |
Thank you for your comments and updated patches, and especially for tests Martin. Added some comments on Rietveld. |
Martin? |
I updated the patch to fix the error handling and memory leak. it also now skips the test in case the locale cannot encode the test data. |
LGTM. But the test fails with PYTHONIOENCODING=ascii. $ PYTHONIOENCODING=ascii ./python -m test test_readline
Run tests sequentially
0:00:00 [1/1] test_readline
test test_readline failed -- Traceback (most recent call last):
File "/home/serhiy/py/cpython/Lib/test/test_readline.py", line 194, in test_nonascii
self.assertIn(b"result " + expected + b"\r\n", output)
AssertionError: b"result '[\\xefnserted]|t\\xebxt[after]'\r\n" not found in bytearray(b'^A^B^B^B^B^B^B^B\t\tx\t\r\n[\xc3\xafnserted]|t\xc3\xab[after]\x08\x08\x08\x08\x08\x08\x08text \'t\\xeb\'\r\nline \'[\\xefnserted]|t\\xeb[after]\'\r\nindexes 11 13\r\n\x07text \'t\\xeb\'\r\nline \'[\\xefnserted]|t\\xeb[after]\'\r\nindexes 11 13\r\nsubstitution \'t\\xeb\'\r\nmatches [\'t\\xebnt\', \'t\\xebxt\']\r\nx[after]\x08\x08\x08\x08\x08\x08\x08t[after]\x08\x08\x08\x08\x08\x08\x08\r\nTraceback (most recent call last):\r\n File "<string>", line 39, in <module>\r\nUnicodeDecodeError: \'ascii\' codec can\'t decode byte 0xc3 in position 1: ordinal not in range(128)\r\n') This is minor problem, since buildbots rarely configured with PYTHONIOENCODING=ascii. |
I get two other test suite failures if I set PYTHONIOENCODING, so I am not going to bother addressing this in test_readline :) FAIL: test_forced_io_encoding (test.test_capi.EmbeddingTests) |
New changeset 5122b3465a38 by Martin Panter in branch '3.5': New changeset 2ae2657d87a6 by Martin Panter in branch 'default': |
Failures from AMD64 Snow Leop buildbots: ====================================================================== Traceback (most recent call last):
File "/Users/buildbot/buildarea/3.5.murray-snowleopard/build/Lib/test/test_readline.py", line 102, in test_nonascii_history
self.assertEqual(readline.get_history_item(1), "entrée 1")
AssertionError: None != 'entrée 1' ====================================================================== Traceback (most recent call last):
File "/Users/buildbot/buildarea/3.5.murray-snowleopard/build/Lib/test/test_readline.py", line 174, in test_nonascii
self.assertIn(b"line '[\\xefnserted]|t\\xeb[after]'\r\n", output)
AssertionError: b"line '[\\xefnserted]|t\\xeb[after]'\r\n" not found in bytearray(b"^A^B^B^B^B^B^B^B\t\tx\t\r\n|t\xc3\xab[after]\x08\x08\x08\x08\x08\x08\x08text \'t\\xeb\'\r\nline \'|t\\xeb[after]\'\r\nindexes 1 3\r\n\x07text \'t\\xeb\'\r\nline \'|t\\xeb[after]\'\r\nindexes 1 3\r\n\r\nt\xc3\xabxt t\xc3\xabnt \r\n\r\r\n|t\xc3\xab[after]\r|t\xc3\xabx[after]\r|t\xc3\xabxt[after]\r|t\xc3\xabxt\r\nresult \'|t\\xebxt[after]\'\r\nhistory \'|t\\xebxt[after]\'\r\n") |
New changeset 0794bbfceec6 by Martin Panter in branch '3.5': New changeset a1ca9c0ebc05 by Martin Panter in branch 'default': |
Also need a fix for missing set_pre_input_hook() on the AIX buildbot: ====================================================================== Traceback (most recent call last):
File "/home/shager/cpython-buildarea/3.5.edelsohn-aix-ppc64/build/Lib/test/test_readline.py", line 173, in test_nonascii
self.assertIn(b"text 't\\xeb'\r\n", output)
AssertionError: b"text 't\\xeb'\r\n" not found in bytearray(b'\x01\x02\x02\x02\x02\x02\x02\x02 x \r\n\x1b[?1034hTraceback (most recent call last):\r\n File "<string>", line 18, in <module>\r\nAttributeError: module \'readline\' has no attribute \'set_pre_input_hook\'\r\n') |
New changeset 005cab4f5629 by Martin Panter in branch '3.5': New changeset c4dd384ee3fa by Martin Panter in branch 'default': New changeset cff695a0b449 by Martin Panter in branch '2.7': |
New changeset ef234a5c5817 by Martin Panter in branch '3.5': New changeset 241bae60cef8 by Martin Panter in branch 'default': |
Think I got all the bugs fixed here. |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
bugs.python.org fields:
The text was updated successfully, but these errors were encountered: