New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Segfault when readline history is more then 2 * history size #74040
Comments
GNU readline let the user select limit the history size by setting: $ cat ~/.inputrc
set history-size 1000 So I cooked this test script: $ cat history.py
from __future__ import print_function
import readline readline.read_history_file(".history")
print("current_history_length", readline.get_current_history_length())
print("history_length", readline.get_history_length())
print("history_get_item(1)", readline.get_history_item(1))
print("history_get_item(1000)", readline.get_history_item(1000))
input()
readline.write_history_file(".history") And this history file generator: $ cat make-history
for i in range(2000):
print("%04d" % i) Generating .history file with 2000 entries: Finally running the test script: $ python3 history.py
current_history_length 1000
history_length -1
history_get_item(1) None
history_get_item(1000) None
please crash
Segmentation fault (core dumped) So we have few issues here:
Running with gdb we see: $ gdb python3
GNU gdb (GDB) Fedora 7.12.1-46.fc25
Copyright (C) 2017 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from python3...Reading symbols from /usr/lib/debug/usr/libexec/system-python.debug...done.
done. (gdb) run history.py Program received signal SIGSEGV, Segmentation fault. (gdb) list So we assume that history_get(length) returns non-null In 2 other usages in Modules/readline.c, we validate If we change the .history contents to 1999 lines, we get: $ python3 make-history | head -1999 > .history
$ python3 history.py
current_history_length 1000
history_length -1
history_get_item(1) None
history_get_item(1000) 0999
crash?
$ wc -l .history
1000 .history
$ head -1 .history
1000
$ tail -1 .history
crash? So now it does not crash, but item 1 is still None. Trying again with history file with 1000 entries: $ python3 make-history | head -1000 > .history
$ python3 history.py
current_history_length 1000
history_length -1
history_get_item(1) 0000
history_get_item(1000) 0999
looks fine!
$ wc -l .history
1000 .history
$ head -1 history
head: cannot open 'history' for reading: No such file or directory
$ head -1 .history
0001
$ tail -1 .history
looks fine! Finally trying with 1001 items: $ python3 make-history | head -1001 > .history
$ python3 history.py
current_history_length 1000
history_length -1
history_get_item(1) None
history_get_item(1000) 0999 And item 1 is wrong. I got same results with python 2.7, 3.5 and master The root cause seems to be a readline bug when history |
The fix LGTM. Any chance to write a test? And please add an entry in Misc/NEWS. |
Sure, I'll add news entry and tests. |
Gnu Readline comes includes its own documentation (e.g. /usr/share/info/history.info.gz on my computer). It is also at <https://cnswww.cns.cwru.edu/php/chet/readline/history.html\>. Perhaps the history_base value is relevant; see some of the comments starting at <https://bugs.python.org/issue6953#msg100466\>. It would be interesting to see if Apple Editline is affected. According to the comment in the get_history_item function, history_get might crash before returning the null pointer. Is there some other workaround that avoids calling history_get? |
I think the issue can be solved in readline or in the code using it, but I don't have more time to dig into this, and I think that python should not crash in this case. I don't have an environment to test Apple editline, so I cannot test this issue. The PR includes a test case now, running the new test on OS X will tell us if this is the case. |
I suspect the test won’t be effective with Editline (either fail, or silently pass without testing the bug). From memory Editline uses an ~/.editrc file, not INPUTRC, with different syntax and configuration. |
This issue does not exist on OS X 10.11.6 (latest my old mac can install). I tested using .editrc file: $ cat ~/.editrc
history size 5 With history file with 10 items that crashes on Linux using GNU readline. This settings is ignored, adding items to the history file without truncating it to 5 items. I tested also truncating the size using readline.set_history_size(). It works correctly, but this means every application need to implement its own readline configuration, instead of reusing the system readline configuration. So this bug is relevant only to GNU readline, need to skip this test when using libedit. |
The test fails on AMD64 FreeBSD 10.x Shared 3.x: ====================================================================== Traceback (most recent call last):
File "/usr/home/buildbot/python/3.x.koobs-freebsd10/build/Lib/test/test_readline.py", line 247, in test_history_size
self.assertEqual(len(lines), history_size)
AssertionError: 21 != 10 |
Similar failure on x86 Tiger 3.x. Maybe we need to skip the test on old macOS and old FreeBSD versions? Maybe it's related to the libncurses version? http://buildbot.python.org/all/builders/x86%20Tiger%203.x/builds/908/steps/test/logs/stdio ====================================================================== Traceback (most recent call last):
File "/Users/db3l/buildarea/3.x.bolen-tiger/build/Lib/test/test_readline.py", line 247, in test_history_size
self.assertEqual(len(lines), history_size)
AssertionError: 21 != 10 |
This one is interesting. I thought we don't have OS X buildbots with readline installed. I would prefer skipping the test based on readline version installed. Side note: I think we should print readline, sqlite3 etc. versions in cpython/Lib/test/libregrtest/main.py Line 421 in fae8f4a
|
I've opened PR 2618 to print readline version and implementation in regrtest's display_header() method. |
The failures looks like libedit failures on OS X, where history size is ignored. The test is skipped if is_editline is set, we should probably skip on these platforms too. |
AMD64 FreeBSD 9.x 3.x, test_history_size() fails: readline version: 0x502 AMD64 FreeBSD 10.x Shared 3.x, test_history_size() fails: readline version: 0x502 x86 Tiger 3.x, test_history_size() fails: readline version: 0x501 -- My Linux box, test_history_size() pass: readline version: 0x603 |
So we have version 0x502 without libedit emulation succeeding on I think we are missing something, or maybe the libedit check is wrong. We need results from all builders to do something with this. I think Maybe switch the failing test to run only on Linux for now? |
test_history_size() fails on FreeBSD 9.x too: ====================================================================== Traceback (most recent call last):
File "/usr/home/buildbot/python/3.x.koobs-freebsd9/build/Lib/test/test_readline.py", line 263, in test_history_size
self.assertEqual(len(lines), history_size)
AssertionError: 21 != 10 http://buildbot.python.org/all/builders/AMD64%20FreeBSD%209.x%203.x/builds/316/steps/test/logs/stdio |
According to https://cnswww.cns.cwru.edu/php/chet/readline/CHANGES, the history-size setting was added in readline 6.0:
The only thing we need to do is skip the test if readline version is older than 6.0. We discussed this with Nir on IRC, and he will send another PR to tweak the test. |
Ok, I think we can finally close this one :) Thank you, everyone! |
FYI I added the test.pythoninfo utility as a follow-up of this issue to log many informations to debug Python, not only the readline version: see bpo-30871. |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
bugs.python.org fields:
The text was updated successfully, but these errors were encountered: