interactive interpreter crashes and test_readline fails on OS X 10.9 Mavericks due to libedit update #62658

ronaldoussoren · 2013-07-15T12:24:21Z

BPO	18458
Nosy	@jcea, @ronaldoussoren, @mkleehammer, @ned-deily
Files	dynamic_editline_workaround_detection.txt issue18458.patch patch_readline_issue_18458.sh

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/ronaldoussoren'
closed_at = <Date 2013-10-12.23:04:06.975>
created_at = <Date 2013-07-15.12:24:20.750>
labels = ['type-crash']
title = 'interactive interpreter crashes and test_readline fails on OS X 10.9 Mavericks due to libedit update'
updated_at = <Date 2021-11-04.14:27:39.838>
user = 'https://github.com/ronaldoussoren'

bugs.python.org fields:

activity = <Date 2021-11-04.14:27:39.838>
actor = 'erlendaasland'
assignee = 'ronaldoussoren'
closed = True
closed_date = <Date 2013-10-12.23:04:06.975>
closer = 'ned.deily'
components = []
creation = <Date 2013-07-15.12:24:20.750>
creator = 'ronaldoussoren'
dependencies = []
files = ['30926', '31634', '32324']
hgrepos = []
issue_num = 18458
keywords = ['patch']
message_count = 24.0
messages = ['193090', '193091', '193467', '193470', '193500', '197113', '197116', '199637', '199638', '201068', '201070', '201087', '201129', '201176', '201184', '201507', '201748', '201772', '201773', '201888', '201902', '244692', '244694', '244701']
nosy_count = 9.0
nosy_names = ['jcea', 'ronaldoussoren', 'nneonneo', 'mkleehammer', 'ned.deily', 'python-dev', 'mrichman', 'Russell.Jurney', 'Etienne Le Sueur']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'crash'
url = 'https://bugs.python.org/issue18458'
versions = []

ronaldoussoren · 2013-07-15T12:24:20Z

Modules/readline.c contains a workaround for a bug in the readline emulation of libedit: that emulation uses a different starting offset for the history than the real readline.

In more recent versions of libedit (such as the one you can now download from <http://www.thrysoee.dk/editline/\>) the bug has been fixed and the workaround causes problems.

The attached patch dynamicly detects if the workaround is necessary.

NOTE: Actually using the trysoee.dk libedit requires a patch to setup.py, that version of libedit does not install libreadline.dylib, only libedit.dylib.

ronaldoussoren · 2013-07-15T12:27:21Z

(set as "crash" because the current workaround causes a crash with recent versions of libedit)

mkleehammer · 2013-07-21T21:08:17Z

Just wanted to chime in that I reported this to Apple and their response was exactly this. (Perhaps Ronald works for Apple?)

Is there a temporary work around in the meantime other than rebuilding everything?

ned-deily · 2013-07-21T21:59:41Z

If you are using an existing binary installation, you could disable readline processing altogether by simply renaming the readline extension module, for example:

cd /Library/Frameworks/Python.framework/Versions/3.3
cd ./lib/python3.3/lib-dynload
sudo mv readline.so readline.so.disabled

Of course, then you will not have interactive command line history and much of the editing features. But it shouldn't crash.

If you don't mind using GNU readline (which is GPL licensed), you *might* be able to install the third-party readline module from PyPI:
https://pypi.python.org/pypi/readline
But extension module building on unreleased systems may be problematic. It may be easier to build on a currently supported system and move it.

As necessary, we will likely need to provide updates for the OS X installers in conjunction with the official release of 10.9.

ronaldoussoren · 2013-07-22T05:43:46Z

Building just the readline extension shouldn't be that hard, using a small custom setup.py script. Then install by manually copying it to the right location.

And no, I don't work for Apple.

ned-deily · 2013-09-06T22:10:12Z

The original patch missed one spot. Here's an updated version. I'm going to apply this to default (for the imminent 3.4.0a2) in case people start running into this on new versions of OS X. It needs to be backported to 2.7 and 3.3; a test would be nice and it could be simplified a bit as part of the generalized support for libedit (bpo-13501).

python-dev · 2013-09-06T22:17:06Z

New changeset b6b8a2171aa3 by Ned Deily in branch 'default':
Issue bpo-18458: Prevent crashes with newer versions of libedit. Its readline
http://hg.python.org/cpython/rev/b6b8a2171aa3

python-dev · 2013-10-12T22:58:13Z

New changeset 1e03fd72e116 by Ned Deily in branch '2.7':
Issue bpo-18458: Prevent crashes with newer versions of libedit. Its readline
http://hg.python.org/cpython/rev/1e03fd72e116

New changeset dfb7cab9f819 by Ned Deily in branch '3.3':
Issue bpo-18458: Prevent crashes with newer versions of libedit. Its readline
http://hg.python.org/cpython/rev/dfb7cab9f819

New changeset 47a7313f079a by Ned Deily in branch 'default':
Issue bpo-18458: merge comments from 3.3
http://hg.python.org/cpython/rev/47a7313f079a

ned-deily · 2013-10-12T23:04:07Z

Backported to 2.7 (for 2.7.6) and 3.3 (for 3.3.3) as well. The current test_readline already covers this case.

nneonneo · 2013-10-23T21:01:50Z

I've attached a fixed readline.so built from today's 2.7 branch, for the convenience of anyone who upgraded to 10.9 and now has crashing Python.

Drop the file in /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload, replacing the original readline.so.

This file is compatible with both 32-bit and 64-bit python.org builds of Python 2.7.5.

ned-deily · 2013-10-23T21:42:54Z

Robert, I appreciate your effort but I do not think it is good practice to recommend downloading and installing binary bits from the bug tracker from an uploaded file. Also, at a cursory examination, the readline.so supplied is not exactly a drop-in replacement for the one supplied by the python.org installer as it is linked with wrong ncurses library. We should have a new set of maintenance releases soon. My preference would be for you to delete that file.

ned-deily · 2013-10-24T03:07:59Z

Until the updated readline module is available in python.org 2.7.6 and 3.3.3 maintenance releases, here is a script that will check python.org versions of Python 2.7.x, 3.2.x, 3.3.x, and 3.4.x for interactive crashes and will patch around the problem as needed. On most systems you will need to run it under an account with administrator privileges.

To use it, open a terminal session in Terminal.app (or other shell), then enter:

curl -O http://bugs.python.org/file32324/patch_readline_issue_18458.sh
openssl sha1 patch_readline_issue_18458.sh
# the digest should be 7cb0ff57820a027dd4ca242eb2418930f8f46b4c
sh ./patch_readline_issue_18458.sh

The output on a 10.9 system with python.org 2.7.5, 3.3.2, and 3.4.0a4 installed should look something like this:

-- running on OS X 10.9
-- 2.7 crashed
-- 2.7 moving readline extension to /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/readline.so.disabled
-- need to patch. Enter password if prompted
Password:
-- 2.7 is now patched
-- 3.2 not found - skipped
-- 3.3 crashed
-- 3.3 moving readline extension to /Library/Frameworks/Python.framework/Versions/3.3/lib/python3.3/lib-dynload/readline.so.disabled
-- 3.3 is now patched
-- 3.4 does not need to be patched - skipped
-- done

warsaw · 2013-10-24T13:43:32Z

Does this affect 2.6? Is there a patch we need to get into 2.6.9 final?

ned-deily · 2013-10-24T20:14:08Z

Barry: yes, 2.6 is affected. See discussion on python-dev.

warsaw · 2013-10-24T20:43:24Z

On Oct 24, 2013, at 08:14 PM, Ned Deily wrote:

Barry: yes, 2.6 is affected. See discussion on python-dev.

Thanks Ned for the background over on python-dev. Unless I hear objections
otherwise, I am not going to apply this to 2.6.

ned-deily · 2013-10-27T23:02:45Z

An update: release candidates for Python 3.3.3 and 2.7.6 are now available, including binary installers for OS X, that include this fix and fully support OS X 10.9.

http://www.python.org/download/releases/3.3.3/

http://www.python.org/download/releases/2.7.6/

mrichman · 2013-10-30T16:55:34Z

I had to do sudo sh ./patch_readline_issue_18458.sh or the patch script itself would cause Python to crash.

After applying this patch, I got the following output, and the problem is still *not* solved for me:

-- running on OS X 10.9
-- 2.7 does not need to be patched - skipped
-- 3.2 not found - skipped
-- 3.3 not found - skipped
-- 3.4 not found - skipped
-- done

ned-deily · 2013-10-30T19:32:25Z

Mark, I'm not sure I understand what you saw but the patch script will cause a Python crash as part of its testing so that is to be expected. You should not have to run the script using sudo. This script also only applies to Pythons installed from python.org (or otherwise installed into /Library/Frameworks). Please check which python you are using. Using whatever command name you enter to start the failing python, try the following (I'll assume you use "python2.7"):

type python2.7
which python2.7
python2.7 -c "import sys;print(sys.version)"
python2.7 -c "import sys;print(sys.executable)"

The value for sys.executable should be:
/Library/Frameworks/Python.framework/Versions/2.7/Resources/Python.app/Contents/MacOS/Python

In any case, you could manually rename _readline.so as shown in one of the earlier messages (substituting "2.7" for "3.3"). Or you could install 2.7.6rc1.

mrichman · 2013-10-30T19:37:17Z

My mistake. I'm using
/System/Library/Frameworks/Python.framework/Versions/2.7/Resources/Python.app/Contents/MacOS/Python

--
*Mark Richman*
web: www.markrichman.com
email: mark@markrichman.com
tel: (954) 234-9049
http://www.linkedin.com/in/mrichman

On Wed, Oct 30, 2013 at 3:32 PM, Ned Deily <report@bugs.python.org> wrote:

Ned Deily added the comment:

Mark, I'm not sure I understand what you saw but the patch script will
cause a Python crash as part of its testing so that is to be expected. You
should not have to run the script using sudo. This script also only
applies to Pythons installed from python.org (or otherwise installed into
/Library/Frameworks). Please check which python you are using. Using
whatever command name you enter to start the failing python, try the
following (I'll assume you use "python2.7"):

type python2.7
which python2.7
python2.7 -c "import sys;print(sys.version)"
python2.7 -c "import sys;print(sys.executable)"

The value for sys.executable should be:

/Library/Frameworks/Python.framework/Versions/2.7/Resources/Python.app/Contents/MacOS/Python

In any case, you could manually rename _readline.so as shown in one of the
earlier messages (substituting "2.7" for "3.3"). Or you could install
2.7.6rc1.

----------

Python tracker <report@bugs.python.org>
<http://bugs.python.org/issue18458\>

RussellJurney · 2013-11-01T09:35:12Z

I have installed 2.7.6 RC1 on OS X 10.9, and this bug still exists. See http://stackoverflow.com/questions/19722580/segfault-11-with-pandas-with-python-v2-7-6-rc1-on-mac-os-x-10-9

nneonneo · 2013-11-01T14:36:27Z

Russell, that's not related to readline. Please file a new bug report.

EtienneLeSueur · 2015-06-02T21:32:50Z

So what's the resolution here? All I see is a script that disables the readline module depending on whether the interactive session crashes...

Has a fix been pushed into the 3.4 release such that linking the readline module against editline results in a working interactive interpreter?

bitdancer · 2015-06-02T21:39:24Z

Yes, this is fixed, as the issue resolution says. If you are curious about the fix, follow the links to the commits starting in msg197116.

EtienneLeSueur · 2015-06-02T22:04:05Z

Thanks, I had missed that message.

On 6/2/15 4:39 PM, R. David Murray wrote:

R. David Murray added the comment:

Yes, this is fixed, as the issue resolution says. If you are curious about the fix, follow the links to the commits starting in msg197116.

----------
nosy: +r.david.murray

Python tracker <report@bugs.python.org>
<http://bugs.python.org/issue18458\>

ronaldoussoren self-assigned this Jul 15, 2013

ronaldoussoren added the type-crash A hard crash of the interpreter, possibly with a core dump label Jul 15, 2013

ned-deily changed the title ~~libedit history offset workaround~~ interactive interpreter crashes and test_readline fails with newer versions of libedit Oct 12, 2013

ned-deily closed this as completed Oct 12, 2013

ned-deily changed the title ~~interactive interpreter crashes and test_readline fails with newer versions of libedit~~ interactive interpreter crashes and test_readline fails on OS X 10.9 Mavericks due to libedit update Oct 24, 2013

ahmedsayeed1982 mannequin added the topic-email label Nov 4, 2021

ahmedsayeed1982 mannequin added the 3.11 only security fixes label Nov 4, 2021

erlend-aasland removed topic-email 3.11 only security fixes labels Nov 4, 2021

ezio-melotti transferred this issue from another repository Apr 10, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

interactive interpreter crashes and test_readline fails on OS X 10.9 Mavericks due to libedit update #62658

interactive interpreter crashes and test_readline fails on OS X 10.9 Mavericks due to libedit update #62658

ronaldoussoren commented Jul 15, 2013

ronaldoussoren commented Jul 15, 2013

ronaldoussoren commented Jul 15, 2013

mkleehammer mannequin commented Jul 21, 2013

ned-deily commented Jul 21, 2013

ronaldoussoren commented Jul 22, 2013

ned-deily commented Sep 6, 2013

python-dev mannequin commented Sep 6, 2013

python-dev mannequin commented Oct 12, 2013

ned-deily commented Oct 12, 2013

nneonneo mannequin commented Oct 23, 2013

ned-deily commented Oct 23, 2013

ned-deily commented Oct 24, 2013

warsaw commented Oct 24, 2013

ned-deily commented Oct 24, 2013

warsaw commented Oct 24, 2013

ned-deily commented Oct 27, 2013

mrichman mannequin commented Oct 30, 2013

ned-deily commented Oct 30, 2013

mrichman mannequin commented Oct 30, 2013

RussellJurney mannequin commented Nov 1, 2013

nneonneo mannequin commented Nov 1, 2013

EtienneLeSueur mannequin commented Jun 2, 2015

bitdancer commented Jun 2, 2015

EtienneLeSueur mannequin commented Jun 2, 2015

interactive interpreter crashes and test_readline fails on OS X 10.9 Mavericks due to libedit update #62658

interactive interpreter crashes and test_readline fails on OS X 10.9 Mavericks due to libedit update #62658

Comments

ronaldoussoren commented Jul 15, 2013

ronaldoussoren commented Jul 15, 2013

ronaldoussoren commented Jul 15, 2013

mkleehammer mannequin commented Jul 21, 2013

ned-deily commented Jul 21, 2013

ronaldoussoren commented Jul 22, 2013

ned-deily commented Sep 6, 2013

python-dev mannequin commented Sep 6, 2013

python-dev mannequin commented Oct 12, 2013

ned-deily commented Oct 12, 2013

nneonneo mannequin commented Oct 23, 2013

ned-deily commented Oct 23, 2013

ned-deily commented Oct 24, 2013

warsaw commented Oct 24, 2013

ned-deily commented Oct 24, 2013

warsaw commented Oct 24, 2013

ned-deily commented Oct 27, 2013

mrichman mannequin commented Oct 30, 2013

ned-deily commented Oct 30, 2013

mrichman mannequin commented Oct 30, 2013

RussellJurney mannequin commented Nov 1, 2013

nneonneo mannequin commented Nov 1, 2013

EtienneLeSueur mannequin commented Jun 2, 2015

bitdancer commented Jun 2, 2015

EtienneLeSueur mannequin commented Jun 2, 2015