classification
Title: interactive interpreter crashes and test_readline fails on OS X 10.9 Mavericks due to libedit update
Type: crash Stage: resolved
Components: Versions: Python 3.4, Python 3.3, Python 2.7
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: ronaldoussoren Nosy List: Russell.Jurney, barry, jcea, mkleehammer, mrichman, ned.deily, nneonneo, python-dev, ronaldoussoren
Priority: normal Keywords: patch

Created on 2013-07-15 12:24 by ronaldoussoren, last changed 2013-11-01 14:36 by nneonneo. This issue is now closed.

Files
File name Uploaded Description Edit
dynamic_editline_workaround_detection.txt ronaldoussoren, 2013-07-15 12:24 review
issue18458.patch ned.deily, 2013-09-06 22:10 review
patch_readline_issue_18458.sh ned.deily, 2013-10-24 02:47
Messages (21)
msg193090 - (view) Author: Ronald Oussoren (ronaldoussoren) * (Python committer) Date: 2013-07-15 12:24
Modules/readline.c contains a workaround for a bug in the readline emulation of libedit: that emulation uses a different starting offset for the history than the real readline.

In more recent versions of libedit (such as the one you can now download from <http://www.thrysoee.dk/editline/>) the bug has been fixed and the workaround causes problems.

The attached patch dynamicly detects if the workaround is necessary.

NOTE: Actually using the trysoee.dk libedit requires a patch to setup.py, that version of libedit does not install libreadline.dylib, only libedit.dylib.
msg193091 - (view) Author: Ronald Oussoren (ronaldoussoren) * (Python committer) Date: 2013-07-15 12:27
(set as "crash" because the current workaround causes a crash with recent versions of libedit)
msg193467 - (view) Author: Michael Kleehammer (mkleehammer) Date: 2013-07-21 21:08
Just wanted to chime in that I reported this to Apple and their response was exactly this.  (Perhaps Ronald works for Apple?)

Is there a temporary work around in the meantime other than rebuilding everything?
msg193470 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2013-07-21 21:59
If you are using an existing binary installation, you could disable readline processing altogether by simply renaming the readline extension module, for example:

  cd /Library/Frameworks/Python.framework/Versions/3.3
  cd ./lib/python3.3/lib-dynload
  sudo mv readline.so readline.so.disabled

Of course, then you will not have interactive command line history and much of the editing features.  But it shouldn't crash.

If you don't mind using GNU readline (which is GPL licensed), you *might* be able to install the third-party readline module from PyPI:
   https://pypi.python.org/pypi/readline
But extension module building on unreleased systems may be problematic.  It may be easier to build on a currently supported system and move it.

As necessary, we will likely need to provide updates for the OS X installers in conjunction with the official release of 10.9.
msg193500 - (view) Author: Ronald Oussoren (ronaldoussoren) * (Python committer) Date: 2013-07-22 05:43
Building just the readline extension shouldn't be that hard, using a small custom setup.py script. Then install by manually copying it to the right location.

And no, I don't work for Apple.
msg197113 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2013-09-06 22:10
The original patch missed one spot.  Here's an updated version. I'm going to apply this to default (for the imminent 3.4.0a2) in case people start running into this on new versions of OS X.  It needs to be backported to 2.7 and 3.3; a test would be nice and it could be simplified a bit as part of the generalized support for libedit (Issue13501).
msg197116 - (view) Author: Roundup Robot (python-dev) Date: 2013-09-06 22:17
New changeset b6b8a2171aa3 by Ned Deily in branch 'default':
Issue #18458: Prevent crashes with newer versions of libedit.  Its readline
http://hg.python.org/cpython/rev/b6b8a2171aa3
msg199637 - (view) Author: Roundup Robot (python-dev) Date: 2013-10-12 22:58
New changeset 1e03fd72e116 by Ned Deily in branch '2.7':
Issue #18458: Prevent crashes with newer versions of libedit.  Its readline
http://hg.python.org/cpython/rev/1e03fd72e116

New changeset dfb7cab9f819 by Ned Deily in branch '3.3':
Issue #18458: Prevent crashes with newer versions of libedit.  Its readline
http://hg.python.org/cpython/rev/dfb7cab9f819

New changeset 47a7313f079a by Ned Deily in branch 'default':
Issue #18458: merge comments from 3.3
http://hg.python.org/cpython/rev/47a7313f079a
msg199638 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2013-10-12 23:04
Backported to 2.7 (for 2.7.6) and 3.3 (for 3.3.3) as well.  The current test_readline already covers this case.
msg201068 - (view) Author: Robert Xiao (nneonneo) * Date: 2013-10-23 21:01
I've attached a fixed readline.so built from today's 2.7 branch, for the convenience of anyone who upgraded to 10.9 and now has crashing Python.

Drop the file in /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload, replacing the original readline.so.

This file is compatible with both 32-bit and 64-bit python.org builds of Python 2.7.5.
msg201070 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2013-10-23 21:42
Robert, I appreciate your effort but I do not think it is good practice to recommend downloading and installing binary bits from the bug tracker from an uploaded file.  Also, at a cursory examination, the readline.so supplied is not exactly a drop-in replacement for the one supplied by the python.org installer as it is linked with wrong ncurses library.  We should have a new set of maintenance releases soon.  My preference would be for you to delete that file.
msg201087 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2013-10-24 03:07
Until the updated readline module is available in python.org 2.7.6 and 3.3.3 maintenance releases, here is a script that will check python.org versions of Python 2.7.x, 3.2.x, 3.3.x, and 3.4.x for interactive crashes and will patch around the problem as needed.  On most systems you will need to run it under an account with administrator privileges.

To use it, open a terminal session in Terminal.app (or other shell), then enter:

curl -O http://bugs.python.org/file32324/patch_readline_issue_18458.sh
openssl sha1 patch_readline_issue_18458.sh
# the digest should be 7cb0ff57820a027dd4ca242eb2418930f8f46b4c
sh ./patch_readline_issue_18458.sh


The output on a 10.9 system with python.org 2.7.5, 3.3.2, and 3.4.0a4 installed should look something like this:

 -- running on OS X 10.9
 -- 2.7 crashed
 -- 2.7 moving readline extension to /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/readline.so.disabled
 -- need to patch. Enter password if prompted
Password:
 -- 2.7 is now patched
 -- 3.2 not found - skipped
 -- 3.3 crashed
 -- 3.3 moving readline extension to /Library/Frameworks/Python.framework/Versions/3.3/lib/python3.3/lib-dynload/readline.so.disabled
 -- 3.3 is now patched
 -- 3.4 does not need to be patched - skipped
 -- done
msg201129 - (view) Author: Barry A. Warsaw (barry) * (Python committer) Date: 2013-10-24 13:43
Does this affect 2.6?  Is there a patch we need to get into 2.6.9 final?
msg201176 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2013-10-24 20:14
Barry: yes, 2.6 is affected. See discussion on python-dev.
msg201184 - (view) Author: Barry A. Warsaw (barry) * (Python committer) Date: 2013-10-24 20:43
On Oct 24, 2013, at 08:14 PM, Ned Deily wrote:

>Barry: yes, 2.6 is affected. See discussion on python-dev.

Thanks Ned for the background over on python-dev.  Unless I hear objections
otherwise, I am not going to apply this to 2.6.
msg201507 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2013-10-27 23:02
An update: release candidates for Python 3.3.3 and 2.7.6 are now available, including binary installers for OS X, that include this fix and fully support OS X 10.9.

http://www.python.org/download/releases/3.3.3/

http://www.python.org/download/releases/2.7.6/
msg201748 - (view) Author: Mark Richman (mrichman) Date: 2013-10-30 16:55
I had to do `sudo sh ./patch_readline_issue_18458.sh` or the patch script itself would cause Python to crash.

After applying this patch, I got the following output, and the problem is still *not* solved for me:

-- running on OS X 10.9
 -- 2.7 does not need to be patched - skipped
 -- 3.2 not found - skipped
 -- 3.3 not found - skipped
 -- 3.4 not found - skipped
 -- done
msg201772 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2013-10-30 19:32
Mark, I'm not sure I understand what you saw but the patch script will cause a Python crash as part of its testing so that is to be expected.  You should not have to run the script using sudo.  This script also only applies to Pythons installed from python.org (or otherwise installed into /Library/Frameworks).  Please check which python you are using.  Using whatever command name you enter to start the failing python, try the following (I'll assume you use "python2.7"):

type python2.7
which python2.7
python2.7 -c "import sys;print(sys.version)"
python2.7 -c "import sys;print(sys.executable)"

The value for sys.executable should be:
/Library/Frameworks/Python.framework/Versions/2.7/Resources/Python.app/Contents/MacOS/Python

In any case, you could manually rename _readline.so as shown in one of the earlier messages (substituting "2.7" for "3.3").  Or you could install 2.7.6rc1.
msg201773 - (view) Author: Mark Richman (mrichman) Date: 2013-10-30 19:37
My mistake. I'm using
/System/Library/Frameworks/Python.framework/Versions/2.7/Resources/Python.app/Contents/MacOS/Python

--
*Mark Richman*
web: www.markrichman.com
email: mark@markrichman.com
tel: (954) 234-9049
http://www.linkedin.com/in/mrichman

On Wed, Oct 30, 2013 at 3:32 PM, Ned Deily <report@bugs.python.org> wrote:

>
> Ned Deily added the comment:
>
> Mark, I'm not sure I understand what you saw but the patch script will
> cause a Python crash as part of its testing so that is to be expected.  You
> should not have to run the script using sudo.  This script also only
> applies to Pythons installed from python.org (or otherwise installed into
> /Library/Frameworks).  Please check which python you are using.  Using
> whatever command name you enter to start the failing python, try the
> following (I'll assume you use "python2.7"):
>
> type python2.7
> which python2.7
> python2.7 -c "import sys;print(sys.version)"
> python2.7 -c "import sys;print(sys.executable)"
>
> The value for sys.executable should be:
>
> /Library/Frameworks/Python.framework/Versions/2.7/Resources/Python.app/Contents/MacOS/Python
>
> In any case, you could manually rename _readline.so as shown in one of the
> earlier messages (substituting "2.7" for "3.3").  Or you could install
> 2.7.6rc1.
>
> ----------
>
> _______________________________________
> Python tracker <report@bugs.python.org>
> <http://bugs.python.org/issue18458>
> _______________________________________
>
msg201888 - (view) Author: Russell Jurney (Russell.Jurney) Date: 2013-11-01 09:35
I have installed 2.7.6 RC1 on OS X 10.9, and this bug still exists. See http://stackoverflow.com/questions/19722580/segfault-11-with-pandas-with-python-v2-7-6-rc1-on-mac-os-x-10-9
msg201902 - (view) Author: Robert Xiao (nneonneo) * Date: 2013-11-01 14:36
Russell, that's not related to readline. Please file a new bug report.
History
Date User Action Args
2014-05-05 05:29:24ned.deilylinkissue21433 superseder
2014-04-15 16:44:11ned.deilylinkissue21241 superseder
2014-04-02 03:19:47zach.warelinkissue21129 superseder
2014-03-25 16:44:25ned.deilylinkissue21064 superseder
2014-02-04 21:51:03ned.deilylinkissue20512 superseder
2013-12-03 05:19:31ned.deilylinkissue19868 superseder
2013-11-18 12:45:45ezio.melottilinkissue19644 superseder
2013-11-12 13:31:31ezio.melottilinkissue19559 superseder
2013-11-07 03:51:08ned.deilylinkissue19516 superseder
2013-11-01 17:10:13ned.deilyunlinkissue19471 superseder
2013-11-01 14:36:27nneonneosetmessages: + msg201902
2013-11-01 14:35:55nneonneosetfiles: - readline.so
2013-11-01 09:35:12Russell.Jurneysetnosy: + Russell.Jurney
messages: + msg201888
2013-11-01 09:26:01ezio.melottilinkissue19471 superseder
2013-10-30 19:37:17mrichmansetmessages: + msg201773
2013-10-30 19:32:25ned.deilysetmessages: + msg201772
2013-10-30 16:55:33mrichmansetnosy: + mrichman
messages: + msg201748
2013-10-27 23:02:44ned.deilysetmessages: + msg201507
2013-10-26 00:33:57ned.deilylinkissue19401 superseder
2013-10-24 21:18:57benjamin.petersonlinkissue19381 superseder
2013-10-24 20:43:23barrysetmessages: + msg201184
2013-10-24 20:14:07ned.deilysetmessages: + msg201176
2013-10-24 13:43:32barrysetnosy: + barry
messages: + msg201129
2013-10-24 07:53:56ned.deilylinkissue19374 superseder
2013-10-24 03:08:00ned.deilysetmessages: + msg201087
title: interactive interpreter crashes and test_readline fails with newer versions of libedit -> interactive interpreter crashes and test_readline fails on OS X 10.9 Mavericks due to libedit update
2013-10-24 02:47:33ned.deilysetfiles: + patch_readline_issue_18458.sh
2013-10-23 22:06:17ned.deilylinkissue19370 superseder
2013-10-23 21:42:54ned.deilysetmessages: + msg201070
2013-10-23 21:01:50nneonneosetfiles: + readline.so
nosy: + nneonneo
messages: + msg201068

2013-10-23 19:48:27ned.deilylinkissue19368 superseder
2013-10-23 18:05:33ned.deilylinkissue19366 superseder
2013-10-12 23:04:06ned.deilysetstatus: open -> closed
title: libedit history offset workaround -> interactive interpreter crashes and test_readline fails with newer versions of libedit
messages: + msg199638

keywords: - needs review
resolution: fixed
stage: patch review -> resolved
2013-10-12 22:58:12python-devsetmessages: + msg199637
2013-09-06 22:17:06python-devsetnosy: + python-dev
messages: + msg197116
2013-09-06 22:10:12ned.deilysetfiles: + issue18458.patch

messages: + msg197113
2013-07-23 14:18:02jceasetnosy: + jcea
2013-07-22 05:43:46ronaldoussorensetmessages: + msg193500
2013-07-21 21:59:41ned.deilysetmessages: + msg193470
2013-07-21 21:08:17mkleehammersetnosy: + mkleehammer
messages: + msg193467
2013-07-16 22:55:29ned.deilylinkissue18289 superseder
2013-07-15 12:27:21ronaldoussorensetkeywords: + patch, needs review
type: crash
messages: + msg193091

stage: patch review
2013-07-15 12:24:20ronaldoussorencreate