The attached patch enables compilation and use of the readline module on 
Mac OS X 10.5 (Leopard) and 10.6 (Snow Leopard).
It utilizes the native editline library (used for emulation of the 
readline library on Mac).

I used the patch for almost two years already with Python 2.4 and since 
December 2008 with Python 2.6.  The only difference is that Python 2.4 
did not need the setup.py changes.

The patch is written in such a way that it does *not* affect the 
compilation on systems that use GNU readline library (e.g., Linux).
However, I don't have access to any other system that uses editline 
emulation of readline library besides Mac.  I believe it should work the 
same but some testing would be welcome if anyone is aware of such a 
system (NetBSD?).

With the readline module compiled in, it is enough to put in .editrc

    python:bind -v

and one gets a vi emulation in the python interactive interpreter.

You can also try it directly from the shell:

    >>> import readline
    >>> readline.parse_and_bind("bind -v")
    >>> # use editing features to change the lines above to
    >>> import rlcompleter
    >>> readline.parse_and_bind("bind ^I rl_complete")
    >>> # now TAB offers the completions

It would be nice if we could get this included into Python-2.6.3 
release.

Changed type to "crash" because compilation of readline module without 
this patch was causing Python to crash with BusError.

When testing the patch make sure that your readline module has been 
actually linked against editline library rather then some copy of GNU 
readline from MacPorts or Fink.

Assuming --prefix=${HOME}/opt, the output of

    otool -L ~/opt/lib/python2.4/lib-dynload/readline.so

should contain /usr/lib/libedit.2.dylib.

I'm +1 on merging this functionality.

See also: issue6872

As I mentioned there we should ensure that readline linked to libedit 
has the same semantics as readline linked to GNU readline, and because 
the configuration file of libedit has a different format as the one of 
readline we should mention that in the documentation as well.

It would also be nice if one could programmaticly detect if readline is 
linked to libedit, that way tools like ipython can load the right 
configuration files without user interaction.


BTW. I'm pretty sure that the readline emultation on Leopard was pretty 
broken, ipython used to cause hard crashes with /usr/bin/python on 
Leopard.

> It would also be nice if one could programmaticly detect if readline
> is linked to libedit

There's this rl_library_version constant defined in editline/readline C 
libraries that the attached patch uses.
Perhaps, if we can expose its value from the Python readline module, one 
could check whether the value startswith("EditLine wrapper") in ipython 
and similar programs.

Regarding your comment about editline being broken on Leopard:  The 
patch worked just fine for me and other people who used it in my 
company.  We used line editing and history in interactive interpreter 
with both Python 2.4 and Python 2.6 on Leopard.

One person used it with Python 2.4 and ipython.  He did not have a 
readline functionality until he compiled with the patch.  After applying 
the patch the line editing, history and TAB completion worked for him in 
ipython.  He's now running Snow Leopard so we couldn't check one more 
time.
 
Perhaps you meant it was broken on Tiger (10.4; Darwin 8).

The attached patch is a slightly cleaner version of your patch. What I 
don't like is that I'm also using runtime detection of libedit, I'd 
prefer compile-time detection but that doesn't seem possible without 
doing that dtection in setup.py.

My changes w.r.t. your patch:
* Do the libedit detection once when the readline module is initialised.
* Place libedit support in "#ifdef __APPLE__" blocks, the code might
  work with other libedit's (such as the one in NetBSD) as well, 
  but I cannot test that and don't want to risk breaking other 
  platforms.
* Also patch readline.get_history_item
* Change readline.__doc__ when using libedit's readline emulation

The patch seems to work on fine, including browsing in ipython's history 
on 10.6.

PS. I mentioned ipython because I know there were issues with ipython 
and Apple's python when Leopard was just out. I don't use ipython myself 
and hence don't know if they have added workarounds in their code (or if 
Apple fixed the issues the ran into)

Marking this issue as 'needs review' for two reasons:
1) I'm not familiar with the readline code
2) I'd like to know if this patch is backport material

The patch `readline-libedit.patch` has the following problems:

- a typo causing an undefined variable error;
- a missing "#endif";
- it doesn't refer to the new __APPLE__ specific doc string anywhere.

It fails to compile `readline.c`. ::

    Modules/readline.c:1073: error:
        ‘using_libedit_emultation’ undeclared (first use in this 
function)
    Modules/readline.c:1059:1: error: unterminated #ifdef

The fixed patch is attached as `readline-libedit-2.patch` above.
Please review it.

There are also a few minor stylistic nits I took a liberty to fix.

In readline.c
=============

Line 496
    Formatted a block comment to a usual C style (used by other comments
    it the patch, FWIW).

Line 505
    The standard style in C (and C-like) languages is to *not* write a
    space between a variable and a unary operator (e.g., "C++").
    Changed it to::

        idx--;

Lines 1085/86
    Removed two unnecessary empty lines added by the first patch.
    There's already one empty line in the original code to space an if
    statement from the next line.

Line 1070
    Line was longer than 79 characters (violates PEP-7).  Reformatted.


In setup.py
===========

Line 561
    A comment block was indented in the first patch and had lines longer
    than 79 characters (violates PEP-8).  I refactored the code to avoid
    repeating the same else statement twice.  This obviated the need to
    indent the said comment block.

    *Please review whether you agree with the refactoring*

Just out of curiosity: does anyone know why the first if statement that
tests for 'darwin' uses ``platform`` and the second one uses
``sys.platform``?  Couldn't we use ``platform`` in both of them?

Do the remove_history_item and replace_history_item functions also need 
the off-by-one adjustment?

Mark: yes those functions need to be changed as well.

Mark: it turns out that GNU readline has a rather odd interface, only 
the index of get_history_item is 1-based, all others are 0-based. This 
is not mentioned in the documentation (neither that of the readline 
module or the GNU documentation).

I've attached a very minimal testcase for the history manipulation 
functions in the readline module that passes with GNU readline.

The tests now also pass using libedit emulation (after fixing the 
compilation issues noted by Zvezdan).

I'll file a new issue about the readline documentation, that's unrelated 
to getting libedit support into the repository.

Committed my latest version of the patch as r74970 (trunk) and r74971 
(3.2)


Barry: what's your opinion on a backport of this to 2.6?

Is this open or closed? Wondering as I just updated my checkout and I am 
now segfaulting at the command-line whenever I import something under 
readline 6.0 which was working fine.


>>> import tokenize

Program received signal EXC_BAD_ACCESS, Could not access memory.
Reason: KERN_INVALID_ADDRESS at address: 0x0000000000000000
0x00000001003effa6 in call_readline (sys_stdin=0x7fff70a230c0, 
sys_stdout=0x7fff70a23158, prompt=0x1005da194 ">>> ") at readline.c:1029
1029					line = history_get(state->length 
- 1)->line;
(gdb) bt
#0  0x00000001003effa6 in call_readline (sys_stdin=0x7fff70a230c0, 
sys_stdout=0x7fff70a23158, prompt=0x1005da194 ">>> ") at readline.c:1029
#1  0x00000001000044d8 in PyOS_Readline (sys_stdin=0x7fff70a230c0, 
sys_stdout=0x7fff70a23158, prompt=0x1005da194 ">>> ") at 
myreadline.c:208
#2  0x0000000100006ce0 in tok_nextc (tok=0x100890c10) at tokenizer.c:781
#3  0x0000000100005464 in tok_get (tok=0x100890c10, 
p_start=0x7fff5fbfe9e8, p_end=0x7fff5fbfe9e0) at tokenizer.c:1128
#4  0x00000001000053a4 in PyTokenizer_Get (tok=0x100890c10, 
p_start=0x7fff5fbfe9e8, p_end=0x7fff5fbfe9e0) at tokenizer.c:1568
#5  0x000000010000388b in parsetok (tok=0x100890c10, g=0x100251f68, 
start=256, err_ret=0x7fff5fbfeb10, flags=0x7fff5fbfeb0c) at 
parsetok.c:159
#6  0x0000000100003fe8 in PyParser_ParseFileFlagsEx (fp=0x7fff70a230c0, 
filename=0x1001fba28 "<stdin>", g=0x100251f68, start=256, 
ps1=0x1005da194 ">>> ", ps2=0x1005da2b4 "... ", err_ret=0x7fff5fbfeb10, 
flags=0x7fff5fbfeb0c) at parsetok.c:106
#7  0x0000000100193b76 in PyParser_ASTFromFile (fp=0x7fff70a230c0, 
filename=0x1001fba28 "<stdin>", start=256, ps1=0x1005da194 ">>> ", 
ps2=0x1005da2b4 "... ", flags=0x7fff5fbfeea0, errcode=0x7fff5fbfebec, 
arena=0x1004427d0) at pythonrun.c:1461
#8  0x00000001001937c9 in PyRun_InteractiveOneFlags (fp=0x7fff70a230c0, 
filename=0x1001fba28 "<stdin>", flags=0x7fff5fbfeea0) at pythonrun.c:823
#9  0x0000000100193091 in PyRun_InteractiveLoopFlags (fp=0x7fff70a230c0, 
filename=0x1001fba28 "<stdin>", flags=0x7fff5fbfeea0) at pythonrun.c:763
#10 0x0000000100192db8 in PyRun_AnyFileExFlags (fp=0x7fff70a230c0, 
filename=0x1001fba28 "<stdin>", closeit=0, flags=0x7fff5fbfeea0) at 
pythonrun.c:732
#11 0x00000001001af808 in Py_Main (argc=1, argv=0x7fff5fbfef40) at 
main.c:603
#12 0x0000000100001325 in main (argc=1, argv=0x7fff5fbfef40) at 
python.c:23

This should have been closed, although readline shouldn't crash either.

Brett: What version of OSX do you use? Readline works fine for me on OSX 
10.6 without GNU readline.

BTW. The crashlog indicates you are no longer using GNU readline, but 
use system readline instead (that is the libedit emulation layer). 

This is one thing we completely failed to look at: how to determine 
which one to use? We currently use the first readline we find, which 
will we the system one on OSX 10.5 or later. 

It may be better to either add a configure flag to explicitly select 
which one is preferential, or use the GNU version when it is found.

Brett,

what does this command return for you?

otool -L /path/to/lib/python2.4/lib-dynload/readline.so

Make that python2.6 in the command above.  :-)

Better yet:

   otool -vL $(python -c 'import readline; print readline.__file__')

(Replace "python" by the interpreter that your actually using).

I'm still interested to know the OS release as well.

I've tested with readline 6 on OSX 10.6 as well. Both that and the system 
readline (libedit emulation) work just fine for me.

The current behaviour for me:
* When GNU readline is present in /usr/local it gets used
* Otherwise libedit gets used

I assume you had to pass extra -I and -L flags when compiling with GNU 
readline.  Right?

Zvezdan: I did not have to pass additional arguments to get readline, the 
default machinery automaticly picks up libraries in /usr/local.

I'd love to have a way to restrict the default compiler and linker search 
paths to system locations (e.g. exclude /usr/local and 
/Library/Frameworks), but that's sadly enough not easily possible (and the 
issue is in the compiler not distutils)

I'm on OS X 10.6 and I *thought* I was using GNU Readline 6. But if my 
backtrace implies otherwise something must have gotten messed up somewhere 
on my end. When I deleted Readline and rebuilt everything worked fine.

I'm going to go ahead and close this as you got it build and working for 
yourself that's good enough for me, Ronald.

Brett,

IMO, your backtrace only implies that readline module was built 
believing it has libedit (i.e., include files were system ones from 
/usr/include).

However, the following scenario is possible.  Some packaging tools 
choose to divide library packages into a runtime part and a development 
part.  If you had a copy of readline that was a runtime part only, it 
would have /usr/local/lib/* files but not /usr/local/include/* files 
(development part would have them).

Because of the way setup.py stashes /usr/local/lib first in the path, 
the build could have used system /usr/include/* file and linked to your 
local copy of readline library.

This is just a wild guess of course.

That's why I was interested in the output of otool command on your build 
of readline module.  That would tell us what it was linked to.