Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

enable compilation of readline module on Mac OS X 10.5 and 10.6 #51126

Closed
zvezdan mannequin opened this issue Sep 10, 2009 · 44 comments
Closed

enable compilation of readline module on Mac OS X 10.5 and 10.6 #51126

zvezdan mannequin opened this issue Sep 10, 2009 · 44 comments
Assignees
Labels
build The build process and cross-build release-blocker

Comments

@zvezdan
Copy link
Mannequin

zvezdan mannequin commented Sep 10, 2009

BPO 6877
Nosy @freddrake, @warsaw, @brettcannon, @birkenfeld, @ronaldoussoren, @mdickinson, @benjaminp, @ned-deily, @zvezdan, @meadori
Files
  • readline-trunk.patch: readline.patch
  • readline-libedit.patch: Updated patch
  • readline-libedit-2.patch: Fixed patch
  • test_readline.py
  • readline-libedit-2.6.5.patch: Patch for 2.6.5
  • issue-6877.patch: patch against 2.7 trunk
  • readline-libedit-2.6.5-fix1.patch: Updated 2.6.5 patch
  • readline-libedit-trunk.patch: patch for trunk and 3.2 branch
  • Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = 'https://github.com/ronaldoussoren'
    closed_at = <Date 2010-02-11.13:16:42.701>
    created_at = <Date 2009-09-10.12:38:28.291>
    labels = ['build', 'release-blocker']
    title = 'enable compilation of readline module on Mac OS X 10.5 and 10.6'
    updated_at = <Date 2011-10-28.20:57:54.717>
    user = 'https://github.com/zvezdan'

    bugs.python.org fields:

    activity = <Date 2011-10-28.20:57:54.717>
    actor = 'ned.deily'
    assignee = 'ronaldoussoren'
    closed = True
    closed_date = <Date 2010-02-11.13:16:42.701>
    closer = 'ronaldoussoren'
    components = ['Build']
    creation = <Date 2009-09-10.12:38:28.291>
    creator = 'zvezdan'
    dependencies = []
    files = ['14871', '14895', '14899', '14938', '16136', '16159', '16162', '16163']
    hgrepos = []
    issue_num = 6877
    keywords = ['patch', '26backport', 'needs review']
    message_count = 44.0
    messages = ['92480', '92481', '92483', '92487', '92489', '92654', '92694', '92747', '92750', '92894', '92895', '92968', '92972', '92988', '92989', '92991', '92996', '92998', '93001', '93015', '93018', '98818', '98819', '98858', '98859', '98904', '98908', '98912', '98913', '98942', '98953', '98977', '98978', '98979', '98980', '98987', '99011', '99012', '99019', '99031', '99204', '99207', '100464', '100697']
    nosy_count = 12.0
    nosy_names = ['fdrake', 'barry', 'brett.cannon', 'georg.brandl', 'ronaldoussoren', 'mark.dickinson', 'benjamin.peterson', 'ned.deily', 'zvezdan', 'srid', 'meador.inge', 'Alexander.Belopolsky']
    pr_nums = []
    priority = 'release blocker'
    resolution = 'fixed'
    stage = 'resolved'
    status = 'closed'
    superseder = None
    type = 'compile error'
    url = 'https://bugs.python.org/issue6877'
    versions = ['Python 2.6', 'Python 3.1', 'Python 2.7', 'Python 3.2']

    @zvezdan
    Copy link
    Mannequin Author

    zvezdan mannequin commented Sep 10, 2009

    The attached patch enables compilation and use of the readline module on
    Mac OS X 10.5 (Leopard) and 10.6 (Snow Leopard).
    It utilizes the native editline library (used for emulation of the
    readline library on Mac).

    I used the patch for almost two years already with Python 2.4 and since
    December 2008 with Python 2.6. The only difference is that Python 2.4
    did not need the setup.py changes.

    The patch is written in such a way that it does *not* affect the
    compilation on systems that use GNU readline library (e.g., Linux).
    However, I don't have access to any other system that uses editline
    emulation of readline library besides Mac. I believe it should work the
    same but some testing would be welcome if anyone is aware of such a
    system (NetBSD?).

    With the readline module compiled in, it is enough to put in .editrc

    python:bind -v
    

    and one gets a vi emulation in the python interactive interpreter.

    You can also try it directly from the shell:

    >>> import readline
    >>> readline.parse_and_bind("bind -v")
    >>> # use editing features to change the lines above to
    >>> import rlcompleter
    >>> readline.parse_and_bind("bind ^I rl_complete")
    >>> # now TAB offers the completions
    

    It would be nice if we could get this included into Python-2.6.3
    release.

    @zvezdan zvezdan mannequin added build The build process and cross-build labels Sep 10, 2009
    @zvezdan
    Copy link
    Mannequin Author

    zvezdan mannequin commented Sep 10, 2009

    Changed type to "crash" because compilation of readline module without
    this patch was causing Python to crash with BusError.

    @zvezdan zvezdan mannequin added type-crash A hard crash of the interpreter, possibly with a core dump and removed build The build process and cross-build labels Sep 10, 2009
    @zvezdan
    Copy link
    Mannequin Author

    zvezdan mannequin commented Sep 10, 2009

    When testing the patch make sure that your readline module has been
    actually linked against editline library rather then some copy of GNU
    readline from MacPorts or Fink.

    Assuming --prefix=${HOME}/opt, the output of

    otool -L ~/opt/lib/python2.4/lib-dynload/readline.so
    

    should contain /usr/lib/libedit.2.dylib.

    @ronaldoussoren
    Copy link
    Contributor

    I'm +1 on merging this functionality.

    See also: bpo-6872

    As I mentioned there we should ensure that readline linked to libedit
    has the same semantics as readline linked to GNU readline, and because
    the configuration file of libedit has a different format as the one of
    readline we should mention that in the documentation as well.

    It would also be nice if one could programmaticly detect if readline is
    linked to libedit, that way tools like ipython can load the right
    configuration files without user interaction.

    BTW. I'm pretty sure that the readline emultation on Leopard was pretty
    broken, ipython used to cause hard crashes with /usr/bin/python on
    Leopard.

    @zvezdan
    Copy link
    Mannequin Author

    zvezdan mannequin commented Sep 10, 2009

    It would also be nice if one could programmaticly detect if readline
    is linked to libedit

    There's this rl_library_version constant defined in editline/readline C
    libraries that the attached patch uses.
    Perhaps, if we can expose its value from the Python readline module, one
    could check whether the value startswith("EditLine wrapper") in ipython
    and similar programs.

    Regarding your comment about editline being broken on Leopard: The
    patch worked just fine for me and other people who used it in my
    company. We used line editing and history in interactive interpreter
    with both Python 2.4 and Python 2.6 on Leopard.

    One person used it with Python 2.4 and ipython. He did not have a
    readline functionality until he compiled with the patch. After applying
    the patch the line editing, history and TAB completion worked for him in
    ipython. He's now running Snow Leopard so we couldn't check one more
    time.

    Perhaps you meant it was broken on Tiger (10.4; Darwin 8).

    @ronaldoussoren
    Copy link
    Contributor

    The attached patch is a slightly cleaner version of your patch. What I
    don't like is that I'm also using runtime detection of libedit, I'd
    prefer compile-time detection but that doesn't seem possible without
    doing that dtection in setup.py.

    My changes w.r.t. your patch:

    • Do the libedit detection once when the readline module is initialised.
    • Place libedit support in "#ifdef __APPLE__" blocks, the code might
      work with other libedit's (such as the one in NetBSD) as well,
      but I cannot test that and don't want to risk breaking other
      platforms.
    • Also patch readline.get_history_item
    • Change readline.__doc__ when using libedit's readline emulation

    The patch seems to work on fine, including browsing in ipython's history
    on 10.6.

    PS. I mentioned ipython because I know there were issues with ipython
    and Apple's python when Leopard was just out. I don't use ipython myself
    and hence don't know if they have added workarounds in their code (or if
    Apple fixed the issues the ran into)

    Marking this issue as 'needs review' for two reasons:

    1. I'm not familiar with the readline code
    2. I'd like to know if this patch is backport material

    @zvezdan
    Copy link
    Mannequin Author

    zvezdan mannequin commented Sep 16, 2009

    The patch readline-libedit.patch has the following problems:

    • a typo causing an undefined variable error;
    • a missing "#endif";
    • it doesn't refer to the new __APPLE__ specific doc string anywhere.

    It fails to compile readline.c. ::

    [Modules/readline.c:1073](https://github.com/python/cpython/blob/main/Modules/readline.c#L1073): error:
        ‘using_libedit_emultation’ undeclared (first use in this 
    

    function)
    Modules/readline.c:1059:1: error: unterminated #ifdef

    The fixed patch is attached as readline-libedit-2.patch above.
    Please review it.

    There are also a few minor stylistic nits I took a liberty to fix.

    In readline.c
    =============

    Line 496
    Formatted a block comment to a usual C style (used by other comments
    it the patch, FWIW).

    Line 505
    The standard style in C (and C-like) languages is to *not* write a
    space between a variable and a unary operator (e.g., "C++").
    Changed it to::

        idx--;
    

    Lines 1085/86
    Removed two unnecessary empty lines added by the first patch.
    There's already one empty line in the original code to space an if
    statement from the next line.

    Line 1070
    Line was longer than 79 characters (violates PEP-7). Reformatted.

    In setup.py
    ===========

    Line 561
    A comment block was indented in the first patch and had lines longer
    than 79 characters (violates PEP-8). I refactored the code to avoid
    repeating the same else statement twice. This obviated the need to
    indent the said comment block.

    *Please review whether you agree with the refactoring*
    

    Just out of curiosity: does anyone know why the first if statement that
    tests for 'darwin' uses platform and the second one uses
    sys.platform? Couldn't we use platform in both of them?

    @mdickinson
    Copy link
    Member

    Do the remove_history_item and replace_history_item functions also need
    the off-by-one adjustment?

    @ronaldoussoren
    Copy link
    Contributor

    Mark: yes those functions need to be changed as well.

    @ronaldoussoren
    Copy link
    Contributor

    Mark: it turns out that GNU readline has a rather odd interface, only
    the index of get_history_item is 1-based, all others are 0-based. This
    is not mentioned in the documentation (neither that of the readline
    module or the GNU documentation).

    I've attached a very minimal testcase for the history manipulation
    functions in the readline module that passes with GNU readline.

    The tests now also pass using libedit emulation (after fixing the
    compilation issues noted by Zvezdan).

    I'll file a new issue about the readline documentation, that's unrelated
    to getting libedit support into the repository.

    @ronaldoussoren
    Copy link
    Contributor

    Committed my latest version of the patch as r74970 (trunk) and r74971
    (3.2)

    Barry: what's your opinion on a backport of this to 2.6?

    @ronaldoussoren ronaldoussoren added build The build process and cross-build and removed type-crash A hard crash of the interpreter, possibly with a core dump labels Sep 20, 2009
    @brettcannon
    Copy link
    Member

    Is this open or closed? Wondering as I just updated my checkout and I am
    now segfaulting at the command-line whenever I import something under
    readline 6.0 which was working fine.

    >> import tokenize

    Program received signal EXC_BAD_ACCESS, Could not access memory.
    Reason: KERN_INVALID_ADDRESS at address: 0x0000000000000000
    0x00000001003effa6 in call_readline (sys_stdin=0x7fff70a230c0, 
    sys_stdout=0x7fff70a23158, prompt=0x1005da194 ">>> ") at readline.c:1029
    1029					line = history_get(state->length 
    - 1)->line;
    (gdb) bt
    #0  0x00000001003effa6 in call_readline (sys_stdin=0x7fff70a230c0, 
    sys_stdout=0x7fff70a23158, prompt=0x1005da194 ">>> ") at readline.c:1029
    #1  0x00000001000044d8 in PyOS_Readline (sys_stdin=0x7fff70a230c0, 
    sys_stdout=0x7fff70a23158, prompt=0x1005da194 ">>> ") at 
    myreadline.c:208
    #2  0x0000000100006ce0 in tok_nextc (tok=0x100890c10) at tokenizer.c:781
    #3  0x0000000100005464 in tok_get (tok=0x100890c10, 
    p_start=0x7fff5fbfe9e8, p_end=0x7fff5fbfe9e0) at tokenizer.c:1128
    #4  0x00000001000053a4 in PyTokenizer_Get (tok=0x100890c10, 
    p_start=0x7fff5fbfe9e8, p_end=0x7fff5fbfe9e0) at tokenizer.c:1568
    #5  0x000000010000388b in parsetok (tok=0x100890c10, g=0x100251f68, 
    start=256, err_ret=0x7fff5fbfeb10, flags=0x7fff5fbfeb0c) at 
    parsetok.c:159
    #6  0x0000000100003fe8 in PyParser_ParseFileFlagsEx (fp=0x7fff70a230c0, 
    filename=0x1001fba28 "<stdin>", g=0x100251f68, start=256, 
    ps1=0x1005da194 ">>> ", ps2=0x1005da2b4 "... ", err_ret=0x7fff5fbfeb10, 
    flags=0x7fff5fbfeb0c) at parsetok.c:106
    #7  0x0000000100193b76 in PyParser_ASTFromFile (fp=0x7fff70a230c0, 
    filename=0x1001fba28 "<stdin>", start=256, ps1=0x1005da194 ">>> ", 
    ps2=0x1005da2b4 "... ", flags=0x7fff5fbfeea0, errcode=0x7fff5fbfebec, 
    arena=0x1004427d0) at pythonrun.c:1461
    #8  0x00000001001937c9 in PyRun_InteractiveOneFlags (fp=0x7fff70a230c0, 
    filename=0x1001fba28 "<stdin>", flags=0x7fff5fbfeea0) at pythonrun.c:823
    #9  0x0000000100193091 in PyRun_InteractiveLoopFlags (fp=0x7fff70a230c0, 
    filename=0x1001fba28 "<stdin>", flags=0x7fff5fbfeea0) at pythonrun.c:763
    #10 0x0000000100192db8 in PyRun_AnyFileExFlags (fp=0x7fff70a230c0, 
    filename=0x1001fba28 "<stdin>", closeit=0, flags=0x7fff5fbfeea0) at 
    pythonrun.c:732
    #11 0x00000001001af808 in Py_Main (argc=1, argv=0x7fff5fbfef40) at 
    main.c:603
    #12 0x0000000100001325 in main (argc=1, argv=0x7fff5fbfef40) at 
    python.c:23

    @ronaldoussoren
    Copy link
    Contributor

    This should have been closed, although readline shouldn't crash either.

    Brett: What version of OSX do you use? Readline works fine for me on OSX
    10.6 without GNU readline.

    BTW. The crashlog indicates you are no longer using GNU readline, but
    use system readline instead (that is the libedit emulation layer).

    This is one thing we completely failed to look at: how to determine
    which one to use? We currently use the first readline we find, which
    will we the system one on OSX 10.5 or later.

    It may be better to either add a configure flag to explicitly select
    which one is preferential, or use the GNU version when it is found.

    @zvezdan
    Copy link
    Mannequin Author

    zvezdan mannequin commented Sep 22, 2009

    Brett,

    what does this command return for you?

    otool -L /path/to/lib/python2.4/lib-dynload/readline.so

    @zvezdan
    Copy link
    Mannequin Author

    zvezdan mannequin commented Sep 22, 2009

    Make that python2.6 in the command above. :-)

    @ronaldoussoren
    Copy link
    Contributor

    Better yet:

    otool -vL $(python -c 'import readline; print readline.__file__')

    (Replace "python" by the interpreter that your actually using).

    I'm still interested to know the OS release as well.

    @ronaldoussoren
    Copy link
    Contributor

    I've tested with readline 6 on OSX 10.6 as well. Both that and the system
    readline (libedit emulation) work just fine for me.

    The current behaviour for me:

    • When GNU readline is present in /usr/local it gets used
    • Otherwise libedit gets used

    @zvezdan
    Copy link
    Mannequin Author

    zvezdan mannequin commented Sep 22, 2009

    I assume you had to pass extra -I and -L flags when compiling with GNU
    readline. Right?

    @ronaldoussoren
    Copy link
    Contributor

    Zvezdan: I did not have to pass additional arguments to get readline, the
    default machinery automaticly picks up libraries in /usr/local.

    I'd love to have a way to restrict the default compiler and linker search
    paths to system locations (e.g. exclude /usr/local and
    /Library/Frameworks), but that's sadly enough not easily possible (and the
    issue is in the compiler not distutils)

    @brettcannon
    Copy link
    Member

    I'm on OS X 10.6 and I *thought* I was using GNU Readline 6. But if my
    backtrace implies otherwise something must have gotten messed up somewhere
    on my end. When I deleted Readline and rebuilt everything worked fine.

    I'm going to go ahead and close this as you got it build and working for
    yourself that's good enough for me, Ronald.

    @zvezdan
    Copy link
    Mannequin Author

    zvezdan mannequin commented Sep 22, 2009

    Brett,

    IMO, your backtrace only implies that readline module was built
    believing it has libedit (i.e., include files were system ones from
    /usr/include).

    However, the following scenario is possible. Some packaging tools
    choose to divide library packages into a runtime part and a development
    part. If you had a copy of readline that was a runtime part only, it
    would have /usr/local/lib/* files but not /usr/local/include/* files
    (development part would have them).

    Because of the way setup.py stashes /usr/local/lib first in the path,
    the build could have used system /usr/include/* file and linked to your
    local copy of readline library.

    This is just a wild guess of course.

    That's why I was interested in the output of otool command on your build
    of readline module. That would tell us what it was linked to.

    @warsaw
    Copy link
    Member

    warsaw commented Feb 4, 2010

    The readline-libedit-2.patch no longer applies against the 2.6 branch because of changes in setup.py since then. If this is fixed and the subsequent patch is reviewed and approved, then this can be landed for Python 2.6.5.

    @warsaw
    Copy link
    Member

    warsaw commented Feb 4, 2010

    release blocker for 2.6.5

    @warsaw warsaw reopened this Feb 4, 2010
    @zvezdan
    Copy link
    Mannequin Author

    zvezdan mannequin commented Feb 4, 2010

    The readline-libedit-2.6.5.patch is attached.

    The patch was applied and python built in several configurations on Mac
    OS X 10.6 (Snow Leopard). There is no regression (details below).

    Can somebody else test on Mac OS X 10.5 (Leopard)?

    32-bit
    ======

    Configuration with system libedit::

    ./configure --prefix=${HOME}/opt/snapshot/2.6.5 \
        BASECFLAGS="-arch i386" \
        CFLAGS="-arch i386" \
        LDFLAGS="-arch i386" \
        MACOSX_DEPLOYMENT_TARGET=10.6
    

    Configuration with GNU readline 6.1::

    ./configure --prefix=${HOME}/opt/snapshot/gnurl/2.6.5 \
        BASECFLAGS="-arch i386" \
        CFLAGS="-arch i386" \
        CPPFLAGS="-I/opt/local/include" \
        LDFLAGS="-arch i386 -L/opt/local/lib" \
        MACOSX_DEPLOYMENT_TARGET=10.6
    

    Failed tests for both builds:

    • test_asynchat
    • test_smtplib

    64-bit
    ======

    Configuration with system libedit::

    ./configure --prefix=${HOME}/opt/snapshot/2.6.5-64 \
        MACOSX_DEPLOYMENT_TARGET=10.6
    

    Configuration with GNU readline 6.1::

    ./configure --prefix=${HOME}/opt/snapshot/gnurl/2.6.5-64 \
        CPPFLAGS="-I/opt/local/include" \
        LDFLAGS="-L/opt/local/lib" \
        MACOSX_DEPLOYMENT_TARGET=10.6checking
    

    Failed tests for both builds:

    • test_asynchat
    • test_macostools
    • test_smtplib

    Unexpected skips:

    • test_dl

    Universal
    =========

    Configuration with system libedit::

    ./configure --prefix=${HOME}/opt/snapshot \
        BASECFLAGS="-arch x86_64 -arch i386" \
        CFLAGS="-arch x86_64 -arch i386" \
        LDFLAGS="-arch x86_64 -arch i386" \
        MACOSX_DEPLOYMENT_TARGET=10.6
    

    Configuration with GNU readline 6.1::

    ./configure --prefix=${HOME}/opt/snapshot/gnurl \
        BASECFLAGS="-arch x86_64 -arch i386" \
        CFLAGS="-arch x86_64 -arch i386" \
        CPPFLAGS="-I/opt/local/include" \
        LDFLAGS="-arch x86_64 -arch i386 -L/opt/local/lib" \
        MACOSX_DEPLOYMENT_TARGET=10.6
    

    Failed tests for both builds:

    • test_asynchat
    • test_macostools
    • test_smtplib

    Unexpected skips:

    • test_dl

    when run as 64-bit or 32-bit executable.

    Errors in both asynchat and smtplib are caused by the same issue in
    asyncore.py. A typical error output::

    error: uncaptured python exception, closing channel
    <test.test_asynchat.echo_client at 0x1b112b0>
    (<class 'socket.error'>:[Errno 9]
     Bad file descriptor
     [/Users/zvezdan/opt/snapshot/2.6.5/lib/python2.6/asyncore.py|
      readwrite|107]
     [/Users/zvezdan/opt/snapshot/2.6.5/lib/python2.6/asyncore.py|
      handle_expt_event|441]
     [<string>|getsockopt|1]
     [/Users/zvezdan/opt/snapshot/2.6.5/lib/python2.6/socket.py|_dummy|165])
    

    but this is not caused by readline patch.

    @zvezdan
    Copy link
    Mannequin Author

    zvezdan mannequin commented Feb 4, 2010

    I forgot to add that the patch for 2.6.5 is based on:
    http://svn.python.org/view?rev=74970&view=rev

    @warsaw
    Copy link
    Member

    warsaw commented Feb 5, 2010

    This worked fine on OS X 10.6, but failed on OS X 10.5:

    gcc -fno-strict-aliasing -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I. -I/Users/barry/projects/python/python26/./Include -I/Users/barry/projects/python/python26/./Mac/Include -I. -IInclude -I./Include -I/usr/local/include -I/Users/barry/projects/python/python26/Include -I/Users/barry/projects/python/python26 -c /Users/barry/projects/python/python26/Modules/readline.c -o build/temp.macosx-10.3-ppc-2.6/Users/barry/projects/python/python26/Modules/readline.o
    /Users/barry/projects/python/python26/Modules/readline.c:41: error: conflicting types for 'completion_matches'
    /usr/include/readline/readline.h:172: error: previous declaration of 'completion_matches' was here
    /Users/barry/projects/python/python26/Modules/readline.c: In function 'py_remove_history':
    /Users/barry/projects/python/python26/Modules/readline.c:378: warning: passing argument 1 of 'free' discards qualifiers from pointer target type
    /Users/barry/projects/python/python26/Modules/readline.c: In function 'py_replace_history':
    /Users/barry/projects/python/python26/Modules/readline.c:415: warning: passing argument 1 of 'free' discards qualifiers from pointer target type
    /Users/barry/projects/python/python26/Modules/readline.c: In function 'call_readline':
    /Users/barry/projects/python/python26/Modules/readline.c:1032: warning: assignment discards qualifiers from pointer target type
    /Users/barry/projects/python/python26/Modules/readline.c:1035: warning: assignment discards qualifiers from pointer target type

    Failed to find the necessary bits to build these modules:
    _bsddb gdbm linuxaudiodev
    ossaudiodev spwd sunaudiodev
    To find the necessary bits, look in setup.py in detect_modules() for the module's name.

    Failed to build these modules:
    readline

    running build_scripts
    @bytor[~/projects/python/python26:1011]%

    @zvezdan
    Copy link
    Mannequin Author

    zvezdan mannequin commented Feb 5, 2010

    This is a configuration bug:

    /Users/barry/projects/python/python26/Modules/readline.c -o build/temp.macosx-10.3-ppc-2.6 ...

    Why is it trying to build using macosx-10.3 target?
    I bet if you specify MACOSX_DEPLOYMENT_TARGET=10.5 on the ./configure line it will work fine. It worked for me when I used Leopard (until September 2009).

    This is a bug in configure.in

    @ned-deily
    Copy link
    Member

    It's not a configuration bug. All current python.org python installers are built with MACOSX_DEPLOYMENT_TARGET=10.3 and with the 10.4u SDK to allow one installer to work on systems from 10.3.9 through 10.6.

    @zvezdan
    Copy link
    Mannequin Author

    zvezdan mannequin commented Feb 5, 2010

    Barry,

    I'm sorry, I only now noticed this line in your compilation on 10.5:

    /Users/barry/projects/python/python26/Modules/readline.c:41: error: conflicting types for 'completion_matches'

    This problem was introduced by a patch in bpo-4204.
    See my comments in msg82619 and msg92482.
    They've been ignored because the issue was closed, and I did not have a right to reopen it.

    Please notice that my original patch attached here (readline-trunk.patch) has this part in it, which is very important for 10.5:

    @@ -6,6 +6,7 @@
     
     /* Standard definitions */
     #include "Python.h"
    +#include <stdlib.h>
     #include <setjmp.h>
     #include <signal.h>
     #include <errno.h>

    Ronald has omitted that part when he was changing the patch and I completely forgot about its importance since I'm running 10.6 and did not have problems.

    Initially my first version of the patch was checking for __APPLE__ and avoided to re-define completion_matches on lines 35-36. I used that patch when 2.6.1 came out in my personal builds.

    Then I found out that including stdlib early avoided the problem.

    I really do not have a 10.5 machine to test on now.
    Can you please try inserting that include <stdlib.h> and see if 10.5 builds readline?

    If yes, I'd be glad to make a new patch against 2.6 branch and trunk/3.x branch (probably need the same include line).

    If not, then we need to make a choice of #ifdef __FreeBSD__ as I suggested in bpo-4204, or #ifndef __APPLE__ as I used in my first personal version of the patch.

    The problem with re-definition of completion_matches did not exist in 2.4 and 2.5 and is definitely introduced by a patch in bpo-4204, which annoyed me because it broke a modern OS to support a very old version of FreeBSD (4.x).
    :-)

    <aside note>
    That said, I still think that configure.in should not treat 10.5 as 10.3 because Leopard was a big change to the UNIX 2003 specification and too many things are different between 10.3/10.4 and 10.5/10.6.

    We had a discussion on pythonmac-sig about it and people mostly agreed but I do not remember if any issue was opened in the tracker to act on it or not.
    </aside note>

    @meadori
    Copy link
    Member

    meadori commented Feb 6, 2010

    Can you please try inserting that include <stdlib.h> and see if 10.5
    builds readline?

    This does not work out of trunk for me:

    euclid:trunk minge$ sw_vers
    ProductName: Mac OS X
    ProductVersion: 10.5.7
    BuildVersion: 9J61
    euclid:trunk minge$ svn diff
    Index: Modules/readline.c
    ===================================================================

    --- Modules/readline.c	(revision 78019)
    +++ Modules/readline.c	(working copy)
    @@ -6,6 +6,7 @@
     
     /* Standard definitions */
     #include "Python.h"
    +#include <stdlib.h>
     #include <setjmp.h>
     #include <signal.h>
     #include <errno.h>
    euclid:trunk minge$ make

    Modules/Setup.dist is newer than Modules/Setup;
    check to make sure you have all the updates you
    need in your Modules/Setup file.
    Usually, copying Modules/Setup.dist to Modules/Setup will work.
    -----------------------------------------------
    running build
    running build_ext
    building dbm using ndbm
    building 'readline' extension
    gcc -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/Users/minge/Research/Languages/python/trunk/Mac/Include -IInclude -I./Include -I/usr/local/include -I/Users/minge/Research/Languages/python/trunk/Include -I. -c /Users/minge/Research/Languages/python/trunk/Modules/readline.c -o build/temp.macosx-10.4-i386-2.7/Users/minge/Research/Languages/python/trunk/Modules/readline.o
    /Users/minge/Research/Languages/python/trunk/Modules/readline.c:42: error: conflicting types for ‘completion_matches’
    /usr/include/readline/readline.h:172: error: previous declaration of ‘completion_matches’ was here
    /Users/minge/Research/Languages/python/trunk/Modules/readline.c: In function ‘py_remove_history’:
    /Users/minge/Research/Languages/python/trunk/Modules/readline.c:379: warning: passing argument 1 of ‘free’ discards qualifiers from pointer target type
    /Users/minge/Research/Languages/python/trunk/Modules/readline.c: In function ‘py_replace_history’:
    /Users/minge/Research/Languages/python/trunk/Modules/readline.c:416: warning: passing argument 1 of ‘free’ discards qualifiers from pointer target type
    /Users/minge/Research/Languages/python/trunk/Modules/readline.c: In function ‘call_readline’:
    /Users/minge/Research/Languages/python/trunk/Modules/readline.c:1033: warning: assignment discards qualifiers from pointer target type
    /Users/minge/Research/Languages/python/trunk/Modules/readline.c:1036: warning: assignment discards qualifiers from pointer target type

    Python build finished, but the necessary bits to build these modules were not found:
    _bsddb gdbm linuxaudiodev
    ossaudiodev spwd sunaudiodev
    To find the necessary bits, look in setup.py in detect_modules() for the module's name.

    Failed to build these modules:
    readline

    running build_scripts

    I am more than happy to run more tests with respect to this issue. I am tired of seeing this build break everyday :)

    @warsaw
    Copy link
    Member

    warsaw commented Feb 6, 2010

    @meador: confirmed. This does not fix the 10.5 build.

    @meadori
    Copy link
    Member

    meadori commented Feb 7, 2010

    If not, then we need to make a choice of #ifdef __FreeBSD__ as I
    suggested in bpo-4204, or #ifndef __APPLE__ as I used in my first
    personal version of the patch.

    In the off chance that this same problem were to occur on another platform, wouldn't be better to test for 'completion_matches' in configure? Perhaps something like the attached patch? (fixes the issue for me on OS X 10.5 in the Python trunk)

    @zvezdan
    Copy link
    Mannequin Author

    zvezdan mannequin commented Feb 7, 2010

    OK, so bpo-4204 patch is causing problems.
    As I mentioned before we could choose to #ifdef __FreeBSD__ or check whether __APPLE__ is not defined.

    I'm attaching a new patch for 2.6.5 (2.6 branch) that uses the __APPLE__ check.

    @zvezdan
    Copy link
    Mannequin Author

    zvezdan mannequin commented Feb 7, 2010

    The patch with the same correction against trunk is attached so that Meador can test it.

    @zvezdan
    Copy link
    Mannequin Author

    zvezdan mannequin commented Feb 7, 2010

    Meador, I just looked at your patch (it seems we were posting patches at about the same time).

    That's a good fix too.

    Barry it's your call.

    @ronaldoussoren
    Copy link
    Contributor

    re: msg98913

    The addition of an #include of stdlib.h is unnecessary because Python.h already does that.

    re: msg98908

    Building against with a 10.3 deployment target should be harmless. A universal build targets 10.3 on purpose because that results in binaries that work on all supported OSX platforms.

    BTW. I've pruned the list of python versions, this patch will never be applied to 2.4.

    BTW2. I'll do the backport later today.

    @warsaw
    Copy link
    Member

    warsaw commented Feb 7, 2010

    Zvezdan, your "fix1" patch works perfectly for me in Python 2.6 on both OS X 10.5 and 10.6. Great work!

    I approve applying this patch for Python 2.6.5. Ronald, would you like to do the honors?

    @ronaldoussoren
    Copy link
    Contributor

    I have a question about this bit of the patch:

    @@ -38,10 +38,31 @@
     #if defined(_RL_FUNCTION_TYPEDEF)
     extern char **completion_matches(char *, rl_compentry_func_t *);
     #else
    +#if !defined(__APPLE__)
     extern char **completion_matches(char *, CPFunction *);
     #endif
     #endif
    +#endif

    That bit is not in the trunk, should it be forward ported to the trunk?

    The actual prototype on 10.5 and 10.6 is:

    char           **completion_matches(const char *, CPFunction *);

    Wouldn't it be better to change the prototype in readline.c to match?

    As that's not a critical change I've committed the fix1 patch as is in r78096.

    @meadori
    Copy link
    Member

    meadori commented Feb 7, 2010

    That bit is not in the trunk, should it be forward ported to the trunk?

    FWIW, I would really like to have it.

    The actual prototype on 10.5 and 10.6 is:

    char **completion_matches(const char *, CPFunction *);

    Wouldn't it be better to change the prototype in readline.c to match?

    I may have missed something, but the patch is actually *excluding* the
    prototype on OS X, i.e. #if *not* defined. Thus the 'const char*' prototype
    that comes in with 'readline.h' is the one that will be used.

    On the other hand, if you are proposing to make them 'match' just to avoid
    the redefinition conflict, then this may break on other systems where 'char
    *' is used.

    Regards,

    -- Meador

    @zvezdan
    Copy link
    Mannequin Author

    zvezdan mannequin commented Feb 7, 2010

    Ronald,

    That bit is not in the trunk, should it be forward ported to the trunk?

    Yes, that should be applied to trunk and 3.x to make it work on Mac OS X 10.5 (Leopard). I indicated that in msg98979.

    The explanation why that part is needed is given in msg98978 and msg98913.
    That ultimately points to re-declaration of completion_matches() inserted in a patch in bpo-4204 (see my comments in the last two messages there).

    We must *avoid* the duplicate declaration on Mac OS X, because the same declaration is already in /usr/include/readline/readline.h.
    Hence #if !defined(APPLE) is used.

    So, the readline-libedit-trunk.patch I attached yesterday should be applied to trunk (2.7) and 3.x.

    As that's not a critical change I've committed the fix1 patch as is in r78096.

    It seems that you forgot to

        svn add [Lib/test/test_readline.py](https://github.com/python/cpython/blob/main/Lib/test/test_readline.py)
    

    before committing.
    If you run svn status on that checkout I bet it will show with '?' in the left column.

    P.S.
    FWIW, you are right about #include <stdlib.h>. I removed it from the patch yesterday (notice attachment, removal, and a new attachment in the history).

    @ronaldoussoren
    Copy link
    Contributor

    I've added the tests to the 2.6 branch and have ported the #ifdef guard around the prototype for completion_matches to the trunk and 3.2.

    I'm therefore closing the issue.

    @meadori
    Copy link
    Member

    meadori commented Feb 11, 2010

    On Thu, Feb 11, 2010 at 7:16 AM, Ronald Oussoren <report@bugs.python.org>wrote:

    I've added the tests to the 2.6 branch and have ported the #ifdef guard
    around the prototype for completion_matches to the trunk and 3.2.

    Verified in trunk. Thanks Ronald!

    @ned-deily
    Copy link
    Member

    See bpo-8066 for additional patches to correct problems when building on 10.5 or 10.6 for deployment targets of 10.4 or earlier.

    @ned-deily
    Copy link
    Member

    Also note that this feature has not been backported to 3.1 which means it will not be in the upcoming 3.1.2 release. (It will not be an issue for the python.org 3.1.2 installer which will be built with GNU readline as usual to support older systems.)

    @ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
    Labels
    build The build process and cross-build release-blocker
    Projects
    None yet
    Development

    No branches or pull requests

    6 participants