Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

make install must not copy python.o into $prefix/lib/python3.10/config-3.10-x86_64-linux-gnu/ #86473

Closed
vstinner opened this issue Nov 10, 2020 · 13 comments
Labels
3.10 only security fixes build The build process and cross-build

Comments

@vstinner
Copy link
Member

BPO 42307
Nosy @nascheme, @ncoghlan, @vstinner, @ned-deily, @hroncok, @pablogsal
PRs
  • bpo-42307: No longer install python.o file #23219
  • Superseder
  • bpo-43103: Add configure --without-static-libpython to not build libpython3.10.a
  • Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = None
    closed_at = <Date 2021-02-17.10:59:40.422>
    created_at = <Date 2020-11-10.10:49:40.121>
    labels = ['build', '3.10']
    title = 'make install must not copy python.o into $prefix/lib/python3.10/config-3.10-x86_64-linux-gnu/'
    updated_at = <Date 2021-02-17.10:59:40.420>
    user = 'https://github.com/vstinner'

    bugs.python.org fields:

    activity = <Date 2021-02-17.10:59:40.420>
    actor = 'vstinner'
    assignee = 'none'
    closed = True
    closed_date = <Date 2021-02-17.10:59:40.422>
    closer = 'vstinner'
    components = ['Build']
    creation = <Date 2020-11-10.10:49:40.121>
    creator = 'vstinner'
    dependencies = []
    files = []
    hgrepos = []
    issue_num = 42307
    keywords = ['patch']
    message_count = 13.0
    messages = ['380650', '380652', '380653', '380655', '380671', '380674', '380682', '380685', '380711', '380722', '380724', '386150', '387144']
    nosy_count = 6.0
    nosy_names = ['nascheme', 'ncoghlan', 'vstinner', 'ned.deily', 'hroncok', 'pablogsal']
    pr_nums = ['23219']
    priority = 'normal'
    resolution = 'duplicate'
    stage = 'resolved'
    status = 'closed'
    superseder = '43103'
    type = None
    url = 'https://bugs.python.org/issue42307'
    versions = ['Python 3.10']

    @vstinner
    Copy link
    Member Author

    vstinner commented Nov 10, 2020

    Commands reproduce the issue:

    cd /path/to/python/source/
    ./configure --prefix /opt/pymaster CFLAGS="-O0"
    make
    make install
    

    make install copies python.o:

    $ find /opt/pymaster/ -name "*.o"
    /opt/pymaster/lib/python3.10/config-3.10-x86_64-linux-gnu/python.o

    This file is useless and must not be installed by make install.

    The issue was first discovered in Fedora:
    https://bugzilla.redhat.com/show_bug.cgi?id=1894462

    @vstinner vstinner added 3.10 only security fixes build The build process and cross-build labels Nov 10, 2020
    @vstinner
    Copy link
    Member Author

    vstinner commented Nov 10, 2020

    python.o is installed by "make libainstall". It is done since 2001 (commit 85515ad).

    Git history:

    commit 49fd7fa4431da299196d74087df4a04f99f9c46f
    Author: Thomas Wouters <thomas@python.org>
    Date:   Fri Apr 21 10:40:58 2006 +0000
    
        Merge p3yk branch with the trunk up to revision 45595. This breaks a fair
        number of tests, all because of the codecs/_multibytecodecs issue described
        here (it's not a Py3K issue, just something Py3K discovers):
        http://mail.python.org/pipermail/python-dev/2006-April/064051.html
        (...)
    
    -       $(INSTALL_DATA) Modules/$(MAINOBJ) $(DESTDIR)$(LIBPL)/$(MAINOBJ)
    +       $(INSTALL_DATA) Modules/python.o $(DESTDIR)$(LIBPL)/python.o
    
    commit a1a84e7d4f7b493bb6fa5415ce55d3f722221470
    Author: Fred Drake <fdrake@acm.org>
    Date:   Tue Mar 6 05:52:16 2001 +0000
    
        Move all knowledge that $(MAINOBJ) is built in the Modules/ directory
        into Makefile.pre.in; the configure script will only determine the basename
        of the file.
        
        This fixes installation of a Python built using C++, reported by Greg
        Wilson.
    
    -       $(INSTALL_DATA) Modules/python.o $(LIBPL)/python.o
    +       $(INSTALL_DATA) Modules/$(MAINOBJ) $(LIBPL)/$(MAINOBJ)
    
    commit 85515ad9795ffc3b676cbddeeea2b003818a2623
    Author: Neil Schemenauer <nascheme@enme.ucalgary.ca>
    Date:   Wed Jan 24 17:11:43 2001 +0000
    
        Flat makefile based on toplevel Makefile.in and makefiles in build
        subdirectories.  Those other makefiles will go away eventually.
    
    +       $(INSTALL_DATA) Modules/python.o $(LIBPL)/python.o
    

    @vstinner
    Copy link
    Member Author

    At commit 85515ad, "make install" installs the following files in the config/ directory:

    $ find /opt/pyold/lib/python2.1/config/
    /opt/pyold/lib/python2.1/config/
    /opt/pyold/lib/python2.1/config/libpython2.1.a
    /opt/pyold/lib/python2.1/config/config.c
    /opt/pyold/lib/python2.1/config/python.o
    /opt/pyold/lib/python2.1/config/config.c.in
    /opt/pyold/lib/python2.1/config/Makefile
    /opt/pyold/lib/python2.1/config/Setup
    /opt/pyold/lib/python2.1/config/Setup.local
    /opt/pyold/lib/python2.1/config/Setup.config
    /opt/pyold/lib/python2.1/config/makesetup
    /opt/pyold/lib/python2.1/config/install-sh
    /opt/pyold/lib/python2.1/config/Makefile.pre.in

    @vstinner
    Copy link
    Member Author

    python.o is the object file build by the C compiler from Programs/python.c. I don't see why anyone would need such object file.

    Programs/python.c:
    ---
    /* Minimal main program -- everything is loaded from the library */

    #include "Python.h"
    
    #ifdef MS_WINDOWS
    int
    wmain(int argc, wchar_t **argv)
    {
        return Py_Main(argc, argv);
    }
    #else
    int
    main(int argc, char **argv)
    {
        return Py_BytesMain(argc, argv);
    }
    #endif

    $ objdump -d /opt/pymaster/lib/python3.10/config-3.10-x86_64-linux-gnu/python.o
    (...)
    Disassembly of section .text:
    0000000000000000 <main>:
       0:	55                   	push   %rbp
       1:	48 89 e5             	mov    %rsp,%rbp
       4:	48 83 ec 10          	sub    $0x10,%rsp
       8:	89 7d fc             	mov    %edi,-0x4(%rbp)
       b:	48 89 75 f0          	mov    %rsi,-0x10(%rbp)
       f:	48 8b 55 f0          	mov    -0x10(%rbp),%rdx
      13:	8b 45 fc             	mov    -0x4(%rbp),%eax
      16:	48 89 d6             	mov    %rdx,%rsi
      19:	89 c7                	mov    %eax,%edi
      1b:	e8 00 00 00 00       	callq  20 <main+0x20>
      20:	c9                   	leaveq 
      21:	c3                   	retq   
    
    $ objdump -t /opt/pymaster/lib/python3.10/config-3.10-x86_64-linux-gnu/python.o
    (...)
    0000000000000000         *UND*	0000000000000000 Py_BytesMain

    @ned-deily
    Copy link
    Member

    I don't know for sure why python.o is included but perhaps the intent was to allow a user to rebuild a static interpreter with an user-supplied extension as is hinted at in https://docs.python.org/dev/extending/extending.html#compilation-and-linkage

    @vstinner
    Copy link
    Member Author

    Oh! I managed to build a static Python with the following commands:

    cd /opt/pymaster/lib/python3.10/config-3.10-x86_64-linux-gnu
    gcc -static -o ~/python-static python.o -L. $(./python-config.py --libs --embed)

    I get a static binary:

    $ ldd ~/python-static 
    	not a dynamic executable

    And it works as expected:

    $ ~/python-static 
    Python 3.10.0a2+ (heads/master:1f73c320e2, Nov 10 2020, 11:47:55) 
    [GCC 10.2.1 20201016 (Red Hat 10.2.1-6)] on linux
    Type "help", "copyright", "credits" or "license" for more information.
    >>> 

    But... does any user really build Python manually after Python is installed? The Python build system couldn't handle that as part of the regular build? Maybe using a new configure --enable-static flag?

    I'm not a fan of static binaries. For example, the linker emits many warnings about static linking:

    /usr/bin/ld: ./libpython3.10.a(dynload_shlib.o): in function _PyImport_FindSharedFuncptr': /home/vstinner/python/master/./Python/dynload_shlib.c:100: warning: Using 'dlopen' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking /usr/bin/ld: ./libpython3.10.a(posixmodule.o): in function os_getgrouplist_impl':
    /home/vstinner/python/master/./Modules/posixmodule.c:7411: warning: Using 'getgrouplist' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking
    /usr/bin/ld: ./libpython3.10.a(posixmodule.o): in function os_initgroups_impl': /home/vstinner/python/master/./Modules/posixmodule.c:7610: warning: Using 'initgroups' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking /usr/bin/ld: ./libpython3.10.a(pwdmodule.o): in function pwd_getpwall_impl':
    /home/vstinner/python/master/./Modules/pwdmodule.c:302: warning: Using 'getpwent' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking
    /usr/bin/ld: /home/vstinner/python/master/./Modules/pwdmodule.c:301: warning: Using 'setpwent' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking
    /usr/bin/ld: /home/vstinner/python/master/./Modules/pwdmodule.c:307: warning: Using 'endpwent' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking
    /usr/bin/ld: ./libpython3.10.a(pwdmodule.o): in function pwd_getpwnam_impl': /home/vstinner/python/master/./Modules/pwdmodule.c:249: warning: Using 'getpwnam_r' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking /usr/bin/ld: ./libpython3.10.a(pwdmodule.o): in function pwd_getpwuid':
    /home/vstinner/python/master/./Modules/pwdmodule.c:166: warning: Using 'getpwuid_r' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking

    @pablogsal
    Copy link
    Member

    But... does any user really build Python manually after Python is installed? The Python build system couldn't handle that as part of the regular build? Maybe using a new configure --enable-static flag?

    Embedding? But they should use the .a not any of the .o. Object files should not be packaged IMHO.

    @ned-deily
    Copy link
    Member

    Embedding?

    The example I cited from the docs was for extending Python by building a new interpreter using an installed Python, not embedding. Dunno how often that is done, though. Perhaps on Windows where it was more difficult to build an interpreter from source? Perhaps Nick might have some ideas since he was involved in that section of the docs.

    @nascheme
    Copy link
    Member

    I think the comments are correct in that it is used to create a new statically linked interpreter that includes a user provided extension module. We could include python.o inside the libpythonXX.a file but then I think it breaks if you are embedding (e.g. linking .a to your own .o that has a main function).

    It seems probable that no one uses python.o anymore but my preference would be to not remove it unless we are going to completely remove support for statically linking. Nothing is really hurt by having that .o file in the install and the feature still works as originally intended.

    It would be good to add a comment in the makefile, describing the possible use for that file.

    @vstinner
    Copy link
    Member Author

    The Fedora package of Python doesn't build the static libpython for 10 years (patch added at 2010-01-18):
    https://src.fedoraproject.org/rpms/python3.9/blob/master/f/00111-no-static-lib.patch

    libpython3.10.a (static) is around 15 MB and libpython3.10.so (dynamic) is around 3 MB. Fedora only installs libpython3.10.so (dynamic).

    @pablogsal
    Copy link
    Member

    Hummm, I think the gains of having the .o over the .a are very very small as you can almost recreate anything you want with the .o using the .a and a small initialization function unless I am missing something.

    @vstinner
    Copy link
    Member Author

    vstinner commented Feb 2, 2021

    I created bpo-43103 "Add configure option to disable build libpython.a and don't install python.o".

    @vstinner
    Copy link
    Member Author

    I consider that this issue is now fixed by bpo-43103 with the addition of a new configure --without-static-libpython option.

    If someone prefers to remove python.o and get it from libpython.a, please open a separated issue.

    @ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
    Labels
    3.10 only security fixes build The build process and cross-build
    Projects
    None yet
    Development

    No branches or pull requests

    4 participants