This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: make install must not copy python.o into $prefix/lib/python3.10/config-3.10-x86_64-linux-gnu/
Type: Stage: resolved
Components: Build Versions: Python 3.10
process
Status: closed Resolution: duplicate
Dependencies: Superseder: Add configure --without-static-libpython to not build libpython3.10.a
View: 43103
Assigned To: Nosy List: hroncok, nascheme, ncoghlan, ned.deily, pablogsal, vstinner
Priority: normal Keywords: patch

Created on 2020-11-10 10:49 by vstinner, last changed 2022-04-11 14:59 by admin. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 23219 closed vstinner, 2020-11-10 11:44
Messages (13)
msg380650 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-11-10 10:49
Commands reproduce the issue:

cd /path/to/python/source/
./configure --prefix /opt/pymaster CFLAGS="-O0"
make
make install

make install copies python.o:

$ find /opt/pymaster/ -name "*.o"
/opt/pymaster/lib/python3.10/config-3.10-x86_64-linux-gnu/python.o

This file is useless and must not be installed by make install.

The issue was first discovered in Fedora:
https://bugzilla.redhat.com/show_bug.cgi?id=1894462
msg380652 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-11-10 11:13
python.o is installed by "make libainstall". It is done since 2001 (commit 85515ad9795ffc3b676cbddeeea2b003818a2623).

Git history:

commit 49fd7fa4431da299196d74087df4a04f99f9c46f
Author: Thomas Wouters <thomas@python.org>
Date:   Fri Apr 21 10:40:58 2006 +0000

    Merge p3yk branch with the trunk up to revision 45595. This breaks a fair
    number of tests, all because of the codecs/_multibytecodecs issue described
    here (it's not a Py3K issue, just something Py3K discovers):
    http://mail.python.org/pipermail/python-dev/2006-April/064051.html
    (...)

-       $(INSTALL_DATA) Modules/$(MAINOBJ) $(DESTDIR)$(LIBPL)/$(MAINOBJ)
+       $(INSTALL_DATA) Modules/python.o $(DESTDIR)$(LIBPL)/python.o

commit a1a84e7d4f7b493bb6fa5415ce55d3f722221470
Author: Fred Drake <fdrake@acm.org>
Date:   Tue Mar 6 05:52:16 2001 +0000

    Move all knowledge that $(MAINOBJ) is built in the Modules/ directory
    into Makefile.pre.in; the configure script will only determine the basename
    of the file.
    
    This fixes installation of a Python built using C++, reported by Greg
    Wilson.

-       $(INSTALL_DATA) Modules/python.o $(LIBPL)/python.o
+       $(INSTALL_DATA) Modules/$(MAINOBJ) $(LIBPL)/$(MAINOBJ)

commit 85515ad9795ffc3b676cbddeeea2b003818a2623
Author: Neil Schemenauer <nascheme@enme.ucalgary.ca>
Date:   Wed Jan 24 17:11:43 2001 +0000

    Flat makefile based on toplevel Makefile.in and makefiles in build
    subdirectories.  Those other makefiles will go away eventually.

+       $(INSTALL_DATA) Modules/python.o $(LIBPL)/python.o
msg380653 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-11-10 11:27
At commit 85515ad9795ffc3b676cbddeeea2b003818a2623, "make install" installs the following files in the config/ directory:

$ find /opt/pyold/lib/python2.1/config/
/opt/pyold/lib/python2.1/config/
/opt/pyold/lib/python2.1/config/libpython2.1.a
/opt/pyold/lib/python2.1/config/config.c
/opt/pyold/lib/python2.1/config/python.o
/opt/pyold/lib/python2.1/config/config.c.in
/opt/pyold/lib/python2.1/config/Makefile
/opt/pyold/lib/python2.1/config/Setup
/opt/pyold/lib/python2.1/config/Setup.local
/opt/pyold/lib/python2.1/config/Setup.config
/opt/pyold/lib/python2.1/config/makesetup
/opt/pyold/lib/python2.1/config/install-sh
/opt/pyold/lib/python2.1/config/Makefile.pre.in
msg380655 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-11-10 11:47
python.o is the object file build by the C compiler from Programs/python.c. I don't see why anyone would need such object file.

Programs/python.c:
---
/* Minimal main program -- everything is loaded from the library */

#include "Python.h"

#ifdef MS_WINDOWS
int
wmain(int argc, wchar_t **argv)
{
    return Py_Main(argc, argv);
}
#else
int
main(int argc, char **argv)
{
    return Py_BytesMain(argc, argv);
}
#endif
---

$ objdump -d /opt/pymaster/lib/python3.10/config-3.10-x86_64-linux-gnu/python.o
(...)
Disassembly of section .text:
0000000000000000 <main>:
   0:	55                   	push   %rbp
   1:	48 89 e5             	mov    %rsp,%rbp
   4:	48 83 ec 10          	sub    $0x10,%rsp
   8:	89 7d fc             	mov    %edi,-0x4(%rbp)
   b:	48 89 75 f0          	mov    %rsi,-0x10(%rbp)
   f:	48 8b 55 f0          	mov    -0x10(%rbp),%rdx
  13:	8b 45 fc             	mov    -0x4(%rbp),%eax
  16:	48 89 d6             	mov    %rdx,%rsi
  19:	89 c7                	mov    %eax,%edi
  1b:	e8 00 00 00 00       	callq  20 <main+0x20>
  20:	c9                   	leaveq 
  21:	c3                   	retq   

$ objdump -t /opt/pymaster/lib/python3.10/config-3.10-x86_64-linux-gnu/python.o
(...)
0000000000000000         *UND*	0000000000000000 Py_BytesMain
msg380671 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2020-11-10 14:17
I don't know for sure why python.o is included but perhaps the intent was to allow a user to rebuild a static interpreter with an user-supplied extension as is hinted at in https://docs.python.org/dev/extending/extending.html#compilation-and-linkage
msg380674 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-11-10 14:30
Oh! I managed to build a static Python with the following commands:

cd /opt/pymaster/lib/python3.10/config-3.10-x86_64-linux-gnu
gcc -static -o ~/python-static python.o -L. $(./python-config.py --libs --embed)

I get a static binary:

$ ldd ~/python-static 
	not a dynamic executable

And it works as expected:

$ ~/python-static 
Python 3.10.0a2+ (heads/master:1f73c320e2, Nov 10 2020, 11:47:55) 
[GCC 10.2.1 20201016 (Red Hat 10.2.1-6)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> 


But... does any user really build Python manually after Python is installed? The Python build system couldn't handle that as part of the regular build? Maybe using a new configure --enable-static flag?


I'm not a fan of static binaries. For example, the linker emits many warnings about static linking:

/usr/bin/ld: ./libpython3.10.a(dynload_shlib.o): in function `_PyImport_FindSharedFuncptr':
/home/vstinner/python/master/./Python/dynload_shlib.c:100: warning: Using 'dlopen' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking
/usr/bin/ld: ./libpython3.10.a(posixmodule.o): in function `os_getgrouplist_impl':
/home/vstinner/python/master/./Modules/posixmodule.c:7411: warning: Using 'getgrouplist' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking
/usr/bin/ld: ./libpython3.10.a(posixmodule.o): in function `os_initgroups_impl':
/home/vstinner/python/master/./Modules/posixmodule.c:7610: warning: Using 'initgroups' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking
/usr/bin/ld: ./libpython3.10.a(pwdmodule.o): in function `pwd_getpwall_impl':
/home/vstinner/python/master/./Modules/pwdmodule.c:302: warning: Using 'getpwent' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking
/usr/bin/ld: /home/vstinner/python/master/./Modules/pwdmodule.c:301: warning: Using 'setpwent' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking
/usr/bin/ld: /home/vstinner/python/master/./Modules/pwdmodule.c:307: warning: Using 'endpwent' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking
/usr/bin/ld: ./libpython3.10.a(pwdmodule.o): in function `pwd_getpwnam_impl':
/home/vstinner/python/master/./Modules/pwdmodule.c:249: warning: Using 'getpwnam_r' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking
/usr/bin/ld: ./libpython3.10.a(pwdmodule.o): in function `pwd_getpwuid':
/home/vstinner/python/master/./Modules/pwdmodule.c:166: warning: Using 'getpwuid_r' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking
msg380682 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2020-11-10 16:03
> But... does any user really build Python manually after Python is installed? The Python build system couldn't handle that as part of the regular build? Maybe using a new configure --enable-static flag?

Embedding? But they should use the .a not any of the .o. Object files should not be packaged IMHO.
msg380685 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2020-11-10 16:08
> Embedding?

The example I cited from the docs was for extending Python by building a new interpreter using an installed Python, not embedding. Dunno how often that is done, though. Perhaps on Windows where it was more difficult to build an interpreter from source? Perhaps Nick might have some ideas since he was involved in that section of the docs.
msg380711 - (view) Author: Neil Schemenauer (nascheme) * (Python committer) Date: 2020-11-10 20:42
I think the comments are correct in that it is used to create a new statically linked interpreter that includes a user provided extension module.  We could include python.o inside the libpythonXX.a file but then I think it breaks if you are embedding (e.g. linking .a to your own .o that has a main function).

It seems probable that no one uses python.o anymore but my preference would be to not remove it unless we are going to completely remove support for statically linking.  Nothing is really hurt by having that .o file in the install and the feature still works as originally intended.

It would be good to add a comment in the makefile, describing the possible use for that file.
msg380722 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-11-10 22:30
The Fedora package of Python doesn't build the static libpython for 10 years (patch added at 2010-01-18):
https://src.fedoraproject.org/rpms/python3.9/blob/master/f/00111-no-static-lib.patch

libpython3.10.a (static) is around 15 MB and libpython3.10.so (dynamic) is around 3 MB. Fedora only installs libpython3.10.so (dynamic).
msg380724 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2020-11-10 22:35
Hummm, I think the gains of having the .o over the .a are very very small as you can almost recreate anything you want with the .o using the .a and a small initialization function unless I am missing something.
msg386150 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2021-02-02 15:24
I created bpo-43103 "Add configure option to disable build libpython.a and don't install python.o".
msg387144 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2021-02-17 10:59
I consider that this issue is now fixed by bpo-43103 with the addition of a new configure --without-static-libpython option.

If someone prefers to remove python.o and get it from libpython.a, please open a separated issue.
History
Date User Action Args
2022-04-11 14:59:37adminsetgithub: 86473
2021-02-17 10:59:40vstinnersetstatus: open -> closed
superseder: Add configure --without-static-libpython to not build libpython3.10.a
messages: + msg387144

resolution: duplicate
stage: patch review -> resolved
2021-02-02 15:24:07vstinnersetmessages: + msg386150
2021-01-27 10:28:02hroncoksetnosy: + hroncok
2020-11-10 22:35:05pablogsalsetmessages: + msg380724
2020-11-10 22:30:20vstinnersetmessages: + msg380722
2020-11-10 20:42:52naschemesetmessages: + msg380711
2020-11-10 16:08:10ned.deilysetnosy: + ncoghlan
messages: + msg380685
2020-11-10 16:03:10pablogsalsetmessages: + msg380682
2020-11-10 14:30:03vstinnersetmessages: + msg380674
2020-11-10 14:17:03ned.deilysetmessages: + msg380671
2020-11-10 11:47:54vstinnersetnosy: + ned.deily, pablogsal
messages: + msg380655
2020-11-10 11:44:07vstinnersetkeywords: + patch
stage: patch review
pull_requests: + pull_request22117
2020-11-10 11:27:46vstinnersetmessages: + msg380653
2020-11-10 11:13:08vstinnersetnosy: + nascheme
messages: + msg380652
2020-11-10 10:49:40vstinnercreate