New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
makesetup: must link C extensions to libpython when compiled in shared mode #78995
Comments
Python can be compiled in "shared" mode: "./configure --enable-shared", Py_ENABLE_SHARED is defined in pyconfig.h. Most Linux distributions use this configuration. By default, Python builds most C extensions using setup.py which is based on distutils. The get_libraries() method of Lib/distutils/command/build_ext.py explicity add a dependency to libpythonX.Y if Py_ENABLE_SHARED is defined. But it is possible to use Modules/Setup to build some C extensions using Makefile rather than setup.py. If "*shared*" is in Modules/Setup, following modules will be compiled as libraries (".so" files on Linux). For example, RHEL and Fedora use this configuration for many C extensions. Problem: C extensions compiled like are not linked to libpython. Example of the issue on Fedora 28 with Python 2.7: $ ldd $(python2 -c 'import _struct; print(_struct.__file__)')
linux-vdso.so.1 (0x00007ffeedf38000)
libpthread.so.0 => /lib64/libpthread.so.0 (0x00007fb4da876000)
libc.so.6 => /lib64/libc.so.6 (0x00007fb4da4b7000)
/lib64/ld-linux-x86-64.so.2 (0x00007fb4daca1000) => notice the lack of libpython Python 3.6 is fine: $ ldd $(python3 -c 'import _struct; print(_struct.__file__)')
linux-vdso.so.1 (0x00007ffd493dd000)
libpython3.6m.so.1.0 => /lib64/libpython3.6m.so.1.0 (0x00007f47b9160000)
... Patch used by Fedora to build _struct (and other modules) using Makefile: https://src.fedoraproject.org/rpms/python2/blob/f27/f/python-2.7.1-config.patch Another example of patch, to build _contextvars as a shared library: diff --git a/Modules/Setup b/Modules/Setup
index a0622cc8c6..975aeff70d 100644
--- a/Modules/Setup
+++ b/Modules/Setup
@@ -148,7 +148,7 @@ _symtable symtablemodule.c
# modules are to be built as shared libraries (see above for more
# detail; also note that *static* or *disabled* cancels this effect):
-#*shared*
+*shared*
# GNU readline. Unlike previous Python incarnations, GNU readline is
# now incorporated in an optional module, configured in the Setup file
@@ -166,7 +166,7 @@ _symtable symtablemodule.c
#array arraymodule.c # array objects
#cmath cmathmodule.c _math.c # -lm # complex math library functions
#math mathmodule.c _math.c # -lm # math library functions, e.g. sin()
-#_contextvars _contextvarsmodule.c # Context Variables
+_contextvars _contextvarsmodule.c # Context Variables
#_struct _struct.c # binary structure packing/unpacking
#_weakref _weakref.c # basic weak reference support
#_testcapi _testcapimodule.c # Python C API test module Attached PR fixes Modules/makesetup to:
|
Setup.patch: Example of patch to modify Modules/Setup to compile _contextvars as a shared library, to test the fix. |
Example of the bug: --- $ git apply ~/Setup.patch
$ ./configure --with-pydebug --enable-shared
$ make
$ grep _contextvars Makefile
(...) Modules/_contextvarsmodule.o: Modules/_contextvars$(EXT_SUFFIX): Modules/_contextvarsmodule.o; $(BLDSHARED) Modules/_contextvarsmodule.o -o Modules/_contextvars$(EXT_SUFFIX) $ find -name "_contextvars.*so"
./Modules/_contextvars.cpython-38dm-x86_64-linux-gnu.so
$ ldd $(find -name "_contextvars.*so")
linux-vdso.so.1 (0x00007ffd27973000)
libpthread.so.0 => /lib64/libpthread.so.0 (0x00007fd081433000)
libc.so.6 => /lib64/libc.so.6 (0x00007fd081074000)
/lib64/ld-linux-x86-64.so.2 (0x00007fd081854000) The _contextvars shared library is not linked to libpython. There is not "-lpythonX.Y" in the Makefile rule. Now with the patch: $ git clean -fdx
$ git apply ~/Setup.patch
$ ./configure --with-pydebug --enable-shared
$ make
$ grep _contextvars Makefile
(...) Modules/_contextvarsmodule.o: Modules/_contextvars$(EXT_SUFFIX): Modules/_contextvarsmodule.o $ find -name "_contextvars.*so"
./Modules/_contextvars.cpython-38dm-x86_64-linux-gnu.so
$ ldd $(find -name "_contextvars.*so")
linux-vdso.so.1 (0x00007ffd1e918000)
libpython3.8dm.so.1.0 => not found
(...) With my patch, _contextvars.cpython-38dm-x86_64-linux-gnu.so is linked to libpython3.8dm.so.1.0 as expected. The Makefile rule adds |
Downstream (RHEL) issue: |
I copied the nosy list from bpo-32430: people who understand and care about the Modules/Setup file :-) |
Why do you call this a bug? I even opened an issue about that: bpo-21536 |
Of course, one workaround to satisfy everyone would be to build a (empty) libpython.so even on static Python builds. But I'm not sure Debian/Ubuntu would package it. |
I search if C extensions of the Python standard libraries are always linked or not to libpython... it's complicated. I tested _ctypes, _hashlib and _struct modules:
It means that using dlopen("libpython2.7.so.1.0", RTLD_LOCAL | RTLD_NOW) may or may not work depending on the Linux distribution and depending on the imported C extensions... If we use the example of Fedora: some C extensions are compiled using Makefile (the Fedora package modifies Modules/Setup, as I showed previously), but others are compiled by setup.py. For example, _ctypes and _hashlib are compiled by setup.py. |
Ah, it seems like the bpo-832799 (reported in 2003) is similar to the RHEL bug: Extract of the RHEL bug report: pythontest.c:
#include <dlfcn.h>
int main(int argc, char *argv[])
{
void *pylib = dlopen("libpython2.7.so.1.0", RTLD_LOCAL | RTLD_NOW);
void (*Py_Initialize)(void) = dlsym(pylib, "Py_Initialize");
Py_Initialize();
int (*PyRun_SimpleStringFlags)(const char *, void *) = dlsym(pylib, "PyRun_SimpleStringFlags");
PyRun_SimpleStringFlags("import json\n", 0);
return 0;
}
Actual results: it will fail with ImportError: /usr/lib64/python2.7/lib-dynload/_struct.so: undefined symbol: PyFloat_Type The reporter is already aware of the fallback on RTLD_GLOBAL: "(optionally) change RTLD_LOCAL to RTLD_GLOBAL and see that it works". |
Le 27/09/2018 à 12:49, STINNER Victor a écrit :
Do you realize libpython.so doesn't exist on Debian / Ubuntu? |
Extract of Antoine's comment on bpo-21536:
Oh. I didn't notice this major difference...
|
Note: _ctypes, _hashlib and _struct are all linked to libpython, on Python 2 and Python 3. Antoine:
No, I didn't :-) |
Antoine:
"./configure --enable-shared && make" links C extensions to libpython. It's surprising that C extensions compiled by Makefile behave differently (not linked to libpython). We need consistency: either *never* link to libpython, or *always* link to libpython. |
Is it a real use case? Why would anyone use a RHEL binary on Debian? Debian already provides the full standard library. C extensions of the standard library are tidily coupled to CPython. For example, it may be dangerous to use a C extension of Python 2.7.5 on Python 2.7.15. I'm talking about the very specific case of C extensions which are part of the stdlib. Third party C extensions distributed as portable wheel packages using the stable ABI is different use case. |
I'm not talking about the standard library obviously. I don't remember my original use case exactly, but I must have been compiling a C extension on a system and expected it to work on another.
I don't believe that. Binary wheels uploaded to PyPI seem to work fine regardless of the exact bugfix version.
Most wheel packages don't use the stable ABI. They are tied to a Python version such as 2.7, but they don't differentiate between e.g. 2.7.5 and 2.7.15. We don't break the ABI between bugfix releases. |
It seems like we are talking about two different things:
|
But as you said, we need consistency: either *never* link to libpython, or *always* link to libpython. You are proposing to always link, I'm arguing we should never link. |
Debian/Ubuntu doesn't link against the library because it would add dependencies on all supported Python versions. Normally this is just during transition times, but e.g. for the upcoming Ubuntu 18.10 release we didn't finish the transition and so ship two Python3 versions. The packaging tools would add package dependencies on both 3.6 and 3.7 what you don't want. |
Not an expert of Python build, but I've been creating a few « reverse engineer challenge » where I had to ship modified version of the interpreter, so played with it a bit. I agree consistency is nice to reason about. It looks better to me to not link with libpython.so directly. This is probably better as this does not make As a short check, I ran
and everything looks fine, so all symbols should already be in the interpreter. I've also checked whether that's an issue or not for user-defined native extensions and everything runs smoothly without the explicit dep. So the argument would be: why adding this dep when it's not needed? |
I wrote the PR 9912 to ensure that C extensions are never linked to libpython. I tested my change using: git clean -fdx I tested 3 configurations on my Fedora 28 (Linux):
(*) I modified Modules/Setup to compile 37 C extensions as shared libraries using Makefile. Extract of Modules/Setup: errno errnomodule.c # posix (UNIX) errno values |
Perhaps you should bring up this proposed change in distutils-sig before committing. It's probably an OK change but it would be good to try to get some feedback from the downstream users who might be affected by it. |
Another Fedora on Python2: |
This issue has been closed as "not a bug". -- Since this issue has been created, no consensus could be found. So I close the issue to keep the status quo. In short, RTLD_LOCAL is not supported. I also close this issue as not a bug. |
What do you mean, "no consensus could be found"? I don't see anyone objecting the change. |
I propose a change to always link and a change to never link. I don't see any tracking towards one option. It seems like there are issues on Android. Anyway, this issue only seems to be theoretical since libpython must not be used with RTLD_LOCAL. |
FYI I modified Python 3.8 to never link C extensions to libpython on Unix (except on Android): bpo-21536, but with a different rationale (better reasons to do so). |
I am sorry, this is not a workable stance. |
reimar: "I am sorry, this is not a workable stance. This does not just affect loading libpython directly. (...)" This issue is now closed, as bpo-21536. Would you mind to open a new issuse to clearly explain your own case? Please mention your platform. |
Sorry for my laziness. I opened bpo-36753. |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
bugs.python.org fields:
The text was updated successfully, but these errors were encountered: