classification
Title: extension built with a shared python cannot be loaded with a static python
Type: enhancement Stage:
Components: Library (Lib) Versions: Python 3.5
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: barry, dstufft, lemburg, loewis, merwok, ncoghlan, piotr.dobrogost, pitrou
Priority: normal Keywords:

Created on 2014-05-19 18:50 by pitrou, last changed 2016-02-07 16:38 by piotr.dobrogost.

Messages (6)
msg218806 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2014-05-19 18:50
When a C extension is built (using distutils) with a shared library Python, it cannot be loaded with an otherwise identical statically linked Python. The other way round works fine. Trivial example using the _ssl module:

>>> import sys
>>> sys.path.insert(0, '/home/antoine/cpython/shared/build/lib.linux-x86_64-3.5/')
>>> import _ssl
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: libpython3.5m.so.1.0: cannot open shared object file: No such file or directory

This is probably because of an additional -L flag that is passed when linking a C extension with a shared library Python. I don't think the flag is useful under Linux (or perhaps under any other OS at all), since the relevant symbols are already loaded when the interpreter tries to load the C extension.

(AFAIK, systems notorious for providing shared library Pythons are RedHat-alike systems, while Debian/Ubuntu provide statically linked Pythons)
msg218809 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2014-05-19 19:15
Actually, it's not a -L flag but a -l flag. Removing the "-lpython3.5m" flag from the linker line works fine under Linux, and allows the resulting extension to be loaded with both a shared libary Python and a statically-linked Python.
msg218813 - (view) Author: Martin v. Löwis (loewis) * (Python committer) Date: 2014-05-19 20:05
I think you are right. It would IMO be useful to research a few comparable systems. E.g. Apache modules don't link a shared library, but still refer to apr_ functions as undefined symbols - but then, there isn't an APR shared library in the first place (at least not on Debian - how about Redhat?)

PHP might be close to our case: Debian includes a libphp5.so (in /usr/lib/php5), yet neither /usr/bin/php5 nor the Apache libphp5.so link against it, and all the PHP modules (in /usr/lib/php5/20100525+lfs/) don't link with the shared library - on Debian. I wonder how it is on systems that actually use the PHP shared library.
msg219181 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2014-05-26 21:34
> PHP might be close to our case: Debian includes a libphp5.so
> (in /usr/lib/php5), yet neither /usr/bin/php5 nor the Apache libphp5.so 
> link against it, and all the PHP modules (in 
> /usr/lib/php5/20100525+lfs/) don't link with the shared library -
> on Debian. I wonder how it is on systems that actually use the PHP 
> shared library.

Regardless of libphp5.so, the modules in /usr/lib/php5/20100525 don't link against /usr/lib/apache2/modules/libphp5.so, though the latter is obviously able to load those modules.
msg219186 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2014-05-26 22:24
Hmm, apparently the -l flag was added in #832799, for a rather complicated case where the interpreter is linked with a library dlopened by an embedding application (I suppose for some kind of plugin system). The OP there also mentions RTLD_GLOBAL as a workaround (or perhaps the right way of achieving the desired effect).

(also, the OP didn't mention why he used a shared library build, instead of linking Python statically with the dlopened library)
msg220293 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2014-06-11 18:00
Martin, what do you think about the aforementioned use case?
History
Date User Action Args
2016-02-07 16:38:17piotr.dobrogostsetnosy: + piotr.dobrogost
2014-08-23 03:26:43pitrousetnosy: + lemburg
2014-07-22 00:45:30pitrousetnosy: + barry
2014-06-11 18:00:06pitrousetmessages: + msg220293
2014-05-26 22:24:49pitrousetmessages: + msg219186
2014-05-26 21:35:00pitrousetmessages: + msg219181
2014-05-19 20:05:09loewissetmessages: + msg218813
2014-05-19 19:15:20pitrousetmessages: + msg218809
2014-05-19 18:50:02pitroucreate