Python cannot be embedded in shared library due to dependency problems
with lib-dynload.

I am implementing a shared library (in C) that implements a specific API
as a dynamically loadable plugin to an application. To implement this
library, I am calling out to Python via the C API.

When my code tries to load a Python module, it fails. This is because
lib-dynload/time.so (for example) cannot resolve symbols that are in
libpython2.5.so because it never looks there.

In this test case I'm trying to import "time", but it seems to fail on
other modules too - presumably anything that is in lib-dynload. It also
fails if I import a pure Python module that then tries to import time.

The error produced is: ImportError:
/usr/lib/python2.5/lib-dynload/time.so: undefined symbol: PyExc_ValueError

From LD_DEBUG:
     29682:file=/usr/lib/python2.5/lib-dynload/time.so [0];  needed by 
/usr/lib/libpython2.5.so.1.0 [0]
...
     29682:relocation processing: /usr/lib/python2.5/lib-dynload/time.so
     29682:symbol=PyExc_ValueError;  lookup in file=./myprog [0]
     29682:symbol=PyExc_ValueError;  lookup in file=/lib/libdl.so.2 [0]
     29682:symbol=PyExc_ValueError;  lookup in file=/lib/libc.so.6 [0]
     29682:symbol=PyExc_ValueError;  lookup in
file=/lib64/ld-linux-x86-64.so.2 [0]
     29682:symbol=PyExc_ValueError;  lookup in 
file=/usr/lib/python2.5/lib-dynload/time.so [0]
     29682:symbol=PyExc_ValueError;  lookup in file=/lib/libm.so.6 [0]
     29682:symbol=PyExc_ValueError;  lookup in file=/lib/libpthread.so.0 [0]
     29682:symbol=PyExc_ValueError;  lookup in file=/lib/libc.so.6 [0]
     29682:symbol=PyExc_ValueError;  lookup in
file=/lib64/ld-linux-x86-64.so.2 [0]
     29682:/usr/lib/python2.5/lib-dynload/time.so: error: symbol lookup
error: 
undefined symbol: PyExc_ValueError (fatal)


$ nm -D /usr/lib/libpython2.5.so.1|grep PyExc_ValueError
000000000033a7e0 D PyExc_ValueError



$ ldd /usr/lib/python2.5/lib-dynload/time.so
libm.so.6 => /lib/libm.so.6 (0x00002afe014c9000)
libpthread.so.0 => /lib/libpthread.so.0 (0x00002afe0174a000)
libc.so.6 => /lib/libc.so.6 (0x00002afe01967000)
/lib64/ld-linux-x86-64.so.2 (0x0000555555554000)



I am told that the problem is that lib-dynload/*.so should depend on
libpython2.5.so.1.


Test case attached. mylib.c is the library that I'm implementing reduced
to the problem case. myprog.c is a stub program that loads mylib.c to
demonstrate the problem. The "real" myprog would be any third-party
application that I have no control over that expects to be able to
dlopen() mylib.so and call functions within it.


I have been given the following workaround: in mylib.c, before
PyInitialize() I can call dlopen("libpython2.5.so", RTLD_LAZY |
RTLD_GLOBAL);

This works, but I believe that lib-dynload/*.so should depend on
libpython2.5.so.1 so this hack should not be necessary.

I am using Ubuntu 8.04 with Python version 2.5.2-2ubuntu4.1.

ubuntu comes with a static libpython.a which you can also link against.
Linking the c modules against the shared python library would make that
static library pretty much useless...

What is the purpose of linking to the static library if you're still
going to depend on shared objects in lib-dynload?

Besides, .so files should always declare their dependencies.

Yes, there is no purpose in linking with a static library when the
dynamically loaded modules still depend on the shared library.
And yes, the dynamically loaded libraries should declare their
dependencies. My point is they do *not* depend on the shared library.
That's why the static lib currently makes sense on ubuntu, but wouldn't
make any sense when your proposal would be implemented.

I recommend closing this issue as invalid.

Linking with the static library really works. You have to link with
'-Xlinker -export-dynamic' however on linux in order to make the symbols
from the static library visible. (just in case if I haven't been clear..)

The problem, and the reason for the existence of this bug, is that I
cannot build a shared object that links to libpython2.5.so.1 and works.

Please don't mark this bug invalid until this problem is fixed.

My proposal of adding dependencies to lib-dynload/*.so is just a
suggestion on how to go about fixing this bug. If this is not a suitable
solution, please suggest a solution that is.


In response to your discussion on why the static library is currently
useful, you haven't really answered the question. In what circumstance
would someone actually want to build against the static library and then
dynamically load other libraries?

"Because it works" isn't an answer, as if this bug were fixed, then
building against the shared library would validly have the same response.

Why static over dynamic, if when static you have to link to dynamic anyway?

If Ubuntu wants to deliver a static libpython, it also needs to deliver
static versions of the Python modules, or accept that they're not usable
from a static libpython.

It makes no sense at all to mix libpython.a with these .so files. Many of
them declare dynamic dependencies on stuff like libkrb5.so or
libsqlite3.so. I can't conceive of a scenario where you need a static
libpython AND you're OK with both dlopen() of the Python modules and any
of its dynamic dependencies. What is this case?

no john, linking with the static library really works. the resulting
executable does not depend on the shared library and it is possible to
import the e.g. the time module.

Ralf,

I'm sure it does work, but what is the point of linking statically to
libpython.a but then having other dependencies, for example on
lib-dynload/time.so? Why not just link to libpython2.5.so in the first
place?

> I'm sure it does work, but what is the point of linking statically
> to libpython.a but then having other dependencies, for example on
> lib-dynload/time.so? Why not just link to libpython2.5.so in the
> first place?

speed. Using a python executable with a statically linked libpython is
about 10% faster on ix86, plus with some compilers on linux profile
based optimization can be used on static objects, which doesn't work
with shared libraries (at least on linux using gcc).

Ralf,

Linking against the static library (with '-Xlinker -export-dynamic') did
_not_ solve the problem for me.

gcc `python2.6-config --cflags` -c -fPIC mylib.c -o mylib.o -Xlinker
-export-dynamic

gcc -Xlinker -export-dynamic -lpthread -ldl -lutil -lm -lz -shared
-Wl,-soname,mylib.so -o mylib.so  mylib.o 
/usr/lib/python2.6/config/libpython2.6.a

gcc `python2.6-config --cflags` -Xlinker -export-dynamic -ldl -lutil -lm
-lz  -o main main.c /usr/lib/python2.6/config/libpython2.6.a
----------------------

`python2.6-config --cflags` returns
-I/usr/include/python2.6 -I/usr/include/python2.6 -fno-strict-aliasing
-DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes

--------
Ubuntu Jaunty
Linux yonas-laptop 2.6.28-11-generic #42-Ubuntu SMP Fri Apr 17 01:57:59
UTC 2009 i686 GNU/Linux

Same error message: 

import dl: ImportError: /usr/lib/python2.6/lib-dynload/dl.so: undefined
symbol: PyExc_ValueError

Here's my log file (`LD_DEBUG=all ./main`)

Updated log file.

rb's workaround works, though.

Note that on Ubuntu Hardy at least, the libpython2.5.so symlink only
exists in the python2.5-dev package. This means that in order for the
library to work without the dev package installed, it should dlopen
libpython2.5.so.1 instead.

Does it help if you add the --whole-archive option when linking with the
static library? As explained there:
http://www.lysium.de/blog/index.php?/archives/222-Lost-static-objects-in-static-libraries-with-GNU-linker-ld.html
This way your main program will contain the whole python interpreter,
and all symbols used by time.so &co will be resolved.

It is not obvious that this issue is about a Python bug, rather than how to use on a specific system, or that it applies to any current version. If this really is a Python bug and applies to a current version, respond to re-open.

Terry, what in the above discussion  leads you to believe there is no bug?

The original complaint "Python cannot be embedded in shared library due to dependency problems with lib-dynload" is still valid.

I carefully and specifically questioned the existence of current 'Python bug', which is what this tracker is for. For Linux, that would mean a bug in any of the source files in a current 2.7, 3.1.2, or 3.2.1a1 tarball release or the newer source repository versions thereof. But I do not see any claim that there is any problem with any of the files. There certainly is no bug report for any specific stdlib module, which is what is meant by the Library (Lib) component.

This issue *is* about lib-dynload that I gather is supplied by Ubuntu. There may or may not be a bug in how Ubuntu packages, compiles, and loads Python 2.5.2. John, your msg76536, beginning "If *Ubuntu*..., it [Ubuntu] also needs..." suggests that. But unless you *know* that that results from a bug in the files downloaded by Ubuntu, you should take that up with them.

So I will follow Ralf's suggestion and close this, and suggest in turn that this issue be taken to an Ubuntu support list where help is more likely. There might even be knowledgeable Ubuntu users on python-list, I don't know.

Another possible suggestion to the OP is compile everything yourself with whatever compile flags you need to get all your stuff to work together. In particular, can you make your own lib-dynload that acts the way you want it to work? (But here is not the place to ask about how.)

Original reporter here.

lib-dynload is part of Python's dynamic loading mechanism. Perhaps somewhere like Python/dynload_dl.c is the relevant code?

Lib seemed like the right place to put the bug at the time, since it was the stdlib module files that were missing dependency declarations. Although perhaps Extension Modules would have been better?

In any case, I've just tried to reproduce the bug with 2.7 (on Debian Squeeze pulling version 2.7-2 of python2.7 and python2.7-dev from experimental), and failed (both 32- and 64-bit).

So it seems that this bug has been fixed at some point since 2.5.

> perhaps Extension Modules would have been better?

For future reference (like if some 3.x is not ok), yes

> So it seems that this bug has been fixed at some point since 2.5.

Great!

The bug is still present on the default build which many distros uses which produces an invalid build. That means

    ./configure && make && make install

is broken! --enable-shared should be turned on by default because many (most?) distros build with the default settings.