Add pkg-config python-3.8-embed and --embed to python3.8-config #80902

vstinner · 2019-04-25T18:25:49Z

BPO	36721
Nosy	@doko42, @vstinner, @ned-deily, @ambv, @hroncok
PRs	bpo-36721: Add --embed option to python-config #13500 bpo-36721: Fix pkg-config symbolic links on "make install" #13551 gh-59795: distinct python-config from pkg-config python #737

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2019-05-27.22:59:53.944>
created_at = <Date 2019-04-25.18:25:48.580>
labels = ['build', '3.8']
title = 'Add pkg-config python-3.8-embed and --embed to python3.8-config'
updated_at = <Date 2019-05-31.17:09:42.980>
user = 'https://github.com/vstinner'

bugs.python.org fields:

activity = <Date 2019-05-31.17:09:42.980>
actor = 'haubi'
assignee = 'none'
closed = True
closed_date = <Date 2019-05-27.22:59:53.944>
closer = 'vstinner'
components = ['Build']
creation = <Date 2019-04-25.18:25:48.580>
creator = 'vstinner'
dependencies = []
files = []
hgrepos = []
issue_num = 36721
keywords = ['patch']
message_count = 11.0
messages = ['340853', '342868', '342905', '343193', '343195', '343265', '343266', '343399', '343401', '343408', '343698']
nosy_count = 6.0
nosy_names = ['doko', 'vstinner', 'ned.deily', 'lukasz.langa', 'hroncok', 'swt2c']
pr_nums = ['13500', '13551', '737']
priority = None
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue36721'
versions = ['Python 3.8']

vstinner · 2019-04-25T18:25:48Z

The bpo-21536 modified how C extensions are built: they are no longer linked to libpython. The problem is that when an application embeds Python: the application wants to be linked to libpython.

commit 8c3ecc6 (HEAD -> master, upstream/master)
Author: Victor Stinner <vstinner@redhat.com>
Date: Thu Apr 25 20:13:10 2019 +0200

bpo-21536: C extensions are no longer linked to libpython (GH-12946)

On Unix, C extensions are no longer linked to libpython.

It is now possible to load a C extension built using a shared library
Python with a statically linked Python.

When Python is embedded, libpython must not be loaded with
RTLD_LOCAL, but RTLD_GLOBAL instead. Previously, using RTLD_LOCAL, it
was already not possible to load C extensions which were not linked
to libpython, like C extensions of the standard library built by the
"*shared*" section of [Modules/Setup](https://github.com/python/cpython/blob/main/Modules/Setup).

distutils, python-config and python-config.py have been modified.

I chose to modify distutils, python-config (shell) and python-config.py (Python), but *not* Misc/python.pc (pkg-config).

Previously, we had:

$ pkg-config python-3.7 --libs
-lpython3.7m 
$ python3.7-config --libs
-lpython3.7m -lcrypt -lpthread -ldl -lutil -lm 
$ python3.7-config.py --libs
-lpython3.7m -lcrypt -lpthread -ldl -lutil -lm

Now, we get:

$ pkg-config python-3.8 --libs
-lpython3.8
$ python3.8-config --libs
-lcrypt -lpthread -ldl -lutil -lm -lm 
$ python-config.py --libs
-lcrypt -lpthread -ldl -lutil -lm -lm

python-config and python-config.py can now be used to build C extensions, but not to build an application embedding Python.

pkg-config should not be used to build a C extenstion since it links to libpython, but we don't want to do that (see bpo-21536).

I'm not sure that different tools should return different results.

I propose:

Add a new command "pkg-config python-3.8-embed"
Add a new "--embed" option to python3.8-config and python3.8-config.py commands
Remove "-lpython@VERSION@@ABIFLAGS@" from "Libs: -L${libdir} -lpython@VERSION@@ABIFLAGS@" of Misc/python.pc.in

On Windows, we already provide different binaries for embedded Python with "-embed" suffix:

Download Windows x86-64 embeddable zip file: python-3.7.3-embed-amd64.zip
Download Windows x86-64 executable installer: python-3.7.3-amd64.exe

https://www.python.org/downloads/windows/

ned-deily · 2019-05-19T19:56:02Z

I don't understand the need for this. AFAICT, the purpose of python-config is to provide configuration info for embedding Python (bpo-1161914 and https://docs.python.org/3/extending/embedding.html#compiling-and-linking-under-unix-like-systems). At some point, a pkg-config template was added (in bpo-3585) but it seems to duplicate the purpose of python-config and thus is also intended for use in embedding Python (although we don't seem to document it, it is familiar to users of automake). I don't know what other purposes either python-config or the pkg-config template serve other than for embedding. AFAIK, they should not be used for building C extension modules. But admittedly this area is not well-documented. In any case, I think pkg-config and python-config should return the same values for similar parameters. Anyone else have an opinion?

hroncok · 2019-05-20T12:03:03Z

As a note, waf seems to use python3-config for both embedded and extension modules. Currently, embedded is broken.

See for example https://bugzilla.redhat.com/show_bug.cgi?id=1711638

vstinner · 2019-05-22T13:58:06Z

I mark this issue as a release blocker: bpo-21536 basically broke the compilation of all applications which embed Python. IMHO this issue is the best solution to fix it.

I don't understand the need for this.

Oh. Let me explain this issue differently. There are two use cases for libpython:

Build a C extension and load it in Python: use case called "pyext" by waf
Embed Python into an application: use case called "pyembed" by waf

My bpo-21536 broke waf "pyembed" which fails to detect Python. waf "pyembed" creates a C program which calls Py_Initialize(), but the linker fails to locate Py_Initialize() symbol. To embed Python into an application, we really need to link the application to libpython: we need -lpython3.8.

In Python 3.7, "pyext" and "pyembed" use cases were covered both by default "python3.7-config --libs" and "pkg-config python3.7 --libs" configuration.

In Python 3.8, we now have to distinguish both use cases and provide *different* configuration depending if binary must be linked to libpython or not.

More info about waf breakage in my waf bug report:
https://gitlab.com/ita1024/waf/issues/2239

FYI the waf breakage was discovered by trying to build libtdb on Fedora Rawhide with Python 3.8a4, it's a dependency of Samba which embeds Python:
https://bugzilla.redhat.com/show_bug.cgi?id=1711638

doko42 · 2019-05-22T14:04:13Z

"AFAICT, the purpose of python-config is to provide configuration info for embedding Python"

If that's the intention, then at least it's not used as such. It's also used to build/configure extensions using automake/cmake based build systems. There is one tool, and two different use cases. I think Victor's suggestion is appropriate

vstinner · 2019-05-23T01:30:29Z

New changeset 0a8e572 by Victor Stinner in branch 'master':
bpo-36721: Add --embed option to python-config (GH-13500)
0a8e572

vstinner · 2019-05-23T01:33:51Z

Even if I'm not confident in my change (add --embed option), I chose to merge it anyway since at least "waf" build system is broken by my other changes (no longer link C extensions to libpython). I would like to get this change into Python 3.8 beta1 to attempt to fix most applications embedding Python.

Anyway, if something goes wrong, we still have plenty of time to decide what to do before 3.8.0 final, scheduled for 2019-10-21: https://www.python.org/dev/peps/pep-0569/

--

Since I merged my change, I reset the priority from Release Blocker to normal.

vstinner · 2019-05-24T15:54:37Z

Miro commented my PR:
#13500 (comment)

"""
This is what it gave me in Fedora, feels a bit inconsistent:

/usr/lib64/pkgconfig/python-3.8-embed.pc
/usr/lib64/pkgconfig/python-embed-3.8d.pc
/usr/lib64/pkgconfig/python3-embed.pc
"""

I installed Python in release mode:

$ ./configure --enable-shared --prefix=/opt/py38 CFLAGS="-O0" && make && make install

It gives me:

vstinner@apu$ ls -l /opt/py38/lib/pkgconfig/
-rw-r--r--. 1 vstinner vstinner 312 24 mai 17:39 python-3.8-embed.pc
-rw-r--r--. 1 vstinner vstinner 286 24 mai 17:39 python-3.8.pc
lrwxrwxrwx. 1 vstinner vstinner 19 24 mai 17:39 python3-embed.pc -> python-3.8-embed.pc
lrwxrwxrwx. 1 vstinner vstinner 13 24 mai 17:39 python3.pc -> python-3.8.pc

Debug build:

$ ./configure --enable-shared --prefix=/opt/py38dbg --with-pydebug CFLAGS="-O0" && make && make install

It gives me:

vstinner@apu$ ls -l /opt/py38dbg/lib/pkgconfig/
lrwxrwxrwx. 1 vstinner vstinner 13 24 mai 17:43 python-3.8d.pc -> python-3.8.pc
-rw-r--r--. 1 vstinner vstinner 317 24 mai 17:43 python-3.8-embed.pc
-rw-r--r--. 1 vstinner vstinner 290 24 mai 17:43 python-3.8.pc
lrwxrwxrwx. 1 vstinner vstinner 19 24 mai 17:43 python3-embed.pc -> python-3.8-embed.pc
lrwxrwxrwx. 1 vstinner vstinner 13 24 mai 17:43 python3.pc -> python-3.8.pc
lrwxrwxrwx. 1 vstinner vstinner 19 24 mai 17:43 python-embed-3.8d.pc -> python-embed-3.8.pc

Oh! "python-embed-3.8d.pc" is a broken symbolic link to "python-embed-3.8.pc" (which doesn't exist).

Ok, now I get it: I messed up in names of symbolic links :-) I wrote PR 13551 which gives me the following files for a debug build:

vstinner@apu$ ls -l /opt/py38dbg/lib/pkgconfig/*.pc
lrwxrwxrwx. 1 vstinner vstinner 19 24 mai 17:51 /opt/py38dbg/lib/pkgconfig/python-3.8d-embed.pc -> python-3.8-embed.pc
lrwxrwxrwx. 1 vstinner vstinner 13 24 mai 17:51 /opt/py38dbg/lib/pkgconfig/python-3.8d.pc -> python-3.8.pc
-rw-r--r--. 1 vstinner vstinner 317 24 mai 17:51 /opt/py38dbg/lib/pkgconfig/python-3.8-embed.pc
-rw-r--r--. 1 vstinner vstinner 290 24 mai 17:51 /opt/py38dbg/lib/pkgconfig/python-3.8.pc
lrwxrwxrwx. 1 vstinner vstinner 19 24 mai 17:51 /opt/py38dbg/lib/pkgconfig/python3-embed.pc -> python-3.8-embed.pc
lrwxrwxrwx. 1 vstinner vstinner 13 24 mai 17:51 /opt/py38dbg/lib/pkgconfig/python3.pc -> python-3.8.pc

The format is now: "python" "version" "embed suffix".

vstinner · 2019-05-24T16:02:02Z

By the way, I'm not sure that the current layout of .pc files. The name "module" give a different configuration. Is that correct?

$ grep ^Libs: /opt/py38/lib/pkgconfig/python-3.8-embed.pc
Libs: -L${libdir} -lpython3.8

$ grep ^Libs: /opt/py38dbg/lib/pkgconfig/python-3.8-embed.pc
Libs: -L${libdir} -lpython3.8d

vstinner · 2019-05-24T16:27:16Z

New changeset bc66fac by Victor Stinner in branch 'master':
bpo-36721: Fix pkg-config symbolic links on "make install" (GH-13551)
bc66fac

vstinner · 2019-05-27T22:59:54Z

The initial issue has been fixed. As far as I know, all known issues have been fixed, so I close the issue. Thanks Miro for the help in reviews and tests!

FYI waf is already fixed for the future Python 3.8 beta1! It now attempts to use --embed for its "pyembed" config ;-) Miro wrote a fix:

https://gitlab.com/ita1024/waf/merge_requests/2236
https://gitlab.com/ita1024/waf/issues/2239

vstinner added 3.8 only security fixes build The build process and cross-build labels Apr 25, 2019

vstinner added the release-blocker label May 22, 2019

vstinner changed the title ~~Add pkg-config python-3.8-embed~~ Add pkg-config python-3.8-embed and --embed to python3.8-config May 22, 2019

vstinner removed the release-blocker label May 23, 2019

vstinner closed this as completed May 27, 2019

ezio-melotti transferred this issue from another repository Apr 10, 2022

mara004 mentioned this issue Jun 23, 2022

Fix build with python3.6 NatronGitHub/Natron#812

Closed

erlend-aasland mentioned this issue Feb 21, 2023

gh-59795: distinct python-config from pkg-config python #737

Closed

eryngion mentioned this issue Dec 30, 2023

gdb: fix cross-compile with python NixOS/nixpkgs#275731

Open

13 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add pkg-config python-3.8-embed and --embed to python3.8-config #80902

Add pkg-config python-3.8-embed and --embed to python3.8-config #80902

vstinner commented Apr 25, 2019

vstinner commented Apr 25, 2019

ned-deily commented May 19, 2019

hroncok mannequin commented May 20, 2019

vstinner commented May 22, 2019

doko42 commented May 22, 2019

vstinner commented May 23, 2019

vstinner commented May 23, 2019

vstinner commented May 24, 2019

vstinner commented May 24, 2019

vstinner commented May 24, 2019

vstinner commented May 27, 2019

Add pkg-config python-3.8-embed and --embed to python3.8-config #80902

Add pkg-config python-3.8-embed and --embed to python3.8-config #80902

Comments

vstinner commented Apr 25, 2019

vstinner commented Apr 25, 2019

ned-deily commented May 19, 2019

hroncok mannequin commented May 20, 2019

vstinner commented May 22, 2019

doko42 commented May 22, 2019

vstinner commented May 23, 2019

vstinner commented May 23, 2019

vstinner commented May 24, 2019

vstinner commented May 24, 2019

vstinner commented May 24, 2019

vstinner commented May 27, 2019