classification
Title: Use pkg-config to find dependencies
Type: enhancement Stage: needs patch
Components: Build, Extension Modules Versions: Python 3.7
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: Santiago Castro, christian.heimes, koobs, ned.deily, yan12125, zach.ware
Priority: normal Keywords:

Created on 2016-09-19 19:34 by Santiago Castro, last changed 2019-12-10 08:13 by xdegaye.

Messages (15)
msg276983 - (view) Author: Santiago Castro (Santiago Castro) Date: 2016-09-19 19:34
When installing Python (for example, version 3.5.2), if SQLite library and headers are not installed in a default location (like $HOME/.local/include instead of /usr/include), it should take it from there, and not fail to find it. This behavior does work with bz2 library and OpenSSL, but not with SQLite.
msg277001 - (view) Author: Chih-Hsuan Yen (yan12125) * Date: 2016-09-20 03:55
Version 3.5.2 works for me:

sqlite: found /home/yen/usr/include/sqlite3.h
/home/yen/usr/include/sqlite3.h: version 3.14.2

How did you compile CPython? Could you paste commands you use?
msg277063 - (view) Author: Santiago Castro (Santiago Castro) Date: 2016-09-20 20:44
I tried with pyenv (https://github.com/yyuu/pyenv): pyenv install 3.5.2. Maybe the error is from their side, but basically it downloads Python and compiles it: https://github.com/yyuu/pyenv/blob/master/plugins/python-build/install.sh#L24
msg277314 - (view) Author: Chih-Hsuan Yen (yan12125) * Date: 2016-09-24 08:26
Could you try this:

CPPFLAGS=-I/home/<username>/local/include/ LDFLAGS=-L/home/<username>/local/lib bash -x /usr/bin/pyenv install 3.5.2
msg277333 - (view) Author: Santiago Castro (Santiago Castro) Date: 2016-09-24 20:21
Okay, that did work. But shouldn't it call pkg-config, so I don't need to set the flags manually?

I mean, I'm running this in my user's home, in a computer which I don't have root access, and I used Linuxbrew (https://github.com/Linuxbrew/brew) for this, and installed libbz2 headers, openssl headers, pkg-config, sqlite3 headers and all worked but finding the sqlite3 headers.

Just fyi, when I run pkg-config --cflags sqlite3 this is what I get:

-I/home/sacastro/.linuxbrew/Cellar/sqlite/3.14.1/include

which is ok.
msg277347 - (view) Author: Chih-Hsuan Yen (yan12125) * Date: 2016-09-25 06:48
Hmmm, currently only _ctypes uses pkg-config to detect libffi's header path, as it's not easy to determine without pkg-config. Is there a magic that enables openssl and bz2 outside standard paths :)

Anyway, using pkg-config is not a bad idea. I'd like to hear from some core developers. Zach, is it a good idea to introduce pkg-config for dependencies?
msg277400 - (view) Author: Santiago Castro (Santiago Castro) Date: 2016-09-25 23:43
Okay, I checked out again and bz2 and openssl were in standard paths in fact, my bad. But I think python should also take into account pkg-config. I left a Dockerfile with how I think it should work with pyenv and Linuxbrew: https://github.com/bryant1410/docker-pyenv-linuxbrew
msg277403 - (view) Author: Zachary Ware (zach.ware) * (Python committer) Date: 2016-09-26 04:54
I don't have any philosophical opposition to using pkg-config, but it would be nice to avoid making configure/Makefile/setup.py any more complex than they already are.  If you can somehow simplify those files (without breaking anything) by using pkg-config, I'd be all for it :)

Resetting the version to 3.7, this feels like a pretty big change.  If it turns out well, we can consider backporting it later.
msg277405 - (view) Author: Chih-Hsuan Yen (yan12125) * Date: 2016-09-26 06:24
Thanks. I'll give it a try.
msg277413 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2016-09-26 12:06
Simple implementation idea:

* add a pkg_config option to Extension()
* run subprocess.call(["pkg-config", self.pkg_config, "--exists"])
* if return value is 0, extend self.extra_compile_args with ["pkg-config", self.pkg_config, "--cflags"] and self.extra_link_args with ["pkg-config", self.pkg_config, "--libs"]
* otherwise print a warning

We could split and parse the output to handle -I, -L and -l in a more elegant way.
msg277424 - (view) Author: Zachary Ware (zach.ware) * (Python committer) Date: 2016-09-26 14:45
Note the bootstrap issue with that idea though; you'll need to make sure _posixsubprocess is built before importing subprocess.
msg277438 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2016-09-26 17:51
Any solution using pkg-config would need to take into account that pkg-config may not be available (by default) on platforms we support; for example, AFAIK, Apple does not ship pkg-config in the base OS or any of its developer tools.  And the solution would need to be careful to not break the ability to use the subset of Distutils the top-level setup.py needs to bootstrap build the standard library.  Also keep in mind that a lot of this proposed functionality is (or should be) available today by using Modules/Setup.local.
msg277488 - (view) Author: Chih-Hsuan Yen (yan12125) * Date: 2016-09-27 05:09
Yes pkg-config is not ubiquitous. My idea is using it as a fallback, so that non-standard paths can be picked up easily.

> Note the bootstrap issue with that idea though; you'll need to make sure _posixsubprocess is built before importing subprocess.

distutils.spawn.spawn or os.system can be used. setup.py uses the latter.
msg277490 - (view) Author: Kubilay Kocak (koobs) (Python triager) Date: 2016-09-27 05:25
This (adding support for pkg-config) should be done in tandem with adding --with-foo-{include,library} arguments for each *external* dependency, which can be used for: libffi, readline, libintl, openssl, sqlite, db*, among others.

Values obtained from pkg-config should then use the same variables, but only if they are/have not been specified at the command line (./configure).

Doing this and a few other best-practice, standard autoconf things will enable us to drastically simplify and remove hacks in the Python configure.ac, Makefile and setup.py's long term.
msg277497 - (view) Author: Chih-Hsuan Yen (yan12125) * Date: 2016-09-27 07:02
Thanks Kubilay I got it.

Note to myself: gdb uses lots --with-foo-{include,lib}.
History
Date User Action Args
2019-12-10 08:13:29xdegayesetnosy: - xdegaye
2016-09-30 16:19:40xdegayesetnosy: + xdegaye
2016-09-27 07:02:48yan12125setmessages: + msg277497
2016-09-27 05:25:51koobssetnosy: + koobs
messages: + msg277490
2016-09-27 05:09:04yan12125setmessages: + msg277488
2016-09-26 17:51:28ned.deilysetnosy: + ned.deily
messages: + msg277438
2016-09-26 14:45:12zach.waresetmessages: + msg277424
2016-09-26 12:06:41christian.heimessetnosy: + christian.heimes
messages: + msg277413
2016-09-26 06:27:35yan12125settype: behavior -> enhancement
2016-09-26 06:24:50yan12125setmessages: + msg277405
2016-09-26 04:54:34zach.waresetversions: + Python 3.7, - Python 2.7, Python 3.3, Python 3.4, Python 3.5
title: SQLite headers are not searched in custom locations -> Use pkg-config to find dependencies
messages: + msg277403

components: + Build
stage: needs patch
2016-09-25 23:43:53Santiago Castrosetmessages: + msg277400
2016-09-25 06:48:50yan12125setnosy: + zach.ware
messages: + msg277347
2016-09-24 20:21:01Santiago Castrosetmessages: + msg277333
2016-09-24 08:26:39yan12125setmessages: + msg277314
2016-09-20 20:44:01Santiago Castrosetmessages: + msg277063
2016-09-20 03:55:33yan12125setnosy: + yan12125
messages: + msg277001
2016-09-19 20:05:08Santiago Castrosettitle: SQLite headers are not -> SQLite headers are not searched in custom locations
2016-09-19 19:34:50Santiago Castrocreate