Title: Implement equivalent to `pip.locations.distutils_scheme` in sysconfig
Type: enhancement Stage:
Components: Library (Lib) Versions: Python 3.10
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: dstufft, eric.araujo, ned.deily, sylvain.corlay, tdsmith
Priority: normal Keywords:

Created on 2016-05-04 16:17 by sylvain.corlay, last changed 2020-10-22 21:18 by eric.araujo.

Messages (7)
msg264836 - (view) Author: Sylvain Corlay (sylvain.corlay) * Date: 2016-05-04 16:17
When installing a python package that has `headers`, these headers are usually installed under the main python include directory, which can be retrieved with `sysconfig.get_path('include')` or directly referred to as 'include' when setting the include directories of an extension module. 

However, on some systems like OS X, headers for extension modules are not located in under the python include directory (/usr/local/Cellar/pythonX/X.Y.Z/Frameworks/Python.framework/Versions/X.Y/include/pythonX.Y) but in `/usr/local/include/pythonX.Y`.

Is there a generic way to find the location where  headers are installed in a  python install?

pip.locations implements `distutils_scheme` which seems to be returning the right thing, but it seems to be a bit overkill to require pip. On the other side, no path returned by sysconfig corresponds to `/usr/local/include/pythonX.Y`
msg266052 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2016-05-22 05:48
"When installing a python package that has `headers`, these headers are usually installed under the main python include directory".  Can you give some specific examples of packages and platforms where that is the case?  Just taking a quick look at some packages I happen to have installed, like lxml and cffi using a vanilla pip with a vanilla Unix-y Python 3.5.1, their header files get installed in the site-packages directory by default (using pip), not the main Python include directory.  And various distributors of Python have their own preferences where to install things, for example, the locations you cite are those used by Homebrew on OS X and differ from other OS X Pythons.  I'm not sure that there is a general solution as things stand now; I think there is probably a lot of variability among packages (e.g. their and Python distributions.
msg266059 - (view) Author: Sylvain Corlay (sylvain.corlay) * Date: 2016-05-22 07:50
Ned, that is because these packages (lxml, cffi) have header files as `package_data`, instead of `headers`. This is why they are being copied into site-packages. People willing to include them must either rely on knowledge of their location, or a python functions returning if (like numpy's get_include)

Distutils provides a standard way of distributing headers. These are passed to the setup function via the `headers` keyword argument, which triggers the `install_headers` command.
msg266224 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2016-05-24 04:32
Sylvain, OK.  I suggest you bring the matter up on the distutils-sig mailing list (if you haven't already) because that's where discussions and decisions are made about features for Python packaging including distutils.  Otherwise, the issue is likely to languish here.  Good luck!
msg267888 - (view) Author: Tim Smith (tdsmith) * Date: 2016-06-08 20:41
As a Homebrew maintainer I'm happy to consider improving Homebrew's configuration if someone can point me to an extant package that uses this mechanism.
msg267956 - (view) Author: Sylvain Corlay (sylvain.corlay) * Date: 2016-06-09 04:17
We use it in the pybind11 project. I am not sure that homebrew does anything wrong.

It is just that it is the only case I am aware of where the `install_headers` command does not install headers in a subdirectory of `sysconfig.get_path('include')`. On the other hand, pip.locations.distutil_scheme provides the location used by install_headers.
msg379350 - (view) Author: √Čric Araujo (eric.araujo) * (Python committer) Date: 2020-10-22 21:18
If added, this should be in sysconfig, not distutils.
Date User Action Args
2020-10-22 21:18:37eric.araujosettitle: Implement equivalent to `pip.locations.distutils_scheme` in distutils -> Implement equivalent to `pip.locations.distutils_scheme` in sysconfig
messages: + msg379350
components: + Library (Lib), - Distutils
versions: + Python 3.10, - Python 3.6
2016-06-09 04:17:51sylvain.corlaysetmessages: + msg267956
2016-06-08 20:41:13tdsmithsetnosy: + tdsmith
messages: + msg267888
2016-05-24 04:32:39ned.deilysetmessages: + msg266224
2016-05-22 07:50:25sylvain.corlaysetmessages: + msg266059
2016-05-22 05:48:05ned.deilysetnosy: + ned.deily
messages: + msg266052
2016-05-04 16:48:06SilentGhostsetnosy: + eric.araujo, dstufft
components: + Distutils
2016-05-04 16:17:44sylvain.corlaycreate