classification
Title: Compiler build paths and related environment variables are ignored for native builds
Type: compile error Stage:
Components: Build Versions: Python 3.9, Python 3.8, Python 3.7, Python 3.6, Python 3.5, Python 2.7
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: Alex Grund, vstinner
Priority: normal Keywords:

Created on 2019-10-21 14:21 by Alex Grund, last changed 2019-10-21 14:35 by vstinner.

Messages (3)
msg355073 - (view) Author: Alex Grund (Alex Grund) Date: 2019-10-21 14:21
In e.g. Linux users can set CPATH and LIBRARY_PATH to a list of paths considered by e.g. GCC, Clang, ... as if they were passed to `-I`, `-L`

These paths show up in the verbose compiler output too.

However in native builds (not cross-compiling) these variables/directories are ignored which leads to failure of e.g. building the BZip2 extension (among others).

Use case: On HPC modules are used, which means e.g. BZip2 is not installed into a system directory but into some other folder and CPATH, LIBRARY_PATH, LD_LIBRARY_PATH, ... are modified so it can be found by compiler and runtime. However the Python build script does not find it as it ignores those "compiler paths".

This did work ages ago, but was broken when it was decided that compiler paths (output of `gcc -E -v`) is no longer used for native builds and that function was later even renamed: https://github.com/python/cpython/commit/32f5fdd7f4213743fe2f6eedd0fe2108f3157021#diff-2eeaed663bd0d25b7e608891384b7298R514

I propose to rename the function back to its original name and use it always.
msg355075 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-10-21 14:32
> This did work ages ago, but was broken when it was decided that compiler paths (output of `gcc -E -v`) is no longer used for native builds and that function was later even renamed: https://github.com/python/cpython/commit/32f5fdd7f4213743fe2f6eedd0fe2108f3157021#diff-2eeaed663bd0d25b7e608891384b7298R514

Can you please elaborate?

setup.py of Python 2.7:

        if cross_compiling:
            self.add_gcc_paths()

setup.py of Python 3.5:

        # only change this for cross builds for 3.3, issues on Mageia
        if cross_compiling:
            self.add_gcc_paths()

In the 2.7, this code was changed in 2013 to fix bpo-17990:

"For Issue17086, 8ee6d96a1019 backported some cross-build patches to the 2.7 branch.  The changes to setup.py detect_modules differ in the backport from those in default. In particular, in default, changes to the library and include directory lists used to build extensions modules are only made conditionally when cross-compiling.  But the 2.7 backport makes these changes unconditionally."

commit 6166709eec557f52075417b19676dccc9f405fda
Author: Ned Deily <nad@acm.org>
Date:   Wed May 15 18:00:45 2013 -0700

    Issue #17990: Only modify include and library search paths when cross-compiling.
msg355076 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-10-21 14:35
> However in native builds (not cross-compiling) these variables/directories are ignored which leads to failure of e.g. building the BZip2 extension (among others).

Python does not explicitly ignore these environment variable, so I'm not sure how setup.py prevent that.

For example, the _bz2 module is built by setup.py by default in the master branch of Python.
History
Date User Action Args
2019-10-21 14:35:20vstinnersetmessages: + msg355076
2019-10-21 14:32:13vstinnersetnosy: + vstinner
messages: + msg355075
2019-10-21 14:21:13Alex Grundcreate