classification
Title: sysconfig --generate-posix-vars creates wrong file when cross compiling
Type: behavior Stage:
Components: Build, Cross-Build Versions: Python 3.11
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: Alex.Willmer, christian.heimes, ethan smith
Priority: normal Keywords:

Created on 2021-11-27 12:05 by christian.heimes, last changed 2021-11-27 19:03 by christian.heimes.

Messages (2)
msg407141 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2021-11-27 12:05
"sysconfig --generate-posix-vars" creates pybuilddir.txt and a platform-specific sysconfig data file like build/lib.linux-x86_64-3.11/_sysconfigdata__linux_x86_64-linux-gnu.py

When creating a cross-compile build of Python, sysconfig mixes data from the cross compile build and the host build. It creates a pybuilddir.txt and build/lib directory with correct values (e.g. wasm32-unknown-emscripten) but sysconfigdata file with name values from the host Python PYTHON_FOR_BUILD (e.g x86_64-unknown-linux-gnu). 

$ cat pybuilddir.txt 
build/lib.wasm32-unknown-emscripten-3.11

$ ls build/lib.wasm32-unknown-emscripten-3.11/_sysconfigdata*                           
build/lib.wasm32-unknown-emscripten-3.11/_sysconfigdata__linux_x86_64-linux-gnu.py

$ grep CC build/lib.wasm32-unknown-emscripten-3.11/_sysconfigdata__linux_x86_64-linux-gnu.py 
 'CC': 'gcc',

$ grep ^CC Makefile
CC=             emcc
msg407163 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2021-11-27 19:03
I just realized that PYTHON_FOR_BUILD is far more complicated. Our WASM build system sets the env var to "$(pwd)/cpython/builddir/build/python". The configure script expects us to set several additional env vars.

The logic goes like this for cross compiling:

if PYTHON_FOR_BUILD is empty:
    find Python interpreter with same version as PACKAGE_VERSION
    set PYTHON_FOR_BUILD='_PYTHON_PROJECT_BASE=$(abs_builddir) _PYTHON_HOST_PLATFORM=$(_PYTHON_HOST_PLATFORM) PYTHONPATH=$(shell test -f pybuilddir.txt && echo $(abs_builddir)/`cat pybuilddir.txt`:)$(srcdir)/Lib _PYTHON_SYSCONFIGDATA_NAME=_sysconfigdata_$(ABIFLAGS)_$(MACHDEP)_$(MULTIARCH) '$interp
else:
    use user-supplied PYTHON_FOR_BUILD

Since we don't set any of the variables _PYTHON_PROJECT_BASE, _PYTHON_HOST_PLATFORM, and _PYTHON_SYSCONFIGDATA_NAME, we get wrong sysconfig data file.
History
Date User Action Args
2021-11-27 19:03:41christian.heimessetmessages: + msg407163
2021-11-27 13:14:11ethan smithsetnosy: + ethan smith
2021-11-27 12:06:24christian.heimeslinkissue45881 dependencies
2021-11-27 12:05:59christian.heimescreate