classification
Title: Module source files not found when cross-compiling
Type: compile error Stage:
Components: Build, Cross-Build Versions: Python 3.9, Python 3.8
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: Alex.Willmer, bill9889, doko, iliis, ned.deily, steve.dower, wscullin, xdegaye
Priority: normal Keywords:

Created on 2014-10-22 15:49 by bill9889, last changed 2020-03-20 20:22 by steve.dower.

Messages (16)
msg229828 - (view) Author: Billy (bill9889) Date: 2014-10-22 15:49
Who knows to cross-compile Python 3.4?
msg248441 - (view) Author: William Scullin (wscullin) Date: 2015-08-12 01:37
I thought this was originally a help request and was going to re-direct this. Cross-compile with 3.4.3 and later seems broken.

Procedure followed:

wget https://www.python.org/ftp/python/3.5.0/Python-3.5.0rc1.tgz
tar -xf Python-3.5.0rc1.tgz
mkdir buildpowerpc64-linux-gnu
cd buildpowerpc64-linux-gnu
../Python-3.5.0rc1/configure \
--disable-shared \
--prefix=/local/soft/python/3.5.0rc1/powerpc64-linux-gnu/gcc-4.4.7
make
make install

# now for the actual cross-compile build

cd ..
mkdir buildpowerpc64-bgq-linux
export PYTHON_FOR_BUILD=/local/soft/python/3.5.0rc1/powerpc64-linux-gnu/gcc-4.4.7/bin/python3.5
../Python-3.5.0rc1/configure \
--host=powerpc64-bgq-linux \
--build=powerpc64-linux-gnu \
--disable-ipv6 \
--disable-shared \
ac_cv_pthread_system_supported=yes \
ac_cv_file__dev_ptmx=no \
ac_cv_file__dev_ptc=no \
ac_cv_big_endian_double=yes
make

which succeeds in building a cross-compiled interpreter, then fails to build modules as setup.py gets the srcdir wrong:

[wscullin@vestalac1 buildpowerpc64-bgq-linux]$ make
running build
running build_ext
building '_struct' extension
powerpc64-bgq-linux-gcc -fPIC -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -Werror=declaration-after-statement -I../Python-3.5.0rc1/Include -I/local/soft/python/3.5.0rc1/powerpc64-linux-gnu
/gcc-4.4.7/include -I. -IInclude -I/usr/local/include -I/local/soft/python/3.5.0rc1/powerpc64-linux-gnu/gcc-4.4.7/include/python3.5m -c _struct.c -o build/temp.linux-ppc64-3.5/_struct.o
powerpc64-bgq-linux-gcc: _struct.c: No such file or directory
powerpc64-bgq-linux-gcc: no input files
msg350532 - (view) Author: Sam (iliis) Date: 2019-08-26 14:20
Has there been any update on this? I've run into this issue trying to cross-compile python for Android. I've tried 3.7.4, 3.8 and current master, both in-source and out of source builds, all ending with the modules failing to compile due to wrong paths.



...
CC='/home/samuel/android/sdk/ndk-bundle/toolchains/llvm/prebuilt/linux-x86_64/bin/aarch64-linux-android26-clang' LDSHARED='/home/samuel/android/sdk/ndk-bundle/toolchains/llvm/prebuilt/linux-x86_64/bin/aarch64-linux-android26-clang -shared -Wl,--exclude-libs,libgcc.a     -Wl,--exclude-libs,libatomic.a     -static-libstdc++     -Wl,--build-id     -Wl,--warn-shared-textrel     -Wl,--fatal-warnings -Wl,--no-undefined -Qunused-arguments -Wl,-z,noexecstack -Wl,--gc-sections      -lc++ -lm -lgcc -ldl -lc -lgcc -ldl -latomic -lm     -Wl,--exclude-libs,libgcc.a     -Wl,--exclude-libs,libatomic.a     -static-libstdc++     -Wl,--build-id     -Wl,--warn-shared-textrel     -Wl,--fatal-warnings -Wl,--no-undefined -Qunused-arguments -Wl,-z,noexecstack -Wl,--gc-sections      -lc++ -lm -lgcc -ldl -lc -lgcc -ldl -latomic -lm  ' OPT='-DNDEBUG -g -fwrapv -O3 -Wall' 	_TCLTK_INCLUDES='' _TCLTK_LIBS='' 	_PYTHON_PROJECT_BASE=/home/samuel/android/cpython _PYTHON_HOST_PLATFORM=linux-aarch64 PYTHONPATH=/home/samuel/android/cpython/build/lib.linux-aarch64-3.7:./Lib _PYTHON_SYSCONFIGDATA_NAME=_sysconfigdata_m_linux_ python3 ./setup.py  build
running build
running build_ext
INFO: Can't locate Tcl/Tk libs and/or headers
building '_struct' extension
/home/samuel/android/sdk/ndk-bundle/toolchains/llvm/prebuilt/linux-x86_64/bin/aarch64-linux-android26-clang -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall --target=aarch64-none-linux-android26 --gcc-toolchain=/home/samuel/android/sdk/ndk-bundle/toolchains/llvm/prebuilt/linux-x86_64 --sysroot=/home/samuel/android/sdk/ndk-bundle/toolchains/llvm/prebuilt/linux-x86_64/sysroot -g -DANDROID -fdata-sections -ffunction-sections -funwind-tables -fstack-protector-strong -no-canonical-prefixes -fno-addrsig -Wa,--noexecstack -Wformat -Werror=format-security -O3 -fPIE --target=aarch64-none-linux-android26 --gcc-toolchain=/home/samuel/android/sdk/ndk-bundle/toolchains/llvm/prebuilt/linux-x86_64 --sysroot=/home/samuel/android/sdk/ndk-bundle/toolchains/llvm/prebuilt/linux-x86_64/sysroot -g -DANDROID -fdata-sections -ffunction-sections -funwind-tables -fstack-protector-strong -no-canonical-prefixes -fno-addrsig -Wa,--noexecstack -Wformat -Werror=format-security -O3 -fPIE -std=c99 -Wextra -Wno-unused-result -Wno-unused-parameter -Wno-missing-field-initializers -Wstrict-prototypes -Werror=implicit-function-declaration -I./Include -I. -I/home/samuel/android/cpython_build/host/venv/include -I/home/samuel/android/cpython_build/host/install/include/python3.7m -c _struct.c -o build/temp.linux-aarch64-3.7/_struct.o
clang: error: no such file or directory: '_struct.c'
clang: error: no input files
...
Failed to build these modules:
_asyncio              _bisect               _blake2            
_bz2                  _codecs_cn            _codecs_hk         
_codecs_iso2022       _codecs_jp            _codecs_kr         
_codecs_tw            _contextvars          _crypt             
_csv                  _ctypes               _ctypes_test       
_datetime             _decimal              _heapq             
_json                 _lsprof               _md5               
_multibytecodec       _multiprocessing      _opcode            
_pickle               _posixsubprocess      _queue             
_random               _sha1                 _sha256            
_sha3                 _sha512               _socket            
_struct               _testbuffer           _testcapi          
_testimportmultiple   _testmultiphase       _uuid              
_xxtestfuzz           array                 audioop            
binascii              cmath                 fcntl              
grp                   math                  mmap               
ossaudiodev           parser                pyexpat            
resource              select                spwd               
syslog                termios               unicodedata        
xxlimited                                                      
...
msg364574 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2020-03-18 22:39
I'm now hitting this too, on nearly all the compiled modules in the stdlib. One example:

building '_codecs_jp' extension
x86_64-my-linux-gcc -m64 -march=core2 -mtune=core2 -msse3 -mfpmath=sse --sysroot=/opt/my-generic/0.0.1-dev/sysroots/core2-64-my-linux -fPIC -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -std=c99 -Wextra -Wno-unused-result -Wno-unused-parameter -Wno-missing-field-initializers -Werror=implicit-function-declaration -I../Include/internal -I. -IObjects -IInclude -IPython -I/usr/include/x86_64-linux-gnu -I/usr/local/include -I/usr/include/python3.8 -c cjkcodecs/_codecs_jp.c -o build/temp.linux-x86_64-3.8/cjkcodecs/_codecs_jp.o
x86_64-my-linux-gcc: error: cjkcodecs/_codecs_jp.c: No such file or directory
x86_64-my-linux-gcc: fatal error: no input files
compilation terminated.

I think the working directory is incorrect at some stage - note that the first two -I would make sense from Modules, but the next three do not.
msg364575 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2020-03-18 22:56
Trimmed a bit from setup.py here, but I think this is the cause:

    def build_extensions(self):
        self.srcdir = sysconfig.get_config_var('srcdir')
        self.srcdir = os.path.abspath(self.srcdir)

        [SNIP]

        # Fix up the autodetected modules, prefixing all the source files
        # with Modules/.
        moddirlist = [os.path.join(self.srcdir, 'Modules')]

In my build, I've set PYTHON_FOR_BUILD=python3.8, and the contents of moddirlist is '/usr/lib/python3.8/config-3.8-x86_64-linux-gnu/Modules'. So it's picking the wrong source directory.

I replaced the second line of the function above with:
    self.srcdir = os.path.dirname(os.path.abspath(__file__))

Which made things a little better (I can launch my build now), but all the -I/usr/* options need to be replaced as well or there are no dynamically loaded modules.
msg364577 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2020-03-18 23:06
Perhaps sysconfig needs an overhaul to be able to handle _PYTHON_PROJECT_BASE properly? I see so many values in there that are taken from the running interpreter (sys.prefix, for example) that ought to come from the cross-compilation target. Wouldn't surprise me if the host's CFLAGS are leaking through and adding those extra includes.
msg364635 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2020-03-19 20:19
So ultimately this is a major sysconfig design flaw, and it likely requires a rewrite.

The mix of information sources between _sysconfigdata (even when overriding the name and PYTHONPATH) and the Makefile, and the order in which you can rely them to be loaded (you can't) and the fact that INSTALL_SCHEMES depends on _neither_ (it only uses the running interpreter's sys module) means you can't possibly get a coherent view of the settings for anything other than the running interpreter.

Since we need something else for cross-compiling, we should either fix sysconfig so that it's possible to override the source of information reliably, or we avoid using it completely for building the CPython extension modules. The former would likely help third-party cross-compilation more.

I'm not sure who needs to be in on this, so I'll post to python-dev before going any further.
msg364658 - (view) Author: Xavier de Gaye (xdegaye) * (Python triager) Date: 2020-03-20 08:29
Steve wrote:
> In my build, I've set PYTHON_FOR_BUILD=python3.8

This will not work, PYTHON_FOR_BUILD when cross-compiling is set by configure as:

PYTHON_FOR_BUILD='_PYTHON_PROJECT_BASE=$(abs_builddir) _PYTHON_HOST_PLATFORM=$(_PYTHON_HOST_PLATFORM) PYTHONPATH=$(shell test -f pybuilddir.txt && echo $(abs_builddir)/`cat pybuilddir.txt`:)$(srcdir)/Lib _PYTHON_SYSCONFIGDATA_NAME=_sysconfigdata_$(ABIFLAGS)_$(MACHDEP)_$(MULTIARCH) '$interp

FWIW there is no problem cross-compiling Python and the standard library extension modules on Android, see for example the 'termux' Android application and its Python build.

On the other hand there are problems when cross-compiling third party extension modules due to:
* PYTHON_FOR_BUILD must be set correctly.
* distutils is still refering to sysconfig values of the build system instead of the target system.
But setup.py handles this correctly and does not need to be modified.
msg364662 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2020-03-20 09:56
Is that epic line automatically generated? It didn't work for me for some reason (was trying to launch the built Python, not the host).

Even so, I got to the point of filling in all those values manually while testing sysconfig, and while I'm 100% sure sysconfig still gave some wrong values, I'm 99% sure they weren't being fixed up by setup.py. Need to get back on my work machine to check.
msg364683 - (view) Author: Xavier de Gaye (xdegaye) * (Python triager) Date: 2020-03-20 16:55
> Is that epic line automatically generated?

When cross compiling it is generated by the configure script and configure replaces the right hand side of "PYTHON_FOR_BUILD=@PYTHON_FOR_BUILD@" of Makefile.pre.in with the generated value into Makefile.pre.

I think you may be missing the point that when this 'epic line' is being used then PYTHONPATH points to the location of the sysconfig module that has been newly built for the target platform, and so the values should be correct.
msg364685 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2020-03-20 17:01
So it's possible that my first few attempts didn't have a matched build of Python on the host - the cross-build certainly relies on mixing the installed runtime with the source stdlib, so that could have been an early cause of issues. I can only suspect that it was a factor in all the earlier reports too. Maybe we should detect this situation and fail faster?

The final problem in my case is that the multiarch paths conflict with the cross-compiling paths. The change below to configure_compiler() got me through a full build:

         if CROSS_COMPILING:
             self.add_cross_compiling_paths()
-        self.add_multiarch_paths()
+        else:
+            self.add_multiarch_paths()
         self.add_ldflags_cppflags()

I have no idea whether this is a safe change in general. Anyone able to weigh in?
msg364686 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2020-03-20 17:16
The values that are used are (apparently) correct, but most of sysconfig is not. That's very confusing, and makes it difficult to debug. I'm glad I get to maintain the *other* monster (MSBuild) :)

I would much rather sysconfig had a single variable pointing directly to whatever source of information it needs to tell everything about any Python install, whether that's _sysconfigdata or something else.

But I've also got my stuff working with only a minimal patch, so I'm not too concerned here. I'll go invest my time in improving the Windows cross-compilation support.
msg364687 - (view) Author: Matthias Klose (doko) * (Python committer) Date: 2020-03-20 17:19
I think the patch is wrong.  why would you want to differentiate between native and cross builds?  There should be a generic way to pick up those for both cross and native builds.
msg364693 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2020-03-20 17:44
> I think the patch is wrong.  why would you want to differentiate between native and cross builds?

Because the multiarch headers are incompatible with my SDK's headers, and because of the -I ordering they override mine. I tried swapping the order of calls without a difference, but excluding them entirely made it work.

I'll happily take suggestions for a better patch. I don't even know what the multiarch headers are for, but I don't seem to need them here.
msg364706 - (view) Author: Matthias Klose (doko) * (Python committer) Date: 2020-03-20 19:07
the multiarch approach allows you to install libraries and headers for many architectures in the same installation/chroot.  This can then be used to cross-build stuff.

you can check that on WSL with having Debian or Ubuntu installed. A short reference is https://wiki.debian.org/Multiarch/HOWTO
you don't need to build packages, you can build upstream sources as well, just use apt to install the required packages for the target architecture.
msg364715 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2020-03-20 20:22
> the multiarch approach allows you to install libraries and headers for many architectures in the same installation/chroot

That makes sense, but I'm not using it. So presumably I've added a configure option that I didn't need (either "--host=x86_64-my-linux" or "--build=x86_64").

I'm building against a Yocto SDK (like described at https://www.yoctoproject.org/docs/2.1/sdk-manual/sdk-manual.html#sdk-installing-the-sdk), and the --host option identifies the sysroot.

When I add just --host, I also have to add --build, and IIRC only x86_64 worked. Could that have enabled the multiarch settings? Or it is always enabled because I'm building on an Ubuntu VM?
History
Date User Action Args
2020-03-20 20:22:43steve.dowersetmessages: + msg364715
2020-03-20 19:07:48dokosetmessages: + msg364706
2020-03-20 17:44:40steve.dowersetmessages: + msg364693
2020-03-20 17:19:38dokosetmessages: + msg364687
2020-03-20 17:16:47steve.dowersetmessages: + msg364686
2020-03-20 17:01:44steve.dowersetmessages: + msg364685
2020-03-20 16:55:24xdegayesetmessages: + msg364683
2020-03-20 09:56:01steve.dowersetmessages: + msg364662
2020-03-20 08:29:15xdegayesetnosy: + xdegaye
messages: + msg364658
2020-03-19 21:18:03ned.deilysetnosy: + ned.deily
2020-03-19 20:19:19steve.dowersetmessages: + msg364635
2020-03-18 23:06:22steve.dowersetmessages: + msg364577
2020-03-18 22:56:40steve.dowersetmessages: + msg364575
title: cross-compilation of Python3.4 -> Module source files not found when cross-compiling
2020-03-18 22:39:19steve.dowersetnosy: + steve.dower

messages: + msg364574
versions: + Python 3.8, Python 3.9, - Python 3.4, Python 3.5
2019-08-26 14:20:33iliissetnosy: + iliis
messages: + msg350532
2016-03-02 00:25:42Alex.Willmersetnosy: + Alex.Willmer
2015-09-02 19:51:17wscullinsettype: resource usage -> compile error
2015-08-12 01:37:05wscullinsetmessages: + msg248441
components: + Build, Cross-Build
versions: + Python 3.5
2015-02-11 17:29:33wscullinsetnosy: + wscullin
2014-10-22 15:53:00vstinnersetnosy: + doko
2014-10-22 15:49:06bill9889create