When cross compiling Python for ARM many of the extension modules are not build

However when compiling for the native platform the extension modules are properly build.

Cross Compilation Steps
=======================

CONFIG_SITE=config.site CC=arm-linux-gnueabihf-gcc CXX=arm-linux-gnueabihf-g++ AR=arm-linux-gnueabihf-ar RANLIB=arm-linux-gnueabihf-ranlib READELF=arm-linux-gnueabihf-readelf ./configure --enable-shared --host=arm-linux --build=x86_64-linux-gnu --disable-ipv6 --prefix=/opt/python3


make

sudo PATH=/home/benny/workspace/projects/webshield/src/dntl_ws/sw/toolchain/gcc-linaro-4.9-2016.02-x86_64_arm-linux-gnueabihf/bin:$PATH make install


Extension Modules Built when cross compiled
===========================================

building '_ctypes_test' extension
building 'cmath' extension
building '_json' extension
building '_testcapi' extension
building '_testbuffer' extension
building '_testimportmultiple' extension
building '_testmultiphase' extension
building '_lsprof' extension
building '_opcode' extension
building 'parser' extension
building 'mmap' extension
building 'audioop' extension
building '_crypt' extension
building '_csv' extension
building 'termios' extension
building 'resource' extension
building 'nis' extension
building '_multibytecodec' extension
building '_codecs_kr' extension
building '_codecs_jp' extension
building '_codecs_cn' extension
building '_codecs_tw' extension
building '_codecs_hk' extension
building '_codecs_iso2022' extension
building '_decimal' extension
building '_multiprocessing' extension
building 'ossaudiodev' extension
building 'xxlimited' extension
building '_ctypes' extension


Compilation Steps on x86 Machine
================================

CONFIG_SITE=config.site ./configure --enable-shared --disable-ipv6 --prefix=/opt/python3

make

sudo make install

Extension Modules Built when natively compiled
===========================================
building '_struct' extension
building '_ctypes_test' extension
building 'array' extension
building 'cmath' extension
building 'math' extension
building '_datetime' extension
building '_random' extension
building '_bisect' extension
building '_heapq' extension
building '_pickle' extension
building '_json' extension
building '_testcapi' extension
building '_testbuffer' extension
building '_testimportmultiple' extension
building '_testmultiphase' extension
building '_lsprof' extension
building 'unicodedata' extension
building '_opcode' extension
building 'fcntl' extension
building 'grp' extension
building 'spwd' extension
building 'select' extension
building 'parser' extension
building 'mmap' extension
building 'syslog' extension
building 'audioop' extension
building 'readline' extension
building '_crypt' extension
building '_csv' extension
building '_posixsubprocess' extension
building '_socket' extension
building '_sha256' extension
building '_sha512' extension
building '_md5' extension
building '_sha1' extension
building 'termios' extension
building 'resource' extension
building 'nis' extension
building 'binascii' extension
building 'pyexpat' extension
building '_elementtree' extension
building '_multibytecodec' extension
building '_codecs_kr' extension
building '_codecs_jp' extension
building '_codecs_cn' extension
building '_codecs_tw' extension
building '_codecs_hk' extension
building '_codecs_iso2022' extension
building '_decimal' extension
building '_multiprocessing' extension
building 'ossaudiodev' extension
building 'xxlimited' extension
building '_ctypes' extension

I've further tried building for ARM natively on ARM machine and the extensions was build successfully.

Tool chain used for cross compilation
=======================================

Using built-in specs.
COLLECT_GCC=arm-linux-gnueabihf-gcc
COLLECT_LTO_WRAPPER=/home/benny/workspace/projects/webshield/src/dntl_ws/sw/toolchain/gcc-linaro-4.9-2016.02-x86_64_arm-linux-gnueabihf/bin/../libexec/gcc/arm-linux-gnueabihf/4.9.4/lto-wrapper
Target: arm-linux-gnueabihf
Configured with: /home/tcwg-buildslave/workspace/tcwg-make-release/label/tcwg-x86_64-ex40/target/arm-linux-gnueabihf/snapshots/gcc-linaro-4.9-2016.02/configure SHELL=/bin/bash --with-bugurl=https://bugs.linaro.org --with-mpc=/home/tcwg-buildslave/workspace/tcwg-make-release/label/tcwg-x86_64-ex40/target/arm-linux-gnueabihf/_build/builds/destdir/x86_64-unknown-linux-gnu --with-mpfr=/home/tcwg-buildslave/workspace/tcwg-make-release/label/tcwg-x86_64-ex40/target/arm-linux-gnueabihf/_build/builds/destdir/x86_64-unknown-linux-gnu --with-gmp=/home/tcwg-buildslave/workspace/tcwg-make-release/label/tcwg-x86_64-ex40/target/arm-linux-gnueabihf/_build/builds/destdir/x86_64-unknown-linux-gnu --with-gnu-as --with-gnu-ld --disable-libstdcxx-pch --disable-libmudflap --with-cloog=no --with-ppl=no --with-isl=no --disable-nls --enable-c99 --with-tune=cortex-a9 --with-arch=armv7-a --with-fpu=vfpv3-d16 --with-float=hard --with-mode=thumb --disable-multilib --enable-multiarch --with-build-sysroot=/home/tcwg-buildslave/workspace/tcwg-make-release/label/tcwg-x86_64-ex40/target/arm-linux-gnueabihf/_build/sysroots/arm-linux-gnueabihf --enable-lto --enable-linker-build-id --enable-long-long --enable-shared --with-sysroot=/home/tcwg-buildslave/workspace/tcwg-make-release/label/tcwg-x86_64-ex40/target/arm-linux-gnueabihf/_build/builds/destdir/x86_64-unknown-linux-gnu/arm-linux-gnueabihf/libc --enable-languages=c,c++,fortran,lto --enable-checking=release --disable-bootstrap --with-bugurl=https://bugs.linaro.org --build=x86_64-unknown-linux-gnu --host=x86_64-unknown-linux-gnu --target=arm-linux-gnueabihf --prefix=/home/tcwg-buildslave/workspace/tcwg-make-release/label/tcwg-x86_64-ex40/target/arm-linux-gnueabihf/_build/builds/destdir/x86_64-unknown-linux-gnu
Thread model: posix
gcc version 4.9.4 20151028 (prerelease) (Linaro GCC 4.9-2016.02) 


Host Machine
=============
Ubuntu 16.04.1 LTS 

Linux whachamacallit 4.4.0-42-generic #62-Ubuntu SMP Fri Oct 7 23:11:45 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

Please upload the output of make, this is where the error messages are printed when extension modules fail to build.
Where is set the root directory for arm headers and libraries in the configure command line ?

Thank you for the response.

I'm not too sure how to set the root directory for arm headers and libraries in the configure command line.

So I've tried the below, but still not all of the extensions modules was not build (e.g math, socket, select)

export CROSS_PATH=/home/benny/workspace/projects/webshield/src/dntl_ws/sw/toolchain/gcc-linaro-4.9-2016.02-x86_64_arm-linux-gnueabihf/bin/../arm-linux-gnueabihf/libc


CONFIG_SITE=config.site CC=arm-linux-gnueabihf-gcc CXX=arm-linux-gnueabihf-g++ AR=arm-linux-gnueabihf-ar RANLIB=arm-linux-gnueabihf-ranlib READELF=arm-linux-gnueabihf-readelf CFLAGS="-I${CROSS_PATH}/usr/include" LDFLAGS="-L${CROSS_PATH}/usr/lib" CPPFLAGS="-I${CROSS_PATH}/usr/include" ./configure --enable-shared --host=arm-linux --build=x86_64-linux-gnu --disable-ipv6 --prefix=/opt/python3

Attached please find the build log

Could you please tell me if the decision to make the extension modules like math, socket, select etc taken in the configure step or is it left to the Makefile.

Also could you please give me some hints on where to look for on how this decision is made (e.g file name / function name.)

Wierd. Modules like 'math', 'select' or '_socket' are added unconditionally in detect_modules() of setup.py. Did you disable modules in Modules/Setup.local or Modules/Setup or via the global variable `disabled_module_list` in setup.py?

Thank you for your response.

Please note that I haven't touched the Modules/Setup.local or Modules/Setup or the global variable `disabled_module_list` in setup.py

However I'm noticing that the math is commented in the Modules/Setup on both cross compiling on x86_64 for ARM as well as natively compiling on x86_64 for x86_64. But on the native build for x86_64 the math, socket, select are all build properly.

Attached please find the modules/setup file

Attaching the modules/set-up for natively building on x86_64

Your Setup file matches the one distributed with Python 3.5.2.
Did you get the Python source from https://www.python.org/ ?

Thank you for the response.

Yes I've downloaded the XZ compressed source from python.org

Based on your comment I've double checked the MD5 with

https://www.python.org/downloads/release/python-352/



benny@whachamacallit:~/Downloads$ md5sum Python-3.5.2.tar.xz
8906efbacfcdc7c3c9198aeefafd159e  Python-3.5.2.tar.xz

please let me know if there is any additional steps I need to follow when building from the source downloaded from python.org

I assume from your logs that a native (not the cross-built one) python3.5 already exists and is on your PATH.
What is the value of sys.builtin_module_names as given by this native python3.5 interpreter ?

Thank you for your response

Yes. There is a python 3.5 that is installed on the system (from apt-get)


benny@whachamacallit:~$ which python3
/usr/bin/python3


benny@whachamacallit:~$ python3
Python 3.5.2 (default, Sep 10 2016, 08:21:44) 
[GCC 5.4.0 20160609] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> 
>>> 
>>> import sys
>>> 
>>> sys.builtin_module_names
('_ast', '_bisect', '_codecs', '_collections', '_datetime', '_elementtree', '_functools', '_heapq', '_imp', '_io', '_locale', '_md5', '_operator', '_pickle', '_posixsubprocess', '_random', '_sha1', '_sha256', '_sha512', '_signal', '_socket', '_sre', '_stat', '_string', '_struct', '_symtable', '_thread', '_tracemalloc', '_warnings', '_weakref', 'array', 'atexit', 'binascii', 'builtins', 'errno', 'faulthandler', 'fcntl', 'gc', 'grp', 'itertools', 'marshal', 'math', 'posix', 'pwd', 'pyexpat', 'select', 'spwd', 'sys', 'syslog', 'time', 'unicodedata', 'xxsubtype', 'zipimport', 'zlib')
>>>

So the problem is that setup.py in build_extensions() does not build the extensions that have been already built statically into the native Ubuntu interpreter.

The solution is to build first natively from source python3.5 and set the PATH environment variable so that the newly built interpreter is fisrt on the PATH, before the one distributed by Ubuntu. And then to run the cross-build.

I think that should be the standard procedure for cross-compilation:
    * Configure the python source, for example update Modules/Setup.
    * Build python natively out of the source tree [1].
    * Set the new interpreter first on the PATH.
    * Cross-build python out of the same source tree, in another directory.

Unless someone has a better idea to fix this problem, I will propose a patch so that extension modules that are removed by build_extensions(), are not removed anymore silently.

[1] Out of the source tree:
For example, assuming the source tree is at 'src' and the current working directory is its parent, to build python in the 'build' directory without modifying any file in the 'src' directory:
    $ mkdir build
    $ cd build
    $ $(cd ../src && pwd)/configure arg1 arg2 ... && make

To eliminate the issues introduced by the native python on the Ubuntu 16.04 host, I've set up a new Ubuntu Host with all the native python packages removed.

But now when I try to build the make fails saying:

arm-linux-gnueabihf-gcc -shared -L/home/ubuntu/workspace/toolchain/gcc-linaro-4.9-2016.02-x86_64_arm-linux-gnueabihf/bin/../arm-linux-gnueabihf/libc/usr/lib  -Wl,--no-as-needed -o libpython3.so -Wl,-hlibpython3.so libpython3.5m.so
arm-linux-gnueabihf-gcc -L/home/ubuntu/workspace/toolchain/gcc-linaro-4.9-2016.02-x86_64_arm-linux-gnueabihf/bin/../arm-linux-gnueabihf/libc/usr/lib  -Xlinker -export-dynamic -o python Programs/python.o -L. -lpython3.5m -lpthread -ldl  -lpthread -lutil   -lm  
_PYTHON_PROJECT_BASE=/home/ubuntu/workspace/python/Python-3.5.2 _PYTHON_HOST_PLATFORM=linux-arm PYTHONPATH=./Lib:./Lib/plat-linux python -S -m sysconfig --generate-posix-vars ;\
if test $? -ne 0 ; then \
	echo "generate-posix-vars failed" ; \
	rm -f ./pybuilddir.txt ; \
	exit 1 ; \
fi
/bin/sh: 1: python: not found
generate-posix-vars failed
Makefile:598: recipe for target 'pybuilddir.txt' failed
make: *** [pybuilddir.txt] Error 1


Looking at the configure.log it complains of missing python as well:

python: not-found! cannot run $(srcdir)/Parser/asdl_c.py

Could you please comment if it is not possible to build python3 without first installing a native python on the build machine?

Maybe I have not set all the options to the ./configure correctly (e.g _PYTHON_HOST_PLATFORM).

@Xavier de Gaye : Thanks a lot for helping. 

Unfortunately I didn't notice your last post (Date: 2016-10-15 11:13). Kindly ignore my last quires on the native Python. 

I'll follow your steps and report back the results asap.

The cross-build uses a native python to run setup.py to build the extension modules, and to run 'python -S -m sysconfig --generate-posix-vars' and to byte compile the modules from the standard library. So you do need a native python. That is why you should first build it natively, set the PATH so that the cross-build can find it and use it, and then run the cross-build.

Thank you for the support and the detailed explanation on why the native python is necessary.

I'm now able to successfully compile all the extension modules on an Ubuntu Machine where there isn't any Python packages installed.

I'll go ahead and compile the python on the default Ubuntu 16.04 which has both python3 and python2.7. I'll update this thread with my results.

The below steps has been followed:


-----------------------------------------------------------
Building Python 3.5.2 natively for x86_64 HOST
-----------------------------------------------------------
./configure --enable-shared --disable-ipv6 --prefix=/opt/python3

make

sudo make install 


export PATH=/opt/python3/bin:$PATH

#this is a work around for the missing library error, needs to find a better solution
sudo ln -sf /opt/python3/lib/libpython3.5m.so.1.0 /usr/lib/libpython3.5m.so.1.0
sudo ldconfig

-----------------------------------------------------------
#Cros-compiling for ARM
-----------------------------------------------------------
make distclean

CONFIG_SITE=config.site CC=arm-linux-gnueabihf-gcc CXX=arm-linux-gnueabihf-g++ AR=arm-linux-gnueabihf-ar RANLIB=arm-linux-gnueabihf-ranlib READELF=arm-linux-gnueabihf-readelf CFLAGS="-I${CROSS_PATH}/usr/include" LDFLAGS="-L${CROSS_PATH}/usr/lib" CPPFLAGS="-I${CROSS_PATH}/usr/include" ./configure --enable-shared --host=arm-linux --build=x86_64-linux-gnu --disable-ipv6 --prefix=/opt/arm-linux-gnueabihf-python

make

sudo PATH=/opt/python3/bin:$PATH make install


It would be quite helpful if the extension modules are not removed silently. Unfortunately I do not have the know-how how to do this but would like to test it once someone makes available.

Once again thanks a lot for the support all through out the day. I'll revert back with the results of my testing in a while.

I've confirmed that the suggested steps works fine on Ubuntu 14.04 as well as 16.04 versions

standard procedure for cross-compilation:
    * Configure the python source, for example update Modules/Setup.
    * Build python natively out of the source tree [1].
    * Set the new interpreter first on the PATH.
    * Cross-build python out of the same source tree, in another directory.

Thanks a lot for the support.

Could you please comment if we could add these instructions to the README file under the Build Instructions. I'll make a a small patch for updating the documentation.

The attached patch fixes the problem and allows cross-building the extension modules independently of the configuration of the native interpreter that may have set some modules to be statically built. The patch also prints now the list of modules that are detected by setup.py and configured in one of the Setup files to be statically built into the interpreter.

So you don't need to build from scratch a native interpreter anymore. Can you please test the patch using the Ubuntu interpreter for the cross-build and check that the extension modules are correctly built now.

It's my turn to thank you for your reports and your tests and reactivity that are very helpful in fixing this problem :)

Thank you for the response and really appreciate the quick patch.

The "removed_modules.patch" applied cleanly on the Python sources downloaded from:

https://www.python.org/ftp/python/3.5.2/Python-3.5.2.tar.xz

and worked SUCCESSFULLY on HOST:

Ubuntu 16.04 LTS

Linux ip-172-31-16-99 4.4.0-28-generic #47-Ubuntu SMP Fri Jun 24 10:09:13 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

for CROSS COMPILING Python 3.5.2 for ARM (armhf) using

Toolchain: gcc-linaro-4.9-2016.02-x86_64_arm-linux-gnueabihf

attached please find the logs.

I'll now go ahead and try it on the 3.6 sources and update this thread

The "removed_modules.patch" applied cleanly on the Python 3.6.0b2 sources downloaded from:

https://www.python.org/ftp/python/3.6.0/Python-3.6.0b2.tar.xz

and worked SUCCESSFULLY 

The environment is the same as in my last post

Attached please find the build logs and the config logs.

Please let me know if there is any further tests that you would like me to perform. Thank you

Thanks for the reports. The next step is having the patch reviewed by one of the Python build machinery experts.
Nosying Martin, Zachary and Matthias.

Well, I am not really an expert on the setup.py stuff, but I will ask a question anyway that may help the review process: Why do you remove the code that loops over Modules/Setup? Maybe is it redundant with the other code for removing the already-built-in modules?

Looking at the repository history, the code for avoiding already-built-in modules was first added as part of revision c503fa9b265e; see the __import__() call. Later, the second chunk of code looping over Setup was added in revision 90e90c92198b, with discussion at <https://marc.info/?i=E14WL9X-0000cP-00@usw-sf-web3.sourceforge.net>.

The logic matching MODOBJS doesn’t look super robust. E.g. I suspect it will get confused if there are two Python modules that happen to use the same C filename in different subdirectories. Also, I suspect it could get confused by _math.c, which is shared by the “math” and “cmath” modules.

Perhaps I don’t know what I am talking about, but if you added Modules/Setup.config to the list of Setup files to process, would that eliminate the need to look at both MODOBJS and sys.builtin_module_names?

PS: I agree it would be good to add more documentation for cross-compiling. I tried to suggest something in an outdated patch once before; see the bottom of <https://bugs.python.org/file42143/cross-override.patch>.

Thanks for reviewing the patch Martin.

> Why do you remove the code that loops over Modules/Setup? Maybe is it redundant with the other code for removing the already-built-in modules?

Yes because this is redundant, maybe not the case when this was written 15 years ago.


> The logic matching MODOBJS doesn’t look super robust.

Agreed on both points.


> if you added Modules/Setup.config to the list of Setup files to process, would that eliminate the need to look at both MODOBJS and sys.builtin_module_names?

The processing of the Setup files done by setup.py is not robust either. It does not handle the lines of the form '<name> = <value>'. The syntax described in Setup.dist allows for:
    mod_form1 mod_form2 _mod_source.c
where both the 'mod_form1' and 'mod_form2' modules depend on the same source file and one would like to build them statically. makesetup handles that case while setup.py does not.
There is no guarantee that a change in the makesetup machinery would be systematically reflected in a change to setup.py. And indeed this is the case now - as you point it out - setup.py is currently missing the parsing of Setup.config.

Here is a new patch that uses the result of the parsing done by makesetup in setup.py.

Your second patch looks better, given my limited understanding of the scripts involved. :) I left one more suggestion though.

New patch taking into account Martin last review and some updated comments.

>> Why do you remove the code that loops over Modules/Setup? Maybe is it redundant with the other code for removing the already-built-in modules?
> Yes because this is redundant, maybe not the case when this was written 15 years ago.

* sys.builtin_module_names are the modules listed in the _PyImport_Inittab[] array that is built by Modules/makesetup from Modules/config.c.in and the '*static*' modules configured in the Setup files (those Setup files that are listed in the rule of the 'Makefile' target of the Makefile).  This list is missing the '*shared*' modules configured in the Setup files and that should not be built by setup.py.
* The setup.py code that loops over and loosely parses some of the Modules/Setup files, excludes both '*static*' and '*shared*' modules from the built modules.
* 'MODNAMES' does the same thing in the patch accurately as the list is built by makesetup.

New issue 28542 to document the cross-compilation.

Looks good to me :)

New changeset 4b2679a06ace by Xavier de Gaye in branch '3.5':
Issue #28444: Fix missing extensions modules when cross compiling.
https://hg.python.org/cpython/rev/4b2679a06ace

New changeset cddb7b2aba34 by Xavier de Gaye in branch '3.6':
Issue #28444: Merge with 3.5.
https://hg.python.org/cpython/rev/cddb7b2aba34

New changeset a87d4324e804 by Xavier de Gaye in branch 'default':
Issue #28444: Merge with 3.6.
https://hg.python.org/cpython/rev/a87d4324e804

The same issue in Python 2.7.12. Some extensions modules are not built if cross-compiling.

AFAIK many changes that have been made in Python 3 for the support of the cross compilation have not been backported to 2.7, including this one.