Missing extensions modules when cross compiling python 3.5.2 for arm on Linux #72630
Comments
|
When cross compiling Python for ARM many of the extension modules are not build However when compiling for the native platform the extension modules are properly build. Cross Compilation Steps CONFIG_SITE=config.site CC=arm-linux-gnueabihf-gcc CXX=arm-linux-gnueabihf-g++ AR=arm-linux-gnueabihf-ar RANLIB=arm-linux-gnueabihf-ranlib READELF=arm-linux-gnueabihf-readelf ./configure --enable-shared --host=arm-linux --build=x86_64-linux-gnu --disable-ipv6 --prefix=/opt/python3make sudo PATH=/home/benny/workspace/projects/webshield/src/dntl_ws/sw/toolchain/gcc-linaro-4.9-2016.02-x86_64_arm-linux-gnueabihf/bin:$PATH make install Extension Modules Built when cross compiled building '_ctypes_test' extension Compilation Steps on x86 Machine CONFIG_SITE=config.site ./configure --enable-shared --disable-ipv6 --prefix=/opt/python3make sudo make install Extension Modules Built when natively compiled I've further tried building for ARM natively on ARM machine and the extensions was build successfully. Tool chain used for cross compilation Using built-in specs. Host Machine Linux whachamacallit 4.4.0-42-generic #62-Ubuntu SMP Fri Oct 7 23:11:45 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
bennykj
mannequin
added
extension-modules
|
Please upload the output of make, this is where the error messages are printed when extension modules fail to build. |
|
Thank you for the response. I'm not too sure how to set the root directory for arm headers and libraries in the configure command line. So I've tried the below, but still not all of the extensions modules was not build (e.g math, socket, select) export CROSS_PATH=/home/benny/workspace/projects/webshield/src/dntl_ws/sw/toolchain/gcc-linaro-4.9-2016.02-x86_64_arm-linux-gnueabihf/bin/../arm-linux-gnueabihf/libc CONFIG_SITE=config.site CC=arm-linux-gnueabihf-gcc CXX=arm-linux-gnueabihf-g++ AR=arm-linux-gnueabihf-ar RANLIB=arm-linux-gnueabihf-ranlib READELF=arm-linux-gnueabihf-readelf CFLAGS="-I${CROSS_PATH}/usr/include" LDFLAGS="-L${CROSS_PATH}/usr/lib" CPPFLAGS="-I${CROSS_PATH}/usr/include" ./configure --enable-shared --host=arm-linux --build=x86_64-linux-gnu --disable-ipv6 --prefix=/opt/python3Attached please find the build log |
|
Could you please tell me if the decision to make the extension modules like math, socket, select etc taken in the configure step or is it left to the Makefile. Also could you please give me some hints on where to look for on how this decision is made (e.g file name / function name.) |
|
Wierd. Modules like 'math', 'select' or '_socket' are added unconditionally in detect_modules() of setup.py. Did you disable modules in Modules/Setup.local or Modules/Setup or via the global variable |
|
Thank you for your response. Please note that I haven't touched the Modules/Setup.local or Modules/Setup or the global variable However I'm noticing that the math is commented in the Modules/Setup on both cross compiling on x86_64 for ARM as well as natively compiling on x86_64 for x86_64. But on the native build for x86_64 the math, socket, select are all build properly. Attached please find the modules/setup file |
|
Attaching the modules/set-up for natively building on x86_64 |
|
Your Setup file matches the one distributed with Python 3.5.2. |
|
Thank you for the response. Yes I've downloaded the XZ compressed source from python.org Based on your comment I've double checked the MD5 with https://www.python.org/downloads/release/python-352/ benny@whachamacallit:~/Downloads$ md5sum Python-3.5.2.tar.xz |
|
please let me know if there is any additional steps I need to follow when building from the source downloaded from python.org |
|
I assume from your logs that a native (not the cross-built one) python3.5 already exists and is on your PATH. |
|
Thank you for your response Yes. There is a python 3.5 that is installed on the system (from apt-get) benny@whachamacallit:~$ which python3 benny@whachamacallit:~$ python3
Python 3.5.2 (default, Sep 10 2016, 08:21:44)
[GCC 5.4.0 20160609] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>>
>>>
>>> import sys
>>>
>>> sys.builtin_module_names
('_ast', '_bisect', '_codecs', '_collections', '_datetime', '_elementtree', '_functools', '_heapq', '_imp', '_io', '_locale', '_md5', '_operator', '_pickle', '_posixsubprocess', '_random', '_sha1', '_sha256', '_sha512', '_signal', '_socket', '_sre', '_stat', '_string', '_struct', '_symtable', '_thread', '_tracemalloc', '_warnings', '_weakref', 'array', 'atexit', 'binascii', 'builtins', 'errno', 'faulthandler', 'fcntl', 'gc', 'grp', 'itertools', 'marshal', 'math', 'posix', 'pwd', 'pyexpat', 'select', 'spwd', 'sys', 'syslog', 'time', 'unicodedata', 'xxsubtype', 'zipimport', 'zlib')
>>> |
|
So the problem is that setup.py in build_extensions() does not build the extensions that have been already built statically into the native Ubuntu interpreter. The solution is to build first natively from source python3.5 and set the PATH environment variable so that the newly built interpreter is fisrt on the PATH, before the one distributed by Ubuntu. And then to run the cross-build. I think that should be the standard procedure for cross-compilation: Unless someone has a better idea to fix this problem, I will propose a patch so that extension modules that are removed by build_extensions(), are not removed anymore silently. [1] Out of the source tree: |
|
To eliminate the issues introduced by the native python on the Ubuntu 16.04 host, I've set up a new Ubuntu Host with all the native python packages removed. But now when I try to build the make fails saying: arm-linux-gnueabihf-gcc -shared -L/home/ubuntu/workspace/toolchain/gcc-linaro-4.9-2016.02-x86_64_arm-linux-gnueabihf/bin/../arm-linux-gnueabihf/libc/usr/lib -Wl,--no-as-needed -o libpython3.so -Wl,-hlibpython3.so libpython3.5m.so Looking at the configure.log it complains of missing python as well: python: not-found! cannot run $(srcdir)/Parser/asdl_c.py Could you please comment if it is not possible to build python3 without first installing a native python on the build machine? Maybe I have not set all the options to the ./configure correctly (e.g _PYTHON_HOST_PLATFORM). |
|
@xavier de Gaye : Thanks a lot for helping. Unfortunately I didn't notice your last post (Date: 2016-10-15 11:13). Kindly ignore my last quires on the native Python. I'll follow your steps and report back the results asap. |
|
The cross-build uses a native python to run setup.py to build the extension modules, and to run 'python -S -m sysconfig --generate-posix-vars' and to byte compile the modules from the standard library. So you do need a native python. That is why you should first build it natively, set the PATH so that the cross-build can find it and use it, and then run the cross-build. |
|
Thank you for the support and the detailed explanation on why the native python is necessary. I'm now able to successfully compile all the extension modules on an Ubuntu Machine where there isn't any Python packages installed. I'll go ahead and compile the python on the default Ubuntu 16.04 which has both python3 and python2.7. I'll update this thread with my results. The below steps has been followed: ----------------------------------------------------------- make sudo make install export PATH=/opt/python3/bin:$PATH #this is a work around for the missing library error, needs to find a better solution ----------------------------------------------------------- CONFIG_SITE=config.site CC=arm-linux-gnueabihf-gcc CXX=arm-linux-gnueabihf-g++ AR=arm-linux-gnueabihf-ar RANLIB=arm-linux-gnueabihf-ranlib READELF=arm-linux-gnueabihf-readelf CFLAGS="-I${CROSS_PATH}/usr/include" LDFLAGS="-L${CROSS_PATH}/usr/lib" CPPFLAGS="-I${CROSS_PATH}/usr/include" ./configure --enable-shared --host=arm-linux --build=x86_64-linux-gnu --disable-ipv6 --prefix=/opt/arm-linux-gnueabihf-pythonmake sudo PATH=/opt/python3/bin:$PATH make install It would be quite helpful if the extension modules are not removed silently. Unfortunately I do not have the know-how how to do this but would like to test it once someone makes available. Once again thanks a lot for the support all through out the day. I'll revert back with the results of my testing in a while. |
|
I've confirmed that the suggested steps works fine on Ubuntu 14.04 as well as 16.04 versions standard procedure for cross-compilation: Thanks a lot for the support. Could you please comment if we could add these instructions to the README file under the Build Instructions. I'll make a a small patch for updating the documentation. |
|
The attached patch fixes the problem and allows cross-building the extension modules independently of the configuration of the native interpreter that may have set some modules to be statically built. The patch also prints now the list of modules that are detected by setup.py and configured in one of the Setup files to be statically built into the interpreter. So you don't need to build from scratch a native interpreter anymore. Can you please test the patch using the Ubuntu interpreter for the cross-build and check that the extension modules are correctly built now. It's my turn to thank you for your reports and your tests and reactivity that are very helpful in fixing this problem :) |
|
Thank you for the response and really appreciate the quick patch. The "removed_modules.patch" applied cleanly on the Python sources downloaded from: https://www.python.org/ftp/python/3.5.2/Python-3.5.2.tar.xz and worked SUCCESSFULLY on HOST: Ubuntu 16.04 LTS Linux ip-172-31-16-99 4.4.0-28-generic #47-Ubuntu SMP Fri Jun 24 10:09:13 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux for CROSS COMPILING Python 3.5.2 for ARM (armhf) using Toolchain: gcc-linaro-4.9-2016.02-x86_64_arm-linux-gnueabihf attached please find the logs. I'll now go ahead and try it on the 3.6 sources and update this thread |
|
The "removed_modules.patch" applied cleanly on the Python 3.6.0b2 sources downloaded from: https://www.python.org/ftp/python/3.6.0/Python-3.6.0b2.tar.xz and worked SUCCESSFULLY The environment is the same as in my last post Attached please find the build logs and the config logs. Please let me know if there is any further tests that you would like me to perform. Thank you |
|
Thanks for the reports. The next step is having the patch reviewed by one of the Python build machinery experts. |
|
Well, I am not really an expert on the setup.py stuff, but I will ask a question anyway that may help the review process: Why do you remove the code that loops over Modules/Setup? Maybe is it redundant with the other code for removing the already-built-in modules? Looking at the repository history, the code for avoiding already-built-in modules was first added as part of revision c503fa9b265e; see the __import__() call. Later, the second chunk of code looping over Setup was added in revision 90e90c92198b, with discussion at <https://marc.info/?i=E14WL9X-0000cP-00@usw-sf-web3.sourceforge.net\>. The logic matching MODOBJS doesn’t look super robust. E.g. I suspect it will get confused if there are two Python modules that happen to use the same C filename in different subdirectories. Also, I suspect it could get confused by _math.c, which is shared by the “math” and “cmath” modules. Perhaps I don’t know what I am talking about, but if you added Modules/Setup.config to the list of Setup files to process, would that eliminate the need to look at both MODOBJS and sys.builtin_module_names? |
|
PS: I agree it would be good to add more documentation for cross-compiling. I tried to suggest something in an outdated patch once before; see the bottom of <https://bugs.python.org/file42143/cross-override.patch\>. |
|
Thanks for reviewing the patch Martin.
Yes because this is redundant, maybe not the case when this was written 15 years ago.
Agreed on both points.
The processing of the Setup files done by setup.py is not robust either. It does not handle the lines of the form '<name> = <value>'. The syntax described in Setup.dist allows for: Here is a new patch that uses the result of the parsing done by makesetup in setup.py. |
|
Your second patch looks better, given my limited understanding of the scripts involved. :) I left one more suggestion though. |
|
New patch taking into account Martin last review and some updated comments.
New bpo-28542 to document the cross-compilation. |
|
Looks good to me :) |
|
New changeset 4b2679a06ace by Xavier de Gaye in branch '3.5': New changeset cddb7b2aba34 by Xavier de Gaye in branch '3.6': New changeset a87d4324e804 by Xavier de Gaye in branch 'default': |
|
The same issue in Python 2.7.12. Some extensions modules are not built if cross-compiling. |
|
AFAIK many changes that have been made in Python 3 for the support of the cross compilation have not been backported to 2.7, including this one. |
Misc/NEWSso that it is managed by towncrier #552Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
bugs.python.org fields:
The text was updated successfully, but these errors were encountered: