classification
Title: Cross-compile fails trying to execute foreign pgen on build host
Type: compile error Stage:
Components: Cross-Build Versions: Python 3.3, Python 2.7
process
Status: closed Resolution: out of date
Dependencies: Superseder: When cross-compiling, don’t try to execute binaries
View: 22625
Assigned To: Nosy List: Marc Duponcheel, Trevor.Bowen, gregory.p.smith, martin.panter, pitrou, r.david.murray, xdegaye
Priority: normal Keywords: patch

Created on 2013-10-01 23:41 by Trevor.Bowen, last changed 2016-07-18 00:03 by martin.panter. This issue is now closed.

Files
File name Uploaded Description Edit
cross-compile.log Trevor.Bowen, 2013-10-01 23:41 Build log on an x86-64 host for Python-2.7.5 for a PowerPC-linux target
Python-2.7.5-xcompile.patch Trevor.Bowen, 2013-10-07 23:01 Applied *during* build process.
cross-compile.sh Trevor.Bowen, 2013-10-07 23:13 top-level cross-compile script
Python-2.7.5-xcompile_improvements.patch Trevor.Bowen, 2013-10-07 23:33 Failed attempt to completely integrate 2-stage build process into Python's source.
Messages (17)
msg198813 - (view) Author: Trevor Bowen (Trevor.Bowen) Date: 2013-10-01 23:41
FWIW, I'm using a Freescale cross-compile tool-chain on a Linux x86-64 build host, although I have duplicated the cross-compile error on an x86 Ubunutu 10.04 build host.

Steps to reproduce:

$ wget http://www.python.org/ftp/python/2.7.5/Python-2.7.5.tar.bz2
$ tar -jxf Python-2.7.5.tar.bz2
$ cd Python-2.7.5
$ export PATH="/opt/freescale/usr/local/gcc-4.3.74-eglibc-2.8.74-dp-2/powerpc-none-linux-gnuspe/bin:${PATH}"
# For info only:
$ ls -l /opt/freescale/usr/local/gcc-4.3.74-eglibc-2.8.74-dp-2/powerpc-none-linux-gnuspe/bin 
total 10472
-rwxrwxrwx 1 root root  590485 Mar  5  2012 powerpc-none-linux-gnuspe-addr2line
-rwxrwxrwx 1 root root  614647 Mar  5  2012 powerpc-none-linux-gnuspe-ar
-rwxrwxrwx 1 root root  897161 Mar  5  2012 powerpc-none-linux-gnuspe-as
-rwxrwxrwx 1 root root  235382 Mar  5  2012 powerpc-none-linux-gnuspe-c++
-rwxrwxrwx 1 root root  589227 Mar  5  2012 powerpc-none-linux-gnuspe-c++filt
-rwxrwxrwx 1 root root  234277 Mar  5  2012 powerpc-none-linux-gnuspe-cpp
-rwxrwxrwx 1 root root    8503 Mar  5  2012 powerpc-none-linux-gnuspe-embedspu
-rwxrwxrwx 1 root root  235382 Mar  5  2012 powerpc-none-linux-gnuspe-g++
-rwxrwxrwx 1 root root  233126 Mar  5  2012 powerpc-none-linux-gnuspe-gcc
-rwxrwxrwx 1 root root  233126 Mar  5  2012 powerpc-none-linux-gnuspe-gcc-4.3.2
-rwxrwxrwx 1 root root   16512 Mar  5  2012 powerpc-none-linux-gnuspe-gccbug
-rwxrwxrwx 1 root root   28017 Mar  5  2012 powerpc-none-linux-gnuspe-gcov
-rwxrwxrwx 1 root root  655127 Mar  5  2012 powerpc-none-linux-gnuspe-gprof
-rwxrwxrwx 1 root root 1036372 Mar  5  2012 powerpc-none-linux-gnuspe-ld
-rwxrwxrwx 1 root root  603678 Mar  5  2012 powerpc-none-linux-gnuspe-nm
-rwxrwxrwx 1 root root  750617 Mar  5  2012 powerpc-none-linux-gnuspe-objcopy
-rwxrwxrwx 1 root root  895336 Mar  5  2012 powerpc-none-linux-gnuspe-objdump
-rwxrwxrwx 1 root root  614647 Mar  5  2012 powerpc-none-linux-gnuspe-ranlib
-rwxrwxrwx 1 root root  264063 Mar  5  2012 powerpc-none-linux-gnuspe-readelf
-rwxrwxrwx 1 root root  593901 Mar  5  2012 powerpc-none-linux-gnuspe-size
-rwxrwxrwx 1 root root  591853 Mar  5  2012 powerpc-none-linux-gnuspe-strings
-rwxrwxrwx 1 root root  750617 Mar  5  2012 powerpc-none-linux-gnuspe-strip
$ ./configure --host=powerpc-none-linux-gnuspe \
    --build=i586-linux-gnu --prefix=/          \
    --disable-ipv6 ac_cv_file__dev_ptmx=no     \
    ac_cv_file__dev_ptc=no
$ make

Make fails because the foreign (powerpc, in this case) pgen binary is used to build the $(GRAMMAR_H).  Instead, a native version of the python interpreter (preferrable) and pgen (essential) need to be built first using the host's toolchain.  Then, these tools can be used to build the full foreign suite.

FWIW, This 2 step process is well documented here:

http://randomsplat.com/id5-cross-compiling-python-for-embedded-linux.html

However, the cross-compile fixes from Issue17086 may have helped this process.  (It's hard for me to determine how to take advantage of these updates.)  Certainly, they changed the above process dramatically.

Incidentally, Python 3.3.2 exhibits almost the exact same build error, if not the same.

Build log on an x86-64 host for Python-2.7.5 for a PowerPC-linux target is attached.

Tail of log:

make Parser/pgen
make[1]: Entering directory `/home/user/Python-2.7.5'
powerpc-none-linux-gnuspe-gcc -c -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes  -I. -IInclude -I./Include   -DPy_BUILD_CORE -o Python/mysnprintf.o Python/mysnprintf.c
powerpc-none-linux-gnuspe-gcc -c -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes  -I. -IInclude -I./Include   -DPy_BUILD_CORE -o Python/pyctype.o Python/pyctype.c
powerpc-none-linux-gnuspe-gcc -c -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes  -I. -IInclude -I./Include   -DPy_BUILD_CORE -o Parser/tokenizer_pgen.o Parser/tokenizer_pgen.c
powerpc-none-linux-gnuspe-gcc -c -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes  -I. -IInclude -I./Include   -DPy_BUILD_CORE -o Parser/printgrammar.o Parser/printgrammar.c
powerpc-none-linux-gnuspe-gcc -c -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes  -I. -IInclude -I./Include   -DPy_BUILD_CORE -o Parser/pgenmain.o Parser/pgenmain.c
powerpc-none-linux-gnuspe-gcc -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes  Parser/acceler.o Parser/grammar1.o Parser/listnode.o Parser/node.o Parser/parser.o Parser/parsetok.o Parser/bitset.o Parser/metagrammar.o Parser/firstsets.o Parser/grammar.o Parser/pgen.o Objects/obmalloc.o Python/mysnprintf.o Python/pyctype.o Parser/tokenizer_pgen.o Parser/printgrammar.o Parser/pgenmain.o -lpthread -ldl  -lpthread -lutil -o Parser/pgen
make[1]: Leaving directory `/home/user/Python-2.7.5'
Parser/pgen ./Grammar/Grammar Include/graminit.h Python/graminit.c
Parser/pgen: Parser/pgen: cannot execute binary file
make: *** [Include/graminit.h] Error 126
make Parser/pgen
make[1]: Entering directory `/home/user/Python-2.7.5'
make[1]: `Parser/pgen' is up to date.
make[1]: Leaving directory `/home/user/Python-2.7.5'
Parser/pgen ./Grammar/Grammar Include/graminit.h Python/graminit.c
Parser/pgen: Parser/pgen: cannot execute binary file
make: *** [Include/graminit.h] Error 126
msg198815 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2013-10-02 01:38
It is a known issue that the Python build infrastructure does not currently support cross compiling.  There are a number of issues and patches in this tracker that address pieces of this puzzle.  Help sorting it all out will be welcome.
msg198816 - (view) Author: Trevor Bowen (Trevor.Bowen) Date: 2013-10-02 03:14
Sorry, I do not mean to compound an already open and complex problem.  I thought that the fixes in 2.7.4 were meant in part to help alleviate this problem.  I had not found any feedback or tutorials, so I wanted to provide a status update of sorts.  I'm sure there is some.  I just have not been able to find it myself.

If the configure script passed in the cross-compile tool-chain *and* the build-host tool-chain, could the Makefile be enhanced to build pgen and python first with the build-host and then use them to build the full cross-compiled versions?  Or, would there be a better general approach?

I am not familiar with the MINGW requirements.  Would this satisfy their needs?
msg198834 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2013-10-02 13:04
I have no idea, frankly :)

I guess my point is that the tool chain was not designed with cross compilation in mind, so sorting out how to make it work and writing a howto is something that needs to be done.  The patches that have already been applied address a narrow need felt and fixed by the Debian folk, dealing with cross compiling 64 bit vs 32 bit, if I understand correctly.  I don't know if it is a general cross compile fix, but in theory it should at least have moved us closer.

Now, that said, as far as build order goes, what we do in general is to check in the sources that are built by the tools.  If you get your sources out of a tarball, make shouldn't try to rebuild those.  If you are working from a checkout, try 'make touch' before the make, to update the timestamps the way they would be in a release tarball.
msg198854 - (view) Author: Trevor Bowen (Trevor.Bowen) Date: 2013-10-02 17:16
Ok, thanks for the tips.  I'm new to developing on Python itself.  I'll start simple by trying to develop a set of patches for the 2.7.5 source tar-ball, which I usually use to build Python. If that succeeds, I'll look into pushing it into the related source files.  So far, I'm anticipating modifying these files: configure, Makefile.pre.in, and setup.py. ... Thanks!
msg199165 - (view) Author: Trevor Bowen (Trevor.Bowen) Date: 2013-10-07 23:01
In the vein of:

http://randomsplat.com/id5-cross-compiling-python-for-embedded-linux.html

I have created a patch and top-level build script, which builds the requisite python interpreter and Parser/pgen binary to run on the build system, which are both used during the cross-compile process.

In some ways, this patch is simpler, because of the new cross-compile capabilities in 2.7.4 and 2.7.5.  However, previous work is still insufficient, because the Makefile assumes the cross-compiled Parser/pgen can be executed on the build system during build.  This patch primarily compensates for that, but it also works around some misplaced installation file issues.

My top-level cross-compile script, which applies the patch *during* the build process is also attached.  It depends on a VCS to restore the original version, but I could have just as easily made 2 copies of the patched files (before and after patching) and copied the original versions as needed.

I hope this helps others.  Hopefully, bits of this can be integrated to help further simplify future cross-compile efforts.
msg199166 - (view) Author: Trevor Bowen (Trevor.Bowen) Date: 2013-10-07 23:13
Short version of cross-compile script without error checking:

#!/bin/bash
export RFS=/local/my_root_file_system
make distclean
rm -rf python_for_build Parser/pgen_for_build
git checkout -- Makefile.pre.in Modules/Setup.dist configure setup.py
./configure
make python Parser/pgen
mv python python_for_build
mv Parser/pgen Parser/pgen_for_build
patch -p3 < Python-2.7.5-xcompile.patch
export PATH="/opt/my_cross_compile_toolchain/gcc-4.3.74-eglibc-2.8.74-dp-2/powerpc-none-linux-gnuspe/bin:${PATH}"
make distclean
./configure --host=powerpc-none-linux-gnuspe --build=i586-linux-gnu --prefix=/  \
    --disable-ipv6 ac_cv_file__dev_ptmx=no ac_cv_file__dev_ptc=no ac_cv_have_long_long_format=yes
make --jobs=4
sudo make install DESTDIR=${RFS} PATH="${PATH}"
msg199167 - (view) Author: Trevor Bowen (Trevor.Bowen) Date: 2013-10-07 23:33
FWIW, I also explored my original proposal, which essentially moved the above script and modifications into the Python configure.ac and Makefile.pre.in files, so that Python's internal build process would create the native build system versions of Python and Parser/pgen, if necessary.  I made some progress in creating additional Make targets just for the native build system, which redefined the toolchain after completion; however, I encountered unexepected, overwhelming configure contamination in the auto-generated header files.

Maybe someone can push this farther than me?  I'm attaching my patch, in case it helps someone else and just as reference.

As for me, I think it is much easier to use the above solution, which would require very little patching to the sources to make a 2-step compilation to be trivial.
msg199185 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2013-10-08 09:22
Have you tried "make touch" to avoid rebuilding pgen and stuff?
msg199196 - (view) Author: Trevor Bowen (Trevor.Bowen) Date: 2013-10-08 12:00
I thought "make touch" was only for those trying to build from the Mecurial source, as opposed to building from the released tar-ball source.  I thought my efforts laid on the other side of the need for that command.  If I understood wrong, when would I use it and why? ... Thanks!
msg199213 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2013-10-08 16:09
make touch avoids rebuilding "pgen and stuff", and just uses what was checked out or provided in the tarball.  The release tarballs are supposed to have the time stamps in the correct order so that the compiletime/boostrapping utilities don't get built/rebuilt.

So for cross compilation, you should be able to use the existing files to build the target python.  If you then also want pgen *on the target*, you will need to build it for the target.  Like I said, I don't have much experience with this stuff, but if I understand correctly, you only really the utilities if you want to be able to do development *of python* on the target.

So, if running make in cross-compile "mode" (whatever that looks like, I have no clue) tries to rebuild pgen in your scenario, then *that* is what needs to be fixed, I think.

Of course, I could be completely wrong, which is why I've been pretty tentative about handing out advice :)
msg199220 - (view) Author: Trevor Bowen (Trevor.Bowen) Date: 2013-10-08 17:06
Thanks, David!  I have no interest in running pgen on the target/host.  My only interest is building python and its various modules to run on my embedded host.  I do not want to develop Python on the embedded host.  Unfortunately, the build process requires Parser/pgen to build the grammar files, which are needed for several object files.  Here's the relevant snippet from the Makefile.pre.in:

$(GRAMMAR_H): $(GRAMMAR_INPUT) $(PGENSRCS)
        @$(MKDIR_P) Include
        $(MAKE) $(PGEN)
        $(PGEN) $(GRAMMAR_INPUT) $(GRAMMAR_H) $(GRAMMAR_C)
$(GRAMMAR_C): $(GRAMMAR_H) $(GRAMMAR_INPUT) $(PGENSRCS)
        $(MAKE) $(GRAMMAR_H)
        touch $(GRAMMAR_C)

$(PGEN):    $(PGENOBJS)
        $(CC) $(OPT) $(LDFLAGS) $(PGENOBJS) $(LIBS) -o $(PGEN)

Parser/grammar.o:   $(srcdir)/Parser/grammar.c \
                $(srcdir)/Include/token.h \
                $(srcdir)/Include/grammar.h
...
Python/compile.o Python/symtable.o Python/ast.o: $(GRAMMAR_H) $(AST_H)

If there is a way to eliminate the need for Parser/pgen to run on the build system to cross-compile the default "all" target, that would be great. ... I'll experiment with "make touch". ... Thanks!
msg199222 - (view) Author: Trevor Bowen (Trevor.Bowen) Date: 2013-10-08 17:24
I executed "make touch" between "configure" and "make", but the build process still created Parser/pgen and then tried to use it, which of course crashed the build, since pgen was compiled for the embedded host not the build system.  :(  Was that the wrong usage?
msg199241 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2013-10-08 19:46
I believe that's the correct usage, in which case there must be a bug in the process somewhere.  My guess would be that it is looking for a file in the "wrong" place when doing a cross compile, but that's just a guess.
msg261261 - (view) Author: Marc Duponcheel (Marc Duponcheel) Date: 2016-03-06 16:24
one workaround is to build native in separate directory and to use the native build Parser/pgen and Programs/_freeze_importlib for cross compilation host build.
msg270290 - (view) Author: Martin Panter (martin.panter) * (Python committer) Date: 2016-07-13 02:37
Since 2.7.12 and 3.5.2, pgen should not be executed when in cross-compilation mode, thanks to Issue 22359 and Issue 22625. So I suspect the main problem is basically solved now.

Even though it should no longer be used, pgen is still cross-compiled depending on the timestamps and “make touch”. (Evidence: 
Issue 27490.)

My guess of why “make touch” did not work for Trevor was that he was compiling Python 2. In this case, “make touch” was added in revision da3f4774b939, and still only affects two files, neither of which are generated by pgen. Maybe Python 2’s “make touch” rule could be expanded for other generated files including Include/graminit.h etc.
msg270311 - (view) Author: Xavier de Gaye (xdegaye) * (Python triager) Date: 2016-07-13 14:40
I confirm that, on 3.6 and after the changesets that fixed issue 22359, pgen is always cross-compiled whatever the timestamps and that pgen is not run on a cross-build for Android. So I think it should work as well now for Trevor with v2.7.12.
History
Date User Action Args
2016-07-18 00:03:40martin.pantersetstatus: open -> closed
resolution: out of date
2016-07-13 14:40:47xdegayesetnosy: + xdegaye
messages: + msg270311
2016-07-13 02:37:49martin.pantersetsuperseder: When cross-compiling, don’t try to execute binaries

messages: + msg270290
nosy: + martin.panter
2016-03-06 16:24:54Marc Duponcheelsetnosy: + Marc Duponcheel
messages: + msg261261
2014-03-04 08:19:32gregory.p.smithsetnosy: + gregory.p.smith
2013-10-08 19:46:38r.david.murraysetmessages: + msg199241
2013-10-08 17:24:21Trevor.Bowensetmessages: + msg199222
2013-10-08 17:06:24Trevor.Bowensetmessages: + msg199220
2013-10-08 16:09:45r.david.murraysetmessages: + msg199213
2013-10-08 12:00:34Trevor.Bowensetmessages: + msg199196
2013-10-08 09:22:19pitrousetnosy: + pitrou
messages: + msg199185
2013-10-07 23:33:42Trevor.Bowensetfiles: + Python-2.7.5-xcompile_improvements.patch

messages: + msg199167
2013-10-07 23:13:03Trevor.Bowensetfiles: + cross-compile.sh

messages: + msg199166
2013-10-07 23:01:14Trevor.Bowensetfiles: + Python-2.7.5-xcompile.patch
keywords: + patch
messages: + msg199165
2013-10-02 17:16:36Trevor.Bowensetmessages: + msg198854
2013-10-02 13:04:13r.david.murraysetmessages: + msg198834
2013-10-02 03:14:51Trevor.Bowensetmessages: + msg198816
2013-10-02 01:38:11r.david.murraysetnosy: + r.david.murray
messages: + msg198815
2013-10-01 23:41:38Trevor.Bowencreate