classification
Title: When cross-compiling, don’t try to execute binaries
Type: compile error Stage: test needed
Components: Cross-Build Versions: Python 3.6, Python 3.5, Python 2.7
process
Status: closed Resolution: duplicate
Dependencies: Superseder: Remove incorrect uses of recursive make
View: 22359
Assigned To: Nosy List: Alex.Willmer, Arfrever, Link Mauve, benjamin.peterson, doko, freakboy3742, georg.brandl, koobs, mancoast, martin.panter, moreati, pitrou, rbcollins
Priority: normal Keywords: patch

Created on 2014-10-13 18:50 by Link Mauve, last changed 2016-04-20 05:10 by martin.panter. This issue is now closed.

Files
File name Uploaded Description Edit
makefile-regen-fix.patch georg.brandl, 2014-10-13 19:23 review
cross-override.patch martin.panter, 2016-03-12 06:00 review
Messages (15)
msg229261 - (view) Author: (Link Mauve) Date: 2014-10-13 18:50
```
% make
./Programs/_freeze_importlib \
        ./Lib/importlib/_bootstrap.py Python/importlib.h
./Programs/_freeze_importlib: ./Programs/_freeze_importlib: cannot execute binary file
Makefile:710: recipe for target 'Python/importlib.h' failed
make: *** [Python/importlib.h] Error 126
```

I tried `make touch` as it was suggested to me on #python-dev, but it didn’t fix that issue.
msg229264 - (view) Author: Georg Brandl (georg.brandl) * (Python committer) Date: 2014-10-13 19:23
You're right, this can be avoided (although you will also have to patch the extension module build, which uses the just-built python executable).

This problem occurs twice in the build process, once for pgen and once for _freeze_importlib.  The problem is that the Makefile rules for the generated files depend on the binary (which is of course not checked in and cannot be touched by "make touch").

Attached patch should fix this.
msg229271 - (view) Author: (Link Mauve) Date: 2014-10-13 21:08
Thanks, this patch worked fine. :)

I used to build pgen natively first, then make distclean and rebuild until it failed, and then put the native version back here, but this works much better!
msg229273 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2014-10-13 22:10
This was removed in c2a53aa27cad, to fix #22359.
msg229274 - (view) Author: Georg Brandl (georg.brandl) * (Python committer) Date: 2014-10-14 00:11
Indeed, that is an issue :(
msg238198 - (view) Author: Russell Keith-Magee (freakboy3742) * Date: 2015-03-16 13:32
I'm looking into this issue because of issue23670 (iOS support). 

Am I correct in assuming that the right fix here is to identify a $(CC_FOR_BUILD) analog for $(PYTHON_FOR_BUILD) that will identify the build host's CC, enabling a build-host native $(PGEN) and _freeze_importlib to be compiled?
msg238200 - (view) Author: Matthias Klose (doko) * (Python committer) Date: 2015-03-16 13:41
there are two approaches here, one to check in generated files and try to avoid the rebuild, or completely fix the cross build. I think the patch for the parallel build just was a bit over-eager
msg245367 - (view) Author: Kubilay Kocak (koobs) (Python triager) Date: 2015-06-15 06:01
Incorrect recursive use of make will be fixed in default, 3.5, 3.4 (?), 2.7 in issue 22359, reflect the versions correctly here so they're not forgotten.
msg247652 - (view) Author: Martin Panter (martin.panter) * (Python committer) Date: 2015-07-30 06:36
So is there somewhere that documents how cross compilation is meant to be done? I tried looking at <https://docs.python.org/2/using/unix.html#building-python>, <https://hg.python.org/cpython/file/2.7/README>, and “./configure --help”, but found no hints for cross compilation.

If it is just word-of-mouth, would someone mind explaining to me how it is meant to be done?

The configure script lists CC as a “C compiler command”. But for cross compilation it sounds like you would need to differentiate host and target compilers. From the errors that have been posted, it sounds like you guys may be setting CC to the target compiler, causing the build to try and use the target compiler to build host programs.
msg258994 - (view) Author: ƦOB COASTN (mancoast) * Date: 2016-01-27 03:28
Similar patch used for 3.5
Additionally required:
-Python/importlib_external.h: $(srcdir)/Lib/importlib/_bootstrap_external.py Programs/_freeze_importlib
+Python/importlib_external.h: $(srcdir)/Lib/importlib/_bootstrap_external.py
+	$(MAKE) Programs/_freeze_importlib
 	./Programs/_freeze_importlib \
 		$(srcdir)/Lib/importlib/_bootstrap_external.py Python/importlib_external.h
msg261635 - (view) Author: Martin Panter (martin.panter) * (Python committer) Date: 2016-03-12 06:00
In <http://thread.gmane.org/gmane.comp.python.devel/156550/focus=156663> I suggested adding some makefile settings to manually override the pgen and _frozen_importlib programs. I imagine the changes would look something like cross-override.patch. Can someone who does cross compiling see if it is any use, and/or also indicate what commands or process they use to cross compile?
msg261705 - (view) Author: Robert Collins (rbcollins) * (Python committer) Date: 2016-03-14 01:05
So in general: https://www.gnu.org/software/autoconf/manual/autoconf-2.69/html_node/System-Type.html#System-Type and https://www.gnu.org/software/autoconf/manual/autoconf-2.69/html_node/Hosts-and-Cross_002dCompilation.html 

There are three platforms in play: target, host, build.

Host is the platform where what you build should run on.
build is the platform we are building on.
target is the platform where the *output* of the build thing itself should run on. Baby steps though: lets assume target==host always.

Now, the pathological case of building things is the canadian cross - https://en.wikipedia.org/wiki/Cross_compiler#Canadian_Cross

Note here that you actually build multiple different entire compilers, - and thats what we need here.

We need to build two python's. 

One, for build, which pgen and _freeze_importlib can depend on.

One, for host, which is the output, and can depend on the output of running pgen and _freeze_importlib

I don't have examples of Makefile parameterisation to support this offhand, but gcc would be the obvious (if perhaps overwhelming) place to look at it.

The key things I'd expect are that:
 - when host==build, the dependencies and outputs are identical, so we only build one copy of Python and everything else.
 - when host!=build, we get a chain - the host Python -> pgenoutput -> pgen -> build Python -> pgenoutput
msg261710 - (view) Author: Martin Panter (martin.panter) * (Python committer) Date: 2016-03-14 02:02
Thanks Robert. Russell also pointed out on python-dev that the patches at Issue 23670 show how he is doing cross compilation, via a new high-level iOS/Makefile file. Here is a cut-down version of the parts that I find relevant:

# Build for the native build host
host/bin/python*:
	make -C .. distclean
	cd .. && ./configure --prefix=host ...
	make -C .. Programs/_freeze_importlib
	# Main makefile is patched to use this copy (3.5+ only):
	cp ../Programs/_freeze_importlib .
	make -C .. install
	tar czf host.tar.gz host && rm -rf host

# Build for a foreign target host
ios-%.tar.gz: host/bin/python*
	make -C .. distclean
	tar xzf host.tar.gz
	cd .. && PATH=host/bin:$(PATH) ./configure \
	    --host=%-apple-ios --build=$(BUILD_OS_ID) CC=% LD=% \
	    --prefix=$* ...
	PATH=host/bin:$(PATH) make -C .. install
	tar czf $*.tar.gz $* && rm -rf $*

Something I just realized is that the output of pgen is actually committed to the Mercurial repository. It seems that Russell’s cross-compilation was relying on this to avoid rerunning pgen. Same with the output of _freeze_importlib, except Russell is working around that differently. It does not seem right to have files in the repository that are also generated by the normal build process.
msg261736 - (view) Author: Alex Willmer (moreati) * Date: 2016-03-14 09:21
On 14 March 2016 at 01:05, Robert Collins <report@bugs.python.org> wrote:
> There are three platforms in play: target, host, build.
>
> Host is the platform where what you build should run on.
> build is the platform we are building on.
> target is the platform where the *output* of the build thing itself should run on. Baby steps though: lets assume target==host always.

To be 100% explicit: CPython doesn't need to worry about the third
one, the target platform. That only matters when the thing being
compiled, will itself cross-compile/process binaries at runtime e.g.
gcc, binutils. So if

 - I'm on Linux and
 - I want to compile a gcc that runs on BeOS
 - that in turn compiles binaries for TempleOS

only then would target matter.
msg263794 - (view) Author: Martin Panter (martin.panter) * (Python committer) Date: 2016-04-20 05:10
In Issue 22359, Xavier has posted a patch that looks like a reasonable solution to both cross compiling and allowing parallel make.
History
Date User Action Args
2016-07-13 02:37:48martin.panterlinkissue19142 superseder
2016-04-20 05:10:22martin.pantersetstatus: open -> closed
superseder: Remove incorrect uses of recursive make
resolution: duplicate
messages: + msg263794
2016-04-20 05:07:32martin.panterunlinkissue22359 dependencies
2016-03-14 09:21:28moreatisetnosy: + moreati
messages: + msg261736
2016-03-14 02:03:00martin.pantersetmessages: + msg261710
2016-03-14 01:05:11rbcollinssetmessages: + msg261705
2016-03-12 06:00:33martin.pantersetfiles: + cross-override.patch

messages: + msg261635
versions: - Python 3.4
2016-03-02 00:25:13Alex.Willmersetnosy: + Alex.Willmer
2016-01-27 03:28:53mancoastsetnosy: + mancoast
messages: + msg258994
2015-08-01 04:43:30rbcollinssetnosy: + rbcollins
2015-07-30 06:36:24martin.pantersetnosy: + martin.panter

messages: + msg247652
stage: test needed
2015-07-30 06:16:27martin.panterlinkissue22359 dependencies
2015-07-21 07:09:45ethan.furmansetnosy: - ethan.furman
2015-06-15 06:01:17koobssetnosy: + koobs

messages: + msg245367
versions: + Python 2.7, Python 3.4, Python 3.6
2015-03-16 18:39:01ethan.furmansetnosy: + ethan.furman
2015-03-16 13:41:45dokosetmessages: + msg238200
2015-03-16 13:32:49freakboy3742setnosy: + freakboy3742
messages: + msg238198
2014-11-07 09:40:40Arfreversetnosy: + Arfrever
2014-11-06 21:37:12georg.brandllinkissue22809 superseder
2014-10-14 00:11:03georg.brandlsetnosy: + doko
messages: + msg229274
2014-10-13 22:10:25pitrousetmessages: + msg229273
2014-10-13 21:08:03Link Mauvesetmessages: + msg229271
2014-10-13 19:23:21georg.brandlsetfiles: + makefile-regen-fix.patch

nosy: + georg.brandl, pitrou, benjamin.peterson
messages: + msg229264

keywords: + patch
2014-10-13 18:50:19Link Mauvecreate