% make
./Programs/_freeze_importlib \
        ./Lib/importlib/_bootstrap.py Python/importlib.h
./Programs/_freeze_importlib: ./Programs/_freeze_importlib: cannot execute binary file
Makefile:710: recipe for target 'Python/importlib.h' failed
make: *** [Python/importlib.h] Error 126

I tried make touch as it was suggested to me on #python-dev, but it didn’t fix that issue.

You're right, this can be avoided (although you will also have to patch the extension module build, which uses the just-built python executable).

This problem occurs twice in the build process, once for pgen and once for _freeze_importlib. The problem is that the Makefile rules for the generated files depend on the binary (which is of course not checked in and cannot be touched by "make touch").

Attached patch should fix this.

Thanks, this patch worked fine. :)

I used to build pgen natively first, then make distclean and rebuild until it failed, and then put the native version back here, but this works much better!

This was removed in c2a53aa27cad, to fix bpo-22359.

Indeed, that is an issue :(

I'm looking into this issue because of bpo-23670 (iOS support).

Am I correct in assuming that the right fix here is to identify a $(CC_FOR_BUILD) analog for $(PYTHON_FOR_BUILD) that will identify the build host's CC, enabling a build-host native $(PGEN) and _freeze_importlib to be compiled?

there are two approaches here, one to check in generated files and try to avoid the rebuild, or completely fix the cross build. I think the patch for the parallel build just was a bit over-eager

Incorrect recursive use of make will be fixed in default, 3.5, 3.4 (?), 2.7 in bpo-22359, reflect the versions correctly here so they're not forgotten.

So is there somewhere that documents how cross compilation is meant to be done? I tried looking at <https://docs.python.org/2/using/unix.html#building-python\>, <https://hg.python.org/cpython/file/2.7/README\>, and “./configure --help”, but found no hints for cross compilation.

If it is just word-of-mouth, would someone mind explaining to me how it is meant to be done?

The configure script lists CC as a “C compiler command”. But for cross compilation it sounds like you would need to differentiate host and target compilers. From the errors that have been posted, it sounds like you guys may be setting CC to the target compiler, causing the build to try and use the target compiler to build host programs.

In <http://thread.gmane.org/gmane.comp.python.devel/156550/focus=156663\> I suggested adding some makefile settings to manually override the pgen and _frozen_importlib programs. I imagine the changes would look something like cross-override.patch. Can someone who does cross compiling see if it is any use, and/or also indicate what commands or process they use to cross compile?

So in general: https://www.gnu.org/software/autoconf/manual/autoconf-2.69/html_node/System-Type.html#System-Type and https://www.gnu.org/software/autoconf/manual/autoconf-2.69/html_node/Hosts-and-Cross_002dCompilation.html

There are three platforms in play: target, host, build.

Host is the platform where what you build should run on.
build is the platform we are building on.
target is the platform where the *output* of the build thing itself should run on. Baby steps though: lets assume target==host always.

Now, the pathological case of building things is the canadian cross - https://en.wikipedia.org/wiki/Cross_compiler#Canadian_Cross

Note here that you actually build multiple different entire compilers, - and thats what we need here.

We need to build two python's.

One, for build, which pgen and _freeze_importlib can depend on.

One, for host, which is the output, and can depend on the output of running pgen and _freeze_importlib

I don't have examples of Makefile parameterisation to support this offhand, but gcc would be the obvious (if perhaps overwhelming) place to look at it.

The key things I'd expect are that:

• when host==build, the dependencies and outputs are identical, so we only build one copy of Python and everything else.
• when host!=build, we get a chain - the host Python -> pgenoutput -> pgen -> build Python -> pgenoutput

Thanks Robert. Russell also pointed out on python-dev that the patches at bpo-23670 show how he is doing cross compilation, via a new high-level iOS/Makefile file. Here is a cut-down version of the parts that I find relevant:

# Build for the native build host
host/bin/python*:
make -C .. distclean
cd .. && ./configure --prefix=host ...
make -C .. Programs/_freeze_importlib
# Main makefile is patched to use this copy (3.5+ only):
cp ../Programs/_freeze_importlib .
make -C .. install
tar czf host.tar.gz host && rm -rf host

# Build for a foreign target host
ios-%.tar.gz: host/bin/python*
make -C .. distclean
tar xzf host.tar.gz
cd .. && PATH=host/bin:$(PATH) ./configure \
--host=%-apple-ios --build=$(BUILD_OS_ID) CC=% LD=% \
--prefix=$* ...
PATH=host/bin:$(PATH) make -C .. install
tar czf $.tar.gz $ && rm -rf $*

Something I just realized is that the output of pgen is actually committed to the Mercurial repository. It seems that Russell’s cross-compilation was relying on this to avoid rerunning pgen. Same with the output of _freeze_importlib, except Russell is working around that differently. It does not seem right to have files in the repository that are also generated by the normal build process.

On 14 March 2016 at 01:05, Robert Collins <report@bugs.python.org> wrote:

There are three platforms in play: target, host, build.

Host is the platform where what you build should run on.
build is the platform we are building on.
target is the platform where the *output* of the build thing itself should run on. Baby steps though: lets assume target==host always.

To be 100% explicit: CPython doesn't need to worry about the third
one, the target platform. That only matters when the thing being
compiled, will itself cross-compile/process binaries at runtime e.g.
gcc, binutils. So if

• I'm on Linux and
• I want to compile a gcc that runs on BeOS
• that in turn compiles binaries for TempleOS

only then would target matter.

In bpo-22359, Xavier has posted a patch that looks like a reasonable solution to both cross compiling and allowing parallel make.