classification
Title: [Patch] Fix the ability to cross compile Python when doing a rebuild of importlib.h
Type: Stage: resolved
Components: Interpreter Core Versions: Python 3.6, Python 3.5
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: EdSchouten, martin.panter, python-dev
Priority: normal Keywords: patch

Created on 2016-09-10 19:02 by EdSchouten, last changed 2016-09-12 03:35 by martin.panter. This issue is now closed.

Files
File name Uploaded Description Edit
patch-freeze-importlib EdSchouten, 2016-09-10 19:02 Patch to make cross compilation work review
patch-freeze-importlib EdSchouten, 2016-09-11 06:25 Revised patch: make sure that _freeze_importlib writes to the right location review
srcdir-check.patch martin.panter, 2016-09-11 09:24 review
Messages (6)
msg275674 - (view) Author: Ed Schouten (EdSchouten) * Date: 2016-09-10 19:02
For CloudABI (https://mail.python.org/pipermail/python-dev/2016-July/145708.html) we're providing packages containing a precompiled copy of Python. As we had to make some changes to importlib (namely to deal with directory file descriptors), we have to do a rebuild of importlib.h during the cross compilation process, using Programs/_freeze_importlib.

We've run into a couple of issues that require us to patch up the build system to make this work:

- First of all, Programs/_freeze_importlib is built using the compiler for the target. There is no way to override this. The OpenWRT folks have also run into this issue and have patched up the Makefile to add a $FREEZE_IMPORTLIB that allows you to use a different executable:

https://github.com/openwrt/packages/blob/master/lang/python3/patches/013-make-freeze-import-lib-into-an-override-able-var.patch

This patch is almost correct; it doesn't prevent _freeze_importlib from being built, which is needed in our case as it doesn't build cleanly on CloudABI.

- Second, if an out-of-tree build is performed, we need to make sure that Python/frozen.c imports the right copy of importlib.h. We must add -IPython to the compiler flags to realise.

Attached is a patch that contains the modifications that we've made to Makefile.pre.in to make cross compilation work for us.
msg275712 - (view) Author: Martin Panter (martin.panter) * (Python committer) Date: 2016-09-10 23:21
Can you get your importlib changes so they are compatible with a native build? If so, maybe you can build the frozen module with a native build rather than a cross build.

“freeze_importlib is built using the compiler for the target”: I may be mistaken, but I thought we recently prevented this from being cross-compiled: Issue 27641, revision bc677cb34889, Jul 2016. Your patch seems to be based ontop of this change. Perhaps you have found some way to bypass that cross-compilation detection logic.

The addition of $(FREEZE_IMPORTLIB) is a bit like the outdated cross-override.patch I once proposed for Issue 22625.
msg275735 - (view) Author: Ed Schouten (EdSchouten) * Date: 2016-09-11 06:25
The nice thing is that in our case, the importlib changes are already compatible with the native build. So yes, we can reuse the frozen module from the native build. :-)

Ah, yes. Issue 27641 already prevents that it's cross compiled. This patch was written prior to that and simply rebased.

Still, one problem remains: our strategy for creating importlib.h and importlib_external.h doesn't take out-of-tree builds into account. Even if we regenerate it, frozen.c won't pick it up because its directory is not part of the compiler search path.

Attached is a patch that makes us write the results of _freeze_importlib into $(srcdir)/Python/importlib*.h instead, so that even with an out-of-tree build, we actually update the header files used by frozen.c.
msg275754 - (view) Author: Martin Panter (martin.panter) * (Python committer) Date: 2016-09-11 09:20
There are various tricky cases to be considered with the regenerated files like importlib.h. One of them is that you are supposed to be able to build Python from a read-only source tree. See Issue 15819. Writing files into $(srcdir) would break this.

Also, as I just wrote in that bug, the built Python/importlib.h is already supposed to be searchable. However the logic seems to be broken. Can you see if this patch helps?
msg275799 - (view) Author: Ed Schouten (EdSchouten) * Date: 2016-09-11 14:27
It does. I can now cross build Python for CloudABI by copying importlib.h and importlib_external.h from the native build directory to the target build directory. Thanks!
msg275911 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2016-09-12 01:56
New changeset c26dce72a4da by Martin Panter in branch '3.5':
Issue #28066: Fix include search directory logic for out-of-tree builds
https://hg.python.org/cpython/rev/c26dce72a4da

New changeset 3ef078f96494 by Martin Panter in branch 'default':
Issue #28066: Merge srcdir fix from 3.5
https://hg.python.org/cpython/rev/3ef078f96494
History
Date User Action Args
2016-09-12 03:35:03martin.pantersetstatus: open -> closed
stage: patch review -> resolved
resolution: fixed
versions: + Python 3.5, Python 3.6, - Python 3.7
2016-09-12 01:56:30python-devsetnosy: + python-dev
messages: + msg275911
2016-09-11 14:27:12EdSchoutensetmessages: + msg275799
2016-09-11 09:24:14martin.pantersetfiles: + srcdir-check.patch
2016-09-11 09:22:07martin.pantersetfiles: - srcdir-check.patch
2016-09-11 09:20:46martin.pantersetfiles: + srcdir-check.patch

messages: + msg275754
stage: patch review
2016-09-11 09:08:20martin.panterlinkissue15819 dependencies
2016-09-11 07:56:58serhiy.storchakasetkeywords: + patch
2016-09-11 06:25:05EdSchoutensetfiles: + patch-freeze-importlib

messages: + msg275735
2016-09-10 23:21:01martin.pantersetnosy: + martin.panter
messages: + msg275712
2016-09-10 19:02:01EdSchoutencreate