With this patch, cross compiling a third-party extension module is done with the command:

  XBUILD_PYTHON_DIR=/path/to/python/dir python setup.py build

where XBUILD_PYTHON_DIR is the location of the directory of the cross-compiled python executable.
It may be:
a) The build tree, which is the source tree when the cross compilation is not out of the source tree.
b) '$DESTDIR/$exec_prefix/bin' when the cross built python has been installed with 'make DESTDIR=/some/path install'. In that case 'prefix' and 'exec_prefix' may be different.
c) When the result of the cross compilation has been manually copied (for example to /some/path) and if 'prefix' and 'exec_prefix' are identical, this is /some/path/bin.
In case b), one can use the 'install' setup.py command instead of the 'build' command, to build and install the extension module to '$DESTDIR/$exec_prefix/lib/python$VERSION/site-packages' with the appropriate 'egg-info' file.

The patch uses the 'python-config' shell script (created at issue 16235 [1]) that is initialized with the proper variables at the configure stage to provide the minimum information required by the sysconfig module to create (with the option --generate-posix-vars) the sysconfigdata file or to import the sysconfigdata module.
The patch also fixes issue 22724 [2] as sys.path is only modified during the time needed to import the sysconfigdata module.

The patch fixes two minor problems:
* '_PYTHON_HOST_PLATFORM' in Makefile.pre.in was not added to the sysconfig variables as intended, since those variables may not be prefixed with an underscore.
* The sysconfigdata file name was terminated with a dangling underscore when 'multiarch' is not defined.
Patch also tested with pyephem on an Android emulator.
The patch misses the documentation.
Please run autoconf after installing the patch.

[1] issue 16235: add python-config.sh for use during cross compilation
    http://bugs.python.org/issue16235
[2] issue 22724: byte-compile fails for cross-builds
    http://bugs.python.org/issue22724

This approach will not work with a "multiarch" enabled environment, and break cross builds on Debian and Ubuntu.

Afaics, the proposal assumes that the python executable for the target architecture is installed (which it is not for the multiarch cross-build layout), and that each target architecture has to be identified by it's own directory tree (again, not in the multiarch environment, you can install multiple targets into the same path).

I don't think that identifying the target by some path is the right way to go, and you should be able to identify the target by giving the target triplet as used by the configure parameters and then deduce the location from the target (or have an explicit location given).

The idea here is to use the platform identifier which we already had in the trunk before the PLATDIR was removed (the multiarch id or if not defined the platform). So by having a <target> specifier, you could deduce a path, or a <target>-python-config executable and you're done. No need to know about a path directly.  Of course python cross builds would have to install the <target>-python-config executable or symlink.

Minor issue: please s/xbuild/cross_build/.

> * The sysconfigdata file name was terminated with a dangling
>   underscore when 'multiarch' is not defined.

That only solves part of the problem in that the kernel/os version gets encoded as well, e.g. gnukfreebsd9, gnukfreebsd10, which is nasty when the version of your runtime and build time kernel differ.

> This approach will not work with a "multiarch" enabled environment, and break cross builds on Debian and Ubuntu.

No, the patch does not break cross builds on Debian and Ubuntu, unless you can demonstrate it does.

> Afaics, the proposal assumes that the python executable for the target architecture is installed (which it is not for the multiarch cross-build layout), and that each target architecture has to be identified by it's own directory tree (again, not in the multiarch environment, you can install multiple targets into the same path).

No, you are mistaken, the path name of the build tree may be used to cross build third-party extension modules as stated in case a) of msg281993, and this should also work with debian. BTW the same code is run to cross build a standard library extension module and a third-party extension module, and obviously python is not yet installed by the time the standard library extension modules are built ;-)

> The idea here is to use the platform identifier which we already had in the trunk before the PLATDIR was removed (the multiarch id or if not defined the platform). So by having a <target> specifier, you could deduce a path, or a <target>-python-config executable and you're done. No need to know about a path directly.  Of course python cross builds would have to install the <target>-python-config executable or symlink.

It is not clear why, because debian has this multiarch design, this should constrain our build system to follow their paradigm. After all, debian has already found some ways to cross-build the extension modules for their supported multiarch platforms since it is not possible, as of today, with upstream CPython.

> Minor issue: please s/xbuild/cross_build/.

Agreed.

> > * The sysconfigdata file name was terminated with a dangling
> >   underscore when 'multiarch' is not defined.
> 
> That only solves part of the problem in that the kernel/os version gets encoded as well, e.g. gnukfreebsd9, gnukfreebsd10, which is nasty when the version of your runtime and build time kernel differ.

No idea what is the problem you are refering to. It cannot be the "dangling underscore" problem since this is a cosmetic issue. Please explain.

> I don't think that identifying the target by some path is the right way to go, and you should be able to identify the target by giving the target triplet as used by the configure parameters and then deduce the location from the target (or have an explicit location given).
> 
> The idea here is to use the platform identifier which we already had in the trunk before the PLATDIR was removed (the multiarch id or if not defined the platform). So by having a <target> specifier, you could deduce a path, or a <target>-python-config executable and you're done. No need to know about a path directly.  Of course python cross builds would have to install the <target>-python-config executable or symlink.

Hum, you still need to provide the native python interpreter with the _path_ to the <target>-python-config executable that can be anywhere on the file system (its location is relative to the cross compiled build, not the native interpreter), so it seems that this contradicts your previous sentence saying "I don't think that identifying the target by some path is the right way to go".

So I suggest we start with this patch as it works for:
* The standard library extension modules (for debian as well).
* The third-party extension modules on platforms that have multiarch defined and on platforms that do not have multiarch defined (on debian systems one can only use the build directory for XBUILD_PYTHON_DIR, the other systems can use the three options described in msg281993).

Then we can later extend the semantics of XBUILD_PYTHON_DIR to include the multiarch triplet if this is necessary for debian systems, unless the debian developers want to continue using the system they have developed and are currently using to cross compile third-party extension modules.

Well, cross compiling extension modules already works fine...

$ _PYTHON_SYSCONFIGDATA_NAME=_sysconfigdata_m_linux_aarch64-linux-android PYTHONHOME=~/Projects/python3-android/build/21-aarch64-linux-android-4.9/usr python3.7 setup.py build_ext
running build_ext
building 'xx' extension
creating build
creating build/temp.linux-x86_64-3.7
/opt/android-ndk/toolchains/llvm/prebuilt/linux-x86_64/bin/clang -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -target aarch64-none-linux-android -gcc-toolchain /opt/android-ndk/toolchains/aarch64-linux-android-4.9/prebuilt/linux-x86_64 --sysroot=/opt/android-ndk/platforms/android-21/arch-arm64/usr -fPIE -fno-integrated-as -target aarch64-none-linux-android -gcc-toolchain /opt/android-ndk/toolchains/aarch64-linux-android-4.9/prebuilt/linux-x86_64 --sysroot=/opt/android-ndk/platforms/android-21/arch-arm64/usr -fPIE -fno-integrated-as -fPIC -I/home/yen/Projects/python3-android/build/21-aarch64-linux-android-4.9/usr/include/python3.7m -c xxmodule.c -o build/temp.linux-x86_64-3.7/xxmodule.o
creating build/lib.linux-x86_64-3.7
/opt/android-ndk/toolchains/llvm/prebuilt/linux-x86_64/bin/clang -shared -target aarch64-none-linux-android -gcc-toolchain /opt/android-ndk/toolchains/aarch64-linux-android-4.9/prebuilt/linux-x86_64 --sysroot=/opt/android-ndk/platforms/android-21/arch-arm64/usr -pie -L/home/yen/Projects/python3-android/build/21-aarch64-linux-android-4.9/usr/lib -target aarch64-none-linux-android -gcc-toolchain /opt/android-ndk/toolchains/aarch64-linux-android-4.9/prebuilt/linux-x86_64 --sysroot=/opt/android-ndk/platforms/android-21/arch-arm64/usr -pie -L/home/yen/Projects/python3-android/build/21-aarch64-linux-android-4.9/usr/lib build/temp.linux-x86_64-3.7/xxmodule.o -o build/lib.linux-x86_64-3.7/xx.cpython-37m-aarch64-linux-android.so
clang: warning: argument unused during compilation: '-pie'
clang: warning: argument unused during compilation: '-pie'

And it's running fine, too:

$ adb push build/lib.linux-x86_64-3.7/xx.cpython-37m-aarch64-linux-android.so /data/local/tmp
[100%] /data/local/tmp/xx.cpython-37m-aarch64-linux-android.so

$ adb shell
shell@ASUS_Z00E_2:/ $ cd /data/local/tmp
shell@ASUS_Z00E_2:/data/local/tmp $ . ./python3/tools/env.sh
shell@ASUS_Z00E_2:/data/local/tmp $ python3.7m
Python 3.7.0a0 (default:3d660ed2a60e+, Nov 23 2016, 20:22:14) 
[GCC 4.2.1 Compatible Android Clang 3.8.256229 ] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import xx
>>> xx
<module 'xx' from '/data/local/tmp/xx.cpython-37m-aarch64-linux-android.so'>
>>>

Here's my xxmodule.c:

#include "Python.h"

PyDoc_STRVAR(module_doc,
"This is a template module just for instruction.");

static struct PyModuleDef xxmodule = {
    PyModuleDef_HEAD_INIT,
    "xx",
    module_doc,
    0,
    NULL,
    NULL,
    NULL,
    NULL,
    NULL
};

PyMODINIT_FUNC
PyInit_xx(void)
{
    return PyModuleDef_Init(&xxmodule);
}

And setup.py:

from distutils.core import setup, Extension

module1 = Extension('xx',
                    sources=['xxmodule.c'])

setup(name='PackageName',
      version='1.0',
      description='This is a demo package',
      ext_modules=[module1])

Both the host and target Python are built from a6e59a2e880e

There's a little bug in the built filename: build/lib.linux-x86_64-3.7/xx.cpython-37m-aarch64-linux-android.so, which should be build/lib.linux-aarch64-3.7/xx.cpython-37m-aarch64-linux-android.so.

Also, I guess there may be quirks if NDK or the cross-built CPython is at different locations than built-time configurations. Those issues can be left to the future.

$ _PYTHON_SYSCONFIGDATA_NAME=_sysconfigdata_m_linux_aarch64-linux-android PYTHONHOME=~/Projects/python3-android/build/21-aarch64-linux-android-4.9/usr python3.7 setup.py build_ext
>

Variables prefixed by an underscore such as '_PYTHON_SYSCONFIGDATA_NAME' are private, please do not use them, they may be changed or removed at any time.

I know. I guess it can be determined without manually specifying but haven't investigated into details. Maybe some patches are necessary.

again, I don't think relying on a specific target path for a cross target is a good idea. and now deciding that the last possibility to use a target id to identify is better is internal doesn't make it better.  I'd appreciate if we could sit together where your work will lead to, and see how we accomplish everybody's goals.

> Hum, you still need to provide the native python interpreter
> with the _path_ to the <target>-python-config executable that
> can be anywhere on the file system

No, it's found in the same path. No contradiction.

> So I suggest we start with this patch as it works for:

Please do not.

> Then we can later extend the semantics of XBUILD_PYTHON_DIR

Again, I think that's the wrong way to go rely on any path as the primary id for the target, then "extending" on it makes it worse.

PR 17420 fixes cross-compilation of third-party extension modules.

The PYTHON_PROJECT_BASE environment variable is the path to the directory where Python has been cross-compiled. It is used by the native python interpreter to find the target sysconfigdata module.

For example the following command builds a wheel file to be transfered and installed with pip on the target platform, provided the native python interpreter and the cross-compiled one both have the wheel package installed:

  $ PYTHON_PROJECT_BASE=/path/to/builddir python setup.py bdist_wheel