classification
Title: document cross compilation
Type: enhancement Stage: patch review
Components: Cross-Build, Documentation Versions: Python 3.7, Python 3.6
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: xdegaye Nosy List: Alex.Willmer, martin.panter
Priority: normal Keywords: patch

Created on 2016-10-27 10:17 by xdegaye, last changed 2019-12-10 08:13 by xdegaye.

Files
File name Uploaded Description Edit
readme.patch xdegaye, 2016-10-27 10:17 review
readme_2.patch xdegaye, 2016-10-29 10:01 review
readme_3.patch xdegaye, 2016-10-29 19:29 review
readme_4.patch xdegaye, 2016-10-31 15:39 review
readme_5.patch xdegaye, 2016-11-02 20:24 review
Messages (12)
msg279532 - (view) Author: Xavier de Gaye (xdegaye) * (Python triager) Date: 2016-10-27 10:17
Patch adding a section to the README.
msg279648 - (view) Author: Martin Panter (martin.panter) * (Python committer) Date: 2016-10-29 03:26
Thanks, this looks like a decent start. I’m not that familiar with cross-compiling Python, but here are a couple questions:

$(cd /path/to/source && pwd)/configure
Why do you use $(pwd) here? I have mainly seen relative paths used (my own experiments, and reports from others), so I don’t think you need to force an absolute path. The instructions further up already mention “../configure” from a subdirectory.

--prefix=$(cd ../install && pwd)
Won’t that embed the path of the install directory in e.g. sys.prefix or something? Can’t you use “DESTDIR=../install” instead?

I remember there are a bunch of extra things you have to manually configure to cross-compile. See e.g. revision 12a56a349af2 which asks for /dev/ptmx and /dev/ptc settings in a CONFIG_SITE file (although I have had success just adding them to the “configure” command line). Perhaps we could add something like

'''
configure --host=[. . .] \
    ac_cv_file__dev_ptmx=no \
    ac_cv_file__dev_ptc=no
. . .

If the target Python could use /dev/ptmx or /dev/ptc to implement os.openpty(), set the corresponding argument to "yes".
'''

I suspect some people cross compile 2.7, so it may be worth applying something like this to that branch.
msg279668 - (view) Author: Xavier de Gaye (xdegaye) * (Python triager) Date: 2016-10-29 09:36
> $(cd /path/to/source && pwd)/configure
> Why do you use $(pwd) here?

'configure' fails with the following error message when used with '--prefix=../install':
    configure: error: expected an absolute directory name for --prefix: ../install
So for consistency, the example used absolute paths everywhere.

> --prefix=$(cd ../install && pwd)
> Won’t that embed the path of the install directory in e.g. sys.prefix or something? Can’t you use “DESTDIR=../install” instead?

You are right.
On an Android device where /usr and /bin do not exist, the platform independent Python files are installed on the sdcard.  So the build system I am using for Android does the following:
* uses '--prefix=/sdcard/org.bitbucket.pyona
* before running 'make install', it creates temporarily the /sdcard directory on the build system (with sudo) and use 'sudo mount --bind $INSTALL_ROOT /sdcard' so that 'make install' actually copies the files to $INSTALL_ROOT and so that the modules are byte compiled modules with the proper '-d' option to compileall. This allows also for multiple cross-compilation using different $INSTALL_ROOT directory names for each Android API level or architecture (x86, arm, armv7, ...)
* creates a tar file from the files in $INSTALL_ROOT that is later copied and expanded on the device using the Android swiss army knife 'adb shell'.


> I remember there are a bunch of extra things you have to manually configure to cross-compile. See e.g. revision 12a56a349af2 which asks for /dev/ptmx and /dev/ptc settings in a CONFIG_SITE file (although I have had success just adding them to the “configure” command line). Perhaps we could add something like
> 
> '''
> configure --host=[. . .] \
>     ac_cv_file__dev_ptmx=no \
>     ac_cv_file__dev_ptc=no
> . . .
> 
> If the target Python could use /dev/ptmx or /dev/ptc to implement os.openpty(), set the corresponding argument to "yes".

Ok

> I suspect some people cross compile 2.7, so it may be worth applying something like this to that branch.

Not sure that all the recent cross-compilation changes have been applied to 2.7. Would not documenting cross-compilation in 2.7 entail that we support it on 2.7 ?
msg279669 - (view) Author: Xavier de Gaye (xdegaye) * (Python triager) Date: 2016-10-29 10:01
New patch.
msg279695 - (view) Author: Xavier de Gaye (xdegaye) * (Python triager) Date: 2016-10-29 19:29
I have changed my Android build system to use DESTDIR, it is simpler than using 'mount --bind'. Thanks for the suggestion Martin.

Here is a new patch adding some text for the Android cross compilation.
msg279751 - (view) Author: Xavier de Gaye (xdegaye) * (Python triager) Date: 2016-10-30 21:20
One must also add '--(en|dis)able-ipv6' to configure, otherwise configure exits with:
    Fatal: You must get working getaddrinfo() function.
         or you can specify "--disable-ipv6".

The ACTION-IF-CROSS-COMPILING parameter of the AC_RUN_IFELSE that checks for getaddrinfo is $ac_cv_buggy_getaddrinfo="no -- configured with --(en|dis)able-ipv6".
msg279752 - (view) Author: Xavier de Gaye (xdegaye) * (Python triager) Date: 2016-10-30 21:24
> The ACTION-IF-CROSS-COMPILING parameter of the AC_RUN_IFELSE that checks for getaddrinfo is $ac_cv_buggy_getaddrinfo="no -- configured with --(en|dis)able-ipv6".

Hum, I should have written:
The ACTION-IF-CROSS-COMPILING parameter of the AC_RUN_IFELSE that checks for getaddrinfo is $ac_cv_buggy_getaddrinfo="no -- configured with --(en|dis)able-ipv6" when configure is run with --disable-ipv6 or --enable-ipv6.
msg279800 - (view) Author: Xavier de Gaye (xdegaye) * (Python triager) Date: 2016-10-31 15:39
New patch, less verbose and taking into account the previous posts.
msg279845 - (view) Author: Martin Panter (martin.panter) * (Python committer) Date: 2016-11-01 01:23
Regarding 2.7, I guess it depends on your definition of “support”. People have been cross-compiling Python 2.7 longer than I have been working on Python. In the past I have tried to apply cross compilation fixes to 2.7. But if you are only confident about 3.6, let’s just stick with that for the moment.

FWIW, DESTDIR, --prefix, etc are not specific to cross compilation or Android. DESTDIR is also useful when building a package/tarball or whatever, rather than installing directly. And --prefix would be useful if you don’t have root access and want to install to $HOME or somewhere. These settings are mentioned in the 2.7 README, but it looks like Guido trimmed them out for Python 3.0 (revision 07d67a9725a7).
msg279942 - (view) Author: Xavier de Gaye (xdegaye) * (Python triager) Date: 2016-11-02 20:24
About 2.7, this is a list of the issues I am aware of, that may have to be fixed also in 2.7 (and 3.5 for some of them): issues #26884 #27434 #28444 and #22724. Issue #28589 was entered today and is a duplicate of the previous one.

> FWIW, DESTDIR, --prefix, etc are not specific to cross compilation or Android.

I must be seeing everything through my little Android hammer these days :)
This new patch removes the last paragraph.
msg280098 - (view) Author: Martin Panter (martin.panter) * (Python committer) Date: 2016-11-05 00:31
Actually I think it is good to document DESTDIR (you weren’t seem aware of it before?). I was just pointing out that it is useful to more than just Android. Perhaps Guido shouldn’t have trimmed it from the 2.7 readme (original text: https://hg.python.org/cpython/diff/988b3e807043/README). On the other hand, --prefix etc are already mentioned in “./configure --help”, and more people already seem to know about it.
msg280121 - (view) Author: Xavier de Gaye (xdegaye) * (Python triager) Date: 2016-11-05 19:18
Yes, I was not aware of DESTDIR. The 2.7 README contains also useful information on the build process, for example about some use cases of the '*shared*' entries in the Setup files (for when setup.py cannot detect/build an extension module) or about the '*static' entries (to be able to profile extension modules). It also explains Setup.local whereas the only information on Setup.local in the Python3 documentation, is restricted to https://docs.python.org/3/extending/extending.html#compilation-and-linkage :(

There is also a lot of noise in the 2.7 README, this possibly explains the trimming. Maybe the cross-build section in the 3.6 README could be a link to a new build section in the documentation ?

> --prefix etc are already mentioned in “./configure --help”

The documentation on sys.prefix and sys.exec_prefix gives details about where go which files which is useful when trying to figure out where to copy the files on the target after a cross-build.
History
Date User Action Args
2019-12-10 08:13:06xdegayesetnosy: - xdegaye
2016-11-05 19:18:50xdegayesetmessages: + msg280121
2016-11-05 00:31:27martin.pantersetmessages: + msg280098
2016-11-02 20:24:43xdegayesetfiles: + readme_5.patch

messages: + msg279942
2016-11-01 01:24:00martin.pantersetmessages: + msg279845
2016-10-31 15:39:18xdegayesetfiles: + readme_4.patch

messages: + msg279800
2016-10-30 21:24:40xdegayesetmessages: + msg279752
2016-10-30 21:20:10xdegayesetmessages: + msg279751
2016-10-29 19:29:44xdegayesetfiles: + readme_3.patch

messages: + msg279695
2016-10-29 10:01:22xdegayesetfiles: + readme_2.patch

messages: + msg279669
2016-10-29 09:36:45xdegayesetmessages: + msg279668
2016-10-29 03:26:08martin.pantersetnosy: + martin.panter
messages: + msg279648
2016-10-27 14:58:28xdegayesetnosy: + Alex.Willmer
components: + Documentation, Cross-Build
2016-10-27 10:17:40xdegayecreate