classification
Title: Add a build infrastructure for Android
Type: enhancement Stage: patch review
Components: Cross-Build Versions: Python 3.7
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: Alex.Willmer, haypo, martin.panter, ncoghlan, xdegaye, zach.ware
Priority: normal Keywords: patch

Created on 2017-05-17 13:57 by xdegaye, last changed 2017-05-27 19:57 by xdegaye.

Files
File name Uploaded Description Edit
issue29619.patch xdegaye, 2017-05-17 14:23
android-api-24.patch xdegaye, 2017-05-17 14:23
Pull Requests
URL Status Linked Edit
PR 1629 open xdegaye, 2017-05-17 14:17
Messages (4)
msg293852 - (view) Author: Xavier de Gaye (xdegaye) * (Python committer) Date: 2017-05-17 13:57
This PR adds a build infrastructure for Android on linux that allows a developer to check that source code changes do not break the existing state of this support by testing the changes on Android with a simple command. For example after a change to the socket module:
    $ /path/to/python_src/Android/makesetup && make python PYTHON_ARGS="-m test test_socket"

The README in the PR describes how to install the Android NDK and SDK, how to cross-compile Python, how to run python interactively or to run the Python test suite on the emulator, how to debug python on the emulator with gdb.

This build system has been tested with android-ndk-r13 and android-ndk-r14 which is the latest stable release of the NDK.  UnifiedHeaders [1] introduced in android-ndk-r14 is not yet used by this version of the build system.

[1] https://android.googlesource.com/platform/ndk.git/+/master/docs/UnifiedHeaders.md
msg293856 - (view) Author: Xavier de Gaye (xdegaye) * (Python committer) Date: 2017-05-17 14:23
The build system for the x86_64, x86 and armv7 architectures has been tested on API 21 and for the x86_64 and arm64 architectures on API 24.  Of these combinations of architecture/api, only x86_64 on API 21 builds correctly today, the others were building correctly few weeks ago but recent changes in Python seem to have broken those builds.

To fix this temporarily:
* Build the 32 bits architectures (x86 and armv7) with the attached issue29619.patch. This is a hack, the problem will be fixed in issue 29619.
* Build on API 24 with the attached android-api-24.patch. This is another hack, I will enter a new issue for those problems which are of a kind that is recurrent with the NDK: a function implemented in libc and whose declaration is missing in the NDK headers, a problem that is supposed to be fixed by UnifiedHeaders.
msg293956 - (view) Author: Xavier de Gaye (xdegaye) * (Python committer) Date: 2017-05-19 13:21
Commited 'bpo-30386: Support the new NDK Unified Headers'.

This does fix the missing declarations in the NDK headers and android-api-24.patch is not needed now.
It also fixes the missing declaration of sethostname() at API 24 that caused the import of the _socket module to fail.
More surprisingly it also fixes the compilation failure with 32 bits architectures (the size of off_t is now 8 instead of 4 previously) and issue29619.patch is not needed now.
Only one version of the NDK is supported, currently it is android-ndk-r14 the latest stable release.
The '-Wno-nullability-completeness' compilation flag is needed to avoid a flow of warnings in the processing of some NDK headers.

Unified Headers comes with a new annoying problem: langinfo.h exists now but nl_langinfo() is intentionaly not declared except for the __ANDROID_API_FUTURE__ API level
which is reserved for the development of the NDK itself. The solution chosen here is:
* Undefine HAVE_LANGINFO_H and CODESET in Include/pyport.h to maintain the status quo in the Python code.
* The Makefile generated by makesetup runs a script at configure time that checks that nl_langinfo() is not declared and aborts the build otherwise.  This allows to detect when langinfo.h starts being not broken anymore when swicthing to a new version of the NDK. In that case the following steps should be taken upon swicthing to this new version:
  + Remove this script from the build system (obviously).
  + Remove the changes in Include/pyport.h.
  + Revert the changes made in issue 28596.
msg294610 - (view) Author: Xavier de Gaye (xdegaye) * (Python committer) Date: 2017-05-27 19:57
Details of commit "Add Makefile targets to implement the buildbot steps" in 543a287c32a714a668e426ceb80f4add92cc5de4:

The following buildbot steps, as defined by the UnixBuild class of the master.cfg buildbot configuration file, have now their corresponding commands:
* Configure: makesetup configureFlags && make configure
* Compile:   make host
* Test:      make buildbottest
* Clean:     make hostclean

The 'configure' Makefile target builds the native interpreter, the external libraries and runs configure for the target host on its first execution. Afterwards it mainly runs configure.  As it is, the native interpreter is not distclean(ed) by the above four steps and therefore changes made in Setup.dist are ignored by the successive rebuilds of the native interpreter.
The 'buildbottest' Makefile target invokes the 'buildbottest' target of the host Python Makefile and runs using also the variables: TESTPYTHONOPTS, TESTTIMEOUT, TESTOPTS.

The following changes have also been made:
* The new Makefile target 'kill_emulator' allows to shutdown the emulator.
* All the shell and python scripts are now copied to the emulator at <sys.exec_prefix>/bin (i.e. /data/local/tmp/python/bin).
* The python_shell.sh script run by the user in interactive sessions ('install', 'emulator', 'adb_shell') and the shell scripts run by the 'python' and 'buildbottest' targets now change the current directory to <sys.exec_prefix> (i.e. /data/local/tmp/python) instead of its parent previously.  As a result, the temporary files and directories left over by 'buildbottest' are removed on the next run with the removal of the content of the previous <sys.exec_prefix> directory before the new installation.
* Fix file names whose size exceed MAX_FNAME_SIZE.
History
Date User Action Args
2017-05-27 19:57:12xdegayesetmessages: + msg294610
2017-05-19 13:21:01xdegayesetmessages: + msg293956
2017-05-18 21:58:54zach.waresetnosy: + zach.ware
2017-05-17 14:23:57xdegayesetfiles: + android-api-24.patch
2017-05-17 14:23:36xdegayesetfiles: + issue29619.patch
keywords: + patch
messages: + msg293856
2017-05-17 14:17:08xdegayesetpull_requests: + pull_request1720
2017-05-17 13:57:53xdegayecreate