classification
Title: Android's incomplete locale.h implementation prevents cross-compilation
Type: compile error Stage: resolved
Components: Cross-Build Versions: Python 3.6
process
Status: closed Resolution: out of date
Dependencies: Superseder:
Assigned To: Nosy List: lemburg, lizhenhua, loewis, pitrou, shiz, skrah, vstinner, xdegaye, yan12125
Priority: normal Keywords: patch

Created on 2014-01-19 18:13 by shiz, last changed 2016-04-25 19:09 by skrah. This issue is now closed.

Files
File name Uploaded Description Edit
Python-3.4.0tip-workaround-android-locale-issues.patch shiz, 2014-01-19 18:13 Patch to work around locale.h issues in Android. review
Python-3.4-d51d6f1f9db8-workaround-android-locale-issues.patch shiz, 2014-01-20 19:08 review
Messages (15)
msg208486 - (view) Author: Shiz (shiz) * Date: 2014-01-19 18:13
As a result of Android's relatively incomplete locale.h implementation[1], some functions are not defined and some standard structs are lacking fields (e.g. decimal_point, thousand_sep).
This prevents proper cross-compilation using the Android NDK[2] from succeeding.

Attached is a patch which works around the various issues posed by Android's locale.h implementation, mainly involving falling back to default locale values, causing compilation in that department for the author to succeed.

[1]: https://groups.google.com/forum/#!topic/android-ndk/PBK4KkRpDoU
[2]: https://developer.android.com/tools/sdk/ndk/index.html
msg208489 - (view) Author: Marc-Andre Lemburg (lemburg) * (Python committer) Date: 2014-01-19 18:29
I'd be +1 on such a patch if we were to officially support Android, but we'd need a volunteer to champion for this (which would be a good thing, IMO).

Otherwise, such changes need to be maintained externally.
msg208541 - (view) Author: Stefan Krah (skrah) * (Python committer) Date: 2014-01-20 14:12
I agree with Marc-Andre.  Also, we should have an Android buildbot.
msg208542 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2014-01-20 14:22
Is there the only patch required to compile Python 3.4 on Android?
msg208561 - (view) Author: Shiz (shiz) * Date: 2014-01-20 19:08
I managed to cross-compile Python 3.3.3 for arm-linux-androideabi (using the Android NDK r9c) with the patches in this issue, issue 20306 and issue 20307. It did require a small additional patch which addressed the fact that the host system can't run the generated parser generator, which I'll make an issue for as soon as I refactored the patch. The patch allows the user to specify a host parser generator that points to a compiled version of pgen that is runnable on the host through the HOSTPGEN environment variable in ./configure.

Compiling the tip proved somewhat harder as the cross-compilation requires a host Python of the same version, which I didn't have. I formatted a patch which allows the user to supply a special host python path using the HOSTPYTHON environment variable to ./configure. I'll make an issue for that soon, and will bundle it with the HOSTPGEN patch if it's seen as a sufficient approach.

As far as maintaining an Android port for CPython goes; I may be interested in this as I'd need to regularly use it anyway. Can anyone tell me what the possibilities are here?

Since in the meanwhile the tip was updated to unconditionally include locale.h in Python/fileutils.c according to issue 19036, here's an updated patch for this issue that applies cleanly against the tip.
msg208583 - (view) Author: Stefan Krah (skrah) * (Python committer) Date: 2014-01-20 22:17
Shiz <report@bugs.python.org> wrote:
> As far as maintaining an Android port for CPython goes; I may be interested
> in this as I'd need to regularly use it anyway. Can anyone tell me what the
> possibilities are here?

There seem to be some external ports already. -- On the other hand, if we
integrate your patches, we'd sort of make Android officially supported.

The problems with this are:

   a) In the past proponents of non-mainstream platforms have not been
      very active after an initial enthusiasm.

   b) There were no build slaves (See http://www.python.org/dev/buildbot/)
      for these platforms.

   c) Many (or all) core committers did not have access to the platform
      in question.

Wikipedia claims that "QEMU also powers the Android emulator which is part of
the Android SDK."  Can you run a build slave with network access and also run
the test suite in this emulator?

Running a build slave is rather easy, see:

https://wiki.python.org/moin/BuildBot
http://bugs.python.org/file24399/buildslave_install.txt
msg208603 - (view) Author: Shiz (shiz) * Date: 2014-01-21 02:16
>   c) Many (or all) core committers did not have access to the platform
>      in question.

I'd say Android is quite a common platform these days, although I'd concur that it's not particularly easy to run Python OOTB on. :)

>Wikipedia claims that "QEMU also powers the Android emulator which is part of
>the Android SDK."  Can you run a build slave with network access and also run
>the test suite in this emulator?

I've been looking into this, and the only option provided by the Android NDK involves cross-compiling Python on a regular machine. This would require some custom commands before and in between the builds, to build a host Python first, then cross-compile the Android Python, and then start the emulator, transfer the binaries there and run the tests over adb. Can buildbot provide functionality so this can be configured on-slave, or would it require setting up a seperate master?
msg208660 - (view) Author: Stefan Krah (skrah) * (Python committer) Date: 2014-01-21 16:01
"I've been looking into this, and the only option provided by the Android NDK involves cross-compiling Python on a regular machine. This would require some custom commands before and in between the builds, to build a host Python first, then cross-compile the Android Python, and then start the emulator, transfer the binaries there and run the tests over adb. Can buildbot provide functionality so this can be configured on-slave, or would it require setting up a seperate master?"

I'm not sure. Antoine, do you think this is possible?
msg208661 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2014-01-21 16:10
> "I've been looking into this, and the only option provided by the
> Android NDK involves cross-compiling Python on a regular machine. This
> would require some custom commands before and in between the builds,
> to build a host Python first, then cross-compile the Android Python,
> and then start the emulator, transfer the binaries there and run the
> tests over adb. Can buildbot provide functionality so this can be
> configured on-slave, or would it require setting up a seperate
> master?"

The master can ask the slave to execute arbitrary commands. So, as long
as you implement the necessary steps as well-known commands executable
by the slave, you can then tell me how to insert them inside the build
process.

However, the only point of setting up an Android buildbot would be if:
1) Python actually builds correctly on Android
2) someone monitors the buildbot for failures and tries to act on them
msg236888 - (view) Author: Stefan Krah (skrah) * (Python committer) Date: 2015-02-28 14:24
Consider the libmpdec part rejected.  It is too much work given that
external libmpdecs need to be compatible and Android can use the Python
version of decimal.
msg262074 - (view) Author: Chih-Hsuan Yen (yan12125) * Date: 2016-03-20 13:24
Android NDK provides an "android-support" package [1]. There is no documentation other than README in its git repository, I guess it's a 'compatibility layer' between applications targetting a complete C library and Android's BioniC. With my patchset [2], Python builds and runs on my ASUS Zenfone 2 (Android 5.0.2). Patches proposed in this issue are no longer necessary.

[1] https://android.googlesource.com/platform/ndk.git/+/master/sources/android/support/
[2] https://github.com/yan12125/python3-android, see commit 82c971a96021e8f8006755b6963ac39f8a5fa412 and 73c2889c19c0bf259021fcc52a3baf5613b1e075
msg262075 - (view) Author: Stefan Krah (skrah) * (Python committer) Date: 2016-03-20 13:33
Thank you, this is excellent!  Does that mean that all locale patches are no longer needed? Here is one left (perhaps accidentally):

https://github.com/yan12125/python3-android/blob/cpython-hg/mk/ncurses/android-locale-fixes.patch
msg262088 - (view) Author: Chih-Hsuan Yen (yan12125) * Date: 2016-03-20 19:40
curses was broken and now it's fixed. Yes - Android's locale patch is no longer necessary in _curses either. However, there are runtime errors:

shell@ASUS_Z00E_2:/data/local/tmp $ python3.6 -c 'import curses; curses.initscr()'                           
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/data/local/tmp/python3/lib/python3.6/curses/__init__.py", line 30, in initscr
    fd=_sys.__stdout__.fileno())
_curses.error: setupterm: could not find terminfo database
msg264162 - (view) Author: Xavier de Gaye (xdegaye) * (Python triager) Date: 2016-04-25 08:07
This error:
    _curses.error: setupterm: could not find terminfo database

does not occur for me after removing the '--disable-database --disable-home-terminfo' options from the configure options used to cross-compile ncurses, and after setting the TERMINFO environment variable to a location where the the compiled description of the vt100 terminal is set. See https://bitbucket.org/xdegaye/pyona.

Also, with API level 21 (Android 5.0), the build of python3 with the official android toolchains (that is, without resorting to external libraries for wide character support) runs correctly and the python test suite is run with few errors, so IMHO this issue may be closed.
msg264194 - (view) Author: Stefan Krah (skrah) * (Python committer) Date: 2016-04-25 19:09
Thank you, closing this.
History
Date User Action Args
2016-04-25 19:09:35skrahsetstatus: open -> closed
versions: + Python 3.6, - Python 3.4
messages: + msg264194

resolution: out of date
stage: resolved
2016-04-25 08:07:53xdegayesetnosy: + xdegaye
messages: + msg264162
2016-03-20 19:40:26yan12125setmessages: + msg262088
2016-03-20 13:33:55skrahsetmessages: + msg262075
2016-03-20 13:24:01yan12125setnosy: + yan12125
messages: + msg262074
2015-02-28 14:24:13skrahsetnosy: + skrah
messages: + msg236888
2015-02-23 14:38:36vstinnerlinkissue23496 dependencies
2014-10-14 16:14:19skrahsetnosy: - skrah
2014-04-28 09:03:22skrahlinkissue21371 superseder
2014-04-28 09:02:23skrahsetnosy: + lizhenhua
2014-01-21 16:10:16pitrousetmessages: + msg208661
2014-01-21 16:01:21skrahsetnosy: + pitrou
messages: + msg208660
2014-01-21 02:16:45shizsetmessages: + msg208603
2014-01-20 22:17:09skrahsetmessages: + msg208583
2014-01-20 19:08:09shizsetfiles: + Python-3.4-d51d6f1f9db8-workaround-android-locale-issues.patch

messages: + msg208561
2014-01-20 14:22:10vstinnersetmessages: + msg208542
2014-01-20 14:12:14skrahsetmessages: + msg208541
2014-01-20 13:42:33pitrousetnosy: + vstinner, skrah
2014-01-19 18:29:22lemburgsetmessages: + msg208489
2014-01-19 18:19:09serhiy.storchakasetnosy: + lemburg, loewis
2014-01-19 18:13:49shizsettitle: Android's incomplete locale.h implementation prevents compilation -> Android's incomplete locale.h implementation prevents cross-compilation
2014-01-19 18:13:22shizcreate