classification
Title: Fixes Python for Android API 19
Type: Stage: resolved
Components: Versions: Python 3.8
process
Status: closed Resolution: wont fix
Dependencies: Superseder:
Assigned To: Nosy List: ned.deily, pmpp, vstinner, yan12125
Priority: normal Keywords: patch

Created on 2018-01-24 17:31 by vstinner, last changed 2018-05-29 21:46 by vstinner. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 5305 closed vstinner, 2018-01-24 17:56
Messages (14)
msg310619 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2018-01-24 17:31
I'm working on a pull request to make https://github.com/pmp-p/droid-pydk/tree/master/sources.32/cpython-bpo-30386.patchset changes upstream.
msg310682 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2018-01-25 16:09
I wrote PR 5305 for Paul Peny who uses cheap devices (less than 50$, maybe less 20$? I'm not sure) for development, but these devices use old Android versions.

It seems like some people are exchanging patches, in private or in public, for Android API 19. My intent is to make patches upstream, especially small patches which fix compilation of Python on Android API 19.

I'm not sure that we should support API 19.

Xavier de Gaye wants to focus on the future, support API 24 and newer:
https://github.com/xdegaye/cagibi/blob/master/doc/android_support.rst

His document gives many good reasons to not support API < 24.

My intent is not to "fully support" Python on Android API 19. Just to make sure that we can compile Python and that python3 -c pass doesn't crash :-) So only merge the most critical fixes for API 19.

IMHO merging further changes to fix other Python functions should be discussed on a case by base basis.
msg310685 - (view) Author: Chih-Hsuan Yen (yan12125) * Date: 2018-01-25 16:36
Here's another argument for supporting android < 21: according to Google [1], 12.8% devices on the world run android-19. Furthermore, upgrading Android OS for a device is almost impossible for non-hackers unless the manufacturer provides an image for newer Android, and it's common for manufacturers not upgrading a phone soon after its debut.
msg310687 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2018-01-25 17:05
I don't think we should be adding this platform-version-specific code unless we are going to fully support that particular platform and version.  That means at a minimum adding a buildbot for it and then having a core developer signed up to maintain the platform version.  It's not fair to other code developers and to downstream users to falsely imply that we support something that we are not set up to support, especially an already out-of-date version when we don't yet support any version of Android.  We will end up with another support headache as has been the case with other non-mainstream (with regard to Python) platforms.
msg310689 - (view) Author: pmp-p (pmpp) * Date: 2018-01-25 17:22
Thanks ! I tested your PR it just built out of the box with the help of a one line change to Xavier de Gaye's android build support tool ( configure-android script + Android folder from bpo-30386 ). 

I'll forward to you the result of testsuite on a $10 Arm cortex-a7 board running Android kitkat 4.4.2 ( H3droid ) as soon as it finishes as i run a testbot on it.

My point of view is that "out of date" doesn't mean people don't actually *use* it : vstinner is indeed right , vendors provides almost no updates for hardware which actually can live long and have the power of some low power i*&a* cpu but without the actual drawbacks we heard about recently.
msg310692 - (view) Author: Chih-Hsuan Yen (yan12125) * Date: 2018-01-25 17:32
Agree with Ned Deily here. If CPython is going to be support android-19, it's better to fully support it. I'm optimistic here - I believe there won't be many android-19-specific patches beyond locale-related ones. Once android-24 is done, we can revisit this part and see how many patches are needed.
msg310694 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2018-01-25 18:01
Ned Deily: "I don't think we should be adding this platform-version-specific code unless we are going to fully support that particular platform and version."

I know the theory and I know the practice :-)

I started to list platforms that are "supported" by Python, with various support levels:
http://vstinner.readthedocs.io/cpython.html#supported-platforms

For example, Linux has an excellent support, whereas OpenBSD is more on the "best effort" side. "grep OpenBSD" in CPython shows me at least 51 lines of code, whereas OpenBSD has no buildbot and no dedicated developer to handle OpenBSD specific issues. For example, test_crypt is broken since at least 2 years on OpenBSD, as many test_socket tests.

For OpenBSD, IMHO it's ok to have a best effort support, basically apply proposed patches.

For Android, the situation is somehow different. There are more and more people working on supporting officially Python on Android, like Xavier de Gaye and me who are core developers. Xavier is working on that for 2 years, and IMHO the port is done at 95%. The last part is just to setup a buildbot for Android API 24.

The question is the added maintenance cost of Android API 19 #ifdef code, compared to pain of building Python on Android.

Technically, it should be possible to setup a buildbot for Android API 19, but only with a subset of tests. Either explicitly only run a subset of tests on this hypothetic buildbot, or even modify the Python test suite to skip some tests on Android API 19. While I expect a few patches for fix Python compilation on Android API 19, I do expect a lot of @skipIf() in tests, since API 19 has many broken or missing features. The thing is that most people don't care of these missing or broken features, it's not an issue for most applications on Android.

Note: Kivy is using Python on Android for years, but they use CrystaX NDK and not Google NDK. This issue is about supporting Google NDK: have the maximum compatibility.
msg310695 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2018-01-25 18:04
Another example of platform with best effort support is AIX. Until very recently, ctypes.util.find_library() didn't work on AIX which causes many tests failure. The AIX buildbot is red as far as I can remember. *Many* tests are failing on AIX for various reasons. While a few people sometimes propose fixes, basically the AIX support didn't evolved much last years in AIX...

AIX is a platform with a buildbot, but no real dedicated developer to fix all AIX specific issues.
msg310697 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2018-01-25 18:12
By the way, I started to take notes on Python on Android, since the topic is wide and complex:
http://vstinner.readthedocs.io/python_android.html
msg310952 - (view) Author: Chih-Hsuan Yen (yan12125) * Date: 2018-01-28 15:11
Here are some ideas after testing:

* Testing environment:
 - My building scripts at https://github.com/yan12125/python3-android/
 - Android NDK r16b
 - Google's emulator image 'system-images;android-19;default;x86'

* Ideas/findings:
 - Locale emulation for setlocale() no longer works as _locale builds fine.

>>> locale.setlocale(locale.LC_ALL, None)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/data/local/tmp/python3/usr/lib/python3.7/locale.py", line 608, in setlocale
    return _setlocale(category, locale)
locale.Error: locale query failed

 - Declarations for mmap/sendfile shouldn't be necessary. In Android NDK r16b, sendfile() is defined if __USE_FILE_OFFSET64 is not defined: (from $ANDROID_NDK/sysroot/usr/include/sys/sendfile.h)

#if defined(__USE_FILE_OFFSET64)

#if __ANDROID_API__ >= 21
ssize_t sendfile(int __out_fd, int __in_fd, off_t* __offset, size_t __count) __RENAME(sendfile64) __INTRODUCED_IN(21);
#endif /* __ANDROID_API__ >= 21 */

#else
ssize_t sendfile(int __out_fd, int __in_fd, off_t* __offset, size_t __count);
#endif

#if __ANDROID_API__ >= 21
ssize_t sendfile64(int __out_fd, int __in_fd, off64_t* __offset, size_t __count) __INTRODUCED_IN(21);
#endif /* __ANDROID_API__ >= 21 */

__USE_FILE_OFFSET64 is defined as _FILE_OFFSET_BITS is defined to 64 in pyconfig.h. An NDK developer suggest "stop defining _FILE_OFFSET_BITS=64 on 32-bit Android." (https://github.com/android-ndk/ndk/issues/536#issuecomment-333197557) Either disabling large file support on Android or simply don't define _FILE_OFFSET_BITS on Android should work.   (Android does not use _LARGEFILE_SOURCE)

 - SIGRTMIN/SIGRTMAX: This is a bug on older NDK. r15 has fixed it (https://github.com/android-ndk/ndk/issues/352). I propose to wait until CPython builds with NDK r15+. I don't like adding patches for a known bug in older toolchains and revert them later.
msg310959 - (view) Author: pmp-p (pmpp) * Date: 2018-01-28 15:54
@yan12125
Hi, I ran some tests too and here are my findings:
 https://github.com/pmp-p/droid-pydk/tree/master/sources.32/build_logs

* I used some dirty patchset to cover more tests https://github.com/pmp-p/droid-pydk/tree/master/sources.32/cpython-bpo-30386.patchset i would be glad if you could have a look at them, as i am not a C/C++ programmer (i'm "pmpp" on IRC where we met last time).

* Maybe SIG* stuff is fixed only for unified headers API>=21 regardless of NDK that would need checking. 

* python build quite fine will all NDK ( sorry to say but especially the non vendor one can support cpython for prehistoric api and  version 3.5/3.6 *without patches* under certain condition of use )

my thoughts :

* Only third parties libs that have trouble with NDK versions, like openssl but libssl and libcrypto are provided by OS.

* Third parties are not required to embed libpython. Xavier de Gaye's script use shared linking. you should be able to provide whatever version of those you need that's not a cpython's build script job to deal with that, imho it was just for demontrasting that android build can be easy. I personnally would not build my third parties that way.

*  non official cpython build runs very well after android 5.0 there's almost nothing to do left, and micropython is able to run everywhere cpython can't (yet) so nothing is impossible.

* old api are frozen, there won't be suddenly a lot to support, when devices using them will vanish from asian stores it will mark the end of them and their need of support : imho there's no "end of life/support" concept for android os.

* I'd like a possible api19 "best effort", because android kitkat vendor statistics are wrong they don't take into account the *huge* amount of gservice-less devices.

unclear if unrelated to this:

* platform=='android' shoud be set, regardless of any API version : Android platform exists and is already running some pythons, it is a fact.

Thanks Victor for trying to clean my patchset : i did not realize my embed use of libpython could help official cpython too. Anyway I would be glad to help this way or another.
msg310966 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2018-01-28 16:39
Since we won't officially support Android in 3.7, these changes should wait for 3.8 and until there is a PEP for Android support.
msg311026 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2018-01-28 22:50
"""
>>> locale.setlocale(locale.LC_ALL, None)
...
locale.Error: locale query failed
"""

IMHO the correct fix here is to not define _locale.setlocale() to get the pure Python locale.setlocale() emulation.
msg318100 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2018-05-29 21:46
I'm sorry, but I'm not longer interested to work on supporting Android API 19. I close the bug. If someone wants to continue to work on the Android support, it's probably better to first focus on most recent Android versions.
History
Date User Action Args
2018-05-29 21:46:33vstinnersetstatus: open -> closed
resolution: wont fix
messages: + msg318100

stage: patch review -> resolved
2018-01-28 22:50:10vstinnersetmessages: + msg311026
2018-01-28 16:39:02ned.deilysetmessages: + msg310966
versions: + Python 3.8, - Python 3.7
2018-01-28 15:54:48pmppsetmessages: + msg310959
2018-01-28 15:11:26yan12125setmessages: + msg310952
2018-01-25 18:12:28vstinnersetmessages: + msg310697
2018-01-25 18:04:20vstinnersetmessages: + msg310695
2018-01-25 18:01:32vstinnersetmessages: + msg310694
2018-01-25 17:32:08yan12125setmessages: + msg310692
2018-01-25 17:22:30pmppsetmessages: + msg310689
2018-01-25 17:05:51ned.deilysetnosy: + ned.deily
messages: + msg310687
2018-01-25 16:36:08yan12125setnosy: + yan12125
messages: + msg310685
2018-01-25 16:09:52vstinnersetmessages: + msg310682
2018-01-25 13:36:53vstinnersetnosy: + pmpp
2018-01-24 17:56:50vstinnersetkeywords: + patch
stage: patch review
pull_requests: + pull_request5152
2018-01-24 17:31:20vstinnercreate