classification
Title: _bootlocale imports locale at startup on Android, causing test_site to fail
Type: behavior Stage: resolved
Components: Tests Versions: Python 3.7, Python 3.6
process
Status: closed Resolution: not a bug
Dependencies: Superseder:
Assigned To: xdegaye Nosy List: Alex.Willmer, Chi Hsuan Yen, vstinner, xdegaye
Priority: normal Keywords: patch

Created on 2016-05-03 14:06 by xdegaye, last changed 2016-12-17 08:24 by xdegaye. This issue is now closed.

Files
File name Uploaded Description Edit
test_output.txt xdegaye, 2016-05-03 14:06 test results
refactor_locale.patch xdegaye, 2016-05-21 16:23 review
no-locale-envvar.patch xdegaye, 2016-05-21 16:28 review
skip_test.patch xdegaye, 2016-11-03 09:55 review
skip_test_2.patch xdegaye, 2016-11-18 10:31 review
Messages (14)
msg264728 - (view) Author: Xavier de Gaye (xdegaye) * (Python committer) Date: 2016-05-03 14:06
One test of test_site fails on an android emulator running an x86 system image at API level 21.

See the attached test_output.txt file.
msg264903 - (view) Author: Xavier de Gaye (xdegaye) * (Python committer) Date: 2016-05-05 10:44
The problem is caused by the fact that android does not have HAVE_LANGINFO_H and CODESET set, hence in the _bootlocale module, the statement '_locale.CODESET' raises AttributeError and the locale module is imported upon interpreter startup. The locale module imports re.

See issue #19205 for why we would rather not import re and locale on startup.

This seems difficult to fix without either skipping most part of the test as it is done with Mac OS X, or having a specific sys.platform for android to handle the AttributeError in _bootlocale by having the getpreferredencoding() fuction returning 'UTF-8' ('UTF-8' is the file system encoding on android).
msg264904 - (view) Author: Xavier de Gaye (xdegaye) * (Python committer) Date: 2016-05-05 10:48
BTW the test runs fine on android when the AttributeError in _bootlocale is hard-coded with a getpreferredencoding() fuction returning 'UTF-8' and not importing locale.
msg266007 - (view) Author: Xavier de Gaye (xdegaye) * (Python committer) Date: 2016-05-21 16:23
Sorry for the confusion, the file system encoding is not the locale encoding.

In issue #9548, Antoine proposed a patch that avoids the import of the re, collections and functools modules by the _io module on startup, by refactoring and moving code from locale to _bootlocale. The attached refactor_locale.patch does that for python 3.6. The reasons for why Antoine patch has not been pushed still apply to this patch :(

The patch does fix the problem for Android though.
msg266008 - (view) Author: Xavier de Gaye (xdegaye) * (Python committer) Date: 2016-05-21 16:28
An improvement to Python startup time on Android (Android does not have nl_langinfo()) is to have _bootlocale.getpreferredencoding() return 'ascii' without importing locale, when none of the locale environment variables is set. With patch no-locale-envvar.patch, test_site runs ok on android-21-x86 emulator when the locale environment variables are not set.

Committing this patch while leaving the current issue open would also allow removing this issue from the dependencies of the Android meta-issue #26865.
msg279982 - (view) Author: Xavier de Gaye (xdegaye) * (Python committer) Date: 2016-11-03 09:55
This patch fixes test_startup_imports when the platform does not have langinfo.h.

Entered new issue 28596: "on Android _bootlocale on startup relies on too many library modules".
msg281090 - (view) Author: Xavier de Gaye (xdegaye) * (Python committer) Date: 2016-11-18 10:31
Patch that follows closely the conditionals in the __bootlocale module.
msg281091 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2016-11-18 10:55
> The problem is caused by the fact that android does not have HAVE_LANGINFO_H and CODESET set

Hum, is it possible to get the locale encoding by another way?

If not, what is the locale encoding?

Does Android provide mbstowcs() and wcstombs() functions?
msg281103 - (view) Author: Chi Hsuan Yen (Chi Hsuan Yen) * Date: 2016-11-18 12:04
Seems Android/BioniC always uses UTF-8: https://android.googlesource.com/platform/bionic/+/master/libc/bionic/mbrtoc32.cpp#83
msg281107 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2016-11-18 12:22
If it is not possible to change the locale, it makes sense to hardcode utf8.

Note: to avoid mojibake, it's better if sys.getfilesystemencoding() and
locale.getpreferredencoding(False) are equal. I understand that both must
be utf8.
msg281110 - (view) Author: Chi Hsuan Yen (Chi Hsuan Yen) * Date: 2016-11-18 12:52
There are some locale strings supported in setlocale(): https://android.googlesource.com/platform/bionic/+/master/libc/bionic/locale.cpp#104. However, seems mbstowcs just ignores such a setting on Android. Here's an example:

#include <locale.h>
#include <stdlib.h>
#include <string.h>
#include <stdio.h>

#define BUFFER_SIZE 10

void test_mbstowcs()
{
    wchar_t dest[BUFFER_SIZE];
    memset(dest, 0, sizeof(dest));
    printf("mbstowcs: %ld\n", mbstowcs(dest, "中文", BUFFER_SIZE));
    printf("dest: %x %x\n", dest[0], dest[1]);
}

int main()
{
    printf("setlocale: %d\n",  setlocale(LC_ALL, "en_US.UTF-8") != NULL);
    test_mbstowcs();
    printf("setlocale: %d\n",  setlocale(LC_ALL, "C") != NULL);
    test_mbstowcs();
    return 0;
}

On Linux (glibc 2.24) the result is:

$ ./a.out 
setlocale: 1
mbstowcs: 2
dest: 4e2d 6587
setlocale: 1
mbstowcs: -1
dest: 0 0

On Android (6.0 Marshmallow) the result is:
shell@ASUS_Z00E_2:/ $ /data/local/tmp/a.out
setlocale: 1
mbstowcs: 2
dest: 4e2d 6587
setlocale: 1
mbstowcs: 2
dest: 4e2d 6587

A quick search indicates setlocale() affects *scanf functions only, so I guess it's safe to force UTF-8 in CPython.
msg281115 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2016-11-18 13:28
In Python, the most important functions are Py_DecodeLocale() and
Py_EncodeLocale() which use mbstowcs() and wvstombs().
msg281132 - (view) Author: Chi Hsuan Yen (Chi Hsuan Yen) * Date: 2016-11-18 15:52
Submitted a patch to issue28596
msg283473 - (view) Author: Xavier de Gaye (xdegaye) * (Python committer) Date: 2016-12-17 08:24
Closing as invalid, it is useful to have the test failing on platforms that do not have CODESET and detect that too many modules are imported on startup. For Android, this problem is fixed in issue 28596.
History
Date User Action Args
2016-12-17 08:24:54xdegayesetstatus: open -> closed
resolution: not a bug
messages: + msg283473

stage: commit review -> resolved
2016-11-18 15:52:31Chi Hsuan Yensetmessages: + msg281132
2016-11-18 13:28:31vstinnersetmessages: + msg281115
2016-11-18 12:52:19Chi Hsuan Yensetmessages: + msg281110
2016-11-18 12:22:30vstinnersetmessages: + msg281107
2016-11-18 12:04:02Chi Hsuan Yensetnosy: + Chi Hsuan Yen
messages: + msg281103
2016-11-18 10:55:49vstinnersetnosy: + vstinner
messages: + msg281091
2016-11-18 10:31:45xdegayesetfiles: + skip_test_2.patch

messages: + msg281090
stage: patch review -> commit review
2016-11-03 09:55:52xdegayesetfiles: + skip_test.patch
versions: + Python 3.7
messages: + msg279982

assignee: xdegaye
components: + Tests, - Library (Lib), Cross-Build
stage: patch review
2016-05-21 21:16:36ppperrysettitle: _bootlocale imports locale at startup on Android -> _bootlocale imports locale at startup on Android, causing test_site to fail
2016-05-21 16:28:31xdegayesetfiles: + no-locale-envvar.patch

messages: + msg266008
2016-05-21 16:23:27xdegayesetfiles: + refactor_locale.patch
keywords: + patch
messages: + msg266007

title: android: test_site fails -> _bootlocale imports locale at startup on Android
2016-05-21 07:06:39xdegayelinkissue26865 dependencies
2016-05-05 10:48:07xdegayesetmessages: + msg264904
2016-05-05 10:44:40xdegayesetmessages: + msg264903
2016-05-03 14:06:05xdegayecreate