New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
_bootlocale imports locale at startup on Android, causing test_site to fail #71115
Comments
One test of test_site fails on an android emulator running an x86 system image at API level 21. See the attached test_output.txt file. |
The problem is caused by the fact that android does not have HAVE_LANGINFO_H and CODESET set, hence in the _bootlocale module, the statement '_locale.CODESET' raises AttributeError and the locale module is imported upon interpreter startup. The locale module imports re. See issue bpo-19205 for why we would rather not import re and locale on startup. This seems difficult to fix without either skipping most part of the test as it is done with Mac OS X, or having a specific sys.platform for android to handle the AttributeError in _bootlocale by having the getpreferredencoding() fuction returning 'UTF-8' ('UTF-8' is the file system encoding on android). |
BTW the test runs fine on android when the AttributeError in _bootlocale is hard-coded with a getpreferredencoding() fuction returning 'UTF-8' and not importing locale. |
Sorry for the confusion, the file system encoding is not the locale encoding. In issue bpo-9548, Antoine proposed a patch that avoids the import of the re, collections and functools modules by the _io module on startup, by refactoring and moving code from locale to _bootlocale. The attached refactor_locale.patch does that for python 3.6. The reasons for why Antoine patch has not been pushed still apply to this patch :( The patch does fix the problem for Android though. |
An improvement to Python startup time on Android (Android does not have nl_langinfo()) is to have _bootlocale.getpreferredencoding() return 'ascii' without importing locale, when none of the locale environment variables is set. With patch no-locale-envvar.patch, test_site runs ok on android-21-x86 emulator when the locale environment variables are not set. Committing this patch while leaving the current issue open would also allow removing this issue from the dependencies of the Android meta-issue bpo-26865. |
This patch fixes test_startup_imports when the platform does not have langinfo.h. Entered new bpo-28596: "on Android _bootlocale on startup relies on too many library modules". |
Patch that follows closely the conditionals in the __bootlocale module. |
Hum, is it possible to get the locale encoding by another way? If not, what is the locale encoding? Does Android provide mbstowcs() and wcstombs() functions? |
Seems Android/BioniC always uses UTF-8: https://android.googlesource.com/platform/bionic/+/master/libc/bionic/mbrtoc32.cpp#83 |
If it is not possible to change the locale, it makes sense to hardcode utf8. Note: to avoid mojibake, it's better if sys.getfilesystemencoding() and |
There are some locale strings supported in setlocale(): https://android.googlesource.com/platform/bionic/+/master/libc/bionic/locale.cpp#104. However, seems mbstowcs just ignores such a setting on Android. Here's an example: #include <locale.h>
#include <stdlib.h>
#include <string.h>
#include <stdio.h>
#define BUFFER_SIZE 10
void test_mbstowcs()
{
wchar_t dest[BUFFER_SIZE];
memset(dest, 0, sizeof(dest));
printf("mbstowcs: %ld\n", mbstowcs(dest, "中文", BUFFER_SIZE));
printf("dest: %x %x\n", dest[0], dest[1]);
}
int main()
{
printf("setlocale: %d\n", setlocale(LC_ALL, "en_US.UTF-8") != NULL);
test_mbstowcs();
printf("setlocale: %d\n", setlocale(LC_ALL, "C") != NULL);
test_mbstowcs();
return 0;
} On Linux (glibc 2.24) the result is: $ ./a.out
setlocale: 1
mbstowcs: 2
dest: 4e2d 6587
setlocale: 1
mbstowcs: -1
dest: 0 0 On Android (6.0 Marshmallow) the result is: A quick search indicates setlocale() affects *scanf functions only, so I guess it's safe to force UTF-8 in CPython. |
In Python, the most important functions are Py_DecodeLocale() and |
Submitted a patch to bpo-28596 |
Closing as invalid, it is useful to have the test failing on platforms that do not have CODESET and detect that too many modules are imported on startup. For Android, this problem is fixed in bpo-28596. |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
bugs.python.org fields:
The text was updated successfully, but these errors were encountered: