Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

_bootlocale imports locale at startup on Android, causing test_site to fail #71115

Closed
xdegaye mannequin opened this issue May 3, 2016 · 14 comments
Closed

_bootlocale imports locale at startup on Android, causing test_site to fail #71115

xdegaye mannequin opened this issue May 3, 2016 · 14 comments
Assignees
Labels
3.7 (EOL) end of life tests Tests in the Lib/test dir type-bug An unexpected behavior, bug, or error

Comments

@xdegaye
Copy link
Mannequin

xdegaye mannequin commented May 3, 2016

BPO 26928
Nosy @vstinner, @xdegaye, @moreati, @yan12125
Files
  • test_output.txt: test results
  • refactor_locale.patch
  • no-locale-envvar.patch
  • skip_test.patch
  • skip_test_2.patch
  • Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = 'https://github.com/xdegaye'
    closed_at = <Date 2016-12-17.08:24:54.119>
    created_at = <Date 2016-05-03.14:06:05.484>
    labels = ['3.7', 'invalid', 'type-bug', 'tests']
    title = '_bootlocale imports locale at startup on Android, causing test_site to fail'
    updated_at = <Date 2016-12-17.08:24:54.117>
    user = 'https://github.com/xdegaye'

    bugs.python.org fields:

    activity = <Date 2016-12-17.08:24:54.117>
    actor = 'xdegaye'
    assignee = 'xdegaye'
    closed = True
    closed_date = <Date 2016-12-17.08:24:54.119>
    closer = 'xdegaye'
    components = ['Tests']
    creation = <Date 2016-05-03.14:06:05.484>
    creator = 'xdegaye'
    dependencies = []
    files = ['42696', '42928', '42929', '45334', '45530']
    hgrepos = []
    issue_num = 26928
    keywords = ['patch']
    message_count = 14.0
    messages = ['264728', '264903', '264904', '266007', '266008', '279982', '281090', '281091', '281103', '281107', '281110', '281115', '281132', '283473']
    nosy_count = 4.0
    nosy_names = ['vstinner', 'xdegaye', 'Alex.Willmer', 'yan12125']
    pr_nums = []
    priority = 'normal'
    resolution = 'not a bug'
    stage = 'resolved'
    status = 'closed'
    superseder = None
    type = 'behavior'
    url = 'https://bugs.python.org/issue26928'
    versions = ['Python 3.6', 'Python 3.7']

    @xdegaye
    Copy link
    Mannequin Author

    xdegaye mannequin commented May 3, 2016

    One test of test_site fails on an android emulator running an x86 system image at API level 21.

    See the attached test_output.txt file.

    @xdegaye xdegaye mannequin added stdlib Python modules in the Lib dir build The build process and cross-build type-bug An unexpected behavior, bug, or error labels May 3, 2016
    @xdegaye
    Copy link
    Mannequin Author

    xdegaye mannequin commented May 5, 2016

    The problem is caused by the fact that android does not have HAVE_LANGINFO_H and CODESET set, hence in the _bootlocale module, the statement '_locale.CODESET' raises AttributeError and the locale module is imported upon interpreter startup. The locale module imports re.

    See issue bpo-19205 for why we would rather not import re and locale on startup.

    This seems difficult to fix without either skipping most part of the test as it is done with Mac OS X, or having a specific sys.platform for android to handle the AttributeError in _bootlocale by having the getpreferredencoding() fuction returning 'UTF-8' ('UTF-8' is the file system encoding on android).

    @xdegaye
    Copy link
    Mannequin Author

    xdegaye mannequin commented May 5, 2016

    BTW the test runs fine on android when the AttributeError in _bootlocale is hard-coded with a getpreferredencoding() fuction returning 'UTF-8' and not importing locale.

    @xdegaye
    Copy link
    Mannequin Author

    xdegaye mannequin commented May 21, 2016

    Sorry for the confusion, the file system encoding is not the locale encoding.

    In issue bpo-9548, Antoine proposed a patch that avoids the import of the re, collections and functools modules by the _io module on startup, by refactoring and moving code from locale to _bootlocale. The attached refactor_locale.patch does that for python 3.6. The reasons for why Antoine patch has not been pushed still apply to this patch :(

    The patch does fix the problem for Android though.

    @xdegaye xdegaye mannequin changed the title android: test_site fails _bootlocale imports locale at startup on Android May 21, 2016
    @xdegaye
    Copy link
    Mannequin Author

    xdegaye mannequin commented May 21, 2016

    An improvement to Python startup time on Android (Android does not have nl_langinfo()) is to have _bootlocale.getpreferredencoding() return 'ascii' without importing locale, when none of the locale environment variables is set. With patch no-locale-envvar.patch, test_site runs ok on android-21-x86 emulator when the locale environment variables are not set.

    Committing this patch while leaving the current issue open would also allow removing this issue from the dependencies of the Android meta-issue bpo-26865.

    @pppery pppery mannequin changed the title _bootlocale imports locale at startup on Android _bootlocale imports locale at startup on Android, causing test_site to fail May 21, 2016
    @xdegaye
    Copy link
    Mannequin Author

    xdegaye mannequin commented Nov 3, 2016

    This patch fixes test_startup_imports when the platform does not have langinfo.h.

    Entered new bpo-28596: "on Android _bootlocale on startup relies on too many library modules".

    @xdegaye xdegaye mannequin added tests Tests in the Lib/test dir 3.7 (EOL) end of life and removed stdlib Python modules in the Lib dir build The build process and cross-build labels Nov 3, 2016
    @xdegaye xdegaye mannequin self-assigned this Nov 3, 2016
    @xdegaye
    Copy link
    Mannequin Author

    xdegaye mannequin commented Nov 18, 2016

    Patch that follows closely the conditionals in the __bootlocale module.

    @vstinner
    Copy link
    Member

    The problem is caused by the fact that android does not have HAVE_LANGINFO_H and CODESET set

    Hum, is it possible to get the locale encoding by another way?

    If not, what is the locale encoding?

    Does Android provide mbstowcs() and wcstombs() functions?

    @yan12125
    Copy link
    Mannequin

    yan12125 mannequin commented Nov 18, 2016

    @vstinner
    Copy link
    Member

    If it is not possible to change the locale, it makes sense to hardcode utf8.

    Note: to avoid mojibake, it's better if sys.getfilesystemencoding() and
    locale.getpreferredencoding(False) are equal. I understand that both must
    be utf8.

    @yan12125
    Copy link
    Mannequin

    yan12125 mannequin commented Nov 18, 2016

    There are some locale strings supported in setlocale(): https://android.googlesource.com/platform/bionic/+/master/libc/bionic/locale.cpp#104. However, seems mbstowcs just ignores such a setting on Android. Here's an example:

    #include <locale.h>
    #include <stdlib.h>
    #include <string.h>
    #include <stdio.h>
    
    #define BUFFER_SIZE 10
    
    void test_mbstowcs()
    {
        wchar_t dest[BUFFER_SIZE];
        memset(dest, 0, sizeof(dest));
        printf("mbstowcs: %ld\n", mbstowcs(dest, "中文", BUFFER_SIZE));
        printf("dest: %x %x\n", dest[0], dest[1]);
    }
    
    int main()
    {
        printf("setlocale: %d\n",  setlocale(LC_ALL, "en_US.UTF-8") != NULL);
        test_mbstowcs();
        printf("setlocale: %d\n",  setlocale(LC_ALL, "C") != NULL);
        test_mbstowcs();
        return 0;
    }

    On Linux (glibc 2.24) the result is:

    $ ./a.out 
    setlocale: 1
    mbstowcs: 2
    dest: 4e2d 6587
    setlocale: 1
    mbstowcs: -1
    dest: 0 0

    On Android (6.0 Marshmallow) the result is:
    shell@ASUS_Z00E_2:/ $ /data/local/tmp/a.out
    setlocale: 1
    mbstowcs: 2
    dest: 4e2d 6587
    setlocale: 1
    mbstowcs: 2
    dest: 4e2d 6587

    A quick search indicates setlocale() affects *scanf functions only, so I guess it's safe to force UTF-8 in CPython.

    @vstinner
    Copy link
    Member

    In Python, the most important functions are Py_DecodeLocale() and
    Py_EncodeLocale() which use mbstowcs() and wvstombs().

    @yan12125
    Copy link
    Mannequin

    yan12125 mannequin commented Nov 18, 2016

    Submitted a patch to bpo-28596

    @xdegaye
    Copy link
    Mannequin Author

    xdegaye mannequin commented Dec 17, 2016

    Closing as invalid, it is useful to have the test failing on platforms that do not have CODESET and detect that too many modules are imported on startup. For Android, this problem is fixed in bpo-28596.

    @xdegaye xdegaye mannequin closed this as completed Dec 17, 2016
    @xdegaye xdegaye mannequin added the invalid label Dec 17, 2016
    @ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
    Labels
    3.7 (EOL) end of life tests Tests in the Lib/test dir type-bug An unexpected behavior, bug, or error
    Projects
    None yet
    Development

    No branches or pull requests

    1 participant