Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

test_io large file test failure on 32 bits Android platforms #71113

Closed
xdegaye mannequin opened this issue May 3, 2016 · 12 comments
Closed

test_io large file test failure on 32 bits Android platforms #71113

xdegaye mannequin opened this issue May 3, 2016 · 12 comments
Assignees
Labels
3.7 (EOL) end of life tests Tests in the Lib/test dir type-bug An unexpected behavior, bug, or error

Comments

@xdegaye
Copy link
Mannequin

xdegaye mannequin commented May 3, 2016

BPO 26926
Nosy @benjaminp, @xdegaye, @serhiy-storchaka, @moreati
Files
  • large_file.patch
  • skip-large-file.patch
  • skip-large-file_2.patch
  • Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = 'https://github.com/xdegaye'
    closed_at = <Date 2016-11-17.10:21:14.019>
    created_at = <Date 2016-05-03.13:55:37.113>
    labels = ['3.7', 'type-bug', 'tests']
    title = 'test_io large file test failure on 32 bits Android platforms'
    updated_at = <Date 2016-11-17.10:21:14.019>
    user = 'https://github.com/xdegaye'

    bugs.python.org fields:

    activity = <Date 2016-11-17.10:21:14.019>
    actor = 'xdegaye'
    assignee = 'xdegaye'
    closed = True
    closed_date = <Date 2016-11-17.10:21:14.019>
    closer = 'xdegaye'
    components = ['Tests']
    creation = <Date 2016-05-03.13:55:37.113>
    creator = 'xdegaye'
    dependencies = []
    files = ['42720', '42941', '45350']
    hgrepos = []
    issue_num = 26926
    keywords = ['patch']
    message_count = 12.0
    messages = ['264725', '264770', '264798', '264799', '264849', '264893', '264895', '264899', '266063', '266064', '280053', '281027']
    nosy_count = 6.0
    nosy_names = ['benjamin.peterson', 'stutzbach', 'xdegaye', 'python-dev', 'serhiy.storchaka', 'Alex.Willmer']
    pr_nums = []
    priority = 'normal'
    resolution = 'fixed'
    stage = 'resolved'
    status = 'closed'
    superseder = None
    type = 'behavior'
    url = 'https://bugs.python.org/issue26926'
    versions = ['Python 3.6', 'Python 3.7']

    @xdegaye
    Copy link
    Mannequin Author

    xdegaye mannequin commented May 3, 2016

    test_io fails on an android emulator running an x86 system image at API level 21.

    ======================================================================
    ERROR: test_large_file_ops (test.test_io.CIOTest)
    ----------------------------------------------------------------------

    Traceback (most recent call last):
      File "/sdcard/org.bitbucket.pyona/lib/python3.6/test/test_io.py", line 537, in test_large_file_ops
        self.large_file_ops(f)
      File "/sdcard/org.bitbucket.pyona/lib/python3.6/test/test_io.py", line 330, in large_file_ops
        self.assertEqual(f.seek(self.LARGE), self.LARGE)
    OverflowError: Python int too large to convert to C long

    ======================================================================
    ERROR: test_large_file_ops (test.test_io.PyIOTest)
    ----------------------------------------------------------------------

    Traceback (most recent call last):
      File "/sdcard/org.bitbucket.pyona/lib/python3.6/test/test_io.py", line 537, in test_large_file_ops
        self.large_file_ops(f)
      File "/sdcard/org.bitbucket.pyona/lib/python3.6/test/test_io.py", line 330, in large_file_ops
        self.assertEqual(f.seek(self.LARGE), self.LARGE)
      File "/sdcard/org.bitbucket.pyona/lib/python3.6/_pyio.py", line 1632, in seek
        return os.lseek(self._fd, pos, whence)
    OverflowError: Python int too large to convert to C long

    Ran 541 tests in 32.829s

    FAILED (errors=2, skipped=8)
    test test_io failed
    test_io took 33 sec
    1 test failed:
    test_io
    Total duration: 0:00:33

    @xdegaye xdegaye mannequin added stdlib Python modules in the Lib dir build The build process and cross-build type-bug An unexpected behavior, bug, or error labels May 3, 2016
    @serhiy-storchaka
    Copy link
    Member

    This looks similar to bpo-11184.

    @serhiy-storchaka serhiy-storchaka changed the title android: test_io fails Large files are not supported on Android May 4, 2016
    @xdegaye
    Copy link
    Mannequin Author

    xdegaye mannequin commented May 4, 2016

    The relevant part of the output of configure on x86:

    checking size of int... 4
    checking size of long... 4
    checking size of void *... 4
    checking size of short... 2
    checking size of float... 4
    checking size of double... 8
    checking size of fpos_t... 4
    checking size of size_t... 4
    checking size of pid_t... 4
    checking for long long support... yes
    checking size of long long... 8
    checking for long double support... yes
    checking size of long double... 8
    checking for _Bool support... yes
    checking size of _Bool... 1
    checking for uintptr_t... yes
    checking size of uintptr_t... 4
    checking size of off_t... 4
    checking whether to enable large file support... no

    And as a consequence in pyconfig.h:
    /* Defined to enable large file support when an off_t is bigger than a long
    and long long is available and at least as big as an off_t. You may need to
    add some flags for configuration and compilation to enable this mode. (For
    Solaris and Linux, the necessary defines are already defined.) */
    /* #undef HAVE_LARGEFILE_SUPPORT */

    @xdegaye
    Copy link
    Mannequin Author

    xdegaye mannequin commented May 4, 2016

    On android x86:

    >>> import resource
    >>> resource.setrlimit(resource.RLIMIT_FSIZE, (2**31, 2**31))
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
    OverflowError: Python int too large to convert to C long
    >>> resource.setrlimit(resource.RLIMIT_FSIZE, (2**31-1, 2**31-1))
    >>> resource.getrlimit(resource.RLIMIT_FSIZE)
    (2147483647, 2147483647)
    >>> 

    The test_io tests that fail may be skipped by trying first to set this resource limit (if existing) to the max file size tested, in a spawned interpreter.

    @xdegaye
    Copy link
    Mannequin Author

    xdegaye mannequin commented May 4, 2016

    Here is a patch that uses resource.setrlimit() to check whether large file may not be supported.

    @serhiy-storchaka
    Copy link
    Member

    The resource module is not always available. And resource.setrlimit() itself can fail.

    May be check HAVE_LARGEFILE_SUPPORT directly using sysconfig.get_config_vars()?

    test_mmap needs similar fix.

    @serhiy-storchaka
    Copy link
    Member

    Skipping test is the one way.

    Other way is to enable large files support on Android (like it was done in bpo-11184 for AIX). It is hard to believe that Android's libc doesn't support files larger 2GiB. Should be an option to enable this support in a program.

    @xdegaye
    Copy link
    Mannequin Author

    xdegaye mannequin commented May 5, 2016

    Using the glibc compilation flags recommended for large file support '-D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE=1 -D_LARGEFILE64_SOURCE=1' does not enable large file support. See Suse's 'Large File Support in Linux' document [1].

    In the android bpo-64613 [2] one of the last comment said, only few days ago, "fixed in the platform. (partially in L, mostly in M, completely in N.)", L standing for Lollipop (API 21 or 22) and M for Marshmallow (API 23). So it seems these tests should be skipped until platform N is available.

    [1] http://users.suse.com/~aj/linux_lfs.html
    [2] https://code.google.com/p/android/issues/detail?id=64613

    @xdegaye
    Copy link
    Mannequin Author

    xdegaye mannequin commented May 22, 2016

    Size of off_t is also 4 when building on API 23 with or without
    '-D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE=1 -D_LARGEFILE64_SOURCE=1'
    added to the CC envt variable.

    @xdegaye
    Copy link
    Mannequin Author

    xdegaye mannequin commented May 22, 2016

    This patch skips the large file support test.

    @xdegaye
    Copy link
    Mannequin Author

    xdegaye mannequin commented Nov 4, 2016

    The tests are run on an ext4 file system.
    test_large_file_ops does not fail on the x86_64 and arm64 (aka aarch64) Android 64 bits platforms.
    test_large_file_ops still fails on x86 with the latest Android API level 24 (i.e. the latest released libc).

    FWIW, the second item in https://android.googlesource.com/platform/bionic.git/#32_bit-ABI-bugs may explain why there is still no large file support on Android 32 bits platforms.
    HAVE_LARGEFILE_SUPPORT cannot be used to skip the test since it is also undefined on the Android 64 bits platforms and on linux x86_64.
    This new patch is not specific to Android and uses the same method as the one used in test_mmap to skip the test on platforms that do not support large files.

    @xdegaye xdegaye mannequin added 3.7 (EOL) end of life tests Tests in the Lib/test dir and removed stdlib Python modules in the Lib dir build The build process and cross-build labels Nov 4, 2016
    @xdegaye xdegaye mannequin changed the title Large files are not supported on Android test_io large file test failure on 32 bits Android platforms Nov 4, 2016
    @xdegaye xdegaye mannequin self-assigned this Nov 4, 2016
    @python-dev
    Copy link
    Mannequin

    python-dev mannequin commented Nov 17, 2016

    New changeset d616304b82aa by Xavier de Gaye in branch '3.6':
    Issue bpo-26926: Skip some test_io tests on platforms without large file support
    https://hg.python.org/cpython/rev/d616304b82aa

    New changeset 878f91b4ad19 by Xavier de Gaye in branch 'default':
    Issue bpo-26926: Merge 3.6
    https://hg.python.org/cpython/rev/878f91b4ad19

    @xdegaye xdegaye mannequin closed this as completed Nov 17, 2016
    @ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
    Labels
    3.7 (EOL) end of life tests Tests in the Lib/test dir type-bug An unexpected behavior, bug, or error
    Projects
    None yet
    Development

    No branches or pull requests

    1 participant