classification
Title: test_io large file test failure on 32 bits Android platforms
Type: behavior Stage: resolved
Components: Tests Versions: Python 3.7, Python 3.6
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: xdegaye Nosy List: Alex.Willmer, benjamin.peterson, python-dev, serhiy.storchaka, stutzbach, xdegaye
Priority: normal Keywords: patch

Created on 2016-05-03 13:55 by xdegaye, last changed 2016-11-17 10:21 by xdegaye. This issue is now closed.

Files
File name Uploaded Description Edit
large_file.patch xdegaye, 2016-05-04 19:14 review
skip-large-file.patch xdegaye, 2016-05-22 09:44 review
skip-large-file_2.patch xdegaye, 2016-11-04 15:47 review
Messages (12)
msg264725 - (view) Author: Xavier de Gaye (xdegaye) * (Python committer) Date: 2016-05-03 13:55
test_io fails on an android emulator running an x86 system image at API level 21.

======================================================================
ERROR: test_large_file_ops (test.test_io.CIOTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/sdcard/org.bitbucket.pyona/lib/python3.6/test/test_io.py", line 537, in test_large_file_ops
    self.large_file_ops(f)
  File "/sdcard/org.bitbucket.pyona/lib/python3.6/test/test_io.py", line 330, in large_file_ops
    self.assertEqual(f.seek(self.LARGE), self.LARGE)
OverflowError: Python int too large to convert to C long

======================================================================
ERROR: test_large_file_ops (test.test_io.PyIOTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/sdcard/org.bitbucket.pyona/lib/python3.6/test/test_io.py", line 537, in test_large_file_ops
    self.large_file_ops(f)
  File "/sdcard/org.bitbucket.pyona/lib/python3.6/test/test_io.py", line 330, in large_file_ops
    self.assertEqual(f.seek(self.LARGE), self.LARGE)
  File "/sdcard/org.bitbucket.pyona/lib/python3.6/_pyio.py", line 1632, in seek
    return os.lseek(self._fd, pos, whence)
OverflowError: Python int too large to convert to C long

----------------------------------------------------------------------
Ran 541 tests in 32.829s

FAILED (errors=2, skipped=8)
test test_io failed
test_io took 33 sec
1 test failed:
    test_io
Total duration: 0:00:33
msg264770 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2016-05-03 21:32
This looks similar to issue11184.
msg264798 - (view) Author: Xavier de Gaye (xdegaye) * (Python committer) Date: 2016-05-04 09:37
The relevant part of the output of configure on x86:

checking size of int... 4
checking size of long... 4
checking size of void *... 4
checking size of short... 2
checking size of float... 4
checking size of double... 8
checking size of fpos_t... 4
checking size of size_t... 4
checking size of pid_t... 4
checking for long long support... yes
checking size of long long... 8
checking for long double support... yes
checking size of long double... 8
checking for _Bool support... yes
checking size of _Bool... 1
checking for uintptr_t... yes
checking size of uintptr_t... 4
checking size of off_t... 4
checking whether to enable large file support... no

And as a consequence in pyconfig.h:
/* Defined to enable large file support when an off_t is bigger than a long
   and long long is available and at least as big as an off_t. You may need to
   add some flags for configuration and compilation to enable this mode. (For
   Solaris and Linux, the necessary defines are already defined.) */
/* #undef HAVE_LARGEFILE_SUPPORT */
msg264799 - (view) Author: Xavier de Gaye (xdegaye) * (Python committer) Date: 2016-05-04 10:00
On android x86:

>>> import resource
>>> resource.setrlimit(resource.RLIMIT_FSIZE, (2**31, 2**31))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
OverflowError: Python int too large to convert to C long
>>> resource.setrlimit(resource.RLIMIT_FSIZE, (2**31-1, 2**31-1))
>>> resource.getrlimit(resource.RLIMIT_FSIZE)
(2147483647, 2147483647)
>>> 


The test_io tests that fail may be skipped by trying first to set this resource limit (if existing) to the max file size tested, in a spawned interpreter.
msg264849 - (view) Author: Xavier de Gaye (xdegaye) * (Python committer) Date: 2016-05-04 19:14
Here is a patch that uses resource.setrlimit() to check whether large file may not be supported.
msg264893 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2016-05-05 08:18
The resource module is not always available. And resource.setrlimit() itself can fail.

May be check HAVE_LARGEFILE_SUPPORT directly using sysconfig.get_config_vars()?

test_mmap needs similar fix.
msg264895 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2016-05-05 08:21
Skipping test is the one way.

Other way is to enable large files support on Android (like it was done in issue11184 for AIX). It is hard to believe that Android's libc doesn't support files larger 2GiB. Should be an option to enable this support in a program.
msg264899 - (view) Author: Xavier de Gaye (xdegaye) * (Python committer) Date: 2016-05-05 09:10
Using the glibc compilation flags recommended for large file support '-D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE=1 -D_LARGEFILE64_SOURCE=1' does not enable large file support. See Suse's 'Large File Support in Linux' document [1].

In the android issue 64613 [2] one of the last comment said, only few days ago,  "fixed in the platform. (partially in L, mostly in M, completely in N.)", L standing for Lollipop (API 21 or 22) and M for Marshmallow (API 23). So it seems these tests should be skipped until platform N is available.

[1] http://users.suse.com/~aj/linux_lfs.html
[2] https://code.google.com/p/android/issues/detail?id=64613
msg266063 - (view) Author: Xavier de Gaye (xdegaye) * (Python committer) Date: 2016-05-22 09:28
Size of off_t is also 4 when building on API 23 with or without
'-D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE=1 -D_LARGEFILE64_SOURCE=1'
added to the CC envt variable.
msg266064 - (view) Author: Xavier de Gaye (xdegaye) * (Python committer) Date: 2016-05-22 09:44
This patch skips the large file support test.
msg280053 - (view) Author: Xavier de Gaye (xdegaye) * (Python committer) Date: 2016-11-04 15:47
The tests are run on an ext4 file system.
test_large_file_ops does not fail on the x86_64 and arm64 (aka aarch64) Android 64 bits platforms.
test_large_file_ops still fails on x86 with the latest Android API level 24 (i.e. the latest released libc).

FWIW, the second item in https://android.googlesource.com/platform/bionic.git/#32_bit-ABI-bugs may explain why there is still no large file support on Android 32 bits platforms.
HAVE_LARGEFILE_SUPPORT cannot be used to skip the test since it is also undefined on the Android 64 bits platforms and on linux x86_64.
This new patch is not specific to Android and uses the same method as the one used in test_mmap to skip the test on platforms that do not support large files.
msg281027 - (view) Author: Roundup Robot (python-dev) Date: 2016-11-17 08:23
New changeset d616304b82aa by Xavier de Gaye in branch '3.6':
Issue #26926: Skip some test_io tests on platforms without large file support
https://hg.python.org/cpython/rev/d616304b82aa

New changeset 878f91b4ad19 by Xavier de Gaye in branch 'default':
Issue #26926: Merge 3.6
https://hg.python.org/cpython/rev/878f91b4ad19
History
Date User Action Args
2016-11-17 10:21:14xdegayesetstatus: open -> closed
resolution: fixed
stage: commit review -> resolved
2016-11-17 08:23:18python-devsetnosy: + python-dev
messages: + msg281027
2016-11-16 06:51:56xdegayesetstage: patch review -> commit review
2016-11-04 15:47:23xdegayesetfiles: + skip-large-file_2.patch

dependencies: - add the 'is_android' attribute to test.support
assignee: xdegaye
components: + Tests, - Library (Lib), Cross-Build
title: Large files are not supported on Android -> test_io large file test failure on 32 bits Android platforms
versions: + Python 3.7
messages: + msg280053
stage: patch review
2016-05-22 09:44:27xdegayesetfiles: + skip-large-file.patch

dependencies: + add the 'is_android' attribute to test.support
messages: + msg266064
2016-05-22 09:28:11xdegayesetmessages: + msg266063
2016-05-21 07:06:39xdegayelinkissue26865 dependencies
2016-05-05 09:10:22xdegayesetmessages: + msg264899
2016-05-05 08:21:34serhiy.storchakasetmessages: + msg264895
2016-05-05 08:18:23serhiy.storchakasetmessages: + msg264893
2016-05-04 19:14:27xdegayesetfiles: + large_file.patch
keywords: + patch
messages: + msg264849
2016-05-04 10:00:09xdegayesetmessages: + msg264799
2016-05-04 09:37:16xdegayesetmessages: + msg264798
2016-05-04 06:04:36serhiy.storchakasettitle: android: test_io fails -> Large files are not supported on Android
2016-05-03 21:35:47serhiy.storchakalinkissue26927 superseder
2016-05-03 21:32:06serhiy.storchakasetnosy: + serhiy.storchaka
messages: + msg264770
2016-05-03 13:55:37xdegayecreate