classification
Title: Broken large file support on AIX
Type: Stage: patch review
Components: IO Versions: Python 2.7
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: georg.brandl Nosy List: georg.brandl, pitrou, rhettinger, sable
Priority: normal Keywords: patch

Created on 2011-02-11 11:37 by sable, last changed 2011-02-25 11:21 by georg.brandl. This issue is now closed.

Files
File name Uploaded Description Edit
patch_aix_largefile.diff sable, 2011-02-16 10:16
Messages (19)
msg128374 - (view) Author: Sébastien Sablé (sable) Date: 2011-02-11 11:37
I get 2 errors when running test_io.py with trunk on AIX. 

======================================================================
ERROR: test_large_file_ops (__main__.CIOTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "./Lib/test/test_io.py", line 418, in test_large_file_ops
    self.large_file_ops(f)
  File "./Lib/test/test_io.py", line 321, in large_file_ops
    self.assertEqual(f.seek(self.LARGE), self.LARGE)
OverflowError: Python int too large to convert to C long

======================================================================
ERROR: test_large_file_ops (__main__.PyIOTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "./Lib/test/test_io.py", line 418, in test_large_file_ops
    self.large_file_ops(f)
  File "./Lib/test/test_io.py", line 321, in large_file_ops
    self.assertEqual(f.seek(self.LARGE), self.LARGE)
OverflowError: Python int too large to convert to C long

----------------------------------------------------------------------
Ran 395 tests in 27.958s

FAILED (errors=2, skipped=8)

thanks in advance
msg128376 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2011-02-11 12:00
Hmm, strange. Is it a 32-bit build? Is HAVE_LARGEFILE_SUPPORT defined in pyconfig.h?
msg128377 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2011-02-11 12:14
Apparently AIX needs a specific #define to enable large file support:
http://publib.boulder.ibm.com/infocenter/pseries/v5r3/index.jsp?topic=/com.ibm.aix.genprogc/doc/genprogc/prg_lrg_files.htm

Python defines _LARGEFILE_SOURCE by default.
msg128385 - (view) Author: Sébastien Sablé (sable) Date: 2011-02-11 13:16
OK, so the following patch should help:

Index: configure.in
===================================================================
--- configure.in        (revision 88393)
+++ configure.in        (working copy)
@@ -1375,6 +1375,8 @@
 
 if test "$use_lfs" = "yes"; then
 # Two defines needed to enable largefile support on various platforms
+AC_DEFINE(_LARGEFILES, 1, 
+[This must be defined on some systems to enable large file support.])
 # These may affect some typedefs
 AC_DEFINE(_LARGEFILE_SOURCE, 1, 
 [This must be defined on some systems to enable large file support.])
msg128386 - (view) Author: Sébastien Sablé (sable) Date: 2011-02-11 13:39
the error is different now that _LARGEFILES is defined:

======================================================================
ERROR: test_large_file_ops (__main__.CIOTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "./Lib/test/test_io.py", line 418, in test_large_file_ops
    self.large_file_ops(f)
  File "./Lib/test/test_io.py", line 321, in large_file_ops
    self.assertEqual(f.seek(self.LARGE), self.LARGE)
IOError: [Errno 22] Invalid argument

======================================================================
ERROR: test_large_file_ops (__main__.PyIOTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "./Lib/test/test_io.py", line 418, in test_large_file_ops
    self.large_file_ops(f)
  File "./Lib/test/test_io.py", line 321, in large_file_ops
    self.assertEqual(f.seek(self.LARGE), self.LARGE)
IOError: [Errno 22] Invalid argument
msg128387 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2011-02-11 13:53
Hmm, interesting. Can you post the results of the two following snippets:

>>> f = open('foo', 'wb')
>>> f.seek(2**32)
# should be 4294967296

>>> f = open('foo', 'wb')
>>> f.truncate(2**32)
# should be 4294967296
msg128388 - (view) Author: Sébastien Sablé (sable) Date: 2011-02-11 14:06
Sorry I made a mistake in my previous patch (_LARGEFILES instead of _LARGE_FILES).

Here is a better one:

Index: configure.in
===================================================================
--- configure.in	(révision 88395)
+++ configure.in	(copie de travail)
@@ -1376,6 +1376,14 @@
 if test "$use_lfs" = "yes"; then
 # Two defines needed to enable largefile support on various platforms
 # These may affect some typedefs
+    case $ac_sys_system/$ac_sys_release in
+    AIX*)
+        AC_DEFINE(_LARGE_FILES, 1, 
+        [This must be defined on AIX systems to enable large file support.])
+	;;
+    *)
+        ;;
+    esac
 AC_DEFINE(_LARGEFILE_SOURCE, 1, 
 [This must be defined on some systems to enable large file support.])
 AC_DEFINE(_FILE_OFFSET_BITS, 64,

The test fails in a different way now:


======================================================================
ERROR: test_large_file_ops (__main__.CIOTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "./Lib/test/test_io.py", line 418, in test_large_file_ops
    self.large_file_ops(f)
  File "./Lib/test/test_io.py", line 323, in large_file_ops
    self.assertEqual(f.write(b"xxx"), 3)
IOError: [Errno 27] File too large

======================================================================
ERROR: test_large_file_ops (__main__.PyIOTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "./Lib/test/test_io.py", line 418, in test_large_file_ops
    self.large_file_ops(f)
  File "./Lib/test/test_io.py", line 323, in large_file_ops
    self.assertEqual(f.write(b"xxx"), 3)
IOError: [Errno 27] File too large

----------------------------------------------------------------------
Ran 395 tests in 27.958s


Here is your trace:
phenix:~/.buildbot/python-aix6/3.x.phenix.xlc/build\> ./python 
Python 3.2rc2+ (py3k:88393M, Feb 11 2011, 14:56:34) [C] on aix6
Type "help", "copyright", "credits" or "license" for more information.
>>> f = open('foo', 'wb')
[55983 refs]
>>> f.seek(2**32)
4294967296
[55987 refs]
>>> f = open('foo', 'wb')
__main__:1: ResourceWarning: unclosed file <_io.BufferedWriter name='foo'>
[55994 refs]
>>> f.truncate(2**32)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
IOError: [Errno 27] File too large
[56027 refs]
msg128390 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2011-02-11 14:19
Thanks for the patch.

> The test fails in a different way now:
> [...]
> IOError: [Errno 27] File too large

This seems to mean that your file system isn't configured for large
files. According to
http://publib.boulder.ibm.com/infocenter/pseries/v5r3/index.jsp?topic=/com.ibm.aix.genprogc/doc/genprogc/prg_lrg_files.htm :

        For the JFS, the maximum file size is determined by the
        parameters used at the time the file system was made. For JFS
        file systems that are enabled for large files, the maximum file
        size is slightly less than 64 gigabytes (0xff8400000). For all
        other JFS file systems, the maximum file size is 2Gb-1
        (0x7fffffff). Attempts to write a file in excess of the maximum
        file size in any file system format will fail, and errno will be
        set to EFBIG.

(I'm not under AIX, but EFBIG is 27 here)

What does test_largefile output?
msg128391 - (view) Author: Sébastien Sablé (sable) Date: 2011-02-11 14:30
test_largefile complains about the filesystem having no largefile support.
It is probably the case, I will ask a sysadmin, and see if he can get me a file system with large file support so that I can test this feature.


> ./python -Wd -E -bb ./Lib/test/test_largefile.py 
Traceback (most recent call last):
  File "./Lib/test/test_largefile.py", line 163, in test_main
    f.write(b'x')
IOError: [Errno 27] File too large

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "./Lib/test/test_largefile.py", line 192, in <module>
    test_main()
  File "./Lib/test/test_largefile.py", line 168, in test_main
    raise unittest.SkipTest("filesystem does not have largefile support")
unittest.case.SkipTest: filesystem does not have largefile support
[81125 refs]

In the meantime, this test should probably be skipped just like in test_largefile.py and the patch with _LARGE_FILES should probably be applied too.

Merci pour l'aide
msg128392 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2011-02-11 14:36
Yes, I think the skipping code in test_largefile should be factored out and used both in test_io and test_largefile (to be honest I don't know why test_io has large file tests as well; perhaps I should merge them together).
msg128394 - (view) Author: Sébastien Sablé (sable) Date: 2011-02-11 14:40
Also:

is it OK if I open a new issue for each broken unit test on AIX even if I have not investigated them at the moment?

I have a dozen broken unit tests on AIX that need to be investigated, but I don't want to spam the bug tracker followers too much.
So tell me if it is a good thing to report each problem as soon as possible or if I should do it step by step.
msg128395 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2011-02-11 14:46
> is it OK if I open a new issue for each broken unit test on AIX even
> if I have not investigated them at the moment?

Yes. That way they get recorded somewhere and other people can chime in.
If you plan to investigate them you can add a sentence saying so.

Thanks!
msg128643 - (view) Author: Sébastien Sablé (sable) Date: 2011-02-16 10:16
Here is the patch.
It only impacts AIX systems and is minimalist, so I think it should be safe for Python 3.2.
msg128682 - (view) Author: Georg Brandl (georg.brandl) * (Python committer) Date: 2011-02-16 17:43
Antoine, do you agree?  I don't want waves of AIX changes going into 3.2 now...
msg128683 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2011-02-16 17:49
Assuming it doesn't break other platforms, I'm fine with it.
msg128709 - (view) Author: Raymond Hettinger (rhettinger) * (Python committer) Date: 2011-02-17 01:59
This looks to be a low risk fix-up (confined to an "if $use_lfs" block in configure.in and further guarded with a case statement restricting it to AIX builds).
msg128836 - (view) Author: Georg Brandl (georg.brandl) * (Python committer) Date: 2011-02-19 08:58
Okay, committed to py3k in r88440.  Does this need backporting?
msg128837 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2011-02-19 10:00
> Okay, committed to py3k in r88440.  Does this need backporting?

Certainly.
msg129356 - (view) Author: Georg Brandl (georg.brandl) * (Python committer) Date: 2011-02-25 11:21
Backported to 3.1 in r88562, 2.7 in r88569.
History
Date User Action Args
2011-02-25 11:21:10georg.brandlsetstatus: open -> closed
nosy: georg.brandl, rhettinger, pitrou, sable
messages: + msg129356
2011-02-21 11:33:51pitrousetassignee: georg.brandl
nosy: georg.brandl, rhettinger, pitrou, sable
versions: + Python 2.7, - Python 3.2
2011-02-19 10:00:27pitrousetnosy: georg.brandl, rhettinger, pitrou, sable
messages: + msg128837
2011-02-19 08:58:34georg.brandlsetresolution: fixed
messages: + msg128836
nosy: georg.brandl, rhettinger, pitrou, sable
2011-02-17 01:59:04rhettingersetnosy: + rhettinger
messages: + msg128709
2011-02-16 17:49:52pitrousetnosy: georg.brandl, pitrou, sable
messages: + msg128683
2011-02-16 17:43:15georg.brandlsetnosy: georg.brandl, pitrou, sable
messages: + msg128682
2011-02-16 15:49:27pitrousetnosy: + georg.brandl
2011-02-16 10:16:25sablesetfiles: + patch_aix_largefile.diff

messages: + msg128643
keywords: + patch
nosy: pitrou, sable
2011-02-11 14:46:07pitrousetnosy: pitrou, sable
messages: + msg128395
2011-02-11 14:40:13sablesetnosy: pitrou, sable
messages: + msg128394
2011-02-11 14:36:27pitrousetnosy: pitrou, sable
messages: + msg128392
stage: patch review
2011-02-11 14:31:28sablesetnosy: pitrou, sable
title: test_io error on AIX -> Broken large file support on AIX
2011-02-11 14:30:09sablesetnosy: pitrou, sable
messages: + msg128391
2011-02-11 14:19:40pitrousetnosy: pitrou, sable
messages: + msg128390
2011-02-11 14:06:01sablesetnosy: pitrou, sable
messages: + msg128388
2011-02-11 13:53:46pitrousetnosy: pitrou, sable
messages: + msg128387
2011-02-11 13:39:39sablesetnosy: pitrou, sable
messages: + msg128386
2011-02-11 13:16:18sablesetnosy: pitrou, sable
messages: + msg128385
2011-02-11 12:14:03pitrousetnosy: pitrou, sable
messages: + msg128377
2011-02-11 12:00:05pitrousetnosy: + pitrou
messages: + msg128376
2011-02-11 11:37:51sablecreate