Broken large file support on AIX #55393

sable · 2011-02-11T11:37:52Z

BPO	11184
Nosy	@birkenfeld, @rhettinger, @pitrou
Files	patch_aix_largefile.diff

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/birkenfeld'
closed_at = <Date 2011-02-25.11:21:10.166>
created_at = <Date 2011-02-11.11:37:51.690>
labels = ['expert-IO']
title = 'Broken large file support on AIX'
updated_at = <Date 2011-02-25.11:21:10.165>
user = 'https://bugs.python.org/sable'

bugs.python.org fields:

activity = <Date 2011-02-25.11:21:10.165>
actor = 'georg.brandl'
assignee = 'georg.brandl'
closed = True
closed_date = <Date 2011-02-25.11:21:10.166>
closer = 'georg.brandl'
components = ['IO']
creation = <Date 2011-02-11.11:37:51.690>
creator = 'sable'
dependencies = []
files = ['20767']
hgrepos = []
issue_num = 11184
keywords = ['patch']
message_count = 19.0
messages = ['128374', '128376', '128377', '128385', '128386', '128387', '128388', '128390', '128391', '128392', '128394', '128395', '128643', '128682', '128683', '128709', '128836', '128837', '129356']
nosy_count = 4.0
nosy_names = ['georg.brandl', 'rhettinger', 'pitrou', 'sable']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = 'patch review'
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue11184'
versions = ['Python 2.7']

sable · 2011-02-11T11:37:51Z

I get 2 errors when running test_io.py with trunk on AIX.

======================================================================
ERROR: test_large_file_ops (main.CIOTest)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "./Lib/test/test_io.py", line 418, in test_large_file_ops
    self.large_file_ops(f)
  File "./Lib/test/test_io.py", line 321, in large_file_ops
    self.assertEqual(f.seek(self.LARGE), self.LARGE)
OverflowError: Python int too large to convert to C long

======================================================================
ERROR: test_large_file_ops (main.PyIOTest)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "./Lib/test/test_io.py", line 418, in test_large_file_ops
    self.large_file_ops(f)
  File "./Lib/test/test_io.py", line 321, in large_file_ops
    self.assertEqual(f.seek(self.LARGE), self.LARGE)
OverflowError: Python int too large to convert to C long

Ran 395 tests in 27.958s

FAILED (errors=2, skipped=8)

thanks in advance

pitrou · 2011-02-11T12:00:05Z

Hmm, strange. Is it a 32-bit build? Is HAVE_LARGEFILE_SUPPORT defined in pyconfig.h?

pitrou · 2011-02-11T12:14:03Z

Apparently AIX needs a specific #define to enable large file support:
http://publib.boulder.ibm.com/infocenter/pseries/v5r3/index.jsp?topic=/com.ibm.aix.genprogc/doc/genprogc/prg_lrg_files.htm

Python defines _LARGEFILE_SOURCE by default.

sable · 2011-02-11T13:16:18Z

OK, so the following patch should help:

Index: configure.in
===================================================================

--- configure.in        (revision 88393)
+++ configure.in        (working copy)
@@ -1375,6 +1375,8 @@
 
 if test "$use_lfs" = "yes"; then
 # Two defines needed to enable largefile support on various platforms
+AC_DEFINE(_LARGEFILES, 1, 
+[This must be defined on some systems to enable large file support.])
 # These may affect some typedefs
 AC_DEFINE(_LARGEFILE_SOURCE, 1, 
 [This must be defined on some systems to enable large file support.])

sable · 2011-02-11T13:39:39Z

the error is different now that _LARGEFILES is defined:

======================================================================
ERROR: test_large_file_ops (main.CIOTest)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "./Lib/test/test_io.py", line 418, in test_large_file_ops
    self.large_file_ops(f)
  File "./Lib/test/test_io.py", line 321, in large_file_ops
    self.assertEqual(f.seek(self.LARGE), self.LARGE)
IOError: [Errno 22] Invalid argument

======================================================================
ERROR: test_large_file_ops (main.PyIOTest)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "./Lib/test/test_io.py", line 418, in test_large_file_ops
    self.large_file_ops(f)
  File "./Lib/test/test_io.py", line 321, in large_file_ops
    self.assertEqual(f.seek(self.LARGE), self.LARGE)
IOError: [Errno 22] Invalid argument

pitrou · 2011-02-11T13:53:46Z

Hmm, interesting. Can you post the results of the two following snippets:

>>> f = open('foo', 'wb')
>>> f.seek(2**32)
# should be 4294967296

>>> f = open('foo', 'wb')
>>> f.truncate(2**32)
# should be 4294967296

sable · 2011-02-11T14:06:01Z

Sorry I made a mistake in my previous patch (_LARGEFILES instead of _LARGE_FILES).

Here is a better one:

Index: configure.in
===================================================================

--- configure.in	(révision 88395)
+++ configure.in	(copie de travail)
@@ -1376,6 +1376,14 @@
 if test "$use_lfs" = "yes"; then
 # Two defines needed to enable largefile support on various platforms
 # These may affect some typedefs
+    case $ac_sys_system/$ac_sys_release in
+    AIX*)
+        AC_DEFINE(_LARGE_FILES, 1, 
+        [This must be defined on AIX systems to enable large file support.])
+	;;
+    *)
+        ;;
+    esac
 AC_DEFINE(_LARGEFILE_SOURCE, 1, 
 [This must be defined on some systems to enable large file support.])
 AC_DEFINE(_FILE_OFFSET_BITS, 64,

The test fails in a different way now:

======================================================================
ERROR: test_large_file_ops (main.CIOTest)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "./Lib/test/test_io.py", line 418, in test_large_file_ops
    self.large_file_ops(f)
  File "./Lib/test/test_io.py", line 323, in large_file_ops
    self.assertEqual(f.write(b"xxx"), 3)
IOError: [Errno 27] File too large

======================================================================
ERROR: test_large_file_ops (main.PyIOTest)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "./Lib/test/test_io.py", line 418, in test_large_file_ops
    self.large_file_ops(f)
  File "./Lib/test/test_io.py", line 323, in large_file_ops
    self.assertEqual(f.write(b"xxx"), 3)
IOError: [Errno 27] File too large

Ran 395 tests in 27.958s

Here is your trace:
phenix:~/.buildbot/python-aix6/3.x.phenix.xlc/build\> ./python 
Python 3.2rc2+ (py3k:88393M, Feb 11 2011, 14:56:34) [C] on aix6
Type "help", "copyright", "credits" or "license" for more information.
>>> f = open('foo', 'wb')
[55983 refs]
>>> f.seek(2**32)
4294967296
[55987 refs]
>>> f = open('foo', 'wb')
__main__:1: ResourceWarning: unclosed file <_io.BufferedWriter name='foo'>
[55994 refs]
>>> f.truncate(2**32)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
IOError: [Errno 27] File too large
[56027 refs]

pitrou · 2011-02-11T14:19:40Z

Thanks for the patch.

The test fails in a different way now:
[...]
IOError: [Errno 27] File too large

This seems to mean that your file system isn't configured for large
files. According to
http://publib.boulder.ibm.com/infocenter/pseries/v5r3/index.jsp?topic=/com.ibm.aix.genprogc/doc/genprogc/prg_lrg_files.htm :

    For the JFS, the maximum file size is determined by the
    parameters used at the time the file system was made. For JFS
    file systems that are enabled for large files, the maximum file
    size is slightly less than 64 gigabytes (0xff8400000). For all
    other JFS file systems, the maximum file size is 2Gb-1
    (0x7fffffff). Attempts to write a file in excess of the maximum
    file size in any file system format will fail, and errno will be
    set to EFBIG.

(I'm not under AIX, but EFBIG is 27 here)

What does test_largefile output?

sable · 2011-02-11T14:30:09Z

test_largefile complains about the filesystem having no largefile support.
It is probably the case, I will ask a sysadmin, and see if he can get me a file system with large file support so that I can test this feature.

> ./python -Wd -E -bb ./Lib/test/test_largefile.py 
Traceback (most recent call last):
  File "./Lib/test/test_largefile.py", line 163, in test_main
    f.write(b'x')
IOError: [Errno 27] File too large

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "./Lib/test/test_largefile.py", line 192, in <module>
    test_main()
  File "./Lib/test/test_largefile.py", line 168, in test_main
    raise unittest.SkipTest("filesystem does not have largefile support")
unittest.case.SkipTest: filesystem does not have largefile support
[81125 refs]

In the meantime, this test should probably be skipped just like in test_largefile.py and the patch with _LARGE_FILES should probably be applied too.

Merci pour l'aide

pitrou · 2011-02-11T14:36:28Z

Yes, I think the skipping code in test_largefile should be factored out and used both in test_io and test_largefile (to be honest I don't know why test_io has large file tests as well; perhaps I should merge them together).

sable · 2011-02-11T14:40:13Z

Also:

is it OK if I open a new issue for each broken unit test on AIX even if I have not investigated them at the moment?

I have a dozen broken unit tests on AIX that need to be investigated, but I don't want to spam the bug tracker followers too much.
So tell me if it is a good thing to report each problem as soon as possible or if I should do it step by step.

pitrou · 2011-02-11T14:46:07Z

is it OK if I open a new issue for each broken unit test on AIX even
if I have not investigated them at the moment?

Yes. That way they get recorded somewhere and other people can chime in.
If you plan to investigate them you can add a sentence saying so.

Thanks!

sable · 2011-02-16T10:16:25Z

Here is the patch.
It only impacts AIX systems and is minimalist, so I think it should be safe for Python 3.2.

birkenfeld · 2011-02-16T17:43:16Z

Antoine, do you agree? I don't want waves of AIX changes going into 3.2 now...

pitrou · 2011-02-16T17:49:53Z

Assuming it doesn't break other platforms, I'm fine with it.

rhettinger · 2011-02-17T01:59:04Z

This looks to be a low risk fix-up (confined to an "if $use_lfs" block in configure.in and further guarded with a case statement restricting it to AIX builds).

birkenfeld · 2011-02-19T08:58:34Z

Okay, committed to py3k in r88440. Does this need backporting?

pitrou · 2011-02-19T10:00:27Z

Okay, committed to py3k in r88440. Does this need backporting?

Certainly.

birkenfeld · 2011-02-25T11:21:10Z

Backported to 3.1 in r88562, 2.7 in r88569.

sable mannequin added the topic-IO label Feb 11, 2011

sable mannequin changed the title ~~test_io error on AIX~~ Broken large file support on AIX Feb 11, 2011

pitrou assigned birkenfeld Feb 21, 2011

birkenfeld closed this as completed Feb 25, 2011

ezio-melotti transferred this issue from another repository Apr 10, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Broken large file support on AIX #55393

Broken large file support on AIX #55393

sable mannequin commented Feb 11, 2011

sable mannequin commented Feb 11, 2011

pitrou commented Feb 11, 2011

pitrou commented Feb 11, 2011

sable mannequin commented Feb 11, 2011

sable mannequin commented Feb 11, 2011

pitrou commented Feb 11, 2011

sable mannequin commented Feb 11, 2011

pitrou commented Feb 11, 2011

sable mannequin commented Feb 11, 2011

pitrou commented Feb 11, 2011

sable mannequin commented Feb 11, 2011

pitrou commented Feb 11, 2011

sable mannequin commented Feb 16, 2011

birkenfeld commented Feb 16, 2011

pitrou commented Feb 16, 2011

rhettinger commented Feb 17, 2011

birkenfeld commented Feb 19, 2011

pitrou commented Feb 19, 2011

birkenfeld commented Feb 25, 2011

Broken large file support on AIX #55393

Broken large file support on AIX #55393

Comments

sable mannequin commented Feb 11, 2011

sable mannequin commented Feb 11, 2011

pitrou commented Feb 11, 2011

pitrou commented Feb 11, 2011

sable mannequin commented Feb 11, 2011

sable mannequin commented Feb 11, 2011

pitrou commented Feb 11, 2011

sable mannequin commented Feb 11, 2011

pitrou commented Feb 11, 2011

sable mannequin commented Feb 11, 2011

pitrou commented Feb 11, 2011

sable mannequin commented Feb 11, 2011

pitrou commented Feb 11, 2011

sable mannequin commented Feb 16, 2011

birkenfeld commented Feb 16, 2011

pitrou commented Feb 16, 2011

rhettinger commented Feb 17, 2011

birkenfeld commented Feb 19, 2011

pitrou commented Feb 19, 2011

birkenfeld commented Feb 25, 2011