test_zipfile fails on AIX due to time.localtime #83683

EGuesnet · 2020-01-30T13:51:38Z

BPO	39502
Nosy	@vstinner, @aixtools, @miss-islington, @EGuesnet
PRs	bpo-39502: Skip test_zipfile.test_add_file_after_2107() on AIX #18282 [3.8] bpo-39502: Skip test_zipfile.test_add_file_after_2107() on AIX (GH-18282) #18283 bpo-39502: Fix 64-bit Python PyTime_localtime() on AIX #18285
Files	test_zipfile.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2020-01-30.13:51:37.691>
labels = ['interpreter-core', 'type-bug', '3.8', '3.9', '3.7', 'tests']
title = 'test_zipfile fails on AIX due to time.localtime'
updated_at = <Date 2020-03-26.09:42:37.096>
user = 'https://github.com/EGuesnet'

bugs.python.org fields:

activity = <Date 2020-03-26.09:42:37.096>
actor = 'Michael.Felt'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Interpreter Core', 'Tests']
creation = <Date 2020-01-30.13:51:37.691>
creator = 'EGuesnet'
dependencies = []
files = ['48872']
hgrepos = []
issue_num = 39502
keywords = ['patch']
message_count = 21.0
messages = ['361058', '361059', '361061', '361062', '361063', '361064', '361065', '361066', '361067', '361068', '361069', '361070', '361071', '361116', '361118', '361272', '361303', '361323', '361603', '361604', '365056']
nosy_count = 5.0
nosy_names = ['vstinner', 'David.Edelsohn', 'Michael.Felt', 'miss-islington', 'EGuesnet']
pr_nums = ['18282', '18283', '18285']
priority = 'normal'
resolution = None
stage = 'patch review'
status = 'open'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue39502'
versions = ['Python 3.7', 'Python 3.8', 'Python 3.9']

EGuesnet · 2020-01-30T13:51:37Z

Hi,
I have an error during regression tests with Python3.8.1 on AIX 6.1 compiled with GCC 8.3.
It occurs only on 64 bit. Test passes on 32 bit.

======================================================================
ERROR: test_add_file_after_2107 (test.test_zipfile.StoredTestsWithSourceFile)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/opt/freeware/src/packages/BUILD/Python-3.8.1/64bit/Lib/test/test_zipfile.py", line 606, in test_add_file_after_2107
    self.assertRaises(struct.error, zipfp.write, TESTFN)
  File "/opt/freeware/src/packages/BUILD/Python-3.8.1/64bit/Lib/unittest/case.py", line 816, in assertRaises
    return context.handle('assertRaises', args, kwargs)
  File "/opt/freeware/src/packages/BUILD/Python-3.8.1/64bit/Lib/unittest/case.py", line 202, in handle
    callable_obj(*args, **kwargs)
  File "/opt/freeware/src/packages/BUILD/Python-3.8.1/64bit/Lib/zipfile.py", line 1739, in write
    zinfo = ZipInfo.from_file(filename, arcname,
  File "/opt/freeware/src/packages/BUILD/Python-3.8.1/64bit/Lib/zipfile.py", line 523, in from_file
    mtime = time.localtime(st.st_mtime)
OverflowError: localtime argument out of range

The PR associated to the new behavior is: #12726 (new on Python 3.8). Code is AIX specific. Is the code 32 bit only, or maybe the test was not updated?

-----

I can reproduce the behavior as follow:

$ python3.8_32
Python 3.8.1 (default, Jan 27 2020, 11:34:59) 
[GCC 8.3.0] on aix
Type "help", "copyright", "credits" or "license" for more information.
>>> import time
>>> time.localtime(4325562452)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
OverflowError: timestamp out of range for platform time_t

$ python3.8_64
Python 3.8.1 (default, Jan 27 2020, 11:30:15) 
[GCC 8.3.0] on aix
Type "help", "copyright", "credits" or "license" for more information.
>>> import time
>>> time.localtime(4325562452)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
OverflowError: localtime argument out of range

$ python3.7_32
Python 3.7.4 (default, Jan 15 2020, 15:50:53) 
[GCC 8.3.0] on aix6
Type "help", "copyright", "credits" or "license" for more information.
>>> import time
>>> time.localtime(4325562452)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
OverflowError: timestamp out of range for platform time_t

$ python3.7_64
Python 3.7.4 (default, Jan 15 2020, 15:46:22) 
[GCC 8.3.0] on aix6
Type "help", "copyright", "credits" or "license" for more information.
>>> import time
>>> time.localtime(4325562452)
time.struct_time(tm_year=2107, tm_mon=1, tm_mday=27, tm_hour=10, tm_min=7, tm_sec=32, tm_wday=3, tm_yday=27, tm_isdst=0)

vstinner · 2020-01-30T13:55:20Z

I suggest to skip the test if time.localtime() fails with OverflowError.

Can you please try attached patch on AIX, on the master branch if possible? (Or on Python 3.8 at least.)

EGuesnet · 2020-01-30T14:03:17Z

Tested on Python 3.8.1 on both 32 and 64 bits.
Test is now skipped.

test_add_file_after_2107 (test.test_zipfile.StoredTestsWithSourceFile) ... skipped 'time.localtime(4386268800) raises OverflowError'

vstinner · 2020-01-30T14:08:17Z

Tested on Python 3.8.1 on both 32 and 64 bits. Test is now skipped.

Good. I created PR 18282 from my patch.

aixtools · 2020-01-30T14:20:04Z

Probably this broke the 64-bit usage.

diff --git a/Python/pytime.c b/Python/pytime.c
index 54ddfc952b..6f13e62490 100644
--- a/Python/pytime.c
+++ b/Python/pytime.c
@@ -1059,7 +1059,7 @@ _PyTime_localtime(time_t t, struct tm *tm)
return 0;
#else /* !MS_WINDOWS */

-#ifdef _AIX
+#ifdef defined(_AIX) and (SIZEOF_LONG ==4)
     /* bpo-34373: AIX does not return NULL if t is too small or too large */
     if (t < -2145916800 /* 1902-01-01 */
        || t > 2145916800 /* 2038-01-01 */) {

Testing...

vstinner · 2020-01-30T14:36:51Z

Probably this broke the 64-bit usage.

I'm not sure of the meaning of your patch. Are you saying that localtime() supports timestamp after the year 2038 on 64-bit AIX? Did you test that time.localtime(2**32) actually works as expected?

In my timezone, Linux, I get:

$ python3
Python 3.7.6 (default, Dec 19 2019, 22:52:49) 
>>> import time; time.localtime(2**32)
time.struct_time(tm_year=2106, tm_mon=2, tm_mday=7, tm_hour=7, tm_min=28, tm_sec=16, tm_wday=6, tm_yday=38, tm_isdst=0)

aixtools · 2020-01-30T14:40:47Z

From memory I recall the 64-bit version worked with values above the threshold value that broke the 32-bit library.

And the additional test was needed because the AIX library (iirc did not return NULL on error) - so had to test range before the call to get an overflow.

Working up a test of 32 and 64-bit enviornments.

vstinner · 2020-01-30T14:47:57Z

New changeset c232c91 by Victor Stinner in branch 'master':
bpo-39502: Skip test_zipfile.test_add_file_after_2107() on AIX (GH-18282)
c232c91

EGuesnet · 2020-01-30T14:52:32Z

I'm not sure of the meaning of your patch. Are you saying that localtime() supports timestamp after the year 2038 on 64-bit AIX? Did you test that time.localtime(2**32) actually works as expected?

I think it worked as expected before 3.8 on 64 bit.

On AIX 64bit, with Python3.7.4,

$ python3_64
Python 3.7.4 (default, Jan 15 2020, 15:46:22)
[GCC 8.3.0] on aix6
Type "help", "copyright", "credits" or "license" for more information.
>>> import time; time.localtime(2**32)
time.struct_time(tm_year=2106, tm_mon=2, tm_mday=7, tm_hour=7, tm_min=28, tm_sec=16, tm_wday=6, tm_yday=38, tm_isdst=0)

and on 32bit

$ python3_32 
Python 3.7.4 (default, Jan 15 2020, 15:50:53) 
[GCC 8.3.0] on aix6
Type "help", "copyright", "credits" or "license" for more information.
>>> import time; time.localtime(2**32)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
OverflowError: timestamp out of range for platform time_t

With the new Python 3.8.1, on both 32 and 64bits,

Python 3.8.1 (default, Jan 30 2020, 11:23:14) 
[GCC 8.3.0] on aix
Type "help", "copyright", "credits" or "license" for more information.
>>> import time; time.localtime(2**32)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
OverflowError: localtime argument out of range

vstinner · 2020-01-30T14:57:40Z

bpo-39502: Skip test_zipfile.test_add_file_after_2107() on AIX (GH-18282)

Even if localtime() is updated to support larger timestamp, this change is still needed for 32-bit AIX.

miss-islington · 2020-01-30T15:05:14Z

New changeset b841633 by Miss Islington (bot) in branch '3.8':
bpo-39502: Skip test_zipfile.test_add_file_after_2107() on AIX (GH-18282)
b841633

aixtools · 2020-01-30T15:06:29Z

Seems to be working on 64-bit, starting 32-bit test (with overflow expected). Once finished will post a PR.

root@x065:[/data/prj/python/python3-3.9]./python
Python 3.9.0a3+ (heads/bpo-39502-dirty:8d49f7ceb4, Jan 30 2020, 14:47:52) [C] on aix
Type "help", "copyright", "credits" or "license" for more information.
>>> import time
>>> time.localtime(4325562452)
time.struct_time(tm_year=2107, tm_mon=1, tm_mday=27, tm_hour=9, tm_min=7, tm_sec=32, tm_wday=3, tm_yday=27, tm_isdst=0)
>>> time.gmtime(4325562452)
time.struct_time(tm_year=2107, tm_mon=1, tm_mday=27, tm_hour=9, tm_min=7, tm_sec=32, tm_wday=3, tm_yday=27, tm_isdst=0)
>>>

vstinner · 2020-01-30T15:08:53Z

Seems to be working on 64-bit, starting 32-bit test (with overflow expected). Once finished will post a PR.

test_time already has tests for timestamps after year 2038, no?

aixtools · 2020-01-31T15:02:19Z

OK. There have been some comments/questions re: PR18282.

a) my PR is missing that PR, seems we just missed each other.

b) when using my patch I took a suggestion from bpo-39460 to test again:

root@x065:[/home/python/python3-3.9]./python -m test -v test_zipfile -m test_add_file_after_2107 --tempdir /tmp
== CPython 3.9.0a3+ (heads/bpo-39502-dirty:c54fe56362, Jan 30 2020, 15:52:37) [C]
== AIX-3-00C291F54C00-powerpc-64bit big-endian
== cwd: /tmp/test_python_303120
== CPU count: 4
== encodings: locale=ISO8859-1, FS=iso8859-1
0:00:00 Run tests sequentially
0:00:00 [1/1] test_zipfile
test_add_file_after_2107 (test.test_zipfile.StoredTestsWithSourceFile) ... ok

----------------------------------------------------------------------

Ran 1 test in 0.257s

OK

== Tests result: SUCCESS ==

1 test OK.

Total duration: 653 ms
Tests result: SUCCESS

Without the --tempdir setting it "skips" to PASS:

root@x065:[/home/python/python3-3.9]./python -m test -v test_zipfile -m test_add_file_after_2107
== CPython 3.9.0a3+ (heads/bpo-39502-dirty:c54fe56362, Jan 30 2020, 15:52:37) [C]
== AIX-3-00C291F54C00-powerpc-64bit big-endian
== cwd: /data/prj/python/python3-3.9/build/test_python_364784
== CPU count: 4
== encodings: locale=ISO8859-1, FS=iso8859-1
0:00:00 Run tests sequentially
0:00:00 [1/1] test_zipfile
test_add_file_after_2107 (test.test_zipfile.StoredTestsWithSourceFile) ... skipped 'Linux VFS/XFS kernel bug detected: mtime_ns=91301504000000000'

----------------------------------------------------------------------

Ran 1 test in 0.010s

OK (skipped=1)

== Tests result: SUCCESS ==

1 test OK.

Total duration: 252 ms
Tests result: SUCCESS

FYI

aixtools · 2020-01-31T15:08:50Z

p.s., I manually added bpo-18282 to the test, and the results are the same as without - using 64-bit. Will rebuild the 32-bit and try test again.

vstinner · 2020-02-03T08:40:07Z

test_add_file_after_2107 (test.test_zipfile.StoredTestsWithSourceFile) ... skipped 'Linux VFS/XFS kernel bug detected: mtime_ns=91301504000000000'

AIX kernel seems to have a bug. You should report it to AIX.

>>> time.localtime(91301504)
time.struct_time(tm_year=1972, tm_mon=11, tm_mday=22, tm_hour=18, tm_min=31, tm_sec=44, tm_wday=2, tm_yday=327, tm_isdst=0)

aixtools · 2020-02-03T14:45:15Z

Not sure I understand what bug I am supposed to report. I apologize if
my message https://bugs.python.org/issue39502#msg361116.

I assume your comment re: time.localtime(91301504) comes from this bit
of the test message (mtime_ns=91301504000000000).

Assuming our TZ used are different I am verifying AIX works correctly
for the constant 91301504

root@x065:[/data/prj/python/python3-3.9]./python
Python 3.9.0a3+ (heads/master-dirty:78c7183f47, Feb  3 2020, 14:10:40) [C] on aix
Type "help", "copyright", "credits" or "license" for more information.
>>> import time

>>> time.gmtime(91301504)
time.struct_time(tm_year=1972, tm_mon=11, tm_mday=22, tm_hour=17, tm_min=31, tm_sec=44, tm_wday=2, tm_yday=327, tm_isdst=0)
>>> time.localtime(91301504)
time.struct_time(tm_year=1972, tm_mon=11, tm_mday=22, tm_hour=17, tm_min=31, tm_sec=44, tm_wday=2, tm_yday=327, tm_isdst=0)

root@x065:[/data/prj/python/python3-3.9]set | grep TZ
TZ=UTC0

Again, my apologies for the confusion: the message
"skipped 'Linux VFS/XFS kernel bug detected: mtime_ns=91301504000000000" only occurs when the test is run on a NFS
mount hosted by a SAN based on a Linux kernel.

When I specify --tempdir=/tmp (i.e., using jfs2 as filesystem) the test passes normally.

On 03/02/2020 09:40, STINNER Victor wrote:
> STINNER Victor <vstinner@python.org> added the comment:
>
>> test_add_file_after_2107 (test.test_zipfile.StoredTestsWithSourceFile) ... skipped 'Linux VFS/XFS kernel bug detected: mtime_ns=91301504000000000'
> AIX kernel seems to have a bug. You should report it to AIX.
>
>>>> time.localtime(91301504)
> time.struct_time(tm_year=1972, tm_mon=11, tm_mday=22, tm_hour=18, tm_min=31, tm_sec=44, tm_wday=2, tm_yday=327, tm_isdst=0)
>
>

Python tracker <report@bugs.python.org>
<https://bugs.python.org/issue39502\>

DavidEdelsohn · 2020-02-03T19:30:12Z

I think that Victor means AIX kernel and filesystems are not prepared for Y2038.

vstinner · 2020-02-07T17:56:20Z

New changeset de6f38d by Michael Felt in branch 'master':
bpo-39502: Fix 64-bit Python PyTime_localtime() on AIX (GH-18285)
de6f38d

vstinner · 2020-02-07T17:57:19Z

Does time.gmtime() accept year after 2038 on 64-bit AIX? Example on Linux:

>>> time.gmtime(4386268800)
time.struct_time(tm_year=2108, tm_mon=12, tm_mday=30, tm_hour=0, tm_min=0, tm_sec=0, tm_wday=6, tm_yday=365, tm_isdst=0)

aixtools · 2020-03-26T09:42:37Z

My apologies for the late reply -

Here is 3.6.10:

Python 3.6.10 (default, Mar 24 2020, 14:12:31) [C] on aix5
Type "help", "copyright", "credits" or "license" for more information.
>>> import time
>>> time.gmtime(4386268800)
time.struct_time(tm_year=2108, tm_mon=12, tm_mday=30, tm_hour=0,
tm_min=0, tm_sec=0, tm_wday=6, tm_yday=365, tm_isdst=0)

And, a patched 3.9:

root@x064:[/data/prj/python/python-3.9]./python
Python 3.9.0a4+ (default, Mar 13 2020, 08:03:36) [C] on aix
Type "help", "copyright", "credits" or "license" for more information.
>>> import time
>>> time.gmtime(4386268800)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
OverflowError: timestamp out of range for platform time_t

So, it should be working - but is not after the patches. I'll work on an
update asap.

On 07/02/2020 18:57, STINNER Victor wrote:
> STINNER Victor <vstinner@python.org> added the comment:
>
> Does time.gmtime() accept year after 2038 on 64-bit AIX? Example on Linux:
>
>>>> time.gmtime(4386268800)
> time.struct_time(tm_year=2108, tm_mon=12, tm_mday=30, tm_hour=0, tm_min=0, tm_sec=0, tm_wday=6, tm_yday=365, tm_isdst=0)
>
>

Python tracker <report@bugs.python.org>
<https://bugs.python.org/issue39502\>

iritkatriel · 2022-06-24T22:31:30Z

Is there anything left to do here?

EGuesnet mannequin added 3.8 only security fixes tests Tests in the Lib/test dir type-bug An unexpected behavior, bug, or error labels Jan 30, 2020

aixtools added interpreter-core (Objects, Python, Grammar, and Parser dirs) 3.7 (EOL) end of life 3.9 only security fixes labels Jan 30, 2020

ezio-melotti transferred this issue from another repository Apr 10, 2022

iritkatriel added the pending The issue will be closed if no feedback is provided label Jun 24, 2022

kumaraditya303 closed this as not planned Won't fix, can't repro, duplicate, stale Jul 12, 2022

erlend-aasland removed the pending The issue will be closed if no feedback is provided label Jul 20, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

test_zipfile fails on AIX due to time.localtime #83683

test_zipfile fails on AIX due to time.localtime #83683

EGuesnet mannequin commented Jan 30, 2020

EGuesnet mannequin commented Jan 30, 2020

vstinner commented Jan 30, 2020

EGuesnet mannequin commented Jan 30, 2020

vstinner commented Jan 30, 2020

aixtools commented Jan 30, 2020

vstinner commented Jan 30, 2020

aixtools commented Jan 30, 2020

vstinner commented Jan 30, 2020

EGuesnet mannequin commented Jan 30, 2020

vstinner commented Jan 30, 2020

miss-islington commented Jan 30, 2020

aixtools commented Jan 30, 2020

vstinner commented Jan 30, 2020

aixtools commented Jan 31, 2020

aixtools commented Jan 31, 2020

vstinner commented Feb 3, 2020

aixtools commented Feb 3, 2020

DavidEdelsohn mannequin commented Feb 3, 2020

vstinner commented Feb 7, 2020

vstinner commented Feb 7, 2020

aixtools commented Mar 26, 2020

iritkatriel commented Jun 24, 2022

test_zipfile fails on AIX due to time.localtime #83683

test_zipfile fails on AIX due to time.localtime #83683

Comments

EGuesnet mannequin commented Jan 30, 2020

EGuesnet mannequin commented Jan 30, 2020

vstinner commented Jan 30, 2020

EGuesnet mannequin commented Jan 30, 2020

vstinner commented Jan 30, 2020

aixtools commented Jan 30, 2020

vstinner commented Jan 30, 2020

aixtools commented Jan 30, 2020

vstinner commented Jan 30, 2020

EGuesnet mannequin commented Jan 30, 2020

vstinner commented Jan 30, 2020

miss-islington commented Jan 30, 2020

aixtools commented Jan 30, 2020

vstinner commented Jan 30, 2020

aixtools commented Jan 31, 2020

aixtools commented Jan 31, 2020

vstinner commented Feb 3, 2020

aixtools commented Feb 3, 2020

DavidEdelsohn mannequin commented Feb 3, 2020

vstinner commented Feb 7, 2020

vstinner commented Feb 7, 2020

aixtools commented Mar 26, 2020

iritkatriel commented Jun 24, 2022