classification
Title: test_zipfile fails on AIX due to time.localtime
Type: behavior Stage: patch review
Components: Interpreter Core, Tests Versions: Python 3.9, Python 3.8, Python 3.7
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: David.Edelsohn, EGuesnet, Michael.Felt, miss-islington, vstinner
Priority: normal Keywords: patch

Created on 2020-01-30 13:51 by EGuesnet, last changed 2020-03-26 09:42 by Michael.Felt.

Files
File name Uploaded Description Edit
test_zipfile.patch vstinner, 2020-01-30 13:55
Pull Requests
URL Status Linked Edit
PR 18282 merged vstinner, 2020-01-30 14:05
PR 18283 merged miss-islington, 2020-01-30 14:48
PR 18285 merged Michael.Felt, 2020-01-30 15:26
Messages (21)
msg361058 - (view) Author: EGuesnet (EGuesnet) Date: 2020-01-30 13:51
Hi,
I have an error during regression tests with Python3.8.1 on AIX 6.1 compiled with GCC 8.3.
It occurs only on 64 bit. Test passes on 32 bit.

```
======================================================================
ERROR: test_add_file_after_2107 (test.test_zipfile.StoredTestsWithSourceFile)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/opt/freeware/src/packages/BUILD/Python-3.8.1/64bit/Lib/test/test_zipfile.py", line 606, in test_add_file_after_2107
    self.assertRaises(struct.error, zipfp.write, TESTFN)
  File "/opt/freeware/src/packages/BUILD/Python-3.8.1/64bit/Lib/unittest/case.py", line 816, in assertRaises
    return context.handle('assertRaises', args, kwargs)
  File "/opt/freeware/src/packages/BUILD/Python-3.8.1/64bit/Lib/unittest/case.py", line 202, in handle
    callable_obj(*args, **kwargs)
  File "/opt/freeware/src/packages/BUILD/Python-3.8.1/64bit/Lib/zipfile.py", line 1739, in write
    zinfo = ZipInfo.from_file(filename, arcname,
  File "/opt/freeware/src/packages/BUILD/Python-3.8.1/64bit/Lib/zipfile.py", line 523, in from_file
    mtime = time.localtime(st.st_mtime)
OverflowError: localtime argument out of range
```

The PR associated to the new behavior is: https://github.com/python/cpython/pull/12726 (new on Python 3.8). Code is AIX specific. Is the code 32 bit only, or maybe the test was not updated?

-----

I can reproduce the behavior as follow:

```
$ python3.8_32
Python 3.8.1 (default, Jan 27 2020, 11:34:59) 
[GCC 8.3.0] on aix
Type "help", "copyright", "credits" or "license" for more information.
>>> import time
>>> time.localtime(4325562452)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
OverflowError: timestamp out of range for platform time_t

$ python3.8_64
Python 3.8.1 (default, Jan 27 2020, 11:30:15) 
[GCC 8.3.0] on aix
Type "help", "copyright", "credits" or "license" for more information.
>>> import time
>>> time.localtime(4325562452)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
OverflowError: localtime argument out of range

$ python3.7_32
Python 3.7.4 (default, Jan 15 2020, 15:50:53) 
[GCC 8.3.0] on aix6
Type "help", "copyright", "credits" or "license" for more information.
>>> import time
>>> time.localtime(4325562452)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
OverflowError: timestamp out of range for platform time_t

$ python3.7_64
Python 3.7.4 (default, Jan 15 2020, 15:46:22) 
[GCC 8.3.0] on aix6
Type "help", "copyright", "credits" or "license" for more information.
>>> import time
>>> time.localtime(4325562452)
time.struct_time(tm_year=2107, tm_mon=1, tm_mday=27, tm_hour=10, tm_min=7, tm_sec=32, tm_wday=3, tm_yday=27, tm_isdst=0)
```
msg361059 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-01-30 13:55
I suggest to skip the test if time.localtime() fails with OverflowError.

Can you please try attached patch on AIX, on the master branch if possible? (Or on Python 3.8 at least.)
msg361061 - (view) Author: EGuesnet (EGuesnet) Date: 2020-01-30 14:03
Tested on Python 3.8.1 on both 32 and 64 bits.
Test is now skipped.


test_add_file_after_2107 (test.test_zipfile.StoredTestsWithSourceFile) ... skipped 'time.localtime(4386268800) raises OverflowError'
msg361062 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-01-30 14:08
> Tested on Python 3.8.1 on both 32 and 64 bits. Test is now skipped.

Good. I created PR 18282 from my patch.
msg361063 - (view) Author: Michael Felt (Michael.Felt) * Date: 2020-01-30 14:20
Probably this broke the 64-bit usage.

diff --git a/Python/pytime.c b/Python/pytime.c
index 54ddfc952b..6f13e62490 100644
--- a/Python/pytime.c
+++ b/Python/pytime.c
@@ -1059,7 +1059,7 @@ _PyTime_localtime(time_t t, struct tm *tm)
     return 0;
 #else /* !MS_WINDOWS */

-#ifdef _AIX
+#ifdef defined(_AIX) and (SIZEOF_LONG ==4)
     /* bpo-34373: AIX does not return NULL if t is too small or too large */
     if (t < -2145916800 /* 1902-01-01 */
        || t > 2145916800 /* 2038-01-01 */) {


Testing...
msg361064 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-01-30 14:36
> Probably this broke the 64-bit usage.

I'm not sure of the meaning of your patch. Are you saying that localtime() supports timestamp after the year 2038 on 64-bit AIX? Did you test that time.localtime(2**32) actually works as expected?

In my timezone, Linux, I get:

$ python3
Python 3.7.6 (default, Dec 19 2019, 22:52:49) 
>>> import time; time.localtime(2**32)
time.struct_time(tm_year=2106, tm_mon=2, tm_mday=7, tm_hour=7, tm_min=28, tm_sec=16, tm_wday=6, tm_yday=38, tm_isdst=0)
msg361065 - (view) Author: Michael Felt (Michael.Felt) * Date: 2020-01-30 14:40
From memory I recall the 64-bit version worked with values above the threshold value that broke the 32-bit library.

And the additional test was needed because the AIX library (iirc did not return NULL on error) - so had to test range before the call to get an overflow.

Working up a test of 32 and 64-bit enviornments.
msg361066 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-01-30 14:47
New changeset c232c9110cfefa0935cbf158e35e91746a8a9361 by Victor Stinner in branch 'master':
bpo-39502: Skip test_zipfile.test_add_file_after_2107() on AIX (GH-18282)
https://github.com/python/cpython/commit/c232c9110cfefa0935cbf158e35e91746a8a9361
msg361067 - (view) Author: EGuesnet (EGuesnet) Date: 2020-01-30 14:52
> I'm not sure of the meaning of your patch. Are you saying that localtime() supports timestamp after the year 2038 on 64-bit AIX? Did you test that time.localtime(2**32) actually works as expected?

I think it worked as expected before 3.8 on 64 bit.

On AIX 64bit, with Python3.7.4,

$ python3_64
Python 3.7.4 (default, Jan 15 2020, 15:46:22)
[GCC 8.3.0] on aix6
Type "help", "copyright", "credits" or "license" for more information.
>>> import time; time.localtime(2**32)
time.struct_time(tm_year=2106, tm_mon=2, tm_mday=7, tm_hour=7, tm_min=28, tm_sec=16, tm_wday=6, tm_yday=38, tm_isdst=0)

and on 32bit

$ python3_32 
Python 3.7.4 (default, Jan 15 2020, 15:50:53) 
[GCC 8.3.0] on aix6
Type "help", "copyright", "credits" or "license" for more information.
>>> import time; time.localtime(2**32)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
OverflowError: timestamp out of range for platform time_t

With the new Python 3.8.1, on both 32 and 64bits,

Python 3.8.1 (default, Jan 30 2020, 11:23:14) 
[GCC 8.3.0] on aix
Type "help", "copyright", "credits" or "license" for more information.
>>> import time; time.localtime(2**32)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
OverflowError: localtime argument out of range
msg361068 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-01-30 14:57
> bpo-39502: Skip test_zipfile.test_add_file_after_2107() on AIX (GH-18282)

Even if localtime() is updated to support larger timestamp, this change is still needed for 32-bit AIX.
msg361069 - (view) Author: miss-islington (miss-islington) Date: 2020-01-30 15:05
New changeset b841633cc2d7619cf4a7db108d91b14926450a6e by Miss Islington (bot) in branch '3.8':
bpo-39502: Skip test_zipfile.test_add_file_after_2107() on AIX (GH-18282)
https://github.com/python/cpython/commit/b841633cc2d7619cf4a7db108d91b14926450a6e
msg361070 - (view) Author: Michael Felt (Michael.Felt) * Date: 2020-01-30 15:06
Seems to be working on 64-bit, starting 32-bit test (with overflow expected). Once finished will post a PR.

root@x065:[/data/prj/python/python3-3.9]./python
Python 3.9.0a3+ (heads/bpo-39502-dirty:8d49f7ceb4, Jan 30 2020, 14:47:52) [C] on aix
Type "help", "copyright", "credits" or "license" for more information.
>>> import time
>>> time.localtime(4325562452)
time.struct_time(tm_year=2107, tm_mon=1, tm_mday=27, tm_hour=9, tm_min=7, tm_sec=32, tm_wday=3, tm_yday=27, tm_isdst=0)
>>> time.gmtime(4325562452)
time.struct_time(tm_year=2107, tm_mon=1, tm_mday=27, tm_hour=9, tm_min=7, tm_sec=32, tm_wday=3, tm_yday=27, tm_isdst=0)
>>>
msg361071 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-01-30 15:08
> Seems to be working on 64-bit, starting 32-bit test (with overflow expected). Once finished will post a PR.

test_time already has tests for timestamps after year 2038, no?
msg361116 - (view) Author: Michael Felt (Michael.Felt) * Date: 2020-01-31 15:02
OK. There have been some comments/questions re: PR18282.

a) my PR is missing that PR, seems we just missed each other.

b) when using my patch I took a suggestion from issue39460 to test again:

root@x065:[/home/python/python3-3.9]./python -m test -v test_zipfile -m test_add_file_after_2107 --tempdir /tmp
== CPython 3.9.0a3+ (heads/bpo-39502-dirty:c54fe56362, Jan 30 2020, 15:52:37) [C]
== AIX-3-00C291F54C00-powerpc-64bit big-endian
== cwd: /tmp/test_python_303120
== CPU count: 4
== encodings: locale=ISO8859-1, FS=iso8859-1
0:00:00 Run tests sequentially
0:00:00 [1/1] test_zipfile
test_add_file_after_2107 (test.test_zipfile.StoredTestsWithSourceFile) ... ok

----------------------------------------------------------------------

Ran 1 test in 0.257s

OK

== Tests result: SUCCESS ==

1 test OK.

Total duration: 653 ms
Tests result: SUCCESS

Without the --tempdir setting it "skips" to PASS:

root@x065:[/home/python/python3-3.9]./python -m test -v test_zipfile -m test_add_file_after_2107
== CPython 3.9.0a3+ (heads/bpo-39502-dirty:c54fe56362, Jan 30 2020, 15:52:37) [C]
== AIX-3-00C291F54C00-powerpc-64bit big-endian
== cwd: /data/prj/python/python3-3.9/build/test_python_364784
== CPU count: 4
== encodings: locale=ISO8859-1, FS=iso8859-1
0:00:00 Run tests sequentially
0:00:00 [1/1] test_zipfile
test_add_file_after_2107 (test.test_zipfile.StoredTestsWithSourceFile) ... skipped 'Linux VFS/XFS kernel bug detected: mtime_ns=91301504000000000'

----------------------------------------------------------------------

Ran 1 test in 0.010s

OK (skipped=1)

== Tests result: SUCCESS ==

1 test OK.

Total duration: 252 ms
Tests result: SUCCESS

FYI
msg361118 - (view) Author: Michael Felt (Michael.Felt) * Date: 2020-01-31 15:08
p.s., I manually added #18282 to the test, and the results are the same as without - using 64-bit. Will rebuild the 32-bit and try test again.
msg361272 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-02-03 08:40
> test_add_file_after_2107 (test.test_zipfile.StoredTestsWithSourceFile) ... skipped 'Linux VFS/XFS kernel bug detected: mtime_ns=91301504000000000'

AIX kernel seems to have a bug. You should report it to AIX.

>>> time.localtime(91301504)
time.struct_time(tm_year=1972, tm_mon=11, tm_mday=22, tm_hour=18, tm_min=31, tm_sec=44, tm_wday=2, tm_yday=327, tm_isdst=0)
msg361303 - (view) Author: Michael Felt (Michael.Felt) * Date: 2020-02-03 14:45
Not sure I understand what bug I am supposed to report. I apologize if 
my message https://bugs.python.org/issue39502#msg361116.

I assume your comment re: time.localtime(91301504) comes from this bit 
of the test message (mtime_ns=91301504000000000).

Assuming our TZ used are different I am verifying AIX works correctly 
for the constant 91301504

root@x065:[/data/prj/python/python3-3.9]./python
Python 3.9.0a3+ (heads/master-dirty:78c7183f47, Feb  3 2020, 14:10:40) [C] on aix
Type "help", "copyright", "credits" or "license" for more information.
>>> import time

>>> time.gmtime(91301504)
time.struct_time(tm_year=1972, tm_mon=11, tm_mday=22, tm_hour=17, tm_min=31, tm_sec=44, tm_wday=2, tm_yday=327, tm_isdst=0)
>>> time.localtime(91301504)
time.struct_time(tm_year=1972, tm_mon=11, tm_mday=22, tm_hour=17, tm_min=31, tm_sec=44, tm_wday=2, tm_yday=327, tm_isdst=0)

root@x065:[/data/prj/python/python3-3.9]set | grep TZ
TZ=UTC0

Again, my apologies for the confusion: the message
"skipped 'Linux VFS/XFS kernel bug detected: mtime_ns=91301504000000000" only occurs when the test is run on a NFS
mount hosted by a SAN based on a Linux kernel.

When I specify --tempdir=/tmp (i.e., using jfs2 as filesystem) the test passes normally.

On 03/02/2020 09:40, STINNER Victor wrote:
> STINNER Victor <vstinner@python.org> added the comment:
>
>> test_add_file_after_2107 (test.test_zipfile.StoredTestsWithSourceFile) ... skipped 'Linux VFS/XFS kernel bug detected: mtime_ns=91301504000000000'
> AIX kernel seems to have a bug. You should report it to AIX.
>
>>>> time.localtime(91301504)
> time.struct_time(tm_year=1972, tm_mon=11, tm_mday=22, tm_hour=18, tm_min=31, tm_sec=44, tm_wday=2, tm_yday=327, tm_isdst=0)
>
> ----------
>
> _______________________________________
> Python tracker <report@bugs.python.org>
> <https://bugs.python.org/issue39502>
> _______________________________________
>
msg361323 - (view) Author: David Edelsohn (David.Edelsohn) * Date: 2020-02-03 19:30
I think that Victor means AIX kernel and filesystems are not prepared for Y2038.
msg361603 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-02-07 17:56
New changeset de6f38db4859f7b8fe4da4556f06c52675fff24a by Michael Felt in branch 'master':
bpo-39502: Fix 64-bit Python PyTime_localtime() on AIX  (GH-18285)
https://github.com/python/cpython/commit/de6f38db4859f7b8fe4da4556f06c52675fff24a
msg361604 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-02-07 17:57
Does time.gmtime() accept year after 2038 on 64-bit AIX? Example on Linux:

>>> time.gmtime(4386268800)
time.struct_time(tm_year=2108, tm_mon=12, tm_mday=30, tm_hour=0, tm_min=0, tm_sec=0, tm_wday=6, tm_yday=365, tm_isdst=0)
msg365056 - (view) Author: Michael Felt (Michael.Felt) * Date: 2020-03-26 09:42
My apologies for the late reply -

Here is 3.6.10:

Python 3.6.10 (default, Mar 24 2020, 14:12:31) [C] on aix5
Type "help", "copyright", "credits" or "license" for more information.
>>> import time
>>> time.gmtime(4386268800)
time.struct_time(tm_year=2108, tm_mon=12, tm_mday=30, tm_hour=0,
tm_min=0, tm_sec=0, tm_wday=6, tm_yday=365, tm_isdst=0)

And, a patched 3.9:

root@x064:[/data/prj/python/python-3.9]./python
Python 3.9.0a4+ (default, Mar 13 2020, 08:03:36) [C] on aix
Type "help", "copyright", "credits" or "license" for more information.
>>> import time
>>> time.gmtime(4386268800)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
OverflowError: timestamp out of range for platform time_t

So, it should be working - but is not after the patches. I'll work on an
update asap.

On 07/02/2020 18:57, STINNER Victor wrote:
> STINNER Victor <vstinner@python.org> added the comment:
>
> Does time.gmtime() accept year after 2038 on 64-bit AIX? Example on Linux:
>
>>>> time.gmtime(4386268800)
> time.struct_time(tm_year=2108, tm_mon=12, tm_mday=30, tm_hour=0, tm_min=0, tm_sec=0, tm_wday=6, tm_yday=365, tm_isdst=0)
>
> ----------
>
> _______________________________________
> Python tracker <report@bugs.python.org>
> <https://bugs.python.org/issue39502>
> _______________________________________
>
History
Date User Action Args
2020-03-26 09:42:37Michael.Feltsetmessages: + msg365056
2020-02-07 17:57:18vstinnersetmessages: + msg361604
2020-02-07 17:56:19vstinnersetmessages: + msg361603
2020-02-03 19:30:11David.Edelsohnsetnosy: + David.Edelsohn
messages: + msg361323
2020-02-03 14:45:15Michael.Feltsetmessages: + msg361303
2020-02-03 08:40:07vstinnersetmessages: + msg361272
2020-01-31 15:08:49Michael.Feltsetmessages: + msg361118
2020-01-31 15:02:19Michael.Feltsetnosy: vstinner, Michael.Felt, miss-islington, EGuesnet
messages: + msg361116
2020-01-30 15:26:41Michael.Feltsetpull_requests: + pull_request17660
2020-01-30 15:08:53vstinnersetmessages: + msg361071
2020-01-30 15:06:29Michael.Feltsetmessages: + msg361070
components: + Interpreter Core
versions: + Python 3.7, Python 3.9
2020-01-30 15:05:14miss-islingtonsetnosy: + miss-islington
messages: + msg361069
2020-01-30 14:57:39vstinnersetmessages: + msg361068
2020-01-30 14:52:32EGuesnetsetmessages: + msg361067
2020-01-30 14:48:16miss-islingtonsetpull_requests: + pull_request17658
2020-01-30 14:47:57vstinnersetmessages: + msg361066
2020-01-30 14:40:46Michael.Feltsetmessages: + msg361065
2020-01-30 14:36:51vstinnersetmessages: + msg361064
2020-01-30 14:20:03Michael.Feltsetmessages: + msg361063
2020-01-30 14:08:17vstinnersetmessages: + msg361062
2020-01-30 14:05:48vstinnersetstage: patch review
pull_requests: + pull_request17657
2020-01-30 14:03:16EGuesnetsetmessages: + msg361061
2020-01-30 13:55:19vstinnersetfiles: + test_zipfile.patch

nosy: + vstinner
messages: + msg361059

keywords: + patch
2020-01-30 13:52:58Michael.Feltsetnosy: + Michael.Felt
2020-01-30 13:51:37EGuesnetcreate