This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: Improvement mktime error reporting
Type: behavior Stage: needs patch
Components: Versions: Python 3.11, Python 3.10, Python 3.9
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: andrei.avk, belopolsky, christian.heimes, corona10, kj, lemburg, p-ganssle, psijic, terry.reedy, wasade
Priority: normal Keywords:

Created on 2021-06-13 18:02 by psijic, last changed 2022-04-11 14:59 by admin.

Pull Requests
URL Status Linked Edit
PR 30124 closed andrei.avk, 2021-12-15 17:29
Messages (15)
msg395758 - (view) Author: Vyacheslav (psijic) Date: 2021-06-13 18:02
date_str = "2019-06-06 10:02:00"
    datetime_obj = datetime.strptime(date_str, "%Y-%m-%d %H:%M:%S")
    datetime_obj_utc1 = datetime_obj.replace(tzinfo=pytz.timezone("America/New_York"))
    datetime_obj_utc2 = pytz.timezone("America/New_York").localize(datetime_obj)
    print(datetime_obj_utc1, datetime_obj_utc2)
    ts1 = time.mktime(datetime_obj_utc1.timetuple())
    ts2 = time.mktime(datetime_obj_utc2.timetuple())

>>> Output:
    ts2 = int(time.mktime(datetime_obj_utc2.timetuple()))
OverflowError: mktime argument out of range


OS: Linux OpenSuse TW
msg396102 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2021-06-18 22:05
An exception is not a 'crash', as defined for this tracker.  3.8 only gets security fixes.  A 3.9 or 3.10 test is needed and Windows does not have pytz.
msg397600 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2021-07-16 06:09
The posted code is incomplete, improperly indented, and a bit jumbled with respect to prompts.  If the bug is real, it should be possible to reproduce without pytz.  I do not have it on either Windows or macOS.  Vyacheslav, can you post a better failing example?
msg407346 - (view) Author: Andrei Kulakov (andrei.avk) * (Python triager) Date: 2021-11-30 01:12
Cannot reproduce with 3.8.6 on MacOS. The code runs without any errors. As OP hasn't responded in 4 months I think we should close as "works for me".
msg408615 - (view) Author: Daniel McDonald (wasade) Date: 2021-12-15 16:15
I'd like to politely request this issue be reopened. 

We recently observed a similar, if not the same, OverflowError during continuous integration using Github Actions on ubuntu-latest (20.04). We can produce the error using pure Python without pytz.

We have been unable to replicate the error using Github Actions on macos-latest, Centos 7.9, Ubuntu 18.04 or OSX 11.6.1.

Specifically, the following code, which I believe is pure python, will trigger the overflow. We've observed this using Python 3.7 and 3.9. 

import time
print(time.mktime((2017,5,26,15,30,16,4,146,1)))

Exact error output from CI can be found at the link below: 

https://github.com/biocore/microsetta-private-api/runs/4536611219?check_suite_focus=true#step:4:117

On a passing system, we receive "1495837816.0" as output. On a failing system, we observe "OverflowError: mktime argument out of range". 

More detail can be found on our issue with pytz and stub42 who helped guide a more definitive example of the bug showing it could be pure Python. That issue can be found here:

https://github.com/stub42/pytz/issues/66

Last, I apologize in advance if any detail is missing here or if this is not formatted as well as it should be. This is my first bug report with Python and am unfamiliar with the norms. Please let me know if further information may be helpful.
msg408627 - (view) Author: Andrei Kulakov (andrei.avk) * (Python triager) Date: 2021-12-15 17:57
Confirmed on Ubuntu buildbot:

https://github.com/python/cpython/runs/4537544103?check_suite_focus=true
msg408642 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2021-12-15 20:23
Thank you both.  Status: Failures on Open Suse TW (Vyacheslav) and Ubuntu 20-04 (Daniel McDonald, Github Actions, and Azure Pipelines).
Success on Windows (me), macOS (Catalina-me and 11.6.1-DM), Centos 7.9 and Ubuntu 18-04 (both DM) 

I verified that the sample tuple is consistent and round-trips with time.localtime.

>>> import time
>>> time.mktime((2017,5,26,15,30,16,4,146,1))
1495827016.0
>>> time.localtime(time.mktime((2017,5,26,15,30,16,4,146,1)))
time.struct_time(tm_year=2017, tm_mon=5, tm_mday=26, tm_hour=15, tm_min=30, tm_sec=16, tm_wday=4, tm_yday=146, tm_isdst=1)

While OverflowError is documented as a legitimate response, it is not for a contemporary valid datetime such as the above.  Someone with a failure machine could try to determine what tuple vales do and don't result in the error.  Someone with both failure and success machines could add debug prints to mktime (or use C debugger) to see where the behavior of the C code diverges on the two machines.  If no one active on the issue can do either, a request could be made for help on python-list.
msg408644 - (view) Author: Daniel McDonald (wasade) Date: 2021-12-15 20:28
Thank you, Terry. I'm currently exploring modifications to the test Andrei made within a fork of CPython using Github Actions (ubuntu-latest). These modifications include debug prints in the CPython mktime call, and some parameter exploration. I expect to have a summary of the observations this afternoon.
msg408650 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2021-12-15 21:55
time.mktime() is a thin wrapper around the libc function mktime(3). I can confirm that glibc's mktime() on Debian 11 with glibc 2.31 is failing and returning error code -1. Fedora 35 with glibc 2.34, Alpine with musl libc 1.2.2, and RHEL 8 with glibc 2.28 are working fine.

This suggests that it is not a bug in Python, but Debian-specific platform issue. Could you please open a bug with Debian and Ubuntu?
msg408658 - (view) Author: Daniel McDonald (wasade) Date: 2021-12-15 23:44
The use of tm_isdst=1 appears to trigger the overflow, and occurs when varying other aspects of the timetuple.  

Python's mktime wrapper can throw OverflowError in two places. The thrown error is the second location, following the call to glibc's mktime occurring in this specific if block:

https://github.com/python/cpython/blob/f62420c3d3f5d87f2b57e54b2a98682bc835f7b6/Modules/timemodule.c#L1038-L1046

Modification of Modules/timemodule.c as to printf tm struct members confirms that when tm_isdst=1, the result from mktime() is -1, and the tm_wday member is set to -1, the combination of which triggers the conditional. 

mktime within the Github Action ubuntu-latest (20.04) environment is sourced from glibc 2.31 as confirmed by ldd and nm on the compiled Python binary.

As a proof of concept, C program was written that calls mktime independent of CPython (can be compiled with "gcc bug.c -o bug"). The code below succeeds on both OSX 11.6.1 and Centos 7.9:

#include <time.h>
#include <stdio.h>

void do_test() {
  struct tm tm_works = { .tm_year=117,
                         .tm_mon=4,
                         .tm_mday=26,
                         .tm_hour=15,
                         .tm_min=30,
                         .tm_sec=16,
                         .tm_wday=4,
                         .tm_yday=145,
                         .tm_isdst=-1 };
  
  struct tm tm_fails = { .tm_year=117,
                         .tm_mon=4,
                         .tm_mday=26,
                         .tm_hour=15,
                         .tm_min=30,
                         .tm_sec=16,
                         .tm_wday=4,
                         .tm_yday=145,
                         .tm_isdst=1 };

  time_t works = mktime(&tm_works);
  time_t fails = mktime(&tm_fails);

  if(works == -1) {
      printf("Unexpected failure\n");
  } else {
      if(works == fails) {
          printf("Test passed\n");
      } else {
          printf("Test failed: works=%d; fails=%d\n", (int)works, (int)fails);
      }
  }
}

int main(int argc, char **argv) {
    do_test();
}

When compiled and run within in the Github Actions ubuntu-latest (20.04) environment, the erroneous behavior occurs:

https://github.com/wasade/cpython/runs/4541212472?check_suite_focus=true#step:17:57

The use of tm_isdst=1 is valid according to multiple sources, a few examples are linked below. The sources are consistent, and indicate a positive value means Daylight Savings Time is in effect. 

https://www.gnu.org/software/libc/manual/html_node/Broken_002ddown-Time.html
https://www.cplusplus.com/reference/ctime/tm/

At this time, I believe this remains unexpected behavior, however the source appears upstream of CPython as mktime is part of glibc. Unless suggested otherwise, I'll open an issue with on the glibc tracker.
msg408659 - (view) Author: Daniel McDonald (wasade) Date: 2021-12-15 23:46
Thank you, Christian, I apologize for missing your reply. That is great advice, and I will do so.
msg408660 - (view) Author: Daniel McDonald (wasade) Date: 2021-12-16 00:12
For reference, the bug reports with Debian and Ubuntu are at the following URLs:

https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1001774
https://bugs.launchpad.net/ubuntu/+source/glibc/+bug/1954963
msg408977 - (view) Author: Daniel McDonald (wasade) Date: 2021-12-20 22:32
A definitive assessment was obtained by Aurelien with Debian:

https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1001774

In brief, the OverflowError is expected and correct. 

Two factors contribute to the difference in observed behavior. First, the production of the -1 from mktime which Python uses to raise OverflowError depends on whether timezone is or is not set (e.g. "Etc/UTC"). This was confirmed on the Github Actions 20.04 ubuntu instance independent of Python directly with the mktime() function from C. Second, mktime returning -1 to indicate overflow is dependent on glibc >= 2.29, and versions prior "do not report any error if the date is not representable." This second factor is supported by testing the lack of the -1 (and subsequent OverflowError) on Github Actions ubuntu 18.04, which uses an older version of glibc.

The full thread with Debian is here:

https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1001774

With regards to CPython, a possible change to consider would be to test if tm_isdst < 0 prior to the mktime call, and if that evaluates true to provide an error indicating use requires a timezone being set. I think that change would make Python's behavior consistent with glibc >= 2.29, and smooth over a potential platform dependency, while also providing an informative error message. However, I have limited knowledge of how timezones are implemented and am unsure if more than "Etc/UTC" would result in tm_isdst being < 0. 

If that change makes sense, I'd be happy to submit a pull request to that effect. However, I recognize there may be considerations not accounted for here particularly as this is a central module, so I'll hold off issuing a pull request until hearing back from on this matter.
msg408978 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2021-12-20 22:42
Daniel, I believe your proposal is an enhancement request that could only be implemented in a future version.  I changed headers accordingly.  I leave it to Christian or someone else to evaluate the request itself.
msg408979 - (view) Author: Daniel McDonald (wasade) Date: 2021-12-20 22:45
Sounds good, thank you, Terry
History
Date User Action Args
2022-04-11 14:59:46adminsetgithub: 88579
2021-12-20 22:45:30wasadesettype: enhancement -> behavior
messages: + msg408979
2021-12-20 22:42:24terry.reedysettype: behavior -> enhancement
messages: + msg408978
title: OverflowError: mktime argument out of range after 2019 -> Improvement mktime error reporting
2021-12-20 22:32:30wasadesetmessages: + msg408977
2021-12-16 00:12:54wasadesetmessages: + msg408660
2021-12-15 23:46:15wasadesetmessages: + msg408659
2021-12-15 23:44:33wasadesetmessages: + msg408658
2021-12-15 21:55:47christian.heimessetnosy: + christian.heimes
messages: + msg408650
2021-12-15 20:28:38wasadesetmessages: + msg408644
2021-12-15 20:23:00terry.reedysetmessages: + msg408642
2021-12-15 17:58:13andrei.avksetstatus: closed -> open
stage: resolved -> needs patch
resolution: works for me ->
versions: + Python 3.10, Python 3.11, - Python 3.7, Python 3.8
2021-12-15 17:57:37andrei.avksetmessages: + msg408627
2021-12-15 17:29:08andrei.avksetpull_requests: + pull_request28343
2021-12-15 16:15:50wasadesetnosy: + wasade

messages: + msg408615
versions: + Python 3.7, Python 3.9
2021-11-30 06:19:30iritkatrielsetstatus: open -> closed
resolution: works for me
stage: resolved
2021-11-30 01:12:58andrei.avksetnosy: + kj
2021-11-30 01:12:26andrei.avksetnosy: + andrei.avk
messages: + msg407346
2021-07-16 06:09:56terry.reedysetmessages: + msg397600
2021-07-16 03:51:40corona10setnosy: + corona10
2021-06-18 22:05:24terry.reedysetnosy: + terry.reedy, belopolsky, p-ganssle, lemburg
type: crash -> behavior
messages: + msg396102
2021-06-13 18:02:13psijiccreate