classification
Title: datetime.fromtimestamp() doesn't check min/max year anymore: regression of Python 3.6
Type: behavior Stage: resolved
Components: Library (Lib) Versions: Python 3.7, Python 3.6
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: Eli Collins, Jordon Phillips, belopolsky, dstufft, lemburg, python-dev, vstinner
Priority: normal Keywords: 3.6regression, patch

Created on 2016-12-29 00:50 by Jordon Phillips, last changed 2017-03-31 16:36 by dstufft. This issue is now closed.

Files
File name Uploaded Description Edit
test.py Jordon Phillips, 2016-12-29 00:50
datetime_check_args.patch vstinner, 2017-01-03 14:07
unittest.patch vstinner, 2017-01-03 23:14
timestamp_limits.patch vstinner, 2017-01-04 16:39 review
timestamp_limits-2.patch vstinner, 2017-01-27 16:19 review
Pull Requests
URL Status Linked Edit
PR 552 closed dstufft, 2017-03-31 16:36
Messages (18)
msg284218 - (view) Author: Jordon Phillips (Jordon Phillips) Date: 2016-12-29 00:50
In Python 3.6.0 if you give datetime.fromtimestamp a very bad value you either get a core dump or an OverflowError. The core dump occurs when no tzinfo is provided, the OverflowError occurs when a tzinfo is provided (such as tzlocal from dateutil). Attached is a minimal script to reproduce the error. Note that this behavior only occurs on certain systems. It does not happen on OSX 10.11.6, but it does happen on Ubuntu 12.04. I imagine it happens on other systems as well, but I haven't tested beyond those two.

Here are the specific errors I get on Ubuntu 12.04. When no tzinfo is provided:

python: /tmp/python-build.20161228223921.28011/Python-3.6.0/Modules/_datetimemodule.c:251: days_before_year: Assertion `year >= 1' failed.
Aborted (core dumped)

When a tzinfo is provided:

Traceback (most recent call last):
  File "test1.py", line 11, in <module>
    datetime.fromtimestamp(bad_st_mtime, local_time_zone)
  File "~/.pyenv/versions/venv36d/lib/python3.6/site-packages/dateutil/tz/_common.py", line 210, in fromutc
    dt_wall = self._fromutc(dt)
  File "~/.pyenv/versions/venv36d/lib/python3.6/site-packages/dateutil/tz/_common.py", line 195, in _fromutc
    return dt + dtdst
OverflowError: date value out of range


I imagine this is related to the fold changes.
msg284563 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-01-03 14:07
Attached datetime_check_args.patch fixes the issue: move argument checks into time(), date() and datetime() constructors of the _datetime module.

The patch requires the fix of the issue #29140 (time_hash bug), otherwise unit tests fail.

This issue is a regression introduced the implementation of the PEP 495. In Python 3.5, datetime_from_timet_and_us() calls datetime constructor which checks argument. In Python 3.6, datetime_from_timet_and_us() calls directly new_datetime_ex2() which doesn't check arguments anymore.

datetime_check_args.patch makes the _datetime module generally safer, not only for the report bug, because new_datetime_ex2() can be called by different ways, for example from the C API: "PyDateTime_CAPI".

The bug always shows a major lack of unit test in datetime.datetime.fromtimestamp() on dates out of the 0001-01-01 .. 9999-12-31 range. In Python 3.6, datetime.fromtimestamp() can create invalid datetime object greater than datetime.max!

$ python3.6 -c 'import datetime; print(datetime.datetime.fromtimestamp(2**38))'
10680-07-14 08:09:04

TODO: add unit tests.
msg284600 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-01-03 23:12
Oh, datetime.datetime.timestamp() also has a bug for values close to 0001-01-01 (year 1), at least in my timezone (CET).

$ ./python -c 'import datetime; datetime.datetime.min.timestamp()'
python: /home/haypo/prog/python/default/Modules/_datetimemodule.c:251: days_before_year: Assertion `year >= 1' failed.
Aborted (core dumped)

datetime_check_args.patch doesn't fix this issue. A solution is to check that year>=1 in utc_to_seconds():

+    if (year < MINYEAR || year > MAXYEAR) {
+        PyErr_SetString(PyExc_ValueError,
+                        "year is out of range");
+        return -1;
+    }

Maybe it would solve the following local() comment:

   /* XXX: add bounds checking */
msg284601 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-01-03 23:14
unitttest.patch: my attempt to write an unit test, but I was bitten by the datetime.timestamp() bug.
msg284650 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-01-04 16:39
timestamp_limits.patch: Add many checks in the _datetime module to avoid overflows or creating a object out of the range. The patch adds unit tests.

The unit test checks dates around year 1 and year 9999. I expect failures on platforms which only work well on year in the range 1970..2038. My plan is to wait for buildbot reports to check on which platforms the test fails and with which exception.

Since the patch is likely to break buildbots, I will apply first to the default branch, fix it, and then backport to Python 3.6.
msg286361 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-01-27 14:58
The issue #29346 has been marked as a duplicate of this one.
msg286362 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-01-27 15:00
timestamp_limits.patch is waiting for your review! This issue is a major regression of Python 3.6. I'm going to push the patch in one week if I don't get any review.
msg286376 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-01-27 16:19
timestamp_limits-2.patch: catch also OverflowError in unit tests, and uses a context manager rather than self.assertRaises(..., func, ...) syntax.
msg287494 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2017-02-10 09:35
New changeset 383c0238b5b0 by Victor Stinner in branch '3.6':
Fix datetime.fromtimestamp(): check bounds
https://hg.python.org/cpython/rev/383c0238b5b0
msg287495 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-02-10 09:36
I pushed  timestamp_limits-2.patch. Thanks for the bug report Jordon Phillips, thanks for the review Serhiy Storshaka.
msg287500 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2017-02-10 10:00
New changeset 5b804b2fb0eaa2bacb4afcfe4cfa85b31475e87f by Victor Stinner in branch '3.6':
Fix datetime.fromtimestamp(): check bounds
https://github.com/python/cpython/commit/5b804b2fb0eaa2bacb4afcfe4cfa85b31475e87f
msg287501 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2017-02-10 10:00
New changeset 5b804b2fb0eaa2bacb4afcfe4cfa85b31475e87f by Victor Stinner in branch 'master':
Fix datetime.fromtimestamp(): check bounds
https://github.com/python/cpython/commit/5b804b2fb0eaa2bacb4afcfe4cfa85b31475e87f
msg287507 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2017-02-10 10:45
New changeset 1c6b87b07586 by Victor Stinner in branch '3.6':
Fix test_datetime on system with 32-bit time_t
https://hg.python.org/cpython/rev/1c6b87b07586
msg287513 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2017-02-10 11:00
New changeset a88f9614c672c3369cf03cdf51d012a42ae5665f by Victor Stinner in branch 'master':
Fix test_datetime on system with 32-bit time_t
https://github.com/python/cpython/commit/a88f9614c672c3369cf03cdf51d012a42ae5665f
msg287519 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2017-02-10 11:38
New changeset 1555e7776321 by Victor Stinner in branch '3.6':
Fix test_datetime on Windows
https://hg.python.org/cpython/rev/1555e7776321
msg287523 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2017-02-10 12:00
New changeset be363764cad68ad7b608ceed837cc01c2c3b4efc by Victor Stinner in branch '3.6':
Fix test_datetime on Windows
https://github.com/python/cpython/commit/be363764cad68ad7b608ceed837cc01c2c3b4efc
msg287524 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2017-02-10 12:00
New changeset be363764cad68ad7b608ceed837cc01c2c3b4efc by Victor Stinner in branch 'master':
Fix test_datetime on Windows
https://github.com/python/cpython/commit/be363764cad68ad7b608ceed837cc01c2c3b4efc
msg290708 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-03-28 13:11
m-parry reported the issue #29921 "datetime validation is stricter in 3.6.1 than previous versions". He is right, the C API of Python 3.6.1 is now stricter than Python 2.7 and 3.5. The C API doesn't allow anymore to create datetime objects outside the [datetime.datetime.min; datetime.datetime.max] range (what I would call "invalid" datetime objects). I closed the bug as NOT A BUG since I consider that it's a deliberate design choice and not a regression.
History
Date User Action Args
2017-03-31 16:36:38dstufftsetpull_requests: + pull_request1108
2017-03-28 13:11:33vstinnersetmessages: + msg290708
2017-02-10 12:00:31python-devsetmessages: + msg287524
2017-02-10 12:00:29python-devsetmessages: + msg287523
2017-02-10 11:38:32python-devsetmessages: + msg287519
2017-02-10 11:00:22python-devsetmessages: + msg287513
2017-02-10 10:45:51python-devsetmessages: + msg287507
2017-02-10 10:00:23python-devsetmessages: + msg287501
2017-02-10 10:00:22python-devsetmessages: + msg287500
2017-02-10 09:36:06vstinnersetstatus: open -> closed
resolution: fixed
messages: + msg287495

stage: resolved
2017-02-10 09:35:02python-devsetnosy: + python-dev
messages: + msg287494
2017-01-27 16:19:32vstinnersetfiles: + timestamp_limits-2.patch

messages: + msg286376
2017-01-27 15:00:42vstinnersetmessages: + msg286362
2017-01-27 14:58:33vstinnersetmessages: + msg286361
2017-01-27 14:58:14vstinnersetnosy: + lemburg, Eli Collins
2017-01-27 14:58:10vstinnerlinkissue29346 superseder
2017-01-04 16:39:03vstinnersetfiles: + timestamp_limits.patch

messages: + msg284650
2017-01-03 23:14:17vstinnersetfiles: + unittest.patch

messages: + msg284601
2017-01-03 23:12:41vstinnersetmessages: + msg284600
2017-01-03 14:08:06vstinnersetkeywords: + 3.6regression
2017-01-03 14:07:45vstinnersettitle: datetime.fromtimestamp() does crash for very small or very big timestamp -> datetime.fromtimestamp() doesn't check min/max year anymore: regression of Python 3.6
2017-01-03 14:07:25vstinnersetfiles: + datetime_check_args.patch


title: Core dump / OverflowError for datetime.fromtimestamp with overly large timestamp in Ubuntu 12.04 -> datetime.fromtimestamp() does crash for very small or very big timestamp
keywords: + patch
nosy: + vstinner
versions: + Python 3.7
messages: + msg284563
2016-12-29 04:31:10xiang.zhangsetnosy: + belopolsky
2016-12-29 00:52:25dstufftsetnosy: + dstufft
2016-12-29 00:50:47Jordon Phillipscreate