classification
Title: datetime validation is stricter in 3.6.1 than previous versions
Type: behavior Stage: resolved
Components: Library (Lib) Versions: Python 3.6
process
Status: closed Resolution: not a bug
Dependencies: Superseder:
Assigned To: Nosy List: belopolsky, m-parry, ned.deily, vstinner
Priority: normal Keywords:

Created on 2017-03-27 13:48 by m-parry, last changed 2017-03-28 14:19 by ned.deily. This issue is now closed.

Messages (9)
msg290609 - (view) Author: (m-parry) Date: 2017-03-27 13:48
The change in issue #29100 - intended AFAICS simply to fix a regression in 3.6 - seems to have made datetime validation via certain code paths stricter than it was in 2.7 or 3.5. I think it's the case that some routes via the C API now reject out of range values that were previously permitted. Even if this previous behaviour was incorrect, was it intentional to alter that in a maintenance release?

Here's a quick example using pywin32:

---

> import getpass, sspi, sspicon, win32security
> client_name = getpass.getuser()
> auth_info = (client_name, 'wherever.com', None)
> pkg_info = win32security.QuerySecurityPackageInfo('Kerberos')
> win32security.AcquireCredentialsHandle(
>     client_name, pkg_info['Name'],
>     sspicon.SECPKG_CRED_OUTBOUND,
>     None, auth_info)

ValueError: year 30828 is out of range

---

Of course, this is probably a mishandling of the 'never expires' value returned by the Windows API in this case, and indeed I have also created a pywin32 ticket. However, I'm guessing that the linked issue wasn't supposed to break such code.
msg290646 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-03-27 20:55
> The change in issue #29100 - intended AFAICS simply to fix a regression in 3.6 - seems to have made datetime validation via certain code paths stricter than it was in 2.7 or 3.5.

What do you mean by "stricter than 2.7 & 3.5"? The year 30828 was never valid.

Python 2.7 and 3.5:

>>> datetime.datetime(30828, 1, 1)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ValueError: year is out of range
>>> datetime.datetime.fromtimestamp(2**37)
datetime.datetime(6325, 4, 8, 17, 4, 32)
>>> datetime.datetime.fromtimestamp(2**38)
Traceback (most recent call last):
  ...
ValueError: year is out of range

Python 3.6.1 should only be stricter than Python 3.6.0.
msg290647 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-03-27 20:57
The range of valid timestamp is the same since the module was added to Python 2.3:

>>> datetime.datetime.min
datetime.datetime(1, 1, 1, 0, 0)
>>> datetime.datetime.max
datetime.datetime(9999, 12, 31, 23, 59, 59, 999999)

https://docs.python.org/2/library/datetime.html#datetime.datetime.max
https://docs.python.org/dev/library/datetime.html#datetime.datetime.max
msg290688 - (view) Author: (m-parry) Date: 2017-03-28 08:17
From my opening comment (with new emphasis):

"I think it's the case that **some routes via the C API** now reject out of range values that were previously permitted."

The pywin32 repro I gave above eventually calls PyDateTimeAPI->DateTime_FromDateAndTime():

http://pywin32.hg.sourceforge.net/hgweb/pywin32/pywin32/file/85c1c99b1cb8/win32/src/PyTime.cpp#l980

AFAICT, under Python < 3.6.1 such out of range values were tolerated there. Under Python 2.7, for example, the datetime that results from this call is somewhere in 1899.

I am not claiming that these invalid values should be tolerated forever more, or that this was ever the correct behaviour, but I would have expected a backwards incompatible change like this to happen in, say, Python 3.7, rather than a maintenance release. (Particularly when it breaks a library that's practically standard on Windows.)
msg290698 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-03-28 11:07
2017-03-28 10:17 GMT+02:00 m-parry <report@bugs.python.org>:
> "I think it's the case that **some routes via the C API** now reject out of range values that were previously permitted."
>
> The pywin32 repro I gave above eventually calls PyDateTimeAPI->DateTime_FromDateAndTime():
>
> http://pywin32.hg.sourceforge.net/hgweb/pywin32/pywin32/file/85c1c99b1cb8/win32/src/PyTime.cpp#l980

You can please identify which C function is called and dump which
parameters are passed to the function?
msg290699 - (view) Author: (m-parry) Date: 2017-03-28 12:23
That's just a Python C API call. It looks like it eventually resolves to new_datetime_ex(30828, 9, 13, 3, 48, 5, 480000, Py_None, PyDateTime_DateTimeType).

We also have some internal code that sees a similar problem from calling PyTime_FromTime(), that was similarly affected by this change. In one example, that appears to resolve to new_time_ex(24, 0, 0, 0, Py_None, PyDateTime_TimeType). Again, under <3.6.1, that was accepted. (And again, I am making no argument about the validity of this code, just with regards to backwards compatibility.)
msg290707 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-03-28 13:09
> Again, under <3.6.1, that was accepted. (And again, I am making no argument about the validity of this code, just with regards to backwards compatibility.)

You are right that I modified the C API of datetime in Python 3.6.1 to make it stricter and reject invalid dates.

I disagree that the "backward incompatibility" part: I consider that it's a bugfix, and not a behaviour change.

The datetime module was enhanced in Python 3.6 with the PEP 495 to handle better DST changes: a new "fold" attribute was added. Computing this attribute requires to handle valid dates in the [datetime.datetime.min; datetime.datetime.max] range. Otherwise, you get strange errors like OverflowError: see issue #29100.

If Python older than 3.6.1 allowed creating invalid dates, it was a bug, and now this bug can lead to new bugs because of the implementation of the PEP 495.

Please fix you code. I now close this issue as NOTABUG. Stricter input validation was a deliberate choice.
msg290709 - (view) Author: (m-parry) Date: 2017-03-28 13:12
pywin32 is not my code. It is a ubiquitous Python library on Windows that cannot be used under Python 3.6.1.
msg290721 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2017-03-28 14:19
FTR, there is now a pywin32 issue (opened by the OP) on this:  https://sourceforge.net/p/pywin32/bugs/748/
History
Date User Action Args
2017-03-28 14:19:14ned.deilysetnosy: + ned.deily
messages: + msg290721
2017-03-28 13:12:52m-parrysetmessages: + msg290709
2017-03-28 13:09:58vstinnersetstatus: open -> closed

nosy: + belopolsky
messages: + msg290707

resolution: not a bug
stage: resolved
2017-03-28 12:23:58m-parrysetmessages: + msg290699
2017-03-28 11:07:07vstinnersetmessages: + msg290698
2017-03-28 08:17:34m-parrysetmessages: + msg290688
2017-03-27 20:57:29vstinnersetmessages: + msg290647
2017-03-27 20:55:25vstinnersetmessages: + msg290646
2017-03-27 15:39:38xiang.zhangsetnosy: + vstinner
2017-03-27 13:48:47m-parrycreate