classification
Title: datetime operations spanning MINYEAR give bad results
Type: behavior Stage: resolved
Components: Library (Lib) Versions: Python 3.3, Python 3.2, Python 2.7, Python 2.6
process
Status: closed Resolution: accepted
Dependencies: Superseder:
Assigned To: belopolsky Nosy List: amaury.forgeotdarc, belopolsky, mark.dickinson, mark.leander, pythonhacker
Priority: normal Keywords: patch

Created on 2009-10-16 09:45 by mark.leander, last changed 2010-05-27 22:08 by belopolsky. This issue is now closed.

Files
File name Uploaded Description Edit
datetimemodule.c.svndiff pythonhacker, 2009-10-25 09:29
issue7150.diff Alexander.Belopolsky, 2010-02-25 22:36
issue7150a.diff belopolsky, 2010-05-26 05:03
Messages (15)
msg94132 - (view) Author: (mark.leander) Date: 2009-10-16 09:45
The datetime module documentation would imply that operations that cause
dates to fall outside the MINYEAR--MAXYEAR range should raise
OverflowError. The interpreter session below shows that this is not
always the case, and that such operations may cause bogus and
inconsistent results.

Python 2.6.3 (r263rc1:75186, Oct  2 2009, 20:40:30) [MSC v.1500 32 bit
(Intel)]
on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import datetime
>>> t0=datetime.datetime(1,1,1)
>>> d1, d2, d3 = map(datetime.timedelta, range(1,4))
# The following is expected and accoring to the docs:
>>> t0-d1
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
OverflowError: date value out of range

# The following is completely bogus:
>>> t0-d2
datetime.datetime(1, 0, 255, 0, 0)

# The two following behaving differently may be very confusing,
# the second one is correct
>>> t0-d2+d3
datetime.datetime(1, 8, 15, 0, 0)
>>> t0+d3-d2
datetime.datetime(1, 1, 2, 0, 0)
>>>
msg94384 - (view) Author: Anand B Pillai (pythonhacker) Date: 2009-10-23 13:06
The issue is present in Python 3.0 and 2.5 as well.

Python 2.5.1 (r251:54863, Jul 17 2008, 13:21:31)
[GCC 4.3.1 20080708 (Red Hat 4.3.1-4)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import datetime
>>> t0=datetime.datetime(1,1,1)
>>> d1,d2,d3=map(datetime.timedelta, range(1,4))
>>> t0-d1
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
OverflowError: date value out of range
>>> t0-d2
datetime.datetime(1, 0, 255, 0, 0)

I think this is bug in datetime for all Python versions
msg94449 - (view) Author: Anand B Pillai (pythonhacker) Date: 2009-10-25 09:29
The problem seems to be in the "normalize_date" function in
datetimemodule.c. It is checking for a valid year range, but not
checking for a valid month or day range. 

I have a patch which fixes this problem. It checks for month range
(1<=m<=12) and day range(1<=d<=31). Here is Python with the patch.

anand@anand-laptop:~/projects/python/py3k$ ./python
Python 3.2a0 (py3k:75627, Oct 25 2009, 14:28:21) 
[GCC 4.3.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
Traceback (most recent call last):
  File "/home/anand/.pythonrc", line 2, in <module>
    import readline
ImportError: No module named readline
>>> import datetime
>>> t0=datetime.datetime(1,1,1)
>>> d1,d2,d3=map(datetime.timedelta, range(1,4))
>>> t0-d1
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
OverflowError: date value out of range
>>> t0-d2
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
OverflowError: date value out of range
>>> t0-d3
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
OverflowError: date value out of range
>>> d0=datetime.timedelta(0)
>>> t0-d0
datetime.datetime(1, 1, 1, 0, 0)
>>> 

Svn diff is attached.
msg100102 - (view) Author: Amaury Forgeot d'Arc (amaury.forgeotdarc) * (Python committer) Date: 2010-02-25 20:39
The patch seems OK, but needs tests.
msg100104 - (view) Author: Alexander Belopolsky (Alexander.Belopolsky) Date: 2010-02-25 21:20
Given

        assert(*m > 0);
        assert(*d > 0);

at the end of normalize_y_m_d(), it looks like at lest 1 <=*month and 1 <=*day are redundant.

A closer look also reveals 

        assert(1 <= *m && *m <= 12);

in the middle of normalize_y_m_d().  This seems to leave only *day <=31 possibly relevant.

I suspect that out of bounds day surviving normalize_y_m_d() is a logical error in that function that needs to be fixed and an assert() added at the end.  The proposed patch appears to cure the symptom rather than the actual flaw.
msg100105 - (view) Author: Alexander Belopolsky (Alexander.Belopolsky) Date: 2010-02-25 21:24
Aha!  My reliance on asserts() was misguided.  With the debug build:

>>> t0-d2
Assertion failed: (ordinal >= 1), function ord_to_ymd, file /Users/sasha/Work/python-svn/trunk/Modules/datetimemodule.c, line 269.
Abort trap

Should we reclassify this bug as a crash?
msg100107 - (view) Author: Amaury Forgeot d'Arc (amaury.forgeotdarc) * (Python committer) Date: 2010-02-25 21:57
No, because normally distributions do not use debug builds.
but that's the reason why tests are needed: they must pass with asserts enabled.
msg100118 - (view) Author: Alexander Belopolsky (Alexander.Belopolsky) Date: 2010-02-25 22:36
I am attaching my variant of the patch including additional unit tests.

Note that my changes make normalize_y_m_d() and normalize_date() the same, but I am leaving the current structure intact to make reviewer's job easier.
msg102945 - (view) Author: Anand B Pillai (pythonhacker) Date: 2010-04-12 13:25
Can someone update this issue ? Is the 2nd patch tested... ?
msg103718 - (view) Author: Alexander Belopolsky (Alexander.Belopolsky) Date: 2010-04-20 14:27
My patch includes unit tests and I tested it on Mac OS X.  Anand, what kind of testing do you have in mind?
msg106509 - (view) Author: Alexander Belopolsky (belopolsky) * (Python committer) Date: 2010-05-26 05:03
I've untabified my last patch and added a NEWS entry.  I believe it is ready for commit review.  Mark?
msg106511 - (view) Author: Mark Dickinson (mark.dickinson) * (Python committer) Date: 2010-05-26 07:28
I'll take a look at this patch later today.
msg106529 - (view) Author: Mark Dickinson (mark.dickinson) * (Python committer) Date: 2010-05-26 15:02
The patch looks good to me.  Please apply!
msg106531 - (view) Author: Mark Dickinson (mark.dickinson) * (Python committer) Date: 2010-05-26 15:20
As an aside, I dislike the fact that the datetime module uses a C 'int' for date ordinals, and clearly assumes that it'll be at least 32 bits.  int could be as small as 16 bits on some systems (small embedded systems?).  But that's another issue.
msg106618 - (view) Author: Alexander Belopolsky (belopolsky) * (Python committer) Date: 2010-05-27 22:08
Committed in r81566 (trunk), r81568 (py3k), r81569 (release26-maint), r81570 (release31-maint).
History
Date User Action Args
2010-09-23 22:31:45amaury.forgeotdarclinkissue9888 superseder
2010-05-27 22:08:58belopolskysetstatus: open -> closed

messages: + msg106618
stage: commit review -> resolved
2010-05-26 15:20:26mark.dickinsonsetmessages: + msg106531
2010-05-26 15:02:40mark.dickinsonsetassignee: mark.dickinson -> belopolsky
resolution: accepted
messages: + msg106529
2010-05-26 07:28:37mark.dickinsonsetassignee: belopolsky -> mark.dickinson
messages: + msg106511
2010-05-26 05:03:11belopolskysetfiles: + issue7150a.diff

nosy: + mark.dickinson
messages: + msg106509

stage: patch review -> commit review
2010-05-26 00:55:33belopolskysetassignee: belopolsky
nosy: + belopolsky, - Alexander.Belopolsky
2010-04-20 14:27:56Alexander.Belopolskysetmessages: + msg103718
versions: + Python 2.7, Python 3.2, Python 3.3
2010-04-12 13:25:34pythonhackersetmessages: + msg102945
2010-02-25 22:36:50Alexander.Belopolskysetfiles: + issue7150.diff
keywords: + patch
messages: + msg100118
2010-02-25 21:57:27amaury.forgeotdarcsetmessages: + msg100107
2010-02-25 21:24:53Alexander.Belopolskysetmessages: + msg100105
2010-02-25 21:20:36Alexander.Belopolskysetnosy: + Alexander.Belopolsky
messages: + msg100104
2010-02-25 20:39:23amaury.forgeotdarcsetnosy: + amaury.forgeotdarc

messages: + msg100102
stage: patch review
2009-10-25 09:29:51pythonhackersetfiles: + datetimemodule.c.svndiff

messages: + msg94449
2009-10-23 13:06:01pythonhackersetnosy: + pythonhacker
messages: + msg94384
2009-10-16 09:45:28mark.leandercreate