classification
Title: imaplib: Internaldate2tuple produces wrong result if date is near a DST change
Type: behavior Stage: resolved
Components: Library (Lib) Versions: Python 3.4, Python 3.2, Python 3.3, Python 2.7
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: belopolsky Nosy List: belopolsky, georg.brandl, lavajoe, python-dev, r.david.murray
Priority: normal Keywords: patch

Created on 2011-01-18 23:41 by lavajoe, last changed 2014-06-29 20:25 by r.david.murray. This issue is now closed.

Files
File name Uploaded Description Edit
imaplib_Internaldate2tuple_dst_fix_python32.patch lavajoe, 2011-01-18 23:46
imaplib_Internaldate2tuple_dst_fix_python27.patch lavajoe, 2011-01-18 23:47 review
issue10941.diff lavajoe, 2011-01-30 02:08 review
issue10941_python3.diff lavajoe, 2012-04-20 06:08 review
issue10941_python2.diff lavajoe, 2012-04-21 00:55 review
issue10941.diff belopolsky, 2012-06-22 23:58 review
Messages (27)
msg126503 - (view) Author: Joe Peterson (lavajoe) Date: 2011-01-18 23:41
DST is not handled correctly.  Specifically, when the input date/time, ignoring the TZ offset (and treated as if it is local time) is on the other side of the DST changeover from the actual local time represented, the result will be off by one hour.  This can happen, e.g., when the input date/time is actually UTC (offset +0000).

This is because the check for DST is done after converting the time into a local time tuple, thereby treating as real local time.  This can be corrected by keeping the time in real UTC (by using calendar.timegm() instead of checking the DST flag) until the final conversion to local time.

Here is an example:  Run the following two dates, that represent exactly the same time, through Internaldate2tuple:

'25 (INTERNALDATE "01-Apr-2000 19:02:23 -0700")'
'101 (INTERNALDATE "02-Apr-2000 02:02:23 +0000")'

Note that a variant of this issue (but identifying it as a different problem) was addressed in a similar way in an old post I found by Colin Brown in 2004 (http://www.velocityreviews.com/forums/t336162-imaplib-function-bug.html).

Patch also adds unit test to check the above example dates.  Python 3 version of patch assumes patch from issue 10939 has been applied.

Patch also corrects code python doc that currently states time is UT, not local (see issue 10934).
msg126573 - (view) Author: Alexander Belopolsky (belopolsky) * (Python committer) Date: 2011-01-19 22:50
I agree that this is a bug and the solution is right, I am not sure this should be applied during RC period.  The bug has been present and known for a long time, so affected parties probably have a work-around in place already.
msg127490 - (view) Author: Joe Peterson (lavajoe) Date: 2011-01-29 21:45
Here is a new patch that adds a test to the tests committed for issue 10939.

This new test case is needed to trigger the DST issue.  This test is not timezone-dependent (in the sense that one does not have to have a specific timezone set for it to work), but it does depend on the DST change happening at 02:00 on the first Sunday in April, 2000.  This is the case for most of the United States, but to make it work everywhere, I have added code that sets the TZ to one that works, and then reverts the temporary TZ change after the test.

The bug is triggered when the time described in the internal date string, ignoring the time offset, is between 02:00 and 03:00.  This is because the raw time in the string is interpreted as local time, and local times in this range are basically invalid, since time advances to 03:00 when 02:00 is reached because of the DST change.  This causes the final result to be off by one hour.

[This patch is only for Python 3]
msg127502 - (view) Author: Alexander Belopolsky (belopolsky) * (Python committer) Date: 2011-01-30 00:48
On Sat, Jan 29, 2011 at 4:45 PM, Joe Peterson <report@bugs.python.org> wrote:
..
> I have added code that sets the TZ to one that works, and then reverts the temporary
> TZ change after the test.
>

Your test will not restore TZ if it fails.  This is not nice.  To fix
that, code restoring the TZ should go into a finally clause.  Even
better, set TZ in a setUp() method and restore it in a tearDown()
method of the test case class.  Ultimately, if you want to be fancy,
take a look at the @run_with_locale decorator defined in
Lib/test/support.py.  We may have more than one use for @run_with_tz.
msg127507 - (view) Author: Joe Peterson (lavajoe) Date: 2011-01-30 02:08
I like the idea of adding the decorator.  New patch added.
msg127508 - (view) Author: Joe Peterson (lavajoe) Date: 2011-01-30 02:28
[just carrying over some info from issue 10939 that is related to this issue]

Here is another manifestation of this issue, related to the local time assumption, but not to DST, per se:

Here is the definition for Europe/London in the unix tz data:

# Zone	NAME		GMTOFF	RULES	FORMAT	[UNTIL]
Zone	Europe/London	-0:01:15 -	LMT	1847 Dec  1 0:00s
			 0:00	GB-Eire	%s	1968 Oct 27
			 1:00	-	BST	1971 Oct 31 2:00u
			 0:00	GB-Eire	%s	1996
			 0:00	EU	GMT/BST

So London's local time was always 1 hour ahead of UTC (BST time) from 1968 Oct 27 until 1971 Oct 31 2:00u.

Because of this, Internaldate2tuple() returns the wrong local time (00:00 rather than [the correct] 01:00) in its tuple when input is "01-Jan-1970 00:00:00 +0000" (the epoch).
msg158645 - (view) Author: Joe Peterson (lavajoe) Date: 2012-04-18 18:16
It's been over a year since any activity on this bug, and it is still in the "patch review" stage.  Any news?  Anything else I can provide to help to get this moved to the next stage?  Thanks!
msg158646 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2012-04-18 18:26
Someone needs to find time to review it.

You could try recruiting reviewers on python-list.  Anyone can do a review.  Obviously the more knowledge they have in this area the better, but any good review is likely to move the issue along.
msg158661 - (view) Author: Joe Peterson (lavajoe) Date: 2012-04-18 20:53
Ah.  I figured that one of the Python devs would be who would review it.  Is this the normal path for bugs?  Not to question the process, but I find it surprising, since I would think that it would cause a lot of bugs to stall out.  Also, I had no idea it was the submitter's responsibility to recruit reviewers.  And I am not quite sure how to go about this.  I do care about this bug (as it's pretty serious that it returns wrong results), so I am willing to do it.  Any wiki page or anything that describes the recruiting process or has more info?
msg158662 - (view) Author: Alexander Belopolsky (Alexander.Belopolsky) Date: 2012-04-18 20:59
I'll review your patch. 

On Apr 18, 2012, at 4:53 PM, Joe Peterson <report@bugs.python.org> wrote:

> 
> Joe Peterson <joe@skyrush.com> added the comment:
> 
> Ah.  I figured that one of the Python devs would be who would review it.  Is this the normal path for bugs?  Not to question the process, but I find it surprising, since I would think that it would cause a lot of bugs to stall out.  Also, I had no idea it was the submitter's responsibility to recruit reviewers.  And I am not quite sure how to go about this.  I do care about this bug (as it's pretty serious that it returns wrong results), so I am willing to do it.  Any wiki page or anything that describes the recruiting process or has more info?
> 
> ----------
> 
> _______________________________________
> Python tracker <report@bugs.python.org>
> <http://bugs.python.org/issue10941>
> _______________________________________
msg158663 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2012-04-18 21:05
I see Alexander is going to take care of this.  But to clarify what I suggested for your information:

In an ideal world it would be a committer doing the patch review, followed by a checkin.  But in the real world there aren't enough of us with enough time to get to all the bugs with patches.  You asked how to move it along and I suggested one way: get someone else to do a review.  I wouldn't say that the submitter recruiting a reviewer was a "normal" process, but it is a way to get bugs unstuck.  And we get reviews from non-committers frequently (it is a step along the path to becoming a committer...quality reviews are as important as quality patches).

I don't think there's anything in the devguide about this particular nuance.
msg158664 - (view) Author: Joe Peterson (lavajoe) Date: 2012-04-18 21:05
Thanks!!  :)
msg158665 - (view) Author: Joe Peterson (lavajoe) Date: 2012-04-18 21:16
David, I understand - thanks for the details.  Hopefully I can return the favor and review one in the future.
msg158733 - (view) Author: Alexander Belopolsky (belopolsky) * (Python committer) Date: 2012-04-19 15:14
Joe,

Your changes to the test suit don't apply cleanly anymore.  I can probably fix the conflicts, but if you could post an updated patch it will help.

Thanks.
msg158798 - (view) Author: Joe Peterson (lavajoe) Date: 2012-04-20 06:08
OK, fixed patch to apply cleanly to current code.  BTW, this is only for python3.  Is it still appropriate to patch python2?  And if so, what is the correct code repo to check out for that?
msg158901 - (view) Author: Joe Peterson (lavajoe) Date: 2012-04-21 00:55
I have now included a patch for 2.7.  Here are the two latest patches:

Python 2: issue10941_python2.diff
Python 3: issue10941_python3.diff
msg159105 - (view) Author: Alexander Belopolsky (belopolsky) * (Python committer) Date: 2012-04-24 00:42
I still get conflicts:


$ patch -p0 < imaplib_Internaldate2tuple_dst_fix_python32.patch 
patching file Lib/imaplib.py
Hunk #2 FAILED at 1312.
Hunk #3 succeeded at 1347 (offset 8 lines).
Hunk #4 FAILED at 1379.
2 out of 4 hunks FAILED -- saving rejects to file Lib/imaplib.py.rej
patching file Lib/test/test_imaplib.py
Hunk #1 FAILED at 24.
1 out of 1 hunk FAILED -- saving rejects to file Lib/test/test_imaplib.py.rej


Are you sure your patch is against the head of the hg repository?
msg159112 - (view) Author: Joe Peterson (lavajoe) Date: 2012-04-24 02:21
The one you tried is the old patch.  Here are the two new patches.  Use these:

Python 2: issue10941_python2.diff
Python 3: issue10941_python3.diff
msg159144 - (view) Author: Alexander Belopolsky (belopolsky) * (Python committer) Date: 2012-04-24 14:36
My bad.  For some reason I assumed that the latest patch would be at the top of the files list.

David, the patch is good.  Should I go ahead and commit?
msg159146 - (view) Author: Joe Peterson (lavajoe) Date: 2012-04-24 14:42
Great to hear, Alexander.  Thanks for reviewing!  Is it also possible to get the pyhton2.7 version one in?
msg159148 - (view) Author: Alexander Belopolsky (belopolsky) * (Python committer) Date: 2012-04-24 14:47
On Tue, Apr 24, 2012 at 10:42 AM, Joe Peterson <report@bugs.python.org> wrote:
..
>  Is it also possible to get the pyhton2.7 version one in?

I don't see any reason not to.  This is a bug fix and should go into a
maintenance release.  I will wait to hear from David, who is the
maintainer of this module, though.
msg159151 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2012-04-24 15:04
If you are satisfied with the time logic then yes, please apply it.  I suspect that the number of people using the code and not aware of the problem (or not caring enough to do anything about it) exceeds the number aware of it who have coded a workaround, so I think putting it in 2.7 (and 3.2) is better than not doing so.
msg159649 - (view) Author: Roundup Robot (python-dev) Date: 2012-04-29 20:42
New changeset 90b9781ccb5f by Alexander Belopolsky in branch '3.2':
Issue #10941: Fix imaplib.Internaldate2tuple to produce correct result near
http://hg.python.org/cpython/rev/90b9781ccb5f

New changeset 6e029b6c142a by Alexander Belopolsky in branch 'default':
Issue #10941: Fix imaplib.Internaldate2tuple to produce correct result near
http://hg.python.org/cpython/rev/6e029b6c142a
msg163505 - (view) Author: Alexander Belopolsky (belopolsky) * (Python committer) Date: 2012-06-22 23:58
I updated the fix to take advantage of resolved issue 9527.  I also noticed and fixed another bug: Internaldate2tuple was using locale-dependent %b directive for strftime.
msg163510 - (view) Author: Alexander Belopolsky (belopolsky) * (Python committer) Date: 2012-06-23 00:56
The latest patch belongs to issue 11024.
msg221896 - (view) Author: Alexander Belopolsky (belopolsky) * (Python committer) Date: 2014-06-29 20:18
David,

Is there anything left to do here that is not covered by issue 11024?
msg221897 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2014-06-29 20:25
Not that I can see.
History
Date User Action Args
2014-06-29 20:25:14r.david.murraysetstatus: open -> closed
resolution: fixed
messages: + msg221897

stage: patch review -> resolved
2014-06-29 20:18:32belopolskysetmessages: + msg221896
2012-06-23 00:56:58belopolskysetmessages: + msg163510
2012-06-22 23:59:00belopolskysetfiles: + issue10941.diff

messages: + msg163505
2012-06-22 18:31:28belopolskysetassignee: belopolsky
2012-04-29 20:42:09python-devsetnosy: + python-dev
messages: + msg159649
2012-04-24 15:04:57r.david.murraysetmessages: + msg159151
2012-04-24 14:47:32belopolskysetmessages: + msg159148
2012-04-24 14:42:24lavajoesetmessages: + msg159146
2012-04-24 14:36:54belopolskysetmessages: + msg159144
2012-04-24 02:21:12lavajoesetmessages: + msg159112
2012-04-24 00:42:33belopolskysetmessages: + msg159105
2012-04-21 00:55:39lavajoesetfiles: + issue10941_python2.diff

messages: + msg158901
2012-04-20 06:08:57lavajoesetfiles: + issue10941_python3.diff

messages: + msg158798
versions: + Python 3.4
2012-04-19 15:14:27belopolskysetnosy: - Alexander.Belopolsky
messages: + msg158733
2012-04-18 21:16:52lavajoesetmessages: + msg158665
2012-04-18 21:05:20lavajoesetmessages: + msg158664
2012-04-18 21:05:19r.david.murraysetmessages: + msg158663
2012-04-18 20:59:43Alexander.Belopolskysetnosy: + Alexander.Belopolsky
messages: + msg158662
2012-04-18 20:53:43lavajoesetmessages: + msg158661
2012-04-18 18:26:51r.david.murraysetnosy: + r.david.murray
messages: + msg158646
2012-04-18 18:16:44lavajoesetmessages: + msg158645
2011-01-30 02:28:45lavajoesetnosy: georg.brandl, belopolsky, lavajoe
messages: + msg127508
2011-01-30 02:08:59lavajoesetfiles: + issue10941.diff
nosy: georg.brandl, belopolsky, lavajoe
messages: + msg127507
2011-01-30 02:08:33lavajoesetfiles: - issue10941.diff
nosy: georg.brandl, belopolsky, lavajoe
2011-01-30 00:48:55belopolskysetnosy: georg.brandl, belopolsky, lavajoe
messages: + msg127502
2011-01-29 21:45:14lavajoesetfiles: + issue10941.diff
nosy: georg.brandl, belopolsky, lavajoe
messages: + msg127490
2011-01-19 22:50:20belopolskysetversions: + Python 3.3
nosy: + georg.brandl, belopolsky

messages: + msg126573

stage: patch review
2011-01-18 23:47:06lavajoesetfiles: + imaplib_Internaldate2tuple_dst_fix_python27.patch
2011-01-18 23:46:45lavajoesetfiles: + imaplib_Internaldate2tuple_dst_fix_python32.patch
keywords: + patch
2011-01-18 23:41:44lavajoecreate