classification
Title: imaplib: Internaldate2tuple raises KeyError parsing month and does not work with negative TZ offset due to bytes/str issues
Type: behavior Stage: resolved
Components: Library (Lib) Versions: Python 3.1, Python 3.2
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: georg.brandl Nosy List: belopolsky, benjamin.peterson, georg.brandl, lavajoe
Priority: release blocker Keywords: patch

Created on 2011-01-18 22:31 by lavajoe, last changed 2011-01-29 23:07 by belopolsky. This issue is now closed.

Files
File name Uploaded Description Edit
imaplib_Internaldate2tuple_bytes_fixes_python32.patch lavajoe, 2011-01-28 22:22
issue10939.diff belopolsky, 2011-01-29 16:35
Messages (12)
msg126499 - (view) Author: Joe Peterson (lavajoe) Date: 2011-01-18 22:31
There are two issues with conversion to Python 3:

1. It raise "KeyError".  This is because the Mon2num dictionary keys are strings (str), not bytes objects (note that many other strings in imaplib have been updated, but not Mon2num).

2. The sign character of the TZ offset (e.g. -0700) is compared to the string (str) '-', not bytes array b'-', so the compare is never true, causing a large error when the TZ offset is negative.

Patch attached that also adds a unit test.
msg126572 - (view) Author: Alexander Belopolsky (belopolsky) * (Python committer) Date: 2011-01-19 22:35
Georg,

This is an obvious artifact of an untested py3k port.  Can this go in 3.2?

The patch looks good except for a long line in the test which I can take care of.
msg127393 - (view) Author: Georg Brandl (georg.brandl) * (Python committer) Date: 2011-01-29 09:50
The new test is faulty; it appears to be specific to the timezone of the patch submitter.

The library fix should go in nevertheless, if you could add a correct test, Alexander, it would be great.
msg127437 - (view) Author: Alexander Belopolsky (belopolsky) * (Python committer) Date: 2011-01-29 16:33
Attached patch fixes the test (hopefully we don't have to support systems with non-POSIX epoch) and cleans up whitespace.
msg127444 - (view) Author: Alexander Belopolsky (belopolsky) * (Python committer) Date: 2011-01-29 17:19
Committed in revision 88231.
msg127462 - (view) Author: Alexander Belopolsky (belopolsky) * (Python committer) Date: 2011-01-29 18:45
Merged to 3.1 in r88233.
msg127470 - (view) Author: Joe Peterson (lavajoe) Date: 2011-01-29 19:16
Good catch on the test.  Note that for issue 10941, we'll want a new non-timezone-dependent test case that can catch the DST-related problem.  I'll post a new patch to issue 10941 now and describe this some more there.
msg127471 - (view) Author: Alexander Belopolsky (belopolsky) * (Python committer) Date: 2011-01-29 19:20
It looks like my test is not robust enough:

http://www.python.org/dev/buildbot/all/builders/x86%20OpenIndiana%203.1/builds/298
msg127485 - (view) Author: Alexander Belopolsky (belopolsky) * (Python committer) Date: 2011-01-29 20:37
Some systems don't like times too close to epoch.  Fixed in revision 88239.
msg127489 - (view) Author: Joe Peterson (lavajoe) Date: 2011-01-29 21:38
Alexander, looks like you hit a weirdness in the Europe/London timezone for dates before 31-Oct-1971, as if DST was in effect even in winter.  And the fail of the test is caused by the same bug that causes issue 10941: the use of mktime to interpret the values from an internal time string (which should not be dealing with timezones, but rather explicit offsets from UTC, before converting to localtime).
msg127494 - (view) Author: Joe Peterson (lavajoe) Date: 2011-01-29 22:47
More info on the Europe/London "near the epoch" thing (there is no weirdness in the tz stuff - it's just issue 10941 causing the test to fail)...

I looked at the sources for the tz data, and here is the definition for Europe/London:

# Zone	NAME		GMTOFF	RULES	FORMAT	[UNTIL]
Zone	Europe/London	-0:01:15 -	LMT	1847 Dec  1 0:00s
			 0:00	GB-Eire	%s	1968 Oct 27
			 1:00	-	BST	1971 Oct 31 2:00u
			 0:00	GB-Eire	%s	1996
			 0:00	EU	GMT/BST

I appears that London was always 1 hour ahead (BST time) from 1968 Oct 27 until 1971 Oct 31 2:00u, thus triggering issue 10941:

Internaldate2tuple() actually returns the wrong local time (00:00 rather than [the correct] 01:00) in its tuple for the epoch, so mktime(), doing the right thing, returns -3600.  The patch in issue 10941 fixes this, so the right local time in London (01:00) is returned for the epoch (internal date string "01-Jan-1970 00:00:00 +0000").

Note that this also exposes another problem with Time2Internaldate(), since it uses time.timezone/time.altzone, which are only valid for the current rules, not old rules as in the London case near the epoch.
msg127496 - (view) Author: Alexander Belopolsky (belopolsky) * (Python committer) Date: 2011-01-29 23:07
On Sat, Jan 29, 2011 at 5:48 PM, Joe Peterson <report@bugs.python.org> wrote:
..
> Note that this also exposes another problem with Time2Internaldate(), since it uses
> time.timezone/time.altzone, which are only valid for the current rules, not old rules as in the London case
> near the epoch.

This is a known issue.  Hopefully this example will raise more
interest for issue 9527.  See also issue 1647654 which suggests a
python-only work-around.
History
Date User Action Args
2011-01-29 23:07:11belopolskysetmessages: + msg127496
2011-01-29 22:47:59lavajoesetmessages: + msg127494
2011-01-29 21:38:06lavajoesetmessages: + msg127489
2011-01-29 20:43:16belopolskysetstatus: open -> closed
2011-01-29 20:37:09belopolskysetmessages: + msg127485
2011-01-29 19:20:42belopolskysetmessages: + msg127471
2011-01-29 19:16:42lavajoesetmessages: + msg127470
2011-01-29 18:45:09belopolskysetversions: + Python 3.1
nosy: + benjamin.peterson

messages: + msg127462

resolution: accepted -> fixed
stage: commit review -> resolved
2011-01-29 17:19:36belopolskysetnosy: georg.brandl, belopolsky, lavajoe
messages: + msg127444
2011-01-29 16:35:35belopolskysetfiles: + issue10939.diff
nosy: georg.brandl, belopolsky, lavajoe
2011-01-29 16:34:35belopolskysetfiles: - issue10939.diff
nosy: georg.brandl, belopolsky, lavajoe
2011-01-29 16:33:55belopolskysetfiles: + issue10939.diff
nosy: georg.brandl, belopolsky, lavajoe
messages: + msg127437

resolution: accepted
stage: patch review -> commit review
2011-01-29 09:50:20georg.brandlsetpriority: normal -> release blocker
nosy: georg.brandl, belopolsky, lavajoe
messages: + msg127393
2011-01-28 23:53:20belopolskysetassignee: belopolsky -> georg.brandl
nosy: georg.brandl, belopolsky, lavajoe
2011-01-28 22:22:39lavajoesetfiles: + imaplib_Internaldate2tuple_bytes_fixes_python32.patch
nosy: georg.brandl, belopolsky, lavajoe
2011-01-28 22:20:11lavajoesetfiles: - imaplib_Internaldate2tuple_bytes_fixes_python32.patch
nosy: georg.brandl, belopolsky, lavajoe
2011-01-19 22:35:08belopolskysetnosy: + georg.brandl, belopolsky
messages: + msg126572

assignee: belopolsky
stage: patch review
2011-01-18 22:37:58lavajoesettitle: imaplib: Internaldate2tuple fails to parse month and does not work with negative TZ offset due to bytes/str issues -> imaplib: Internaldate2tuple raises KeyError parsing month and does not work with negative TZ offset due to bytes/str issues
2011-01-18 22:31:34lavajoecreate