classification
Title: imaplib: Time2Internaldate() returns localized strings
Type: behavior Stage: resolved
Components: Library (Lib) Versions: Python 3.1, Python 3.2, Python 2.7
process
Status: closed Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: Pilessio, belopolsky, floriankisser, lavajoe, python-dev, r.david.murray, spaetz
Priority: normal Keywords: patch

Created on 2011-01-27 13:35 by spaetz, last changed 2018-10-23 13:50 by floriankisser. This issue is now closed.

Files
File name Uploaded Description Edit
imaplib_Time2Internaldate_locale_fix.patch lavajoe, 2011-01-29 00:06 review
Messages (22)
msg127186 - (view) Author: Sebastian Spaeth (spaetz) Date: 2011-01-27 13:35
imaplib's Time2Internaldate returns invalid (as localized) INTERNALDATE strings. Appending a message with such a time string leads to a:
19 BAD Command Argument Error. 11 (for MS Exchange IMAP servers)

it returned "26-led-2011 18:23:44 +0100", however:

http://tools.ietf.org/html/rfc2060.html defines:
date_month      ::= "Jan" / "Feb" / "Mar" / "Apr" / "May" / "Jun" /
                    "Jul" / "Aug" / "Sep" / "Oct" / "Nov" / "Dec"

so it expects an English date format.

imaplib's Time2Internaldate uses time.strftime() to create the final string which uses the current locale, returning things such as:

"26-led-2011 18:23:44 +0100" rather than "26-Jan-2011 18:23:44 +0100".

For the right thing to do, we would need to set locale.setlocale(locale.LC_TIME, '') to get English formatting or we would need to use some home-grown parser that hardcodes the proper terms.
msg127187 - (view) Author: Sebastian Spaeth (spaetz) Date: 2011-01-27 13:40
P.S. To replicate this in ipython:

import locale, imaplib

locale.setlocale(locale.LC_ALL,'de_CH.utf8')
imaplib.Time2Internaldate(220254431)
Out[1]: '"24-Dez-1976 06:47:11 +0100"'

(Note the German 'Dez' rather than 'Dec')
msg127189 - (view) Author: Sebastian Spaeth (spaetz) Date: 2011-01-27 14:15
CC'ing lavajoe as he seemed to be busy with some of imaplib's Date stuff the last couple of days.
msg127195 - (view) Author: Joe Peterson (lavajoe) Date: 2011-01-27 14:39
Sebastian,

Yes, in fact Alexander Belopolsky (belopolsky) brought up the the locale issue for this very function in one of the other issue comments.

The invert function, Internaldate2tuple(), actually does its own parsing using a regex match (and so does not have the problem), but you are right, Time2Internaldate() has this issue.
msg127199 - (view) Author: Sebastian Spaeth (spaetz) Date: 2011-01-27 15:43
I think I found the issue he mentioned, however it was about the functions taking the local time (rather than UTC), which is fine.

The problem is that Time2Internaldate is used for every .append() operation internally, producing invalid dates which are handed to the IMAP server. So in most cases, the IMAP server will silently ignore the time and use the current time (as per IMAP RFC) or it will complain and barf out (as the MS Exchange server rightly does.

So this is more than just an inconvenience, it outright prevents intenational users from APPENDing new messages to a server (or silently bodges the message date) as there is no way around using that function...

Sorry if this sounds like whining :-) I don't even have a patch handy...
msg127200 - (view) Author: Joe Peterson (lavajoe) Date: 2011-01-27 15:52
Yes, that's serious, certainly.

A patch should be fairly straightforward, given that part of the formatting logic is already there (for the TZ offset at the end).  You just need to format the 6 values, and do a lookup for the month name.

If you want to try to work up one, I can take a look, or maybe, if I have some time today, I'll try to do one quickly...
msg127225 - (view) Author: Joe Peterson (lavajoe) Date: 2011-01-27 21:06
OK, I attached a patch that should work.  Note that this patch works for Python 2 and Python 3.

As an aside, the str type is still returned as before (even in Python 3), and the _month_names list uses str.  As has been discussed, it may be more proper to return a bytes array and be consistent throughout imaplib, but this is not addressed here.

Also, I return a leading zero on the day instead of a leading space, since this appears to be what is returned by two IMAP servers I have just tested (gmail's and dovecot).
msg127257 - (view) Author: Sebastian Spaeth (spaetz) Date: 2011-01-28 08:58
> Added file: imaplib_Time2Internaldate_locale_fix.patch

The patch looks very good to me and works. I agree that we should be
returning a bytearray but this is should not be part of this issue.

For all that it's worth:
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
msg127328 - (view) Author: Alexander Belopolsky (belopolsky) * (Python committer) Date: 2011-01-28 19:43
Two nitpicks:

1. To avoid repetition, I would now define Mon2num as 

Mon2num = dict(zip(_month_names, range(1, 13)))

2. Please keep lines under 79 characters long.

This does not seem important enough to push to RC2, but if you think otherwise please get RM approval.
msg127330 - (view) Author: Alexander Belopolsky (belopolsky) * (Python committer) Date: 2011-01-28 19:44
Also, isn't day supposed to be space- rather than 0- padded?
msg127339 - (view) Author: Alexander Belopolsky (belopolsky) * (Python committer) Date: 2011-01-28 20:31
On Fri, Jan 28, 2011 at 2:44 PM, Alexander Belopolsky
<report@bugs.python.org> wrote:
..
> Also, isn't day supposed to be space- rather than 0- padded?

To the best of my understanding, rfc 2060 requires space-padded day
(strftime code %e):

"""
date_day_fixed  ::= (SPACE digit) / 2digit
                    ;; Fixed-format version of date_day
...
date_time       ::= <"> date_day_fixed "-" date_month "-" date_year
                    SPACE time SPACE zone <">
...
msg_att         ::= ...
                    "INTERNALDATE" SPACE date_time /
...
"""

See http://tools.ietf.org/html/rfc2060.html
msg127341 - (view) Author: Joe Peterson (lavajoe) Date: 2011-01-28 20:34
> Also, isn't day supposed to be space- rather than 0- padded?

This is not clear to me.  RFC2822 (referenced from RFC3501 for internal date) discusses date formats, but as used in the header.  In this case, day is specified as "([FWS] 1*2DIGIT)", which implies optional space and 1 or 2 digit day.  I am not sure this disallows leading-zero format.  But this date spec also says dates should be space-separated (like "12 Jan 2011"), and clearly INTERNALDATE needs "-" (like "12-Jan-2011").  Therefore, I cannot see this date format as being authoritative fro INTERNALDATE.

Also, RFC3501, in chage #71, is extra confusing in that it puts the 3-letter month in all-caps.  Python's Internaldate2tuple(), e.g., cannot handle this currently (nor can it handle a single-digit day with no space or 0, but its regex does handle a leading zero, which led me to think 0 is OK).

Also, it seems that gmail's imap server and Dovecot imap server return leading zero, not leading space, when you fetch INTERNALDATE.  So I concluded from all this that 0 might actually be preferred.  If this is true, leading zero is better also in that it is less error-prone (e.g., strip can remove the leading space, which will cause problems).

I'll keep looking for definitive info, but if you know of some I missed, please let me know.
msg127342 - (view) Author: Joe Peterson (lavajoe) Date: 2011-01-28 20:47
Our messages crossed...  :)

Hm, I see that in RFC 3501, as well (which obsoletes 2060).

But...  I wonder: does "(SP DIGIT) / 2DIGIT" mean that " 1" and "01" are both OK?  It seems ambiguous to me.

I still don't see why major IMAP servers are returning leading zeros if now allowed...
msg127356 - (view) Author: Joe Peterson (lavajoe) Date: 2011-01-28 22:17
Here's a new patch.  I would still like to discuss the leading space vs. leading zero issue, but I have reverted to using a leading space in this patch - fewer changes that way.

The long line is also fixed (sorry about that - yes, long lines are ugly).  And I have used your suggested Mon2num dict creation.  Note that I do an encode() in there for compatibility with Python 3, which throws an exception if the keys are not bytes arrays (consistent with the fix in issue 10939).
msg127361 - (view) Author: Alexander Belopolsky (belopolsky) * (Python committer) Date: 2011-01-28 23:33
I would write the formatting code as follows:

('"%2d-%s-%04d %02d:%02d:%02d %+03d%02d"' %
 ((tt[2], _month_names[tt[1]], tt[0]) +
   tt[3:6] + divmod(zone//60, 60)))

The above also assumes that month names are stored in a 1-based array:

_month_names = [None, 'Jan', ...]

Note that %2d format code takes care of space-padding.

If you think the expression that I conjured is too cryptic, get the temporal data from timetuple first with say

y, m, d, H, M, S = tt[:6]

and use named variables in the formatting expression.
msg127363 - (view) Author: Joe Peterson (lavajoe) Date: 2011-01-29 00:06
Not cryptic at all - looks great!  New patch attached with associated tweaks.
msg130850 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2011-03-14 16:33
The tests for this function are...not sufficient.  I don't think I'm comfortable committing a patch without improving the tests.  Ideally there would also be a test that the locale does not affect the result, which would need to be skipped if the chosen test locale was not available on the machine running the tests.
msg163513 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2012-06-23 01:11
New changeset 42b9d9d795f7 by Alexander Belopolsky in branch 'default':
Issues #11024: Fixes and additional tests for Time2Internaldate.
http://hg.python.org/cpython/rev/42b9d9d795f7
msg234507 - (view) Author: Alessio (Pilessio) Date: 2015-01-22 18:36
Not working patch, if I use this method on append I've all messages with 1970 year
msg234554 - (view) Author: Alessio (Pilessio) Date: 2015-01-23 13:23
Is anybody working with this case?
msg234556 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2015-01-23 14:41
I'm not sure why this issue is still open.  It looks like Alexander committed the fix.

If you are seeing a problem, I think that would be a new bug, and you should open a new issue giving details on how to reproduce the problem you are seeing.
msg328317 - (view) Author: Florian Kisser (floriankisser) Date: 2018-10-23 13:50
imaplib_Time2Internaldate_locale_fix.patch was never applied to Python 2.7. Regarding the comments I found no reason why, so this is still an issue, I guess.
History
Date User Action Args
2018-10-23 13:50:16floriankissersetnosy: + floriankisser
messages: + msg328317
2015-01-23 14:41:22r.david.murraysetstatus: open -> closed

messages: + msg234556
stage: needs patch -> resolved
2015-01-23 13:23:02Pilessiosetmessages: + msg234554
2015-01-22 18:36:06Pilessiosetnosy: + Pilessio
messages: + msg234507
2012-06-23 01:11:02python-devsetnosy: + python-dev
messages: + msg163513
2011-03-14 16:33:43r.david.murraysetnosy: belopolsky, r.david.murray, lavajoe, spaetz
messages: + msg130850
2011-01-29 00:06:40lavajoesetfiles: + imaplib_Time2Internaldate_locale_fix.patch
nosy: belopolsky, r.david.murray, lavajoe, spaetz
messages: + msg127363
2011-01-29 00:02:56lavajoesetfiles: - imaplib_Time2Internaldate_locale_fix.patch
nosy: belopolsky, r.david.murray, lavajoe, spaetz
2011-01-28 23:33:23belopolskysetnosy: belopolsky, r.david.murray, lavajoe, spaetz
messages: + msg127361
2011-01-28 22:27:19lavajoesetfiles: + imaplib_Time2Internaldate_locale_fix.patch
nosy: belopolsky, r.david.murray, lavajoe, spaetz
2011-01-28 22:21:02lavajoesetfiles: - imaplib_Time2Internaldate_locale_fix.patch
nosy: belopolsky, r.david.murray, lavajoe, spaetz
2011-01-28 22:17:46lavajoesetfiles: + imaplib_Time2Internaldate_locale_fix.patch
nosy: belopolsky, r.david.murray, lavajoe, spaetz
messages: + msg127356
2011-01-28 22:12:45lavajoesetfiles: - imaplib_Time2Internaldate_locale_fix.patch
nosy: belopolsky, r.david.murray, lavajoe, spaetz
2011-01-28 20:47:29lavajoesetnosy: belopolsky, r.david.murray, lavajoe, spaetz
messages: + msg127342
2011-01-28 20:34:33lavajoesetnosy: belopolsky, r.david.murray, lavajoe, spaetz
messages: + msg127341
2011-01-28 20:31:01belopolskysetnosy: belopolsky, r.david.murray, lavajoe, spaetz
messages: + msg127339
2011-01-28 19:44:36belopolskysetnosy: belopolsky, r.david.murray, lavajoe, spaetz
messages: + msg127330
2011-01-28 19:43:03belopolskysetnosy: + belopolsky
messages: + msg127328
2011-01-28 08:58:29spaetzsetnosy: r.david.murray, lavajoe, spaetz
messages: + msg127257
2011-01-27 21:06:43lavajoesetfiles: + imaplib_Time2Internaldate_locale_fix.patch
nosy: r.david.murray, lavajoe, spaetz
2011-01-27 21:06:31lavajoesetfiles: - imaplib_Time2Internaldate_locale_fix.patch
nosy: r.david.murray, lavajoe, spaetz
2011-01-27 21:06:01lavajoesetfiles: + imaplib_Time2Internaldate_locale_fix.patch

messages: + msg127225
keywords: + patch
nosy: r.david.murray, lavajoe, spaetz
2011-01-27 18:12:59r.david.murraysetnosy: + r.david.murray
stage: needs patch

versions: + Python 3.1, Python 2.7, Python 3.2, - 3rd party
2011-01-27 15:52:29lavajoesetnosy: lavajoe, spaetz
messages: + msg127200
2011-01-27 15:43:51spaetzsetnosy: lavajoe, spaetz
messages: + msg127199
2011-01-27 14:39:51lavajoesetnosy: lavajoe, spaetz
messages: + msg127195
versions: + 3rd party, - Python 2.6
2011-01-27 14:15:35spaetzsetnosy: + lavajoe
messages: + msg127189
2011-01-27 13:40:09spaetzsetmessages: + msg127187
2011-01-27 13:35:00spaetzcreate