msg127186 - (view) |
Author: Sebastian Spaeth (spaetz) |
Date: 2011-01-27 13:35 |
imaplib's Time2Internaldate returns invalid (as localized) INTERNALDATE strings. Appending a message with such a time string leads to a:
19 BAD Command Argument Error. 11 (for MS Exchange IMAP servers)
it returned "26-led-2011 18:23:44 +0100", however:
http://tools.ietf.org/html/rfc2060.html defines:
date_month ::= "Jan" / "Feb" / "Mar" / "Apr" / "May" / "Jun" /
"Jul" / "Aug" / "Sep" / "Oct" / "Nov" / "Dec"
so it expects an English date format.
imaplib's Time2Internaldate uses time.strftime() to create the final string which uses the current locale, returning things such as:
"26-led-2011 18:23:44 +0100" rather than "26-Jan-2011 18:23:44 +0100".
For the right thing to do, we would need to set locale.setlocale(locale.LC_TIME, '') to get English formatting or we would need to use some home-grown parser that hardcodes the proper terms.
|
msg127187 - (view) |
Author: Sebastian Spaeth (spaetz) |
Date: 2011-01-27 13:40 |
P.S. To replicate this in ipython:
import locale, imaplib
locale.setlocale(locale.LC_ALL,'de_CH.utf8')
imaplib.Time2Internaldate(220254431)
Out[1]: '"24-Dez-1976 06:47:11 +0100"'
(Note the German 'Dez' rather than 'Dec')
|
msg127189 - (view) |
Author: Sebastian Spaeth (spaetz) |
Date: 2011-01-27 14:15 |
CC'ing lavajoe as he seemed to be busy with some of imaplib's Date stuff the last couple of days.
|
msg127195 - (view) |
Author: Joe Peterson (lavajoe) |
Date: 2011-01-27 14:39 |
Sebastian,
Yes, in fact Alexander Belopolsky (belopolsky) brought up the the locale issue for this very function in one of the other issue comments.
The invert function, Internaldate2tuple(), actually does its own parsing using a regex match (and so does not have the problem), but you are right, Time2Internaldate() has this issue.
|
msg127199 - (view) |
Author: Sebastian Spaeth (spaetz) |
Date: 2011-01-27 15:43 |
I think I found the issue he mentioned, however it was about the functions taking the local time (rather than UTC), which is fine.
The problem is that Time2Internaldate is used for every .append() operation internally, producing invalid dates which are handed to the IMAP server. So in most cases, the IMAP server will silently ignore the time and use the current time (as per IMAP RFC) or it will complain and barf out (as the MS Exchange server rightly does.
So this is more than just an inconvenience, it outright prevents intenational users from APPENDing new messages to a server (or silently bodges the message date) as there is no way around using that function...
Sorry if this sounds like whining :-) I don't even have a patch handy...
|
msg127200 - (view) |
Author: Joe Peterson (lavajoe) |
Date: 2011-01-27 15:52 |
Yes, that's serious, certainly.
A patch should be fairly straightforward, given that part of the formatting logic is already there (for the TZ offset at the end). You just need to format the 6 values, and do a lookup for the month name.
If you want to try to work up one, I can take a look, or maybe, if I have some time today, I'll try to do one quickly...
|
msg127225 - (view) |
Author: Joe Peterson (lavajoe) |
Date: 2011-01-27 21:06 |
OK, I attached a patch that should work. Note that this patch works for Python 2 and Python 3.
As an aside, the str type is still returned as before (even in Python 3), and the _month_names list uses str. As has been discussed, it may be more proper to return a bytes array and be consistent throughout imaplib, but this is not addressed here.
Also, I return a leading zero on the day instead of a leading space, since this appears to be what is returned by two IMAP servers I have just tested (gmail's and dovecot).
|
msg127257 - (view) |
Author: Sebastian Spaeth (spaetz) |
Date: 2011-01-28 08:58 |
> Added file: imaplib_Time2Internaldate_locale_fix.patch
The patch looks very good to me and works. I agree that we should be
returning a bytearray but this is should not be part of this issue.
For all that it's worth:
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
|
msg127328 - (view) |
Author: Alexander Belopolsky (belopolsky) * ![Python committer (Python committer)](@@file/committer.png) |
Date: 2011-01-28 19:43 |
Two nitpicks:
1. To avoid repetition, I would now define Mon2num as
Mon2num = dict(zip(_month_names, range(1, 13)))
2. Please keep lines under 79 characters long.
This does not seem important enough to push to RC2, but if you think otherwise please get RM approval.
|
msg127330 - (view) |
Author: Alexander Belopolsky (belopolsky) * ![Python committer (Python committer)](@@file/committer.png) |
Date: 2011-01-28 19:44 |
Also, isn't day supposed to be space- rather than 0- padded?
|
msg127339 - (view) |
Author: Alexander Belopolsky (belopolsky) * ![Python committer (Python committer)](@@file/committer.png) |
Date: 2011-01-28 20:31 |
On Fri, Jan 28, 2011 at 2:44 PM, Alexander Belopolsky
<report@bugs.python.org> wrote:
..
> Also, isn't day supposed to be space- rather than 0- padded?
To the best of my understanding, rfc 2060 requires space-padded day
(strftime code %e):
"""
date_day_fixed ::= (SPACE digit) / 2digit
;; Fixed-format version of date_day
...
date_time ::= <"> date_day_fixed "-" date_month "-" date_year
SPACE time SPACE zone <">
...
msg_att ::= ...
"INTERNALDATE" SPACE date_time /
...
"""
See http://tools.ietf.org/html/rfc2060.html
|
msg127341 - (view) |
Author: Joe Peterson (lavajoe) |
Date: 2011-01-28 20:34 |
> Also, isn't day supposed to be space- rather than 0- padded?
This is not clear to me. RFC2822 (referenced from RFC3501 for internal date) discusses date formats, but as used in the header. In this case, day is specified as "([FWS] 1*2DIGIT)", which implies optional space and 1 or 2 digit day. I am not sure this disallows leading-zero format. But this date spec also says dates should be space-separated (like "12 Jan 2011"), and clearly INTERNALDATE needs "-" (like "12-Jan-2011"). Therefore, I cannot see this date format as being authoritative fro INTERNALDATE.
Also, RFC3501, in chage #71, is extra confusing in that it puts the 3-letter month in all-caps. Python's Internaldate2tuple(), e.g., cannot handle this currently (nor can it handle a single-digit day with no space or 0, but its regex does handle a leading zero, which led me to think 0 is OK).
Also, it seems that gmail's imap server and Dovecot imap server return leading zero, not leading space, when you fetch INTERNALDATE. So I concluded from all this that 0 might actually be preferred. If this is true, leading zero is better also in that it is less error-prone (e.g., strip can remove the leading space, which will cause problems).
I'll keep looking for definitive info, but if you know of some I missed, please let me know.
|
msg127342 - (view) |
Author: Joe Peterson (lavajoe) |
Date: 2011-01-28 20:47 |
Our messages crossed... :)
Hm, I see that in RFC 3501, as well (which obsoletes 2060).
But... I wonder: does "(SP DIGIT) / 2DIGIT" mean that " 1" and "01" are both OK? It seems ambiguous to me.
I still don't see why major IMAP servers are returning leading zeros if now allowed...
|
msg127356 - (view) |
Author: Joe Peterson (lavajoe) |
Date: 2011-01-28 22:17 |
Here's a new patch. I would still like to discuss the leading space vs. leading zero issue, but I have reverted to using a leading space in this patch - fewer changes that way.
The long line is also fixed (sorry about that - yes, long lines are ugly). And I have used your suggested Mon2num dict creation. Note that I do an encode() in there for compatibility with Python 3, which throws an exception if the keys are not bytes arrays (consistent with the fix in issue 10939).
|
msg127361 - (view) |
Author: Alexander Belopolsky (belopolsky) * ![Python committer (Python committer)](@@file/committer.png) |
Date: 2011-01-28 23:33 |
I would write the formatting code as follows:
('"%2d-%s-%04d %02d:%02d:%02d %+03d%02d"' %
((tt[2], _month_names[tt[1]], tt[0]) +
tt[3:6] + divmod(zone//60, 60)))
The above also assumes that month names are stored in a 1-based array:
_month_names = [None, 'Jan', ...]
Note that %2d format code takes care of space-padding.
If you think the expression that I conjured is too cryptic, get the temporal data from timetuple first with say
y, m, d, H, M, S = tt[:6]
and use named variables in the formatting expression.
|
msg127363 - (view) |
Author: Joe Peterson (lavajoe) |
Date: 2011-01-29 00:06 |
Not cryptic at all - looks great! New patch attached with associated tweaks.
|
msg130850 - (view) |
Author: R. David Murray (r.david.murray) * ![Python committer (Python committer)](@@file/committer.png) |
Date: 2011-03-14 16:33 |
The tests for this function are...not sufficient. I don't think I'm comfortable committing a patch without improving the tests. Ideally there would also be a test that the locale does not affect the result, which would need to be skipped if the chosen test locale was not available on the machine running the tests.
|
msg163513 - (view) |
Author: Roundup Robot (python-dev) ![Python triager (Python triager)](@@file/triager.png) |
Date: 2012-06-23 01:11 |
New changeset 42b9d9d795f7 by Alexander Belopolsky in branch 'default':
Issues #11024: Fixes and additional tests for Time2Internaldate.
http://hg.python.org/cpython/rev/42b9d9d795f7
|
msg234507 - (view) |
Author: Alessio (Pilessio) |
Date: 2015-01-22 18:36 |
Not working patch, if I use this method on append I've all messages with 1970 year
|
msg234554 - (view) |
Author: Alessio (Pilessio) |
Date: 2015-01-23 13:23 |
Is anybody working with this case?
|
msg234556 - (view) |
Author: R. David Murray (r.david.murray) * ![Python committer (Python committer)](@@file/committer.png) |
Date: 2015-01-23 14:41 |
I'm not sure why this issue is still open. It looks like Alexander committed the fix.
If you are seeing a problem, I think that would be a new bug, and you should open a new issue giving details on how to reproduce the problem you are seeing.
|
msg328317 - (view) |
Author: Florian Kisser (floriankisser) |
Date: 2018-10-23 13:50 |
imaplib_Time2Internaldate_locale_fix.patch was never applied to Python 2.7. Regarding the comments I found no reason why, so this is still an issue, I guess.
|
|
Date |
User |
Action |
Args |
2022-04-11 14:57:11 | admin | set | github: 55233 |
2018-10-23 13:50:16 | floriankisser | set | nosy:
+ floriankisser messages:
+ msg328317
|
2015-01-23 14:41:22 | r.david.murray | set | status: open -> closed
messages:
+ msg234556 stage: needs patch -> resolved |
2015-01-23 13:23:02 | Pilessio | set | messages:
+ msg234554 |
2015-01-22 18:36:06 | Pilessio | set | nosy:
+ Pilessio messages:
+ msg234507
|
2012-06-23 01:11:02 | python-dev | set | nosy:
+ python-dev messages:
+ msg163513
|
2011-03-14 16:33:43 | r.david.murray | set | nosy:
belopolsky, r.david.murray, lavajoe, spaetz messages:
+ msg130850 |
2011-01-29 00:06:40 | lavajoe | set | files:
+ imaplib_Time2Internaldate_locale_fix.patch nosy:
belopolsky, r.david.murray, lavajoe, spaetz messages:
+ msg127363
|
2011-01-29 00:02:56 | lavajoe | set | files:
- imaplib_Time2Internaldate_locale_fix.patch nosy:
belopolsky, r.david.murray, lavajoe, spaetz |
2011-01-28 23:33:23 | belopolsky | set | nosy:
belopolsky, r.david.murray, lavajoe, spaetz messages:
+ msg127361 |
2011-01-28 22:27:19 | lavajoe | set | files:
+ imaplib_Time2Internaldate_locale_fix.patch nosy:
belopolsky, r.david.murray, lavajoe, spaetz |
2011-01-28 22:21:02 | lavajoe | set | files:
- imaplib_Time2Internaldate_locale_fix.patch nosy:
belopolsky, r.david.murray, lavajoe, spaetz |
2011-01-28 22:17:46 | lavajoe | set | files:
+ imaplib_Time2Internaldate_locale_fix.patch nosy:
belopolsky, r.david.murray, lavajoe, spaetz messages:
+ msg127356
|
2011-01-28 22:12:45 | lavajoe | set | files:
- imaplib_Time2Internaldate_locale_fix.patch nosy:
belopolsky, r.david.murray, lavajoe, spaetz |
2011-01-28 20:47:29 | lavajoe | set | nosy:
belopolsky, r.david.murray, lavajoe, spaetz messages:
+ msg127342 |
2011-01-28 20:34:33 | lavajoe | set | nosy:
belopolsky, r.david.murray, lavajoe, spaetz messages:
+ msg127341 |
2011-01-28 20:31:01 | belopolsky | set | nosy:
belopolsky, r.david.murray, lavajoe, spaetz messages:
+ msg127339 |
2011-01-28 19:44:36 | belopolsky | set | nosy:
belopolsky, r.david.murray, lavajoe, spaetz messages:
+ msg127330 |
2011-01-28 19:43:03 | belopolsky | set | nosy:
+ belopolsky messages:
+ msg127328
|
2011-01-28 08:58:29 | spaetz | set | nosy:
r.david.murray, lavajoe, spaetz messages:
+ msg127257 |
2011-01-27 21:06:43 | lavajoe | set | files:
+ imaplib_Time2Internaldate_locale_fix.patch nosy:
r.david.murray, lavajoe, spaetz |
2011-01-27 21:06:31 | lavajoe | set | files:
- imaplib_Time2Internaldate_locale_fix.patch nosy:
r.david.murray, lavajoe, spaetz |
2011-01-27 21:06:01 | lavajoe | set | files:
+ imaplib_Time2Internaldate_locale_fix.patch
messages:
+ msg127225 keywords:
+ patch nosy:
r.david.murray, lavajoe, spaetz |
2011-01-27 18:12:59 | r.david.murray | set | nosy:
+ r.david.murray stage: needs patch
versions:
+ Python 3.1, Python 2.7, Python 3.2, - 3rd party |
2011-01-27 15:52:29 | lavajoe | set | nosy:
lavajoe, spaetz messages:
+ msg127200 |
2011-01-27 15:43:51 | spaetz | set | nosy:
lavajoe, spaetz messages:
+ msg127199 |
2011-01-27 14:39:51 | lavajoe | set | nosy:
lavajoe, spaetz messages:
+ msg127195 versions:
+ 3rd party, - Python 2.6 |
2011-01-27 14:15:35 | spaetz | set | nosy:
+ lavajoe messages:
+ msg127189
|
2011-01-27 13:40:09 | spaetz | set | messages:
+ msg127187 |
2011-01-27 13:35:00 | spaetz | create | |