classification
Title: datetime-RFC2822 roundtripping
Type: enhancement Stage: resolved
Components: email, Library (Lib) Versions: Python 3.3
process
Status: closed Resolution: fixed
Dependencies: 5094 Superseder:
Assigned To: belopolsky Nosy List: Alexander.Belopolsky, ajaksu2, akuchling, barry, belopolsky, doerwalter, eric.araujo, python-dev, r.david.murray, tim.peters
Priority: normal Keywords: easy, patch

Created on 2003-01-09 18:24 by doerwalter, last changed 2012-08-23 04:10 by belopolsky. This issue is now closed.

Files
File name Uploaded Description Edit
datetime-mimeformat.diff doerwalter, 2007-03-07 19:55 review
formatdate_datetime_support.patch r.david.murray, 2011-05-04 19:56 review
util_datetime.patch r.david.murray, 2011-07-13 23:04 review
issue665194.diff belopolsky, 2012-06-22 21:10 review
localtime.patch r.david.murray, 2012-08-22 01:39 review
localtime.patch r.david.murray, 2012-08-22 02:24 review
Messages (36)
msg53725 - (view) Author: Walter Dörwald (doerwalter) * (Python committer) Date: 2003-01-09 18:24
It would be good to have a simply way to convert
between datetime objects and RFC2822 style strings.
From string to datetime is easy with
datetime.datetime(*email.Utils.parsedate(m)[:7]) (but
this drops the timezone), but the other direction seems
impossible. email.Utils.formatdate takes a timestamp
argument, but AFAICT there's no way to get a timestamp
out of a datetime object.

Of course the best solution (ignoring backwards
compatibility) would be for parsedate und formatdate to
return/accept datetime objects or for datetime to have
the appropriate methods.
msg53726 - (view) Author: Tim Peters (tim.peters) * (Python committer) Date: 2003-01-11 05:10
Logged In: YES 
user_id=31435

You can get a timestamp like so:

>>> time.mktime(datetime.date(2002, 1, 1).timetuple())
1009861200.0
>>>

The dates for which this works depends on the platform 
mktime implementation, though.

BTW, this sounds a lot more like a new feature request 
than a bug!
msg53727 - (view) Author: Walter Dörwald (doerwalter) * (Python committer) Date: 2003-01-11 12:33
Logged In: YES 
user_id=89016

OK, I'll mark this a feature request.

datetime has fromordinal() and toordinal(), it has 
fromtimestamp(), so I'd say a totimestamp() method would be 
a logical addition.
msg53728 - (view) Author: Tim Peters (tim.peters) * (Python committer) Date: 2003-01-11 16:32
Logged In: YES 
user_id=31435

Define what totimestamp() should do.  The range of 
timestamps supported by the *platform* C library (and so 
indirectly by Python's time module) isn't defined by any 
standard, and isn't easily discoverable either.  It may or 
may not work before 1970, may or may not after 2038.  
datetime covers days far outside that range.  Note that 
even a double doesn't have enough bits of precision to 
cover the full range of datetime values, either.

In contrast, ordinals are wholly under Python's control, so 
we can promise surprise-free conversion in both directions.  
All we can promise about timestamps is that if the platform 
supports a timestamp for a time in datetime's range, 
datetime can make sense of it.
msg53729 - (view) Author: Walter Dörwald (doerwalter) * (Python committer) Date: 2003-01-13 13:05
Logged In: YES 
user_id=89016

totimestamp() should be the inverse of fromtimestamp(), i.e.
foo.totimestamp() should be the same as
time.mktime(foo.timetuple()).
datetime.datetime.totimestamp() should raise OverflowError
iff time.mktime() raises OverflowError.

But as this may lose precision, I'd say it would be better,
if datetime supported RFC1123 conversion directly, i.e. two
methods frommime() and tomime(), that parse and format
strings like "Sun, 06 Nov 1994 08:49:37 GMT" (what
rfc822.parsedate() and rfc822.formatdate() do)
msg53730 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2003-05-23 22:46
Logged In: YES 
user_id=357491

time.strptime doesn't solve your problem?
msg53731 - (view) Author: Walter Dörwald (doerwalter) * (Python committer) Date: 2003-05-26 16:56
Logged In: YES 
user_id=89016

time.strptime() is locale aware, but RFC1123 & RFC822
require english day and month names, so time.strptime()
can't be used as-is. (And of course time.strptime() only
works for formatting, not for parsing)
msg53732 - (view) Author: A.M. Kuchling (akuchling) * (Python committer) Date: 2006-12-21 15:16
Moving to feature requests.
msg53733 - (view) Author: Walter Dörwald (doerwalter) * (Python committer) Date: 2007-03-07 19:55
Here is a patch that implements to formatting part. It adds a method mimeformat() to the datetime class. The datetime object must have a tzinfo for this to work.
File Added: datetime-mimeformat.diff
msg81810 - (view) Author: Daniel Diniz (ajaksu2) Date: 2009-02-12 19:58
Patch includes tests, should be updated.
msg107436 - (view) Author: Alexander Belopolsky (belopolsky) * (Python committer) Date: 2010-06-09 22:58
-1 on adding more formatting/parsing methods to datetime. +1 on adding functions to email.utils that work with datetime objects.  Adding #5094 as a dependency because RFC 2822 requires timezone information for proper formatting.
msg135154 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2011-05-04 19:56
Here is a patch that adds datetime support to email.utils.formatdate.  Ultimately the email package will give programs access to datetime+timezone representations of the dates in various headers, so this provides the output end of the round trip.

Alexander, if you could review this that would be great.  There may be bugs in the logic of formatdate, but my hope is that I haven't added any new ones.
msg135211 - (view) Author: Alexander Belopolsky (belopolsky) * (Python committer) Date: 2011-05-05 15:33
Rather than shoehorning datetime class support into existing functions, I think a separate set of functions should be written to convert between RFC 2822 timestamps and datetime instances.  Currently, email.utils has three functions dealing with date-time specification in email messages:

formatdate(timeval=None, localtime=False, usegmt=False),
parsedate(data),
parsedate_tz(data)

To work with datetime instances, we can just provide two functions.  For lack of better names, I'll call them format_datetime and parse_datetime.  Rather than using a localtime flag in the format function, I suggest to always interpret naive datetime instances (those with tzinfo = None) as time given in UTC with no information about the local time zone. Per RFC  2822, this should be formatted with "-0000" in the timezone field.  The format_datetime() may take usegmt flag, but it may be better to handle usegmt=True case by a separate format_datetime_gmt() function.

The parse_datetime() function should use a similar convention and produce aware datetime instances unless TZ field contains "-0000".  In this case a naive datetime containing unchanged time data should be produced.

The problem of guessing the local timezone offset or converting naive datetime instance presumed to be in local time to an aware instance does not belong to the email package.  This functionality should be added to the datetime module.  See issue 9527.

There is a remaining question to which I don't have an immediate answer: How should parse_datetime() handle valid RFC 2882 date-time specifications that cannot be represented as a valid datetime.  For example, a spec with seconds=60 or timezone > 2400.
msg135222 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2011-05-05 16:44
Do you think we can get 9527 in?  My patch was based on the non-existence of a LocalTimezone facility in the stdlib.

Good point about the special interpretation of -0000.  Given 9527 (and only given 9527) it would indeed make sense to restrict email to aware datetimes plus naive datetimes for -0000 timestamps.

I'll have to keep a flag for the 60th second  outside of the datetime instance (or pretend it doesn't exist :)
msg135226 - (view) Author: Alexander Belopolsky (belopolsky) * (Python committer) Date: 2011-05-05 17:36
On Thu, May 5, 2011 at 12:44 PM, R. David Murray <report@bugs.python.org> wrote:
..
> Do you think we can get 9527 in?

I hope we can.  Pure Python implementation can be improved by deducing
the TZ offset from localtime() and gmtime() calls.  In C we can use
additional struct tm fields when they are available to do even better.
 Would you like to add your voice to support #9527?

..
> I'll have to keep a flag for the 60th second  outside of the datetime instance (or pretend it doesn't exist :)

If you can find an e-mail message archived somewhere with 60 seconds
in the timestamp, it will be a powerful argument to extend seconds
range that can be stored in datetime objects.   I doubt such messages
exist, though.  Few systems can produce such a timestamp even if they
happen to process an e-mail during a leap second.  In
parse_datetime(), your choice will be between raising an error and
approximating the leap second with the nearest representable time.  I
think clamping 60 seconds to 59 is the best option and this is what
datetime.fromtimestamp does if the system happens to produce a leap
second in the timetuple.
msg135228 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2011-05-05 18:18
Yes, since the package will save the original text anyway, I think just clamping to 59 is best.

Hmm.  Maybe instead I could put in an assert that says "please report this incident to bugs.python.org so we can argue that datetime should get support for leap seconds) :)
msg140317 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2011-07-13 23:04
OK, here is a new patch proposal.  As suggested by Alexander, it adds two functions to email.utils: format_datetime and parsedate_to_datetime.  Adding these does not require having localtime.  In fact, with this patch you can get an aware datetime representing email's current best guess at localtime by doing:

   dt = parsedate_to_datetime(formatdate(localtime=True))

Reviews welcome, but this is simple enough I'll probably just commit it if no one objects.
msg140747 - (view) Author: Roundup Robot (python-dev) Date: 2011-07-20 15:41
New changeset 5f7b03dcd523 by R David Murray in branch 'default':
#665194: support roundtripping RFC2822 date stamps in the email.utils module
http://hg.python.org/cpython/rev/5f7b03dcd523
msg140748 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2011-07-20 15:46
Fixed a bug (parsedate_to_datetime wasn't producing naive datetimes for -0000) and checked this in.

Note that I have added Alexander's 'localtime' function to email.utils in my feature branch/pypi release, to facilitate testing date headers.  If localtime gets added to datetime before 3.3 I'll remove it from email.utils.  The email package itself does not depend on having localtime, but it will be wanted by application programs that use email.
msg161644 - (view) Author: Roundup Robot (python-dev) Date: 2012-05-26 03:23
New changeset df12ce0c96eb by R David Murray in branch 'default':
#665194: Add a localtime function to email.utils.
http://hg.python.org/cpython/rev/df12ce0c96eb
msg163486 - (view) Author: Alexander Belopolsky (belopolsky) * (Python committer) Date: 2012-06-22 21:10
Most of the localtime() logic is now implemented (correctly) in datetime.astimezone().  Attached patch fixes email.utils.localtime() implementation.
msg163516 - (view) Author: Alexander Belopolsky (belopolsky) * (Python committer) Date: 2012-06-23 01:27
David,

issue665194.diff patch is a bug fix for localtime().  If you decide to keep localtime(), there is not much of a rush because bug fixes can go in after beta.
msg163620 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2012-06-23 15:00
Well, I guess I'd like to keep it since it does satisfy a need in the email package: to manipulate dates in the local timezone when composing messages.  It isn't a critical need, though; the critical need is just to be able to have a datetime that has the correct timezone as its tzinfo for 'now', and your astimezone change supplies that.  On the third hand, 'util.localtime()' is a lot more intuitive then 'datetime.now().astimezone()', so I'd probably still have a util.localtime() helper even if I restricted it to only generating 'now'.

So, on balance, since the method is in and tested with the extra functionality, and that functionality will probably prove useful to programs manipulating emails, I'm coming down in favor of keeping it.
msg168834 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2012-08-22 01:39
Alexander, this slipped off my radar.

Some of the tests Brian added fail with the patch applied.  I fixed most of them by adding back the support for converting an aware datetime to an aware localtime.  The remaining two I believe are what you are fixing with this patch (in addition to converting to using astimezone for the current-localtime case).  So there I fixed the tests.

Hopefully I got this right, and hopefully you can find time to review it.  I'll probably check it in before the RC regardless, since it looks like an improvement to me.
msg168835 - (view) Author: Alexander Belopolsky (belopolsky) * (Python committer) Date: 2012-08-22 01:55
I'll take a look tomorrow morning. (EDT:-)
msg168837 - (view) Author: Alexander Belopolsky (belopolsky) * (Python committer) Date: 2012-08-22 02:04
I noticed this part:

+        # We have an aware datetime.  Use aware datetime arithmetic to find the
+        # seconds since the epoch.
+        delta = dt - datetime.datetime(*time.gmtime(0)[:6],
+                                       tzinfo=datetime.timezone.utc)
+        seconds = delta.total_seconds()

Why don't you just return dt.astimezone() here?  A round trip through a floating point timestamp always makes me nervous.  I doubt loss of precision is a problem for the e-mail package, but who knows.
msg168838 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2012-08-22 02:10
Heh, I was just copying the previous code, and didn't think about being able to update it.  Will make that change.
msg168840 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2012-08-22 02:24
And my tests and code were wrong, and I was wrong about what you were trying to fix.  So since the other tests were passing before, presumably there is some test that could be added to exercise the bug you were fixing.  Do you remember what that was?
msg168841 - (view) Author: Alexander Belopolsky (belopolsky) * (Python committer) Date: 2012-08-22 02:38
> So since the other tests were passing before, presumably there
> is some test that could be added to exercise the bug you were
> fixing.  Do you remember what that was?

Yes, the issue was the one that was mentioned in an XXX comment: in many places UTC offset was different at different historical times but standard C library and Python's time module provides a single constant for it.  There are plenty of examples that can be found in Olson's database (Europe/Kiev comes to mind), but it is not easy to devise a test that will work cross-platform.  Maybe we should just restrict the test to Linux/BSD family of OSes?
msg168912 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2012-08-23 01:11
I think restricting the test is fine.  If we find a platform-specific bug on another platform we can add a test specific to that platform when we fix the bug.

Can you provide the test?  I'm going to commit what I've got at this point to make sure I don't miss the RC.
msg168913 - (view) Author: Alexander Belopolsky (Alexander.Belopolsky) Date: 2012-08-23 01:24
Please commit.  I'll add the test.

On Wed, Aug 22, 2012 at 9:11 PM, R. David Murray <report@bugs.python.org> wrote:
>
> R. David Murray added the comment:
>
> I think restricting the test is fine.  If we find a platform-specific bug on another platform we can add a test specific to that platform when we fix the bug.
>
> Can you provide the test?  I'm going to commit what I've got at this point to make sure I don't miss the RC.
>
> ----------
>
> _______________________________________
> Python tracker <report@bugs.python.org>
> <http://bugs.python.org/issue665194>
> _______________________________________
msg168914 - (view) Author: Roundup Robot (python-dev) Date: 2012-08-23 01:34
New changeset 71b9cca81598 by R David Murray in branch 'default':
#665194: Update email.utils.localtime to use astimezone, and fix bug.
http://hg.python.org/cpython/rev/71b9cca81598
msg168916 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2012-08-23 01:36
Leaving open until the test is committed.
msg168919 - (view) Author: Roundup Robot (python-dev) Date: 2012-08-23 01:52
New changeset a2d83fba8fd8 by R David Murray in branch 'default':
#665194: fix variable name in exception code path.
http://hg.python.org/cpython/rev/a2d83fba8fd8
msg168920 - (view) Author: Roundup Robot (python-dev) Date: 2012-08-23 02:06
New changeset 604222c1f8a0 by Alexander Belopolsky in branch 'default':
Added test for a bug fixed in issue #665194.
http://hg.python.org/cpython/rev/604222c1f8a0
msg168922 - (view) Author: Roundup Robot (python-dev) Date: 2012-08-23 03:01
New changeset 4c134e6ba0df by Alexander Belopolsky in branch 'default':
Issue #665194: Added a small optimization
http://hg.python.org/cpython/rev/4c134e6ba0df
History
Date User Action Args
2012-08-23 04:10:12belopolskysetstatus: open -> closed
2012-08-23 03:01:58python-devsetmessages: + msg168922
2012-08-23 02:06:06python-devsetmessages: + msg168920
2012-08-23 01:52:55python-devsetmessages: + msg168919
2012-08-23 01:37:09belopolskysetassignee: belopolsky
2012-08-23 01:36:19r.david.murraysetassignee: r.david.murray -> (no value)
resolution: accepted -> fixed
messages: + msg168916
2012-08-23 01:34:28python-devsetmessages: + msg168914
2012-08-23 01:24:50Alexander.Belopolskysetnosy: + Alexander.Belopolsky
messages: + msg168913
2012-08-23 01:11:29r.david.murraysetmessages: + msg168912
2012-08-22 02:38:30belopolskysetmessages: + msg168841
2012-08-22 02:24:31r.david.murraysetfiles: + localtime.patch

messages: + msg168840
2012-08-22 02:10:34r.david.murraysetmessages: + msg168838
2012-08-22 02:04:37belopolskysetmessages: + msg168837
2012-08-22 01:55:29belopolskysetmessages: + msg168835
2012-08-22 01:39:32r.david.murraysetfiles: + localtime.patch

messages: + msg168834
2012-07-20 17:53:13brett.cannonsetnosy: - brett.cannon
2012-06-23 15:00:05r.david.murraysetmessages: + msg163620
2012-06-23 01:27:10belopolskysetmessages: + msg163516
2012-06-22 21:19:39r.david.murraysetnosy: + barry
components: + email
2012-06-22 21:10:42belopolskysetstatus: closed -> open
files: + issue665194.diff
messages: + msg163486
2012-05-26 03:37:38r.david.murraylinkissue9864 superseder
2012-05-26 03:23:32python-devsetmessages: + msg161644
2011-11-07 18:28:55r.david.murraylinkissue13284 superseder
2011-07-20 15:46:22r.david.murraysetstatus: open -> closed
messages: + msg140748

dependencies: - Add aware local time support to datetime module
resolution: accepted
stage: patch review -> resolved
2011-07-20 15:41:44python-devsetnosy: + python-dev
messages: + msg140747
2011-07-13 23:04:37r.david.murraysetfiles: + util_datetime.patch
assignee: belopolsky -> r.david.murray
messages: + msg140317
2011-05-05 18:18:06r.david.murraysetmessages: + msg135228
2011-05-05 17:36:34belopolskysetmessages: + msg135226
2011-05-05 16:44:03r.david.murraysetdependencies: + Add aware local time support to datetime module
messages: + msg135222
2011-05-05 15:33:38belopolskysetmessages: + msg135211
2011-05-04 19:56:46r.david.murraysetfiles: + formatdate_datetime_support.patch
versions: + Python 3.3, - Python 3.2
nosy: + r.david.murray

messages: + msg135154

stage: needs patch -> patch review
2010-11-20 17:46:30eric.araujosetnosy: + eric.araujo
2010-08-17 22:53:30BreamoreBoysetversions: + Python 3.2, - Python 2.7
2010-06-09 22:58:34belopolskysetdependencies: + datetime lacks concrete tzinfo implementation for UTC
messages: + msg107436
stage: patch review -> needs patch
2010-05-25 20:26:47belopolskysetassignee: belopolsky

nosy: + belopolsky
2009-04-22 18:47:39ajaksu2setkeywords: + easy
stage: needs patch -> patch review
2009-02-12 19:58:21ajaksu2setversions: + Python 2.7
nosy: + ajaksu2
messages: + msg81810
components: + Library (Lib), - None
keywords: + patch
stage: needs patch
2003-01-09 18:24:53doerwaltercreate