classification
Title: datetime module missing some important methods
Type: enhancement Stage:
Components: Library (Lib) Versions: Python 2.7
process
Status: closed Resolution: duplicate
Dependencies: Superseder: datetime needs an "epoch" method
View: 2736
Assigned To: Nosy List: Alexander.Belopolsky, amaury.forgeotdarc, catlee, erik.stephens, guettli, jribbens, mark.dickinson, pitrou, skip.montanaro, srittau, steve.roberts, tim.peters, tomster, vstinner
Priority: normal Keywords: patch

Created on 2007-03-04 11:59 by jribbens, last changed 2010-04-21 18:52 by mark.dickinson. This issue is now closed.

Files
File name Uploaded Description Edit
timedelta.diff catlee, 2007-11-30 20:50
Messages (39)
msg55042 - (view) Author: Jon Ribbens (jribbens) * Date: 2007-03-04 11:59
The datetime module is missing some important methods for interacting with timestamps (i.e. seconds since 1970-01-01T00:00:00).

There are methods to convert from a timestamp, i.e. date.fromtimestamp and datetime.fromtimestamp, but there are no methods to convert back.

In addition, timedelta has no method for returning the number of seconds it represents (i.e. days*86400+seconds+microseconds/1000000).
msg55043 - (view) Author: Thomas Guettler (guettli) Date: 2007-03-22 22:05
Yes, that's true.
msg55573 - (view) Author: Skip Montanaro (skip.montanaro) * (Python triager) Date: 2007-09-02 03:22
There is no datetime.totimestamp because the range
of time represented by a datetime object far
exceeds the range of a normal int-based Unix
timestamp (roughly 1970-2038).  Datetime objects
before the start of the Unix epoch would be
represented by negative numbers.  As far as I
know, the common Unix library functions which
accept epoch times wouldn't know what to do
with a negative number.

That said, you stated these missing methods
were important.  Can you offer some use
cases which would support that contention?
I personally don't think a argument for
symmetry would be a convincing use case and
that's the only one I can think of.
msg55585 - (view) Author: Jon Ribbens (jribbens) * Date: 2007-09-02 14:03
Almost everything you just said about time_t is wrong. time_t is signed,
and always has been (otherwise the 'end of time' for 32-bit time_t would
be 2106, not 2038). Also, time_t does not end at 2038 because nothing
says it must be 32 bits. Also, Python has 'long integers' which do not
overflow.

Also, I don't understand what you mean about use cases. The "use case"
is "dealing with anything which expects standard Unix time_t, for
example the Python standard library". The use case I have personally is
the program I was working on when I encountered the problem described in
this bug report. Also I think symmetry is a darn good argument. Why does
fromtimestamp exist if, as you claim, nobody uses time_t?
msg55586 - (view) Author: Skip Montanaro (skip.montanaro) * (Python triager) Date: 2007-09-02 16:07
Jon> Almost everything you just said about time_t is wrong. time_t is
    Jon> signed, and always has been (otherwise the 'end of time' for 32-bit
    Jon> time_t would be 2106, not 2038). Also, time_t does not end at 2038
    Jon> because nothing says it must be 32 bits. Also, Python has 'long
    Jon> integers' which do not overflow.

My apologies about goofing up on the signedness of time_t.  What are you
going to do with a long integer that you can't do with a datetime object?
You clearly can't pass it directly to any Unix library functions which
expect time_t.  Converting it can overflow.

    Jon> Also, I don't understand what you mean about use cases. The "use
    Jon> case" is "dealing with anything which expects standard Unix time_t,
    Jon> for example the Python standard library". The use case I have
    Jon> personally is the program I was working on when I encountered the
    Jon> problem described in this bug report. Also I think symmetry is a
    Jon> darn good argument. Why does fromtimestamp exist if, as you claim,
    Jon> nobody uses time_t?

What should datetime.datetime(9999, 1, 1).totimestamp() return?  How would
you pass it to something which accepts a time_t?  The fromtimestamp
functions work simply because the range of time_t is a proper subset of the
range of Python's datetime objects.  Symmetry gets you little.  In
situations where you need Unix timestamps and you know your datetime objects
are within the bounds representable by time_t, you can define a convenience
function:

    def totimestamp(dt):
        return time.mktime(dt.timetuple()) + dt.microsecond/1e6

This will, of course, fail if the year is too big or too small (and will
fail in platform-dependent ways if the underlying platform's range of
representable dates has different bounds than Unix does).  Doing it without
resorting to calling time.mktime is also nontrivial.  Under the covers the
datetime module currently uses platform functions to get time information
anyway.  It doesn't do a lot of low-level time arithmethic itself.
Implementing fromtimestamp would require a fair amount of effort unless you
were willing to punt and just raise OverflowError for dates outside the
system's range.

Skip
msg57050 - (view) Author: Tom Lazar (tomster) Date: 2007-11-02 15:02
unless I'm missing something important this will do the trick quite 
nicely:

    >>> from datetime import datetime
    >>> datetime(2007, 12, 24, 20, 0).strftime("%s")
    '1198522800'

For most imaginable use cases, where an epoch style time stamp is 
required this should be enough.

HTH,

Tom
msg57051 - (view) Author: Skip Montanaro (skip.montanaro) * (Python triager) Date: 2007-11-02 15:42
Tom> unless I'm missing something important this will do the trick quite
    Tom> nicely:

    >>>> from datetime import datetime
    >>>> datetime(2007, 12, 24, 20, 0).strftime("%s")
    Tom>     '1198522800'

    Tom> For most imaginable use cases, where an epoch style time stamp is
    Tom> required this should be enough.

No fractions of a second...

Skip
msg57052 - (view) Author: Jon Ribbens (jribbens) * Date: 2007-11-02 15:51
> No fractions of a second...

If we're expecting floating-point, then everything you said earlier
about the limitations of ints was a bit redundant ;-)
msg57057 - (view) Author: Skip Montanaro (skip.montanaro) * (Python triager) Date: 2007-11-02 16:55
>> No fractions of a second...

    Jon> If we're expecting floating-point, then everything you said earlier
    Jon> about the limitations of ints was a bit redundant ;-)

Yes, sorry.  I responded to the mail without going back and reviewing the
entire thread.  (It's been a couple months.)  Note however that my example
to_timestamp() function converts the microseconds field to seconds and
include them in the result.

So, are we concluding that nothing needs to be added to the datetime module?

Skip
msg57058 - (view) Author: Skip Montanaro (skip.montanaro) * (Python triager) Date: 2007-11-02 17:00
One other thing worth noting is that %s is not universally
available.  Solaris, for example, lacks it in its strftime()
implementation.  (It has a bazillion other non-standard
format strings but not %s.)

Skip
msg57059 - (view) Author: Jon Ribbens (jribbens) * Date: 2007-11-02 18:18
Well, I still think that a convert-to-time_t function is essential, and
I don't buy any of the counter-arguments so far. The only argument I can
see is "should it return float or integer?" - floats are inaccurate and
integers can't represent partial seconds.
msg57078 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2007-11-02 23:40
This is not going to go anywhere without someone offering a patch to
implement this.  Plus I suspect this has a much greater use in the C API
than in the Python API for datetime.
msg57080 - (view) Author: Jon Ribbens (jribbens) * Date: 2007-11-03 00:08
Skip has already provided what amounts to a patch. It just needs to be
decided whether to (a) not include it, (b) include it with the floating
point part, or (c) include it without the floating point part.

I couldn't comment as to how many people need it. I can say that I need
it, and anyone else who's used to manipulating dates and times either
"in a unixy sort of way" or with libraries that are expecting time_t's
will need it.
msg58007 - (view) Author: Chris AtLee (catlee) Date: 2007-11-30 20:50
I keep needing to know the number of seconds that a timedelta represents,
so I implemented the following patch.

This returns only the integer portion, but could be modified to return a
floating point value.
msg58009 - (view) Author: Skip Montanaro (skip.montanaro) * (Python triager) Date: 2007-11-30 21:13
Chris> I keep needing to know the number of seconds that a timedelta
    Chris> represents, so I implemented the following patch.

I can sympathize, but if we accept this patch, for symmetry reasons
shouldn't we also add .todays, .tomicroseconds and maybe even .toweeks?

Skip
msg58068 - (view) Author: Chris AtLee (catlee) Date: 2007-12-01 21:01
timedelta.todays() could be useful in general I suppose.  I think
timedelta.toweeks() could be omitted since it's simple division by an
easy to recognize constant...also the timedelta docs say that it stores
data in terms of microseconds, seconds, and days.

OTOH tohours(), tominutes(), etc. (all the units that the constructor
uses) could be useful in some cases.

My feeling is that adding a method for each time unit bloats the API too
much.  Personally I've only ever wanted to know the number of seconds a
timedelta represents. It seems like seconds are a good base unit to
support since it allows easier interoperability with other python types
and external data.
msg69516 - (view) Author: Erik Stephens (erik.stephens) Date: 2008-07-10 18:35
I'm not sure this is still being considered, so I just wanted to add my
opinion.  This seems like such a simple request being made overly
complicated.  We just want seconds since epoch since that is used in
many other functions and libraries (i.e. the time module,
os.stat().st_mtime).  Why is there toordinal()?!?  If there is going to
be any toxxx() methods, there should definitely be a
tosecs()/totimestamp() method or an 'epoch/timestamp' attribute.  For
boundary cases, I would look to the std time module for guidance.
msg75408 - (view) Author: Steve Roberts (steve.roberts) Date: 2008-10-31 08:17
I would like to add a use case, generating random dates. Especially
generating random dates/times according to different probability
distributions and within ranges. It would be nicest if operations could
be done directly with datetime, date, time and timedelta objects, but
easy conversion to and from timestamps as requested here would also make
for reasonably convenient code (i.e. get timestamp conversions from
objects to represent ranges, generate random numbers and convert back)
msg75410 - (view) Author: Skip Montanaro (skip.montanaro) * (Python triager) Date: 2008-10-31 10:22
Unassigning.  Haven't heard from Tim in quite awhile and he's made no 
input on this issue.  Also bump to 2.7.
msg75411 - (view) Author: Marc-Andre Lemburg (lemburg) * (Python committer) Date: 2008-10-31 10:23
You can have all this by using the time module's functions.

It is true that these don't work for all dates, but they are still
useful to have.

FWIW: mxDateTime has always had methods to convert the date/time values
to ticks and back again:

    http://www.egenix.com/products/python/mxBase/mxDateTime/

Since most of the datetime module was inspired by mxDateTime, I wonder
why these were left out.
msg75423 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2008-10-31 17:20
I wrote this function is my program:

def timedelta2seconds(delta):
    """
    Convert a datetime.timedelta() objet to a number of second
    (floatting point number).

    >>> timedelta2seconds(timedelta(seconds=2, microseconds=40000))
    2.04
    >>> timedelta2seconds(timedelta(minutes=1, milliseconds=250))
    60.25
    """
    return delta.microseconds / 1000000.0 \
        + delta.seconds + delta.days * 60*60*24

About the use cases: I use it to compute the compression rate of an 
audio song (bytes / seconds), to compute the bit rate of a video 
(bytes / seconds). I'm using datetime.timedelta() to store the 
audio/video duration.

It's not related to time_t: see issue #2736 for 
datetime.datetime.totimestamp(). And about time_t: I don't about 31 
bits signed integer. It's not beacuse other programs have arbitrary 
limits than Python should also be limited.

About the patch: I don't like the name "tosecs", it's not consistent 
with the constructor: timedelta(seconds=...).tosec[ond]s(). And why 
dropping the microseconds? For short duration, microseconds are 
useful.
msg75426 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2008-10-31 18:12
New patch based on catlee's patch: add toseconds() method to timedelta 
but use the float type to keep the microseconds. The error between the 
exact value and IEEE754 value is small: the error is smaller than one 
second even with 999999999 days:

 - with less than 16 days, the microsecond error is smaller than 
0,01%.
 - with less than 1000 days, the microsecond error is smaller than 1%
 - with 99500 days or more, the microsecond error is bigger than 90%

To compute the error in microseconds:
>>> d=100; m=1
>>> e=abs((timedelta(days=d, microseconds=m).toseconds() - 
d*86400)*1e6 - m)
>>> e*100/m
0.024044513702392578

For the seconds, this is no error even with 999999999 days. Error in 
seconds:

>>> d=999999999; s=1; timedelta(days=d, seconds=m).toseconds() - 
d*86400, s
(1.0, 1)

(no error, both values are the same)
msg75724 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2008-11-11 03:12
See also issue2736
msg75754 - (view) Author: Alexander Belopolsky (belopolsky) * (Python committer) Date: 2008-11-11 18:29
Chris> I keep needing to know the number of seconds that a timedelta
Chris> represents.

I propose an alternative approach that I believe will neatly solve 
fractional vs. whole seconds and multitude of conceivable toxxx methods:

Let's extend timedelta's div and floordiv methods to allow 

>>> (t - epoch) // timedelta(seconds=1)
--> whole seconds

and  

>>> (t - epoch) / timedelta(seconds=1)
--> fractional seconds
msg75859 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2008-11-14 11:46
>>> (t - epoch) // timedelta(seconds=1)

I don't like this syntax, because I can't guess the result unit:
  datetime - datetime -> timedelta
but:
  timedelta / timedelta -> seconds? days? nanoseconds?

If you example, you used timedelta(seconds=1), but what is the result 
unit if you use timedelta(hours=1)? or timedelta(days=1, 
microseconds=1)?

The problem is that timedelta has no unit (or has multiple units), 
whereas timedelta.toseconds() are seconds. So about your example:

>>> (t - epoch).toseconds()
--> fractional seconds
>>> int((t - epoch).toseconds())
--> whole seconds
msg75860 - (view) Author: Amaury Forgeot d'Arc (amaury.forgeotdarc) * (Python committer) Date: 2008-11-14 12:17
> timedelta / timedelta -> seconds? days? nanoseconds?

The quotient of two timedelta is a dimensionless number with no unit:
    timedelta(hours=1) / timedelta(minutes=5) == 12.0
This seems well defined, where is the ambiguity?
msg75863 - (view) Author: Alexander Belopolsky (belopolsky) * (Python committer) Date: 2008-11-14 15:18
I was going to say the same as Amaury: timedelta / timedelta is
dimensionless (time units cancel each other in division) and the
advantage of this notation is that you get a way to express
timedelta.toxxx for all units accepted in constructor and even toweeks
becomes simple d / timedelta(7).

I've started flashing out a patch and then recalled that I've seen one
at issue2706 .  So instead of attaching a new patch here, I am going
to review the one in issue2706 now.

There was also some related discussion at issue4291 .   Apparently it
has been suggested that timedelta to integer and float conversions
would be a better alternative to timedelta / timedelta division.   I
disagree.  Integer conversion is ambiguous - should it be to seconds,
to microseconds or to timedelta.resolution (whatever that will become
in the future)?  Floating point conversion may loose precision as Tim
pointed out in msg26266 .  That would lead users to scratching their
heads over what to use float(d1)/float(d2) or float(d1)/int(d2) or
even int(d1)/int(d2) with true division on.

This said, let's move this discussion to issue2706 now.

On Fri, Nov 14, 2008 at 7:17 AM, Amaury Forgeot d'Arc
<report@bugs.python.org> wrote:
>
> Amaury Forgeot d'Arc <amauryfa@gmail.com> added the comment:
>
>> timedelta / timedelta -> seconds? days? nanoseconds?
>
> The quotient of two timedelta is a dimensionless number with no unit:
>    timedelta(hours=1) / timedelta(minutes=5) == 12.0
> This seems well defined, where is the ambiguity?
>
> ----------
> nosy: +amaury.forgeotdarc
>
> _______________________________________
> Python tracker <report@bugs.python.org>
> <http://bugs.python.org/issue1673409>
> _______________________________________
>
msg77859 - (view) Author: Sebastian Rittau (srittau) * Date: 2008-12-15 11:28
A timedelta.toseconds method (or equivalent) makes no sense. The number
of seconds in a day is not fixed (due to leap seconds) and relying on
such a method would introduce subtle bugs. The only way to find out the
number of seconds in a range of dates is if you have a concrete date range.
msg77861 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2008-12-15 12:03
> The number of seconds in a day is not fixed (due to leap seconds)

POSIX timestamp doesn't count leap seconds. It looks like the datetime 
module is not aware of the leap seconds:

>>> print datetime.datetime(2006, 1, 1) - datetime.datetime(2005, 12, 
31)
1 day, 0:00:00

About my method: I finally prefer datetime/datetime or 
datetime//datetime instead of a toseconds() method. And to convert a 
timestamp to a timestamp: see my patch attached to issue #2736.
msg77862 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2008-12-15 12:04
I removed my ".toseconds() method" patch because I prefer division. 
See issue #2706 for divison, divmod, etc.
msg77865 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2008-12-15 12:57
I also think totimestamp() on datetime objects would be useful, I've
missed it myself a couple of times. The return value should be similar
to that of time.time(), i.e. a float.
msg77867 - (view) Author: Sebastian Rittau (srittau) * Date: 2008-12-15 13:14
Leap second handling is usually configurable. Default on Debian Linux
(but similar on RHEL and SuSE):

>>> int(date(1994,1,1).strftime("%s")) - int(date(1993,1,1).strftime("%s"))
31536000

After doing "cp /usr/share/zoneinfo/right/Europe/Berlin /etc/localtime":

>>> int(date(1994,1,1).strftime("%s")) - int(date(1993,1,1).strftime("%s"))
31536001

Also, NTP servers usually get this right. I don't think, Python should
promote a wrong date handling by default, even if it's convenient.
msg77868 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2008-12-15 13:15
> (...) totimestamp() (...) return value should be similar
> to that of time.time(), i.e. a float

float is a source of many problems (rounding problems), especially for 
huge values: float is unable to store correctly microseconds for big 
values: see msg75426. A simple tuple (int, int) is simple and there is 
no rounding/float limit.
msg77869 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2008-12-15 14:18
What you say is "with 99500 days or more, the microsecond error is
bigger than 90%". It means that with epoch starting at 1970, you can
still return timestamps with a 1-2 microsecond accuracy for the year 2242.

Additional precision would be overkill.
msg77871 - (view) Author: Jon Ribbens (jribbens) * Date: 2008-12-15 14:58
> A timedelta.toseconds method (or equivalent) makes no sense.
> The number of seconds in a day is not fixed (due to leap seconds) and
> relying on such a method would introduce subtle bugs.

You are misunderstanding what timedelta is. It is a fixed-length period
of time. It is stored as a number of (24x3600-second) days, seconds and
microseconds. There is no "start date" or "end date", so the concept of
leap seconds just does not apply.
msg103868 - (view) Author: Alexander Belopolsky (Alexander.Belopolsky) Date: 2010-04-21 17:49
With issue2706 accepted, I think this issue can now be rejected because proposed td.tosecs() can now be spelled simply as td/timedelta(seconds=1).

The other RFE for a totimestamp() method is a duplicate of issue2736.
msg103874 - (view) Author: Mark Dickinson (mark.dickinson) * (Python committer) Date: 2010-04-21 18:03
Closing this as a duplicate of issue 2736, as suggested.  I'll combine the nosy lists.

(BTW, as well as the newly introduced division methods, td.tosecs already exists, except that it's spelt td.total_seconds.)
msg103878 - (view) Author: Alexander Belopolsky (Alexander.Belopolsky) Date: 2010-04-21 18:18
On Wed, Apr 21, 2010 at 2:03 PM, Mark Dickinson <report@bugs.python.org> wrote:
..
> (BTW, as well as the newly introduced division methods, td.tosecs already exists, except that it's spelt td.total_seconds.)

Aggrrr!  How did that slip in? :-)

86399999913600.0

Shouldn't td.total_seconds() return Decimal?  Maybe in py4k ...
msg103888 - (view) Author: Mark Dickinson (mark.dickinson) * (Python committer) Date: 2010-04-21 18:52
> Aggrrr!  How did that slip in? :-)

Blame Antoine.  :)  (See issue 5788 and revision 76529.)

> Shouldn't td.total_seconds() return Decimal?  Maybe in py4k ...

Yes, that would have been nice.

I'm not sure that the Decimal type is well-established enough yet that it's okay to return Decimal instances from random unrelated modules, though.  Maybe someday.
History
Date User Action Args
2010-04-21 18:52:38mark.dickinsonsetmessages: + msg103888
2010-04-21 18:18:03Alexander.Belopolskysetmessages: + msg103878
2010-04-21 18:03:30mark.dickinsonsetstatus: open -> closed
resolution: duplicate
superseder: datetime needs an "epoch" method
messages: + msg103874
2010-04-21 17:49:02Alexander.Belopolskysetnosy: + Alexander.Belopolsky, mark.dickinson, - belopolsky
messages: + msg103868
2008-12-16 11:08:17lemburgsetnosy: - lemburg
2008-12-15 19:24:08brett.cannonsetnosy: - brett.cannon
2008-12-15 14:58:40jribbenssetmessages: + msg77871
2008-12-15 14:18:43pitrousetmessages: + msg77869
2008-12-15 13:15:46vstinnersetmessages: + msg77868
2008-12-15 13:14:11srittausetmessages: + msg77867
2008-12-15 12:57:24pitrousetnosy: + pitrou
messages: + msg77865
2008-12-15 12:04:39vstinnersetmessages: + msg77862
2008-12-15 12:03:25vstinnersetfiles: - timedelta_toseconds_float.patch
2008-12-15 12:03:12vstinnersetmessages: + msg77861
2008-12-15 11:28:48srittausetnosy: + srittau
messages: + msg77859
2008-11-14 15:18:25belopolskysetmessages: + msg75863
2008-11-14 12:17:12amaury.forgeotdarcsetnosy: + amaury.forgeotdarc
messages: + msg75860
2008-11-14 11:46:26vstinnersetmessages: + msg75859
2008-11-11 18:29:45belopolskysetnosy: + belopolsky
messages: + msg75754
2008-11-11 03:12:53vstinnersetmessages: + msg75724
2008-10-31 18:12:25vstinnersetfiles: + timedelta_toseconds_float.patch
keywords: + patch
messages: + msg75426
2008-10-31 17:20:28vstinnersetnosy: + vstinner
messages: + msg75423
2008-10-31 10:23:46lemburgsetnosy: + lemburg
messages: + msg75411
2008-10-31 10:22:57skip.montanarosetassignee: tim.peters ->
messages: + msg75410
versions: + Python 2.7, - Python 2.6
2008-10-31 08:17:24steve.robertssetnosy: + steve.roberts
messages: + msg75408
2008-07-10 18:35:42erik.stephenssetnosy: + erik.stephens
messages: + msg69516
2007-12-01 21:01:03catleesetmessages: + msg58068
2007-11-30 21:13:21skip.montanarosetmessages: + msg58009
2007-11-30 20:50:13catleesetfiles: + timedelta.diff
nosy: + catlee
messages: + msg58007
2007-11-03 00:08:58jribbenssetmessages: + msg57080
2007-11-02 23:41:00brett.cannonsetnosy: + brett.cannon
messages: + msg57078
2007-11-02 18:18:39jribbenssetmessages: + msg57059
2007-11-02 17:00:00skip.montanarosetmessages: + msg57058
2007-11-02 16:55:24skip.montanarosetmessages: + msg57057
2007-11-02 15:51:36jribbenssetmessages: + msg57052
2007-11-02 15:42:34skip.montanarosetmessages: + msg57051
2007-11-02 15:02:16tomstersetnosy: + tomster
messages: + msg57050
2007-09-02 16:07:11skip.montanarosetmessages: + msg55586
2007-09-02 14:03:10jribbenssetmessages: + msg55585
2007-09-02 03:22:52skip.montanarosetnosy: + skip.montanaro
messages: + msg55573
2007-03-04 11:59:05jribbenscreate