classification
Title: datetime: support leap seconds
Type: Stage:
Components: Library (Lib) Versions: Python 2.7
process
Status: closed Resolution: wont fix
Dependencies: Superseder:
Assigned To: Nosy List: akira, belopolsky, dlroo, doughellmann, haypo, lemburg
Priority: normal Keywords: patch

Created on 2015-03-03 14:42 by haypo, last changed 2015-07-31 18:32 by belopolsky. This issue is now closed.

Files
File name Uploaded Description Edit
datetime_leapsecond.patch haypo, 2015-03-03 14:43 review
support_leap_seconds.patch haypo, 2015-03-03 15:45 review
Messages (13)
msg237142 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2015-03-03 14:42
A leap second will be added in June 2015:
http://www.usatoday.com/story/tech/2015/01/08/computer-chaos-feares/21433363/

The datetime module explicitly doesn't support leap seconds:
https://docs.python.org/dev/library/datetime.html#datetime.date.fromtimestamp
"Note that on non-POSIX systems that include leap seconds in their notion of a timestamp, leap seconds are ignored by fromtimestamp()."

The following bug in oslo.utils was reported because datetime is indirectly used to unserialize a date, but it fails with ValueError("second must be in 0..59") if the second is 60:
https://bugs.launchpad.net/oslo.utils/+bug/1427212

Would it be possible to silently drop ignore leap seconds in datetime.datetime constructor, as already done in datetime.datetime.fromtimestamp?

Attached patch modified datetime constructor to drop leap seconds: replace second=60 with second=59. I also changed the error message for second (valid range is now 0..60).
msg237144 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2015-03-03 14:52
Leap seconds are ignored, so a difference of <datetime before the leap second> and <datetime with the leap second> is zero:

>>> import datetime
>>> t1=datetime.datetime(2012, 6, 30, 23, 59, 59)
>>> t2=datetime.datetime(2012, 6, 30, 23, 59, 59)
>>> t2-t1
datetime.timedelta(0)

Supporting leap seconds might be possible, but it requires much more work.
msg237146 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2015-03-03 15:37
Ignoring leap seconds introduces unexpected result.

datetime.timestamp -> datetime.fromtimestamp drops one second:

$ ./python
Python 3.5.0a1+ (default:760f222103c7+, Mar  3 2015, 15:36:36) 
>>> t=datetime.datetime(2012, 6, 30, 23, 59, 60).timestamp()
>>> datetime.datetime.fromtimestamp(t)
datetime.datetime(2012, 6, 30, 23, 59, 59)

time and datetime modules behave differently:

$ ./python
Python 3.5.0a1+ (default:760f222103c7+, Mar  3 2015, 15:36:36) 
>>> import datetime, time
>>> t1=datetime.datetime(2012, 6, 30, 23, 59, 59).timestamp()
>>> t2=datetime.datetime(2012, 6, 30, 23, 59, 60).timestamp()
>>> t2-t1
0.0

>>> t3=time.mktime((2012, 6, 30, 23, 59, 59, -1, -1, -1))
>>> t4=time.mktime((2012, 6, 30, 23, 59, 60, -1, -1, -1))
>>> t4-t3
1.0

>>> t1 == t2 == t3
True
>>> t3, t4
(1341093599.0, 1341093600.0)
msg237147 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2015-03-03 15:45
support_leap_seconds.patch: different approach, accept second=60. Problem: fromtimestamp() returns the wrong day.

haypo@smithers$ ./python
Python 3.5.0a1+ (default:760f222103c7+, Mar  3 2015, 15:36:36) 
>>> import datetime
>>> datetime.datetime(2012, 6, 30, 23, 59, 60)
datetime.datetime(2012, 6, 30, 23, 59, 60)
>>> dt1=datetime.datetime(2012, 6, 30, 23, 59, 60)
>>> t1=datetime.datetime(2012, 6, 30, 23, 59, 60).timestamp()
>>> dt2=datetime.datetime.fromtimestamp(t1)
>>> dt2
datetime.datetime(2012, 7, 1, 0, 0)
>>> dt2 == dt1
False
>>> dt1
datetime.datetime(2012, 6, 30, 23, 59, 60)
>>> print(dt1)
2012-06-30 23:59:60
>>> print(dt2)
2012-07-01 00:00:00

>>> import time
>>> time.mktime((2012, 6, 30, 23, 59, 60, -1, -1, -1))
1341093600.0
>>> t1
1341093600.0
>>> t2=time.mktime((2012, 6, 30, 23, 59, 60, -1, -1, -1))
>>> t2 == t1
True
>>> time.localtime(time.mktime((2012, 6, 30, 23, 59, 60, -1, -1, -1)))
time.struct_time(tm_year=2012, tm_mon=7, tm_mday=1, tm_hour=0, tm_min=0, tm_sec=0, tm_wday=6, tm_yday=183, tm_isdst=1)


http://cr.yp.to/proto/utctai.html

"""
For many years, the UNIX localtime() time-display routine didn't support leap seconds.
(...)
Why not fix it?
(...)
The main obstacle is POSIX. POSIX is a ``standard'' designed by a vendor consortium several years ago to eliminate progress and protect the installed base. The behavior of the broken localtime() libraries was documented and turned into a POSIX requirement.
"""
msg237148 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2015-03-03 15:48
Oh, mktime() returns the same timestamp with and without the leap second:

>>> time.mktime((2012, 6, 30, 23, 59, 59, -1, -1, -1))
1341093599.0
>>> time.mktime((2012, 6, 30, 23, 59, 60, -1, -1, -1))
1341093600.0
>>> time.mktime((2012, 7, 1, 0, 0, 0, -1, -1, -1))
1341093600.0
msg237413 - (view) Author: Alexander Belopolsky (belopolsky) * (Python committer) Date: 2015-03-07 03:39
> POSIX is a ``standard'' designed by a vendor consortium several years ago to eliminate progress and protect the installed base.


No, POSIX is an attempt to bring some sanity to the installed base of human calendars.  The established standard tell's us that a year is 365 days.  Wait, every 4-th year is 366 days, except some other rule every 400 years.

POSIX says: fine as long as we can enumerate all YYYY-MM-DD's, we can live  with it.  But the line is drawn where each day is divided into 86,400 seconds.

The problem is that unlike ancient astronomers who were finding better and better approximations to the ratio of two Earth's rotation periods (around the Sun and around itself) every few hundred years, modern astronomers will tell us how many seconds there will be in any given year with only a six month notice.
msg238004 - (view) Author: Akira Li (akira) * Date: 2015-03-13 03:13
POSIX timestamp doesn't count (literally) past/future leap seconds.
It allows to find out that the timestamp 2**31-1 corresponds to
2038-01-19T03:14:07Z (UTC) regardless of how many leap seconds will
occur before 2038:

  >>> from datetime import datetime, timedelta
  >>> str(datetime(1970,1,1) + timedelta(seconds=2**31-1))
  '2038-01-19 03:14:07'

If you use "right" timezone then mktime() may count leap seconds:

  $ TZ=right/UTC ./python
  >>> import time
  >>> time.mktime((2012, 6, 30, 23, 59, 59, -1, -1, -1))
  1341100823.0
  >>> time.mktime((2012, 6, 30, 23, 59, 60, -1, -1, -1))
  1341100824.0
  >>> time.mktime((2012, 7, 1, 0, 0, 0, -1, -1, -1))
  1341100825.0

It is a different time scale. There are no leap seconds in TAI:

  >>> str(datetime(1970,1,1, 0,0, 10) + timedelta(seconds=1341100825))
  '2012-07-01 00:00:35'

i.e., 2012-07-01 00:00:35 TAI that corresponds to 2012-07-01 00:00:00
UTC. Each positive leap second increases the difference TAI-UTC (on
2015-07-01UTC it will be 36 [1]).

TAI-UTC in the future (more than 6 months) is unknown but it is less
than ~200 seconds until 2100 [2].

It might be convenient to think about datetime as a broken-down
timestamp and therefore

  (datetime(2012,6,30,23,59,60) - epoch) == 
  (datetime(2012,7, 1, 0, 0, 0) - epoch)

The code [3] that silently truncates 60 to 59 when datetime
constructor is called implicitly should retire.

Use case: parse timestamps that might include a leap second [4]

[1] https://hpiers.obspm.fr/iers/bul/bulc/bulletinc.dat
[2] http://www.ucolick.org/~sla/leapsecs/year2100.html
[3] https://bugs.python.org/msg155689
[4] http://stackoverflow.com/q/21027639
msg244064 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2015-05-25 22:58
Sorry, I give up on this issue. I don't know how to fix it, nor if it's possible to fix it.
msg244151 - (view) Author: Marc-Andre Lemburg (lemburg) * (Python committer) Date: 2015-05-27 09:12
Here's what mxDateTime uses:

>>> import mx.DateTime
>>>
>>> t1 = mx.DateTime.DateTime(2012,6,30,23,59,60)
>>> t2 = mx.DateTime.DateTime(2012,7,1,0,0,0)
>>>
>>> t1
<mx.DateTime.DateTime object for '2012-06-30 23:59:60.00' at 7fbb36008d68>
>>> t2
<mx.DateTime.DateTime object for '2012-07-01 00:00:00.00' at 7fbb36008d20>
>>>
>>> t2-t1
<mx.DateTime.DateTimeDelta object for '00:00:00.00' at 7fbb35ff0540>
>>> (t2-t1).seconds
0.0
>>>
>>> t1 + mx.DateTime.oneSecond
<mx.DateTime.DateTime object for '2012-07-01 00:00:01.00' at 7fbb360083d8>

It preserves the broken down values, but uses POSIX days of 86400 seconds per day to calculate time deltas.

It's a compromise, not a perfect solution, but it prevents applications from failing for that one second every now and then.

I don't believe there is a perfect solution, since what your application or users expect may well be different. All I can say is that raising exceptions in these rare cases is not what your users typically want :-)
msg247059 - (view) Author: (dlroo) Date: 2015-07-21 20:15
If you are using mx.DateTime make certain you do not use the .strftime method.  If you use .strftime method and have a 60th second in your DateTime object it will crash python with no error message.  This occurs because the .strftime method is fully inherited from Python's datetime.datetime.
msg247743 - (view) Author: Marc-Andre Lemburg (lemburg) * (Python committer) Date: 2015-07-31 11:33
On 21.07.2015 22:15, dlroo wrote:
> 
> dlroo added the comment:
> 
> If you are using mx.DateTime make certain you do not use the .strftime method.  If you use .strftime method and have a 60th second in your DateTime object it will crash python with no error message.  This occurs because the .strftime method is fully inherited from Python's datetime.datetime.

Thanks for the report. We will fix this in the next mxDateTime release.
msg247760 - (view) Author: (dlroo) Date: 2015-07-31 18:11
Is it possible to modify datetime so that the check_time_args function in the datetimemodule.c does not error when given a seconds value of greater than 59?  I was thinking that if the seconds were greater than 59, the seconds are set to 59 and any extra seconds are kept in a book keeping "attribute" (not a real attribute because its C) that is accessible from the Python side?  You would have to make the seconds argue passed by reference (thus returning a modified second).  Also would want the book keeping value to be zero in nominal conditions.
This would do nothing for any of the datetime arithmetic, but that can be handled externally.
msg247762 - (view) Author: Alexander Belopolsky (belopolsky) * (Python committer) Date: 2015-07-31 18:32
Please redirect this discussion to the recently opened datetime-sig mailing list.

https://mail.python.org/pipermail/datetime-sig/
History
Date User Action Args
2015-07-31 18:32:55belopolskysetmessages: + msg247762
2015-07-31 18:11:50dlroosetmessages: + msg247760
2015-07-31 11:33:12lemburgsetmessages: + msg247743
2015-07-21 20:15:09dlroosetnosy: + dlroo

messages: + msg247059
versions: + Python 2.7, - Python 3.5
2015-05-27 09:12:20lemburgsetnosy: + lemburg
messages: + msg244151
2015-05-25 22:58:18hayposetstatus: open -> closed
resolution: wont fix
messages: + msg244064
2015-03-13 03:13:40akirasetnosy: + akira
messages: + msg238004
2015-03-07 03:39:56belopolskysetmessages: + msg237413
2015-03-03 15:48:15hayposetmessages: + msg237148
2015-03-03 15:45:19hayposetfiles: + support_leap_seconds.patch

messages: + msg237147
2015-03-03 15:37:21hayposetmessages: + msg237146
2015-03-03 15:02:18doughellmannsetnosy: + doughellmann
2015-03-03 14:52:00hayposetmessages: + msg237144
2015-03-03 14:43:29hayposetfiles: + datetime_leapsecond.patch
keywords: + patch
2015-03-03 14:42:23haypocreate