classification
Title: Calling timestamp() on a datetime object modifies the timestamp of a different datetime object.
Type: enhancement Stage:
Components: Library (Lib) Versions: Python 3.6
process
Status: closed Resolution: duplicate
Dependencies: Superseder: Do not call localtime (gmtime) in datetime module
View: 28067
Assigned To: belopolsky Nosy List: Arfrever, akira, barry, belopolsky, lemburg, pitrou, thomir, veebers
Priority: normal Keywords:

Created on 2014-10-14 01:31 by veebers, last changed 2016-09-10 19:39 by belopolsky. This issue is now closed.

Files
File name Uploaded Description Edit
mkbug.py pitrou, 2014-10-14 07:58
Messages (12)
msg229276 - (view) Author: Christopher Lee (veebers) Date: 2014-10-14 01:31
I have an example script here[1].
This script creates 2 datetime objects (using a timedelta work around to deal with large timestamps).
It then makes 2 assertions, that the timestamp of the created object is the same as the one that was used to create it. (when run with no arguments this script passes both assertions).
However, if the argument 'breakme' is passed to the script then after the first assertion the method 'timestamp()' is called on a different (un-asserted) datetime which will now make the 2nd assertion fail.

[1] http://paste.ubuntu.com/8556130/
msg229279 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2014-10-14 07:58
This has nothing to do with the datetime module. Attached script reduces the issue to a bug (?) in time.mktime() with the corresponding timezone. time.mktime() is a thin wrapper around the C library's mktime() function, so it is probably not a bug in Python at all.

Note your script is fixed by removing ".replace(tzinfo=None)". It seems conter-productive to take the pain to create an aware timezone and then make it naive, IMHO. datetime.timestamp() falls back on time.mktime() when the datetime is naive (i.e. it asks the OS to do the computation).

(I did my tests under Ubuntu 13.10, btw)
msg229344 - (view) Author: Christopher Lee (veebers) Date: 2014-10-14 20:13
Hi Antoine, thanks for taking a look.

I should explain further. This code is for an introspection tool[1] that provides an interface to write tests in python against an application.

This code is a workaround for using large timestamps[2] (i.e. larger than 32bit time_t) the original suggested code (from SO, using timedelta) has an inconsistancy with the hour in some cases (this example in the NZ timezone):
>>> datetime.datetime.fromtimestamp(2047570047)
datetime.datetime(2034, 11, 19, 17, 27, 27)

>>> datetime.datetime.fromtimestamp(0) + datetime.timedelta(seconds=2047570047)
datetime.datetime(2034, 11, 19, 18, 27, 27)

The code you see here in my example, creating a timezone aware object, is a result of 'fixing' this behaviour I've seen.
The reason for then removing the tzinfo is because currently our API states that dates returned from the introspected objects are naive.


[1] https://launchpad.net/autopilot
[2] https://bugs.launchpad.net/autopilot/+bug/1328600
msg229388 - (view) Author: Alexander Belopolsky (belopolsky) * (Python committer) Date: 2014-10-15 00:17
In general, you cannot expect datetime.fromtimestamp(0) + timedelta(seconds) to return the same value as datetime.fromtimestamp(seconds).  This will only be true if you are lucky enough to live in an area where local government did not mess with timezones since 1970.  In your particular case, I think what is happening is that you have DST in January, but you did not have that on January 1, 1970.

$ zdump -v NZ | grep NZDT | head
NZ  Sat Nov  2 14:00:00 1974 UTC = Sun Nov  3 03:00:00 1974 NZDT isdst=1
NZ  Sat Feb 22 13:59:59 1975 UTC = Sun Feb 23 02:59:59 1975 NZDT isdst=1
NZ  Sat Oct 25 14:00:00 1975 UTC = Sun Oct 26 03:00:00 1975 NZDT isdst=1
NZ  Sat Mar  6 13:59:59 1976 UTC = Sun Mar  7 02:59:59 1976 NZDT isdst=1
NZ  Sat Oct 30 14:00:00 1976 UTC = Sun Oct 31 03:00:00 1976 NZDT isdst=1
NZ  Sat Mar  5 13:59:59 1977 UTC = Sun Mar  6 02:59:59 1977 NZDT isdst=1
NZ  Sat Oct 29 14:00:00 1977 UTC = Sun Oct 30 03:00:00 1977 NZDT isdst=1
NZ  Sat Mar  4 13:59:59 1978 UTC = Sun Mar  5 02:59:59 1978 NZDT isdst=1
NZ  Sat Oct 28 14:00:00 1978 UTC = Sun Oct 29 03:00:00 1978 NZDT isdst=1
NZ  Sat Mar  3 13:59:59 1979 UTC = Sun Mar  4 02:59:59 1979 NZDT isdst=1
msg229390 - (view) Author: Alexander Belopolsky (belopolsky) * (Python committer) Date: 2014-10-15 01:01
Antoine,

I don't think the behavior that you have shown is a bug in strict sense.

On my Mac, I get

$ python mkbug.py breakme
1396702800.0
1396702800.0

but on Linux,

$ python mkbug.py breakme
1396706400.0
1396702800.0

The problem here is that time.mktime((2014, 4, 6, 2, 0, 0, -1, -1, -1))
is undefined  and both 1396706400 and 1396702800 timestamps are valid guesses for what 2014-04-06T02 was in NZ: 

$ TZ=NZ date -d @1396706400
Sun Apr  6 02:00:00 NZST 2014
$ TZ=NZ date -d @1396702800
Sun Apr  6 02:00:00 NZDT 2014

It is unfortunate that Linux C library (glibc?) makes different guess at different time, but I don't think this violates any applicable standards.
msg229391 - (view) Author: Christopher Lee (veebers) Date: 2014-10-15 01:05
Alexander,

Ah ok thanks for clarifying that. Am I wrong then to think that this code[1] should work as I think it should (i.e. datetime_from_large_timestamp(example_ts) == datetime.fromtimestamp(example_ts))

I'm trying to be able to handle timestamps larger than the 32bit time_t limit, which is why I'm doing these gymnastic steps.

[1] http://paste.ubuntu.com/8562027/
msg229393 - (view) Author: Alexander Belopolsky (belopolsky) * (Python committer) Date: 2014-10-15 01:45
Your code as good as your timezone library, but you should realize that by discarding tzinfo you are making your "local_stamp" ambiguous.

I am not familiar with dateutil.tz, but pytz I believe uses some tricks to make sure astimezone() result remembers the isdst flag.

If you API requires naive local datetime objects, you need to carry isdst flag separately if you want to disambiguate between  2014-04-06 02:00 NZST and 2014-04-06 02:00 NZDT.  On top of that, you will not be able to use datetime.timestamp() method and will have to use time.mktime or whatever equivalent utility your timezone library provides.

Note that I was against adding datetime.timestamp() for this specific reason: it is supposed to be inverse of datetime.fromtimestamp(), but since the later is not monotonic, no such inverse exists in the strict mathematical sense.  See msg133039 in issue 2736.

BTW, if you open a feature request to add isdst=-1 optional argument to datetime.timestamp(), you will have my +1.
msg229395 - (view) Author: Akira Li (akira) * Date: 2014-10-15 02:07
Christopher, 

About your script http://paste.ubuntu.com/8562027/

dateutil may break if the local timezone had different UTC offset in the past.
You could  use tzlocal module to get pytz timezone that can handle such 
timezones. 

To get the correct time for 1414274400 timezone in Europe/Moscow timezone,
you need the latest tz database e.g., `pytz` works but fromtimestamp, dateutil
that use the local tz database fail (2:00 instead of 1:00):

  >>> import time
  >>> import os
  >>> os.environ['TZ'] = 'Europe/Moscow'
  >>> time.tzset()
  >>> from datetime import datetime, timezone
  >>> from tzlocal import get_localzone 
  >>> datetime.fromtimestamp(1414274400, get_localzone())
  datetime.datetime(2014, 10, 26, 1, 0, tzinfo=<DstTzInfo 'Europe/Moscow' MSK+3:00:00 STD>)
  >>> datetime.utcfromtimestamp(1414274400).replace(tzinfo=timezone.utc).astimezone(get_localzone())
  datetime.datetime(2014, 10, 26, 1, 0, tzinfo=<DstTzInfo 'Europe/Moscow' MSK+3:00:00 STD>)
  >>> datetime.fromtimestamp(1414274400) # wrong
  datetime.datetime(2014, 10, 26, 2, 0)
  >>> datetime.fromtimestamp(1414274400, timezone.utc).astimezone() # wrong
  datetime.datetime(2014, 10, 26, 2, 0, tzinfo=datetime.timezone(datetime.timedelta(0, 14400), 'MSK'))
  >>> datetime.utcfromtimestamp(1414274400).replace(tzinfo=timezone.utc).astimezone() # wrong
  datetime.datetime(2014, 10, 26, 2, 0, tzinfo=datetime.timezone(datetime.timedelta(0, 14400), 'MSK'))
  >>> from dateutil.tz import gettz, tzutc
  >>> datetime.fromtimestamp(1414274400, gettz()) # wrong
  datetime.datetime(2014, 10, 26, 2, 0, tzinfo=tzfile('/usr/share/zoneinfo/Europe/Moscow'))
  >>> datetime.fromtimestamp(1414274400, tzutc()).astimezone(gettz()) # wrong
  datetime.datetime(2014, 10, 26, 2, 0, tzinfo=tzfile('/usr/share/zoneinfo/Europe/Moscow'))
  >>> datetime.utcfromtimestamp(1414274400).replace(tzinfo=tzutc()).astimezone(gettz()) # wrong
  datetime.datetime(2014, 10, 26, 2, 0, tzinfo=tzfile('/usr/share/zoneinfo/Europe/Moscow'))


To avoid surprises, always use UTC time to perform date arithmetics:

  EPOCH = datetime(1970, 1,1, tzinfo=pytz.utc)
  utc_dt = EPOCH + timedelta(seconds=timestamp)

should work for dates after 2038 too.

To convert it to the local timezone:

  from tzlocal import get_localzone

  local_dt = utc_dt.astimezone(get_localzone())
  ts = (local_dt - EPOCH) // timedelta(seconds=1)
  assert ts == timestamp # if timestamp is an integer

Python stdlib assumes POSIX encoding for time.time() value (issue22356)  
therefore the formulae work on all platforms where Python works.
msg229401 - (view) Author: Christopher Lee (veebers) Date: 2014-10-15 04:10
Thanks Akira, everyone for all the info.

It looks like I've highjacked this bug comments to trying to solve my first problem (i.e. datetime objects for large timestamps) instead of the bug at hand, I feel should move that conversation elsewhere.

It appears from Alexander's comment that it might not be a bug. I need to figure out if I need to work around this or use some other mechanism.
msg229405 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2014-10-15 08:00
Christopher, I've already pointed out a fix in another message: just remove ".replace(tzinfo=None)". Doing computations on UTC datetimes (rather than naive) should ensure you don't encounter any ambiguities.
msg250859 - (view) Author: Alexander Belopolsky (belopolsky) * (Python committer) Date: 2015-09-16 18:48
I am reopening this issue as an "enhancement" because I would like to revisit it in light of PEP 495.
msg275684 - (view) Author: Alexander Belopolsky (belopolsky) * (Python committer) Date: 2016-09-10 19:39
Superseded by #28067.
History
Date User Action Args
2016-09-10 19:39:45belopolskysetstatus: open -> closed
superseder: Do not call localtime (gmtime) in datetime module
resolution: duplicate
messages: + msg275684
2015-09-16 18:48:19belopolskysetstatus: closed -> open
versions: + Python 3.6, - Python 3.4
type: behavior -> enhancement
messages: + msg250859

assignee: belopolsky
resolution: not a bug -> (no value)
2014-10-15 08:00:52pitrousetstatus: open -> closed
resolution: not a bug
messages: + msg229405
2014-10-15 04:10:12veeberssetmessages: + msg229401
2014-10-15 02:07:28akirasetnosy: + akira
messages: + msg229395
2014-10-15 01:45:11belopolskysetmessages: + msg229393
2014-10-15 01:05:36veeberssetmessages: + msg229391
2014-10-15 01:01:23belopolskysetmessages: + msg229390
2014-10-15 00:17:34belopolskysetmessages: + msg229388
2014-10-15 00:02:33Arfreversetnosy: + Arfrever
2014-10-14 20:13:21veeberssetmessages: + msg229344
2014-10-14 07:59:25pitrousetnosy: + lemburg
2014-10-14 07:58:29pitrousetfiles: + mkbug.py
nosy: + pitrou, belopolsky
messages: + msg229279

2014-10-14 01:47:35barrysetnosy: + barry
2014-10-14 01:31:45veeberscreate