classification
Title: datetime.strftime('%s') should respect tzinfo
Type: enhancement Stage: commit review
Components: Extension Modules, Library (Lib) Versions: Python 3.5
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: belopolsky Nosy List: Daniel.O'Connor, akira, belopolsky, bignose, haypo, mumino, r.david.murray, santa4nt, tim.peters
Priority: normal Keywords: easy, patch

Created on 2011-08-15 02:31 by Daniel.O'Connor, last changed 2014-07-01 15:26 by akira.

Files
File name Uploaded Description Edit
strftime.patch mumino, 2012-09-20 12:23 patch for strftime("%s") review
strftime2.patch mumino, 2014-06-26 14:56 rounding problem fixed with math.floor review
strftime3.patch mumino, 2014-07-01 13:45 more tests and some documentation added. review
Messages (25)
msg142095 - (view) Author: Daniel O'Connor (Daniel.O'Connor) Date: 2011-08-15 02:31
It isn't possible to add a timezone to a naive datetime object which means that if you are getting them from some place you can't directly control there is no way to set the TZ.

eg pywws' DataStore returns naive datetime's which are in UTC. There is no way to set this and hence strftime seems to think they are in local time.

I can sort of see why you would disallow changing a TZ once set but it doesn't make sense to prevent this for naive DTs.

Also, utcnow() returns a naive DT whereas it would seem to be more sensible to return it with a UTC TZ.
msg142129 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2011-08-15 14:09
In what way does 'replace' not satisfy your need to set the tzinfo?

As for utcnow, we can't change what it returns for backward compatibility reasons, but you can get a non-naive utc datatime by doing datetime.now(timezone.utc).  (I must admit, however, that at least this morning I can't wrap my head around how that works based on the docs :(.
msg142130 - (view) Author: Daniel O'Connor (Daniel.O'Connor) Date: 2011-08-15 14:23
On 15/08/2011, at 23:39, R. David Murray wrote:
> R. David Murray <rdmurray@bitdance.com> added the comment:
> 
> In what way does 'replace' not satisfy your need to set the tzinfo?

Ahh that would work, although it is pretty clumsy since you have to specify everything else as well.

In the end I used calendar.timegm (which I only found out about after this).

> As for utcnow, we can't change what it returns for backward compatibility reasons, but you can get a non-naive utc datatime by doing ´

That is a pity :(

> datetime.now(timezone.utc).  (I must admit, however, that at least this morning I can't wrap my head around how that works based on the docs :(.

OK.. I am only using 2.7 so I can't try that :)

> 
> ----------
> nosy: +r.david.murray
> 
> _______________________________________
> Python tracker <report@bugs.python.org>
> <http://bugs.python.org/issue12750>
> _______________________________________
>
msg142131 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2011-08-15 15:36
Ah.  Well, pre-3.2 datetime itself did not generate *any* non-naive datetimes.

Nor do you need to specify everything for replace.  dt.replace(tzinfo=tz) should work just fine.
msg142150 - (view) Author: Daniel O'Connor (Daniel.O'Connor) Date: 2011-08-15 22:57
On 16/08/2011, at 1:06, R. David Murray wrote:
> R. David Murray <rdmurray@bitdance.com> added the comment:
> 
> Ah.  Well, pre-3.2 datetime itself did not generate *any* non-naive datetimes.
> 
> Nor do you need to specify everything for replace.  dt.replace(tzinfo=tz) should work just fine.

OK.

I did try this and it seems broken though..
In [19]: now = datetime.datetime.utcnow()

In [21]: now.replace(tzinfo = pytz.utc)
Out[21]: datetime.datetime(2011, 8, 15, 22, 54, 13, 173110, tzinfo=<UTC>)

In [22]: datetime.datetime.strftime(now, "%s")
Out[22]: '1313414653'

In [23]: now
Out[23]: datetime.datetime(2011, 8, 15, 22, 54, 13, 173110)

[ur 8:22] ~ >date -ujr 1313414653
Mon 15 Aug 2011 13:24:13 UTC

i.e. it appears that replace() applies the TZ offset to a naive datetime object effectively assuming it is local time rather than un-timezoned (which is what the docs imply to me)

> ----------
> resolution:  -> invalid
> stage:  -> committed/rejected
> status: open -> closed
> 
> _______________________________________
> Python tracker <report@bugs.python.org>
> <http://bugs.python.org/issue12750>
> _______________________________________
>
msg142190 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2011-08-16 12:30
OK. At a minimum there is a doc issue here, so I'm reopening.
msg142244 - (view) Author: Alexander Belopolsky (belopolsky) * (Python committer) Date: 2011-08-17 01:00
> i.e. it appears that replace() applies the TZ offset to a naive datetime
> object effectively assuming it is local time rather than un-timezoned
> (which is what the docs imply to me)

I don't understand your issue.  The replace method does not assume anything, it just replaces whatever fields you specify with new values.  You can replace tzinfo just like any other field, year, month, day, etc while preserving the other fields.  I think this is fairly well documented. I think what you are looking for is the astimezone() method which, however may not work well on naive datetime instances simply because a naive instance may be ambiguous in presence of DST.  However, if you start with an aware UTC datetime, you should be able to use astimezone() to convert to any local TZ.
msg142245 - (view) Author: Daniel O'Connor (Daniel.O'Connor) Date: 2011-08-17 01:55
On 17/08/2011, at 10:30, Alexander Belopolsky wrote:
> Alexander Belopolsky <alexander.belopolsky@gmail.com> added the comment:
> 
>> i.e. it appears that replace() applies the TZ offset to a naive datetime
>> object effectively assuming it is local time rather than un-timezoned
>> (which is what the docs imply to me)
> 
> I don't understand your issue.  The replace method does not assume anything, it just replaces whatever fields you specify with new values.  You can replace tzinfo just like any other field, year, month, day, etc while preserving the other fields.  I think this is fairly well documented. I think what you are looking for is the astimezone() method which, however may not work well on naive datetime instances simply because a naive instance may be ambiguous in presence of DST.  However, if you start with an aware UTC datetime, you should be able to use astimezone() to convert to any local TZ.

Hmm I see, it would appear the problem lies with strftime().

[ur 10:34] ~ >ipython-2.7 
Python 2.7.2 (default, Aug  6 2011, 23:46:16) 
Type "copyright", "credits" or "license" for more information.
IPython 0.10.2 -- An enhanced Interactive Python.
?         -> Introduction and overview of IPython's features.
%quickref -> Quick reference.
help      -> Python's own help system.
object?   -> Details about 'object'. ?object also works, ?? prints more.

In [48]: now = datetime.datetime.utcnow()
In [49]: nowtz = now.replace(tzinfo = pytz.utc)
In [50]: nowadl = now.replace(tzinfo = pytz.timezone('Australia/Adelaide'))
In [51]: now
Out[51]: datetime.datetime(2011, 8, 17, 1, 53, 51, 451118)
In [52]: nowtz
Out[52]: datetime.datetime(2011, 8, 17, 1, 53, 51, 451118, tzinfo=<UTC>)
In [53]: nowadl
Out[53]: datetime.datetime(2011, 8, 17, 1, 53, 51, 451118, tzinfo=<DstTzInfo 'Australia/Adelaide' CST+9:30:00 STD>)
In [54]: now.strftime("%F %r %s")
Out[54]: '2011-08-17 01:53:51 AM 1313511831'
In [55]: nowtz.strftime("%F %r %s")
Out[55]: '2011-08-17 01:53:51 AM 1313511831'
In [56]: nowadl.strftime("%F %r %s")
Out[56]: '2011-08-17 01:53:51 AM 1313511831'

Wed 17 Aug 2011 01:54:52 UTC
[ur 11:24] ~ >date +%s
1313546093
[ur 11:24] ~ >date -ujr `date +%s`
Wed 17 Aug 2011 01:54:59 UTC
[ur 11:24] ~ >date -ujr 1313511831
Tue 16 Aug 2011 16:23:51 UTC

i.e. strftime disregards tzinfo and seems to treat the time as LT (I think).

It certainly doesn't behave the way I'd expect after using strftime(3) et al :)
msg142249 - (view) Author: Alexander Belopolsky (belopolsky) * (Python committer) Date: 2011-08-17 03:12
> it would appear the problem lies with strftime()

Yes, strftime('%s') ignores tzinfo at the moment.  This is not a bug. Support for '%s' format code is incidental and not documented in Python.

Nevertheless I think this is a good feature request.  I am changing the title to make it more explicit.
msg142250 - (view) Author: Daniel O'Connor (Daniel.O'Connor) Date: 2011-08-17 03:16
On 17/08/2011, at 12:42, Alexander Belopolsky wrote:
> Alexander Belopolsky <alexander.belopolsky@gmail.com> added the comment:
> 
>> it would appear the problem lies with strftime()
> 
> Yes, strftime('%s') ignores tzinfo at the moment.  This is not a bug. Support for '%s' format code is incidental and not documented in Python.
> 
> Nevertheless I think this is a good feature request.  I am changing the title to make it more explicit.

OK thanks!
msg170808 - (view) Author: Mümin Öztürk (mumino) Date: 2012-09-20 12:23
I made a patch for datetime.strftime('%s'). it takes tzinfo into consideration.


>>> datetime.datetime(1970,1,1).strftime("%s")   
'-7200'

>>> datetime.datetime(1970,1,1, tzinfo=datetime.timezone.utc).strftime("%s")
'0'

datetime.date still behave as naive datetime.datetime
>>> datetime.date(1970,1,1).strftime("%s")
'-7200'
msg221353 - (view) Author: Alexander Belopolsky (belopolsky) * (Python committer) Date: 2014-06-23 13:08
I would like to hear from others on this feature.  One concern that I have is whether it is wise to truncate the fractional seconds part in '%s'.  Also, if we support '%s' in strftime we should probably support it in strptime as well.
msg221385 - (view) Author: (akira) * Date: 2014-06-24 00:09
*If* the support for %s strftime format code is added then it should
keep backward compatibility on Linux, OSX: it should produce an
integer string with the correct rounding.

Currently, datetime.strftime delegates to a platform strftime(3) for
format specifiers that are not described explicitly [1]:

> The full set of format codes supported varies across platforms,
> because Python calls the platform C library’s strftime() function,
> and platform variations are common. To see the full set of format
> codes supported on your platform, consult the strftime(3)
> documentation.

[1]: https://docs.python.org/3.4/library/datetime.html#strftime-strptime-behavior

%s is not defined in C, POSIX but is already defined on Linux, BSD [2]
where `datetime.now().strftime('%s')` can print an integer timestamp.

>  %s    is replaced by the number of seconds since the Epoch, UTC (see
>  mktime(3)).

[2]: http://www.openbsd.org/cgi-bin/man.cgi?query=strftime

Unsupported format code is *undefined behavior* (crash, launch a
missile is a valid behavior) otherwise.

Support for additional codes on some platforms is explicitly mentioned
in datetime docs therefore %s behavior shouldn't change if it is
well-defined on a given platform i.e., `datetime.now().strftime('%s')`
should keep producing an integer string on Linux, BSD.

- old code: `aware_dt.astimezone().strftime('%s')`
- proposed code: `aware_dt.strftime('%s')` (all platforms)

'%d' produces the wrong rounding on my machine:

  >>> from datetime import datetime, timezone
  >>> dt = datetime(1969, 1, 1, 0,0,0, 600000, tzinfo=timezone.utc)
  >>> '%d' % dt.timestamp()
  '-31535999'
  >>> dt.astimezone().strftime('%s')
  '-31536000'

`math.floor` could be used instead:

  >>> '%d' % math.floor(dt.timestamp())
  '-31536000'

There is no issue with the round-trip via a float timestamp for
datetime.min...datetime.max range on my machine. `calendar.timegm`
could be used to avoid floats if desired:

  >>> import calendar
  >>> calendar.timegm(dt.astimezone(timezone.utc).timetuple())
  -31536000

Note: dt.utctimetuple() is not used to avoid producing the wrong
result silently if dt is a naive datetime object; an exception is
raised instead.

The result is equivalent to `time.strftime('%s',
dt.astimezone().timetuple())` (+/- date/time range issues).

---

It is not clear what the returned value for %s strptime should be:
naive or timezone-aware datetime object and what timezone e.g.,

- old code: `datetime.fromtimestamp(int('-31536000'), timezone.utc)`
- proposed code: `datetime.strptime('-31536000', '%s')`

The result is an aware datetime object in UTC timezone.
msg221386 - (view) Author: Alexander Belopolsky (belopolsky) * (Python committer) Date: 2014-06-24 00:26
> It is not clear what the returned value for %s strptime should be:

I would start conservatively and require %z to be used with %s.  In this case, we can easily produce aware datetime objects.

I suspect that in the absence of %z, the most useful option would be to return naive datetime in the local timezone, but that can be added later.
msg221602 - (view) Author: (akira) * Date: 2014-06-26 13:15
> I suspect that in the absence of %z, the most useful option would be to return naive datetime in the local timezone, but that can be added later.

Naive datetime in the local timezone may lose information that is contained in the input timestamp:

  >>> import os
  >>> import time
  >>> from datetime import datetime
  >>> import pytz
  >>> os.environ['TZ'] = ':America/New_York'
  >>> time.tzset()
  >>> naive_dt = datetime(2014, 11, 2, 1, 30)
  >>> naive_dt.timestamp()
  1414906200.0
  >>> naive_dt.strftime('%s')
  '1414906200'
  >>> pytz.timezone('America/New_York').localize(naive_dt, is_dst=False).timestamp()
  1414909800.0
  >>> pytz.timezone('America/New_York').localize(naive_dt, is_dst=True).timestamp()
  1414906200.0
  >>> pytz.timezone('America/New_York').localize(naive_dt, is_dst=None)
  Traceback (most recent call last):
    File "<stdin>", line 1, in <module>
    File "~/.virtualenvs/py3.4/lib/python3.4/site-packages/pytz/tzinfo.py", line 349, in localize
      raise AmbiguousTimeError(dt)
  pytz.exceptions.AmbiguousTimeError: 2014-11-02 01:30:00

1414906200 timestamp corresponds to 2014-11-02 01:30:00-04:00
but datetime(2014, 11, 2, 1, 30) along is ambiguous -- 
it may correspond to both 1414906200 and 1414909800 if local timezone is America/New_York.

It would be nice if datetime.strptime() would allow the round-trip whatever the local timezone is:

  >>> ts = '1414906800'
  >>> datetime.strptime(ts, '%s').strftime('%s') == ts

it is possible if strptime() returns timezone-aware datetime object.
msg221606 - (view) Author: Mümin Öztürk (mumino) Date: 2014-06-26 14:56
I added an improved patch according to akira's explanation for strftime and rounding problem.
msg221620 - (view) Author: Alexander Belopolsky (belopolsky) * (Python committer) Date: 2014-06-26 17:25
On the second thought, I don't think accepting this should be contingent on any decision with respect to strptime.
msg221622 - (view) Author: Alexander Belopolsky (belopolsky) * (Python committer) Date: 2014-06-26 17:32
> rounding problem fixed with math.floor

Can you explain why math.floor rather than builtin round is the correct function to use?
msg221868 - (view) Author: (akira) * Date: 2014-06-29 17:58
> Can you explain why math.floor rather than builtin round is the correct function to use?

To avoid breaking existing scripts that use `.strftime('%s')` on Linux, OSX,
see msg221385:

  >>> from datetime import datetime, timezone
  >>> dt = datetime(1969, 1, 1, 0,0,0, 600000, tzinfo=timezone.utc)
  >>> '%d' % dt.timestamp()
  '-31535999'
  >>> round(dt.timestamp())
  -31535999
  >>> dt.astimezone().strftime('%s') # <-- existing behavior
  '-31536000'
  >>> '%d' % math.floor(dt.timestamp())
  '-31536000'
  >>> import calendar
  >>> calendar.timegm(dt.astimezone(timezone.utc).timetuple())
  -31536000
msg221872 - (view) Author: Alexander Belopolsky (belopolsky) * (Python committer) Date: 2014-06-29 18:09
Here is the simpler demonstration of the "floor" behavior on Linux:

>>> from datetime import datetime
>>> datetime.fromtimestamp(-0.1).strftime('%s')
'-1'
>>> datetime.fromtimestamp(-1.1).strftime('%s')
'-2'
>>> datetime.fromtimestamp(0.1).strftime('%s')
'0'
>>> datetime.fromtimestamp(1.1).strftime('%s')
'1'
msg221873 - (view) Author: Alexander Belopolsky (belopolsky) * (Python committer) Date: 2014-06-29 18:17
Could you, please add tests for non-fixed offset timezones?  There are several defined in datetimetester.py already.
msg221875 - (view) Author: Alexander Belopolsky (belopolsky) * (Python committer) Date: 2014-06-29 18:26
The patch should update documentation.

See https://docs.python.org/3.5/library/datetime.html#strftime-and-strptime-behavior
msg221877 - (view) Author: Alexander Belopolsky (belopolsky) * (Python committer) Date: 2014-06-29 18:30
+        t = datetime(1969, 1, 1, 0,0,0, 600000, tzinfo=timezone.utc)

Please add spaces after commas.
msg222030 - (view) Author: Mümin Öztürk (mumino) Date: 2014-07-01 13:45
more tests and some documentation added.
msg222047 - (view) Author: (akira) * Date: 2014-07-01 15:26
> ``%s`` format code behaviour was undefined and incidental.

strftime('%s') is not portable but it *is* supported on some
platforms i.e., it is *not* undefined and it is *not* incidental
on these platforms. datetime.strftime *delegates* to the platform
strftime(3) and some platforms do support %s format code. See the
quote from the datetime docs in msg221385.

It would be preferable that datetime.strftime would reject format
codes that it doesn't support explicitly (like datetime.strptime
does) so that datetime.strftime were portable but that ship
has sailed.

This issue could be titled: add cross-platform support for %s
strftime-format code (and fix its behavior (add support) for
timezone-aware datetime objects).

---

If the implementation uses floats to get an integer result; it should
have tests for edge cases (datetime.min, datetime.max at least). I
don't see such tests, please, correct me if I'm wrong.
History
Date User Action Args
2014-07-01 15:26:48akirasetmessages: + msg222047
2014-07-01 13:45:23muminosetfiles: + strftime3.patch

messages: + msg222030
2014-06-29 18:30:02belopolskysetmessages: + msg221877
2014-06-29 18:26:24belopolskysetmessages: + msg221875
2014-06-29 18:17:38belopolskysetmessages: + msg221873
versions: + Python 3.5, - Python 3.3
2014-06-29 18:09:55belopolskysetmessages: + msg221872
2014-06-29 17:58:05akirasetmessages: + msg221868
2014-06-26 17:32:47belopolskysetmessages: + msg221622
2014-06-26 17:25:22belopolskysetassignee: belopolsky
messages: + msg221620
stage: needs patch -> commit review
2014-06-26 14:56:11muminosetfiles: + strftime2.patch

messages: + msg221606
2014-06-26 13:15:35akirasetmessages: + msg221602
2014-06-24 00:26:57belopolskysetmessages: + msg221386
2014-06-24 00:09:07akirasetnosy: + akira
messages: + msg221385
2014-06-23 13:08:24belopolskysetnosy: + tim.peters, haypo
messages: + msg221353
2012-09-20 12:23:51muminosetfiles: + strftime.patch

nosy: + mumino
messages: + msg170808

keywords: + patch
2011-08-17 07:02:29santa4ntsetnosy: + santa4nt
2011-08-17 03:16:59Daniel.O'Connorsetmessages: + msg142250
2011-08-17 03:12:25belopolskysetversions: - Python 2.7, Python 3.2
title: datetime.datetime how to correctly attach a timezone to an existing naive datetime -> datetime.strftime('%s') should respect tzinfo
messages: + msg142249

components: + Extension Modules
keywords: + easy
stage: resolved -> needs patch
2011-08-17 01:55:54Daniel.O'Connorsetmessages: + msg142245
2011-08-17 01:00:55belopolskysetmessages: + msg142244
2011-08-16 12:30:50r.david.murraysetstatus: closed -> open


title: datetime.datetime timezone problems -> datetime.datetime how to correctly attach a timezone to an existing naive datetime
nosy: + belopolsky
versions: + Python 3.2, Python 3.3
messages: + msg142190
resolution: not a bug ->
2011-08-16 06:39:46bignosesetnosy: + bignose
2011-08-15 22:57:03Daniel.O'Connorsetmessages: + msg142150
2011-08-15 15:36:17r.david.murraysetstatus: open -> closed
resolution: not a bug
messages: + msg142131

stage: resolved
2011-08-15 14:23:44Daniel.O'Connorsetmessages: + msg142130
2011-08-15 14:09:45r.david.murraysetnosy: + r.david.murray
messages: + msg142129
2011-08-15 02:31:12Daniel.O'Connorcreate