classification
Title: datetime.datetime.strftime('%s') always uses local timezone, even with aware datetimes
Type: Stage: resolved
Components: Library (Lib) Versions: Python 3.6, Python 2.7
process
Status: closed Resolution: duplicate
Dependencies: Superseder:
Assigned To: Nosy List: adamwill, p-ganssle
Priority: normal Keywords:

Created on 2018-03-03 00:43 by adamwill, last changed 2018-03-03 01:53 by adamwill. This issue is now closed.

Messages (7)
msg313169 - (view) Author: Adam Williamson (adamwill) Date: 2018-03-03 00:43
Test script:

import pytz
import datetime
utc = pytz.timezone('UTC')
print(datetime.datetime(2017, 1, 1, tzinfo=utc).strftime('%s'))

Try running it with various system timezones:

[adamw@xps13k pagure (more-timezone-fun %)]$ TZ='UTC' python /tmp/test2.py
1483228800
[adamw@xps13k pagure (more-timezone-fun %)]$ TZ='America/Winnipeg' python /tmp/test2.py
1483250400
[adamw@xps13k pagure (more-timezone-fun %)]$ TZ='America/Vancouver' python /tmp/test2.py
1483257600

That's Python 2.7.14; same results with Python 3.6.4.

This does not seem correct. The correct Unix time for an aware datetime object should be a constant: for 2017-01-01 00:00 UTC it *is* 1483228800 . No matter what the system's local timezone, that should be the output of strftime('%s'), surely. What it seems to be doing instead is just outputting the Unix time for 2017-01-01 00:00 in the system timezone.

I *do* note that strftime('%s') is completely undocumented in Python; neither https://docs.python.org/2/library/datetime.html#strftime-and-strptime-behavior nor https://docs.python.org/3/library/datetime.html#strftime-and-strptime-behavior mentions it. However, it does exist, and is used in the real world; I found this usage of it, and the bug, in a real project, Pagure.
msg313171 - (view) Author: Paul Ganssle (p-ganssle) * (Python committer) Date: 2018-03-03 01:01
adamwill: I think datetime's strftime is a wrapper around the system strftime, so it varies between platforms. Might be useful to specify which platform you are seeing this behavior on.
msg313172 - (view) Author: Paul Ganssle (p-ganssle) * (Python committer) Date: 2018-03-03 01:05
It seems that %s is not supported and the fact that it works at all is incidental. See issue 12750 and this SO thread:

https://stackoverflow.com/questions/11743019/convert-python-datetime-to-epoch-with-strftime
msg313173 - (view) Author: Adam Williamson (adamwill) Date: 2018-03-03 01:14
Paul: right. This is on Linux - specifically Fedora Linux, but I don't think it matters. glibc strftime and strptime depend on an underlying struct called 'tm'. 'man strftime' says:

       %s     The number of seconds since the Epoch, 1970-01-01 00:00:00 +0000 (UTC). (TZ) (Calculated from mktime(tm).)

And 'man mktime' says:

The  mktime() function converts a broken-down time structure, expressed as local time, to calendar time representation. ... On success, mktime() returns the calendar time (seconds since the Epoch), expressed as a value of type time_t."

I am finding it hard to determine whether various C standards require the tm struct and mktime and strftime and so on to handle timezones, but I'm sort of inclining to the answer that "no they don't".

Basically I suspect what's going on in this case is that the timezone information gets lost somewhere in the chain down from Python to system strftime to system mktime, and Python doesn't make any adjustment to the actual date / time values before calling system strftime to try and account for this.

I think Python must do *something* more than purely converting to a tm and calling system strftime, though, as %Z does work, which it wouldn't if Python was purely converting to a non-timezone-aware tm struct and calling system strftime, I don't think...
msg313174 - (view) Author: Adam Williamson (adamwill) Date: 2018-03-03 01:18
I'd suggest that if that is the case, it would be better for the docs to *specifically mention* that `%s` is not supported and should not be used, rather than simply not mentioning it.

When it's used in real code (note someone in the SO issue mentions "I have been going crazy trying to figure out why i see strftime("%s") a lot, yet it's not in the docs") and just *not mentioned* in the docs, this tends to give the impression that it's something usable that was perhaps just forgotten from the docs, or something. The situation would be much clearer if the docs said "DO NOT USE THIS, IT'S DANGEROUS AND DOESN'T DO WHAT YOU THINK" in big letters. (And suggested using .timestamp() on Python 3.3+, and possibly arrow's .timestamp on 2.7?)
msg313175 - (view) Author: Paul Ganssle (p-ganssle) * (Python committer) Date: 2018-03-03 01:24
I suspect discussion should be centralized in issue 12750, but if it were up to me %s would either work as expected or throw an error. Silently giving the wrong answer is a terrible compromise.
msg313177 - (view) Author: Adam Williamson (adamwill) Date: 2018-03-03 01:53
Yeah, I've added a comment there. I agree we can keep subsequent discussion in that issue. Closing this as a dupe.

I actually have the same thought as you, but I suspect making something that "worked" before start throwing an error might be a hard sell for some. Perhaps at least some kind of warning?
History
Date User Action Args
2018-03-03 01:53:40adamwillsetstatus: open -> closed
resolution: duplicate
messages: + msg313177

stage: resolved
2018-03-03 01:24:34p-gansslesetmessages: + msg313175
2018-03-03 01:18:50adamwillsetmessages: + msg313174
2018-03-03 01:14:32adamwillsetmessages: + msg313173
2018-03-03 01:05:24p-gansslesetmessages: + msg313172
2018-03-03 01:01:28p-gansslesetnosy: + p-ganssle
messages: + msg313171
2018-03-03 00:43:22adamwillcreate