classification
Title: fromutc does not respect datetime subclasses
Type: Stage: resolved
Components: Versions: Python 3.8
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: belopolsky, eric.snow, jaraco, lukasz.langa, p-ganssle, vstinner
Priority: normal Keywords: patch

Created on 2017-12-23 18:15 by p-ganssle, last changed 2019-11-08 20:23 by jaraco. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 10902 merged p-ganssle, 2018-12-04 18:30
PR 11790 merged p-ganssle, 2019-02-08 15:24
PR 11790 merged p-ganssle, 2019-02-08 15:24
PR 11790 merged p-ganssle, 2019-02-08 15:24
PR 11790 merged p-ganssle, 2019-02-08 15:24
Messages (14)
msg308961 - (view) Author: Paul Ganssle (p-ganssle) * (Python committer) Date: 2017-12-23 18:15
When preparing some tests for how subclasses of date and datetime react as part of a fix for issue 32403, I noticed a fairly big example of where subclass is not preserved - `tzinfo.fromutc`:

    from datetime import datetime, timezone

    class DateTimeSubclass(datetime):
        pass

    dt = DateTimeSubclass(2012, 1, 1)
    dt2 = dt.astimezone(timezone.utc)

    print(type(dt))
    print(type(dt2))

    # Result:
    # <class '__main__.DateTimeSubclass'>
    # <class 'datetime.datetime'>

This also affects `datetime.fromtimestamp` and `datetime.now`, since both of these, when passed a time zone argument, will call `fromutc` internally. I personally think that Python's `tzinfo.fromutc` should preserve the object's class, but this *is* counter to the current API.

And either way, it's quite inconsistent to have `DateTimeSubclass.now()` return `DateTimeSubclass` but have `DateTimeSubclass.now(timezone.utc)` return `datetime.datetime`.

There is probably a somewhat inelegant way to get the alternate constructors working properly (ignore the type of the argument up until the last return and then construct the subclass from the components of the datetime), but I think it might be better to fix the behavior of tzinfo.fromutc.

Somewhat related to issue 32404 and 31222, in that this concerns which operations preserve type in subclasses.
msg308962 - (view) Author: Paul Ganssle (p-ganssle) * (Python committer) Date: 2017-12-23 18:19
I've noticed that there's another complicating factor here, which is that addition of `timedelta` does not respect subclasses either, which means that third party libraries implementing fromutc (as was recommended in issue 28602), who will likely make use of timedelta addition, are going to have a harder time providing something that respects subclass.
msg331341 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2018-12-07 17:54
Since the C code is only a few lines, what do you think of also fixing Python 2.7?
msg331342 - (view) Author: Paul Ganssle (p-ganssle) * (Python committer) Date: 2018-12-07 18:06
I am somewhat uneasy about backporting this to Python 2.7 because changing the return type of `SomeDateTime + timedelta` could be seen as a breaking change. I have sent a message to the datetime-SIG mailing list about this for more visibility.

If it is decided that this is just a bugfix, I'm OK with creating a backport for 2.7 (provided that there's nothing so different about 2.7 that the fix becomes much bigger).
msg331343 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2018-12-07 18:12
> I am somewhat uneasy about backporting this to Python 2.7 because changing the return type of `SomeDateTime + timedelta` could be seen as a breaking change. I have sent a message to the datetime-SIG mailing list about this for more visibility.

You asked to backport up to Python 3.6. The change is either fine to be backported to 2.7 and 3.6, or should not be backported. I prefer to have the same policy for stable branches... but I also understand that 2.7 requires even more stability.
msg331347 - (view) Author: Paul Ganssle (p-ganssle) * (Python committer) Date: 2018-12-07 19:49
Ah, that's my mistake. I have always been under the impression that "Versions" meant "versions affected", not "versions that this needs to be fixed for". I usually just selected the ones where I had verified that it's a problem.

I do not think this should be backported to 3.6. From the discussion in the datetime-SIG mailing list, we have realized that this change will *also* break anyone whose default constructor does not support the same signature as the base datetime. I think this is probably not a major problem (many other alternate constructors assume that the constructor accepts arguments as datetime does), but it's not something that I think we should be changing in a patch version.
msg331349 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2018-12-07 20:07
What's the use case for subclassing DateTime? These classes were not designed with subclassing as a use case in mind.
msg331353 - (view) Author: Paul Ganssle (p-ganssle) * (Python committer) Date: 2018-12-07 20:47
> What's the use case for subclassing DateTime? These classes were not designed with subclassing as a use case in mind.

There are several reasons for doing it, of various levels of legitimacy. The overall theme is that people want different behaviors from their datetime classes and they want to maintain drop-in compatibility with datetime so that you don't need to re-build the whole world of datetime-handling libraries if you want to adopt one of these alternative datetime providers.

Ideally, you would tell people to just write API-compatible code and use duck-typing, but there's a lot of code in the standard library that uses `isinstance` checks, so things like `some_tzinfo.utcoffset(MyCoolDatetime.now())` will raise a TypeError.

Two popular datetime frameworks arrow and pendulum, both use datetime subclasses. A lot of what they are providing is convenience methods that could easily be free functions, but they also need to be subclasses so that they can change things like the semantics of arithmetic. For example, one motivation for the creation of pendulum was that the creator wanted this invariant to hold true:

    assert dt1 == (dt1 - dt2) + dt2

This is basically due to the fact that in Python's datetime library, no distinction is made between "absolute deltas" (the absolute time between two events) and "calendar deltas", which makes subtraction or addition across DST boundaries ambiguous and occasionally lossy. Arithmetic semantics are one of the things about datetime I'd most love to change but for backwards compatibility reasons it's just not feasible.

Another reason I've seen for subclassing datetime is that this is how dateutil provides its backport of PEP 495 (ambiguous datetime support). We have a datetime subclass called _DatetimeWithFold that supports the `fold` attribute, and is generated only when necessary (and does not exist in Python 3.6+). _DatetimeWithFold is not affected by this problem because PEP 495 specifies that the result of an arithmetic operation always sets fold to 0, but it *was* affected by the earlier (now fixed) bug where the subclass did not survive a `replace` operation.

One last place I've seen datetime subclasses used is when you have a thin wrapper used for dispatch or other purposes where you are mapping between types. For example, at work we had to create mappings between python types and the types specified by a standard (developed for another language), but that standard specified both a datetime type (with millisecond precision) and a datetimeus type (with microsecond precision). The solution was a thin wrapper around datetime called DatetimeUs: https://github.com/bloomberg/python-comdb2/blob/master/comdb2/_cdb2_types.py#L62

Preventing operations from reverting to datetime was a bit of a pain, which is why we have a bunch of tests to check that the subclass survives basic operations: https://github.com/bloomberg/python-comdb2/blob/master/tests/test_cdb2_datetimeus.py#L95

Although it was not originally *designed* to be subclassed, support for datetime subclasses is already quite good. This timedelta issue is one of the last major issues to fix to make them truly subclass-friendly. I'll note also that for the past 9 years, the test suite has run all datetime tests against a "thin wrapper" subclass of datetime: https://github.com/python/cpython/blame/028f0ef4f3111d2b3fc5b971642e337ba7990873/Lib/test/datetimetester.py#L2802
msg331355 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2018-12-07 21:13
OK.
msg334837 - (view) Author: Alexander Belopolsky (belopolsky) * (Python committer) Date: 2019-02-04 19:42
New changeset 89427cd0feae25bbc8693abdccfa6a8c81a2689c by Alexander Belopolsky (Paul Ganssle) in branch 'master':
bpo-32417: Make timedelta arithmetic respect subclasses (#10902)
https://github.com/python/cpython/commit/89427cd0feae25bbc8693abdccfa6a8c81a2689c
msg335088 - (view) Author: Paul Ganssle (p-ganssle) * (Python committer) Date: 2019-02-08 15:38
Hm, when I made the "What's new" issue, it added the same PR to the "Pull requests" 4 times instead of once, and in the history it seems like it *tried* to actually add PR 11790, 11791, 11792 and 11793 (only the first one exists at the moment). Not sure why that happened and where I'd report that bug.
msg335089 - (view) Author: Paul Ganssle (p-ganssle) * (Python committer) Date: 2019-02-08 15:42
Ah, sorry for the noise, this is a known issue: https://github.com/python/bugs.python.org/issues/12
msg335091 - (view) Author: Łukasz Langa (lukasz.langa) * (Python committer) Date: 2019-02-08 16:02
New changeset d9503c307a8b6a7b73f6344183602ffb014d3356 by Łukasz Langa (Paul Ganssle) in branch 'master':
Add What's New entry for date subclass behavior (#11790)
https://github.com/python/cpython/commit/d9503c307a8b6a7b73f6344183602ffb014d3356
msg356259 - (view) Author: Jason R. Coombs (jaraco) * (Python committer) Date: 2019-11-08 20:23
This issue broke a date subclass in the calendra project (https://github.com/jaraco/calendra/issues/11). I acknowledge this change was a known breakage, but I mention it here and link the downstream issue for your information.
History
Date User Action Args
2019-11-08 20:23:01jaracosetnosy: + jaraco
messages: + msg356259
2019-02-14 16:05:46p-gansslesetstatus: open -> closed
resolution: fixed
stage: patch review -> resolved
2019-02-08 16:02:10lukasz.langasetnosy: + lukasz.langa
messages: + msg335091
2019-02-08 15:42:44p-gansslesetmessages: + msg335089
2019-02-08 15:38:46p-gansslesetmessages: + msg335088
2019-02-08 15:25:02p-gansslesetpull_requests: + pull_request11793
2019-02-08 15:24:53p-gansslesetpull_requests: + pull_request11792
2019-02-08 15:24:43p-gansslesetpull_requests: + pull_request11791
2019-02-08 15:24:33p-gansslesetpull_requests: + pull_request11790
2019-02-05 15:45:39eric.snowsetnosy: + eric.snow
2019-02-04 19:42:09belopolskysetmessages: + msg334837
2018-12-07 21:13:07gvanrossumsetnosy: - gvanrossum
2018-12-07 21:13:02gvanrossumsetmessages: + msg331355
2018-12-07 20:47:01p-gansslesetmessages: + msg331353
2018-12-07 20:07:45gvanrossumsetnosy: + gvanrossum
messages: + msg331349
2018-12-07 19:49:07p-gansslesetmessages: + msg331347
versions: - Python 3.6, Python 3.7
2018-12-07 18:12:43vstinnersetmessages: + msg331343
versions: - Python 3.5
2018-12-07 18:06:34p-gansslesetmessages: + msg331342
2018-12-07 17:54:51vstinnersetnosy: + vstinner
messages: + msg331341
2018-12-04 18:30:47p-gansslesetkeywords: + patch
stage: patch review
pull_requests: + pull_request10142
2017-12-23 18:19:08p-gansslesetmessages: + msg308962
2017-12-23 18:15:39p-gansslecreate