classification
Title: fromutc does not respect datetime subclasses
Type: Stage: patch review
Components: Versions: Python 3.8
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: belopolsky, p-ganssle, vstinner
Priority: normal Keywords: patch

Created on 2017-12-23 18:15 by p-ganssle, last changed 2018-12-07 21:13 by gvanrossum.

Pull Requests
URL Status Linked Edit
PR 10902 open p-ganssle, 2018-12-04 18:30
Messages (9)
msg308961 - (view) Author: Paul Ganssle (p-ganssle) * Date: 2017-12-23 18:15
When preparing some tests for how subclasses of date and datetime react as part of a fix for issue 32403, I noticed a fairly big example of where subclass is not preserved - `tzinfo.fromutc`:

    from datetime import datetime, timezone

    class DateTimeSubclass(datetime):
        pass

    dt = DateTimeSubclass(2012, 1, 1)
    dt2 = dt.astimezone(timezone.utc)

    print(type(dt))
    print(type(dt2))

    # Result:
    # <class '__main__.DateTimeSubclass'>
    # <class 'datetime.datetime'>

This also affects `datetime.fromtimestamp` and `datetime.now`, since both of these, when passed a time zone argument, will call `fromutc` internally. I personally think that Python's `tzinfo.fromutc` should preserve the object's class, but this *is* counter to the current API.

And either way, it's quite inconsistent to have `DateTimeSubclass.now()` return `DateTimeSubclass` but have `DateTimeSubclass.now(timezone.utc)` return `datetime.datetime`.

There is probably a somewhat inelegant way to get the alternate constructors working properly (ignore the type of the argument up until the last return and then construct the subclass from the components of the datetime), but I think it might be better to fix the behavior of tzinfo.fromutc.

Somewhat related to issue 32404 and 31222, in that this concerns which operations preserve type in subclasses.
msg308962 - (view) Author: Paul Ganssle (p-ganssle) * Date: 2017-12-23 18:19
I've noticed that there's another complicating factor here, which is that addition of `timedelta` does not respect subclasses either, which means that third party libraries implementing fromutc (as was recommended in issue 28602), who will likely make use of timedelta addition, are going to have a harder time providing something that respects subclass.
msg331341 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2018-12-07 17:54
Since the C code is only a few lines, what do you think of also fixing Python 2.7?
msg331342 - (view) Author: Paul Ganssle (p-ganssle) * Date: 2018-12-07 18:06
I am somewhat uneasy about backporting this to Python 2.7 because changing the return type of `SomeDateTime + timedelta` could be seen as a breaking change. I have sent a message to the datetime-SIG mailing list about this for more visibility.

If it is decided that this is just a bugfix, I'm OK with creating a backport for 2.7 (provided that there's nothing so different about 2.7 that the fix becomes much bigger).
msg331343 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2018-12-07 18:12
> I am somewhat uneasy about backporting this to Python 2.7 because changing the return type of `SomeDateTime + timedelta` could be seen as a breaking change. I have sent a message to the datetime-SIG mailing list about this for more visibility.

You asked to backport up to Python 3.6. The change is either fine to be backported to 2.7 and 3.6, or should not be backported. I prefer to have the same policy for stable branches... but I also understand that 2.7 requires even more stability.
msg331347 - (view) Author: Paul Ganssle (p-ganssle) * Date: 2018-12-07 19:49
Ah, that's my mistake. I have always been under the impression that "Versions" meant "versions affected", not "versions that this needs to be fixed for". I usually just selected the ones where I had verified that it's a problem.

I do not think this should be backported to 3.6. From the discussion in the datetime-SIG mailing list, we have realized that this change will *also* break anyone whose default constructor does not support the same signature as the base datetime. I think this is probably not a major problem (many other alternate constructors assume that the constructor accepts arguments as datetime does), but it's not something that I think we should be changing in a patch version.
msg331349 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2018-12-07 20:07
What's the use case for subclassing DateTime? These classes were not designed with subclassing as a use case in mind.
msg331353 - (view) Author: Paul Ganssle (p-ganssle) * Date: 2018-12-07 20:47
> What's the use case for subclassing DateTime? These classes were not designed with subclassing as a use case in mind.

There are several reasons for doing it, of various levels of legitimacy. The overall theme is that people want different behaviors from their datetime classes and they want to maintain drop-in compatibility with datetime so that you don't need to re-build the whole world of datetime-handling libraries if you want to adopt one of these alternative datetime providers.

Ideally, you would tell people to just write API-compatible code and use duck-typing, but there's a lot of code in the standard library that uses `isinstance` checks, so things like `some_tzinfo.utcoffset(MyCoolDatetime.now())` will raise a TypeError.

Two popular datetime frameworks arrow and pendulum, both use datetime subclasses. A lot of what they are providing is convenience methods that could easily be free functions, but they also need to be subclasses so that they can change things like the semantics of arithmetic. For example, one motivation for the creation of pendulum was that the creator wanted this invariant to hold true:

    assert dt1 == (dt1 - dt2) + dt2

This is basically due to the fact that in Python's datetime library, no distinction is made between "absolute deltas" (the absolute time between two events) and "calendar deltas", which makes subtraction or addition across DST boundaries ambiguous and occasionally lossy. Arithmetic semantics are one of the things about datetime I'd most love to change but for backwards compatibility reasons it's just not feasible.

Another reason I've seen for subclassing datetime is that this is how dateutil provides its backport of PEP 495 (ambiguous datetime support). We have a datetime subclass called _DatetimeWithFold that supports the `fold` attribute, and is generated only when necessary (and does not exist in Python 3.6+). _DatetimeWithFold is not affected by this problem because PEP 495 specifies that the result of an arithmetic operation always sets fold to 0, but it *was* affected by the earlier (now fixed) bug where the subclass did not survive a `replace` operation.

One last place I've seen datetime subclasses used is when you have a thin wrapper used for dispatch or other purposes where you are mapping between types. For example, at work we had to create mappings between python types and the types specified by a standard (developed for another language), but that standard specified both a datetime type (with millisecond precision) and a datetimeus type (with microsecond precision). The solution was a thin wrapper around datetime called DatetimeUs: https://github.com/bloomberg/python-comdb2/blob/master/comdb2/_cdb2_types.py#L62

Preventing operations from reverting to datetime was a bit of a pain, which is why we have a bunch of tests to check that the subclass survives basic operations: https://github.com/bloomberg/python-comdb2/blob/master/tests/test_cdb2_datetimeus.py#L95

Although it was not originally *designed* to be subclassed, support for datetime subclasses is already quite good. This timedelta issue is one of the last major issues to fix to make them truly subclass-friendly. I'll note also that for the past 9 years, the test suite has run all datetime tests against a "thin wrapper" subclass of datetime: https://github.com/python/cpython/blame/028f0ef4f3111d2b3fc5b971642e337ba7990873/Lib/test/datetimetester.py#L2802
msg331355 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2018-12-07 21:13
OK.
History
Date User Action Args
2018-12-07 21:13:07gvanrossumsetnosy: - gvanrossum
2018-12-07 21:13:02gvanrossumsetmessages: + msg331355
2018-12-07 20:47:01p-gansslesetmessages: + msg331353
2018-12-07 20:07:45gvanrossumsetnosy: + gvanrossum
messages: + msg331349
2018-12-07 19:49:07p-gansslesetmessages: + msg331347
versions: - Python 3.6, Python 3.7
2018-12-07 18:12:43vstinnersetmessages: + msg331343
versions: - Python 3.5
2018-12-07 18:06:34p-gansslesetmessages: + msg331342
2018-12-07 17:54:51vstinnersetnosy: + vstinner
messages: + msg331341
2018-12-04 18:30:47p-gansslesetkeywords: + patch
stage: patch review
pull_requests: + pull_request10142
2017-12-23 18:19:08p-gansslesetmessages: + msg308962
2017-12-23 18:15:39p-gansslecreate