This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: Inconsistency between datetime.now() and datetime.fromtimestamp(time.time(), None)
Type: behavior Stage:
Components: Library (Lib), Windows Versions: Python 3.11
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: Miksus, belopolsky, larry, p-ganssle, paul.moore, rhettinger, steve.dower, steven.daprano, tim.golden, vstinner, zach.ware
Priority: normal Keywords:

Created on 2021-08-04 17:49 by Miksus, last changed 2022-04-11 14:59 by admin.

Messages (10)
msg398919 - (view) Author: Mikael Koli (Miksus) Date: 2021-08-04 17:49
I am trying to measure time twice and the second measurement gives a time that is 1 microsecond before the first measurement about half of the time.

My experiment in short:
---------------------------------------------------
import time, datetime
start = time.time()
end = datetime.datetime.now()

start = datetime.datetime.fromtimestamp(start, None)
assert end >= start # fails about half the time.
---------------------------------------------------


The problem is somewhat interesting. This does not fail:
---------------------------------------------------
import time, datetime
start = time.time()
end = time.time()

start = datetime.datetime.fromtimestamp(start, None)
end = datetime.datetime.fromtimestamp(end, None)
assert end >= start
---------------------------------------------------

And neither does this:
---------------------------------------------------
import datetime
start = datetime.datetime.now()
end = datetime.datetime.now()
assert end >= start
---------------------------------------------------

And it seems datetime.datetime.now() works the same way as to how I handled the "start" time in my first experiment: 
https://github.com/python/cpython/blob/3.6/Lib/datetime.py#L1514
and therefore the issue seems to be under the hood.

I have tested this on two Windows 10 machines (Python 3.6 & 3.8) in which cases this occurred. This did not happen on Raspberry Pi OS using Python 3.7.

In short:
- The time module imported in datetime.datetime.now() seems to measure time slightly differently than the time module imported by a Python user.
- This seems to be Windows specific.

My actual application has some code in between the measurements suffering from the same problem thus this is not an issue affecting only toy examples.
msg398922 - (view) Author: Mikael Koli (Miksus) Date: 2021-08-04 17:55
I accidentially posted Python 3.6 link to the declaration of datetime.datetime.now() but this has been unchanged: https://github.com/python/cpython/blob/3d2b4c6f18d7e644e5850d2af74ac5dc530eb24c/Lib/datetime.py#L1696

The actual piece of code as of now:
...
import time as _time
...
    @classmethod
    def now(cls, tz=None):
        "Construct a datetime from time.time() and optional time zone info."
        t = _time.time()
        return cls.fromtimestamp(t, tz)
msg399068 - (view) Author: Steven D'Aprano (steven.daprano) * (Python committer) Date: 2021-08-06 10:53
Confirmed on Windows with Python 3.9.6.
msg399070 - (view) Author: Steven D'Aprano (steven.daprano) * (Python committer) Date: 2021-08-06 11:07
I've replicated this under Linux as well.

def test():
    start = time.time()
    end = datetime.datetime.now()
    start = datetime.datetime.fromtimestamp(start, None)
    assert end >= start


Then run it in a loop:


>>> for i in range(10000000):
...     test()
... 
Traceback (most recent call last):
  File "<stdin>", line 2, in <module>
  File "<stdin>", line 5, in test
AssertionError
>>> i
22


So while it is not as frequent as on Windows, it does occur on Linux as well.
msg399090 - (view) Author: Alexander Belopolsky (belopolsky) * (Python committer) Date: 2021-08-06 13:56
Can someone try to replicate this while disabling the C acceleration:

import sys
sys.modules[‘_datetime’] = None

(Before any other imports.)

If anything, this is likely to be a problem with the C implementation.
msg399096 - (view) Author: Raymond Hettinger (rhettinger) * (Python committer) Date: 2021-08-06 15:27
The cause of the problem is inconsistent rounding modes:

   end = datetime.datetime.now()                          # always rounds down
   start = datetime.datetime.fromtimestamp(start, None)   # sometimes rounds_up

From the C code in Modules/_datetimemodule.c:

* The fromtimestamp() code uses ROUND_HALF_EVEN which can round-up.

* The datetime.now() code calls datetime_best_possible() which uses ROUND_FLOOR, always rounding down.

-------------------------------

static PyObject *
datetime_from_timestamp(PyObject *cls, TM_FUNC f, PyObject *timestamp,
                        PyObject *tzinfo)
{
    time_t timet;
    long us;

    if (_PyTime_ObjectToTimeval(timestamp,
                                &timet, &us, _PyTime_ROUND_HALF_EVEN) == -1)
        return NULL;

    return datetime_from_timet_and_us(cls, f, timet, (int)us, tzinfo);
}

--------------------------------

static PyObject *
datetime_datetime_now_impl(PyTypeObject *type, PyObject *tz)
/*[clinic end generated code: output=b3386e5345e2b47a input=80d09869c5267d00]*/
{
    PyObject *self;

    /* Return best possible local time -- this isn't constrained by the
     * precision of a timestamp.
     */
    if (check_tzinfo_subclass(tz) < 0)
        return NULL;

    self = datetime_best_possible((PyObject *)type,
                                  tz == Py_None ? _PyTime_localtime :
                                  _PyTime_gmtime,
                                  tz);
    if (self != NULL && tz != Py_None) {
        /* Convert UTC to tzinfo's zone. */
        self = _PyObject_CallMethodId(tz, &PyId_fromutc, "N", self);
    }
    return self;
}

------------------------

static PyObject *
datetime_best_possible(PyObject *cls, TM_FUNC f, PyObject *tzinfo)
{
    _PyTime_t ts = _PyTime_GetSystemClock();
    time_t secs;
    int us;

    if (_PyTime_AsTimevalTime_t(ts, &secs, &us, _PyTime_ROUND_FLOOR) < 0)
        return NULL;
    assert(0 <= us && us <= 999999);

    return datetime_from_timet_and_us(cls, f, secs, us, tzinfo);
}
msg399097 - (view) Author: Paul Ganssle (p-ganssle) * (Python committer) Date: 2021-08-06 15:28
I think this is a rounding issue. `time.time()` returns an epoch timestamp as a float and at the current epoch time, floats are spaced ~500ns apart.

`datetime.datetime.now` does a floor division when rounding: https://github.com/python/cpython/blob/8bdf12e99a3dc7ada5f85bba79c2a9eb9931f5b0/Modules/_datetimemodule.c#L5056

`datetime.fromtimestamp` uses the standard banker's round (round above half, tie goes to the nearest even number): https://github.com/python/cpython/blob/8bdf12e99a3dc7ada5f85bba79c2a9eb9931f5b0/Modules/_datetimemodule.c#L5038-L5039

Presumably if we change these two to be consistent, this issue will go away. I am not entirely sure if anyone is relying on a particular rounding behavior for one or both of these, and I'm not sure which one is the right one to harmonize on.

For now I'm going to say that we should target 3.11 on this, since it will change an existing observable behavior for at least one of these functions in a way that isn't necessarily going from "obviously wrong" to "obviously right", so I think we should be cautious and not change this in a patch release.
msg403063 - (view) Author: Raymond Hettinger (rhettinger) * (Python committer) Date: 2021-10-02 19:40
Related:  https://bugs.python.org/issue45347
msg403124 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2021-10-04 08:56
> Presumably if we change these two to be consistent, this issue will go away. I am not entirely sure if anyone is relying on a particular rounding behavior for one or both of these, and I'm not sure which one is the right one to harmonize on.

We already changed datetime rounding once in Python 3.4.4, see bpo-23517.

commit 511491ade0bb77febb176bc75f049797f0c71ed0
Author: Victor Stinner <victor.stinner@gmail.com>
Date:   Fri Sep 18 14:42:05 2015 +0200

    Issue #23517: Fix rounding in fromtimestamp() and utcfromtimestamp() methods
    of datetime.datetime: microseconds are now rounded to nearest with ties going
    to nearest even integer (ROUND_HALF_EVEN), instead of being rounding towards
    zero (ROUND_DOWN). It's important that these methods use the same rounding
    mode than datetime.timedelta to keep the property:
    
       (datetime(1970,1,1) + timedelta(seconds=t)) == datetime.utcfromtimestamp(t)
    
    It also the rounding mode used by round(float) for example.
    
    Add more unit tests on the rounding mode in test_datetime.

Since that time, I wrote a lot of time in Python/pytime.c to handle various rounding methods, and handle various time formats. See Include/cpython/pytime.h, I added documentation at the top recently ;-)

https://github.com/python/cpython/blob/main/Include/cpython/pytime.h

The _datetime module doesn't use _PyTime_t type but time_t to support the time_t full range (larger than _PyTime_t with 64-bit time_t).

I wrote an article about rounding timestamps:
https://vstinner.github.io/pytime.html

It seems like not all issues have been fixed yet :-)
msg403125 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2021-10-04 08:57
> Since that time, I wrote a lot of time

I wrote a lot of *code* :-)
History
Date User Action Args
2022-04-11 14:59:48adminsetgithub: 88994
2021-10-04 08:57:15vstinnersetmessages: + msg403125
2021-10-04 08:56:27vstinnersetmessages: + msg403124
2021-10-02 19:40:11rhettingersetmessages: + msg403063
2021-08-06 15:31:08rhettingersetnosy: + larry
2021-08-06 15:28:17p-gansslesetmessages: + msg399097
versions: + Python 3.11, - Python 3.6, Python 3.8, Python 3.9
2021-08-06 15:27:24rhettingersetnosy: + rhettinger
messages: + msg399096
2021-08-06 13:56:25belopolskysetmessages: + msg399090
2021-08-06 12:28:08serhiy.storchakasetnosy: + belopolsky, vstinner, p-ganssle
2021-08-06 11:07:47steven.dapranosetmessages: + msg399070
2021-08-06 10:53:58steven.dapranosetversions: + Python 3.9
nosy: + steven.daprano

messages: + msg399068

type: behavior
2021-08-04 17:55:51Miksussetmessages: + msg398922
2021-08-04 17:49:51Miksuscreate