Issue44831
This issue tracker has been migrated to GitHub,
and is currently read-only.
For more information,
see the GitHub FAQs in the Python's Developer Guide.
Created on 2021-08-04 17:49 by Miksus, last changed 2022-04-11 14:59 by admin.
Messages (10) | |||
---|---|---|---|
msg398919 - (view) | Author: Mikael Koli (Miksus) | Date: 2021-08-04 17:49 | |
I am trying to measure time twice and the second measurement gives a time that is 1 microsecond before the first measurement about half of the time. My experiment in short: --------------------------------------------------- import time, datetime start = time.time() end = datetime.datetime.now() start = datetime.datetime.fromtimestamp(start, None) assert end >= start # fails about half the time. --------------------------------------------------- The problem is somewhat interesting. This does not fail: --------------------------------------------------- import time, datetime start = time.time() end = time.time() start = datetime.datetime.fromtimestamp(start, None) end = datetime.datetime.fromtimestamp(end, None) assert end >= start --------------------------------------------------- And neither does this: --------------------------------------------------- import datetime start = datetime.datetime.now() end = datetime.datetime.now() assert end >= start --------------------------------------------------- And it seems datetime.datetime.now() works the same way as to how I handled the "start" time in my first experiment: https://github.com/python/cpython/blob/3.6/Lib/datetime.py#L1514 and therefore the issue seems to be under the hood. I have tested this on two Windows 10 machines (Python 3.6 & 3.8) in which cases this occurred. This did not happen on Raspberry Pi OS using Python 3.7. In short: - The time module imported in datetime.datetime.now() seems to measure time slightly differently than the time module imported by a Python user. - This seems to be Windows specific. My actual application has some code in between the measurements suffering from the same problem thus this is not an issue affecting only toy examples. |
|||
msg398922 - (view) | Author: Mikael Koli (Miksus) | Date: 2021-08-04 17:55 | |
I accidentially posted Python 3.6 link to the declaration of datetime.datetime.now() but this has been unchanged: https://github.com/python/cpython/blob/3d2b4c6f18d7e644e5850d2af74ac5dc530eb24c/Lib/datetime.py#L1696 The actual piece of code as of now: ... import time as _time ... @classmethod def now(cls, tz=None): "Construct a datetime from time.time() and optional time zone info." t = _time.time() return cls.fromtimestamp(t, tz) |
|||
msg399068 - (view) | Author: Steven D'Aprano (steven.daprano) * | Date: 2021-08-06 10:53 | |
Confirmed on Windows with Python 3.9.6. |
|||
msg399070 - (view) | Author: Steven D'Aprano (steven.daprano) * | Date: 2021-08-06 11:07 | |
I've replicated this under Linux as well. def test(): start = time.time() end = datetime.datetime.now() start = datetime.datetime.fromtimestamp(start, None) assert end >= start Then run it in a loop: >>> for i in range(10000000): ... test() ... Traceback (most recent call last): File "<stdin>", line 2, in <module> File "<stdin>", line 5, in test AssertionError >>> i 22 So while it is not as frequent as on Windows, it does occur on Linux as well. |
|||
msg399090 - (view) | Author: Alexander Belopolsky (belopolsky) * | Date: 2021-08-06 13:56 | |
Can someone try to replicate this while disabling the C acceleration: import sys sys.modules[‘_datetime’] = None (Before any other imports.) If anything, this is likely to be a problem with the C implementation. |
|||
msg399096 - (view) | Author: Raymond Hettinger (rhettinger) * | Date: 2021-08-06 15:27 | |
The cause of the problem is inconsistent rounding modes: end = datetime.datetime.now() # always rounds down start = datetime.datetime.fromtimestamp(start, None) # sometimes rounds_up From the C code in Modules/_datetimemodule.c: * The fromtimestamp() code uses ROUND_HALF_EVEN which can round-up. * The datetime.now() code calls datetime_best_possible() which uses ROUND_FLOOR, always rounding down. ------------------------------- static PyObject * datetime_from_timestamp(PyObject *cls, TM_FUNC f, PyObject *timestamp, PyObject *tzinfo) { time_t timet; long us; if (_PyTime_ObjectToTimeval(timestamp, &timet, &us, _PyTime_ROUND_HALF_EVEN) == -1) return NULL; return datetime_from_timet_and_us(cls, f, timet, (int)us, tzinfo); } -------------------------------- static PyObject * datetime_datetime_now_impl(PyTypeObject *type, PyObject *tz) /*[clinic end generated code: output=b3386e5345e2b47a input=80d09869c5267d00]*/ { PyObject *self; /* Return best possible local time -- this isn't constrained by the * precision of a timestamp. */ if (check_tzinfo_subclass(tz) < 0) return NULL; self = datetime_best_possible((PyObject *)type, tz == Py_None ? _PyTime_localtime : _PyTime_gmtime, tz); if (self != NULL && tz != Py_None) { /* Convert UTC to tzinfo's zone. */ self = _PyObject_CallMethodId(tz, &PyId_fromutc, "N", self); } return self; } ------------------------ static PyObject * datetime_best_possible(PyObject *cls, TM_FUNC f, PyObject *tzinfo) { _PyTime_t ts = _PyTime_GetSystemClock(); time_t secs; int us; if (_PyTime_AsTimevalTime_t(ts, &secs, &us, _PyTime_ROUND_FLOOR) < 0) return NULL; assert(0 <= us && us <= 999999); return datetime_from_timet_and_us(cls, f, secs, us, tzinfo); } |
|||
msg399097 - (view) | Author: Paul Ganssle (p-ganssle) * | Date: 2021-08-06 15:28 | |
I think this is a rounding issue. `time.time()` returns an epoch timestamp as a float and at the current epoch time, floats are spaced ~500ns apart. `datetime.datetime.now` does a floor division when rounding: https://github.com/python/cpython/blob/8bdf12e99a3dc7ada5f85bba79c2a9eb9931f5b0/Modules/_datetimemodule.c#L5056 `datetime.fromtimestamp` uses the standard banker's round (round above half, tie goes to the nearest even number): https://github.com/python/cpython/blob/8bdf12e99a3dc7ada5f85bba79c2a9eb9931f5b0/Modules/_datetimemodule.c#L5038-L5039 Presumably if we change these two to be consistent, this issue will go away. I am not entirely sure if anyone is relying on a particular rounding behavior for one or both of these, and I'm not sure which one is the right one to harmonize on. For now I'm going to say that we should target 3.11 on this, since it will change an existing observable behavior for at least one of these functions in a way that isn't necessarily going from "obviously wrong" to "obviously right", so I think we should be cautious and not change this in a patch release. |
|||
msg403063 - (view) | Author: Raymond Hettinger (rhettinger) * | Date: 2021-10-02 19:40 | |
Related: https://bugs.python.org/issue45347 |
|||
msg403124 - (view) | Author: STINNER Victor (vstinner) * | Date: 2021-10-04 08:56 | |
> Presumably if we change these two to be consistent, this issue will go away. I am not entirely sure if anyone is relying on a particular rounding behavior for one or both of these, and I'm not sure which one is the right one to harmonize on. We already changed datetime rounding once in Python 3.4.4, see bpo-23517. commit 511491ade0bb77febb176bc75f049797f0c71ed0 Author: Victor Stinner <victor.stinner@gmail.com> Date: Fri Sep 18 14:42:05 2015 +0200 Issue #23517: Fix rounding in fromtimestamp() and utcfromtimestamp() methods of datetime.datetime: microseconds are now rounded to nearest with ties going to nearest even integer (ROUND_HALF_EVEN), instead of being rounding towards zero (ROUND_DOWN). It's important that these methods use the same rounding mode than datetime.timedelta to keep the property: (datetime(1970,1,1) + timedelta(seconds=t)) == datetime.utcfromtimestamp(t) It also the rounding mode used by round(float) for example. Add more unit tests on the rounding mode in test_datetime. Since that time, I wrote a lot of time in Python/pytime.c to handle various rounding methods, and handle various time formats. See Include/cpython/pytime.h, I added documentation at the top recently ;-) https://github.com/python/cpython/blob/main/Include/cpython/pytime.h The _datetime module doesn't use _PyTime_t type but time_t to support the time_t full range (larger than _PyTime_t with 64-bit time_t). I wrote an article about rounding timestamps: https://vstinner.github.io/pytime.html It seems like not all issues have been fixed yet :-) |
|||
msg403125 - (view) | Author: STINNER Victor (vstinner) * | Date: 2021-10-04 08:57 | |
> Since that time, I wrote a lot of time I wrote a lot of *code* :-) |
History | |||
---|---|---|---|
Date | User | Action | Args |
2022-04-11 14:59:48 | admin | set | github: 88994 |
2021-10-04 08:57:15 | vstinner | set | messages: + msg403125 |
2021-10-04 08:56:27 | vstinner | set | messages: + msg403124 |
2021-10-02 19:40:11 | rhettinger | set | messages: + msg403063 |
2021-08-06 15:31:08 | rhettinger | set | nosy:
+ larry |
2021-08-06 15:28:17 | p-ganssle | set | messages:
+ msg399097 versions: + Python 3.11, - Python 3.6, Python 3.8, Python 3.9 |
2021-08-06 15:27:24 | rhettinger | set | nosy:
+ rhettinger messages: + msg399096 |
2021-08-06 13:56:25 | belopolsky | set | messages: + msg399090 |
2021-08-06 12:28:08 | serhiy.storchaka | set | nosy:
+ belopolsky, vstinner, p-ganssle |
2021-08-06 11:07:47 | steven.daprano | set | messages: + msg399070 |
2021-08-06 10:53:58 | steven.daprano | set | versions:
+ Python 3.9 nosy: + steven.daprano messages: + msg399068 type: behavior |
2021-08-04 17:55:51 | Miksus | set | messages: + msg398922 |
2021-08-04 17:49:51 | Miksus | create |