classification
Title: datetime.utcfromtimestamp rounds results incorrectly
Type: behavior Stage: commit review
Components: Library (Lib) Versions: Python 3.6, Python 3.4, Python 3.5
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: belopolsky Nosy List: BreamoreBoy, aconrad, belopolsky, larry, mark.dickinson, python-dev, r.david.murray, serhiy.storchaka, tbarbugli, tim.peters, trcarden, vivanov, vstinner
Priority: normal Keywords: 3.3regression, patch

Created on 2015-02-24 22:38 by tbarbugli, last changed 2015-09-18 13:11 by vstinner. This issue is now closed.

Files
File name Uploaded Description Edit
round_half_even_py34.patch vstinner, 2015-09-09 00:02 review
Messages (100)
msg236552 - (view) Author: Tommaso Barbugli (tbarbugli) Date: 2015-02-24 22:38
Hi,

I am porting a library from python 2.7 to 3.4 and I noticed that the behaviour of datetime.utcfromtimestamp is not consistent between the two versions.

For example on python 2.7.5
datetime.utcfromtimestamp(1424817268.274)
returns a datetime with 274000 microseconds

the same code in python 3.4 returns a datetime with 273999 microseconds.
msg236553 - (view) Author: Ethan Furman (ethan.furman) * (Python committer) Date: 2015-02-24 22:43
This seems to have changed in 3.3 (versions up to 3.2 return 274000).
msg236577 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2015-02-25 14:11
Most likely this was a rounding fix (ie: not a bug), but hopefully Alexander will know for sure.
msg236580 - (view) Author: Alexander Belopolsky (belopolsky) * (Python committer) Date: 2015-02-25 15:00
Let me dig up the history, but this does not look like correct rounding to me:

>>> datetime.utcfromtimestamp(1424817268.274)
datetime.datetime(2015, 2, 24, 22, 34, 28, 273999)
>>> decimal.Decimal(1424817268.274)
Decimal('1424817268.2739999294281005859375')
msg236581 - (view) Author: Alexander Belopolsky (belopolsky) * (Python committer) Date: 2015-02-25 15:13
It looks like it was an intentional change.  See #14180 (changeset 75590:1e9cc1a03365).

I am not sure what the motivation was. Note that this change made utcfromtimestamp(t) different from datetime(1970,1,1) + timedelta(seconds=t).
msg236585 - (view) Author: Alexander Belopolsky (belopolsky) * (Python committer) Date: 2015-02-25 15:33
Victor's motivation for the change was (msg154811):

"""
I chose this rounding method because it is the method used by int(float) and int(time.time()) is a common in programs (more than round(time.time()). Rounding towards zero avoids also producing timestamps in the future.
"""

I recall the earlier discussions of rounding in the datetime module and Mark's explanation that rounding up is fine as long as ordering is preserved. i.e. for x < y round(x) <= round(y).

There are cases when producing times in the future are problematic, for example UNIX make really dislikes when file timestamps are in the future, but if this was the main motivation - rounding towards -infinity would be more appropriate.

In any case, as long as we have the following in the datetime module documentation, I think this behavior is a bug:

"""
On the POSIX compliant platforms, utcfromtimestamp(timestamp) is equivalent to the following expression:

datetime(1970, 1, 1) + timedelta(seconds=timestamp)
"""

>>> timestamp = 1424817268.274
>>> datetime.utcfromtimestamp(timestamp) == datetime(1970, 1, 1) + timedelta(seconds=timestamp)
False
msg236587 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2015-02-25 16:06
I started a large change set to support nanoseconds in the C "pytime" API: see the issue #22117. While working on this change, I noticed that the rounding mode of datetime is currently wrong. Extract of a private patch:

typedef enum {
    /* Round towards zero. */
    _PyTime_ROUND_DOWN=0,
    /* Round away from zero.
       For example, used for timeout to wait "at least" N seconds. */
    _PyTime_ROUND_UP=1,
    /* Round towards minus infinity (-inf).
       For example, used for the system clock with UNIX epoch (time_t). */
    _PyTime_ROUND_FLOOR=2
} _PyTime_round_t;

I changed Modules/_datetimemodule.c to use _PyTime_ROUND_FLOOR, instead of _PyTime_ROUND_DOWN.
msg236589 - (view) Author: Alexander Belopolsky (belopolsky) * (Python committer) Date: 2015-02-25 16:16
> I noticed that the rounding mode of datetime is currently wrong.

What do you mean by "currently"?  What versions of python have it wrong?
msg236597 - (view) Author: Alexander Belopolsky (belopolsky) * (Python committer) Date: 2015-02-25 16:48
Victor,

Would you consider going back to round to nearest?  Mark and I put in a lot of effort to get the rounding in the datetime module right.  (See for example, #8860.)

Sub-microsecond timesources are still rare and users who work with such should avoid FP timestamps in any case.  On the other hand, double precision timestamps are adequate for microsecond resolution now and in the next few decades.

Timestamps like OP's (sec=1424817268, us=274000) should not change when converted to double and back.  IMO, the following behavior is a bug.

>>> dt = datetime(2015, 2, 24, 22, 34, 28, 274000)
>>> datetime.utcfromtimestamp(dt.timestamp())
datetime.datetime(2015, 2, 25, 3, 34, 28, 273999)
msg236607 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2015-02-25 17:31
> Would you consider going back to round to nearest?

I don't understand "nearest". I prefer to use names of decimal rounding modes:
https://docs.python.org/dev/library/decimal.html#rounding-modes

In my local patch, I'm using ROUND_FLOOR in _decimal: "Round towards -Infinity."

> Mark and I put in a lot of effort to get the rounding in the datetime module right.  (See for example, #8860.)

I'm unable right now to say which rounding mode should be used in the decimal module. But it's important to use the same rounding mode for all similar operations. For example, time.time() and datetime.datetime.now() should have the same rounding method (bad example, time.time() returns a float, which doesn't round the result).

For example, in my local patch, I'm using ROUND_FLOOR for:

- datetime.date.fromtimestamp()
- datetime.datetime.fromtimestamp()
- datetime.datetime.now()
- datetime.datetime.utcnow()
- os.utime()
- time.clock_settime()
- time.gmtime()
- time.localtime()
- time.ctime()

Note: the Python implementation of datetime uses time.localtime() and time.gmtime() for fromtimestamp(), so these functions should also have the same rounding method.
msg236608 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2015-02-25 17:33
> What do you mean by "currently"?  What versions of python have it wrong?

I search for "ROUND" in Modules/_datetimemodule.c: in the Python development branch (default), I found _PyTime_ROUND_DOWN (Round towards zero). Since a bug was reported, I understand that it's not the good rounding method?
msg236609 - (view) Author: Alexander Belopolsky (belopolsky) * (Python committer) Date: 2015-02-25 17:38
> I don't understand "nearest".

Sorry for using loose terms.  I was hoping the in the context of "going back", it would be clear.

I believe the correct mode is "ROUND_HALF_EVEN".  This is the mode used by the builtin round() function: 

>>> round(0.5)
0
>>> round(1.5)
2
msg236610 - (view) Author: Alexander Belopolsky (belopolsky) * (Python committer) Date: 2015-02-25 17:57
> For example, in my local patch, I'm using ROUND_FLOOR for:

> - datetime.date.fromtimestamp()
> - datetime.datetime.fromtimestamp()

These should use ROUND_HALF_EVEN

> - datetime.datetime.now()
> - datetime.datetime.utcnow()

These should not involve floating point arithmetics, but when converting from nanoseconds to microseconds, you should round to nearest 1000 ns with 500 ns ties resolved to even number of microseconds.


> - os.utime()

This takes nanoseconds as an optional argument.  Passing floats in times should probably be deprecated. In any case, here you would be rounding floats to nanoseconds and what you do with 0.5 nanoseconds is less important because in most cases they are not even representable as floats.

> - time.clock_settime()

Is this a new method?  I don't see it in 3.5.0a1.

> - time.gmtime()

This should be fixed

>>> time.gmtime(1.999999999).tm_sec
1

is really bad and

>>> time.gmtime(-1.999999999)[:6]
(1969, 12, 31, 23, 59, 59)

is probably even worse. 


> - time.localtime()
> - time.ctime()

Same story as in time.gmtime.
msg246049 - (view) Author: Timothy Cardenas (trcarden) Date: 2015-07-01 23:31
We are seeing this behavior influencing other libraries in python 3.4.

This should never fail if timestamp and fromtimestamp are implemented correctly:

from datetime import datetime
t = datetime.utcnow().timestamp()
t2 = datetime.utcfromtimestamp(t)
assert t == t2, 'Moving from timestamp and back should always work'
msg246073 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2015-07-02 12:23
Because this seems to be a regression, I'm marking this as a release blocker.  The RM can decide is isn't, of course.
msg246097 - (view) Author: Larry Hastings (larry) * (Python committer) Date: 2015-07-02 21:04
Yes, by all means, fix for 3.4, 3.5, and 3.6.  If possible I'd appreciate you getting the fix checked in to 3.5 within the next 48 hours, as I'm tagging the next beta release of 3.5 around then, and it'd be nice if this fix went out in that release.
msg246104 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2015-07-02 22:55
I'm concerned by this example:

>>> dt = datetime(2015, 2, 24, 22, 34, 28, 274000)
>>> dt - datetime.fromtimestamp(dt.timestamp())
datetime.timedelta(0, 0, 1)

I don't know yet if it should be fixed or not.

If we modify .fromtimestamp(), should we use the same rounding method in datetime constructor? And in datetime.now()/.utcnow()?

I would prefer to keep ROUND_DOWN for .now() and .utcnow() to avoid timestamps in the future. I care less for other methods.

What do you think of this plan?

---

Hum, I don't remember the whole story line of rounding timestamps in Python. Some raw data.

Include/pytime.h of Python 3.5+ has:

typedef enum {
    /* Round towards minus infinity (-inf).
       For example, used to read a clock. */
    _PyTime_ROUND_FLOOR=0,
    /* Round towards infinity (+inf).
       For example, used for timeout to wait "at least" N seconds. */
    _PyTime_ROUND_CEILING
} _PyTime_round_t;

Include/pytime.h of Python 3.4 had:

typedef enum {
    /* Round towards zero. */
    _PyTime_ROUND_DOWN=0,
    /* Round away from zero. */
    _PyTime_ROUND_UP
} _PyTime_round_t;

Include/pytime.h of Python 3.3 and older didn't have rounding.

C files using pytime.h rounding in Python 3.4 (grep -l _PyTime_ROUND */*.c):

Modules/_datetimemodule.c
Modules/posixmodule.c
Modules/selectmodule.c
Modules/signalmodule.c
Modules/_testcapimodule.c
Modules/timemodule.c
Python/pytime.c

It is used by 3 mores C files in Python 3.5:

Modules/socketmodule.c
Modules/_ssl.c
Modules/_threadmodule.c

NEAREST was never implemented in pytime.h.

If I recall correctly, there were inconsitencies between the Python and the C implementation of the datetime module. At least in Python 3.5, both implementations should be consistent (even if some people would prefer a different rounding method).

The private pytime API was rewritten in Python 3.5 to get nanosecond resolution. This API is only used by the datetime module to get the current time.

My rationale for ROUND_DOWN was to follow how UNIX rounds timestmaps. As Alexander wrote, UNIX doesn't like timestamps in the future, so rounding towards minus infinity avoids such issue. Rounding issues become more common on file timestamps with filesystems supporting microsecond resolution or event nanosecond resolution.
msg246121 - (view) Author: Alexander Belopolsky (belopolsky) * (Python committer) Date: 2015-07-03 02:07
Victor> I don't know yet if it should be fixed or not.

It is my understanding that datetime -> timestamp -> datetime round-tripping was exact in 3.3 for datetimes not too far in the future (as of 2015), but now it breaks for datetime(2015, 2, 24, 22, 34, 28, 274000).
This is clearly a regression and should be fixed.

> UNIX doesn't like timestamps in the future

I don't think this is a serious consideration.  The problematic scenario would be obtaining high-resolution timestamp (from say time.time()), converting it to datetime and passing it back to OS as a possibly 0.5µs
higher value.  Given that timestamp -> datetime -> timestamp roundtrip by
itself takes over 1µs, it is very unlikely that by the time rounded value hits the OS it is still in the future.
msg246284 - (view) Author: Larry Hastings (larry) * (Python committer) Date: 2015-07-05 01:28
I'm not going to hold up beta 3 while you guys argue about how to round up or down the number of angels that can dance on the head of a pin.
msg246306 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2015-07-05 10:42
Le vendredi 3 juillet 2015, Alexander Belopolsky <report@bugs.python.org> a
écrit :
>
> > UNIX doesn't like timestamps in the future
>
> I don't think this is a serious consideration.  The problematic scenario
> would be obtaining high-resolution timestamp (from say time.time()),
> converting it to datetime and passing it back to OS as a possibly 0.5µs
> higher value.  Given that timestamp -> datetime -> timestamp roundtrip by
> itself takes over 1µs, it is very unlikely that by the time rounded value
> hits the OS it is still in the future.
>

In many cases the resolution is 1 second. For example, a filesystem with a
resolution of 1second. Or an API only supporting a resolution of 1 second.

With a resoltuion of 1 second, timestamps in the future are likely (50%).

Sorry I don't remember all detail of timestamp rounding and all issues that
I saw.
msg246307 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2015-07-05 10:44
My rationale is more general than datetime. But problems araise when
different API use different rounding methods.
msg246334 - (view) Author: Alexander Belopolsky (belopolsky) * (Python committer) Date: 2015-07-05 19:04
I'll let others fight this battle.  In my view, introducing floating point timestamp method for datetime objects was a mistake.  See issue #2736.  

Specifically, I would like to invite Velko Ivanov to rethink his rant at msg124197.

If anyone followed his advise and started using timestamp method to JSON-serialize datetimes around 3.3, have undoubtedly being bitten by the present bug (but may not know it yet.)

For those who need robust code, I will continue recommending (dt - EPOCH)/timedelta(seconds=1) expression over the timestamp method and for JSON serialization (dt - EPOCH) // datetime.resolution to convert to integers and EPOCH + n * datetime.resolution to convert back.
msg248942 - (view) Author: Tim Peters (tim.peters) * (Python committer) Date: 2015-08-21 02:14
It is really bad that roundtripping current microsecond datetimes doesn't work.  About half of all microsecond-resolution datetimes fail to roundtrip correctly now.  While the limited precision of a C double guarantees roundtripping of microsecond datetimes "far enough" in the future will necessarily fail, that point is about 200 years from now.

Rather than argue endlessly about rounding, it's possible instead to make the tiniest possible change to the timestamp _produced_ at the start.  Here's code explaining it:

    ts = d.timestamp()
    # Will microseconds roundtrip correctly?  For times far
    # enough in the future, there aren't enough bits in a C
    # double for that to always work.  But for years through
    # about 2241, there are enough bits.  How does it fail
    # before then?  Very few microsecond datetimes are exactly
    # representable as a binary float.  About half the time, the
    # closest representable binary float is a tiny bit less than
    # the decimal value, and that causes truncating 1e6 times
    # the fraction to be 1 less than the original microsecond
    # value.
    if int((ts - int(ts)) * 1e6) != d.microsecond:
        # Roundtripping fails.  Add 1 ulp to the timestamp (the
        # tiniest possible change) and see whether that repairs
        # it.  It's enough of a change until doubles just plain
        # run out of enough bits.
        mant, exp = math.frexp(ts)
        ulp = math.ldexp(0.5, exp - 52)
        ts2 = ts + ulp
        if int((ts2 - int(ts2)) * 1e6) == d.microsecond:
            ts = ts2
        else:
            # The date is so late in time that a C double's 53
            # bits of precision aren't sufficient to represent
            # microseconds faithfully.  Leave the original
            # timestamp alone.
            pass
    # Now ts exactly reproduces the original datetime,
    # if that's at all possible.

This assumes timestamps are >= 0, and that C doubles have 53 bits of precision.  Note that because a change of 1 ulp is the smallest possible change for a C double, this cannot make closest-possible unequal datetimes produce out-of-order after-adjustment timestamps.

And, yes, this sucks ;-)  But it's far better than having half of timestamps fail to convert back for the next two centuries.  Alas, it does nothing to get the intended datetime from a microsecond-resolution timestamp produced _outside_ of Python.  That requires rounding timestamps on input - which would be a better approach.

Whatever theoretical problems may exist with rounding, the change to use truncation here is causing real problems now.  Practicality beats purity.
msg249282 - (view) Author: Alexander Belopolsky (belopolsky) * (Python committer) Date: 2015-08-28 17:17
I wish we could use the same algorithm in datetime.utcfromtimestamp as we use in float to string conversion.  This may allow the following chain of conversions to round trip in most cases:

float literal -> float -> datetime -> seconds.microseconds string
msg249300 - (view) Author: Tim Peters (tim.peters) * (Python committer) Date: 2015-08-28 22:02
> I wish we could use the same algorithm in
> datetime.utcfromtimestamp as we use in float
> to string conversion.  This may allow the
> following chain of conversions to round trip in most cases:
>
> float literal -> float -> datetime -> seconds.microseconds string

I don't follow.  float->string produces the shortest string that reproduces the float exactly.  Any flavor of changing a timestamp to a microsecond-precision datetime is essentially converting a float * 1e6 to an integer - there doesn't seem to be a coherent concept of "shortest integer" that could apply.  We have to fill every bit a datetime has.

A variant of the code I posted could be "good enough":  take the result we get now (truncate float*1e6).  Also add 1 ulp to the float and do that again.  If the results are the same, we're done.  If the results are different, and the difference is 1, take the second result.  Else keep the first result.  What this "means" is that we're rounding up if and only if the original is so close to the boundary that the tiniest possible amount of floating-point noise is all that's keeping it from giving a different result - but also that the float "has enough bits" to represent a 1-microsecond difference (which is true of current times, but in a couple centuries will become false).

But that's all nuts compared to just rounding float*1e6 to the nearest int, period.  There's nothing wrong with doing that.  Truncating is propagating the tiniest possible binary fp representation error all the way into the microseconds.  It would be defensible _if_ we were using base-10 floats (in which "representation error" doesn't occur for values expressed _in_ base 10).  But we're not.  Truncating a base-2 float _as if_ it were a base-10 float is certain to cause problems.  Like the one this report is about ;-)
msg249301 - (view) Author: Alexander Belopolsky (belopolsky) * (Python committer) Date: 2015-08-28 22:19
I probably misremembered a different issue.  See msg194311.

>>> timedelta(seconds=0.6112295) == timedelta(seconds=1)*0.6112295
False

I thought the problem there was that the same float was converted to one decimal by str() and to a different decimal by timedelta.  But now it looks like it was something else.

Does your algorithm guarantee that any float that is displayed with 6 decimal places or less will convert to a datetime or timedelta with microseconds matching the fractional part?
msg249303 - (view) Author: Alexander Belopolsky (belopolsky) * (Python committer) Date: 2015-08-28 22:28
OK, I looked at the wrong place.  Here is the correct example:

>>> x = float.fromhex('0x1.38f312b1b36bdp-1')
>>> x
0.6112295
>>> round(x, 6)
0.611229
>>> timedelta(0, x).microseconds
611230

but I no longer remember whether we concluded that timedelta got it wrong or round or both or neither. :-)
msg249304 - (view) Author: Tim Peters (tim.peters) * (Python committer) Date: 2015-08-28 22:31
> Does your algorithm guarantee that any float that
> is displayed with 6 decimal places or less will
> convert to a datetime or timedelta with microseconds
> matching the fractional part?

No algorithm can, for datetimes far enough in the future (C doubles just plain run out of enough bits).

Apart from negative timestamps (which I didn't consider - they just blow up on my platform :-) ), the intent is to do the best that _can_ be done.

But _proving_ things in this area isn't simple, and there's no need for it:  check in a change to round the thing, and be done with it.  If Victor wants to rework rounding again, that's fine, but only under a _requirement_ that this particular bug remain fixed.  His change created the problem, and it's still languishing half a year after being reported - there's little sense in continuing to wait for him to do something about it.
msg249307 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2015-08-29 00:35
Hi, I'm trying to write the rationale of the changes that I wrote in pytime.h in Python 3.3-3.5. The rounding of microseconds or nanoseconds is tricky. The code changed a lot and we used and we are still using various rounding methods depending on the case...

Alexander Belopolsky wrote:
> I believe the correct mode is "ROUND_HALF_EVEN".  This is the mode used by the builtin round() function: (...)

Right, round(float) and round(decimal.Decimal) uses the ROUND_HALF_EVEN rounding method.

On Python < 3.3, datetime.datetime.fromtimestamp(float) doesn't use exactly ROUND_HALF_EVEN, but it looks more to "round half away from zero" (the decimal module doesn't seem to support this exact rounding method).

The difference between ROUND_HALF_EVEN and "round half away from zero" is subtle. The two rounding methods only return a different result on the following case:

   divmod(t + us * 1e-6, 1.0)[1] * 1e6 == 0.5

where t and us are integers (t is a number of seconds created by mktime() and us is a number of microseconds in [0; 999999]).

I don't think that the case can occur. I failed to find such case for various values of t between 0 and 2**40, and us=0 or us=1. 1e-6 (10^-6 = 0.000001) cannot be represented exactly in base 2 (IEEE 754).

--

To move forward, we should agree on which rounding method datetime.datetime.fromtimestamp() should use, implement it in pytime.c (add a constant in pytime.h, implement it in pytime.c, and then write unit tests in test_time.py), and then use it in datetime.datetime.fromtimestamp().

IMHO we should only modify the rounding method used by datetime.datetime.fromtimestamp() and datetime.datetime.utcfromtimestamp(), other functions use the "right" rounding method.
msg249308 - (view) Author: Tim Peters (tim.peters) * (Python committer) Date: 2015-08-29 01:40
> >>> x = float.fromhex('0x1.38f312b1b36bdp-1')
> >>> x
> 0.6112295
> >>> round(x, 6)
> 0.611229
> >>> timedelta(0, x).microseconds
> 611230
>
> but I no longer remember whether we concluded that
> timedelta got it wrong or round or both or neither. :-)

Here you go:

>>> import decimal
>>> decimal.Decimal(x)
Decimal('0.61122949999999998116351207499974407255649566650390625')

That's the exact value you're actually using.  What's "correct" depends on what's intended.

round(x, 6) actually rounds to

>>> decimal.Decimal(round(x, 6))
0.6112290000000000222968310481519438326358795166015625

and that's fine.  timedelta's result does not match what using infinite precision would deliver, but I couldn't care much less ;-)

The real lesson to take from all this, when you design your own killer language, is that using a binary floating point type for timestamps comes with many costs and surprises.
msg249309 - (view) Author: Tim Peters (tim.peters) * (Python committer) Date: 2015-08-29 02:08
> IMHO we should only modify the rounding method used by
> datetime.datetime.fromtimestamp() and
> datetime.datetime.utcfromtimestamp(), other functions
> use the "right" rounding method.

Fine by me.  How about today? ;-)

The regression reported here must get repaired.  nearest/even is generally favored when there's a choice.

I personally _prefer_ add-a-half-and-chop in time contexts that need rounding, because it's more uniform.  That is, picturing a context that rounds to 1 digit for clarity, using a decimal system, with a uniformly spaced sequence of inputs on the first line, then a line with add-a-half-and-chop results, and then a line with nearest/even results:

0.0 0.5 1.0 1.5 2.0 2.5 3.0 3.5 input
0.0 1.0 1.0 2.0 2.0 3.0 3.0 4.0 add-a-half-and-chop
0.0 0.0 1.0 2.0 2.0 2.0 3.0 4.0 nearest/even

From the last (nearest/even) line, you'd never guess that the inputs were uniformly spaced; in the second line, you would.  nearest/even's "in a tie, sometimes go up, sometimes go down" is, IMO, unnatural in this context.

But it doesn't make a lick of real difference to timestamp functions.  We're not working in decimal, and people aren't going to be staring at hex patterns in output.  So I'd pick whichever is easier to implement.
msg249519 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2015-09-01 23:52
New changeset abeb625b20c2 by Victor Stinner in branch 'default':
Issue #23517: Add "half up" rounding mode to the _PyTime API
https://hg.python.org/cpython/rev/abeb625b20c2
msg249520 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2015-09-01 23:57
New changeset b690bf218702 by Victor Stinner in branch 'default':
Issue #23517: datetime.datetime.fromtimestamp() and
https://hg.python.org/cpython/rev/b690bf218702
msg249521 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2015-09-02 00:03
Ok, I fixed the issue in Python 3.6. Example with the initial message:

$ python2.7 -c 'import datetime; print(datetime.datetime.utcfromtimestamp(1424817268.274).microsecond); print(datetime.datetime.utcfromtimestamp(-1424817268.274).microsecond)'
274000
726000

$ python3.6 -c 'import datetime; print(datetime.datetime.utcfromtimestamp(1424817268.274).microsecond); print(datetime.datetime.utcfromtimestamp(-1424817268.274).microsecond)'
274000
726000

I wrote:
"On Python < 3.3, datetime.datetime.fromtimestamp(float) doesn't use exactly ROUND_HALF_EVEN, but it looks more to "round half away from zero" (the decimal module doesn't seem to support this exact rounding method)."

I was wrong: it's decimal.ROUND_HALF_UP in fact.

I will backport the change to Python 3.4 and 3.5. Since this issue was defined as a bugfix, it should be fixed in Python 3.5.1 (too late for 3.5.0).
msg249525 - (view) Author: Larry Hastings (larry) * (Python committer) Date: 2015-09-02 01:41
> too late for 3.5.0

How's that?
msg249528 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2015-09-02 08:41
New changeset 30454ef98e81 by Victor Stinner in branch 'default':
Backed out changeset b690bf218702
https://hg.python.org/cpython/rev/30454ef98e81

New changeset 700303850cd7 by Victor Stinner in branch 'default':
Issue #23517: Fix _PyTime_ObjectToDenominator()
https://hg.python.org/cpython/rev/700303850cd7

New changeset 03c97bb04cd2 by Victor Stinner in branch 'default':
Issue #23517: Reintroduce unit tests for the old PyTime API since it's still
https://hg.python.org/cpython/rev/03c97bb04cd2
msg249530 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2015-09-02 08:56
Larry Hasting wrote:
>> too late for 3.5.0
> How's that?

Well, for example... my change broke all buildbots.

I don't think that it's good idea to rush to fix Python 3.5 :-) This part of Python (handling timestamps, especially the rounding mode) is complex, I prefer to check for all buildbots and wait for some feedback from users (wait at least 2 weeks).

I reverted my change, another function must be changed:

$ python2 -c 'import datetime; print(datetime.timedelta(microseconds=0.5))'
0:00:00.000001
$ python3 -c 'import datetime; print(datetime.timedelta(microseconds=0.5))'
0:00:00

datetime.timedelta must also use the ROUND_HALF_UP method, as Python 2, instead of ROUND_HALF_EVEN (datetime.py uses the round() function).

$ python2 -c 'import datetime; print(datetime.timedelta(microseconds=1.5))'
0:00:00.000002
$ python3 -c 'import datetime; print(datetime.timedelta(microseconds=1.5))'
0:00:00.000002

I have to rework my patch to use ROUND_HALF_UP in datetime.timedelta(), datetime.datetime.fromtimestamp() and datetime.datetime.utcfromtimestamp(), and update test_datetime.
msg249532 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2015-09-02 10:01
New changeset df074eb2a5be by Victor Stinner in branch 'default':
Issue #23517: Try to fix test_time on "x86 Ubuntu Shared 3.x" buildbot
https://hg.python.org/cpython/rev/df074eb2a5be
msg249539 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2015-09-02 11:54
New changeset 59185ef69166 by Victor Stinner in branch 'default':
Issue #23517: test_time, skip a test checking a corner case on floating point
https://hg.python.org/cpython/rev/59185ef69166
msg249569 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2015-09-02 20:37
New changeset 0eb8c182131e by Victor Stinner in branch 'default':
Issue #23517: datetime.timedelta constructor now rounds microseconds to nearest
https://hg.python.org/cpython/rev/0eb8c182131e
msg249611 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2015-09-03 07:14
New changeset bf634dfe076f by Victor Stinner in branch 'default':
Issue #23517: fromtimestamp() and utcfromtimestamp() methods of
https://hg.python.org/cpython/rev/bf634dfe076f
msg249728 - (view) Author: Larry Hastings (larry) * (Python committer) Date: 2015-09-04 07:11
I will happy delegate to Tim Peters whether or not this should be fixed in 3.5.0, or whether it should wait until 3.5.1 or even 3.6.

Tim, ball's in your court!
msg249744 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2015-09-04 08:53
Backport to Python 3.4 splitted in 3 patches:

* (1) timedelta_round_half_up_py34.patch, backport changeset 0eb8c182131e: "datetime.timedelta constructor now rounds microseconds to nearest with ties going away from zero (ROUND_HALF_UP)".

* (2) round_half_up.patch: add _PyTime_ROUND_HALF_UP rounding mode to the _PyTime API

* (3) fromtimestamp_round_half_up.patch, backport changeset bf634dfe076f: "fromtimestamp() and utcfromtimestamp() methods of datetime.datetime now round microseconds to nearest with ties going away from zero (ROUND_HALF_UP)"
msg249771 - (view) Author: Tim Peters (tim.peters) * (Python committer) Date: 2015-09-04 15:01
Larry, I appreciate the vote of confidence, but I'm ill-equipped to help at the patch level:  I'm solely on Windows, and (long story) don't even have a C compiler at the moment.  The patch(es) are too broad and delicate to be sure of without kicking the tires (running contrived examples).

So I would sub-delegate to Alexander and/or Mark.  They understand the issues too.  I was just the most annoying about insisting it get fixed ;-)
msg249777 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2015-09-04 15:40
> Larry, I appreciate the vote of confidence, but I'm ill-equipped to help at the patch level:  (...)  The patch(es) are too broad and delicate to be sure of without kicking the tires (running contrived examples).

Well, the patches change how timedelta, .fromtimestamp() and .utcfromtimestamp() round the number of microseconds. It's a deliberate choice since it was decided that the current rounding mode is a bug, and not a feature :-)

The code is well tested. There are unit tests on how numbers are rounded for: timedelta, .(utc)fromtimestamp(), and even the C private API _PyTime. The code is (almost) the same in default and was validated on various platforms. So I'm confident on the change.
msg249778 - (view) Author: Tim Peters (tim.peters) * (Python committer) Date: 2015-09-04 15:52
That's great, Victor!  Another person trying the code with their own critical eyes would still be prudent.  Two days ago you wrote:

> This part of Python (handling timestamps, especially
> the rounding mode) is complex, I prefer to check for
> all buildbots and wait for some feedback from users
> (wait at least 2 weeks).

It's not entirely clear why that switched to "So I'm confident on the change." in 12 days short of 2 weeks ;-)

I have no reason to doubt your confidence.  Just saying some independent checking is prudent (but I can't do it at this time).
msg249779 - (view) Author: Alexander Belopolsky (belopolsky) * (Python committer) Date: 2015-09-04 16:07
I'll try to find the time to kick the tires on this patch this weekend.
msg249786 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2015-09-04 16:52
2015-09-04 17:52 GMT+02:00 Tim Peters <report@bugs.python.org>:
> That's great, Victor!  Another person trying the code with their own critical eyes would still be prudent.

Sure!

> It's not entirely clear why that switched to "So I'm confident on the change." in 12 days short of 2 weeks ;-)

He he. 2 days ago, the buildbots were broken for various reasons. I
fixed a lot of issues (unrelated to this rounding mode issue), so I
now got the confirmation that the test pass on all platforms.

> I have no reason to doubt your confidence.  Just saying some independent checking is prudent (but I can't do it at this time).

Sorry if I wasn't clear. I'm confident, but not enough to not wait for
a review :-)

--

Usually, I don't wait for a review simply because there are too few
reviewers :-( I spent the last 3 years to work alone on the funnny
_PyTime C API project. I started to write an article to tell this
journey ;-)
msg249787 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2015-09-04 16:52
Alexander Belopolsky added the comment:
> I'll try to find the time to kick the tires on this patch this weekend.

Cool! Keep me in touch ;-)
msg249788 - (view) Author: Alexander Belopolsky (belopolsky) * (Python committer) Date: 2015-09-04 16:58
Victor,

Do I understand correctly that this is already committed in 3.4 - 3.6 development branches and we just need to decide whether to cherry-pick this fix to 3.5rc?

Is the "review" link up-to date?
msg249791 - (view) Author: Alexander Belopolsky (belopolsky) * (Python committer) Date: 2015-09-04 17:23
It looks like this patch violates fromtimestamp(s) == EPOCH + timedelta(seconds=s) invariant:

Python 3.6.0a0 (default:73911e6c97c8, Sep  4 2015, 13:14:12)
>>> E = datetime(1970,1,1,tzinfo=timezone.utc)
>>> s = -1/2**7
>>> datetime.fromtimestamp(s, timezone.utc) == E + timedelta(seconds=s)
False
msg249796 - (view) Author: Tim Peters (tim.peters) * (Python committer) Date: 2015-09-04 17:35
FYI, that invariant failed for me just now under the released 3.4.3 too:

Python 3.4.3 (v3.4.3:9b73f1c3e601, Feb 24 2015, 22:44:40) [MSC v.1600 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> from datetime import *
>>> E = datetime(1970,1,1,tzinfo=timezone.utc)
>>> s = -1/2**7
>>> datetime.fromtimestamp(s, timezone.utc)
datetime.datetime(1969, 12, 31, 23, 59, 59, 992187, tzinfo=datetime.timezone.utc)
>>> E + timedelta(seconds=s)
datetime.datetime(1969, 12, 31, 23, 59, 59, 992188, tzinfo=datetime.timezone.utc)

It's an exactly-tied rounding case for the exactly-representable-in-binary s = -0.0078125.
msg249798 - (view) Author: Alexander Belopolsky (belopolsky) * (Python committer) Date: 2015-09-04 17:54
Can someone check 3.3?  I believe that was the release where we tried to get various rounding issues right.
msg249825 - (view) Author: Mark Lawrence (BreamoreBoy) * Date: 2015-09-04 20:30
Python 3.3.5 (v3.3.5:62cf4e77f785, Mar  9 2014, 10:35:05) [MSC v.1600 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> from datetime import *
>>> E = datetime(1970,1,1,tzinfo=timezone.utc)
>>> s = -1/2**7
>>> datetime.fromtimestamp(s, timezone.utc)
datetime.datetime(1969, 12, 31, 23, 59, 59, 992187, tzinfo=datetime.timezone.utc)
>>> E + timedelta(seconds=s)
datetime.datetime(1969, 12, 31, 23, 59, 59, 992187, tzinfo=datetime.timezone.utc)

FWIW Windows 10.
msg249827 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2015-09-04 20:38
Oh, it looks like my implementation of ROUND_HALF_UP is wrong.
Negative numbers are not correctly rounded :-/ I'm working on a fix.
msg249834 - (view) Author: Alexander Belopolsky (belopolsky) * (Python committer) Date: 2015-09-04 21:05
Victor,

I did not quite understand why you've chosen ROUND_HALF_UP over ROUND_HALF_EVEN, but as long as fromtimestamp() uses the same rounding as timedelta() - I am happy.
msg249835 - (view) Author: Tim Peters (tim.peters) * (Python committer) Date: 2015-09-04 21:09
Alex, if you like, I'll take the blame for the rounding method - I did express a preference for it here:

http://bugs.python.org/issue23517#msg249309

When I looked at the code earlier, the round-half-up implementation looked good to me (floor(x+0.5) if x >= 0 else ceil(x-0.5)).
msg249836 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2015-09-04 21:27
> I did not quite understand why you've chosen ROUND_HALF_UP over ROUND_HALF_EVEN, but as long as fromtimestamp() uses the same rounding as timedelta() - I am happy.

Not only Tim prefers this rounding mode, Python 2.7 also uses the same mode, and the original poster basically said that Python 3 doesn't behave like Python 2. So... the most obvious rounding mode is the same than Python 2, ROUND_HALF_UP.
msg249841 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2015-09-04 22:09
New changeset 3c29d05c0710 by Victor Stinner in branch 'default':
Issue #23517: Fix implementation of the ROUND_HALF_UP rounding mode in
https://hg.python.org/cpython/rev/3c29d05c0710
msg249842 - (view) Author: Alexander Belopolsky (belopolsky) * (Python committer) Date: 2015-09-04 22:11
It would be nice to hear from Mark Dickinson on this.  In Python 3, we took a much more systematic approach to rounding than a rather haphazard Python 2. For example, the rounding mode for timedelta(0, float_seconds) is not specified in Python 2, but it is in Python 3:


https://docs.python.org/2/library/datetime.html#datetime.timedelta
https://docs.python.org/3/library/datetime.html#datetime.timedelta
msg249844 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2015-09-04 22:25
> Is the "review" link up-to date?

Oh, I wrote a patch serie, but Rietveld doesn't know the dependencies between my patches...

So I attached combined_py34.patch which combines the 3 patches into a single one, hard to reviewer, but it should work with Rietveld.
msg249845 - (view) Author: Tim Peters (tim.peters) * (Python committer) Date: 2015-09-04 22:31
Yes, it would be good to hear from Mark.  When I first saw this report, I checked to see whether he was on the nosy list.  He is, but is apparently busy elsewhere.

My opinions haven't changed:  nearest/even is unnatural for rounding times ("sometimes up, sometimes down" grates against the nature of monotonically non-decreasing timestamps), but it doesn't make a lick of real difference.  If the inconsequential choice was documented, then sure, that it _was_ documented makes it consequential, so that's the one that must be used.
msg249847 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2015-09-04 22:34
> My opinions haven't changed:  nearest/even is unnatural for rounding times ("sometimes up, sometimes down" grates against the nature of monotonically non-decreasing timestamps),

By the way, why does Python use ROUND_HALF_EVEN for round()? It also feels unnatural to me!

>>> round(0.5)
0
>>> round(1.5)
2
msg249848 - (view) Author: Tim Peters (tim.peters) * (Python committer) Date: 2015-09-04 22:38
Victor, there are good "theoretical" reasons for using half/even rounding in _general_ use.  But this bug report isn't the place to get into them.  Here it just should be enough to note that the IEEE 754 floating point standard _requires_ half/even to be the default rounding mode.  Slowly but surely, all the world's non-business computer applications are moving toward that.  Python is just one of 'em.
msg249849 - (view) Author: Alexander Belopolsky (belopolsky) * (Python committer) Date: 2015-09-04 22:43
> By the way, why does Python use ROUND_HALF_EVEN for round()?

ROUND_HALF_EVEN does not introduce statistical bias in your floating point data.  With this choice a randomly chosen decimal has an equal chance of being rounded up or down.  I think one can prove the same for a binary float being converted to a decimal with a fixed number of places after dot.  The builtin round() is a unique beast and I would have to ask Mark to know what its statistical properties are, but in general round-half-to-even is justified by the desire of not introducing a statistical bias and I don't think the natural asymmetry of the time line matters in this argument.
msg249850 - (view) Author: Alexander Belopolsky (belopolsky) * (Python committer) Date: 2015-09-04 22:46
Here is what Wikipedia has to say on the subject:

"""
Round half to even .. This method treats positive and negative values symmetrically, and is therefore free of sign bias. More importantly, for reasonable distributions of y values, the expected (average) value of the rounded numbers is the same as that of the original numbers. However, this rule will introduce a towards-zero bias when y − 0.5 is even, and a towards-infinity bias for when it is odd.
"""
https://en.wikipedia.org/wiki/Rounding#Round_half_to_even
msg249851 - (view) Author: Alexander Belopolsky (belopolsky) * (Python committer) Date: 2015-09-04 22:47
.. and here is an unbeatable argument: "this variant of the round-to-nearest method is also called unbiased rounding, convergent rounding, statistician's rounding, **Dutch rounding**, Gaussian rounding, odd–even rounding, or bankers' rounding."
msg249853 - (view) Author: Alexander Belopolsky (belopolsky) * (Python committer) Date: 2015-09-04 22:49
.. and "[Round half to even] is the default rounding mode used in IEEE 754 computing functions and operators."
msg249856 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2015-09-04 23:03
Ok, thanks for the explanation :-)
msg249875 - (view) Author: Tim Peters (tim.peters) * (Python committer) Date: 2015-09-05 00:54
Goodness.  It's the properties of "randomly chosen decimals" that have nothing to do with timestamps ;-)  timestamps aren't random, so "statistical bias" is roughly meaningless in this context.  I gave a specific example before of how nearest/even destroys obvious regularities in a _sequence_ of timestamps, where half-up preserves as much of the input regularity as possible.  That's worth more than a million abstract "head arguments" on Wikipedia.

But it doesn't make a lick of real difference either way.  We're rounding to microseconds, and there are only 64 "fractional parts" where the methods could _possibly_ deliver different results:  those of the form i/128 for i in range(1, 128, 2).  All and only those are exactly representable in base 2, and require exactly 7 decimal digits "after the decimal point" to express in decimal, _and_ end with "5" in decimal.  Half end with "25" while the other half with "75".  So Alex's 1/128 is one of the only 32 possible fractional parts where it makes a difference.  We systematically force all these cases to even, and dare think that's _not_ "biased"?  Half-up would leave half the results even and half odd, exactly the same as the _input_ odd/even distribution of the 6th digit.  And preserve the input strict alternation between even and odd in the 6th digit.  nearest/even destroys all of that.

Except that, I agree, there's no arguing with "Dutch rounding" ;-)
msg249879 - (view) Author: Alexander Belopolsky (belopolsky) * (Python committer) Date: 2015-09-05 01:25
> timestamps aren't random, so "statistical bias" is roughly meaningless in this context.

I agree.  I don't think I made any arguments about timestamps specifically other than a consistency with timedeltas.  In the later case, I think all the usual arguments for half-to-even tiebreaker apply.

Let's just leave it at "When in doubt - use Dutch rounding."
msg249885 - (view) Author: Tim Peters (tim.peters) * (Python committer) Date: 2015-09-05 02:28
Bah.  It doesn't matter who's consuming the rounding of a binary float to decimal microseconds:  there are only 32 possible fractional parts where nearest/even and half-up deliver different results.  half-up preserves properties of these specific inputs that nearest/even destroys.  These inputs themselves have no bias - they're utterly uniformly spaced.

Not only does nearest/even _introduce_ bias on these inputs by destroying these properties, it doesn't even preserve the spacing between them.  Half-up leaves them all 5 microseconds apart, while nearest/even creates a bizarre "sometimes 4 microseconds apart, sometimes 6" output spacing out of thin air.

So it's not a question of "when in doubt" to me, it's a question of "live up to what the docs already say".  Although, again, it doesn't make a lick of real difference.  That's why we'll never stop arguing about it ;-)
msg249886 - (view) Author: Tim Peters (tim.peters) * (Python committer) Date: 2015-09-05 02:34
> Half-up leaves them all 5 microseconds apart,

When only looking at the decimal digit in the 6th place after rounding.    Which is all I did look at ;-)
msg249898 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2015-09-05 08:50
New changeset d755f75019c8 by Victor Stinner in branch 'default':
Issue #23517: Skip a datetime test on Windows
https://hg.python.org/cpython/rev/d755f75019c8
msg249909 - (view) Author: Alexander Belopolsky (belopolsky) * (Python committer) Date: 2015-09-05 14:08
> It doesn't matter who's consuming the rounding of a binary
> float to decimal microseconds

That is true, but what does matter is who is producing the incoming floats.   Consider an extreme case of a timer that ticks twice a microsecond and gives you results in a timespec struct style sec, nsec pair.  If you convert those to timedeltas with timedelta(0, sec, nsec/1000), you will see half of nsec/1000 floats ending in .5 and half ending with .0.  If you have a large sample of (sec, nsec) measurements, the mean of corresponding timedeltas will be 0.25µs larger with the half-up rounding than the actual mean of your samples.

Is round-half-to-even a panacea?  Of course not.  In an extreme case if all your measurements are 1.5µs - it will not help at all, but in a more realistic sample you are likely to have an approximately equal number of even and odd µs and the Dutch rounding will affect the mean much less than round-half-up.

As you said, for most uses none of this matters, but we are not discussing a change to timedelta(0, s) rounding here.  All I want from this issue is to keep the promise that we make in the docs:

On the POSIX compliant platforms, utcfromtimestamp(timestamp) is equivalent to the following expression:

datetime(1970, 1, 1) + timedelta(seconds=timestamp)

https://docs.python.org/3/library/datetime.html#datetime.datetime.utcfromtimestamp

Tim, while we have this entertaining theoretical dispute, I am afraid we are leaving Victor confused about what he has to do in his patch.  I think we agree that fromtimestamp() should use Dutch rounding.  We only disagree why.

If this is the case, please close this thread with a clear advise to use round-half-to-even and we can continue discussing the rationale privately or in a different forum.
msg249910 - (view) Author: Tim Peters (tim.peters) * (Python committer) Date: 2015-09-05 16:46
Victor, sorry if I muddied the waters here:  Alex & I do agree nearest/even must be used.  It's documented for timedelta already, and the seconds-from-the-epoch invariant Alex showed is at least as important to preserve as round-tripping.

Alex, agreed about the overall mean in the example, but expect that continuing to exchange contrived examples isn't really a major life goal for either of us ;-)  Thanks for playing along.
msg249925 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2015-09-05 20:49
> Alex & I do agree nearest/even must be used.

Ok, so Python 3.6 should be updated to use ROUND_HALF_EVEN, and then
updated patches should be backported to Python 3.4 and 3.5. Right?

Right now, Python 3.6 uses ROUND_HALF_UP.
msg249926 - (view) Author: Alexander Belopolsky (belopolsky) * (Python committer) Date: 2015-09-05 21:05
You may find it easier to start with a patch for 3.5 which is the only time sensitive task.  I'll be happy to review your code in whatever form you find it easier to submit, but I believe in hg it is easier to forward port than to backport.
msg250043 - (view) Author: Larry Hastings (larry) * (Python committer) Date: 2015-09-07 03:55
So is this considered broken enough that I need to accept a fix for 3.5.0?  And has a consensus been reached about exactly what that fix would be?
msg250047 - (view) Author: Tim Peters (tim.peters) * (Python committer) Date: 2015-09-07 04:06
Universal consensus on ROUND_HALF_EVEN, yes.

I would like to see it repaired for 3.5.0, but that's just me saying so without offering to do a single damned thing to make it happen ;-)
msg250048 - (view) Author: Larry Hastings (larry) * (Python committer) Date: 2015-09-07 04:07
Well, I'm already holding up rc3 on one other issue, might as well fix this too.  Can somebody make me a pull request?
msg250049 - (view) Author: Larry Hastings (larry) * (Python committer) Date: 2015-09-07 04:08
I suspect we're not fixing this in 3.4, so I'm removing 3.4 from the version list.
msg250066 - (view) Author: Larry Hastings (larry) * (Python committer) Date: 2015-09-07 05:33
Okay, this is literally the only thing rc3 is waiting on now.
msg250075 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2015-09-07 05:57
I understand that I have to implement a new rounding mode. The code will be
new, I'm not condifent enough to push it immedialty into python 3.5. IMHO a
buggy rounding mode is worse than keeping the current rounding mode. The
rounding mode changed in python 3.3. There is no urgency to fix it.

I will change python 3.6, then 3.4 and 3.5.1.
Le 7 sept. 2015 07:33, "Larry Hastings" <report@bugs.python.org> a écrit :

>
> Larry Hastings added the comment:
>
> Okay, this is literally the only thing rc3 is waiting on now.
>
> ----------
>
> _______________________________________
> Python tracker <report@bugs.python.org>
> <http://bugs.python.org/issue23517>
> _______________________________________
>
msg250083 - (view) Author: Larry Hastings (larry) * (Python committer) Date: 2015-09-07 06:49
Is it appropriate to make this change as a "bug fix", in already-released versions of Python?  Would you be happy or sad if you updated your Python from 3.x.y to 3.x.y+1 and the rounding method used when converting floats to datetime stamps changed?
msg250101 - (view) Author: Larry Hastings (larry) * (Python committer) Date: 2015-09-07 12:04
Well, for now I assume it really truly genuinely isn't going in 3.5.0.  I suppose we can debate about 3.4.x and 3.5.1 later, once we have a fix everybody is happy with.
msg250103 - (view) Author: Alexander Belopolsky (belopolsky) * (Python committer) Date: 2015-09-07 12:26
Larry> Well, for now I assume it really truly genuinely isn't going in 3.5.0.

This is an unfortunate outcome.

Larry> I suppose we can debate about 3.4.x and 3.5.1 later

It is even more unfortunate that the question of whether this regression is a bug or not is up for debate again.

Victor> The code will be new, I'm not condifent enough to push it immedialty into python 3.5.

I don't understand why any new code is needed to fix a regression.  All you need to do is to revert a few chunks of 1e9cc1a03365 where the regression was introduced.
msg250104 - (view) Author: Larry Hastings (larry) * (Python committer) Date: 2015-09-07 12:38
If the guy doing the work says "don't merge it in 3.5.0", and we're at the final release candidate before 3.5.0 final ships, and we don't even have a patch that everyone likes yet... it does seem like it's not going to happen for 3.5.0.  Unfortunate perhaps but that's the situation we're in.
msg250107 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2015-09-07 12:56
> All you need to do is to revert a few chunks of 1e9cc1a03365 where the regression was introduced.

Hum, please don't revert this change. I spent a lot of days to write pytime.c/.h.

My patches add more unit tests to datetime, to test the exact rounding mode.

> Well, for now I assume it really truly genuinely isn't going in 3.5.0.  I suppose we can debate about 3.4.x and 3.5.1 later, once we have a fix everybody is happy with.

The rounding mode of microseconds only impact a very few people. The rounding mode changed in 3.3, again in 3.4 and again in 3.5. This issue is the first bug report. So I think it's fine to modify 3.4.x and 3.5.x.
msg250108 - (view) Author: Alexander Belopolsky (belopolsky) * (Python committer) Date: 2015-09-07 13:38
Victor> please don't revert this change.

I did not suggest reverting the entire commit.  The change that affects fromdatetime() is just

-        us = round(frac * 1e6)
+        us = int(frac * 1e6)

in datetime.py.  It is probably more involved in _datetimemodule.c, but cannot be that bad.  You can still keep pytime.c/.h.
msg250113 - (view) Author: Mark Dickinson (mark.dickinson) * (Python committer) Date: 2015-09-07 15:09
FWIW, count me as +1 on roundTiesToEven, mostly just for consistency.  It's easier to remember that pretty much everything in Python 3 does round-ties-to-even (round function, string formatting, float literal evaluations, int-to-float conversion, Fraction-to-float conversion, ...) than to have to remember that there's a handful of corner cases where we do roundTiesToAway instead.
msg250259 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2015-09-08 21:59
New changeset d0bb896f9b14 by Victor Stinner in branch 'default':
Revert change 0eb8c182131e:
https://hg.python.org/cpython/rev/d0bb896f9b14
msg250265 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2015-09-08 23:04
New changeset 171d5590ebc3 by Victor Stinner in branch 'default':
Issue #23517: fromtimestamp() and utcfromtimestamp() methods of
https://hg.python.org/cpython/rev/171d5590ebc3
msg250268 - (view) Author: Larry Hastings (larry) * (Python committer) Date: 2015-09-09 00:00
I wish people wouldn't remove old patches.  There's no harm in leaving them, and it may be a useful historical record.
msg250269 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2015-09-09 00:02
I modified Python 3.6 to use ROUND_HALF_EVEN rounding mode in datetime.timedelta, datetime.datetime.fromtimestamp(), datetime.datetime.utcfromtimestamp().

round_half_even_py34.patch: Backport this change to Python 3.4. I tried to write a minimal patch only changing datetime, not pytime.
msg250270 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2015-09-09 00:04
> I wish people wouldn't remove old patches.  There's no harm in leaving them, and it may be a useful historical record.

It became hard to read this issue, it has a long history. My old patches for Python 3.4 were for ROUND_HALF_UP, but it was then decided to use ROUND_HALF_EVEN. Technically, patches are not "removed" but "unlinked" from the issue: you can get from the history at the bottom of this page.
msg250304 - (view) Author: Larry Hastings (larry) * (Python committer) Date: 2015-09-09 12:58
Given Victor's reluctance to get this in to 3.5.0, this can't even be marked as a "deferred blocker" anymore.  Demoting to normal priority.
msg250405 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2015-09-10 19:03
Alexander: can you please review attached round_half_even_py34.patch? It's the minimum patch to fix datetime/timedelta rounding for Python 3.4 (and 3.5).
msg250973 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2015-09-18 12:58
New changeset ee1cf1b188d2 by Victor Stinner in branch '3.4':
Issue #23517: Fix rounding in fromtimestamp() and utcfromtimestamp() methods
https://hg.python.org/cpython/rev/ee1cf1b188d2
msg250974 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2015-09-18 13:01
Ok, I fixed Python 3.4 and 3.5 too: fromtimestamp() and utcfromtimestamp() now uses also ROUND_HALF_EVEN rounding mode, as timedelta constructor. I explained in Misc/NEWS that the change was made to respect the property:

(datetime(1970,1,1) + timedelta(seconds=t)) == datetime.utcfromtimestamp(t)

Thanks Alexander Belopolsky & Tim Peters for your explanation on rounding. It's not a simple problem :-)

Thanks Tommaso Barbugli for the bug report: it's now fixed on all (maintained) Python 3 versions: 3.4, 3.5 and 3.6. The fix will be part of Python 3.5.1.
History
Date User Action Args
2015-09-18 13:11:31vstinnersetstatus: open -> closed
resolution: fixed
2015-09-18 13:01:57vstinnersetmessages: + msg250974
versions: + Python 3.4
2015-09-18 12:58:34python-devsetmessages: + msg250973
2015-09-10 19:04:58serhiy.storchakasetnosy: + serhiy.storchaka
2015-09-10 19:03:23vstinnersetmessages: + msg250405
2015-09-09 12:58:01larrysetpriority: deferred blocker -> normal

messages: + msg250304
2015-09-09 00:04:17vstinnersetmessages: + msg250270
2015-09-09 00:02:08vstinnersetfiles: + round_half_even_py34.patch

messages: + msg250269
2015-09-09 00:00:37larrysetmessages: + msg250268
2015-09-08 23:59:56vstinnersetfiles: - combined_py34.patch
2015-09-08 23:59:55vstinnersetfiles: - fromtimestamp_round_half_up_py34-2.patch
2015-09-08 23:59:54vstinnersetfiles: - round_half_up_py34-2.patch
2015-09-08 23:59:53vstinnersetfiles: - timedelta_round_half_up_py34.patch
2015-09-08 23:04:28python-devsetmessages: + msg250265
2015-09-08 21:59:50python-devsetmessages: + msg250259
2015-09-07 15:09:28mark.dickinsonsetmessages: + msg250113
2015-09-07 13:38:47belopolskysetmessages: + msg250108
2015-09-07 12:56:39vstinnersetmessages: + msg250107
2015-09-07 12:38:39larrysetmessages: + msg250104
2015-09-07 12:26:40belopolskysetmessages: + msg250103
2015-09-07 12:04:12larrysetmessages: + msg250101
2015-09-07 06:49:25larrysetmessages: + msg250083
2015-09-07 05:57:17vstinnersetmessages: + msg250075
2015-09-07 05:33:26larrysetmessages: + msg250066
2015-09-07 04:08:05larrysetmessages: + msg250049
versions: - Python 3.4
2015-09-07 04:07:38larrysetmessages: + msg250048
2015-09-07 04:06:10tim.peterssetmessages: + msg250047
2015-09-07 03:55:27larrysetmessages: + msg250043
2015-09-05 21:05:28belopolskysetmessages: + msg249926
2015-09-05 20:49:31vstinnersetmessages: + msg249925
2015-09-05 16:46:37tim.peterssetmessages: + msg249910
2015-09-05 14:08:38belopolskysetmessages: + msg249909
2015-09-05 08:50:41python-devsetmessages: + msg249898
2015-09-05 02:34:54tim.peterssetmessages: + msg249886
2015-09-05 02:28:11tim.peterssetmessages: + msg249885
2015-09-05 01:25:22belopolskysetmessages: + msg249879
2015-09-05 00:54:27tim.peterssetmessages: + msg249875
2015-09-04 23:03:49vstinnersetmessages: + msg249856
2015-09-04 22:49:21belopolskysetmessages: + msg249853
2015-09-04 22:47:59belopolskysetmessages: + msg249851
2015-09-04 22:46:03belopolskysetmessages: + msg249850
2015-09-04 22:43:50belopolskysetmessages: + msg249849
2015-09-04 22:38:26tim.peterssetmessages: + msg249848
2015-09-04 22:34:49vstinnersetmessages: + msg249847
2015-09-04 22:31:29tim.peterssetmessages: + msg249845
2015-09-04 22:25:06vstinnersetfiles: + combined_py34.patch

messages: + msg249844
2015-09-04 22:21:22vstinnersetfiles: + fromtimestamp_round_half_up_py34-2.patch
2015-09-04 22:21:04vstinnersetfiles: - fromtimestamp_round_half_up_py34-2.patch
2015-09-04 22:19:30vstinnersetfiles: - fromtimestamp_round_half_up_py34.patch
2015-09-04 22:19:29vstinnersetfiles: - round_half_up_py34.patch
2015-09-04 22:19:22vstinnersetfiles: + fromtimestamp_round_half_up_py34-2.patch
2015-09-04 22:18:06vstinnersetfiles: + round_half_up_py34-2.patch
2015-09-04 22:11:28belopolskysetmessages: + msg249842
2015-09-04 22:09:31python-devsetmessages: + msg249841
2015-09-04 21:27:45vstinnersetmessages: + msg249836
2015-09-04 21:09:05tim.peterssetmessages: + msg249835
2015-09-04 21:05:01belopolskysetmessages: + msg249834
2015-09-04 20:38:19vstinnersetmessages: + msg249827
2015-09-04 20:30:24BreamoreBoysetnosy: + BreamoreBoy
messages: + msg249825
2015-09-04 17:54:00belopolskysetmessages: + msg249798
2015-09-04 17:35:09tim.peterssetmessages: + msg249796
2015-09-04 17:23:19belopolskysetmessages: + msg249791
2015-09-04 16:58:28belopolskysetmessages: + msg249788
2015-09-04 16:54:47belopolskysetstage: commit review
2015-09-04 16:52:57vstinnersetmessages: + msg249787
2015-09-04 16:52:37vstinnersetmessages: + msg249786
2015-09-04 16:07:42belopolskysetassignee: belopolsky
messages: + msg249779
2015-09-04 15:52:32tim.peterssetmessages: + msg249778
2015-09-04 15:40:27vstinnersetmessages: + msg249777
2015-09-04 15:01:56tim.peterssetmessages: + msg249771
2015-09-04 08:53:31vstinnersetfiles: + fromtimestamp_round_half_up_py34.patch
2015-09-04 08:53:19vstinnersetfiles: + round_half_up_py34.patch
2015-09-04 08:53:09vstinnersetfiles: + timedelta_round_half_up_py34.patch
keywords: + patch
messages: + msg249744
2015-09-04 07:11:32larrysetmessages: + msg249728
2015-09-03 07:14:37python-devsetmessages: + msg249611
2015-09-02 20:37:33python-devsetmessages: + msg249569
2015-09-02 11:54:44python-devsetmessages: + msg249539
2015-09-02 10:01:46python-devsetmessages: + msg249532
2015-09-02 08:56:47vstinnersetmessages: + msg249530
2015-09-02 08:41:02python-devsetmessages: + msg249528
2015-09-02 01:41:13larrysetmessages: + msg249525
2015-09-02 00:03:56vstinnersetmessages: + msg249521
2015-09-01 23:57:35python-devsetmessages: + msg249520
2015-09-01 23:52:53python-devsetnosy: + python-dev
messages: + msg249519
2015-08-29 02:08:18tim.peterssetmessages: + msg249309
2015-08-29 01:40:11tim.peterssetmessages: + msg249308
2015-08-29 00:35:02vstinnersetmessages: + msg249307
2015-08-28 22:31:58tim.peterssetmessages: + msg249304
2015-08-28 22:28:48belopolskysetmessages: + msg249303
2015-08-28 22:19:07belopolskysetmessages: + msg249301
2015-08-28 22:02:47tim.peterssetmessages: + msg249300
2015-08-28 17:17:30belopolskysetnosy: + belopolsky
messages: + msg249282
2015-08-28 17:08:48belopolskysettitle: datetime.utcfromtimestamp parses timestamps incorrectly -> datetime.utcfromtimestamp rounds results incorrectly
2015-08-27 20:28:19aconradsetnosy: + aconrad
2015-08-21 02:14:48tim.peterssetnosy: + tim.peters
messages: + msg248942
2015-07-21 07:04:57ethan.furmansetnosy: - ethan.furman
2015-07-05 19:04:33belopolskysetnosy: - belopolsky
2015-07-05 19:04:08belopolskysetnosy: + vivanov
messages: + msg246334
2015-07-05 10:44:38vstinnersetmessages: + msg246307
2015-07-05 10:42:37vstinnersetmessages: + msg246306
2015-07-05 01:28:29larrysetpriority: release blocker -> deferred blocker

messages: + msg246284
2015-07-03 02:07:08belopolskysetmessages: + msg246121
2015-07-02 22:55:45vstinnersetmessages: + msg246104
2015-07-02 21:04:44larrysetmessages: + msg246097
2015-07-02 12:23:26r.david.murraysetpriority: normal -> release blocker
versions: + Python 3.5, Python 3.6
nosy: + larry

messages: + msg246073
2015-07-01 23:31:55trcardensetnosy: + trcarden
messages: + msg246049
2015-02-25 17:57:09belopolskysetmessages: + msg236610
2015-02-25 17:38:00belopolskysetmessages: + msg236609
2015-02-25 17:33:48vstinnersetmessages: + msg236608
2015-02-25 17:31:56vstinnersetmessages: + msg236607
2015-02-25 17:30:57belopolskysetkeywords: + 3.3regression, - 3.2regression
2015-02-25 16:48:46belopolskysetmessages: + msg236597
2015-02-25 16:16:17belopolskysetmessages: + msg236589
2015-02-25 16:06:34vstinnersetmessages: + msg236587
2015-02-25 15:33:56belopolskysetnosy: + vstinner
2015-02-25 15:33:09belopolskysetmessages: + msg236585
2015-02-25 15:13:28belopolskysetkeywords: + 3.2regression

messages: + msg236581
2015-02-25 15:00:55belopolskysetnosy: + mark.dickinson
messages: + msg236580
2015-02-25 14:11:57r.david.murraysetnosy: + r.david.murray
messages: + msg236577
2015-02-25 09:30:54ned.deilysetnosy: + belopolsky
2015-02-24 22:43:21ethan.furmansetnosy: + ethan.furman
messages: + msg236553
2015-02-24 22:38:42tbarbuglicreate