classification
Title: Change os.utime &c functions to use nanosecond precision where possible
Type: enhancement Stage:
Components: Extension Modules Versions: Python 3.3
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: larry Nosy List: Arfrever, larry, loewis, python-dev
Priority: normal Keywords:

Created on 2011-09-05 21:08 by larry, last changed 2012-02-26 21:50 by larry. This issue is now closed.

Files
File name Uploaded Description Edit
larry.utimensat.patch.r1 larry, 2011-09-05 21:08 review
Messages (7)
msg143563 - (view) Author: Larry Hastings (larry) * (Python committer) Date: 2011-09-05 21:08
Since Linux 2.5 stat() has supported reading nanosecond resolution (1b/sec) for atime and mtime.  However, the utime() family of functions could only write atime and mtime with microsecond resolution (1m/sec) until relatively recently.

On POSIX platforms Python reads atime and mtime at nanosecond resolution but only writes them at microsecond resolution.  This impedance mismatch can cause undesirable behavior.  Consider the following code:

    import os
    import shutil
    import sys

    def mtime(path):
      return os.stat(path).st_mtime

    src = sys.argv[0]
    dest = src + ".new"
    shutil.copy2(src, dest)
    assert mtime(src) == mtime(dest)

When run under Python on any modern Linux system, the assert fails.  (I think any Python since 2.5 will do; I tested with 2.7.1 and a fresh 3.3 trunk.)

The accompanying patch modifies Modules/posixmodule.c so all functions that write atime and mtime use nanosecond resolution when possible.  With this patch applied, all the regression tests pass (except the ones for extension modules I didn't compile), and the above code runs to completion.

Happy to hoist the patch up on Rietveld if there's interest.

p.s. I have the commit bit set, so I'd like to be the one to check this in if we get that far.
msg143566 - (view) Author: Larry Hastings (larry) * (Python committer) Date: 2011-09-05 21:50
A small aside:

An IEEE 754 double is insufficient to store a modern timestamp with nanosecond accuracy.  We're currently at just over 1.3 billion seconds since the epoch.  This eats 28 bits of the mantissa.  (The value is actually 31 bits to the left of the decimal point, but you get the leading 1 and the subsequent two 0s for free.)  Precisely storing a billion values to the right of the decimal point would require another 30 bits.  The mantissa of a double is sadly only 52 bits.  So we're shy by 6 bits, give or take.

So this patch doesn't make Python 100% accurate with respect to atime/mtime.  What it *does* do is makes Python's loss of accuracy consistent.  I claim that's a definite improvement.
msg143568 - (view) Author: Arfrever Frehtes Taifersar Arahesis (Arfrever) * Date: 2011-09-05 21:57
It's a duplicate of issue #11457.
(Python >=3.3 has os.utimensat() and os.futimens().)
msg143571 - (view) Author: Larry Hastings (larry) * (Python committer) Date: 2011-09-05 22:18
> It's a duplicate of issue #11457.

No.  #11457 is a how-many-angels-on-the-head-of-a-pin discussion proposing new representations of ctime/mtime/atime to preserve nanosecond accuracy.  This patch changes the behavior of os.utime, os.futimes, os.lutimes, and os.futimesat so they conserve all of the existing representation's accuracy, making them consistent with os.stat.

> (Python >=3.3 has os.utimensat() and os.futimens().)

How is that relevant?

I mean, sure, I leveraged some of the work from that support in my patch--it's heavily reliant on the preprocessor macros HAVE_UTIMENSAT and HAVE_FUTIMENS now generated by configure.  (Thanks, Ross!)  But the presence of those functions does not mitigate the loss of accuracy inside os.utime &c.
msg143587 - (view) Author: Martin v. Löwis (loewis) * (Python committer) Date: 2011-09-06 07:15
The patch looks fine to me.
msg143750 - (view) Author: Roundup Robot (python-dev) Date: 2011-09-09 02:29
New changeset 1de6619733d9 by Larry Hastings in branch 'default':
Issue #12904: os.utime, os.futimes, os.lutimes, and os.futimesat now write
http://hg.python.org/cpython/rev/1de6619733d9
msg154399 - (view) Author: Larry Hastings (larry) * (Python committer) Date: 2012-02-26 21:50
This was committed last September; I should have marked it closed then.
History
Date User Action Args
2012-02-26 21:50:32larrysetstatus: open -> closed
resolution: fixed
messages: + msg154399
2011-09-09 02:29:05python-devsetnosy: + python-dev
messages: + msg143750
2011-09-06 07:15:42loewissetnosy: + loewis
messages: + msg143587
2011-09-05 22:18:42larrysetmessages: + msg143571
2011-09-05 21:57:56Arfreversetnosy: + Arfrever
messages: + msg143568
2011-09-05 21:50:13larrysetmessages: + msg143566
2011-09-05 21:08:02larrycreate