classification
Title: Expose C struct timespec (nanosecond resolution) in time module
Type: enhancement Stage: resolved
Components: Extension Modules Versions: Python 3.6
process
Status: closed Resolution: rejected
Dependencies: Superseder:
Assigned To: belopolsky Nosy List: Arfrever, belopolsky, ethan.furman, lemburg, mdcb808@gmail.com, vstinner
Priority: normal Keywords: patch

Created on 2014-12-18 16:34 by mdcb808@gmail.com, last changed 2014-12-19 16:17 by belopolsky. This issue is now closed.

Files
File name Uploaded Description Edit
time.struct_timespec.patch mdcb808@gmail.com, 2014-12-18 16:34 patch review
Messages (21)
msg232889 - (view) Author: mdcb (mdcb808@gmail.com) * Date: 2014-12-18 16:34
nanosecond support has been discussed at length on python-dev and issue 15443.
POSIX.1b defines a struct timespec that is commonly used in C, and seems a good candidate to add core nanosecond support. kernel's time-related structs typically end up in the time module.

Attached patch defines a new type struct_timespec for the time module. A new capsule exports the type along with to/from converters - opening a bridge for C, and for example the datetime module.
msg232892 - (view) Author: Alexander Belopolsky (belopolsky) * (Python committer) Date: 2014-12-18 17:57
See also discussion in #9079, #14127 and #14180.  At some point there was some resistance to use capsule mechanism to share code within stdlib and the functions that were shared between time and datetime modules were moved to the core.

I am -1 on adding struct_timespec type.

POSIX defines [1] the following functions that take struct timespec:

int        clock_getres(clockid_t, struct timespec *);
int        clock_gettime(clockid_t, struct timespec *);
int        clock_settime(clockid_t, const struct timespec *);
int        nanosleep(const struct timespec *, struct timespec *);

and a pair of timer functions that take timespec indirectly through itimerspec struct:

int        timer_gettime(timer_t, struct itimerspec *);
int        timer_settime(timer_t, int, const struct itimerspec *,
               struct itimerspec *);

In addition, struct stat provides timespec members on some systems.

There was a long discussion on how to represent high precession time in Python.  PEP 410 proposed using the decimal type and it was rejected.

Nanosecond support was ultimately added to os.struct() by adding integer st_{a,m,c}time_ns members.  See #14127.

Interface to nanosleep is already available via time.sleep() function.  If we ever need higher precision sleep we can add time.nanosleep() that takes time in nanoseconds.  We have a similar story with time.clock().

Overall, the proposed type is much less convenient than integer nanoseconds because it does not support arithmetics. 

In any case, I think it is premature to discuss adding a new type before some functions are proposed that would produce or consume instances of this type.

[1] http://pubs.opengroup.org/onlinepubs/7908799/xsh/time.h.html
msg232894 - (view) Author: Ethan Furman (ethan.furman) * (Python committer) Date: 2014-12-18 18:12
I haven't reviewed the patch yet, but I believe the intent is not for better sleep support, but simply to be able to create and record time data which contains nano-seconds.

python-dev discussion here:

  https://mail.python.org/pipermail/python-dev/2014-December/137522.html

As far as producing and consuming: time() could be used to produce them, and then any user-function that cares could consume them.
msg232895 - (view) Author: Alexander Belopolsky (belopolsky) * (Python committer) Date: 2014-12-18 18:21
> .. the intent is not for better sleep support, but simply
> to be able to create and record time data which contains nano-seconds.

Can you describe a specific use-case?   What's the advantage of the proposed time.struct_timespec over say

>>> timespec = namedtuple('timespec', 'sec,nsec')

?
msg232896 - (view) Author: Alexander Belopolsky (belopolsky) * (Python committer) Date: 2014-12-18 18:25
> time() could be used to produce them

How?
msg232902 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2014-12-18 20:17
See the issue #22117 which basically implement the PEP 410, but only for private C API. The idea is to avoid loss of precision caused by the float type when it is possible. For example, it would be possible for datetime.datetime.now() to avoid the float time.
msg232905 - (view) Author: Alexander Belopolsky (belopolsky) * (Python committer) Date: 2014-12-18 20:43
> it would be possible for datetime.datetime.now()
> to avoid the float time.

C implementation of datetime.now() does not rely on float time, so this is only an issue for the Python implementation.

Moreover, as long as datetime keeps its microsecond resolution, float timestamps are good until the next century.

In any case, I don't see how struct_timespec is better than integer expressing time in nanoseconds.

We can implement time.nanotime() returning an int without having to invent a new type.
msg232906 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2014-12-18 21:17
> C implementation of datetime.now() does not rely on float time, so this is only an issue for the Python implementation.

Ah yes, but there is another technical issue that I'm trying to address in the issue #22117: datetime.datetime.now() is implemented with _PyTime_gettimeofday() which has a resolution of 1 microsecond, even it is now implemented with clock_gettime(CLOCK_REALTIME) which has a resolution of 1 nanoecond.

On Linux, the effictive resolution of clock_gettime(CLOCK_REALTIME) is better than 1 microsecond, around 1/4 microsecond (250 nanosecond).

> Overall, the proposed type is much less convenient than integer nanoseconds because it does not support arithmetics. 

I'm working on using a number of nanoseconds using an integer type. It's just a signed 64-bit bit integer. If we decide to support nanosecond resolution in Python, a integer number of nanosecond may be enough.

But this issue looks like the PEP 410 which was rejected. If you are motivated enough, you may update the PEP and write a new one. But first, read again the PEP and the discussion explaining why it was rejected.

Basically, the loss of precision is very rare (or may not occur) in practice and the PEP 410 required to modify a lot of functions.

Remember that the effictive resolution of time.time() on Windows is just 1 millisecond (0.001 second)... See the PEP 418 for many numbers, list of hardware clocks, operating system, etc.
https://www.python.org/dev/peps/pep-0418/#system-time
msg232908 - (view) Author: Alexander Belopolsky (belopolsky) * (Python committer) Date: 2014-12-18 22:41
Would anyone object if I rename this issue to "Expose C struct timespec in time module"?  The current title is way too broad.
msg232910 - (view) Author: Ethan Furman (ethan.furman) * (Python committer) Date: 2014-12-18 22:58
Just keep the word nanasecond in there somewhere, as that is the motivating purpose behind the patch.
msg232911 - (view) Author: Alexander Belopolsky (belopolsky) * (Python committer) Date: 2014-12-18 23:28
I would say that rejection note for PEP 410 [1] and the implementation of st_[amc]time_ns fields in os.stats() have established a de-facto standard for representing nanosecond resolution timestamps in Python.

I think this proposal should be rejected.

[1] https://mail.python.org/pipermail/python-dev/2012-February/116837.html
msg232912 - (view) Author: Ethan Furman (ethan.furman) * (Python committer) Date: 2014-12-18 23:49
If I am reading data from an external device that has nanosecond resolution, how would I create such a time stamp in Python right now?
msg232913 - (view) Author: mdcb (mdcb808@gmail.com) * Date: 2014-12-18 23:52
I'm going to be my own devil's advocate:

PyLong_FromLong

...

Thanks for all the links, I hadn't realized there was so much background to the issue, there is some good reading there too.
you've changed the title now, but in fact the intention was the other aspect, core nanosecond support in python. I just happened to pick timespec because it does the job and is reasonably widespread.

reading all this, and threading lighlty, I was playing with

class timestamp(int): pass
  """measure of time expressed as a number of nanoseconds"""

but that seems to loose the type information after doing arithmetics

x=timestamp(10)
y=timestamp(20)
type(x+y) gives <int>
msg232914 - (view) Author: Alexander Belopolsky (belopolsky) * (Python committer) Date: 2014-12-18 23:57
> If I am reading data from an external device that has nanosecond
> resolution, how would I create such a time stamp in Python right now?

seconds * 10**9 + nanoseconds  (translated to C API if necessary)
msg232915 - (view) Author: mdcb (mdcb808@gmail.com) * Date: 2014-12-19 00:00
can we change back the title because in fact, exposing struct timespec wasn't really the intention (but keep the patch if you want it!)
msg232916 - (view) Author: Alexander Belopolsky (belopolsky) * (Python committer) Date: 2014-12-19 00:03
> but that seems to loose the type information after doing arithmetics

Hear, hear.  See #2267.  If you find out why Python was designed this way - let me know.
msg232918 - (view) Author: Alexander Belopolsky (belopolsky) * (Python committer) Date: 2014-12-19 00:16
> can we change back the title because in fact

Please don't.  If you would like to discuss general ideas - the right forum would be the "python-ideas" mailing list.

We have no shortage of issue numbers: once you have another specific proposal - feel free to open a new issue.

If you withdraw your proposal to apply time.struct_timespec.patch, I will close this issue.
msg232920 - (view) Author: mdcb (mdcb808@gmail.com) * Date: 2014-12-19 00:40
I'm fine my patch doesn't resolve the "nanosecond support", but that doesn't mean the issue is closed per say.

Ref. to PEP410 rejection and de facto standard seems a bit expeditive.

assuming it worked, this would somewhat be more agreeable ?

>class timestamp(int): pass
>  """measure of time expressed as a number of nanoseconds"""
msg232921 - (view) Author: mdcb (mdcb808@gmail.com) * Date: 2014-12-19 00:48
firefox did something and changed some fields I did not intend to. I'm trying to undo that now.
msg232922 - (view) Author: Alexander Belopolsky (belopolsky) * (Python committer) Date: 2014-12-19 01:26
"nanosecond support [in Python]" is a PEP-size topic.  As Victor suggested, "If you are motivated enough, you may update the PEP and write a new one."  The tracker does not provide a sufficiently large forum to discuss "nanosecond support" in full generality.

>assuming it worked, this would somewhat be more agreeable ?

>class timestamp(int): pass
>  """measure of time expressed as a number of nanoseconds"""

By "working", I assume you mean "if arithmetic operations worked as expected."  In this case you are reinventing mxDateTime [1], which is similar to stdlib datetime, but supports a much higher resolution.

> Ref. to PEP410 rejection and de facto standard seems a bit expeditive.

I don't think so.  Possibly, I should have referred to PEP 20, as well, but that is usually implicit.  We now have two APIs in stdlib that pass nanosecond-resolution timestamps as plain integers: os.stat() and os.utime().  This is not going to change, so for compatibility reasons, any new type would have to be interchangeable with int.  In theory, your timestamp subclass would qualify, but what advantage would it give you over plain int?  If you start adding functionality to timestamp, you will quickly end up with something that is half-way between int and either datetime or timedelta.


[1] http://www.egenix.com/products/python/mxBase/mxDateTime
msg232930 - (view) Author: mdcb (mdcb808@gmail.com) * Date: 2014-12-19 05:59
naively and coming from C, you have time_t to represent time so even though underneath it's typedef, it gives the casual user like me comfort (2038 not accounted). I don't know the details why struct timespec was chosen rather than nanoseconds as integer, and in fact would rather leave that worry to others. 
maybe a typdef PyTimeStamp would make the defacto you mention more obvious, and maybe class timestamp(int):pass in python that seems to bring even more meaning but I don't know how you map that in the C API.

more practically speaking: you're welcomed to close this issue since it's seeminly going nowhere.
History
Date User Action Args
2014-12-19 16:17:07belopolskysetstatus: open -> closed
components: + Extension Modules, - Library (Lib)
2014-12-19 05:59:49mdcb808@gmail.comsetmessages: + msg232930
2014-12-19 01:26:16belopolskysetstatus: pending -> open

messages: + msg232922
2014-12-19 00:48:05mdcb808@gmail.comsetstatus: open -> pending
resolution: rejected
messages: + msg232921
2014-12-19 00:40:59mdcb808@gmail.comsetstatus: pending -> open
resolution: rejected -> (no value)
messages: + msg232920
2014-12-19 00:16:34belopolskysetstatus: open -> pending
messages: + msg232918

assignee: belopolsky
resolution: rejected
stage: resolved
2014-12-19 00:03:33belopolskysetmessages: + msg232916
2014-12-19 00:00:11mdcb808@gmail.comsetmessages: + msg232915
2014-12-18 23:57:09belopolskysetmessages: + msg232914
2014-12-18 23:55:53Arfreversetnosy: + Arfrever
2014-12-18 23:52:15mdcb808@gmail.comsetmessages: + msg232913
2014-12-18 23:49:37ethan.furmansetmessages: + msg232912
2014-12-18 23:28:54belopolskysetmessages: + msg232911
2014-12-18 23:19:22belopolskysettitle: nanosecond support -> Expose C struct timespec (nanosecond resolution) in time module
2014-12-18 22:58:18ethan.furmansetmessages: + msg232910
2014-12-18 22:41:04belopolskysetmessages: + msg232908
2014-12-18 21:17:24vstinnersetmessages: + msg232906
2014-12-18 20:43:55belopolskysetmessages: + msg232905
2014-12-18 20:17:05vstinnersetmessages: + msg232902
2014-12-18 18:25:18belopolskysetmessages: + msg232896
2014-12-18 18:21:31belopolskysetmessages: + msg232895
2014-12-18 18:12:48ethan.furmansetmessages: + msg232894
2014-12-18 17:57:20belopolskysetnosy: + vstinner
messages: + msg232892
2014-12-18 16:49:51ethan.furmansetnosy: + lemburg, belopolsky, ethan.furman
2014-12-18 16:44:20mdcb808@gmail.comsethgrepos: - hgrepo289
2014-12-18 16:34:38mdcb808@gmail.comcreate