classification
Title: time.sleep (floatsleep()) should use clock_nanosleep() on Linux
Type: behavior Stage: resolved
Components: Library (Lib) Versions: Python 3.11
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: Livius, akira, python-dev, shankarunni, vstinner, yselivanov
Priority: normal Keywords: patch

Created on 2014-04-18 20:12 by shankarunni, last changed 2021-09-25 16:18 by vstinner. This issue is now closed.

Files
File name Uploaded Description Edit
wait.py vstinner, 2021-09-20 21:29
bench.py vstinner, 2021-09-22 14:50
Pull Requests
URL Status Linked Edit
PR 28077 closed python-dev, 2021-08-30 22:49
PR 28111 merged Livius, 2021-09-01 13:00
PR 28311 merged vstinner, 2021-09-13 12:43
PR 28341 closed Livius, 2021-09-14 20:36
PR 28350 merged vstinner, 2021-09-15 10:51
PR 28483 merged vstinner, 2021-09-20 21:29
PR 28526 closed Livius, 2021-09-22 23:38
PR 28545 merged vstinner, 2021-09-24 11:01
Messages (22)
msg216799 - (view) Author: Shankar Unni (shankarunni) Date: 2014-04-18 20:12
I know that an earlier request to use nanosleep() has been rejected as "wontfix", but I'm filing this one for a different reason.

Today, timemodule.c:floatsleep() calls select() on platforms that support it. On Linux, select() with a timeout has an unfortunate property that it is very sensitive to clock jumps, because it computes a sleep end time based on the current kernel timestamp.

If the system clock is yanked back (by ntpd, or other processes), then the process can end up sleeping for a very long time. (E.g. if the clock is yanked back by half an hour while we are in the middle of, say, a sleep(10), then the process will sleep until "original_kernel_clock+10", which will turn into a half-hour sleep.

Yes, systems shouldn't jerk their clocks around, but we can't often control this sort of thing on end-user environments.

Using clock_nanosleep(CLOCK_MONOTONIC, 0, <timespec>, NULL) makes the sleep a much more reliable thing, and mostly insensitive to such jumps. (It'll still be affected by any adjtime(), but that's OK in this case).
msg216814 - (view) Author: Shankar Unni (shankarunni) Date: 2014-04-18 22:47
I'm working on a patch, but I noticed a similar issue in Condition.wait(), which also keeps re-evaluating the "remaining sleep time" based on the current kernel clock, with similar effects.

I'll try to address both issues, or we could open a separate bug for the latter..
msg217225 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2014-04-26 23:38
> I know that an earlier request to use nanosleep() has been rejected as "wontfix"

It was the issue #13981. I created this issue while I worked on the PEP 410 (nanosecond timestamp). I closed the issue myself, it doesn't mean that Python must not use the function, just that I didn't want to work on it anymore at this time.
msg217226 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2014-04-26 23:45
"I'm working on a patch, but I noticed a similar issue in Condition.wait(), which also keeps re-evaluating the "remaining sleep time" based on the current kernel clock, with similar effects."

I see that Lock.acquire(timeout) uses the C function gettimeofday() to recompute the timeout if acquiring the lock was interrupted (C error "EINTR"). It would be better to use a monotonic clock here, but please open a new issue because it's unrelated to nanosleep().

Or did you another bug?

By the way, you didn't mention the Python version. Are you working on Python 2.7 or 3.5?

See also the PEP 418.
msg217227 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2014-04-26 23:59
If you want to modify time.sleep(), you must be careful of the portability: Windows, Linux, but also Mac OS X, FreeBSD, Solaris, etc.

Try to describe the behaviour of each underlying C function on each platform to be able to describe the "portable behaviour" on all platforms, especially the expected behaviour when the system clock is changed (is time.sleep impacted or not? always?) and the expected behaviour when the system is suspended.

For example, it looks like nanosleep() uses a different clock depending on OS (Linux uses CLOCK_MONOTONIC, other UNIX platforms use CLOCK_REALTIME).
http://lists.gnu.org/archive/html/bug-coreutils/2012-08/msg00087.html

I know that you suggest to use clock_nanosleep(), but this function is not available on all platforms. For example, I would not use it on Windows.

Another example (on Fedora?): "sleep() ignores time spent with a suspended system"
http://mjg59.dreamwidth.org/7846.html

You should also decide how to handle interrupted sleep (C error "EINTR"). Currently, the sleep is interrupted, no error is raised.

I began to describe all these functions in the PEP 418, even if I didn't change the implementation with the PEP:
http://legacy.python.org/dev/peps/pep-0418/#sleep
msg217233 - (view) Author: Shankar Unni (shankarunni) Date: 2014-04-27 00:26
> If you want to modify time.sleep(), you must be careful of the portability: Windows, Linux, but also Mac OS X, FreeBSD, Solaris, etc.

Oh, I totally agree. What I'm trying to do is to define another autoconf flag (HAVE_CLOCK_NANOSLEEP), that does a feature test and enable that flag, and just use that if available.

Now that's a good point that if we have clock_nanosleep() on another platform (non-Linux) and it does the wrong thing, then I might have to add further discrimination.

For now, one sticking point that I've stumbled across is that clock_nanosleep() requires "-lrt". Complicates the autoconf check a bit.
msg217235 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2014-04-27 00:36
2014-04-27 2:26 GMT+02:00 Shankar Unni <report@bugs.python.org>:
>> If you want to modify time.sleep(), you must be careful of the portability: Windows, Linux, but also Mac OS X, FreeBSD, Solaris, etc.
>
> Oh, I totally agree. What I'm trying to do is to define another autoconf flag (HAVE_CLOCK_NANOSLEEP), that does a feature test and enable that flag, and just use that if available.

I'm talking about the expected behaviour which can be found in the
documentation of the function:
https://docs.python.org/dev/library/time.html#time.sleep
msg401108 - (view) Author: Benjamin Szőke (Livius) * Date: 2021-09-05 22:27
Can you review my final implementation?
https://github.com/python/cpython/pull/28111
msg401701 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2021-09-13 12:37
New changeset 85a4748118c3793be7047ecbcbfc79dd07cb2a75 by Livius in branch 'main':
bpo-21302: Add clock_nanosleep() implementation for time.sleep() (GH-28111)
https://github.com/python/cpython/commit/85a4748118c3793be7047ecbcbfc79dd07cb2a75
msg401706 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2021-09-13 15:40
New changeset 85dc53a463967659075744ad911d08a32aa70dd5 by Victor Stinner in branch 'main':
bpo-21302: Update time.sleep() doc for clock_nanosleep() (GH-28311)
https://github.com/python/cpython/commit/85dc53a463967659075744ad911d08a32aa70dd5
msg401824 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2021-09-15 12:26
New changeset b49263b698993cad2b8aaddc55cdeaa678412b30 by Victor Stinner in branch 'main':
bpo-21302: Add _PyTime_AsNanoseconds() (GH-28350)
https://github.com/python/cpython/commit/b49263b698993cad2b8aaddc55cdeaa678412b30
msg402279 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2021-09-20 21:29
wait.py: script to test the time.sleep() function. Press CTRL+C multiple times during the sleep to check that even if the sleep is interrupted, time.sleep() sleeps the expected duration.
msg402437 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2021-09-22 14:09
New changeset 58f8adfda3c2b42f654a55500e8e3a6433cb95f2 by Victor Stinner in branch 'main':
bpo-21302: time.sleep() uses waitable timer on Windows (GH-28483)
https://github.com/python/cpython/commit/58f8adfda3c2b42f654a55500e8e3a6433cb95f2
msg402438 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2021-09-22 14:13
Livius: your first PR modified Sleep() in Modules/_tkinter.c to use nanosleep(). I don't see the point since this function has a solution of 1 ms (10^-3). Using select() on Unix is enough: resolution of 1 us (10^-6).
msg402440 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2021-09-22 14:50
bench.py: measure the shortest possible sleep. Use time.sleep(1e-10): 0.1 nanosecond. It should be rounded to the resolution of the used sleep function, like 1 ns on Linux.

On Linux with Fedora 34 Python 3.10 executable, I get:

    Mean +- std dev: 60.5 us +- 12.9 us (80783 values)

On Windows with a Python 3.11 debug build, I get:

    Mean +- std dev: 21.9 ms +- 7.8 ms (228 values)

Sadly, it seems like on Windows 10, one of the following function still uses the infamous 15.6 ms resolution:

* CreateWaitableTimerW()
* SetWaitableTimer()
* WaitForMultipleObjects()
msg402441 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2021-09-22 14:51
> On Windows with a Python 3.11 debug build, I get:
> Mean +- std dev: 21.9 ms +- 7.8 ms (228 values)

I wrote an optimization to cache the Windows timer handle between time.sleep() calls (don't close it). I don't think that it's needed because they shortest sleep is about 15.6 ms. CreateWaitableTimerW() is likely way more fast than 15.6 ms. So this optimization is basically useless.
msg402442 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2021-09-22 14:52
Livius: do you care about using nanosleep(), or can I close the issue?
msg402448 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2021-09-22 15:43
See also bpo-19007: "precise time.time() under Windows 8: use GetSystemTimePreciseAsFileTime".
msg402618 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2021-09-25 12:36
New changeset 7834ff26cbcd4d8394d64d80d9f51a364d38b1c6 by Victor Stinner in branch 'main':
bpo-21302: Add nanosleep() implementation for time.sleep() in Unix (GH-28545)
https://github.com/python/cpython/commit/7834ff26cbcd4d8394d64d80d9f51a364d38b1c6
msg402619 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2021-09-25 12:37
Thanks Livius for all these nice enhancements!
msg402630 - (view) Author: Benjamin Szőke (Livius) * Date: 2021-09-25 15:36
Do you have any information about when will be it released in 3.11?
msg402632 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2021-09-25 16:18
> Do you have any information about when will be it released in 3.11?

Here is a schedule of Python 3.11 releases:
https://www.python.org/dev/peps/pep-0664/

In the meanwhile, you can develop a C extension to get the feature.
History
Date User Action Args
2021-09-25 16:18:40vstinnersetmessages: + msg402632
2021-09-25 15:36:20Liviussetmessages: + msg402630
2021-09-25 12:37:55vstinnersetstatus: open -> closed
resolution: fixed
messages: + msg402619

stage: patch review -> resolved
2021-09-25 12:36:35vstinnersetmessages: + msg402618
2021-09-24 11:01:19vstinnersetpull_requests: + pull_request26929
2021-09-22 23:38:52Liviussetpull_requests: + pull_request26917
2021-09-22 15:43:55vstinnersetmessages: + msg402448
2021-09-22 14:52:28vstinnersetmessages: + msg402442
2021-09-22 14:51:54vstinnersetmessages: + msg402441
2021-09-22 14:50:04vstinnersetfiles: + bench.py

messages: + msg402440
2021-09-22 14:13:34vstinnersetmessages: + msg402438
2021-09-22 14:09:38vstinnersetmessages: + msg402437
2021-09-20 21:29:58vstinnersetpull_requests: + pull_request26882
2021-09-20 21:29:45vstinnersetfiles: + wait.py

messages: + msg402279
2021-09-15 12:26:52vstinnersetmessages: + msg401824
2021-09-15 10:51:14vstinnersetpull_requests: + pull_request26764
2021-09-14 20:36:56Liviussetpull_requests: + pull_request26754
2021-09-13 15:40:36vstinnersetmessages: + msg401706
2021-09-13 12:43:44vstinnersetpull_requests: + pull_request26724
2021-09-13 12:37:42vstinnersetmessages: + msg401701
2021-09-05 22:27:20Liviussetmessages: + msg401108
versions: + Python 3.11
2021-09-01 13:00:26Liviussetnosy: + Livius
pull_requests: + pull_request26552
2021-08-30 22:49:48python-devsetkeywords: + patch
nosy: + python-dev

pull_requests: + pull_request26521
stage: patch review
2014-04-27 00:36:47vstinnersetmessages: + msg217235
2014-04-27 00:26:44shankarunnisetmessages: + msg217233
2014-04-26 23:59:46vstinnersetmessages: + msg217227
2014-04-26 23:45:47vstinnersetmessages: + msg217226
2014-04-26 23:38:52vstinnersetmessages: + msg217225
2014-04-20 00:39:51akirasetnosy: + akira
2014-04-18 22:47:14shankarunnisetmessages: + msg216814
2014-04-18 22:25:03yselivanovsetnosy: + vstinner, yselivanov
2014-04-18 20:12:03shankarunnicreate