classification
Title: Use the monotonic clock for thread conditions on POSIX platforms
Type: Stage: resolved
Components: Interpreter Core Versions: Python 3.8
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: anikey, cstratak, neologix, pablogsal, vstinner
Priority: normal Keywords: patch

Created on 2015-02-09 21:34 by vstinner, last changed 2019-02-20 10:35 by vstinner. This issue is now closed.

Files
File name Uploaded Description Edit
cond_timedwait_monotonic.patch vstinner, 2015-02-09 22:17 review
Pull Requests
URL Status Linked Edit
PR 4964 closed erik.bray, 2017-12-21 17:09
Messages (11)
msg235637 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2015-02-09 21:34
Python 3.5 now requires a monotonic clock to start and has the C function _PyTime_monotonic().

Python/condvar.h and Python/thread_pthread.h should use the monotonic clock CLOCK_MONOTONIC, not the system clock CLOCK_REALTIME. See the PEP 418 for the rationale.

Most platforms support pthread_condattr_setclock(CLOCK_MONOTONIC), except Mac OS X and old versions of Android.

The glib looks to use pthread_cond_timedwait_relative_np() for Mac OS X:
https://mail.gnome.org/archives/commits-list/2014-February/msg07782.html

Note: Android had non-standard pthread_cond_timedwait_monotonic() and pthread_cond_timedwait_monotonic_np() functions. Android is not a official supported platform, and newer Android version now support pthread_condattr_setclock(). I prefer to not support old Android versions (yet).
https://android-review.googlesource.com/#/c/83881/

--

For Windows, SleepConditionVariableSRW() is used on Windows 7 and newer, otherwise WaitForSingleObjectEx() is used. By the way, the check looks to be done during the compilation. I should check which Windows version is used to build Python...

SleepConditionVariableSRW() and WaitForSingleObjectEx() both take a relative timeout, so they don't use (directly) the system clock. I don't see any required change for Windows.

According to the PEP 418: "WaitForSingleObject() uses the same timer as GetTickCount() with the same precision."

Hum, it is not possible to interrupt such wait() with CTRL+c? time.sleep() is implemented with WaitForSingleObjectEx() with _PyOS_SigintEvent(). It doesn't look to be the case here.
msg235639 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2015-02-09 22:12
> Python/condvar.h and Python/thread_pthread.h should use the monotonic clock CLOCK_MONOTONIC

Oh, I forgot that Python/thread_pthread.h only uses pthread_cond_timedwait() if semaphores are emulated with mutexes+conditional variables.

On most platforms, PyThread_acquire_lock_timed() is implemented with sem_timedwait(). Problem: sem_timedwait() requires an absolute time using the CLOCK_REALTIME clock and the clock is not yet configurable on Linux :-(

See the feature request in the glibc: "Bug 14717 - Allow choice of clock source for calls to sem_timedwait() and pthread_mutex_timedwait()" opened in 2012:
https://sourceware.org/bugzilla/show_bug.cgi?id=14717

Note: QNX provides sem_timedwait_monotonic().
msg235640 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2015-02-09 22:17
cond_timedwait_monotonic.patch: Work-in-progress patch. It doesn't change configure.ac yet to check if pthread_condattr_setclock() is supported (with CLOCK_MONOTONIC).
msg236156 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2015-02-17 23:47
Oh, I missed the issue #12822 which looks to fix similar bugs.
msg273565 - (view) Author: Charalampos Stratakis (cstratak) * Date: 2016-08-24 14:17
Hello,

Is there any progress on the issue? Should someone take over?
msg333841 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-01-17 10:05
I marked bpo-31267 "threading.Timer object is affected by changes to system time: Python locks should use a monotonic clock if available" as a duplicate of this issue.
msg333844 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-01-17 10:07
I closed bpo-35747 "Python threading event wait influenced by date change" as a duplicate of the issue.
msg333845 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-01-17 10:11
> Is there any progress on the issue? Should someone take over?

It's a limitation of the libc, not directly of Python.

The problem is that the sem_timedwait() function of the glibc doesn't allow to specify which clock is used:
https://sourceware.org/bugzilla/show_bug.cgi?id=14717

Someone has to contribute to the glibc to add an option to sem_init() or sem_timedwait() to allow to use a different clock than CLOCK_REALTIME.

One workaround is to use Python to use the mutex+cond implementation of pthread locks, since this one is already able to use CLOCK_MONOTONIC:
https://bugs.python.org/issue31267#msg302257
msg333846 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-01-17 10:11
See also bpo-12822: "NewGIL should use CLOCK_MONOTONIC if possible".
msg333851 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2019-01-17 10:57
> One workaround is to use Python to use the mutex+cond implementation of pthread locks, since this one is already able to use CLOCK_MONOTONIC:

Does this have any drawbacks?
msg336060 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-02-20 10:35
INADA-san fixed bpo-12822 with:

New changeset 001fee14e0f2ba5f41fb733adc69d5965925a094 by Inada Naoki in branch 'master':
bpo-12822: use monotonic clock for condvar if possible (GH-11723)
https://github.com/python/cpython/commit/001fee14e0f2ba5f41fb733adc69d5965925a094
History
Date User Action Args
2019-02-20 10:35:46vstinnersetmessages: + msg336060
2019-02-20 01:01:03inada.naokisetstatus: open -> closed
stage: patch review -> resolved
resolution: fixed
versions: + Python 3.8, - Python 3.5
2019-01-17 10:57:17pablogsalsetmessages: - msg333850
2019-01-17 10:57:11pablogsalsetmessages: + msg333851
2019-01-17 10:56:40pablogsalsetnosy: + pablogsal
messages: + msg333850
2019-01-17 10:30:37anikeysetnosy: + anikey
2019-01-17 10:11:42vstinnersetmessages: + msg333846
2019-01-17 10:11:18vstinnersetmessages: + msg333845
2019-01-17 10:07:21vstinnersetmessages: + msg333844
2019-01-17 10:06:55vstinnerlinkissue35747 superseder
2019-01-17 10:05:16vstinnersetmessages: + msg333841
2019-01-17 10:04:50vstinnerlinkissue31267 superseder
2017-12-21 17:09:50erik.braysetstage: patch review
pull_requests: + pull_request4856
2016-08-24 14:17:01cstrataksetnosy: + cstratak
messages: + msg273565
2015-02-17 23:47:49vstinnersetmessages: + msg236156
2015-02-09 22:17:39vstinnersetnosy: + neologix
2015-02-09 22:17:31vstinnersetfiles: + cond_timedwait_monotonic.patch
keywords: + patch
messages: + msg235640
2015-02-09 22:12:06vstinnersetmessages: + msg235639
2015-02-09 21:34:27vstinnercreate