classification
Title: acquire(timeout) of threading.Lock and threading.Condition is affected by jumps in system time: Python should use sem_clockwait(CLOCK_MONOTONIC)
Type: behavior Stage:
Components: Interpreter Core Versions: Python 3.8
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: methane, mikecrowe, vstinner, wocket
Priority: normal Keywords:

Created on 2020-09-04 08:34 by wocket, last changed 2021-09-13 14:59 by vstinner.

Files
File name Uploaded Description Edit
time_test.py wocket, 2020-09-04 08:34
Messages (4)
msg376338 - (view) Author: Jonas Norling (wocket) Date: 2020-09-04 08:34
The timeout for threading.Lock, threading.Condition, etc, is not using a monotonic clock — it is affected if the system time (realtime clock) is set.

The attached program can be used to show the problem. It is expected to print "Took 2.000 s" repeatedly, but if run with permissions to set the system time, it prints:
$ sudo ./time_test.py
Took 2.400 s
Took 1.657 s
Took 2.044 s
Took 2.401 s
...

(the 1.6 s time can be explained by NTP correcting the clock)

There are already a number of closed bugs for this and related issues: bpo 23428, bpo 31267, bpo 35747.

This happens in Python 3.7.7 (ARM32, Yocto Warrior), Python 3.8.2 (AMD64, Ubuntu Linux 20.04) and Python v3.9.0rc1 (AMD64, Ubuntu Linux 20.04).
msg376340 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-09-04 08:49
bpo-23428 modified the pthread implementation of conditional variable to use pthread_condattr_setclock(&ca, CLOCK_MONOTONIC) is available: commit 001fee14e0f2ba5f41fb733adc69d5965925a094. The change should be part of Python 3.8.

What is your sys.thread_info value? Example on Fedora 32 with Python 3.8.5:

>>> sys.thread_info
sys.thread_info(name='pthread', lock='semaphore', version='NPTL 2.31')

Sadly, the semaphore implementation doesn't use monotonic clock. See glibc issues:

* https://sourceware.org/bugzilla/show_bug.cgi?id=14717
* https://bugzilla.kernel.org/show_bug.cgi?id=112521
msg376394 - (view) Author: Jonas Norling (wocket) Date: 2020-09-04 18:55
sys.thread_info = sys.thread_info(name='pthread', lock='semaphore', version='NPTL 2.31') on my system. Looking at the source I think the semaphore implementation will be used on all modern Linux systems.

In my tests it works as expected on a Macintosh (3.8.5 with lock='mutex+cond') and also if I force a Linux build to use the mutex+cond implementation by defining HAVE_BROKEN_POSIX_SEMAPHORES.

Doesn't look like those glibc and Linux bug reports will get any attention anytime soon. I will find a workaround instead :-/
msg399405 - (view) Author: Mike Crowe (mikecrowe) Date: 2021-08-11 16:11
glibc v2.30 onwards provides sem_clockwait which can wait on either CLOCK_MONOTONIC or CLOCK_REALTIME. I failed to notice that https://sourceware.org/bugzilla/show_bug.cgi?id=14717 existed until today. :(
History
Date User Action Args
2021-09-13 14:59:05vstinnersettitle: Timeout is affected by jumps in system time -> acquire(timeout) of threading.Lock and threading.Condition is affected by jumps in system time: Python should use sem_clockwait(CLOCK_MONOTONIC)
2021-08-11 16:11:17mikecrowesetnosy: + mikecrowe
messages: + msg399405
2020-09-04 18:55:50wocketsetmessages: + msg376394
2020-09-04 08:49:30vstinnersetnosy: + vstinner, methane
messages: + msg376340
2020-09-04 08:34:54wocketcreate