classification
Title: Python 3.7 and 3.8 process_time is not reported correctly when built on older macOS versions
Type: Stage: needs patch
Components: Interpreter Core, macOS Versions: Python 3.8, Python 3.7
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: Nitapol, lukasz.langa, ned.deily, ronaldoussoren, vstinner
Priority: deferred blocker Keywords: patch

Created on 2019-03-06 02:23 by Nitapol, last changed 2019-03-25 18:45 by lukasz.langa.

Pull Requests
URL Status Linked Edit
PR 12287 open ned.deily, 2019-03-12 09:40
Messages (8)
msg337270 - (view) Author: Alexander Lopatin (Nitapol) Date: 2019-03-06 02:23
I see this problem only on my iMac (macOS Mojave, Version 10.14.3). My Windows and Linux (CentOS) computers have no such problem.

I asked the question on Stack Overflow today, investigated, and reported here:
 
https://stackoverflow.com/questions/55008397/python-3-5-vs-3-7-process-time

To fix this problem: the build for macOS should be made with compiler Clang 10.0.0 (clang-1000.11.45.5). You made 3.7 and 3.8 with Clang 6.0 (clang-600.0.57). Of course, this could introduce new problems, but also fix some old ones.
msg337277 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2019-03-06 04:50
Thanks for your report.  There does indeed seem to be a problem but, as can be seen if you run your test from SO (please attach it to this issue!) with a current python.org 3.6.x installer for macOS which uses the same build infrastructure as the 3.7 and 3.8 installers, the results are correct.  The macOS Pythons from python.org are built to run on a range of OS versions: these days macOS 10.9+ or 10.6+.  Your test also fails when run on these earlier systems with 3.7.0 or later but not 3.6.8.  There were a number of changes made in 3.7 to the time module and underlying code in Python, specifically Python/pytime.c, so my guess is that the changes are depending on some feature or change that is in later versions of macOS since the expected results are obtained when built directly on and run on a current macOS version.  That needs to be fixed for 3.7.3.

I had time to run a quick check building on earlier macOS versions. It looks like the incorrect results show up when building on macOS 10.11.x and earlier.
10.12 through 10.14 have the correct results.  I don't have time to investigate further today.

BTW, it would be a good idea to adapt your test program as a test case, i.e. ensure the results are "close" to each other.
msg337731 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2019-03-12 09:49
Further investigation shows that several time related functions were added to macOS at 10.12 including clock_gettime. For older systems, timemodule.c falls back to using getrusage. With Python 3.6.x, that fallbacks correctly but it appears that refactoring introduced with the implementation of PEP 564 (bpo-31784, #3989) broke that for 3.7.x (and master/3.8).  I've attached a WIP PR that at the moment just turns Alexander's test into a potential test case.

Since this problem has been around since 3.7.0, I am lowering the priority to "deferred blocker" to not hold up 3.7.3rc1.

Victor, I'd appreciate it if you could take a look at this. Thanks!
msg337758 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-03-12 15:44
Sorry, I don't understand the problem.

Can someone please give the result of these commands on Python 3.6 and 3.7 on macOS?

Example on Linux (Fedora 29):

$ python3
>>> import platform, time
>>> platform.platform()
'Linux-4.20.13-200.fc29.x86_64-x86_64-with-fedora-29-Twenty_Nine'
>>> for clock in ('monotonic', 'perf_counter', 'process_time', 'thread_time', 'time'): print(f'clock: {time.get_clock_info(clock)}')
... 
clock: namespace(adjustable=False, implementation='clock_gettime(CLOCK_MONOTONIC)', monotonic=True, resolution=1e-09)
clock: namespace(adjustable=False, implementation='clock_gettime(CLOCK_MONOTONIC)', monotonic=True, resolution=1e-09)
clock: namespace(adjustable=False, implementation='clock_gettime(CLOCK_PROCESS_CPUTIME_ID)', monotonic=True, resolution=1e-09)
clock: namespace(adjustable=False, implementation='clock_gettime(CLOCK_THREAD_CPUTIME_ID)', monotonic=True, resolution=1e-09)
clock: namespace(adjustable=True, implementation='clock_gettime(CLOCK_REALTIME)', monotonic=False, resolution=1e-09)
msg337759 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-03-12 15:45
Ah, or maybe use test.pythoninfo:

$ ./python -m test.pythoninfo|grep -E '^(platform|time)'
platform.architecture: 64bit ELF
platform.libc_ver: glibc 2.28
platform.platform: Linux-4.20.13-200.fc29.x86_64-x86_64-with-glibc2.28
platform.python_implementation: CPython
time.altzone: -7200
time.daylight: 1
time.get_clock_info(clock): namespace(adjustable=False, implementation='clock()', monotonic=True, resolution=1e-06)
time.get_clock_info(monotonic): namespace(adjustable=False, implementation='clock_gettime(CLOCK_MONOTONIC)', monotonic=True, resolution=1e-09)
time.get_clock_info(perf_counter): namespace(adjustable=False, implementation='clock_gettime(CLOCK_MONOTONIC)', monotonic=True, resolution=1e-09)
time.get_clock_info(process_time): namespace(adjustable=False, implementation='clock_gettime(CLOCK_PROCESS_CPUTIME_ID)', monotonic=True, resolution=1e-09)
time.get_clock_info(thread_time): namespace(adjustable=False, implementation='clock_gettime(CLOCK_THREAD_CPUTIME_ID)', monotonic=True, resolution=1e-09)
time.get_clock_info(time): namespace(adjustable=True, implementation='clock_gettime(CLOCK_REALTIME)', monotonic=False, resolution=1e-09)
time.time: 1552405487.361025
time.timezone: -3600
time.tzname: ('CET', 'CEST')
msg337774 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2019-03-12 16:37
*******************
tip of master built on current macOS 10.14 - correct results:

3.8.0a2+ (heads/master:f45813df52, Mar 12 2019, 12:25:58)
[Clang 10.0.0 (clang-1000.11.45.5)]
macOS-10.14.3-x86_64-i386-64bit
monotonic: namespace(adjustable=False, implementation='mach_absolute_time()', monotonic=True, resolution=1e-09)
perf_counter: namespace(adjustable=False, implementation='mach_absolute_time()', monotonic=True, resolution=1e-09)
process_time: namespace(adjustable=False, implementation='clock_gettime(CLOCK_PROCESS_CPUTIME_ID)', monotonic=True, resolution=1.0000000000000002e-06)
thread_time: namespace(adjustable=False, implementation='clock_gettime(CLOCK_THREAD_CPUTIME_ID)', monotonic=True, resolution=1e-09)
time: namespace(adjustable=True, implementation='clock_gettime(CLOCK_REALTIME)', monotonic=False, resolution=1.0000000000000002e-06)
*******************

*******************
v3.8.0a2 python.org installer built on macOS 10.9 - INCORRECT results:

3.8.0a2 (v3.8.0a2:23f4589b4b, Feb 25 2019, 10:59:08)
[Clang 6.0 (clang-600.0.57)]
macOS-10.14.3-x86_64-i386-64bit
monotonic: namespace(adjustable=False, implementation='mach_absolute_time()', monotonic=True, resolution=1e-09)
perf_counter: namespace(adjustable=False, implementation='mach_absolute_time()', monotonic=True, resolution=1e-09)
process_time: namespace(adjustable=False, implementation='getrusage(RUSAGE_SELF)', monotonic=True, resolution=1e-06)
thread_time: not available
time: namespace(adjustable=True, implementation='gettimeofday()', monotonic=False, resolution=1e-06)
*******************

*******************
v3.6.8 python.org installer built on macOS 10.9 - correct results:

3.6.8 (v3.6.8:3c6b436a57, Dec 24 2018, 02:04:31)
[GCC 4.2.1 Compatible Apple LLVM 6.0 (clang-600.0.57)]
Darwin-18.2.0-x86_64-i386-64bit
monotonic: namespace(adjustable=False, implementation='mach_absolute_time()', monotonic=True, resolution=1e-09)
perf_counter: namespace(adjustable=False, implementation='mach_absolute_time()', monotonic=True, resolution=1e-09)
process_time: namespace(adjustable=False, implementation='getrusage(RUSAGE_SELF)', monotonic=True, resolution=1e-06)
thread_time: not available
time: namespace(adjustable=True, implementation='gettimeofday()', monotonic=False, resolution=1e-06)
*******************
msg338751 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2019-03-24 20:24
@vstinner, ping
msg338818 - (view) Author: Łukasz Langa (lukasz.langa) * (Python committer) Date: 2019-03-25 18:45
Looks like this will have to be broken for 3.8.0a3, too. I will mark this as a release blocker for a4 though.
History
Date User Action Args
2019-03-25 18:45:29lukasz.langasetnosy: + lukasz.langa
messages: + msg338818
2019-03-24 20:24:50ned.deilysetmessages: + msg338751
2019-03-12 16:37:14ned.deilysetmessages: + msg337774
2019-03-12 15:45:03vstinnersetmessages: + msg337759
2019-03-12 15:44:27vstinnersetmessages: + msg337758
2019-03-12 09:49:53ned.deilysetpriority: release blocker -> deferred blocker

nosy: - lukasz.langa
messages: + msg337731

stage: patch review -> needs patch
2019-03-12 09:40:11ned.deilysetkeywords: + patch
stage: needs patch -> patch review
pull_requests: + pull_request12267
2019-03-06 04:50:49ned.deilysetpriority: normal -> release blocker

components: + Interpreter Core, - Build, Installation
title: Python 3.7 and 3.8 process_time is not reported correctly (twice then expected) -> Python 3.7 and 3.8 process_time is not reported correctly when built on older macOS versions
nosy: + vstinner, lukasz.langa

messages: + msg337277
stage: needs patch
2019-03-06 02:23:46Nitapolcreate