classification
Title: os.cpu_count() returns wrong number of processors on system with > 64 logical processors
Type: behavior Stage: resolved
Components: Windows Versions: Python 3.7, Python 3.6
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: crwilcox, giampaolo.rodola, paul.moore, pitrou, robbuckley, steve.dower, tim.golden, tzickel, vstinner, zach.ware
Priority: normal Keywords:

Created on 2017-06-06 12:49 by robbuckley, last changed 2017-12-01 09:45 by pitrou. This issue is now closed.

Files
File name Uploaded Description Edit
ci.txt robbuckley, 2017-06-06 12:56
Pull Requests
URL Status Linked Edit
PR 2934 merged python-dev, 2017-07-28 16:45
PR 3267 merged crwilcox, 2017-09-01 19:05
Messages (18)
msg295254 - (view) Author: (robbuckley) Date: 2017-06-06 12:49
os.cpu_count() seems to report incorrect values on windows systems with >64 logical processors

tried it on 2 similar systems, both running windows 7 / 10 with python 3.6.1 64bit (anaconda):

platform1 - 2x Xeon E5-2698v4. 20 cores/CPU = total 80 logical cpus with hyperthreading
platform2 - 2x Xeon E5-2697v3. 14 cores/CPU = total 56 logical cpus with hyperthreading

os.cpu_count() reports 40 cores on platform1 and 56 on platform2

I would expect 80 and 56 respectively. 

I suppose this is because the windows api call used is not aware of processor groups, and reports only the number of processors in the current processor group ( eg GetSystemInfo vs GetMaximumProcessorCount )
msg295255 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-06-06 12:53
On Windows, os.cpu_count() is currently implemented with:

"GetSystemInfo(&sysinfo); return sysinfo.dwNumberOfProcessors;"

https://msdn.microsoft.com/en-us/library/windows/desktop/ms724958(v=vs.85).aspx

It seems to return the number of *logical* CPUs:

"""
dwNumberOfProcessors

    The number of logical processors in the current group.

Note: For information about the physical processors shared by logical processors, call GetLogicalProcessorInformationEx with the RelationshipType parameter set to RelationProcessorPackage (3).
"""

It seems like you have two physical CPU packages. Maybe the function only returns infos from the first package?
msg295258 - (view) Author: (robbuckley) Date: 2017-06-06 12:56
yes, i believe its reporting the number of processors in the current group only, not across all groups.

attached output of windows sysinternals/coreinfo showing 2 processor groups

see https://github.com/giampaolo/psutil/issues/771 for some further disucssion of this topic

the maintainer of psutil asked me to raise this bug, also had a quick check on #python IRC. Its my first bug on bugs.python.org so if you need more info just let me know
msg295365 - (view) Author: Chris Wilcox (crwilcox) * Date: 2017-06-07 19:41
I am going to work on this if no one else has started.
msg295377 - (view) Author: Giampaolo Rodola' (giampaolo.rodola) * (Python committer) Date: 2017-06-07 22:27
Nobody has AFAIK.
msg299462 - (view) Author: Giampaolo Rodola' (giampaolo.rodola) * (Python committer) Date: 2017-07-29 10:21
MS documentation is not clear on what function should be used as there are many returning different values. Here it is being suggested to use GetLogicalProcessorInformationEx: 
https://stackoverflow.com/questions/31209256/reliable-way-to-programmatically-get-the-number-of-cores-on-windows
msg299476 - (view) Author: Chris Wilcox (crwilcox) * Date: 2017-07-29 16:57
I agree that the MS Docs for this are a bit confusing. I ended up reaching out to the guy who authored the GetMaximumProcessorCount function. I had also written an implementation that iterated over GetProcessorInformationEx and he advised against it. 

One of the things that makes this interesting is that in 32 bit processes (wow64) your processor is simulated to fit in the confines of that old system. This method will only report 32 cores under 32 bit as that is all the program can access in 32 bit mode.
msg299477 - (view) Author: Giampaolo Rodola' (giampaolo.rodola) * (Python committer) Date: 2017-07-29 16:57
About GetMaximumProcessorCount, MS doc states that it returns the "maximum number of logical processors that a processor group or the system can have", so maybe it also includes "empty" CPU sockets.

GetActiveProcessorCount, on the other hand, returns "the number of active processors in a processor group or in the system", which adds even more confusion.
msg299480 - (view) Author: Chris Wilcox (crwilcox) * Date: 2017-07-29 17:15
I was reviewing the docs for the os module and cpu_count should always return the number of cpus on the system, not the usable CPUs. GetMaximumProcessorCount returns a simulated count in WoW64. I have reached back out to the Windows API dev and will see if GetLogicalProcessorInformationEx will allow us to do this. He had thought that my solution that way had other limitations under WoW64.
msg301017 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2017-08-30 09:01
New changeset c67bae04780f9d7590f9f91b4ee5f31c5d75b3c3 by Antoine Pitrou (Christopher Wilcox) in branch 'master':
bpo-30581: Windows: os.cpu_count() returns wrong number of processors (#2934)
https://github.com/python/cpython/commit/c67bae04780f9d7590f9f91b4ee5f31c5d75b3c3
msg301018 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2017-08-30 09:01
Fixed.  Someone might backport this to 3.6 if they want.
msg301033 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-08-30 17:41
I reopen the issue to backport the bugfix to 3.6.
msg301146 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2017-09-01 19:28
New changeset 58521fdba1657f6553a1ead5cbaa100967a167b3 by Antoine Pitrou (Christopher Wilcox) in branch '3.6':
bpo-30581: Windows: os.cpu_count() returns wrong number of processors (#2934) (#3267)
https://github.com/python/cpython/commit/58521fdba1657f6553a1ead5cbaa100967a167b3
msg301147 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2017-09-01 19:29
Backport merged. Thanks Chris!
msg301150 - (view) Author: (tzickel) Date: 2017-09-01 19:52
One should be careful with this modification because of the Windows definition of process groups.

For example, if multi-threaded code thinks that by reading the value of the new os.cpu_count() it can use all the cores returned, by default it cannot as in windows processes by default can run only in a single process group (how it worked before).

We can see such code builtin python stdlib itself:
https://github.com/python/cpython/blob/bc61315377056fe362b744d9c44e17cd3178ce54/Lib/concurrent/futures/thread.py#L102

I think even .NET still uses the old way that python did until now:
https://github.com/dotnet/corefx/blob/aaaffdf7b8330846f6832f43700fbcc060460c9f/src/System.Runtime.Extensions/src/System/Environment.Windows.cs#L71

Although some of this stuff is used in code for python multiprocess code which that might actually get a boost (since different process can get scheduled to different groups)

https://msdn.microsoft.com/en-us/library/windows/desktop/dd405503(v=vs.85).aspx
msg301151 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2017-09-01 20:08
os.cpu_count() is specified to return the total number of processors, not the number of usable processors.  See e.g. https://bugs.python.org/issue26692
msg307374 - (view) Author: (robbuckley) Date: 2017-12-01 09:43
hi, 
as the reporter i just want to say this is working for me with 3.6.3. 

Regarding https://bugs.python.org/issue30581#msg301150, I take your point that a lot of multiprocessing using the standard libraries may not benefit, as processes may be restricted to the processor group of the parent process (python). 

For my use case it works well: I launch a queue of blocking jobs, using a thread pool. Each thread launches 1 jobsubprocess.subprocess.run(), where the thread pool size is equal to number of processors reported by os.cpu_count(). Since the OS controls the scheduling in this case, it works perfectly well with 2 processor groups. 

thanks :-)
msg307376 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2017-12-01 09:45
Thanks for the heads up Rob!
History
Date User Action Args
2017-12-01 09:45:38pitrousetmessages: + msg307376
2017-12-01 09:43:34robbuckleysetmessages: + msg307374
2017-09-01 20:08:32pitrousetmessages: + msg301151
2017-09-01 19:52:24tzickelsetnosy: + tzickel
messages: + msg301150
2017-09-01 19:29:13pitrousetstatus: open -> closed
resolution: fixed
messages: + msg301147
2017-09-01 19:28:49pitrousetmessages: + msg301146
2017-09-01 19:05:55crwilcoxsetpull_requests: + pull_request3312
2017-08-30 17:41:36vstinnersetstatus: closed -> open
resolution: fixed -> (no value)
messages: + msg301033
2017-08-30 09:01:56pitrousetstatus: open -> closed
versions: - Python 3.5
messages: + msg301018

resolution: fixed
stage: patch review -> resolved
2017-08-30 09:01:13pitrousetnosy: + pitrou
messages: + msg301017
2017-07-29 17:15:27crwilcoxsetmessages: + msg299480
2017-07-29 16:57:08giampaolo.rodolasetmessages: + msg299477
2017-07-29 16:57:08crwilcoxsetmessages: + msg299476
2017-07-29 10:21:42giampaolo.rodolasetmessages: + msg299462
2017-07-29 10:12:41pitrousetstage: needs patch -> patch review
2017-07-28 16:45:13python-devsetpull_requests: + pull_request2986
2017-06-29 14:18:25pitrousetstage: needs patch
versions: + Python 3.5, Python 3.7
2017-06-07 22:27:40giampaolo.rodolasetmessages: + msg295377
2017-06-07 19:41:41crwilcoxsetnosy: + crwilcox
messages: + msg295365
2017-06-06 16:10:04giampaolo.rodolasetnosy: + giampaolo.rodola
2017-06-06 12:56:00robbuckleysetfiles: + ci.txt

messages: + msg295258
2017-06-06 12:53:23vstinnersetnosy: + vstinner
messages: + msg295255
2017-06-06 12:49:39robbuckleycreate