os.cpu_count() returns wrong number of processors on specific systems #77347

yanirh · 2018-03-28T09:44:43Z

BPO	33166
Nosy	@pfmoore, @pitrou, @giampaolo, @tjguk, @zware, @eryksun, @zooba, @csabella, @miss-islington
PRs	bpo-33166: Change os.cpu_count to return active (real) processors #15949 [3.8] bpo-33166: Change os.cpu_count to return active (real) processors (GH-15949) #15979
Files	Screenshot.png

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2019-09-11.15:20:09.367>
created_at = <Date 2018-03-28.09:44:42.722>
labels = ['type-bug', '3.8', '3.9', 'OS-windows']
title = 'os.cpu_count() returns wrong number of processors on specific systems'
updated_at = <Date 2019-09-11.15:56:16.043>
user = 'https://bugs.python.org/yanirh'

bugs.python.org fields:

activity = <Date 2019-09-11.15:56:16.043>
actor = 'miss-islington'
assignee = 'none'
closed = True
closed_date = <Date 2019-09-11.15:20:09.367>
closer = 'zach.ware'
components = ['Windows']
creation = <Date 2018-03-28.09:44:42.722>
creator = 'yanirh'
dependencies = []
files = ['47504']
hgrepos = []
issue_num = 33166
keywords = ['patch']
message_count = 20.0
messages = ['314580', '314581', '314583', '314585', '314587', '314589', '314590', '314591', '314592', '314596', '314597', '314599', '314609', '314620', '314631', '314664', '334274', '351931', '351936', '351956']
nosy_count = 10.0
nosy_names = ['paul.moore', 'pitrou', 'giampaolo.rodola', 'tim.golden', 'zach.ware', 'eryksun', 'steve.dower', 'cheryl.sabella', 'miss-islington', 'yanirh']
pr_nums = ['15949', '15979']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue33166'
versions = ['Python 3.8', 'Python 3.9']

yanirh · 2018-03-28T10:00:50Z

wrong number of cpu's is reported on some specific platforms.

first platform:
server with X4 Intel® Xeon® E5-4620 (8 physical, 16 logical), running
a 64bit Windows Server 2012 R2 Standard.
results:
os.cpu_count() reports 64 units
psutil.cpu_count(logical=False) reports 32 units
psutil.cpu_count(logical=True) reports 64 units

multiprocessing using concurrent.futures able to fully utilize the server;

second platform:
server with X2 Intel® Xeon® Gold 6138 (20 physical, 40 logical), running a 64bit Windows Server 2016 Standard.
results:
os.cpu_count() reports 128 units
psutil.cpu_count(logical=False) reports 20 units
psutil.cpu_count(logical=True) reports 40 units

multiprocessing using concurrent.futures able to utilize only 1/4 of the server's power;

yanirh · 2018-03-28T10:01:52Z

wrong number of cpu's is reported on some specific platforms.

first platform:
server with X4 Intel Xeon E5-4620 (8 physical, 16 logical), running
a 64bit Windows Server 2012 R2 Standard.
results:
os.cpu_count() reports 64 units
psutil.cpu_count(logical=False) reports 32 units
psutil.cpu_count(logical=True) reports 64 units

multiprocessing using concurrent.futures able to fully utilize the server;

second platform:
server with X2 Intel Xeon Gold 6138 (20 physical, 40 logical), running a 64bit Windows Server 2016 Standard.
results:
os.cpu_count() reports 128 units
psutil.cpu_count(logical=False) reports 20 units
psutil.cpu_count(logical=True) reports 40 units

multiprocessing using concurrent.futures able to utilize only 1/4 of the server's power;

pitrou · 2018-03-28T11:18:45Z

os.cpu_count() reports 128 units
psutil.cpu_count(logical=False) reports 20 units
psutil.cpu_count(logical=True) reports 40 units

You mean os.cpu_count() reports *more* CPUs than exist on the machine? How can that happen?

yanirh · 2018-03-28T12:19:32Z

Yup.
Attaching a screenshot.

giampaolo · 2018-03-28T13:09:05Z

The difference between os.cpu_count() and psutil.cpu_count() is because one uses GetMaximumProcessorCount() and the other dwNumberOfProcessors.

This is tracked as a bug in psutil bug tracker but it's not fixed yet:
giampaolo/psutil#771

As for Python this is where it was discussed and changed:
https://bugs.python.org/issue30581
c67bae0

In summary: psutil is wrong and you should rely on os.cpu_count().

yanirh · 2018-03-28T13:33:34Z

Maybe i'm missing something, and would appreciate clarification.

Perhaps psutil is wrong, but it gives an answer that has something to do with the actual situation.

On platform 2, i have 2 Intel Xeon Gold 6138, each with 20 physical processors, 40 logicals.

you are saying i need to rely on os.cpu_count(), which outputs '128'. Can you elaborate on this?

Moreover, when attempting to parallelize on the processors, i reach 25% utilization, which suggests Python 'sees' only one processor group.

giampaolo · 2018-03-28T13:37:22Z

Oh! So both os.cpu_count() and psutil.cpu_count() are wrong? How do you determine the actual number of logical/physical CPUs on your machine?

yanirh · 2018-03-28T14:06:57Z

Yes. Both are wrong, and os.cpu_count() is completely off.
Regarding how to determine the number of physical/logical cores in my machine - well, not sure what you mean by that. I've attached a screenshot of Windows' system information. Also used 'cpu-z'.

It seems like aside from the os.cpu_count() issue, Python itself has some problem - it 'sees' only 1 CPU group. It is evident from the fact the when parallelizing, utilization level is only 25%.

pitrou · 2018-03-28T14:12:01Z

Let's not conflate different issues. The parallelization issue is distinct from the os.cpu_count() issue (and I'm skeptical Python is at fault there).

yanirh · 2018-03-28T14:33:37Z

Ok, no problem.
Just to be sure i'm doing the right thing - this thread will be dedicated to the os.cpu_count() issue, and i'll open a new issue on the parallelization problem.

makes sense?

pitrou · 2018-03-28T14:34:12Z

Just to be sure i'm doing the right thing - this thread will be dedicated to the os.cpu_count() issue, and i'll open a new issue on the parallelization problem.

Exactly.

giampaolo · 2018-03-28T14:53:13Z

By re-reading
https://bugs.python.org/issue30581 and
giampaolo/psutil#771 (comment) I now remember why I haven't fixed this issue in psutil yet: because the whole thing (MS APIs and doc basically) is confusing.

GetMaximumProcessorCount (now used by os.cpu_count()) returns "the maximum number of logical processors that a processor group or the system CAN have", not the actual number. That would explain why in OP's case os.cpu_count() returns 128 instead of 40.

As per https://bugs.python.org/issue30581#msg295255 dwNumberOfProcessors wasn't good because it doesn't take multiple processor groups into account (hence the number may be too small) and GetLogicalProcessorInformationEx may be the way to go. This is based on the assumption that os.cpu_count() should report the number of CPUs in the system (including the non-usable ones, like in case of process groups).

pitrou · 2018-03-28T15:38:43Z

Adding Chris Wilcox who wrote the original patch using GetMaximumProcessorCount.

giampaolo · 2018-03-28T21:04:21Z

I created a psutil branch using GetLogicalProcessorInformation() to determine both logical and physical CPUs:
giampaolo/psutil#1257
According to https://stackoverflow.com/questions/31209256 basically all Windows APIs are unreliable and GetLogicalProcessorInformationEx() is what should really be used.
That is available only starting from Windows 7 though so apparently what we want is GetLogicalProcessorInformationEx() and if not available fallback on GetActiveProcessorCount(ALL_PROCESSOR_GROUPS) which may still report the wrong number of CPUs on 32 bit processes.

eryksun · 2018-03-29T03:56:20Z

if not available fallback on
GetActiveProcessorCount(ALL_PROCESSOR_GROUPS)

The fallback for older versions of Windows is dwNumberOfProcessors from GetSystemInfo. This can be removed from 3.7 and 3.8, which no longer support Windows versions prior to Windows 7.

GetLogicalProcessorInformationEx() is what should really be used.

GetActiveProcessorCount and GetMaximumProcessorCount are implemented via GetLogicalProcessorInformationEx (i.e. NtQuerySystemInformation, SystemLogicalProcessorAndGroupInformation). They query the RelationGroup information. For ALL_PROCESSOR_GROUPS, they respectively sum the ActiveProcessorCount and MaximumProcessorCount over all groups.

These functions were added in Windows 7 to support the implementation of logical processor groups, which allows up to 64 logical processors per group. Each process is created in a single group, which is assigned round-robin. A thread can call SetThreadGroupAffinity to manually switch to another group.

Apparently someone at Microsoft advised calling GetMaximumProcessorCount (see bpo-30581), but I don't follow this decision. Why should os.cpu_count() include CPUs that may or may not come online? Also, on POSIX it reports sysconf(_SC_NPROCESSORS_ONLN), not sysconf(_SC_NPROCESSORS_CONF), so for Windows it should instead call GetActiveProcessorCount. I assume on BSD that HW_NCPU is similar, though I'm not sure for MacOS. Also on MacOS it appears to be deprecated in favor of HW_LOGICALCPU, HW_LOGICALCPU_MAX, HW_PHYSICALCPU, and HW_PHYSICALCPU_MAX.

which may still report the wrong number of CPUs on 32 bit processes.

32-bit Windows and WOW64 emulation are limited to 32 CPUs. Applications that need more logical processors should be 64-bit

giampaolo · 2018-03-29T18:03:19Z

That makes sense to me. Thanks for deciphering this.

csabella · 2019-01-24T00:06:13Z

Added bpo-32592 as a dependency since that is removing the Vista code mentioned here. Once that change is merged, then this would be a simpler change to make.

zware · 2019-09-11T15:15:42Z

New changeset aa92927 by Zachary Ware (Steve Dower) in branch 'master':
bpo-33166: Change os.cpu_count to return active (real) processors (GH-15949)
aa92927

zware · 2019-09-11T15:20:09Z

Thank you for clarifying this muddy topic, Eryk!

(Dropping bpo-32592 dependency; we've done the update in a basically-compatible way.)

miss-islington · 2019-09-11T15:56:16Z

New changeset 43ee0e2 by Miss Islington (bot) in branch '3.8':
bpo-33166: Change os.cpu_count to return active (real) processors (GH-15949)
43ee0e2

yanirh mannequin added OS-windows type-bug An unexpected behavior, bug, or error labels Mar 28, 2018

pitrou added 3.7 (EOL) end of life 3.8 only security fixes labels Mar 28, 2018

zooba added 3.9 only security fixes and removed 3.7 (EOL) end of life labels Sep 11, 2019

zware closed this as completed Sep 11, 2019

ezio-melotti transferred this issue from another repository Apr 10, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

os.cpu_count() returns wrong number of processors on specific systems #77347

os.cpu_count() returns wrong number of processors on specific systems #77347

yanirh mannequin commented Mar 28, 2018

yanirh mannequin commented Mar 28, 2018

yanirh mannequin commented Mar 28, 2018

pitrou commented Mar 28, 2018

yanirh mannequin commented Mar 28, 2018

giampaolo commented Mar 28, 2018

yanirh mannequin commented Mar 28, 2018

giampaolo commented Mar 28, 2018

yanirh mannequin commented Mar 28, 2018

pitrou commented Mar 28, 2018

yanirh mannequin commented Mar 28, 2018

pitrou commented Mar 28, 2018

giampaolo commented Mar 28, 2018

pitrou commented Mar 28, 2018

giampaolo commented Mar 28, 2018

eryksun commented Mar 29, 2018

giampaolo commented Mar 29, 2018

csabella commented Jan 24, 2019

zware commented Sep 11, 2019

zware commented Sep 11, 2019

miss-islington commented Sep 11, 2019

os.cpu_count() returns wrong number of processors on specific systems #77347

os.cpu_count() returns wrong number of processors on specific systems #77347

Comments

yanirh mannequin commented Mar 28, 2018

yanirh mannequin commented Mar 28, 2018

yanirh mannequin commented Mar 28, 2018

pitrou commented Mar 28, 2018

yanirh mannequin commented Mar 28, 2018

giampaolo commented Mar 28, 2018

yanirh mannequin commented Mar 28, 2018

giampaolo commented Mar 28, 2018

yanirh mannequin commented Mar 28, 2018

pitrou commented Mar 28, 2018

yanirh mannequin commented Mar 28, 2018

pitrou commented Mar 28, 2018

giampaolo commented Mar 28, 2018

pitrou commented Mar 28, 2018

giampaolo commented Mar 28, 2018

eryksun commented Mar 29, 2018

giampaolo commented Mar 29, 2018

csabella commented Jan 24, 2019

zware commented Sep 11, 2019

zware commented Sep 11, 2019

miss-islington commented Sep 11, 2019