classification
Title: On Linux, os.count() should read cgroup cpu.shares and cpu.cfs (CPU count inside docker container)
Type: performance Stage:
Components: Interpreter Core, Library (Lib) Versions: Python 3.8, Python 3.7
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: Manjusaka, christian.heimes, galen, giampaolo.rodola, jab, matrixise, mcnelsonphd, methane, nargit, vstinner
Priority: normal Keywords:

Created on 2019-02-20 16:56 by keirlawson, last changed 2020-04-20 05:47 by nargit.

Messages (22)
msg336117 - (view) Author: Keir Lawson (keirlawson) Date: 2019-02-20 16:56
There appears to be no way to detect the number of CPUs allotted to a Python program within a docker container.  With the following script:

import os

print("os.cpu_count(): " + str(os.cpu_count()))
print("len(os.sched_getaffinity(0)): " + str(len(os.sched_getaffinity(0))))

when run in a container (from an Ubuntu 18.04 host) I get:

docker run -v "$PWD":/src/ -w /src/ --cpus=1 python:3.7 python detect_cpus.py
os.cpu_count(): 4
len(os.sched_getaffinity(0)): 4

Recent vesions of Java are able to correctly detect the CPU allocation:

docker run -it --cpus 1 openjdk:10-jdk
Feb 20, 2019 4:20:29 PM java.util.prefs.FileSystemPreferences$1 run
INFO: Created user preferences directory.
|  Welcome to JShell -- Version 10.0.2
|  For an introduction type: /help intro

jshell> Runtime.getRuntime().availableProcessors()
$1 ==> 1
msg336126 - (view) Author: Stéphane Wirtel (matrixise) * (Python committer) Date: 2019-02-20 17:12
I would like to work on this issue.
msg336146 - (view) Author: Stéphane Wirtel (matrixise) * (Python committer) Date: 2019-02-20 20:15
so, I have also tested with the last docker image of golang.

> docker run --rm --cpus 1  -it golang /bin/bash

here is the code of golang:

package main

import "fmt"
import "runtime"

func main() {
	cores := runtime.NumCPU()
	fmt.Printf("This machine has %d CPU cores.\n", cores)
}

Here is the output
> ./demo 
This machine has 4 CPU cores.

When I try with grep on /proc/cpuinfo
I get this result

> grep processor /proc/cpuinfo  -c
4


I will test with openjdk because it's related to Java and see if I can get the result of 1
msg336148 - (view) Author: Stéphane Wirtel (matrixise) * (Python committer) Date: 2019-02-20 20:18
ok, I didn't see your test with openjdk:10, sorry
msg336149 - (view) Author: Keir Lawson (keirlawson) Date: 2019-02-20 20:24
I believe this is related to this ticket: https://bugs.python.org/issue26692

Looking at Java's implementation it seems like they are checking if cgroups are enabled via /proc/self/cgroup and then if it is parsing the cgroup information out of the filesystem.
msg338724 - (view) Author: Stéphane Wirtel (matrixise) * (Python committer) Date: 2019-03-24 06:23
I am really sorry but I thought to work on this issue but it's not the case. Feel free to submit a PR.
msg338778 - (view) Author: Manjusaka (Manjusaka) * Date: 2019-03-25 02:47
I think that I may work on a PR for this issue. Is there anybody has worked on it ?
msg338780 - (view) Author: Stéphane Wirtel (matrixise) * (Python committer) Date: 2019-03-25 05:08
Hi Manjusaka,

Could you explain your solution, because I have read the code of
openjdk, (C++) and I am going to be honnest, it was not complex but not
really clear.

Also, if you need my help for the review or for the construction of your
PR, I can help you.

Have a nice day,

Stéphane
msg338783 - (view) Author: Manjusaka (Manjusaka) * Date: 2019-03-25 05:38
Hi Stéphane

Thanks a lot!

In my opinion, I would like to make an independent library that name is cgroups. For ease of use and compatibility, I think it's better than combining code with the os module.

Thanks for you working!

Manjusaka
msg339401 - (view) Author: Manjusaka (Manjusaka) * Date: 2019-04-03 16:48
Hi Stéphane:

I have checked the JVM implantation about container improvements. I confirm that maybe we need a new Libary for container environment. I don't think that combine it into the os module is a good idea. I will make a PR during this week.
msg339404 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2019-04-03 17:05
The JVM parses cgroups information from the proc filesystem and evaluates CPU count from the cgroup cpu.shares and cpu.cfs.

https://github.com/openjdk/jdk/blob/d5686b87f31d6c57ec6b3e5e9c85a04209dbac7a/src/hotspot/os/linux/os_linux.cpp#L5304-L5336

https://github.com/openjdk/jdk/blob/2d5137e403e16b694800b2ffe18c3640396b757e/src/hotspot/os/linux/osContainer_linux.cpp#L517-L591
msg339429 - (view) Author: Manjusaka (Manjusaka) * Date: 2019-04-04 03:43
Yes, not only but also support get real memory limit.

look at https://blogs.oracle.com/java-platform-group/java-se-support-for-docker-cpu-and-memory-limits
msg339439 - (view) Author: Stéphane Wirtel (matrixise) * (Python committer) Date: 2019-04-04 09:37
>Yes, not only but also support get real memory limit.
>
>look at https://blogs.oracle.com/java-platform-group/java-se-support-for-docker-cpu-and-memory-limits

Yep, but in this case, you have to create an other issue for the memory
limit.
msg353690 - (view) Author: Mike (mcnelsonphd) Date: 2019-10-01 12:46
Is this issue still being worked on as a core feature? I needed a solution for this using 2.7.11 to enable some old code to work properly/nicely in a container environment on AWS Batch and was forced to figure out what OpenJDK was doing and came up with a solution. The process in OpenJDK seems to be, find where the cgroups for docker are located in the file system, then depending on the values in different files you can determine the number of CPUs available. 

The inelegant code below is what worked for me:

def query_cpu():
	if os.path.isfile('/sys/fs/cgroup/cpu/cpu.cfs_quota_us'):
		cpu_quota = int(open('/sys/fs/cgroup/cpu/cpu.cfs_quota_us').read().rstrip())
		#print(cpu_quota) # Not useful for AWS Batch based jobs as result is -1, but works on local linux systems
	if cpu_quota != -1 and os.path.isfile('/sys/fs/cgroup/cpu/cpu.cfs_period_us'):
		cpu_period = int(open('/sys/fs/cgroup/cpu/cpu.cfs_period_us').read().rstrip())
		#print(cpu_period)
		avail_cpu = int(cpu_quota / cpu_period) # Divide quota by period and you should get num of allotted CPU to the container, rounded down if fractional.
	elif os.path.isfile('/sys/fs/cgroup/cpu/cpu.shares'):
		cpu_shares = int(open('/sys/fs/cgroup/cpu/cpu.shares').read().rstrip())
		#print(cpu_shares) # For AWS, gives correct value * 1024.
		avail_cpu = int(cpu_shares / 1024)
	return avail_cpu


This solution makes several assumptions about the cgroup locations within the container vs dynamically finding where those files are located as OpenJDK does. I also haven't included the more robust method in case cpu.quota and cpu.shares are -1.

Hopefully this is a start for getting this implemented.
msg364894 - (view) Author: Manjusaka (Manjusaka) * Date: 2020-03-23 20:09
Hello Mike, thanks for your code. 

I think it's a good way 

I think if  cpu.quota and cpu.shares are -1, just return the original value in os.cpu_count() is OK
msg364898 - (view) Author: Manjusaka (Manjusaka) * Date: 2020-03-23 20:18
I will make a PR in this week
msg364901 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-03-23 21:18
I'm not sure that it's a good idea to change os.cpucount(). I suggest to add a new function instead.

os.cpu_count() iss documented as:
"Return the number of CPUs in the system."
https://docs.python.org/dev/library/os.html#os.cpu_count

By the way, the documentation adds:

"This number is not equivalent to the number of CPUs the current process can use. The number of usable CPUs can be obtained with len(os.sched_getaffinity(0))"
msg364902 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2020-03-23 21:28
I suggest that your provide a low solution that returns general information from cgroup v1 and unified cgroup v2 rather than a solution that focuses on CPU only. Then you can provide a high level interface that returns effective CPU cores.

cgroup v2 (unified hierarchy) has been around for 6 years and slowly gains traction as container platforms start to support them.
msg364916 - (view) Author: Manjusaka (Manjusaka) * Date: 2020-03-24 04:03
Actually, we already have some third party libs to support cgroup. But most of them get these questions

1. They are not std lib

2. They are just support cgroup1

But if we want to add a new std lib. Should we create a PEP?
msg365071 - (view) Author: Manjusaka (Manjusaka) * Date: 2020-03-26 13:34
Hello guys, I some ideas about this issue

First, maybe we should add a new API named cpu_usable_count(). I think it's more meaningful than the os.sched_getaffinity(0)

Second, more and more people use docker to run their app today. So people need an official way to get the environment info, not just cpu, but the memory,  the network traffic limit. Because the docker is based on the CGroup in Linux, maybe we can add a cgroup lib as an official supported lib.

but I'm not sure about this idea, because there are some problem.

1. the CGroup is only supported on Linux. I'm not sure that adding a platform-specific lib is a good idea

2. Many languages are not adding cgroup official yet. Maybe there are some languages are optimized for the cgroup (such as Java in JVM)
msg365075 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-03-26 13:53
> Hello guys,

Please try to find a more inclusive way to say hello: https://heyguys.cc/ ;-)
msg366811 - (view) Author: Tigran Tch. (nargit) Date: 2020-04-20 05:47
Do we have any news about this?
History
Date User Action Args
2020-04-20 05:47:26nargitsetnosy: + nargit
messages: + msg366811
2020-03-26 13:53:43vstinnersetmessages: + msg365075
2020-03-26 13:34:48Manjusakasetnosy: + giampaolo.rodola
messages: + msg365071
2020-03-24 04:03:08Manjusakasetmessages: + msg364916
2020-03-23 21:28:47christian.heimessetmessages: + msg364902
2020-03-23 21:21:46keirlawsonsetnosy: - keirlawson
2020-03-23 21:18:09vstinnersetmessages: + msg364901
2020-03-23 20:18:52Manjusakasetmessages: + msg364898
2020-03-23 20:09:47Manjusakasetnosy: + vstinner
messages: + msg364894
2019-12-12 06:10:40methanesetnosy: + methane
2019-11-21 00:26:57galensetnosy: + galen
2019-10-01 12:51:30vstinnersettitle: Way to detect CPU count inside docker container -> On Linux, os.count() should read cgroup cpu.shares and cpu.cfs (CPU count inside docker container)
2019-10-01 12:46:23mcnelsonphdsetnosy: + mcnelsonphd
messages: + msg353690
2019-04-18 22:57:38jabsetnosy: + jab
2019-04-04 09:37:14matrixisesetmessages: + msg339439
2019-04-04 03:43:22Manjusakasetmessages: + msg339429
2019-04-03 17:05:14christian.heimessetnosy: + christian.heimes
messages: + msg339404
2019-04-03 16:48:52Manjusakasetmessages: + msg339401
2019-03-25 05:38:46Manjusakasetmessages: + msg338783
2019-03-25 05:08:49matrixisesetmessages: + msg338780
2019-03-25 02:47:56Manjusakasetnosy: + Manjusaka
messages: + msg338778
2019-03-24 06:23:07matrixisesetmessages: + msg338724
components: + Interpreter Core
versions: + Python 3.8
2019-02-20 20:24:27keirlawsonsetmessages: + msg336149
2019-02-20 20:18:18matrixisesetmessages: + msg336148
2019-02-20 20:15:06matrixisesetmessages: + msg336146
2019-02-20 17:12:51matrixisesetnosy: + matrixise
messages: + msg336126
2019-02-20 16:56:22keirlawsoncreate