classification
Title: On Linux, os.count() should read cgroup cpu.shares and cpu.cfs (CPU count inside docker container)
Type: performance Stage:
Components: Interpreter Core, Library (Lib) Versions: Python 3.8, Python 3.7
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: Manjusaka, christian.heimes, galen, jab, keirlawson, matrixise, mcnelsonphd
Priority: normal Keywords:

Created on 2019-02-20 16:56 by keirlawson, last changed 2019-11-21 00:26 by galen.

Messages (14)
msg336117 - (view) Author: Keir Lawson (keirlawson) Date: 2019-02-20 16:56
There appears to be no way to detect the number of CPUs allotted to a Python program within a docker container.  With the following script:

import os

print("os.cpu_count(): " + str(os.cpu_count()))
print("len(os.sched_getaffinity(0)): " + str(len(os.sched_getaffinity(0))))

when run in a container (from an Ubuntu 18.04 host) I get:

docker run -v "$PWD":/src/ -w /src/ --cpus=1 python:3.7 python detect_cpus.py
os.cpu_count(): 4
len(os.sched_getaffinity(0)): 4

Recent vesions of Java are able to correctly detect the CPU allocation:

docker run -it --cpus 1 openjdk:10-jdk
Feb 20, 2019 4:20:29 PM java.util.prefs.FileSystemPreferences$1 run
INFO: Created user preferences directory.
|  Welcome to JShell -- Version 10.0.2
|  For an introduction type: /help intro

jshell> Runtime.getRuntime().availableProcessors()
$1 ==> 1
msg336126 - (view) Author: Stéphane Wirtel (matrixise) * (Python committer) Date: 2019-02-20 17:12
I would like to work on this issue.
msg336146 - (view) Author: Stéphane Wirtel (matrixise) * (Python committer) Date: 2019-02-20 20:15
so, I have also tested with the last docker image of golang.

> docker run --rm --cpus 1  -it golang /bin/bash

here is the code of golang:

package main

import "fmt"
import "runtime"

func main() {
	cores := runtime.NumCPU()
	fmt.Printf("This machine has %d CPU cores.\n", cores)
}

Here is the output
> ./demo 
This machine has 4 CPU cores.

When I try with grep on /proc/cpuinfo
I get this result

> grep processor /proc/cpuinfo  -c
4


I will test with openjdk because it's related to Java and see if I can get the result of 1
msg336148 - (view) Author: Stéphane Wirtel (matrixise) * (Python committer) Date: 2019-02-20 20:18
ok, I didn't see your test with openjdk:10, sorry
msg336149 - (view) Author: Keir Lawson (keirlawson) Date: 2019-02-20 20:24
I believe this is related to this ticket: https://bugs.python.org/issue26692

Looking at Java's implementation it seems like they are checking if cgroups are enabled via /proc/self/cgroup and then if it is parsing the cgroup information out of the filesystem.
msg338724 - (view) Author: Stéphane Wirtel (matrixise) * (Python committer) Date: 2019-03-24 06:23
I am really sorry but I thought to work on this issue but it's not the case. Feel free to submit a PR.
msg338778 - (view) Author: Manjusaka (Manjusaka) * Date: 2019-03-25 02:47
I think that I may work on a PR for this issue. Is there anybody has worked on it ?
msg338780 - (view) Author: Stéphane Wirtel (matrixise) * (Python committer) Date: 2019-03-25 05:08
Hi Manjusaka,

Could you explain your solution, because I have read the code of
openjdk, (C++) and I am going to be honnest, it was not complex but not
really clear.

Also, if you need my help for the review or for the construction of your
PR, I can help you.

Have a nice day,

Stéphane
msg338783 - (view) Author: Manjusaka (Manjusaka) * Date: 2019-03-25 05:38
Hi Stéphane

Thanks a lot!

In my opinion, I would like to make an independent library that name is cgroups. For ease of use and compatibility, I think it's better than combining code with the os module.

Thanks for you working!

Manjusaka
msg339401 - (view) Author: Manjusaka (Manjusaka) * Date: 2019-04-03 16:48
Hi Stéphane:

I have checked the JVM implantation about container improvements. I confirm that maybe we need a new Libary for container environment. I don't think that combine it into the os module is a good idea. I will make a PR during this week.
msg339404 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2019-04-03 17:05
The JVM parses cgroups information from the proc filesystem and evaluates CPU count from the cgroup cpu.shares and cpu.cfs.

https://github.com/openjdk/jdk/blob/d5686b87f31d6c57ec6b3e5e9c85a04209dbac7a/src/hotspot/os/linux/os_linux.cpp#L5304-L5336

https://github.com/openjdk/jdk/blob/2d5137e403e16b694800b2ffe18c3640396b757e/src/hotspot/os/linux/osContainer_linux.cpp#L517-L591
msg339429 - (view) Author: Manjusaka (Manjusaka) * Date: 2019-04-04 03:43
Yes, not only but also support get real memory limit.

look at https://blogs.oracle.com/java-platform-group/java-se-support-for-docker-cpu-and-memory-limits
msg339439 - (view) Author: Stéphane Wirtel (matrixise) * (Python committer) Date: 2019-04-04 09:37
>Yes, not only but also support get real memory limit.
>
>look at https://blogs.oracle.com/java-platform-group/java-se-support-for-docker-cpu-and-memory-limits

Yep, but in this case, you have to create an other issue for the memory
limit.
msg353690 - (view) Author: Mike (mcnelsonphd) Date: 2019-10-01 12:46
Is this issue still being worked on as a core feature? I needed a solution for this using 2.7.11 to enable some old code to work properly/nicely in a container environment on AWS Batch and was forced to figure out what OpenJDK was doing and came up with a solution. The process in OpenJDK seems to be, find where the cgroups for docker are located in the file system, then depending on the values in different files you can determine the number of CPUs available. 

The inelegant code below is what worked for me:

def query_cpu():
	if os.path.isfile('/sys/fs/cgroup/cpu/cpu.cfs_quota_us'):
		cpu_quota = int(open('/sys/fs/cgroup/cpu/cpu.cfs_quota_us').read().rstrip())
		#print(cpu_quota) # Not useful for AWS Batch based jobs as result is -1, but works on local linux systems
	if cpu_quota != -1 and os.path.isfile('/sys/fs/cgroup/cpu/cpu.cfs_period_us'):
		cpu_period = int(open('/sys/fs/cgroup/cpu/cpu.cfs_period_us').read().rstrip())
		#print(cpu_period)
		avail_cpu = int(cpu_quota / cpu_period) # Divide quota by period and you should get num of allotted CPU to the container, rounded down if fractional.
	elif os.path.isfile('/sys/fs/cgroup/cpu/cpu.shares'):
		cpu_shares = int(open('/sys/fs/cgroup/cpu/cpu.shares').read().rstrip())
		#print(cpu_shares) # For AWS, gives correct value * 1024.
		avail_cpu = int(cpu_shares / 1024)
	return avail_cpu


This solution makes several assumptions about the cgroup locations within the container vs dynamically finding where those files are located as OpenJDK does. I also haven't included the more robust method in case cpu.quota and cpu.shares are -1.

Hopefully this is a start for getting this implemented.
History
Date User Action Args
2019-11-21 00:26:57galensetnosy: + galen
2019-10-01 12:51:30vstinnersettitle: Way to detect CPU count inside docker container -> On Linux, os.count() should read cgroup cpu.shares and cpu.cfs (CPU count inside docker container)
2019-10-01 12:46:23mcnelsonphdsetnosy: + mcnelsonphd
messages: + msg353690
2019-04-18 22:57:38jabsetnosy: + jab
2019-04-04 09:37:14matrixisesetmessages: + msg339439
2019-04-04 03:43:22Manjusakasetmessages: + msg339429
2019-04-03 17:05:14christian.heimessetnosy: + christian.heimes
messages: + msg339404
2019-04-03 16:48:52Manjusakasetmessages: + msg339401
2019-03-25 05:38:46Manjusakasetmessages: + msg338783
2019-03-25 05:08:49matrixisesetmessages: + msg338780
2019-03-25 02:47:56Manjusakasetnosy: + Manjusaka
messages: + msg338778
2019-03-24 06:23:07matrixisesetmessages: + msg338724
components: + Interpreter Core
versions: + Python 3.8
2019-02-20 20:24:27keirlawsonsetmessages: + msg336149
2019-02-20 20:18:18matrixisesetmessages: + msg336148
2019-02-20 20:15:06matrixisesetmessages: + msg336146
2019-02-20 17:12:51matrixisesetnosy: + matrixise
messages: + msg336126
2019-02-20 16:56:22keirlawsoncreate