classification
Title: Way to detect CPU count inside docker container
Type: performance Stage:
Components: Interpreter Core, Library (Lib) Versions: Python 3.8, Python 3.7
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: Manjusaka, christian.heimes, jab, keirlawson, matrixise
Priority: normal Keywords:

Created on 2019-02-20 16:56 by keirlawson, last changed 2019-04-18 22:57 by jab.

Messages (13)
msg336117 - (view) Author: Keir Lawson (keirlawson) Date: 2019-02-20 16:56
There appears to be no way to detect the number of CPUs allotted to a Python program within a docker container.  With the following script:

import os

print("os.cpu_count(): " + str(os.cpu_count()))
print("len(os.sched_getaffinity(0)): " + str(len(os.sched_getaffinity(0))))

when run in a container (from an Ubuntu 18.04 host) I get:

docker run -v "$PWD":/src/ -w /src/ --cpus=1 python:3.7 python detect_cpus.py
os.cpu_count(): 4
len(os.sched_getaffinity(0)): 4

Recent vesions of Java are able to correctly detect the CPU allocation:

docker run -it --cpus 1 openjdk:10-jdk
Feb 20, 2019 4:20:29 PM java.util.prefs.FileSystemPreferences$1 run
INFO: Created user preferences directory.
|  Welcome to JShell -- Version 10.0.2
|  For an introduction type: /help intro

jshell> Runtime.getRuntime().availableProcessors()
$1 ==> 1
msg336126 - (view) Author: Stéphane Wirtel (matrixise) * (Python committer) Date: 2019-02-20 17:12
I would like to work on this issue.
msg336146 - (view) Author: Stéphane Wirtel (matrixise) * (Python committer) Date: 2019-02-20 20:15
so, I have also tested with the last docker image of golang.

> docker run --rm --cpus 1  -it golang /bin/bash

here is the code of golang:

package main

import "fmt"
import "runtime"

func main() {
	cores := runtime.NumCPU()
	fmt.Printf("This machine has %d CPU cores.\n", cores)
}

Here is the output
> ./demo 
This machine has 4 CPU cores.

When I try with grep on /proc/cpuinfo
I get this result

> grep processor /proc/cpuinfo  -c
4


I will test with openjdk because it's related to Java and see if I can get the result of 1
msg336148 - (view) Author: Stéphane Wirtel (matrixise) * (Python committer) Date: 2019-02-20 20:18
ok, I didn't see your test with openjdk:10, sorry
msg336149 - (view) Author: Keir Lawson (keirlawson) Date: 2019-02-20 20:24
I believe this is related to this ticket: https://bugs.python.org/issue26692

Looking at Java's implementation it seems like they are checking if cgroups are enabled via /proc/self/cgroup and then if it is parsing the cgroup information out of the filesystem.
msg338724 - (view) Author: Stéphane Wirtel (matrixise) * (Python committer) Date: 2019-03-24 06:23
I am really sorry but I thought to work on this issue but it's not the case. Feel free to submit a PR.
msg338778 - (view) Author: Manjusaka (Manjusaka) * Date: 2019-03-25 02:47
I think that I may work on a PR for this issue. Is there anybody has worked on it ?
msg338780 - (view) Author: Stéphane Wirtel (matrixise) * (Python committer) Date: 2019-03-25 05:08
Hi Manjusaka,

Could you explain your solution, because I have read the code of
openjdk, (C++) and I am going to be honnest, it was not complex but not
really clear.

Also, if you need my help for the review or for the construction of your
PR, I can help you.

Have a nice day,

Stéphane
msg338783 - (view) Author: Manjusaka (Manjusaka) * Date: 2019-03-25 05:38
Hi Stéphane

Thanks a lot!

In my opinion, I would like to make an independent library that name is cgroups. For ease of use and compatibility, I think it's better than combining code with the os module.

Thanks for you working!

Manjusaka
msg339401 - (view) Author: Manjusaka (Manjusaka) * Date: 2019-04-03 16:48
Hi Stéphane:

I have checked the JVM implantation about container improvements. I confirm that maybe we need a new Libary for container environment. I don't think that combine it into the os module is a good idea. I will make a PR during this week.
msg339404 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2019-04-03 17:05
The JVM parses cgroups information from the proc filesystem and evaluates CPU count from the cgroup cpu.shares and cpu.cfs.

https://github.com/openjdk/jdk/blob/d5686b87f31d6c57ec6b3e5e9c85a04209dbac7a/src/hotspot/os/linux/os_linux.cpp#L5304-L5336

https://github.com/openjdk/jdk/blob/2d5137e403e16b694800b2ffe18c3640396b757e/src/hotspot/os/linux/osContainer_linux.cpp#L517-L591
msg339429 - (view) Author: Manjusaka (Manjusaka) * Date: 2019-04-04 03:43
Yes, not only but also support get real memory limit.

look at https://blogs.oracle.com/java-platform-group/java-se-support-for-docker-cpu-and-memory-limits
msg339439 - (view) Author: Stéphane Wirtel (matrixise) * (Python committer) Date: 2019-04-04 09:37
>Yes, not only but also support get real memory limit.
>
>look at https://blogs.oracle.com/java-platform-group/java-se-support-for-docker-cpu-and-memory-limits

Yep, but in this case, you have to create an other issue for the memory
limit.
History
Date User Action Args
2019-04-18 22:57:38jabsetnosy: + jab
2019-04-04 09:37:14matrixisesetmessages: + msg339439
2019-04-04 03:43:22Manjusakasetmessages: + msg339429
2019-04-03 17:05:14christian.heimessetnosy: + christian.heimes
messages: + msg339404
2019-04-03 16:48:52Manjusakasetmessages: + msg339401
2019-03-25 05:38:46Manjusakasetmessages: + msg338783
2019-03-25 05:08:49matrixisesetmessages: + msg338780
2019-03-25 02:47:56Manjusakasetnosy: + Manjusaka
messages: + msg338778
2019-03-24 06:23:07matrixisesetmessages: + msg338724
components: + Interpreter Core
versions: + Python 3.8
2019-02-20 20:24:27keirlawsonsetmessages: + msg336149
2019-02-20 20:18:18matrixisesetmessages: + msg336148
2019-02-20 20:15:06matrixisesetmessages: + msg336146
2019-02-20 17:12:51matrixisesetnosy: + matrixise
messages: + msg336126
2019-02-20 16:56:22keirlawsoncreate