This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

Author njs
Recipients M J Harvey, Matt Harvey, Matthew Rocklin, davin, njs, pitrou
Date 2018-03-07.22:26:16
SpamBayes Score -1.0
Marked as misclassified Yes
Message-id <1520461576.42.0.467229070634.issue32986@psf.upfronthosting.co.za>
In-reply-to
Content
That stackoverflow thread points to the GNU coreutils 'nproc', which is an interesting compendium of knowledge about this problem.

It looks like their algorithm is roughly:

1. Determine how many CPUs *could* this program access, by going down a list of possible options and using the first that works: pthread_getaffinity_np -> sched_getaffinity -> GetProcessAffinityMask -> sysconf(_SC_NUMPROCESSORS_ONLN) -> some arcane stuff specific to HP-UX, IRIX, etc.

2. Parse the OMP_NUM_THREADS and OMP_THREAD_LIMIT envvars (this is not quite trivial, there's some handling of whitespace and commas and references to the OMP spec)

3. If OMP_NUM_THREADS is set, return min(OMP_NUM_THREADS, OMP_THREAD_LIMIT). Otherwise, return min(available_processors, OMP_THREAD_LIMIT).

Step (1) handles both the old affinity APIs, and also the cpuset system that docker uses. (From cpuset(7): "Cpusets are integrated with the sched_setaffinity(2) scheduling  affinity  mechanism".) Step (2) relies on the quasi-standard OMP_* envvars to let you choose something explicitly.

The PBS Pro docs say that they set both NPROCS and OMP_NUM_THREADS. See section 6.1.7 of https://pbsworks.com/pdfs/PBSUserGuide14.2.pdf

So this seems like a reasonable heuristic approach to me.
History
Date User Action Args
2018-03-07 22:26:16njssetrecipients: + njs, pitrou, davin, Matthew Rocklin, Matt Harvey, M J Harvey
2018-03-07 22:26:16njssetmessageid: <1520461576.42.0.467229070634.issue32986@psf.upfronthosting.co.za>
2018-03-07 22:26:16njslinkissue32986 messages
2018-03-07 22:26:16njscreate