This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

Author M J Harvey
Recipients M J Harvey
Date 2018-03-02.17:51:29
SpamBayes Score -1.0
Marked as misclassified Yes
Message-id <1520013089.33.0.467229070634.issue32986@psf.upfronthosting.co.za>
In-reply-to
Content
Hi,

multiprocessing's default assumption about Pool size is os.cpu_count() ie all the cores visible to the OS.

This is tremendously unhelpful when running multiprocessing code inside an HPC batch system (PBS Pro in my case), as there's no way to hint to the code that the # of cpus actually allocated to it may be fewer.

It's quite tedious to have to explain this to every single person trying to use it.

Proposal: multiprocessing should look for a hint for default Pool size from the environment variable "NCPUS" which most batch systems set. If that's not set, or its value is invalid, fall back to os.cpu_count() as before
History
Date User Action Args
2018-03-02 17:51:29M J Harveysetrecipients: + M J Harvey
2018-03-02 17:51:29M J Harveysetmessageid: <1520013089.33.0.467229070634.issue32986@psf.upfronthosting.co.za>
2018-03-02 17:51:29M J Harveylinkissue32986 messages
2018-03-02 17:51:29M J Harveycreate