Author neologix
Recipients Satrajit Ghosh, jnoller, neologix, pitrou, sbt
Date 2017-07-23.07:56:18
SpamBayes Score -1.0
Marked as misclassified Yes
Message-id <CAH_1eM186pqoQhvVpK+ZJDoUt4DwE-d9Th0Cvi9m=Wx6kfcfeQ@mail.gmail.com>
In-reply-to <1500760793.24.0.182939763232.issue26692@psf.upfronthosting.co.za>
Content
I'm not convinced.
The reason is that using the number of CPU cores is just a heuristic
for a *default value*: the API allows the user to specify the number
of workers to use, so it's not really a limitation.

The problem is that if you try to think about a more "correct" default
value, it gets complicated: here, it's about cgroups, but for example:
- What if they are multiple processes running on the same box?
- What if the process is subject to CPU affinity? Currently, the CPU
affinity mask is ignored.
- What if the code being executed by children is itself multi-threaded
(maybe because it's using a numerical library using BLAS etc)?
- What about hyper-threading? If the code has a lot of cache misses,
it would probably be a good idea to use one worker per logical thread,
but if it's cache-friendly, probably not.
- Etc.

In other words, I think that there's simply not reasonable default
value for the number of workers to use, that any value will make some
class of users/use-case unhappy, and it would add a lot of unnecessary
complexity.

Since the user can always specify the number of workers - if you find
a place where it's not possible, then please report it - I really
think we should let the choice/burden up to the user.
History
Date User Action Args
2017-07-23 07:56:19neologixsetrecipients: + neologix, pitrou, jnoller, sbt, Satrajit Ghosh
2017-07-23 07:56:19neologixlinkissue26692 messages
2017-07-23 07:56:18neologixcreate