This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: Extensions build does not respect --jobs setting
Type: resource usage Stage:
Components: Build Versions: Python 3.9
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: JustAnotherArchivist, ncoghlan, petr.viktorin
Priority: normal Keywords:

Created on 2021-03-26 18:58 by JustAnotherArchivist, last changed 2022-04-11 14:59 by admin.

Messages (1)
msg389558 - (view) Author: (JustAnotherArchivist) * Date: 2021-03-26 18:58
The extension building does not respect the --jobs option passed to make. Specifically, in that step, `python setup.py build` always spawns as many gcc processes as there are CPU cores available regardless of that option. This caused problems for me because I have a VM that sees all host machine CPU cores but only has a limited amount of RAM. Despite running `make -j 4`, many more gcc processes are spawned, and this immediately causes memory starvation and a system freeze after a few seconds.

The reason for this is that setup.py blindly enables parallelism in the extension compilation if '-j' appears in the MAKEFLAGS at 3.9.2/setup.py:355. Later on, distutils uses os.cpu_count to set the worker count, i.e. the '-j' *value* is ignored. This behaviour was first introduced with #5309 as far as I can see, though I haven't tested anything other than version 3.9.2.

Hacky workaround: patching the above setup.py line to `self.parallel = 4`.

Cf. https://github.com/pyenv/pyenv/issues/1857
History
Date User Action Args
2022-04-11 14:59:43adminsetgithub: 87800
2021-04-03 01:20:46terry.reedysetnosy: + ncoghlan, petr.viktorin
2021-03-26 18:58:30JustAnotherArchivistcreate