classification
Title: distutils doesn't parallelize extension module compilation
Type: enhancement Stage: needs patch
Components: Build Versions: Python 3.4
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: Arfrever, bfroehle, christian.heimes, eric.araujo, pitrou, tarek
Priority: low Keywords: gsoc, patch

Created on 2009-02-19 01:39 by pitrou, last changed 2013-10-26 22:50 by christian.heimes.

Files
File name Uploaded Description Edit
build_ext_parallel.patch pitrou, 2013-10-26 20:08 review
build_ext_parallel2.patch pitrou, 2013-10-26 21:09 review
build_ext_parallel3.patch christian.heimes, 2013-10-26 22:50 review
Messages (7)
msg82447 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2009-02-19 01:39
When run with "make -jN", the Python compilation process is able to
parallelize across N concurrent processes for the interpreter core, but
extension modules are still compiled sequentially.
msg116603 - (view) Author: √Čric Araujo (eric.araujo) * (Python committer) Date: 2010-09-16 20:24
Do you want to work on a patch?
msg178993 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2013-01-03 22:27
The original request is really about setup.py, not packaging. I don't care about packaging, and it's not in the stdlib.
msg179010 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2013-01-04 10:01
+1

I think it's sufficient to parallelize the compilation step (.c -> .o) and ignore the linker step. Linking is usually so fast that it doesn't matter.

Idea:
* separate compile step from link step
* run all compile calls for all extensions in parallel until all objects of all extensions are generated
* do linking and error reporting in serial mode
msg201392 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2013-10-26 20:08
Here is a working patch.
msg201396 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2013-10-26 21:09
Here is an updated patch which only enables parallel building in setup.py.
msg201400 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2013-10-26 22:50
Antoine's patch doesn't work for a fresh copy of Python

- add import error report to build_ext parallel (distutils doesn't use logging and doesn't have a working log.exception() function)
- make time a builtin module
- add hack for concurrent.futures.process
- add --parallel=i / -ji (jobs) argument to build_ext
History
Date User Action Args
2013-10-26 22:50:54christian.heimessetfiles: + build_ext_parallel3.patch

messages: + msg201400
2013-10-26 21:09:23pitrousetfiles: + build_ext_parallel2.patch

messages: + msg201396
2013-10-26 20:08:06pitrousetfiles: + build_ext_parallel.patch
keywords: + patch
messages: + msg201392
2013-10-25 22:34:37Arfreversetnosy: + Arfrever

title: setup.py doesn't parallelize extension module compilation -> distutils doesn't parallelize extension module compilation
2013-01-04 10:01:17christian.heimessetnosy: + christian.heimes
messages: + msg179010
2013-01-03 22:27:08pitrousetversions: + Python 3.4, - 3rd party
title: packaging doesn't parallelize extension module compilation -> setup.py doesn't parallelize extension module compilation
messages: + msg178993

assignee: tarek ->
components: + Build, - Distutils2
2013-01-03 21:24:53bfroehlesetnosy: + bfroehle
2011-04-08 15:11:56eric.araujosettitle: setup.py doesn't parallelize extension module compilation -> packaging doesn't parallelize extension module compilation
2011-03-21 22:26:30eric.araujosetkeywords: + gsoc
nosy: pitrou, tarek, eric.araujo
2010-09-29 23:51:32eric.araujosetversions: + 3rd party, - Python 2.6, Python 2.5, Python 3.1, Python 2.7, Python 3.2
2010-09-16 20:24:33eric.araujosetstage: needs patch
messages: + msg116603
versions: + Python 2.6, Python 2.5, Python 3.1, Python 2.7
2010-09-16 16:18:40BreamoreBoysetnosy: + eric.araujo

components: + Distutils2, - Distutils
versions: + Python 3.2, - Python 3.1, Python 2.7
2009-02-19 01:39:51pitroucreate