classification
Title: distutils doesn't parallelize extension module compilation
Type: enhancement Stage: patch review
Components: Build Versions: Python 3.5
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: Arfrever, Sjlver, bfroehle, christian.heimes, eric.araujo, pitrou, scoder, tarek
Priority: low Keywords: gsoc, patch

Created on 2009-02-19 01:39 by pitrou, last changed 2014-09-09 16:40 by Sjlver.

Files
File name Uploaded Description Edit
build_ext_parallel.patch pitrou, 2013-10-26 20:08 review
build_ext_parallel2.patch pitrou, 2013-10-26 21:09 review
build_ext_parallel3.patch christian.heimes, 2013-10-26 22:50 review
build_ext_parallel4.patch Sjlver, 2014-09-05 09:48 review
build_ext_parallel5.patch pitrou, 2014-09-06 23:16 review
Messages (14)
msg82447 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2009-02-19 01:39
When run with "make -jN", the Python compilation process is able to
parallelize across N concurrent processes for the interpreter core, but
extension modules are still compiled sequentially.
msg116603 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2010-09-16 20:24
Do you want to work on a patch?
msg178993 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2013-01-03 22:27
The original request is really about setup.py, not packaging. I don't care about packaging, and it's not in the stdlib.
msg179010 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2013-01-04 10:01
+1

I think it's sufficient to parallelize the compilation step (.c -> .o) and ignore the linker step. Linking is usually so fast that it doesn't matter.

Idea:
* separate compile step from link step
* run all compile calls for all extensions in parallel until all objects of all extensions are generated
* do linking and error reporting in serial mode
msg201392 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2013-10-26 20:08
Here is a working patch.
msg201396 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2013-10-26 21:09
Here is an updated patch which only enables parallel building in setup.py.
msg201400 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2013-10-26 22:50
Antoine's patch doesn't work for a fresh copy of Python

- add import error report to build_ext parallel (distutils doesn't use logging and doesn't have a working log.exception() function)
- make time a builtin module
- add hack for concurrent.futures.process
- add --parallel=i / -ji (jobs) argument to build_ext
msg226412 - (view) Author: Jonas Wagner (Sjlver) * Date: 2014-09-05 09:48
Is there a reason this has not landed? The patch works perfectly for me, except for one issue:

    @@ -268,6 +275,9 @@
             if self.undef:
                 self.undef = self.undef.split(',')
     
    +        if self.parallel:
    +            self.parallel = int(self.parallel)
    +
             if self.swig_opts is None:
                 self.swig_opts = []
             else:

If self.parallel is True, this will set self.parallel to 1, causing it to use one worker instead of n (where n is the number of processors).

An updated patch is attached.
msg226415 - (view) Author: Stefan Behnel (scoder) * Date: 2014-09-05 09:54
Yes, please. The sequential build in distutils is very annoying.

(too late for Py3.4, though, I guess...)
msg226424 - (view) Author: Jonas Wagner (Sjlver) * Date: 2014-09-05 13:59
With this patch, and on Ubuntu 14.04, occasionally modules fail to build with the following error:

*** WARNING: renaming "_testbuffer" since importing it failed: dlopen: cannot load any more object with static TLS

I'm not 100% sure if this is really due to the patch... but I've never seen it before. I'll try investigate a bit more.
msg226521 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2014-09-06 23:16
Updated patch:
- add the -j option to the "build" command as well (even though it's only used by build_ext)
- in setup.py, only force parallel compilation if make was called with -j
msg226523 - (view) Author: Arfrever Frehtes Taifersar Arahesis (Arfrever) * Date: 2014-09-06 23:37
- Is support for infinite jobs planned?
  (GNU make supports it for -j without number argument.)

- self.parallel = int(self.parallel) would raise ValueError when a non-number is passed. I suggest to print user-friendly error message.
msg226546 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2014-09-07 21:23
> self.parallel = int(self.parallel) would raise ValueError when a
> non-number is passed. I suggest to print user-friendly error message.

The distutils idiom would be to catch the TypeError/ValueError and raise DistutilsOptionError.  Higher layers convert that to a message.
msg226646 - (view) Author: Jonas Wagner (Sjlver) * Date: 2014-09-09 16:40
I've checked the `dlopen` issue. This was due to a problem with UndefinedBehaviorSanitizer, and was solved by upgrading to Clang 3.5. It has little to do with this patch.

Using this patch and http://bugs.python.org/issue22359 , I now get reliable parallel builds.
History
Date User Action Args
2014-09-09 16:40:16Sjlversetmessages: + msg226646
2014-09-07 21:23:42eric.araujosetmessages: + msg226546
2014-09-06 23:37:08Arfreversetmessages: + msg226523
2014-09-06 23:23:45pitrousetstage: needs patch -> patch review
2014-09-06 23:16:48pitrousetfiles: + build_ext_parallel5.patch

messages: + msg226521
2014-09-05 13:59:52Sjlversetmessages: + msg226424
2014-09-05 09:54:23scodersetnosy: + scoder

messages: + msg226415
versions: + Python 3.5, - Python 3.4
2014-09-05 09:48:25Sjlversetfiles: + build_ext_parallel4.patch

messages: + msg226412
2014-09-05 09:25:55Sjlversetnosy: + Sjlver
2013-10-26 22:50:54christian.heimessetfiles: + build_ext_parallel3.patch

messages: + msg201400
2013-10-26 21:09:23pitrousetfiles: + build_ext_parallel2.patch

messages: + msg201396
2013-10-26 20:08:06pitrousetfiles: + build_ext_parallel.patch
keywords: + patch
messages: + msg201392
2013-10-25 22:34:37Arfreversetnosy: + Arfrever

title: setup.py doesn't parallelize extension module compilation -> distutils doesn't parallelize extension module compilation
2013-01-04 10:01:17christian.heimessetnosy: + christian.heimes
messages: + msg179010
2013-01-03 22:27:08pitrousetversions: + Python 3.4, - 3rd party
title: packaging doesn't parallelize extension module compilation -> setup.py doesn't parallelize extension module compilation
messages: + msg178993

assignee: tarek ->
components: + Build, - Distutils2
2013-01-03 21:24:53bfroehlesetnosy: + bfroehle
2011-04-08 15:11:56eric.araujosettitle: setup.py doesn't parallelize extension module compilation -> packaging doesn't parallelize extension module compilation
2011-03-21 22:26:30eric.araujosetkeywords: + gsoc
nosy: pitrou, tarek, eric.araujo
2010-09-29 23:51:32eric.araujosetversions: + 3rd party, - Python 2.6, Python 2.5, Python 3.1, Python 2.7, Python 3.2
2010-09-16 20:24:33eric.araujosetstage: needs patch
messages: + msg116603
versions: + Python 2.6, Python 2.5, Python 3.1, Python 2.7
2010-09-16 16:18:40BreamoreBoysetnosy: + eric.araujo

components: + Distutils2, - Distutils
versions: + Python 3.2, - Python 3.1, Python 2.7
2009-02-19 01:39:51pitroucreate