classification
Title: distutils doesn't parallelize extension module compilation
Type: enhancement Stage: resolved
Components: Build Versions: Python 3.5
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: Arfrever, Sjlver, bfroehle, christian.heimes, eric.araujo, haypo, pitrou, python-dev, scoder, tarek
Priority: low Keywords: gsoc, patch

Created on 2009-02-19 01:39 by pitrou, last changed 2014-09-30 13:04 by pitrou. This issue is now closed.

Files
File name Uploaded Description Edit
build_ext_parallel.patch pitrou, 2013-10-26 20:08 review
build_ext_parallel2.patch pitrou, 2013-10-26 21:09 review
build_ext_parallel3.patch christian.heimes, 2013-10-26 22:50 review
build_ext_parallel4.patch Sjlver, 2014-09-05 09:48 review
build_ext_parallel5.patch pitrou, 2014-09-06 23:16 review
build_ext_parallel6.patch pitrou, 2014-09-20 22:26 review
Messages (25)
msg82447 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2009-02-19 01:39
When run with "make -jN", the Python compilation process is able to
parallelize across N concurrent processes for the interpreter core, but
extension modules are still compiled sequentially.
msg116603 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2010-09-16 20:24
Do you want to work on a patch?
msg178993 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2013-01-03 22:27
The original request is really about setup.py, not packaging. I don't care about packaging, and it's not in the stdlib.
msg179010 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2013-01-04 10:01
+1

I think it's sufficient to parallelize the compilation step (.c -> .o) and ignore the linker step. Linking is usually so fast that it doesn't matter.

Idea:
* separate compile step from link step
* run all compile calls for all extensions in parallel until all objects of all extensions are generated
* do linking and error reporting in serial mode
msg201392 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2013-10-26 20:08
Here is a working patch.
msg201396 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2013-10-26 21:09
Here is an updated patch which only enables parallel building in setup.py.
msg201400 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2013-10-26 22:50
Antoine's patch doesn't work for a fresh copy of Python

- add import error report to build_ext parallel (distutils doesn't use logging and doesn't have a working log.exception() function)
- make time a builtin module
- add hack for concurrent.futures.process
- add --parallel=i / -ji (jobs) argument to build_ext
msg226412 - (view) Author: Jonas Wagner (Sjlver) * Date: 2014-09-05 09:48
Is there a reason this has not landed? The patch works perfectly for me, except for one issue:

    @@ -268,6 +275,9 @@
             if self.undef:
                 self.undef = self.undef.split(',')
     
    +        if self.parallel:
    +            self.parallel = int(self.parallel)
    +
             if self.swig_opts is None:
                 self.swig_opts = []
             else:

If self.parallel is True, this will set self.parallel to 1, causing it to use one worker instead of n (where n is the number of processors).

An updated patch is attached.
msg226415 - (view) Author: Stefan Behnel (scoder) * Date: 2014-09-05 09:54
Yes, please. The sequential build in distutils is very annoying.

(too late for Py3.4, though, I guess...)
msg226424 - (view) Author: Jonas Wagner (Sjlver) * Date: 2014-09-05 13:59
With this patch, and on Ubuntu 14.04, occasionally modules fail to build with the following error:

*** WARNING: renaming "_testbuffer" since importing it failed: dlopen: cannot load any more object with static TLS

I'm not 100% sure if this is really due to the patch... but I've never seen it before. I'll try investigate a bit more.
msg226521 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2014-09-06 23:16
Updated patch:
- add the -j option to the "build" command as well (even though it's only used by build_ext)
- in setup.py, only force parallel compilation if make was called with -j
msg226523 - (view) Author: Arfrever Frehtes Taifersar Arahesis (Arfrever) * Date: 2014-09-06 23:37
- Is support for infinite jobs planned?
  (GNU make supports it for -j without number argument.)

- self.parallel = int(self.parallel) would raise ValueError when a non-number is passed. I suggest to print user-friendly error message.
msg226546 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2014-09-07 21:23
> self.parallel = int(self.parallel) would raise ValueError when a
> non-number is passed. I suggest to print user-friendly error message.

The distutils idiom would be to catch the TypeError/ValueError and raise DistutilsOptionError.  Higher layers convert that to a message.
msg226646 - (view) Author: Jonas Wagner (Sjlver) * Date: 2014-09-09 16:40
I've checked the `dlopen` issue. This was due to a problem with UndefinedBehaviorSanitizer, and was solved by upgrading to Clang 3.5. It has little to do with this patch.

Using this patch and http://bugs.python.org/issue22359 , I now get reliable parallel builds.
msg227188 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2014-09-20 22:26
Updated patch raising DistutilsOptionError. I couldn't find any docs so I didn't update them :-)
msg227651 - (view) Author: Roundup Robot (python-dev) Date: 2014-09-26 21:32
New changeset bbe57429eba0 by Antoine Pitrou in branch 'default':
Issue #5309: distutils' build and build_ext commands now accept a ``-j``
https://hg.python.org/cpython/rev/bbe57429eba0
msg227652 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2014-09-26 21:33
This is now pushed.
msg227877 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2014-09-30 10:57
It looks like compilation of Python 3.5 fails on FreeBSD 6.4 because of the changeset bbe57429eba0a9ec21fb0f1178f409f1bba44c22:

http://buildbot.python.org/all/builders/x86%20FreeBSD%206.4%203.x/builds/5061

Compile log:
---

building '_ctypes' extension
creating build/temp.freebsd-6.4-RELEASE-i386-3.5-pydebug/usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Modules/_ctypes/libffi
creating build/temp.freebsd-6.4-RELEASE-i386-3.5-pydebug/usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Modules/_ctypes/libffi/src
creating build/temp.freebsd-6.4-RELEASE-i386-3.5-pydebug/usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Modules/_ctypes/libffi/src/x86
gcc -pthread -fPIC -fno-strict-aliasing -Wsign-compare -g -O0 -Wall -Wstrict-prototypes -Ibuild/temp.freebsd-6.4-RELEASE-i386-3.5-pydebug/libffi/include -Ibuild/temp.freebsd-6.4-RELEASE-i386-3.5-pydebug/libffi -I/usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Modules/_ctypes/libffi/src -I./Include -I. -IInclude -I/usr/local/include -I/usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Include -I/usr/home/db3l/buildarea/3.x.bolen-freebsd/build -c /usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Modules/_ctypes/_ctypes.c -o build/temp.freebsd-6.4-RELEASE-i386-3.5-pydebug/usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Modules/_ctypes/_ctypes.o -Wall -fexceptions
gcc -pthread -fPIC -fno-strict-aliasing -Wsign-compare -g -O0 -Wall -Wstrict-prototypes -Ibuild/temp.freebsd-6.4-RELEASE-i386-3.5-pydebug/libffi/include -Ibuild/temp.freebsd-6.4-RELEASE-i386-3.5-pydebug/libffi -I/usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Modules/_ctypes/libffi/src -I./Include -I. -IInclude -I/usr/local/include -I/usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Include -I/usr/home/db3l/buildarea/3.x.bolen-freebsd/build -c /usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Modules/_ctypes/callbacks.c -o build/temp.freebsd-6.4-RELEASE-i386-3.5-pydebug/usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Modules/_ctypes/callbacks.o -Wall -fexceptions
gcc -pthread -fPIC -fno-strict-aliasing -Wsign-compare -g -O0 -Wall -Wstrict-prototypes -Ibuild/temp.freebsd-6.4-RELEASE-i386-3.5-pydebug/libffi/include -Ibuild/temp.freebsd-6.4-RELEASE-i386-3.5-pydebug/libffi -I/usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Modules/_ctypes/libffi/src -I./Include -I. -IInclude -I/usr/local/include -I/usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Include -I/usr/home/db3l/buildarea/3.x.bolen-freebsd/build -c /usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Modules/_ctypes/callproc.c -o build/temp.freebsd-6.4-RELEASE-i386-3.5-pydebug/usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Modules/_ctypes/callproc.o -Wall -fexceptions
/usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Modules/_ctypes/callproc.c: In function `py_dl_open':
/usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Modules/_ctypes/callproc.c:1336: warning: initialization discards qualifiers from pointer target type
gcc -pthread -fPIC -fno-strict-aliasing -Wsign-compare -g -O0 -Wall -Wstrict-prototypes -Ibuild/temp.freebsd-6.4-RELEASE-i386-3.5-pydebug/libffi/include -Ibuild/temp.freebsd-6.4-RELEASE-i386-3.5-pydebug/libffi -I/usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Modules/_ctypes/libffi/src -I./Include -I. -IInclude -I/usr/local/include -I/usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Include -I/usr/home/db3l/buildarea/3.x.bolen-freebsd/build -c /usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Modules/_ctypes/stgdict.c -o build/temp.freebsd-6.4-RELEASE-i386-3.5-pydebug/usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Modules/_ctypes/stgdict.o -Wall -fexceptions
gcc -pthread -fPIC -fno-strict-aliasing -Wsign-compare -g -O0 -Wall -Wstrict-prototypes -Ibuild/temp.freebsd-6.4-RELEASE-i386-3.5-pydebug/libffi/include -Ibuild/temp.freebsd-6.4-RELEASE-i386-3.5-pydebug/libffi -I/usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Modules/_ctypes/libffi/src -I./Include -I. -IInclude -I/usr/local/include -I/usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Include -I/usr/home/db3l/buildarea/3.x.bolen-freebsd/build -c /usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Modules/_ctypes/cfield.c -o build/temp.freebsd-6.4-RELEASE-i386-3.5-pydebug/usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Modules/_ctypes/cfield.o -Wall -fexceptions
gcc -pthread -fPIC -fno-strict-aliasing -Wsign-compare -g -O0 -Wall -Wstrict-prototypes -Ibuild/temp.freebsd-6.4-RELEASE-i386-3.5-pydebug/libffi/include -Ibuild/temp.freebsd-6.4-RELEASE-i386-3.5-pydebug/libffi -I/usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Modules/_ctypes/libffi/src -I./Include -I. -IInclude -I/usr/local/include -I/usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Include -I/usr/home/db3l/buildarea/3.x.bolen-freebsd/build -c /usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Modules/_ctypes/libffi/src/prep_cif.c -o build/temp.freebsd-6.4-RELEASE-i386-3.5-pydebug/usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Modules/_ctypes/libffi/src/prep_cif.o -Wall -fexceptions
gcc -pthread -fPIC -fno-strict-aliasing -Wsign-compare -g -O0 -Wall -Wstrict-prototypes -Ibuild/temp.freebsd-6.4-RELEASE-i386-3.5-pydebug/libffi/include -Ibuild/temp.freebsd-6.4-RELEASE-i386-3.5-pydebug/libffi -I/usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Modules/_ctypes/libffi/src -I./Include -I. -IInclude -I/usr/local/include -I/usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Include -I/usr/home/db3l/buildarea/3.x.bolen-freebsd/build -c /usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Modules/_ctypes/libffi/src/closures.c -o build/temp.freebsd-6.4-RELEASE-i386-3.5-pydebug/usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Modules/_ctypes/libffi/src/closures.o -Wall -fexceptions
gcc -pthread -fPIC -fno-strict-aliasing -Wsign-compare -g -O0 -Wall -Wstrict-prototypes -Ibuild/temp.freebsd-6.4-RELEASE-i386-3.5-pydebug/libffi/include -Ibuild/temp.freebsd-6.4-RELEASE-i386-3.5-pydebug/libffi -I/usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Modules/_ctypes/libffi/src -I./Include -I. -IInclude -I/usr/local/include -I/usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Include -I/usr/home/db3l/buildarea/3.x.bolen-freebsd/build -c /usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Modules/_ctypes/libffi/src/x86/ffi.c -o build/temp.freebsd-6.4-RELEASE-i386-3.5-pydebug/usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Modules/_ctypes/libffi/src/x86/ffi.o -Wall -fexceptions
gcc -pthread -fPIC -fno-strict-aliasing -Wsign-compare -g -O0 -Wall -Wstrict-prototypes -Ibuild/temp.freebsd-6.4-RELEASE-i386-3.5-pydebug/libffi/include -Ibuild/temp.freebsd-6.4-RELEASE-i386-3.5-pydebug/libffi -I/usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Modules/_ctypes/libffi/src -I./Include -I. -IInclude -I/usr/local/include -I/usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Include -I/usr/home/db3l/buildarea/3.x.bolen-freebsd/build -c /usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Modules/_ctypes/libffi/src/x86/freebsd.S -o build/temp.freebsd-6.4-RELEASE-i386-3.5-pydebug/usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Modules/_ctypes/libffi/src/x86/freebsd.o -Wall -fexceptions
gcc -pthread -shared build/temp.freebsd-6.4-RELEASE-i386-3.5-pydebug/usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Modules/_ctypes/_ctypes.o build/temp.freebsd-6.4-RELEASE-i386-3.5-pydebug/usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Modules/_ctypes/callbacks.o build/temp.freebsd-6.4-RELEASE-i386-3.5-pydebug/usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Modules/_ctypes/callproc.o build/temp.freebsd-6.4-RELEASE-i386-3.5-pydebug/usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Modules/_ctypes/stgdict.o build/temp.freebsd-6.4-RELEASE-i386-3.5-pydebug/usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Modules/_ctypes/cfield.o build/temp.freebsd-6.4-RELEASE-i386-3.5-pydebug/usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Modules/_ctypes/libffi/src/prep_cif.o build/temp.freebsd-6.4-RELEASE-i386-3.5-pydebug/usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Modules/_ctypes/libffi/src/closures.o build/temp.freebsd-6.4-RELEASE-i386-3.5-pydebug/usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Modules/_ctypes/libffi/src/x86/ffi.o build/temp.freebsd-6.4-RELEASE-i386-3.5-pydebug/usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Modules/_ctypes/libffi/src/x86/freebsd.o -L/usr/local/lib -o build/lib.freebsd-6.4-RELEASE-i386-3.5-pydebug/_ctypes.so
*** WARNING: renaming "_ssl" since importing it failed: build/lib.freebsd-6.4-RELEASE-i386-3.5-pydebug/_ssl.so: Undefined symbol "X509_check_ca"
*** WARNING: renaming "_ctypes" since importing it failed: build/lib.freebsd-6.4-RELEASE-i386-3.5-pydebug/_ctypes.so: Undefined symbol "ffi_call_win32"
error: [Errno 2] No such file or directory: 'build/temp.freebsd-6.4-RELEASE-i386-3.5-pydebug/usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Modules/_ctypes/_ctypes.o'
*** Error code 1
---

Before:
---
building '_ctypes' extension
gcc -pthread -fPIC -fno-strict-aliasing -Wsign-compare -g -O0 -Wall -Wstrict-prototypes -Ibuild/temp.freebsd-6.4-RELEASE-i386-3.5-pydebug/libffi/include -Ibuild/temp.freebsd-6.4-RELEASE-i386-3.5-pydebug/libffi -I/usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Modules/_ctypes/libffi/src -I./Include -I. -IInclude -I/usr/local/include -I/usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Include -I/usr/home/db3l/buildarea/3.x.bolen-freebsd/build -c /usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Modules/_ctypes/_ctypes.c -o build/temp.freebsd-6.4-RELEASE-i386-3.5-pydebug/usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Modules/_ctypes/_ctypes.o -Wall -fexceptions
gcc -pthread -fPIC -fno-strict-aliasing -Wsign-compare -g -O0 -Wall -Wstrict-prototypes -Ibuild/temp.freebsd-6.4-RELEASE-i386-3.5-pydebug/libffi/include -Ibuild/temp.freebsd-6.4-RELEASE-i386-3.5-pydebug/libffi -I/usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Modules/_ctypes/libffi/src -I./Include -I. -IInclude -I/usr/local/include -I/usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Include -I/usr/home/db3l/buildarea/3.x.bolen-freebsd/build -c /usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Modules/_ctypes/callbacks.c -o build/temp.freebsd-6.4-RELEASE-i386-3.5-pydebug/usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Modules/_ctypes/callbacks.o -Wall -fexceptions
gcc -pthread -fPIC -fno-strict-aliasing -Wsign-compare -g -O0 -Wall -Wstrict-prototypes -Ibuild/temp.freebsd-6.4-RELEASE-i386-3.5-pydebug/libffi/include -Ibuild/temp.freebsd-6.4-RELEASE-i386-3.5-pydebug/libffi -I/usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Modules/_ctypes/libffi/src -I./Include -I. -IInclude -I/usr/local/include -I/usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Include -I/usr/home/db3l/buildarea/3.x.bolen-freebsd/build -c /usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Modules/_ctypes/callproc.c -o build/temp.freebsd-6.4-RELEASE-i386-3.5-pydebug/usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Modules/_ctypes/callproc.o -Wall -fexceptions
/usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Modules/_ctypes/callproc.c: In function `py_dl_open':
/usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Modules/_ctypes/callproc.c:1336: warning: initialization discards qualifiers from pointer target type
gcc -pthread -fPIC -fno-strict-aliasing -Wsign-compare -g -O0 -Wall -Wstrict-prototypes -Ibuild/temp.freebsd-6.4-RELEASE-i386-3.5-pydebug/libffi/include -Ibuild/temp.freebsd-6.4-RELEASE-i386-3.5-pydebug/libffi -I/usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Modules/_ctypes/libffi/src -I./Include -I. -IInclude -I/usr/local/include -I/usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Include -I/usr/home/db3l/buildarea/3.x.bolen-freebsd/build -c /usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Modules/_ctypes/stgdict.c -o build/temp.freebsd-6.4-RELEASE-i386-3.5-pydebug/usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Modules/_ctypes/stgdict.o -Wall -fexceptions
gcc -pthread -fPIC -fno-strict-aliasing -Wsign-compare -g -O0 -Wall -Wstrict-prototypes -Ibuild/temp.freebsd-6.4-RELEASE-i386-3.5-pydebug/libffi/include -Ibuild/temp.freebsd-6.4-RELEASE-i386-3.5-pydebug/libffi -I/usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Modules/_ctypes/libffi/src -I./Include -I. -IInclude -I/usr/local/include -I/usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Include -I/usr/home/db3l/buildarea/3.x.bolen-freebsd/build -c /usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Modules/_ctypes/cfield.c -o build/temp.freebsd-6.4-RELEASE-i386-3.5-pydebug/usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Modules/_ctypes/cfield.o -Wall -fexceptions
gcc -pthread -fPIC -fno-strict-aliasing -Wsign-compare -g -O0 -Wall -Wstrict-prototypes -Ibuild/temp.freebsd-6.4-RELEASE-i386-3.5-pydebug/libffi/include -Ibuild/temp.freebsd-6.4-RELEASE-i386-3.5-pydebug/libffi -I/usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Modules/_ctypes/libffi/src -I./Include -I. -IInclude -I/usr/local/include -I/usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Include -I/usr/home/db3l/buildarea/3.x.bolen-freebsd/build -c /usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Modules/_ctypes/libffi/src/prep_cif.c -o build/temp.freebsd-6.4-RELEASE-i386-3.5-pydebug/usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Modules/_ctypes/libffi/src/prep_cif.o -Wall -fexceptions
gcc -pthread -fPIC -fno-strict-aliasing -Wsign-compare -g -O0 -Wall -Wstrict-prototypes -Ibuild/temp.freebsd-6.4-RELEASE-i386-3.5-pydebug/libffi/include -Ibuild/temp.freebsd-6.4-RELEASE-i386-3.5-pydebug/libffi -I/usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Modules/_ctypes/libffi/src -I./Include -I. -IInclude -I/usr/local/include -I/usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Include -I/usr/home/db3l/buildarea/3.x.bolen-freebsd/build -c /usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Modules/_ctypes/libffi/src/closures.c -o build/temp.freebsd-6.4-RELEASE-i386-3.5-pydebug/usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Modules/_ctypes/libffi/src/closures.o -Wall -fexceptions
gcc -pthread -fPIC -fno-strict-aliasing -Wsign-compare -g -O0 -Wall -Wstrict-prototypes -Ibuild/temp.freebsd-6.4-RELEASE-i386-3.5-pydebug/libffi/include -Ibuild/temp.freebsd-6.4-RELEASE-i386-3.5-pydebug/libffi -I/usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Modules/_ctypes/libffi/src -I./Include -I. -IInclude -I/usr/local/include -I/usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Include -I/usr/home/db3l/buildarea/3.x.bolen-freebsd/build -c /usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Modules/_ctypes/libffi/src/x86/ffi.c -o build/temp.freebsd-6.4-RELEASE-i386-3.5-pydebug/usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Modules/_ctypes/libffi/src/x86/ffi.o -Wall -fexceptions
gcc -pthread -fPIC -fno-strict-aliasing -Wsign-compare -g -O0 -Wall -Wstrict-prototypes -Ibuild/temp.freebsd-6.4-RELEASE-i386-3.5-pydebug/libffi/include -Ibuild/temp.freebsd-6.4-RELEASE-i386-3.5-pydebug/libffi -I/usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Modules/_ctypes/libffi/src -I./Include -I. -IInclude -I/usr/local/include -I/usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Include -I/usr/home/db3l/buildarea/3.x.bolen-freebsd/build -c /usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Modules/_ctypes/libffi/src/x86/freebsd.S -o build/temp.freebsd-6.4-RELEASE-i386-3.5-pydebug/usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Modules/_ctypes/libffi/src/x86/freebsd.o -Wall -fexceptions
gcc -pthread -shared build/temp.freebsd-6.4-RELEASE-i386-3.5-pydebug/usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Modules/_ctypes/_ctypes.o build/temp.freebsd-6.4-RELEASE-i386-3.5-pydebug/usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Modules/_ctypes/callbacks.o build/temp.freebsd-6.4-RELEASE-i386-3.5-pydebug/usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Modules/_ctypes/callproc.o build/temp.freebsd-6.4-RELEASE-i386-3.5-pydebug/usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Modules/_ctypes/stgdict.o build/temp.freebsd-6.4-RELEASE-i386-3.5-pydebug/usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Modules/_ctypes/cfield.o build/temp.freebsd-6.4-RELEASE-i386-3.5-pydebug/usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Modules/_ctypes/libffi/src/prep_cif.o build/temp.freebsd-6.4-RELEASE-i386-3.5-pydebug/usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Modules/_ctypes/libffi/src/closures.o build/temp.freebsd-6.4-RELEASE-i386-3.5-pydebug/usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Modules/_ctypes/libffi/src/x86/ffi.o build/temp.freebsd-6.4-RELEASE-i386-3.5-pydebug/usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Modules/_ctypes/libffi/src/x86/freebsd.o -L/usr/local/lib -o build/lib.freebsd-6.4-RELEASE-i386-3.5-pydebug/_ctypes.so
*** WARNING: renaming "_ctypes" since importing it failed: build/lib.freebsd-6.4-RELEASE-i386-3.5-pydebug/_ctypes.so: Undefined symbol "ffi_call_win32"
---
msg227883 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2014-09-30 12:04
There is a similar error on OpenIndiana buildbot, Python cannot be compiled since this build:

http://buildbot.python.org/all/builders/AMD64%20OpenIndiana%203.x/builds/8664/steps/compile/logs/stdio

gcc -fPIC -Wsign-compare -g -O0 -Wall -Wstrict-prototypes -I/usr/local/include/ncursesw -m64 -Werror=declaration-after-statement -Ibuild/temp.solaris-2.11-i86pc.64bit-3.5-pydebug/libffi/include -Ibuild/temp.solaris-2.11-i86pc.64bit-3.5-pydebug/libffi -I/export/home/buildbot/64bits/3.x.cea-indiana-amd64/build/Modules/_ctypes/libffi/src -I./Include -I. -IInclude -I/usr/local/include -I/export/home/buildbot/64bits/3.x.cea-indiana-amd64/build/Include -I/export/home/buildbot/64bits/3.x.cea-indiana-amd64/build -c /export/home/buildbot/64bits/3.x.cea-indiana-amd64/build/Modules/_ctypes/libffi/src/x86/sysv.S -o build/temp.solaris-2.11-i86pc.64bit-3.5-pydebug/export/home/buildbot/64bits/3.x.cea-indiana-amd64/build/Modules/_ctypes/libffi/src/x86/sysv.o -Wall -fexceptions
gcc -fPIC -Wsign-compare -g -O0 -Wall -Wstrict-prototypes -I/usr/local/include/ncursesw -m64 -Werror=declaration-after-statement -Ibuild/temp.solaris-2.11-i86pc.64bit-3.5-pydebug/libffi/include -Ibuild/temp.solaris-2.11-i86pc.64bit-3.5-pydebug/libffi -I/export/home/buildbot/64bits/3.x.cea-indiana-amd64/build/Modules/_ctypes/libffi/src -I./Include -I. -IInclude -I/usr/local/include -I/export/home/buildbot/64bits/3.x.cea-indiana-amd64/build/Include -I/export/home/buildbot/64bits/3.x.cea-indiana-amd64/build -c /export/home/buildbot/64bits/3.x.cea-indiana-amd64/build/Modules/_ctypes/libffi/src/x86/win32.S -o build/temp.solaris-2.11-i86pc.64bit-3.5-pydebug/export/home/buildbot/64bits/3.x.cea-indiana-amd64/build/Modules/_ctypes/libffi/src/x86/win32.o -Wall -fexceptions
/var/tmp//ccqBa4n2.s: Assembler messages:
/var/tmp//ccqBa4n2.s:8: Error: suffix or operands invalid for `push'
/var/tmp//ccqBa4n2.s:17: Error: suffix or operands invalid for `push'
/var/tmp//ccqBa4n2.s:18: Error: suffix or operands invalid for `push'
/var/tmp//ccqBa4n2.s:79: Error: suffix or operands invalid for `jmp'
/var/tmp//ccqBa4n2.s:138: Error: suffix or operands invalid for `pop'
/var/tmp//ccqBa4n2.s:147: Error: suffix or operands invalid for `push'
/var/tmp//ccqBa4n2.s:155: Error: suffix or operands invalid for `push'
/var/tmp//ccqBa4n2.s:156: Error: suffix or operands invalid for `push'
/var/tmp//ccqBa4n2.s:163: Error: suffix or operands invalid for `push'
/var/tmp//ccqBa4n2.s:176: Error: suffix or operands invalid for `pop'
/var/tmp//ccqBa4n2.s:210: Error: suffix or operands invalid for `jmp'
/var/tmp//ccqBa4n2.s:251: Error: suffix or operands invalid for `pop'
/var/tmp//ccqBa4n2.s:261: Error: suffix or operands invalid for `pop'
/var/tmp//ccqBa4n2.s:268: Error: suffix or operands invalid for `push'
/var/tmp//ccqBa4n2.s:272: Error: suffix or operands invalid for `push'
/var/tmp//ccqBa4n2.s:315: Error: suffix or operands invalid for `jmp'
/var/tmp//ccqBa4n2.s:358: Error: suffix or operands invalid for `pop'
/var/tmp//ccqBa4n2.s:359: Error: suffix or operands invalid for `pop'
/var/tmp//ccqBa4n2.s:370: Error: suffix or operands invalid for `push'
/var/tmp//ccqBa4n2.s:383: Error: suffix or operands invalid for `pop'
/var/tmp//ccqBa4n2.s:416: Error: suffix or operands invalid for `jmp'
/var/tmp//ccqBa4n2.s:459: Error: suffix or operands invalid for `pop'
/var/tmp//ccqBa4n2.s:460: Error: suffix or operands invalid for `pop'
/var/tmp//ccqBa4n2.s:461: Error: suffix or operands invalid for `pop'
/var/tmp//ccqBa4n2.s:471: Error: suffix or operands invalid for `jmp'
*** WARNING: renaming "_curses" since importing it failed: ld.so.1: python: fatal: relocation error: file build/lib.solaris-2.11-i86pc.64bit-3.5-pydebug/_curses.so: symbol newscr: referenced symbol not found
*** WARNING: renaming "_curses_panel" since importing it failed: No module named '_curses'
error: [Errno 2] No such file or directory: 'build/temp.solaris-2.11-i86pc.64bit-3.5-pydebug/export/home/buildbot/64bits/3.x.cea-indiana-amd64/build/Modules/_decimal/_decimal.o'
make: *** [sharedmods] Error 1
msg227885 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2014-09-30 12:21
Thanks for noticing this.
Looking at an old build, the ctypes and curses compile errors under OpenIndiana are not new:
http://buildbot.python.org/all/builders/AMD64%20OpenIndiana%203.x/builds/8658/steps/compile/logs/stdio

Simply, it seems that a ctypes compile error is now interpreted as a failure of the compilation build step. I'll try to fix that.

(the problem looks similar on the FreeBSD 6.4 buildbot)
msg227887 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2014-09-30 12:24
> the ctypes and curses compile errors under OpenIndiana are not new

FYI the compilation error of curses on OpenIndiana is *old*, at least 3 years old :-(
http://bugs.python.org/issue13552

I just opened the issue #22521 for the compilation error of ctypes on FreeBSD.
msg227888 - (view) Author: Jonas Wagner (Sjlver) * Date: 2014-09-30 12:25
I don't understand where this error comes from...

The compilation commands are exactly the same in both the "before" and "after" logs. The order of commands is also the same.

The only difference is this message:

    *** WARNING: renaming "_ssl" since importing it failed: build/lib.freebsd-6.4-RELEASE-i386-3.5-pydebug/_ssl.so: Undefined symbol "X509_check_ca"

This message probably occurs because _ssl.so is now built in parallel with _ctypes.

In the logs, one can see that the command to build _ctypes.o is executed; because of that, I wonder why the file is missing. Maybe we could check for wrong dependency tracking in the _ctypes makefile (similar to http://bugs.python.org/issue22359)
msg227902 - (view) Author: Roundup Robot (python-dev) Date: 2014-09-30 12:58
New changeset 94af1af93670 by Antoine Pitrou in branch 'default':
Remove pointless "vile hack" that can cause the build step to fail when some extension modules can't be imported.
https://hg.python.org/cpython/rev/94af1af93670
msg227906 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2014-09-30 13:01
The change I just pushed should fix the failures, waiting for the buildbots to compile now.
msg227907 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2014-09-30 13:04
Looks ok on OpenIndiana, closing now.
History
Date User Action Args
2014-09-30 13:04:24pitrousetstatus: open -> closed
resolution: fixed
messages: + msg227907
2014-09-30 13:01:04pitrousetmessages: + msg227906
2014-09-30 12:58:50python-devsetmessages: + msg227902
2014-09-30 12:25:06Sjlversetmessages: + msg227888
2014-09-30 12:24:40hayposetmessages: + msg227887
2014-09-30 12:21:48pitrousetmessages: + msg227885
2014-09-30 12:04:45hayposetmessages: + msg227883
2014-09-30 10:57:16hayposetstatus: closed -> open

nosy: + haypo
messages: + msg227877

resolution: fixed -> (no value)
2014-09-26 21:33:47pitrousetstatus: open -> closed
resolution: fixed
messages: + msg227652

stage: patch review -> resolved
2014-09-26 21:32:17python-devsetnosy: + python-dev
messages: + msg227651
2014-09-20 22:26:09pitrousetfiles: + build_ext_parallel6.patch

messages: + msg227188
2014-09-09 16:40:16Sjlversetmessages: + msg226646
2014-09-07 21:23:42eric.araujosetmessages: + msg226546
2014-09-06 23:37:08Arfreversetmessages: + msg226523
2014-09-06 23:23:45pitrousetstage: needs patch -> patch review
2014-09-06 23:16:48pitrousetfiles: + build_ext_parallel5.patch

messages: + msg226521
2014-09-05 13:59:52Sjlversetmessages: + msg226424
2014-09-05 09:54:23scodersetnosy: + scoder

messages: + msg226415
versions: + Python 3.5, - Python 3.4
2014-09-05 09:48:25Sjlversetfiles: + build_ext_parallel4.patch

messages: + msg226412
2014-09-05 09:25:55Sjlversetnosy: + Sjlver
2013-10-26 22:50:54christian.heimessetfiles: + build_ext_parallel3.patch

messages: + msg201400
2013-10-26 21:09:23pitrousetfiles: + build_ext_parallel2.patch

messages: + msg201396
2013-10-26 20:08:06pitrousetfiles: + build_ext_parallel.patch
keywords: + patch
messages: + msg201392
2013-10-25 22:34:37Arfreversetnosy: + Arfrever

title: setup.py doesn't parallelize extension module compilation -> distutils doesn't parallelize extension module compilation
2013-01-04 10:01:17christian.heimessetnosy: + christian.heimes
messages: + msg179010
2013-01-03 22:27:08pitrousetversions: + Python 3.4, - 3rd party
title: packaging doesn't parallelize extension module compilation -> setup.py doesn't parallelize extension module compilation
messages: + msg178993

assignee: tarek ->
components: + Build, - Distutils2
2013-01-03 21:24:53bfroehlesetnosy: + bfroehle
2011-04-08 15:11:56eric.araujosettitle: setup.py doesn't parallelize extension module compilation -> packaging doesn't parallelize extension module compilation
2011-03-21 22:26:30eric.araujosetkeywords: + gsoc
nosy: pitrou, tarek, eric.araujo
2010-09-29 23:51:32eric.araujosetversions: + 3rd party, - Python 2.6, Python 2.5, Python 3.1, Python 2.7, Python 3.2
2010-09-16 20:24:33eric.araujosetstage: needs patch
messages: + msg116603
versions: + Python 2.6, Python 2.5, Python 3.1, Python 2.7
2010-09-16 16:18:40BreamoreBoysetnosy: + eric.araujo

components: + Distutils2, - Distutils
versions: + Python 3.2, - Python 3.1, Python 2.7
2009-02-19 01:39:51pitroucreate