➜
This issue tracker has been migrated to GitHub,
and is currently read-only.
For more information,
see the GitHub FAQs in the Python's Developer Guide.
Created on 2009-02-19 01:39 by pitrou, last changed 2022-04-11 14:56 by admin. This issue is now closed.
| Files | ||||
|---|---|---|---|---|
| File name | Uploaded | Description | Edit | |
| build_ext_parallel.patch | pitrou, 2013-10-26 20:08 | review | ||
| build_ext_parallel2.patch | pitrou, 2013-10-26 21:09 | review | ||
| build_ext_parallel3.patch | christian.heimes, 2013-10-26 22:50 | review | ||
| build_ext_parallel4.patch | Sjlver, 2014-09-05 09:48 | review | ||
| build_ext_parallel5.patch | pitrou, 2014-09-06 23:16 | review | ||
| build_ext_parallel6.patch | pitrou, 2014-09-20 22:26 | review | ||
| issue5309-doc.diff | berker.peksag, 2015-01-22 21:52 | review | ||
| Messages (30) | |||
|---|---|---|---|
| msg82447 - (view) | Author: Antoine Pitrou (pitrou) *  |
Date: 2009-02-19 01:39 | |
When run with "make -jN", the Python compilation process is able to parallelize across N concurrent processes for the interpreter core, but extension modules are still compiled sequentially. |
|||
| msg116603 - (view) | Author: Éric Araujo (eric.araujo) * |
Date: 2010-09-16 20:24 | |
Do you want to work on a patch? |
|||
| msg178993 - (view) | Author: Antoine Pitrou (pitrou) * |
Date: 2013-01-03 22:27 | |
The original request is really about setup.py, not packaging. I don't care about packaging, and it's not in the stdlib. |
|||
| msg179010 - (view) | Author: Christian Heimes (christian.heimes) * |
Date: 2013-01-04 10:01 | |
+1 I think it's sufficient to parallelize the compilation step (.c -> .o) and ignore the linker step. Linking is usually so fast that it doesn't matter. Idea: * separate compile step from link step * run all compile calls for all extensions in parallel until all objects of all extensions are generated * do linking and error reporting in serial mode |
|||
| msg201392 - (view) | Author: Antoine Pitrou (pitrou) * |
Date: 2013-10-26 20:08 | |
Here is a working patch. |
|||
| msg201396 - (view) | Author: Antoine Pitrou (pitrou) * |
Date: 2013-10-26 21:09 | |
Here is an updated patch which only enables parallel building in setup.py. |
|||
| msg201400 - (view) | Author: Christian Heimes (christian.heimes) * |
Date: 2013-10-26 22:50 | |
Antoine's patch doesn't work for a fresh copy of Python - add import error report to build_ext parallel (distutils doesn't use logging and doesn't have a working log.exception() function) - make time a builtin module - add hack for concurrent.futures.process - add --parallel=i / -ji (jobs) argument to build_ext |
|||
| msg226412 - (view) | Author: Jonas Wagner (Sjlver) * | Date: 2014-09-05 09:48 | |
Is there a reason this has not landed? The patch works perfectly for me, except for one issue:
@@ -268,6 +275,9 @@
if self.undef:
self.undef = self.undef.split(',')
+ if self.parallel:
+ self.parallel = int(self.parallel)
+
if self.swig_opts is None:
self.swig_opts = []
else:
If self.parallel is True, this will set self.parallel to 1, causing it to use one worker instead of n (where n is the number of processors).
An updated patch is attached.
|
|||
| msg226415 - (view) | Author: Stefan Behnel (scoder) * |
Date: 2014-09-05 09:54 | |
Yes, please. The sequential build in distutils is very annoying. (too late for Py3.4, though, I guess...) |
|||
| msg226424 - (view) | Author: Jonas Wagner (Sjlver) * | Date: 2014-09-05 13:59 | |
With this patch, and on Ubuntu 14.04, occasionally modules fail to build with the following error: *** WARNING: renaming "_testbuffer" since importing it failed: dlopen: cannot load any more object with static TLS I'm not 100% sure if this is really due to the patch... but I've never seen it before. I'll try investigate a bit more. |
|||
| msg226521 - (view) | Author: Antoine Pitrou (pitrou) * |
Date: 2014-09-06 23:16 | |
Updated patch: - add the -j option to the "build" command as well (even though it's only used by build_ext) - in setup.py, only force parallel compilation if make was called with -j |
|||
| msg226523 - (view) | Author: Arfrever Frehtes Taifersar Arahesis (Arfrever) *  |
Date: 2014-09-06 23:37 | |
- Is support for infinite jobs planned? (GNU make supports it for -j without number argument.) - self.parallel = int(self.parallel) would raise ValueError when a non-number is passed. I suggest to print user-friendly error message. |
|||
| msg226546 - (view) | Author: Éric Araujo (eric.araujo) * |
Date: 2014-09-07 21:23 | |
> self.parallel = int(self.parallel) would raise ValueError when a > non-number is passed. I suggest to print user-friendly error message. The distutils idiom would be to catch the TypeError/ValueError and raise DistutilsOptionError. Higher layers convert that to a message. |
|||
| msg226646 - (view) | Author: Jonas Wagner (Sjlver) * | Date: 2014-09-09 16:40 | |
I've checked the `dlopen` issue. This was due to a problem with UndefinedBehaviorSanitizer, and was solved by upgrading to Clang 3.5. It has little to do with this patch. Using this patch and http://bugs.python.org/issue22359 , I now get reliable parallel builds. |
|||
| msg227188 - (view) | Author: Antoine Pitrou (pitrou) * |
Date: 2014-09-20 22:26 | |
Updated patch raising DistutilsOptionError. I couldn't find any docs so I didn't update them :-) |
|||
| msg227651 - (view) | Author: Roundup Robot (python-dev) |
Date: 2014-09-26 21:32 | |
New changeset bbe57429eba0 by Antoine Pitrou in branch 'default': Issue #5309: distutils' build and build_ext commands now accept a ``-j`` https://hg.python.org/cpython/rev/bbe57429eba0 |
|||
| msg227652 - (view) | Author: Antoine Pitrou (pitrou) * |
Date: 2014-09-26 21:33 | |
This is now pushed. |
|||
| msg227877 - (view) | Author: STINNER Victor (vstinner) * |
Date: 2014-09-30 10:57 | |
It looks like compilation of Python 3.5 fails on FreeBSD 6.4 because of the changeset bbe57429eba0a9ec21fb0f1178f409f1bba44c22: http://buildbot.python.org/all/builders/x86%20FreeBSD%206.4%203.x/builds/5061 Compile log: --- building '_ctypes' extension creating build/temp.freebsd-6.4-RELEASE-i386-3.5-pydebug/usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Modules/_ctypes/libffi creating build/temp.freebsd-6.4-RELEASE-i386-3.5-pydebug/usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Modules/_ctypes/libffi/src creating build/temp.freebsd-6.4-RELEASE-i386-3.5-pydebug/usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Modules/_ctypes/libffi/src/x86 gcc -pthread -fPIC -fno-strict-aliasing -Wsign-compare -g -O0 -Wall -Wstrict-prototypes -Ibuild/temp.freebsd-6.4-RELEASE-i386-3.5-pydebug/libffi/include -Ibuild/temp.freebsd-6.4-RELEASE-i386-3.5-pydebug/libffi -I/usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Modules/_ctypes/libffi/src -I./Include -I. -IInclude -I/usr/local/include -I/usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Include -I/usr/home/db3l/buildarea/3.x.bolen-freebsd/build -c /usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Modules/_ctypes/_ctypes.c -o build/temp.freebsd-6.4-RELEASE-i386-3.5-pydebug/usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Modules/_ctypes/_ctypes.o -Wall -fexceptions gcc -pthread -fPIC -fno-strict-aliasing -Wsign-compare -g -O0 -Wall -Wstrict-prototypes -Ibuild/temp.freebsd-6.4-RELEASE-i386-3.5-pydebug/libffi/include -Ibuild/temp.freebsd-6.4-RELEASE-i386-3.5-pydebug/libffi -I/usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Modules/_ctypes/libffi/src -I./Include -I. -IInclude -I/usr/local/include -I/usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Include -I/usr/home/db3l/buildarea/3.x.bolen-freebsd/build -c /usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Modules/_ctypes/callbacks.c -o build/temp.freebsd-6.4-RELEASE-i386-3.5-pydebug/usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Modules/_ctypes/callbacks.o -Wall -fexceptions gcc -pthread -fPIC -fno-strict-aliasing -Wsign-compare -g -O0 -Wall -Wstrict-prototypes -Ibuild/temp.freebsd-6.4-RELEASE-i386-3.5-pydebug/libffi/include -Ibuild/temp.freebsd-6.4-RELEASE-i386-3.5-pydebug/libffi -I/usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Modules/_ctypes/libffi/src -I./Include -I. -IInclude -I/usr/local/include -I/usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Include -I/usr/home/db3l/buildarea/3.x.bolen-freebsd/build -c /usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Modules/_ctypes/callproc.c -o build/temp.freebsd-6.4-RELEASE-i386-3.5-pydebug/usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Modules/_ctypes/callproc.o -Wall -fexceptions /usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Modules/_ctypes/callproc.c: In function `py_dl_open': /usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Modules/_ctypes/callproc.c:1336: warning: initialization discards qualifiers from pointer target type gcc -pthread -fPIC -fno-strict-aliasing -Wsign-compare -g -O0 -Wall -Wstrict-prototypes -Ibuild/temp.freebsd-6.4-RELEASE-i386-3.5-pydebug/libffi/include -Ibuild/temp.freebsd-6.4-RELEASE-i386-3.5-pydebug/libffi -I/usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Modules/_ctypes/libffi/src -I./Include -I. -IInclude -I/usr/local/include -I/usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Include -I/usr/home/db3l/buildarea/3.x.bolen-freebsd/build -c /usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Modules/_ctypes/stgdict.c -o build/temp.freebsd-6.4-RELEASE-i386-3.5-pydebug/usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Modules/_ctypes/stgdict.o -Wall -fexceptions gcc -pthread -fPIC -fno-strict-aliasing -Wsign-compare -g -O0 -Wall -Wstrict-prototypes -Ibuild/temp.freebsd-6.4-RELEASE-i386-3.5-pydebug/libffi/include -Ibuild/temp.freebsd-6.4-RELEASE-i386-3.5-pydebug/libffi -I/usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Modules/_ctypes/libffi/src -I./Include -I. -IInclude -I/usr/local/include -I/usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Include -I/usr/home/db3l/buildarea/3.x.bolen-freebsd/build -c /usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Modules/_ctypes/cfield.c -o build/temp.freebsd-6.4-RELEASE-i386-3.5-pydebug/usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Modules/_ctypes/cfield.o -Wall -fexceptions gcc -pthread -fPIC -fno-strict-aliasing -Wsign-compare -g -O0 -Wall -Wstrict-prototypes -Ibuild/temp.freebsd-6.4-RELEASE-i386-3.5-pydebug/libffi/include -Ibuild/temp.freebsd-6.4-RELEASE-i386-3.5-pydebug/libffi -I/usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Modules/_ctypes/libffi/src -I./Include -I. -IInclude -I/usr/local/include -I/usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Include -I/usr/home/db3l/buildarea/3.x.bolen-freebsd/build -c /usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Modules/_ctypes/libffi/src/prep_cif.c -o build/temp.freebsd-6.4-RELEASE-i386-3.5-pydebug/usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Modules/_ctypes/libffi/src/prep_cif.o -Wall -fexceptions gcc -pthread -fPIC -fno-strict-aliasing -Wsign-compare -g -O0 -Wall -Wstrict-prototypes -Ibuild/temp.freebsd-6.4-RELEASE-i386-3.5-pydebug/libffi/include -Ibuild/temp.freebsd-6.4-RELEASE-i386-3.5-pydebug/libffi -I/usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Modules/_ctypes/libffi/src -I./Include -I. -IInclude -I/usr/local/include -I/usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Include -I/usr/home/db3l/buildarea/3.x.bolen-freebsd/build -c /usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Modules/_ctypes/libffi/src/closures.c -o build/temp.freebsd-6.4-RELEASE-i386-3.5-pydebug/usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Modules/_ctypes/libffi/src/closures.o -Wall -fexceptions gcc -pthread -fPIC -fno-strict-aliasing -Wsign-compare -g -O0 -Wall -Wstrict-prototypes -Ibuild/temp.freebsd-6.4-RELEASE-i386-3.5-pydebug/libffi/include -Ibuild/temp.freebsd-6.4-RELEASE-i386-3.5-pydebug/libffi -I/usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Modules/_ctypes/libffi/src -I./Include -I. -IInclude -I/usr/local/include -I/usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Include -I/usr/home/db3l/buildarea/3.x.bolen-freebsd/build -c /usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Modules/_ctypes/libffi/src/x86/ffi.c -o build/temp.freebsd-6.4-RELEASE-i386-3.5-pydebug/usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Modules/_ctypes/libffi/src/x86/ffi.o -Wall -fexceptions gcc -pthread -fPIC -fno-strict-aliasing -Wsign-compare -g -O0 -Wall -Wstrict-prototypes -Ibuild/temp.freebsd-6.4-RELEASE-i386-3.5-pydebug/libffi/include -Ibuild/temp.freebsd-6.4-RELEASE-i386-3.5-pydebug/libffi -I/usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Modules/_ctypes/libffi/src -I./Include -I. -IInclude -I/usr/local/include -I/usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Include -I/usr/home/db3l/buildarea/3.x.bolen-freebsd/build -c /usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Modules/_ctypes/libffi/src/x86/freebsd.S -o build/temp.freebsd-6.4-RELEASE-i386-3.5-pydebug/usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Modules/_ctypes/libffi/src/x86/freebsd.o -Wall -fexceptions gcc -pthread -shared build/temp.freebsd-6.4-RELEASE-i386-3.5-pydebug/usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Modules/_ctypes/_ctypes.o build/temp.freebsd-6.4-RELEASE-i386-3.5-pydebug/usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Modules/_ctypes/callbacks.o build/temp.freebsd-6.4-RELEASE-i386-3.5-pydebug/usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Modules/_ctypes/callproc.o build/temp.freebsd-6.4-RELEASE-i386-3.5-pydebug/usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Modules/_ctypes/stgdict.o build/temp.freebsd-6.4-RELEASE-i386-3.5-pydebug/usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Modules/_ctypes/cfield.o build/temp.freebsd-6.4-RELEASE-i386-3.5-pydebug/usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Modules/_ctypes/libffi/src/prep_cif.o build/temp.freebsd-6.4-RELEASE-i386-3.5-pydebug/usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Modules/_ctypes/libffi/src/closures.o build/temp.freebsd-6.4-RELEASE-i386-3.5-pydebug/usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Modules/_ctypes/libffi/src/x86/ffi.o build/temp.freebsd-6.4-RELEASE-i386-3.5-pydebug/usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Modules/_ctypes/libffi/src/x86/freebsd.o -L/usr/local/lib -o build/lib.freebsd-6.4-RELEASE-i386-3.5-pydebug/_ctypes.so *** WARNING: renaming "_ssl" since importing it failed: build/lib.freebsd-6.4-RELEASE-i386-3.5-pydebug/_ssl.so: Undefined symbol "X509_check_ca" *** WARNING: renaming "_ctypes" since importing it failed: build/lib.freebsd-6.4-RELEASE-i386-3.5-pydebug/_ctypes.so: Undefined symbol "ffi_call_win32" error: [Errno 2] No such file or directory: 'build/temp.freebsd-6.4-RELEASE-i386-3.5-pydebug/usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Modules/_ctypes/_ctypes.o' *** Error code 1 --- Before: --- building '_ctypes' extension gcc -pthread -fPIC -fno-strict-aliasing -Wsign-compare -g -O0 -Wall -Wstrict-prototypes -Ibuild/temp.freebsd-6.4-RELEASE-i386-3.5-pydebug/libffi/include -Ibuild/temp.freebsd-6.4-RELEASE-i386-3.5-pydebug/libffi -I/usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Modules/_ctypes/libffi/src -I./Include -I. -IInclude -I/usr/local/include -I/usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Include -I/usr/home/db3l/buildarea/3.x.bolen-freebsd/build -c /usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Modules/_ctypes/_ctypes.c -o build/temp.freebsd-6.4-RELEASE-i386-3.5-pydebug/usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Modules/_ctypes/_ctypes.o -Wall -fexceptions gcc -pthread -fPIC -fno-strict-aliasing -Wsign-compare -g -O0 -Wall -Wstrict-prototypes -Ibuild/temp.freebsd-6.4-RELEASE-i386-3.5-pydebug/libffi/include -Ibuild/temp.freebsd-6.4-RELEASE-i386-3.5-pydebug/libffi -I/usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Modules/_ctypes/libffi/src -I./Include -I. -IInclude -I/usr/local/include -I/usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Include -I/usr/home/db3l/buildarea/3.x.bolen-freebsd/build -c /usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Modules/_ctypes/callbacks.c -o build/temp.freebsd-6.4-RELEASE-i386-3.5-pydebug/usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Modules/_ctypes/callbacks.o -Wall -fexceptions gcc -pthread -fPIC -fno-strict-aliasing -Wsign-compare -g -O0 -Wall -Wstrict-prototypes -Ibuild/temp.freebsd-6.4-RELEASE-i386-3.5-pydebug/libffi/include -Ibuild/temp.freebsd-6.4-RELEASE-i386-3.5-pydebug/libffi -I/usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Modules/_ctypes/libffi/src -I./Include -I. -IInclude -I/usr/local/include -I/usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Include -I/usr/home/db3l/buildarea/3.x.bolen-freebsd/build -c /usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Modules/_ctypes/callproc.c -o build/temp.freebsd-6.4-RELEASE-i386-3.5-pydebug/usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Modules/_ctypes/callproc.o -Wall -fexceptions /usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Modules/_ctypes/callproc.c: In function `py_dl_open': /usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Modules/_ctypes/callproc.c:1336: warning: initialization discards qualifiers from pointer target type gcc -pthread -fPIC -fno-strict-aliasing -Wsign-compare -g -O0 -Wall -Wstrict-prototypes -Ibuild/temp.freebsd-6.4-RELEASE-i386-3.5-pydebug/libffi/include -Ibuild/temp.freebsd-6.4-RELEASE-i386-3.5-pydebug/libffi -I/usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Modules/_ctypes/libffi/src -I./Include -I. -IInclude -I/usr/local/include -I/usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Include -I/usr/home/db3l/buildarea/3.x.bolen-freebsd/build -c /usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Modules/_ctypes/stgdict.c -o build/temp.freebsd-6.4-RELEASE-i386-3.5-pydebug/usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Modules/_ctypes/stgdict.o -Wall -fexceptions gcc -pthread -fPIC -fno-strict-aliasing -Wsign-compare -g -O0 -Wall -Wstrict-prototypes -Ibuild/temp.freebsd-6.4-RELEASE-i386-3.5-pydebug/libffi/include -Ibuild/temp.freebsd-6.4-RELEASE-i386-3.5-pydebug/libffi -I/usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Modules/_ctypes/libffi/src -I./Include -I. -IInclude -I/usr/local/include -I/usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Include -I/usr/home/db3l/buildarea/3.x.bolen-freebsd/build -c /usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Modules/_ctypes/cfield.c -o build/temp.freebsd-6.4-RELEASE-i386-3.5-pydebug/usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Modules/_ctypes/cfield.o -Wall -fexceptions gcc -pthread -fPIC -fno-strict-aliasing -Wsign-compare -g -O0 -Wall -Wstrict-prototypes -Ibuild/temp.freebsd-6.4-RELEASE-i386-3.5-pydebug/libffi/include -Ibuild/temp.freebsd-6.4-RELEASE-i386-3.5-pydebug/libffi -I/usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Modules/_ctypes/libffi/src -I./Include -I. -IInclude -I/usr/local/include -I/usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Include -I/usr/home/db3l/buildarea/3.x.bolen-freebsd/build -c /usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Modules/_ctypes/libffi/src/prep_cif.c -o build/temp.freebsd-6.4-RELEASE-i386-3.5-pydebug/usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Modules/_ctypes/libffi/src/prep_cif.o -Wall -fexceptions gcc -pthread -fPIC -fno-strict-aliasing -Wsign-compare -g -O0 -Wall -Wstrict-prototypes -Ibuild/temp.freebsd-6.4-RELEASE-i386-3.5-pydebug/libffi/include -Ibuild/temp.freebsd-6.4-RELEASE-i386-3.5-pydebug/libffi -I/usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Modules/_ctypes/libffi/src -I./Include -I. -IInclude -I/usr/local/include -I/usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Include -I/usr/home/db3l/buildarea/3.x.bolen-freebsd/build -c /usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Modules/_ctypes/libffi/src/closures.c -o build/temp.freebsd-6.4-RELEASE-i386-3.5-pydebug/usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Modules/_ctypes/libffi/src/closures.o -Wall -fexceptions gcc -pthread -fPIC -fno-strict-aliasing -Wsign-compare -g -O0 -Wall -Wstrict-prototypes -Ibuild/temp.freebsd-6.4-RELEASE-i386-3.5-pydebug/libffi/include -Ibuild/temp.freebsd-6.4-RELEASE-i386-3.5-pydebug/libffi -I/usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Modules/_ctypes/libffi/src -I./Include -I. -IInclude -I/usr/local/include -I/usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Include -I/usr/home/db3l/buildarea/3.x.bolen-freebsd/build -c /usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Modules/_ctypes/libffi/src/x86/ffi.c -o build/temp.freebsd-6.4-RELEASE-i386-3.5-pydebug/usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Modules/_ctypes/libffi/src/x86/ffi.o -Wall -fexceptions gcc -pthread -fPIC -fno-strict-aliasing -Wsign-compare -g -O0 -Wall -Wstrict-prototypes -Ibuild/temp.freebsd-6.4-RELEASE-i386-3.5-pydebug/libffi/include -Ibuild/temp.freebsd-6.4-RELEASE-i386-3.5-pydebug/libffi -I/usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Modules/_ctypes/libffi/src -I./Include -I. -IInclude -I/usr/local/include -I/usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Include -I/usr/home/db3l/buildarea/3.x.bolen-freebsd/build -c /usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Modules/_ctypes/libffi/src/x86/freebsd.S -o build/temp.freebsd-6.4-RELEASE-i386-3.5-pydebug/usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Modules/_ctypes/libffi/src/x86/freebsd.o -Wall -fexceptions gcc -pthread -shared build/temp.freebsd-6.4-RELEASE-i386-3.5-pydebug/usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Modules/_ctypes/_ctypes.o build/temp.freebsd-6.4-RELEASE-i386-3.5-pydebug/usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Modules/_ctypes/callbacks.o build/temp.freebsd-6.4-RELEASE-i386-3.5-pydebug/usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Modules/_ctypes/callproc.o build/temp.freebsd-6.4-RELEASE-i386-3.5-pydebug/usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Modules/_ctypes/stgdict.o build/temp.freebsd-6.4-RELEASE-i386-3.5-pydebug/usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Modules/_ctypes/cfield.o build/temp.freebsd-6.4-RELEASE-i386-3.5-pydebug/usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Modules/_ctypes/libffi/src/prep_cif.o build/temp.freebsd-6.4-RELEASE-i386-3.5-pydebug/usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Modules/_ctypes/libffi/src/closures.o build/temp.freebsd-6.4-RELEASE-i386-3.5-pydebug/usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Modules/_ctypes/libffi/src/x86/ffi.o build/temp.freebsd-6.4-RELEASE-i386-3.5-pydebug/usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Modules/_ctypes/libffi/src/x86/freebsd.o -L/usr/local/lib -o build/lib.freebsd-6.4-RELEASE-i386-3.5-pydebug/_ctypes.so *** WARNING: renaming "_ctypes" since importing it failed: build/lib.freebsd-6.4-RELEASE-i386-3.5-pydebug/_ctypes.so: Undefined symbol "ffi_call_win32" --- |
|||
| msg227883 - (view) | Author: STINNER Victor (vstinner) * |
Date: 2014-09-30 12:04 | |
There is a similar error on OpenIndiana buildbot, Python cannot be compiled since this build: http://buildbot.python.org/all/builders/AMD64%20OpenIndiana%203.x/builds/8664/steps/compile/logs/stdio gcc -fPIC -Wsign-compare -g -O0 -Wall -Wstrict-prototypes -I/usr/local/include/ncursesw -m64 -Werror=declaration-after-statement -Ibuild/temp.solaris-2.11-i86pc.64bit-3.5-pydebug/libffi/include -Ibuild/temp.solaris-2.11-i86pc.64bit-3.5-pydebug/libffi -I/export/home/buildbot/64bits/3.x.cea-indiana-amd64/build/Modules/_ctypes/libffi/src -I./Include -I. -IInclude -I/usr/local/include -I/export/home/buildbot/64bits/3.x.cea-indiana-amd64/build/Include -I/export/home/buildbot/64bits/3.x.cea-indiana-amd64/build -c /export/home/buildbot/64bits/3.x.cea-indiana-amd64/build/Modules/_ctypes/libffi/src/x86/sysv.S -o build/temp.solaris-2.11-i86pc.64bit-3.5-pydebug/export/home/buildbot/64bits/3.x.cea-indiana-amd64/build/Modules/_ctypes/libffi/src/x86/sysv.o -Wall -fexceptions gcc -fPIC -Wsign-compare -g -O0 -Wall -Wstrict-prototypes -I/usr/local/include/ncursesw -m64 -Werror=declaration-after-statement -Ibuild/temp.solaris-2.11-i86pc.64bit-3.5-pydebug/libffi/include -Ibuild/temp.solaris-2.11-i86pc.64bit-3.5-pydebug/libffi -I/export/home/buildbot/64bits/3.x.cea-indiana-amd64/build/Modules/_ctypes/libffi/src -I./Include -I. -IInclude -I/usr/local/include -I/export/home/buildbot/64bits/3.x.cea-indiana-amd64/build/Include -I/export/home/buildbot/64bits/3.x.cea-indiana-amd64/build -c /export/home/buildbot/64bits/3.x.cea-indiana-amd64/build/Modules/_ctypes/libffi/src/x86/win32.S -o build/temp.solaris-2.11-i86pc.64bit-3.5-pydebug/export/home/buildbot/64bits/3.x.cea-indiana-amd64/build/Modules/_ctypes/libffi/src/x86/win32.o -Wall -fexceptions /var/tmp//ccqBa4n2.s: Assembler messages: /var/tmp//ccqBa4n2.s:8: Error: suffix or operands invalid for `push' /var/tmp//ccqBa4n2.s:17: Error: suffix or operands invalid for `push' /var/tmp//ccqBa4n2.s:18: Error: suffix or operands invalid for `push' /var/tmp//ccqBa4n2.s:79: Error: suffix or operands invalid for `jmp' /var/tmp//ccqBa4n2.s:138: Error: suffix or operands invalid for `pop' /var/tmp//ccqBa4n2.s:147: Error: suffix or operands invalid for `push' /var/tmp//ccqBa4n2.s:155: Error: suffix or operands invalid for `push' /var/tmp//ccqBa4n2.s:156: Error: suffix or operands invalid for `push' /var/tmp//ccqBa4n2.s:163: Error: suffix or operands invalid for `push' /var/tmp//ccqBa4n2.s:176: Error: suffix or operands invalid for `pop' /var/tmp//ccqBa4n2.s:210: Error: suffix or operands invalid for `jmp' /var/tmp//ccqBa4n2.s:251: Error: suffix or operands invalid for `pop' /var/tmp//ccqBa4n2.s:261: Error: suffix or operands invalid for `pop' /var/tmp//ccqBa4n2.s:268: Error: suffix or operands invalid for `push' /var/tmp//ccqBa4n2.s:272: Error: suffix or operands invalid for `push' /var/tmp//ccqBa4n2.s:315: Error: suffix or operands invalid for `jmp' /var/tmp//ccqBa4n2.s:358: Error: suffix or operands invalid for `pop' /var/tmp//ccqBa4n2.s:359: Error: suffix or operands invalid for `pop' /var/tmp//ccqBa4n2.s:370: Error: suffix or operands invalid for `push' /var/tmp//ccqBa4n2.s:383: Error: suffix or operands invalid for `pop' /var/tmp//ccqBa4n2.s:416: Error: suffix or operands invalid for `jmp' /var/tmp//ccqBa4n2.s:459: Error: suffix or operands invalid for `pop' /var/tmp//ccqBa4n2.s:460: Error: suffix or operands invalid for `pop' /var/tmp//ccqBa4n2.s:461: Error: suffix or operands invalid for `pop' /var/tmp//ccqBa4n2.s:471: Error: suffix or operands invalid for `jmp' *** WARNING: renaming "_curses" since importing it failed: ld.so.1: python: fatal: relocation error: file build/lib.solaris-2.11-i86pc.64bit-3.5-pydebug/_curses.so: symbol newscr: referenced symbol not found *** WARNING: renaming "_curses_panel" since importing it failed: No module named '_curses' error: [Errno 2] No such file or directory: 'build/temp.solaris-2.11-i86pc.64bit-3.5-pydebug/export/home/buildbot/64bits/3.x.cea-indiana-amd64/build/Modules/_decimal/_decimal.o' make: *** [sharedmods] Error 1 |
|||
| msg227885 - (view) | Author: Antoine Pitrou (pitrou) * |
Date: 2014-09-30 12:21 | |
Thanks for noticing this. Looking at an old build, the ctypes and curses compile errors under OpenIndiana are not new: http://buildbot.python.org/all/builders/AMD64%20OpenIndiana%203.x/builds/8658/steps/compile/logs/stdio Simply, it seems that a ctypes compile error is now interpreted as a failure of the compilation build step. I'll try to fix that. (the problem looks similar on the FreeBSD 6.4 buildbot) |
|||
| msg227887 - (view) | Author: STINNER Victor (vstinner) * |
Date: 2014-09-30 12:24 | |
> the ctypes and curses compile errors under OpenIndiana are not new FYI the compilation error of curses on OpenIndiana is *old*, at least 3 years old :-( http://bugs.python.org/issue13552 I just opened the issue #22521 for the compilation error of ctypes on FreeBSD. |
|||
| msg227888 - (view) | Author: Jonas Wagner (Sjlver) * | Date: 2014-09-30 12:25 | |
I don't understand where this error comes from...
The compilation commands are exactly the same in both the "before" and "after" logs. The order of commands is also the same.
The only difference is this message:
*** WARNING: renaming "_ssl" since importing it failed: build/lib.freebsd-6.4-RELEASE-i386-3.5-pydebug/_ssl.so: Undefined symbol "X509_check_ca"
This message probably occurs because _ssl.so is now built in parallel with _ctypes.
In the logs, one can see that the command to build _ctypes.o is executed; because of that, I wonder why the file is missing. Maybe we could check for wrong dependency tracking in the _ctypes makefile (similar to http://bugs.python.org/issue22359)
|
|||
| msg227902 - (view) | Author: Roundup Robot (python-dev) |
Date: 2014-09-30 12:58 | |
New changeset 94af1af93670 by Antoine Pitrou in branch 'default': Remove pointless "vile hack" that can cause the build step to fail when some extension modules can't be imported. https://hg.python.org/cpython/rev/94af1af93670 |
|||
| msg227906 - (view) | Author: Antoine Pitrou (pitrou) * |
Date: 2014-09-30 13:01 | |
The change I just pushed should fix the failures, waiting for the buildbots to compile now. |
|||
| msg227907 - (view) | Author: Antoine Pitrou (pitrou) * |
Date: 2014-09-30 13:04 | |
Looks ok on OpenIndiana, closing now. |
|||
| msg234147 - (view) | Author: Julian Taylor (jtaylor) | Date: 2015-01-16 19:59 | |
very nice, thanks for adding this. coincidentally numpy added the same to numpy.distutils independently just a week later, though numpy also accepts an environment variable to set the number of jobs. This is useful for e.g. pip installations where one does not control the command line. Also an environment variable allows parallel jobs in environments where it is not guaranteed that the feature is available. E.g. you could just put it into your .bashrc and when building with 3.5 it will just work and 2.7 will not fail. Is the naming --parallel/j already fixed? I'll change the numpy options to the same name then. Please also add it to the release notes so the feature can be discovered easier. |
|||
| msg234514 - (view) | Author: Berker Peksag (berker.peksag) * |
Date: 2015-01-22 21:52 | |
Here's a doc patch. |
|||
| msg234803 - (view) | Author: Roundup Robot (python-dev) |
Date: 2015-01-27 00:59 | |
New changeset 107669985805 by Berker Peksag in branch 'default': Add whatsnew entry for issue #5309. https://hg.python.org/cpython/rev/107669985805 |
|||
| msg240505 - (view) | Author: Roundup Robot (python-dev) |
Date: 2015-04-11 22:14 | |
New changeset fc28c67fbbd3 by doko in branch 'default': - Modules/Setup.dist: remove time extension duplicate, introduced by the fix for #5309. https://hg.python.org/cpython/rev/fc28c67fbbd3 |
|||
| msg276819 - (view) | Author: Giampaolo Rodola' (giampaolo.rodola) * |
Date: 2016-09-17 19:05 | |
I have 2 complaints about this: 1 - doc is missing: the only way to be aware of this is either by reading the 3.6 what's new doc or by checking the cmdline helper 2 - -j "N" parameter could be optional: if not specified os.cpu_count() can be used. |
|||
| History | |||
|---|---|---|---|
| Date | User | Action | Args |
| 2022-04-11 14:56:45 | admin | set | github: 49559 |
| 2016-09-17 19:05:32 | giampaolo.rodola | set | nosy:
+ giampaolo.rodola messages: + msg276819 |
| 2015-04-11 22:14:07 | python-dev | set | messages: + msg240505 |
| 2015-01-27 00:59:01 | python-dev | set | messages: + msg234803 |
| 2015-01-22 21:52:57 | berker.peksag | set | files:
+ issue5309-doc.diff nosy: + berker.peksag messages: + msg234514 |
| 2015-01-16 19:59:14 | jtaylor | set | nosy:
+ jtaylor messages: + msg234147 |
| 2014-09-30 13:04:24 | pitrou | set | status: open -> closed resolution: fixed messages: + msg227907 |
| 2014-09-30 13:01:04 | pitrou | set | messages: + msg227906 |
| 2014-09-30 12:58:50 | python-dev | set | messages: + msg227902 |
| 2014-09-30 12:25:06 | Sjlver | set | messages: + msg227888 |
| 2014-09-30 12:24:40 | vstinner | set | messages: + msg227887 |
| 2014-09-30 12:21:48 | pitrou | set | messages: + msg227885 |
| 2014-09-30 12:04:45 | vstinner | set | messages: + msg227883 |
| 2014-09-30 10:57:16 | vstinner | set | status: closed -> open nosy: + vstinner messages: + msg227877 resolution: fixed -> (no value) |
| 2014-09-26 21:33:47 | pitrou | set | status: open -> closed resolution: fixed messages: + msg227652 stage: patch review -> resolved |
| 2014-09-26 21:32:17 | python-dev | set | nosy:
+ python-dev messages: + msg227651 |
| 2014-09-20 22:26:09 | pitrou | set | files:
+ build_ext_parallel6.patch messages: + msg227188 |
| 2014-09-09 16:40:16 | Sjlver | set | messages: + msg226646 |
| 2014-09-07 21:23:42 | eric.araujo | set | messages: + msg226546 |
| 2014-09-06 23:37:08 | Arfrever | set | messages: + msg226523 |
| 2014-09-06 23:23:45 | pitrou | set | stage: needs patch -> patch review |
| 2014-09-06 23:16:48 | pitrou | set | files:
+ build_ext_parallel5.patch messages: + msg226521 |
| 2014-09-05 13:59:52 | Sjlver | set | messages: + msg226424 |
| 2014-09-05 09:54:23 | scoder | set | nosy:
+ scoder messages: + msg226415 versions: + Python 3.5, - Python 3.4 |
| 2014-09-05 09:48:25 | Sjlver | set | files:
+ build_ext_parallel4.patch messages: + msg226412 |
| 2014-09-05 09:25:55 | Sjlver | set | nosy:
+ Sjlver |
| 2013-10-26 22:50:54 | christian.heimes | set | files:
+ build_ext_parallel3.patch messages: + msg201400 |
| 2013-10-26 21:09:23 | pitrou | set | files:
+ build_ext_parallel2.patch messages: + msg201396 |
| 2013-10-26 20:08:06 | pitrou | set | files:
+ build_ext_parallel.patch keywords: + patch messages: + msg201392 |
| 2013-10-25 22:34:37 | Arfrever | set | nosy:
+ Arfrever title: setup.py doesn't parallelize extension module compilation -> distutils doesn't parallelize extension module compilation |
| 2013-01-04 10:01:17 | christian.heimes | set | nosy:
+ christian.heimes messages: + msg179010 |
| 2013-01-03 22:27:08 | pitrou | set | versions:
+ Python 3.4, - 3rd party title: packaging doesn't parallelize extension module compilation -> setup.py doesn't parallelize extension module compilation messages: + msg178993 assignee: tarek -> components: + Build, - Distutils2 |
| 2013-01-03 21:24:53 | bfroehle | set | nosy:
+ bfroehle |
| 2011-04-08 15:11:56 | eric.araujo | set | title: setup.py doesn't parallelize extension module compilation -> packaging doesn't parallelize extension module compilation |
| 2011-03-21 22:26:30 | eric.araujo | set | keywords:
+ gsoc nosy: pitrou, tarek, eric.araujo |
| 2010-09-29 23:51:32 | eric.araujo | set | versions: + 3rd party, - Python 2.6, Python 2.5, Python 3.1, Python 2.7, Python 3.2 |
| 2010-09-16 20:24:33 | eric.araujo | set | stage: needs patch messages: + msg116603 versions: + Python 2.6, Python 2.5, Python 3.1, Python 2.7 |
| 2010-09-16 16:18:40 | BreamoreBoy | set | nosy:
+ eric.araujo components: + Distutils2, - Distutils versions: + Python 3.2, - Python 3.1, Python 2.7 |
| 2009-02-19 01:39:51 | pitrou | create | |