distutils doesn't parallelize extension module compilation #49559
Comments
|
When run with "make -jN", the Python compilation process is able to |
|
Do you want to work on a patch? |
merwok
changed the title
|
The original request is really about setup.py, not packaging. I don't care about packaging, and it's not in the stdlib. |
pitrou
changed the title
|
+1 I think it's sufficient to parallelize the compilation step (.c -> .o) and ignore the linker step. Linking is usually so fast that it doesn't matter. Idea:
|
Arfrever
mannequin
changed the title
|
Here is a working patch. |
|
Here is an updated patch which only enables parallel building in setup.py. |
|
Antoine's patch doesn't work for a fresh copy of Python
|
|
Is there a reason this has not landed? The patch works perfectly for me, except for one issue: If self.parallel is True, this will set self.parallel to 1, causing it to use one worker instead of n (where n is the number of processors). An updated patch is attached. |
|
Yes, please. The sequential build in distutils is very annoying. (too late for Py3.4, though, I guess...) |
|
With this patch, and on Ubuntu 14.04, occasionally modules fail to build with the following error: *** WARNING: renaming "_testbuffer" since importing it failed: dlopen: cannot load any more object with static TLS I'm not 100% sure if this is really due to the patch... but I've never seen it before. I'll try investigate a bit more. |
|
Updated patch:
|
|
The distutils idiom would be to catch the TypeError/ValueError and raise DistutilsOptionError. Higher layers convert that to a message. |
|
I've checked the Using this patch and http://bugs.python.org/issue22359 , I now get reliable parallel builds. |
|
Updated patch raising DistutilsOptionError. I couldn't find any docs so I didn't update them :-) |
|
New changeset bbe57429eba0 by Antoine Pitrou in branch 'default': |
|
This is now pushed. |
|
It looks like compilation of Python 3.5 fails on FreeBSD 6.4 because of the changeset bbe57429eba0a9ec21fb0f1178f409f1bba44c22: http://buildbot.python.org/all/builders/x86%20FreeBSD%206.4%203.x/builds/5061 Compile log: building '_ctypes' extension Before: |
|
There is a similar error on OpenIndiana buildbot, Python cannot be compiled since this build: gcc -fPIC -Wsign-compare -g -O0 -Wall -Wstrict-prototypes -I/usr/local/include/ncursesw -m64 -Werror=declaration-after-statement -Ibuild/temp.solaris-2.11-i86pc.64bit-3.5-pydebug/libffi/include -Ibuild/temp.solaris-2.11-i86pc.64bit-3.5-pydebug/libffi -I/export/home/buildbot/64bits/3.x.cea-indiana-amd64/build/Modules/_ctypes/libffi/src -I./Include -I. -IInclude -I/usr/local/include -I/export/home/buildbot/64bits/3.x.cea-indiana-amd64/build/Include -I/export/home/buildbot/64bits/3.x.cea-indiana-amd64/build -c /export/home/buildbot/64bits/3.x.cea-indiana-amd64/build/Modules/_ctypes/libffi/src/x86/sysv.S -o build/temp.solaris-2.11-i86pc.64bit-3.5-pydebug/export/home/buildbot/64bits/3.x.cea-indiana-amd64/build/Modules/_ctypes/libffi/src/x86/sysv.o -Wall -fexceptions |
|
Thanks for noticing this. Simply, it seems that a ctypes compile error is now interpreted as a failure of the compilation build step. I'll try to fix that. (the problem looks similar on the FreeBSD 6.4 buildbot) |
FYI the compilation error of curses on OpenIndiana is *old*, at least 3 years old :-( I just opened the issue bpo-22521 for the compilation error of ctypes on FreeBSD. |
|
I don't understand where this error comes from... The compilation commands are exactly the same in both the "before" and "after" logs. The order of commands is also the same. The only difference is this message: This message probably occurs because _ssl.so is now built in parallel with _ctypes. In the logs, one can see that the command to build _ctypes.o is executed; because of that, I wonder why the file is missing. Maybe we could check for wrong dependency tracking in the _ctypes makefile (similar to http://bugs.python.org/issue22359) |
|
New changeset 94af1af93670 by Antoine Pitrou in branch 'default': |
|
The change I just pushed should fix the failures, waiting for the buildbots to compile now. |
|
Looks ok on OpenIndiana, closing now. |
|
very nice, thanks for adding this. coincidentally numpy added the same to numpy.distutils independently just a week later, though numpy also accepts an environment variable to set the number of jobs. Is the naming --parallel/j already fixed? I'll change the numpy options to the same name then. Please also add it to the release notes so the feature can be discovered easier. |
|
Here's a doc patch. |
|
New changeset 107669985805 by Berker Peksag in branch 'default': |
|
New changeset fc28c67fbbd3 by doko in branch 'default':
|
|
I have 2 complaints about this: 1 - doc is missing: the only way to be aware of this is either by reading the 3.6 what's new doc or by checking the cmdline helper 2 - -j "N" parameter could be optional: if not specified os.cpu_count() can be used. |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
bugs.python.org fields:
The text was updated successfully, but these errors were encountered: