classification
Title: Race condition compiling Modules/_math.c
Type: behavior Stage: resolved
Components: Build, Distutils Versions: Python 3.6, Python 3.5, Python 2.7
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: Arfrever, dstufft, eric.araujo, floppymaster, martin.panter, python-dev, taleinat, twouters
Priority: normal Keywords: patch

Created on 2015-06-10 08:06 by martin.panter, last changed 2016-02-26 21:46 by martin.panter. This issue is now closed.

Files
File name Uploaded Description Edit
slow-cc.py martin.panter, 2015-06-10 08:06
include-math.c.patch martin.panter, 2015-06-18 05:30 review
math-once.patch martin.panter, 2015-12-09 04:46 Compile from makefile review
Messages (11)
msg245121 - (view) Author: Martin Panter (martin.panter) * (Python committer) Date: 2015-06-10 08:06
Once or twice I have had the math or cmath modules fail to build. There seems to be a race condition, because the _math.c file seems to be compiled twice, once for each module. Attached is a GCC wrapper that should help reproduce this fairly consistently. First, build Python normally:

$ ./configure
$ make -j2

Then force Modules/_math.c to be recompiled using my wrapper script to amplify the race condition:

$ touch Modules/_math.c
$ rm build/temp.linux-x86_64-3.6/media/disk/home/proj/python/cpython/Modules/_math.o
$ make -j2 CC=./slow-cc.py
running build
running build_ext
building 'cmath' extension
compiling for  cmath
./slow-cc.py [. . .]/Modules/cmathmodule.o
building 'math' extension
./slow-cc.py [. . .]/Modules/mathmodule.o
./slow-cc.py -fPIC -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -Werror=declaration-after-statement -I./Include -I. -IInclude -I/usr/local/include -I/media/disk/home/proj/python/cpython/Include -I/media/disk/home/proj/python/cpython -c /media/disk/home/proj/python/cpython/Modules/_math.c -o build/temp.linux-x86_64-3.6/media/disk/home/proj/python/cpython/Modules/_math.o
INITIAL COMPILE
./slow-cc.py -fPIC -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -Werror=declaration-after-statement -I./Include -I. -IInclude -I/usr/local/include -I/media/disk/home/proj/python/cpython/Include -I/media/disk/home/proj/python/cpython -c /media/disk/home/proj/python/cpython/Modules/_math.c -o build/temp.linux-x86_64-3.6/media/disk/home/proj/python/cpython/Modules/_math.o
RECOMPILE
RECOMPIILE TRUNCATING
INITIAL COMPILE FINISHED
./slow-cc.py -shared build/temp.linux-x86_64-3.6/media/disk/home/proj/python/cpython/Modules/mathmodule.o build/temp.linux-x86_64-3.6/media/disk/home/proj/python/cpython/Modules/_math.o -L/usr/local/lib -lm -o build/lib.linux-x86_64-3.6/math.cpython-36m-x86_64-linux-gnu.so
build/temp.linux-x86_64-3.6/media/disk/home/proj/python/cpython/Modules/_math.o: file not recognized: File truncated
collect2: error: ld returned 1 exit status
RECOMPILE FINISHED
./slow-cc.py -shared build/temp.linux-x86_64-3.6/media/disk/home/proj/python/cpython/Modules/cmathmodule.o build/temp.linux-x86_64-3.6/media/disk/home/proj/python/cpython/Modules/_math.o -L/usr/local/lib -lm -o build/lib.linux-x86_64-3.6/cmath.cpython-36m-x86_64-linux-gnu.so

Failed to build these modules:
math
msg245298 - (view) Author: Tal Einat (taleinat) * (Python committer) Date: 2015-06-13 06:51
Confirmed on OSX 10.10. Here's my output:

running build
running build_ext
building 'cmath' extension
./slow-cc.py -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -Werror=declaration-after-statement -I./Include -I. -IInclude -I/usr/local/include -I/Users/taleinat/dev/cpython/Include -I/Users/taleinat/dev/cpython -c /Users/taleinat/dev/cpython/Modules/cmathmodule.c -o build/temp.macosx-10.10-x86_64-3.6/Users/taleinat/dev/cpython/Modules/cmathmodule.o
building 'math' extension
./slow-cc.py -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -Werror=declaration-after-statement -I./Include -I. -IInclude -I/usr/local/include -I/Users/taleinat/dev/cpython/Include -I/Users/taleinat/dev/cpython -c /Users/taleinat/dev/cpython/Modules/mathmodule.c -o build/temp.macosx-10.10-x86_64-3.6/Users/taleinat/dev/cpython/Modules/mathmodule.o
./slow-cc.py -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -Werror=declaration-after-statement -I./Include -I. -IInclude -I/usr/local/include -I/Users/taleinat/dev/cpython/Include -I/Users/taleinat/dev/cpython -c /Users/taleinat/dev/cpython/Modules/_math.c -o build/temp.macosx-10.10-x86_64-3.6/Users/taleinat/dev/cpython/Modules/_math.o
./slow-cc.py -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -Werror=declaration-after-statement -I./Include -I. -IInclude -I/usr/local/include -I/Users/taleinat/dev/cpython/Include -I/Users/taleinat/dev/cpython -c /Users/taleinat/dev/cpython/Modules/_math.c -o build/temp.macosx-10.10-x86_64-3.6/Users/taleinat/dev/cpython/Modules/_math.o
INITIAL COMPILE
RECOMPILE
RECOMPIILE TRUNCATING
INITIAL COMPILE FINISHED
./slow-cc.py -bundle -undefined dynamic_lookup build/temp.macosx-10.10-x86_64-3.6/Users/taleinat/dev/cpython/Modules/mathmodule.o build/temp.macosx-10.10-x86_64-3.6/Users/taleinat/dev/cpython/Modules/_math.o -L/usr/local/lib -o build/lib.macosx-10.10-x86_64-3.6/math.cpython-36m-darwin.so
ld: file too small (length=0) file 'build/temp.macosx-10.10-x86_64-3.6/Users/taleinat/dev/cpython/Modules/_math.o' for architecture x86_64
clang: error: linker command failed with exit code 1 (use -v to see invocation)
RECOMPILE FINISHED
./slow-cc.py -bundle -undefined dynamic_lookup build/temp.macosx-10.10-x86_64-3.6/Users/taleinat/dev/cpython/Modules/cmathmodule.o build/temp.macosx-10.10-x86_64-3.6/Users/taleinat/dev/cpython/Modules/_math.o -L/usr/local/lib -o build/lib.macosx-10.10-x86_64-3.6/cmath.cpython-36m-darwin.so
msg245429 - (view) Author: Martin Panter (martin.panter) * (Python committer) Date: 2015-06-17 03:46
I think this may have been introduced when _math.c was added as a source file of the “cmath” module in r76978 (Issue 7518). In /setup.py:585 there are two distutils.core.Extension objects, both mentioning _math.c. In the build_ext._build_extensions_parallel() method at /Lib/distutils/command/build_ext.py:449, it appears to be invoking build_extension() in two concurrent threads, once for each module, including math and cmath.

I don’t know how this code is meant to work, but it looks like either the “build_ext” class should not be creating the same object file in two different threads, or /setup.py should not be giving it the same _math.c file for two different modules.
msg245453 - (view) Author: Martin Panter (martin.panter) * (Python committer) Date: 2015-06-18 05:30
Here is a hacky patch that adds #include "_math.c" so that it is not compiled as a separate object file. This was suggested by Mark in Issue 7518. It is far from a perfect solution, but I cannot suggest anything better without knowing more about Python’s build system.
msg245459 - (view) Author: Tal Einat (taleinat) * (Python committer) Date: 2015-06-18 10:32
Adding Thomas Wouters to the nosy list, since he's listed on the Experts Index under "autoconf/makefiles" as an interest area. Hopefully he can help move this forward.
msg254560 - (view) Author: Martin Panter (martin.panter) * (Python committer) Date: 2015-11-12 21:46
Ideally, I think the solution would be to compile _math.c once, as a dependency shared between both extension modules (perhaps like a static library if necessary). Does anyone know if the build system supports this, or how to do it?

Another semi-favourable solution might be to compile _math.c twice, but send the output to separate directories, e.g. . . ./math-build/_math.o and . . ./cmath-build/_math.o. Again, I don’t know if this is practical.
msg256137 - (view) Author: Martin Panter (martin.panter) * (Python committer) Date: 2015-12-09 04:46
Here is an alternative patch to compile _math.c once from the main makefile, before the extension modules are built. I have only tested this on Linux, building in the source directory, and from a separate directory.
msg258349 - (view) Author: Martin Panter (martin.panter) * (Python committer) Date: 2016-01-16 02:00
I am inclined to try committing my second patch to 3.6, as the race condition keeps bugging me every now and again.

My biggest worry is that it may not work on Windows. I assume Windows does not use the Makefile, but does use setup.py. In the worst case, perhaps I could add conditional code to restore the original setup.py parameters for Windows.
msg259362 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2016-02-02 05:59
New changeset 76624d47ee99 by Martin Panter in branch 'default':
Issue #24421: Compile _math.c separately to avoid race condition
https://hg.python.org/cpython/rev/76624d47ee99
msg260886 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2016-02-26 01:14
New changeset a99580157968 by Martin Panter in branch '3.5':
Issue #24421: Compile _math.c separately to avoid race condition
https://hg.python.org/cpython/rev/a99580157968
msg260888 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2016-02-26 02:15
New changeset d4190ed586a4 by Martin Panter in branch '2.7':
Issue #24421: Compile _math.c separately to avoid race condition
https://hg.python.org/cpython/rev/d4190ed586a4
History
Date User Action Args
2016-02-26 21:46:18martin.pantersetstatus: open -> closed
resolution: fixed
stage: patch review -> resolved
2016-02-26 02:15:01python-devsetmessages: + msg260888
2016-02-26 01:14:07python-devsetmessages: + msg260886
2016-02-07 09:36:29Arfreversetnosy: + Arfrever
2016-02-02 05:59:16python-devsetnosy: + python-dev
messages: + msg259362
2016-01-16 02:00:35martin.pantersetmessages: + msg258349
2015-12-09 04:46:52martin.pantersetfiles: + math-once.patch

messages: + msg256137
versions: - Python 3.4
2015-11-12 21:46:03martin.pantersetmessages: + msg254560
2015-11-12 05:06:22floppymastersetnosy: + floppymaster
2015-06-18 10:32:55taleinatsetnosy: + twouters
messages: + msg245459
2015-06-18 05:30:30martin.pantersetfiles: + include-math.c.patch
keywords: + patch
messages: + msg245453

stage: patch review
2015-06-17 03:46:40martin.pantersetversions: + Python 2.7, Python 3.4, Python 3.5, Python 3.6
nosy: + eric.araujo, dstufft

messages: + msg245429

components: + Distutils
2015-06-13 06:51:15taleinatsetnosy: + taleinat
messages: + msg245298
2015-06-10 08:06:40martin.pantercreate