Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Race condition compiling Modules/_math.c #68609

Closed
vadmium opened this issue Jun 10, 2015 · 11 comments
Closed

Race condition compiling Modules/_math.c #68609

vadmium opened this issue Jun 10, 2015 · 11 comments
Labels
build The build process and cross-build stdlib Python modules in the Lib dir type-bug An unexpected behavior, bug, or error

Comments

@vadmium
Copy link
Member

vadmium commented Jun 10, 2015

BPO 24421
Nosy @Yhg1s, @taleinat, @merwok, @vadmium, @floppym, @dstufft
Files
  • slow-cc.py
  • include-math.c.patch
  • math-once.patch: Compile from makefile
    		•

    Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = None
closed_at = <Date 2016-02-26.21:46:18.055>
created_at = <Date 2015-06-10.08:06:40.639>
labels = ['type-bug', 'library', 'build']
title = 'Race condition compiling Modules/_math.c'
updated_at = <Date 2016-02-26.21:46:18.054>
user = 'https://github.com/vadmium'

    bugs.python.org fields:

    activity = <Date 2016-02-26.21:46:18.054>
actor = 'martin.panter'
assignee = 'none'
closed = True
closed_date = <Date 2016-02-26.21:46:18.055>
closer = 'martin.panter'
components = ['Build', 'Distutils']
creation = <Date 2015-06-10.08:06:40.639>
creator = 'martin.panter'
dependencies = []
files = ['39672', '39724', '41270']
hgrepos = []
issue_num = 24421
keywords = ['patch']
message_count = 11.0
messages = ['245121', '245298', '245429', '245453', '245459', '254560', '256137', '258349', '259362', '260886', '260888']
nosy_count = 8.0
nosy_names = ['twouters', 'taleinat', 'eric.araujo', 'Arfrever', 'python-dev', 'martin.panter', 'floppymaster', 'dstufft']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue24421'
versions = ['Python 2.7', 'Python 3.5', 'Python 3.6']
    @vadmium
    Copy link
    Member Author

    vadmium commented Jun 10, 2015

    Once or twice I have had the math or cmath modules fail to build. There seems to be a race condition, because the _math.c file seems to be compiled twice, once for each module. Attached is a GCC wrapper that should help reproduce this fairly consistently. First, build Python normally:

    $ ./configure
$ make -j2

    Then force Modules/_math.c to be recompiled using my wrapper script to amplify the race condition:

    $ touch Modules/_math.c
$ rm build/temp.linux-x86_64-3.6/media/disk/home/proj/python/cpython/Modules/_math.o
$ make -j2 CC=./slow-cc.py
running build
running build_ext
building 'cmath' extension
compiling for  cmath
./slow-cc.py [. . .]/Modules/cmathmodule.o
building 'math' extension
./slow-cc.py [. . .]/Modules/mathmodule.o
./slow-cc.py -fPIC -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -Werror=declaration-after-statement -I./Include -I. -IInclude -I/usr/local/include -I/media/disk/home/proj/python/cpython/Include -I/media/disk/home/proj/python/cpython -c /media/disk/home/proj/python/cpython/Modules/_math.c -o build/temp.linux-x86_64-3.6/media/disk/home/proj/python/cpython/Modules/_math.o
INITIAL COMPILE
./slow-cc.py -fPIC -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -Werror=declaration-after-statement -I./Include -I. -IInclude -I/usr/local/include -I/media/disk/home/proj/python/cpython/Include -I/media/disk/home/proj/python/cpython -c /media/disk/home/proj/python/cpython/Modules/_math.c -o build/temp.linux-x86_64-3.6/media/disk/home/proj/python/cpython/Modules/_math.o
RECOMPILE
RECOMPIILE TRUNCATING
INITIAL COMPILE FINISHED
./slow-cc.py -shared build/temp.linux-x86_64-3.6/media/disk/home/proj/python/cpython/Modules/mathmodule.o build/temp.linux-x86_64-3.6/media/disk/home/proj/python/cpython/Modules/_math.o -L/usr/local/lib -lm -o build/lib.linux-x86_64-3.6/math.cpython-36m-x86_64-linux-gnu.so
build/temp.linux-x86_64-3.6/media/disk/home/proj/python/cpython/Modules/_math.o: file not recognized: File truncated
collect2: error: ld returned 1 exit status
RECOMPILE FINISHED
./slow-cc.py -shared build/temp.linux-x86_64-3.6/media/disk/home/proj/python/cpython/Modules/cmathmodule.o build/temp.linux-x86_64-3.6/media/disk/home/proj/python/cpython/Modules/_math.o -L/usr/local/lib -lm -o build/lib.linux-x86_64-3.6/cmath.cpython-36m-x86_64-linux-gnu.so

    Failed to build these modules:
    math

    @vadmium vadmium added build The build process and cross-build type-bug An unexpected behavior, bug, or error labels Jun 10, 2015
    @taleinat
    Copy link
    Contributor

    Confirmed on OSX 10.10. Here's my output:

    running build
    running build_ext
    building 'cmath' extension
    ./slow-cc.py -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -Werror=declaration-after-statement -I./Include -I. -IInclude -I/usr/local/include -I/Users/taleinat/dev/cpython/Include -I/Users/taleinat/dev/cpython -c /Users/taleinat/dev/cpython/Modules/cmathmodule.c -o build/temp.macosx-10.10-x86_64-3.6/Users/taleinat/dev/cpython/Modules/cmathmodule.o
    building 'math' extension
    ./slow-cc.py -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -Werror=declaration-after-statement -I./Include -I. -IInclude -I/usr/local/include -I/Users/taleinat/dev/cpython/Include -I/Users/taleinat/dev/cpython -c /Users/taleinat/dev/cpython/Modules/mathmodule.c -o build/temp.macosx-10.10-x86_64-3.6/Users/taleinat/dev/cpython/Modules/mathmodule.o
    ./slow-cc.py -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -Werror=declaration-after-statement -I./Include -I. -IInclude -I/usr/local/include -I/Users/taleinat/dev/cpython/Include -I/Users/taleinat/dev/cpython -c /Users/taleinat/dev/cpython/Modules/_math.c -o build/temp.macosx-10.10-x86_64-3.6/Users/taleinat/dev/cpython/Modules/_math.o
    ./slow-cc.py -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -Werror=declaration-after-statement -I./Include -I. -IInclude -I/usr/local/include -I/Users/taleinat/dev/cpython/Include -I/Users/taleinat/dev/cpython -c /Users/taleinat/dev/cpython/Modules/_math.c -o build/temp.macosx-10.10-x86_64-3.6/Users/taleinat/dev/cpython/Modules/_math.o
    INITIAL COMPILE
    RECOMPILE
    RECOMPIILE TRUNCATING
    INITIAL COMPILE FINISHED
    ./slow-cc.py -bundle -undefined dynamic_lookup build/temp.macosx-10.10-x86_64-3.6/Users/taleinat/dev/cpython/Modules/mathmodule.o build/temp.macosx-10.10-x86_64-3.6/Users/taleinat/dev/cpython/Modules/_math.o -L/usr/local/lib -o build/lib.macosx-10.10-x86_64-3.6/math.cpython-36m-darwin.so
    ld: file too small (length=0) file 'build/temp.macosx-10.10-x86_64-3.6/Users/taleinat/dev/cpython/Modules/_math.o' for architecture x86_64
    clang: error: linker command failed with exit code 1 (use -v to see invocation)
    RECOMPILE FINISHED
    ./slow-cc.py -bundle -undefined dynamic_lookup build/temp.macosx-10.10-x86_64-3.6/Users/taleinat/dev/cpython/Modules/cmathmodule.o build/temp.macosx-10.10-x86_64-3.6/Users/taleinat/dev/cpython/Modules/_math.o -L/usr/local/lib -o build/lib.macosx-10.10-x86_64-3.6/cmath.cpython-36m-darwin.so

    @vadmium
    Copy link
    Member Author

    vadmium commented Jun 17, 2015

    I think this may have been introduced when _math.c was added as a source file of the “cmath” module in r76978 (bpo-7518). In /setup.py:585 there are two distutils.core.Extension objects, both mentioning _math.c. In the build_ext._build_extensions_parallel() method at /Lib/distutils/command/build_ext.py:449, it appears to be invoking build_extension() in two concurrent threads, once for each module, including math and cmath.

    I don’t know how this code is meant to work, but it looks like either the “build_ext” class should not be creating the same object file in two different threads, or /setup.py should not be giving it the same _math.c file for two different modules.

    @vadmium vadmium added the stdlib Python modules in the Lib dir label Jun 17, 2015
    @vadmium
    Copy link
    Member Author

    vadmium commented Jun 18, 2015

    Here is a hacky patch that adds #include "_math.c" so that it is not compiled as a separate object file. This was suggested by Mark in bpo-7518. It is far from a perfect solution, but I cannot suggest anything better without knowing more about Python’s build system.

    @taleinat
    Copy link
    Contributor

    Adding Thomas Wouters to the nosy list, since he's listed on the Experts Index under "autoconf/makefiles" as an interest area. Hopefully he can help move this forward.

    @vadmium
    Copy link
    Member Author

    vadmium commented Nov 12, 2015

    Ideally, I think the solution would be to compile _math.c once, as a dependency shared between both extension modules (perhaps like a static library if necessary). Does anyone know if the build system supports this, or how to do it?

    Another semi-favourable solution might be to compile _math.c twice, but send the output to separate directories, e.g. . . ./math-build/_math.o and . . ./cmath-build/_math.o. Again, I don’t know if this is practical.

    @vadmium
    Copy link
    Member Author

    vadmium commented Dec 9, 2015

    Here is an alternative patch to compile _math.c once from the main makefile, before the extension modules are built. I have only tested this on Linux, building in the source directory, and from a separate directory.

    @vadmium
    Copy link
    Member Author

    vadmium commented Jan 16, 2016

    I am inclined to try committing my second patch to 3.6, as the race condition keeps bugging me every now and again.

    My biggest worry is that it may not work on Windows. I assume Windows does not use the Makefile, but does use setup.py. In the worst case, perhaps I could add conditional code to restore the original setup.py parameters for Windows.

    @python-dev
    Copy link
    Mannequin

    python-dev mannequin commented Feb 2, 2016

    New changeset 76624d47ee99 by Martin Panter in branch 'default':
    Issue bpo-24421: Compile _math.c separately to avoid race condition
    https://hg.python.org/cpython/rev/76624d47ee99

    @python-dev
    Copy link
    Mannequin

    python-dev mannequin commented Feb 26, 2016

    New changeset a99580157968 by Martin Panter in branch '3.5':
    Issue bpo-24421: Compile _math.c separately to avoid race condition
    https://hg.python.org/cpython/rev/a99580157968

    @python-dev
    Copy link
    Mannequin

    python-dev mannequin commented Feb 26, 2016

    New changeset d4190ed586a4 by Martin Panter in branch '2.7':
    Issue bpo-24421: Compile _math.c separately to avoid race condition
    https://hg.python.org/cpython/rev/d4190ed586a4

    @vadmium vadmium closed this as completed Feb 26, 2016
    @ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
    Assignees
    No one assigned
    Labels
    build The build process and cross-build stdlib Python modules in the Lib dir type-bug An unexpected behavior, bug, or error
    Projects
    None yet
    Milestone
    No milestone
    Development

    No branches or pull requests
    2 participants
    @taleinat @vadmium