This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

Author mgorny
Recipients dstufft, eric.araujo, mgorny
Date 2021-03-10.07:42:11
SpamBayes Score -1.0
Marked as misclassified Yes
Message-id <1615362132.43.0.297389622008.issue43459@roundup.psfhosted.org>
In-reply-to
Content
There is a race condition in distutils' build_ext implementation.  When the same source file is used to build multiple extensions, distutils attempts to build it multiple times using the same output file, in parallel.  This means that the link editor can grab the file while another compiler instance is overwriting it.  The results vary from compile errors to cryptic dyld failures when attempting to load the module.

I've created a trivial reproducer that I've attached in a patch form.  For convenience, it's also available on my GitHub: https://github.com/mgorny/distutils-build_ext-race

The reproducer consists of two extension modules sharing the same file.  The race.sh script attempts to build the extension and then import it.  The process is repeated until something fails, e.g.:

+ python3.10 setup.py build_ext -i -j4
running build_ext
building 'bar' extension
creating build
building 'foo' extension
creating build/temp.linux-x86_64-3.10
creating build/temp.linux-x86_64-3.10
x86_64-pc-linux-gnu-gcc-10.2.0 -pthread -fPIC -I/usr/include/python3.10 -c bar.c -o build/temp.linux-x86_64-3.10/bar.o
x86_64-pc-linux-gnu-gcc-10.2.0 -pthread -fPIC -I/usr/include/python3.10 -c foo.c -o build/temp.linux-x86_64-3.10/foo.o
x86_64-pc-linux-gnu-gcc-10.2.0 -pthread -fPIC -I/usr/include/python3.10 -c shared.c -o build/temp.linux-x86_64-3.10/shared.o
x86_64-pc-linux-gnu-gcc-10.2.0 -pthread -fPIC -I/usr/include/python3.10 -c shared.c -o build/temp.linux-x86_64-3.10/shared.o
x86_64-pc-linux-gnu-gcc-10.2.0 -pthread -shared -Wl,-O1 -Wl,--as-needed -Wl,--hash-style=gnu build/temp.linux-x86_64-3.10/foo.o build/temp.linux-x86_64-3.10/shared.o -L/usr/lib64 -o /home/mgorny/git/distutils-build_ext-race/foo.cpython-310-x86_64-linux-gnu.so
x86_64-pc-linux-gnu-gcc-10.2.0 -pthread -shared -Wl,-O1 -Wl,--as-needed -Wl,--hash-style=gnu build/temp.linux-x86_64-3.10/bar.o build/temp.linux-x86_64-3.10/shared.o -L/usr/lib64 -o /home/mgorny/git/distutils-build_ext-race/bar.cpython-310-x86_64-linux-gnu.so
+ python3.10 -c 'import foo; import bar'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ImportError: /home/mgorny/git/distutils-build_ext-race/foo.cpython-310-x86_64-linux-gnu.so: undefined symbol: call_shared
+ echo 'Reproduced at iteration 256'
Reproduced at iteration 256
+ break
History
Date User Action Args
2021-03-10 07:42:12mgornysetrecipients: + mgorny, eric.araujo, dstufft
2021-03-10 07:42:12mgornysetmessageid: <1615362132.43.0.297389622008.issue43459@roundup.psfhosted.org>
2021-03-10 07:42:12mgornylinkissue43459 messages
2021-03-10 07:42:12mgornycreate