➜

This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: Race conditions when the same source file used to build mutliple extensions
Type: compile error Stage:
Components: Distutils Versions: Python 3.10, Python 3.9, Python 3.8, Python 3.7, Python 3.6
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: dstufft, eric.araujo, mgorny
Priority: normal Keywords: patch

Created on 2021-03-10 07:42 by mgorny, last changed 2022-04-11 14:59 by admin.

Files
File name Uploaded Description Edit
0001-A-reproducer-for-distutils-build_ext-race-condition.patch mgorny, 2021-03-10 07:42 A reproducer
Messages (1)
msg388410 - (view) Author: Michał Górny (mgorny) * Date: 2021-03-10 07:42 
There is a race condition in distutils' build_ext implementation.  When the same source file is used to build multiple extensions, distutils attempts to build it multiple times using the same output file, in parallel.  This means that the link editor can grab the file while another compiler instance is overwriting it.  The results vary from compile errors to cryptic dyld failures when attempting to load the module.

I've created a trivial reproducer that I've attached in a patch form.  For convenience, it's also available on my GitHub: https://github.com/mgorny/distutils-build_ext-race

The reproducer consists of two extension modules sharing the same file.  The race.sh script attempts to build the extension and then import it.  The process is repeated until something fails, e.g.:

+ python3.10 setup.py build_ext -i -j4
running build_ext
building 'bar' extension
creating build
building 'foo' extension
creating build/temp.linux-x86_64-3.10
creating build/temp.linux-x86_64-3.10
x86_64-pc-linux-gnu-gcc-10.2.0 -pthread -fPIC -I/usr/include/python3.10 -c bar.c -o build/temp.linux-x86_64-3.10/bar.o
x86_64-pc-linux-gnu-gcc-10.2.0 -pthread -fPIC -I/usr/include/python3.10 -c foo.c -o build/temp.linux-x86_64-3.10/foo.o
x86_64-pc-linux-gnu-gcc-10.2.0 -pthread -fPIC -I/usr/include/python3.10 -c shared.c -o build/temp.linux-x86_64-3.10/shared.o
x86_64-pc-linux-gnu-gcc-10.2.0 -pthread -fPIC -I/usr/include/python3.10 -c shared.c -o build/temp.linux-x86_64-3.10/shared.o
x86_64-pc-linux-gnu-gcc-10.2.0 -pthread -shared -Wl,-O1 -Wl,--as-needed -Wl,--hash-style=gnu build/temp.linux-x86_64-3.10/foo.o build/temp.linux-x86_64-3.10/shared.o -L/usr/lib64 -o /home/mgorny/git/distutils-build_ext-race/foo.cpython-310-x86_64-linux-gnu.so
x86_64-pc-linux-gnu-gcc-10.2.0 -pthread -shared -Wl,-O1 -Wl,--as-needed -Wl,--hash-style=gnu build/temp.linux-x86_64-3.10/bar.o build/temp.linux-x86_64-3.10/shared.o -L/usr/lib64 -o /home/mgorny/git/distutils-build_ext-race/bar.cpython-310-x86_64-linux-gnu.so
+ python3.10 -c 'import foo; import bar'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ImportError: /home/mgorny/git/distutils-build_ext-race/foo.cpython-310-x86_64-linux-gnu.so: undefined symbol: call_shared
+ echo 'Reproduced at iteration 256'
Reproduced at iteration 256
+ break
History
Date User Action Args
2022-04-11 14:59:42adminsetgithub: 87625
2021-03-10 07:42:12mgornycreate