This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

Title: Race conditions when the same source file used to build mutliple extensions
Type: compile error Stage:
Components: Distutils Versions: Python 3.10, Python 3.9, Python 3.8, Python 3.7, Python 3.6
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: dstufft, eric.araujo, mgorny
Priority: normal Keywords: patch

Created on 2021-03-10 07:42 by mgorny, last changed 2022-04-11 14:59 by admin.

File name Uploaded Description Edit
0001-A-reproducer-for-distutils-build_ext-race-condition.patch mgorny, 2021-03-10 07:42 A reproducer
Messages (1)
msg388410 - (view) Author: Michał Górny (mgorny) * Date: 2021-03-10 07:42
There is a race condition in distutils' build_ext implementation.  When the same source file is used to build multiple extensions, distutils attempts to build it multiple times using the same output file, in parallel.  This means that the link editor can grab the file while another compiler instance is overwriting it.  The results vary from compile errors to cryptic dyld failures when attempting to load the module.

I've created a trivial reproducer that I've attached in a patch form.  For convenience, it's also available on my GitHub:

The reproducer consists of two extension modules sharing the same file.  The script attempts to build the extension and then import it.  The process is repeated until something fails, e.g.:

+ python3.10 build_ext -i -j4
running build_ext
building 'bar' extension
creating build
building 'foo' extension
creating build/temp.linux-x86_64-3.10
creating build/temp.linux-x86_64-3.10
x86_64-pc-linux-gnu-gcc-10.2.0 -pthread -fPIC -I/usr/include/python3.10 -c bar.c -o build/temp.linux-x86_64-3.10/bar.o
x86_64-pc-linux-gnu-gcc-10.2.0 -pthread -fPIC -I/usr/include/python3.10 -c foo.c -o build/temp.linux-x86_64-3.10/foo.o
x86_64-pc-linux-gnu-gcc-10.2.0 -pthread -fPIC -I/usr/include/python3.10 -c shared.c -o build/temp.linux-x86_64-3.10/shared.o
x86_64-pc-linux-gnu-gcc-10.2.0 -pthread -fPIC -I/usr/include/python3.10 -c shared.c -o build/temp.linux-x86_64-3.10/shared.o
x86_64-pc-linux-gnu-gcc-10.2.0 -pthread -shared -Wl,-O1 -Wl,--as-needed -Wl,--hash-style=gnu build/temp.linux-x86_64-3.10/foo.o build/temp.linux-x86_64-3.10/shared.o -L/usr/lib64 -o /home/mgorny/git/distutils-build_ext-race/
x86_64-pc-linux-gnu-gcc-10.2.0 -pthread -shared -Wl,-O1 -Wl,--as-needed -Wl,--hash-style=gnu build/temp.linux-x86_64-3.10/bar.o build/temp.linux-x86_64-3.10/shared.o -L/usr/lib64 -o /home/mgorny/git/distutils-build_ext-race/
+ python3.10 -c 'import foo; import bar'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ImportError: /home/mgorny/git/distutils-build_ext-race/ undefined symbol: call_shared
+ echo 'Reproduced at iteration 256'
Reproduced at iteration 256
+ break
Date User Action Args
2022-04-11 14:59:42adminsetgithub: 87625
2021-03-10 07:42:12mgornycreate