Message363165
> And setup.py does use use multiple processes via a thread pool, where each thread calls fork(), if it detects that "make" was invoked with the parallel (-j) option.
Oh ok, now I get it :-) setup.py uses distutils to build extensions and distutils.command.build_ext uses concurrent.futures for parallelism:
if self.parallel:
self._build_extensions_parallel()
else:
self._build_extensions_serial()
with:
def _build_extensions_parallel(self):
workers = self.parallel
if self.parallel is True:
workers = os.cpu_count() # may return None
try:
from concurrent.futures import ThreadPoolExecutor
except ImportError:
workers = None
if workers is None:
self._build_extensions_serial()
return
with ThreadPoolExecutor(max_workers=workers) as executor:
futures = [executor.submit(self.build_extension, ext)
for ext in self.extensions]
for ext, fut in zip(self.extensions, futures):
with self._filter_build_errors(ext):
fut.result()
The problem is not concurrent.futures but the job submitted to the executor: build_extension() uses distutils.spawn() which is unsafe.
distutils.spawn() is the root issue if I understood correctly. It must be rewritten with subprocess. |
|
Date |
User |
Action |
Args |
2020-03-02 12:22:43 | vstinner | set | recipients:
+ vstinner, gvanrossum, pitrou, eric.araujo, dstufft, Elad Lahav |
2020-03-02 12:22:43 | vstinner | set | messageid: <1583151763.95.0.464522577309.issue39763@roundup.psfhosted.org> |
2020-03-02 12:22:43 | vstinner | link | issue39763 messages |
2020-03-02 12:22:43 | vstinner | create | |
|