Author vstinner
Recipients Elad Lahav, dstufft, eric.araujo, gvanrossum, pitrou, vstinner
Date 2020-03-02.12:22:43
SpamBayes Score -1.0
Marked as misclassified Yes
Message-id <1583151763.95.0.464522577309.issue39763@roundup.psfhosted.org>
In-reply-to
Content
> And setup.py does use use multiple processes via a thread pool, where each thread calls fork(), if it detects that "make" was invoked with the parallel (-j) option.

Oh ok, now I get it :-) setup.py uses distutils to build extensions and distutils.command.build_ext uses concurrent.futures for parallelism:

        if self.parallel:
            self._build_extensions_parallel()
        else:
            self._build_extensions_serial()

with:

    def _build_extensions_parallel(self):
        workers = self.parallel
        if self.parallel is True:
            workers = os.cpu_count()  # may return None
        try:
            from concurrent.futures import ThreadPoolExecutor
        except ImportError:
            workers = None

        if workers is None:
            self._build_extensions_serial()
            return

        with ThreadPoolExecutor(max_workers=workers) as executor:
            futures = [executor.submit(self.build_extension, ext)
                       for ext in self.extensions]
            for ext, fut in zip(self.extensions, futures):
                with self._filter_build_errors(ext):
                    fut.result()

The problem is not concurrent.futures but the job submitted to the executor: build_extension() uses distutils.spawn() which is unsafe.

distutils.spawn() is the root issue if I understood correctly. It must be rewritten with subprocess.
History
Date User Action Args
2020-03-02 12:22:43vstinnersetrecipients: + vstinner, gvanrossum, pitrou, eric.araujo, dstufft, Elad Lahav
2020-03-02 12:22:43vstinnersetmessageid: <1583151763.95.0.464522577309.issue39763@roundup.psfhosted.org>
2020-03-02 12:22:43vstinnerlinkissue39763 messages
2020-03-02 12:22:43vstinnercreate