Author vstinner
Recipients jakirkham, vstinner
Date 2022-01-15.00:10:58
SpamBayes Score -1.0
Marked as misclassified Yes
Message-id <1642205458.33.0.328228361035.issue46367@roundup.psfhosted.org>
In-reply-to
Content
> It appears the `multiprocessing`'s "spawn" mode doesn't actually use POSIX spawn, but instead uses fork+exec[1].

The documentation doesn't pretend to use posix_spawn(). It only says: "starts a fresh python interpreter process".
https://docs.python.org/dev/library/multiprocessing.html#contexts-and-start-methods

I suggest to close the issue as "not a bug". I don't see anything wrong in the current documentation.

--

posix_spawn() is a function of the C library. It is implemented as fork+exec on most operating systems. I'm only aware of macOS which has a dedicated syscall. Well, posix_spawn() implementation is usually faster thanks to some optimizations.

Python has os.posix_spawn() since Python 3.8.

The subprocess can use os.posix_spawn() on Linux under some conditions:
https://docs.python.org/dev/whatsnew/3.8.html#optimizations

Sadly, it's not used by default, since close_fds=True remains subprocess.Popen() default.

I'm open to use it on more platforms. os.posix_spawn() can only be used if it reports properly errors to the parent process, and some other things and bugs. It's a complex function!

--

Oh, about multiprocessing. Well, someone has to propose a patch! I don't know why multiprocessing uses directly _posixsubprocess.fork_exec() rather than the subprocess module. It's also a complex module with many specific constraints.

posix_spawn() looks nice, but it cannot be used in many cases :-(
History
Date User Action Args
2022-01-15 00:10:58vstinnersetrecipients: + vstinner, jakirkham
2022-01-15 00:10:58vstinnersetmessageid: <1642205458.33.0.328228361035.issue46367@roundup.psfhosted.org>
2022-01-15 00:10:58vstinnerlinkissue46367 messages
2022-01-15 00:10:58vstinnercreate