classification
Title: Disallow fork in a subinterpreter.
Type: Stage: resolved
Components: Interpreter Core Versions: Python 3.8
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: eric.snow Nosy List: eric.snow, gregory.p.smith, hroncok, vstinner
Priority: normal Keywords: patch

Created on 2018-09-12 21:04 by eric.snow, last changed 2019-08-29 16:38 by vstinner. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 9279 merged eric.snow, 2018-09-13 19:21
Messages (10)
msg325181 - (view) Author: Eric Snow (eric.snow) * (Python committer) Date: 2018-09-12 21:04
os.fork() potentially has some problematic behavior when called from a subinterpreter.  In additional to the normal fork+threads madness, there's the question of what to do with existing subinterpreters.  The simplest solution is to simply disallow fork in a subinterpreter and then wipe out all subinterpreters in the child process post-fork (if os.fork() called in the main interpreter).
msg325302 - (view) Author: Gregory P. Smith (gregory.p.smith) * (Python committer) Date: 2018-09-13 23:05
+1 agreed.  this is the simplest approach to start with.

Code to restrict: os.fork itself and disallowing the use of preexec_fn on subprocess within subinterpreters.

feel free to ignore preexec_fn in subprocess for the time being if desired, we already promise people that it is a bad idea legacy API that we don't want, guaranteed to cause undiagnosable problems and deadlocks at times. :)
msg325400 - (view) Author: Eric Snow (eric.snow) * (Python committer) Date: 2018-09-14 21:17
New changeset 5903296045b586b9cd1fce0b1e02caf896028d1d by Eric Snow in branch 'master':
bpo-34651: Only allow the main interpreter to fork. (gh-9279)
https://github.com/python/cpython/commit/5903296045b586b9cd1fce0b1e02caf896028d1d
msg343364 - (view) Author: Miro Hrončok (hroncok) * Date: 2019-05-24 10:28
It appears that as a result of this, subprocess.Popen cannot be called from within a subinterpreter either. Is that an obvious and desired limitation?
msg343369 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-05-24 10:52
I reopen the issue to let Eric answer ;-)

If the behavior is deliberate, maybe it should just be documented somewhere?
msg343415 - (view) Author: Gregory P. Smith (gregory.p.smith) * (Python committer) Date: 2019-05-24 17:48
I'd start by documenting the limitation and keeping a future feature request of "Allow subprocess to work from subinterpreters".  That is doable.  Supporting os.fork() is not.
msg343421 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-05-24 19:25
> I'd start by documenting the limitation and keeping a future feature request of "Allow subprocess to work from subinterpreters".  That is doable.  Supporting os.fork() is not.

subprocess is able to use os.posix_spawn() in Python 3.8. In that case, it's not limited by os.fork() check ;-) Windows isn't limited neither, CreateProcess() isn't limited in subinterpreters.
msg345599 - (view) Author: Eric Snow (eric.snow) * (Python committer) Date: 2019-06-14 15:49
FYI, I plan on looking into this either today or next Friday.
msg345603 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-06-14 16:01
See also bpo-37266: "Daemon threads must be forbidden in subinterpreters".
msg350652 - (view) Author: Miro Hrončok (hroncok) * Date: 2019-08-28 04:34
The problem with subprocess.Popen has been fixed in https://bugs.python.org/issue37951
History
Date User Action Args
2019-08-29 16:38:15vstinnersetstatus: open -> closed
resolution: fixed
2019-08-28 04:34:40hroncoksetmessages: + msg350652
2019-06-14 16:01:52vstinnersetmessages: + msg345603
2019-06-14 15:49:42eric.snowsetmessages: + msg345599
2019-05-24 19:25:25vstinnersetmessages: + msg343421
2019-05-24 17:48:04gregory.p.smithsetmessages: + msg343415
2019-05-24 10:52:31vstinnersetstatus: closed -> open

nosy: + vstinner
messages: + msg343369

resolution: fixed -> (no value)
2019-05-24 10:28:45hroncoksetnosy: + hroncok
messages: + msg343364
2018-09-14 21:18:10eric.snowsetstatus: open -> closed
resolution: fixed
stage: patch review -> resolved
2018-09-14 21:17:31eric.snowsetmessages: + msg325400
2018-09-13 23:05:46gregory.p.smithsetmessages: + msg325302
2018-09-13 19:21:43eric.snowsetkeywords: + patch
stage: needs patch -> patch review
pull_requests: + pull_request8710
2018-09-12 21:04:23eric.snowcreate