Title: Refactor subprocess.Popen to let a subclass handle IO asynchronously
Type: enhancement Stage: patch review
Components: Versions: Python 3.8
Status: open Resolution:
Dependencies: Superseder:
Assigned To: gregory.p.smith Nosy List: giampaolo.rodola, gregory.p.smith, martius, pitrou, yselivanov
Priority: normal Keywords: patch

Created on 2016-09-27 14:26 by martius, last changed 2018-05-15 21:06 by python-dev.

File name Uploaded Description Edit
popen_execute_child_refactoring.patch martius, 2016-09-27 14:26 popen_execut_child_refactoring.patch, should work with python 3.4+ review
Pull Requests
URL Status Linked Edit
PR 6878 closed python-dev, 2018-05-15 21:06
Messages (4)
msg277517 - (view) Author: Martin Richard (martius) * Date: 2016-09-27 14:26

Currently, subprocess.Popen performs blocking IO in its constructor (at least on Unix): it reads on a pipe in order to detect outcome of the pre-exec and exec phase in the new child. There is no way yet to modify this behavior as this blocking call is part of a long Popen._execute_child() method.

This is a problem in asyncio (asyncio.subprocess_exec and asyncio.subprocess_shell).

I would like to submit a patch which breaks Popen.__init__() and Popen._execute_child() in several methods so it becomes possible to avoid blocking calls (read on pipe and waitpid) by overriding a few private methods without duplicating too much code. The goal is to use it in asyncio, as described in this pull request (which currently monkey-patches Popen):

This patch only targets the unix implementation.

Thanks for your feedback.
msg298358 - (view) Author: Giampaolo Rodola' (giampaolo.rodola) * (Python committer) Date: 2017-07-14 16:09
Is the main goal to just make Popen.__init__ non-blocking? If not, #1191964 aims at providing non-blocking reads/writes (including on Windows).
msg300839 - (view) Author: Martin Richard (martius) * Date: 2017-08-25 14:01
Yes, the goal is to isolate the blocking IO in __init__ into other methods so Popen can be subclassed in asyncio.

The end goal is to ensure that when asyncio calls Popen(), it doesn't block the process. In the context of asyncio, there's no need to make Popen() IOs non-blocking as they will be performed with the asyncio API (rather than the IO methods provided by the Popen object).
msg301024 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2017-08-30 12:35
By the way, we're using GitHub PRs now for development, though you can still submit patch files if you prefer that.
Date User Action Args
2018-05-15 21:06:57python-devsetpull_requests: + pull_request6550
2018-05-14 19:06:36gregory.p.smithsetversions: + Python 3.8, - Python 3.7
2017-08-30 21:09:21gregory.p.smithsetassignee: gregory.p.smith
2017-08-30 12:35:13pitrousetnosy: + pitrou
messages: + msg301024
2017-08-30 12:34:23pitrousetnosy: + gregory.p.smith
2017-08-30 11:48:06pitrousetstage: patch review
2017-08-25 14:01:26martiussetmessages: + msg300839
2017-07-14 16:09:07giampaolo.rodolasetnosy: + giampaolo.rodola
messages: + msg298358
2016-09-30 18:44:38terry.reedysetversions: + Python 3.7, - Python 3.6
2016-09-30 18:43:59terry.reedysetnosy: + yselivanov
2016-09-27 14:26:43martiuscreate