classification
Title: asyncio.subprocess: connect pipes of two programs
Type: enhancement Stage: resolved
Components: asyncio Versions: Python 3.5
process
Status: closed Resolution: out of date
Dependencies: Superseder:
Assigned To: Nosy List: cvrebert, giampaolo.rodola, gvanrossum, vstinner, yselivanov
Priority: normal Keywords:

Created on 2014-03-27 23:10 by vstinner, last changed 2017-06-28 01:31 by vstinner. This issue is now closed.

Messages (7)
msg214994 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2014-03-27 23:10
With the current asyncio API, it's not possible to implement the shell command "ls | wc -l" in Python: connect the stdin of a consumer to the stdin of a producer.
msg214996 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2014-03-27 23:13
How do you do that with the subprocess module, and why doesn't that work with asyncio?
msg214999 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2014-03-27 23:22
> How do you do that with the subprocess module

Something like that:
---
import subprocess
ls = subprocess.Popen(["ls", "-1"], stdout=subprocess.PIPE)
wc = subprocess.Popen(["wc", "-l"], stdin=ls.stdout)
ls.wait()
wc.wait()
---

> and why doesn't that work with asyncio?

It's possible with the two methods of an event loop, but I'm requesting this feature for the high-level API: asyncio.subprocess.

create_subprocess_shell("cat", stdout=subprocess.PIPE) starts immediatly to consume stdout, before I can connect the pipe to "wc" stdin. Currently, the asyncio.subprocess is designed to be able to write:
---
proc = yield from create_subprocess_exec("ls", stdout=subprocess.PIPE)
stdout, _ = yield from proc.communicate()
---
msg215000 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2014-03-27 23:25
Oh, I see. Given that it is possible to do using event loop methods, why don't you write up a complete implementation and then propose to add that to the asyncio.subprocess module?
msg215006 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2014-03-27 23:47
> Oh, I see. Given that it is possible to do using event loop methods, why don't you write up a complete implementation and then propose to add that to the asyncio.subprocess module?

I don't know that a whole new implementation is needed. I guess that a
simpler change can be done on SubprocessStreamProtocol to ask to not
consume the pipe (don't attach a StreamReader to the pipe transport).

I still want to use streams at the end, so use the "high-level API". Example:
---
ls = yield from create_subprocess_exec("ls", stdout=subprocess.PIPE)
grep = yield from create_subprocess_exec("grep", "-v", "-F", ".old",
stdin=ls.stdin, stdout=subprocess.PIPE)
stdout, _ = yield from grep.communicate()
---
msg215009 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2014-03-28 00:20
But how is the first call supposed to know that you don't want a StreamReader? Or the second that you do want one? Maybe we need a new constant instead of PIPE that means "leave it hanging"?
msg297118 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-06-28 01:31
I don't see how to implement this idea, not if it's doable. Anyway, I'm not interested anymore to implement the idea, so I just close this old issue.
History
Date User Action Args
2017-06-28 01:31:56vstinnersetstatus: open -> closed
resolution: out of date
messages: + msg297118

stage: resolved
2014-06-06 11:41:52vstinnersetcomponents: + asyncio
2014-03-28 05:43:52cvrebertsetnosy: + cvrebert
2014-03-28 01:06:13giampaolo.rodolasetnosy: + giampaolo.rodola
2014-03-28 00:20:53gvanrossumsetmessages: + msg215009
2014-03-27 23:47:54vstinnersetmessages: + msg215006
2014-03-27 23:25:10gvanrossumsetmessages: + msg215000
2014-03-27 23:22:07vstinnersetmessages: + msg214999
2014-03-27 23:13:35gvanrossumsetmessages: + msg214996
2014-03-27 23:10:39vstinnercreate