Title: subprocess.pipe function
Type: enhancement Stage:
Components: Library (Lib) Versions: Python 3.3
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: Netto, aht, amaury.forgeotdarc, desbma, eric.araujo, matheus.v.portela, pitrou, r.david.murray, vapier, vincent.legoll, vlegoll
Priority: normal Keywords: patch

Created on 2008-08-12 17:23 by tebeka, last changed 2020-03-05 02:01 by vapier.

File name Uploaded Description Edit
pipe.patch tebeka, 2008-08-12 17:23 Patch file for "subprocess.pipe" review vincent.legoll, 2008-08-26 13:02 alternate implementation (not a patch) vincent.legoll, 2008-08-26 13:51 clean, doc & unit test (still not a patch) vincent.legoll, 2008-09-01 14:57 new version (pylint, license, copyright)
Messages (18)
msg71062 - (view) Author: Miki Tebeka (tebeka) * Date: 2008-08-12 17:23
Attached is a patch that add "pipe" command to the "subprocess" module.

pipe(["ls"], ["grep", "test_"]) will return the output of 
"ls | grep test_".
msg71063 - (view) Author: Miki Tebeka (tebeka) * Date: 2008-08-12 17:35
Not sure about the name, maybe "chain" will be better?
msg71978 - (view) Author: Vincent Legoll (vincent.legoll) Date: 2008-08-26 13:02

I was searching for a bug in subprocess
module when I saw your patch.

I was implementing the exact same functionality
and mixed some of your ideas in what I use now,
which is attached...

Feel free to use it
msg71979 - (view) Author: Vincent Legoll (vincent.legoll) Date: 2008-08-26 13:51
Here's a clean version with doc & test

enjoy !
msg72257 - (view) Author: Vincent Legoll (vincent.legoll) Date: 2008-09-01 14:57
- Added "shut pylint up" comment for ** keyword expansion
- Added Copyright & license header
msg72262 - (view) Author: Amaury Forgeot d'Arc (amaury.forgeotdarc) * (Python committer) Date: 2008-09-01 15:13
GPL licenced code is incompatible with the inclusion into python.
And if I am correct, you should sign a contributor agreement. Then the
licence text is not necessary.
msg72318 - (view) Author: Legoll Vincent (vlegoll) Date: 2008-09-02 08:26
On Mon, Sep 1, 2008 at 5:13 PM, Amaury Forgeot d'Arc
<> wrote:
> Amaury Forgeot d'Arc <> added the comment:
> Vincent,
> GPL licenced code is incompatible with the inclusion into python.
> And if I am correct, you should sign a contributor agreement. Then the
> licence text is not necessary.

This is not a patch against python, this is a standalone script, just
implementing the same functionality as the original bug report in a
slightly different way...

But thanks for the input anyways.

If this functionality is really interesting people and agreed to be integrated
into upstream subprocess module, I can rework it the right way, or work
from the original patch from the bug report.
msg114730 - (view) Author: Anh Hai Trinh (aht) Date: 2010-08-23 08:53
I've written a package which can do this with arbitrary redirection in all subcommands (and some more).

You can, for example, do this:

    >>> Pipe(Sh('echo -n foo >&2', {2: 1}), Sh('cat; echo ERROR >&2', {2: os.devnull})).capture(1).read()

The package is at:
msg122557 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2010-11-27 21:01
pipe.patch looks interesting to me.  I would replace **kwargs with a keyword-only argument named stderr, since that’s the only key used.  This requires more tests and docs.
msg125226 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2011-01-03 20:06
I think this would be more useful if you could pass an optional input string (as in communicate()) and if it returned a (stdout, stderr) tuple. Or perhaps even a (return code, stdout, stderr) tuple; alternately, non-zero return codes could raise an exception.
msg249120 - (view) Author: Matheus Vieira Portela (matheus.v.portela) * Date: 2015-08-25 13:20
I just found this open issue and I can work on it. What is left to do before closing it?
msg249124 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2015-08-25 14:03
Thanks for being willing to work on it.

If what is wanted is a way to pipeline shell commands, Python already has that functionality in the pipes module.

So the interesting thing here would be pipelining *non* shell commands, to avoid the shell exploits that using a shell pipeline invites.

The pipes module already has a worked out API, so perhaps it would be useful to see about re-implementing pipe's command execution using subprocess, and expand the API to allow for argv style command specification that would be fed to subprocess using the default shell=False.  This would also presumably allow pipes to be used when there's no bash shell available.

The downside is that we might break current uses of pipes if we replace the shell version of the pipelining with subprocess shell=True, while using subprocess only if the command specifications are argv lists would result in code with a split personality.  But if I were working on it I'd experiment with that approach to see if it made sense.

Other core devs may have other opinions on this :)
msg249222 - (view) Author: Matheus Vieira Portela (matheus.v.portela) * Date: 2015-08-27 03:23
Let me check whether I understood you suggestion...

What you are saying is that it is already possible to pipeline shell commands through subprocess, such as in the following line?'ls -l | grep Music', shell=True)

However, this requires the command to be executed in a shell environment. Hence, it would be a good idea to extend it to non-shell command execution. Is this right?
msg249232 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2015-08-27 13:50
No, I'm talking about
msg249239 - (view) Author: Matheus Vieira Portela (matheus.v.portela) * Date: 2015-08-27 16:52
Oh, I see... The pipes module uses os.system and os.popen to execute commands under a "/bin/sh" environment. So you are proposing to re-implement it with subprocess in order to execute commands without shell and, also, extend by accepting argv-style parameters. Is this right this time?

How would an argv-style pipe look like?
msg249241 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2015-08-27 16:58
Exactly like the string pipes API, but cmd would be a list of arguments instead of a string.  That would then become the argument list to the subprocess stage.

It is an interesting question whether we'd want to allow mixing stage types.  I'd say no (at least initially, if people complain we can add it as a feature later, but we can't subtract it if we implement it now and decide it is a bad idea after putting it in the field).

Now, keep in mind that you've only got my opinion so far that this is even a good idea :)
msg249254 - (view) Author: Matheus Vieira Portela (matheus.v.portela) * Date: 2015-08-27 19:10
Sure. I'll leave this issue for a while before others can emit their opinions.
msg363412 - (view) Author: Mike Frysinger (vapier) Date: 2020-03-05 02:01
this would have been nice to have more than once in my personal projects, and in build/infra tooling for Chromium OS.  our alternatives so far have been the obvious ones: use to capture & return the output, then manually feed that as the input to the next command, and so on.  we know it has obvious overhead so we've avoided with large output.

we strongly discourage/ban attempts to write shell code, so the vast majority of our commands are argv style (list of strings), so the pipes module wouldn't help us.

handling of SIGPIPE tends to be where things get tricky.  that'll have to be handled by the API explicitly i think.
Date User Action Args
2020-03-05 02:01:55vapiersetnosy: + vapier
messages: + msg363412
2015-09-10 18:27:11desbmasetnosy: + desbma
2015-08-27 19:10:49matheus.v.portelasetmessages: + msg249254
2015-08-27 16:58:37r.david.murraysetmessages: + msg249241
2015-08-27 16:52:16matheus.v.portelasetmessages: + msg249239
2015-08-27 13:50:24r.david.murraysetmessages: + msg249232
2015-08-27 03:23:52matheus.v.portelasetmessages: + msg249222
2015-08-25 14:03:53r.david.murraysetnosy: + r.david.murray
messages: + msg249124
2015-08-25 13:20:14matheus.v.portelasetnosy: + matheus.v.portela
messages: + msg249120
2011-01-03 23:59:57tebekasetnosy: - tebeka
2011-01-03 20:06:02pitrousetnosy: + pitrou

messages: + msg125226
versions: + Python 3.3, - Python 3.2
2010-11-27 21:01:19eric.araujosetnosy: + eric.araujo
messages: + msg122557
2010-08-23 08:53:40ahtsetnosy: + aht
messages: + msg114730
2010-08-23 05:00:25Nettosetnosy: + Netto
2010-08-21 23:07:50georg.brandlsetversions: + Python 3.2, - Python 3.1, Python 2.7
2008-09-02 08:26:01vlegollsetnosy: + vlegoll
messages: + msg72318
2008-09-01 15:13:54amaury.forgeotdarcsetnosy: + amaury.forgeotdarc
messages: + msg72262
2008-09-01 14:57:59vincent.legollsetfiles: +
messages: + msg72257
2008-08-26 13:55:47pitrousetpriority: normal
versions: + Python 3.1, Python 2.7, - Python 2.6, Python 3.0
2008-08-26 13:51:19vincent.legollsetfiles: +
messages: + msg71979
2008-08-26 13:02:31vincent.legollsetfiles: +
nosy: + vincent.legoll
messages: + msg71978
2008-08-12 17:35:22tebekasetmessages: + msg71063
2008-08-12 17:23:58tebekacreate