classification
Title: subprocess.Popen pipeline example code in the documentation is lacking
Type: behavior Stage: needs patch
Components: Documentation Versions: Python 3.1, Python 3.2, Python 2.7
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: gregory.p.smith Nosy List: docs@python, eric.araujo, gregory.p.smith, rosslagerwall, steven.k.wong
Priority: normal Keywords:

Created on 2010-01-11 21:26 by steven.k.wong, last changed 2011-02-05 21:49 by gregory.p.smith. This issue is now closed.

Messages (4)
msg97606 - (view) Author: Steven K. Wong (steven.k.wong) Date: 2010-01-11 21:26
The example code at http://www.python.org/doc/2.6.2/library/subprocess.html#replacing-shell-pipeline should be updated to add the close() call noted below:

output=`dmesg | grep hda`
==>
p1 = Popen(["dmesg"], stdout=PIPE)
p2 = Popen(["grep", "hda"], stdin=p1.stdout, stdout=PIPE)
p1.stdout.close()  # <----- this line is not in the doc, but we should add it
output = p2.communicate()[0]

The parent process does not use p1.stdout after passing it to p2, so it should close p1.stdout after that. If it is not closed, there is a possibility for p1's process to never terminate if p2's process fails. The following code demonstrate this situation:

from subprocess import Popen, PIPE
import time
# p1 is a program that writes lots of data to the pipe
cmd = ['cat', 'a_large_file']
p1 = Popen(cmd, stdout=PIPE)
# p2 is a program that fails without reading much data from the pipe
cmd = ['python', '-c', 'import time; time.sleep(5); asdf']
p2 = Popen(cmd, stdin=p1.stdout, stdout=PIPE)
output = p2.communicate()[0]
while p1.poll() is None:  # it loops forever
    print 'sleep a bit'
    time.sleep(1)
msg124657 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2010-12-26 01:58
As a non-expert user of subprocess, calling close before communicate seems strange to me.  Does the example code with a bug work if close is called after communicate?

In the 3.2 docs, we could update the example to use a with statement, unless it is deemed a distraction for this simple introductory section.
msg127787 - (view) Author: Ross Lagerwall (rosslagerwall) (Python committer) Date: 2011-02-03 10:01
The docs should be updated. This has been noted in msg54949 and http://www.enricozini.org/2009/debian/python-pipes/

Perhaps this example will make it clear:
import subprocess

p1 = subprocess.Popen(["yes"], stdout=subprocess.PIPE)
p2 = subprocess.Popen(["head"], stdin=p1.stdout, stdout=subprocess.PIPE)
#p1.stdout.close()
p1.wait()

This example hangs. "yes" writes to "head" and head reads the first 10 lines and then exits. But, "yes" does not receive a SIGPIPE because the python process still has a p1.stdout open. Thus, p1.stdout should be closed after being passed to p2.
msg128028 - (view) Author: Gregory P. Smith (gregory.p.smith) * (Python committer) Date: 2011-02-05 21:49
documentation updated in r88352.  thanks!
History
Date User Action Args
2011-02-05 21:49:27gregory.p.smithsetstatus: open -> closed
nosy: gregory.p.smith, eric.araujo, steven.k.wong, docs@python, rosslagerwall
messages: + msg128028

assignee: docs@python -> gregory.p.smith
resolution: fixed
2011-02-03 10:01:29rosslagerwallsetnosy: + rosslagerwall
messages: + msg127787
2010-12-26 01:58:58eric.araujosetversions: + Python 3.1, Python 2.7, Python 3.2, - Python 2.6
nosy: + gregory.p.smith, eric.araujo, - georg.brandl

messages: + msg124657

type: behavior
stage: needs patch
2010-08-04 21:38:51BreamoreBoysetassignee: georg.brandl -> docs@python

nosy: + docs@python
2010-01-11 21:26:43steven.k.wongcreate