classification
Title: subprocess should have an option to restore SIGPIPE to default action
Type: behavior Stage: test needed
Components: Library (Lib) Versions: Python 2.7
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: gregory.p.smith Nosy List: anacrolix, barry, cjwatson, flox, georg.brandl, gregory.p.smith, inducer, jwilk, loewis, mitar, pitrou, r.david.murray, rcronk
Priority: normal Keywords:

Created on 2007-12-18 15:41 by cjwatson, last changed 2014-01-14 20:08 by jwilk. This issue is now closed.

Files
File name Uploaded Description Edit
subprocess-sigpipe.patch cjwatson, 2007-12-18 15:41
Messages (16)
msg58750 - (view) Author: Colin Watson (cjwatson) Date: 2007-12-18 15:41
On Unix, Python sets SIGPIPE to SIG_IGN on startup, because it prefers
to check every write and raise an IOError exception rather than taking
SIGPIPE. This is all well and good for Python itself. However,
non-Python Unix subprocesses generally expect to have SIGPIPE set to the
default action, since that's what will happen if they're started from a
normal shell. If this is not the case, then rather than taking SIGPIPE
when a write fails due to the reading end of a pipe having been closed,
write will return EPIPE and it's up to the process to check that. Many
programs are lazy and fail to check for write errors, but in the
specific case of pipe closure they are fine when invoked from a normal
shell because they rely on taking the signal. Even correctly written
programs that diligently check for write errors will typically produce
different (and confusing) output when SIGPIPE is ignored. For instance,
an example only very slightly adapted from one in the subprocess
documentation:

  $ dd if=/dev/zero of=bigfile bs=1024 seek=10000 count=1
  1+0 records in
  1+0 records out
  1024 bytes (1.0 kB) copied, 0.000176709 seconds, 5.8 MB/s
  $ cat bigfile | head -n0
  $ cat t.py
  from subprocess import Popen, PIPE
  p1 = Popen(["cat", "bigfile"], stdout=PIPE)
  p2 = Popen(["head", "-n0"], stdin=p1.stdout, stdout=PIPE)
  output = p2.communicate()[0]
  $ python t.py
  cat: write error: Broken pipe

In some cases this problem can be much more significant. For instance,
it is very common for shell scripts to rely on SIGPIPE's default action
in order to behave correctly. A year or two ago I ran into this in OS
installer code I was writing in Python, which called some underlying
utility code in shell and C to deal with disk partitioning. In the event
that the Python code failed to handle an exception, the shell script
being run in a subprocess would spiral out of control rather than
cleanly exiting at the first sign of trouble. This actually caused
massive data loss on several testers' systems and required a quick
release to fix
(https://bugs.launchpad.net/ubuntu/+source/ubiquity/+bug/40464 and
https://wiki.ubuntu.com/DapperBeta/PartitionTableCorruption). Now
obviously this was ultimately my fault for failing to catch the
exceptional condition in testing, but this misdesign in Python and the
lack of any documentation of this gotcha really didn't help at the time.
For the record, the fix was to call this in a preexec_fn:

  signal.signal(signal.SIGPIPE, signal.SIG_DFL)

This is an area of Unix that's very easy to get wrong. I've written my
own subprocess handling library in C for another project, and I still
found it non-trivial to track this down when it bit me. Since it
essentially arises due to an implementation detail of the Python
language, I think it should also be Python's responsibility to fix it up
when subprocesses are involved.

There are many ways to invoke subprocesses in Python, but the new,
all-singing, all-dancing one is of course the subprocess module. I think
it would be sufficient for that to do the right thing, or at least have
an option to do so, and it's certainly the easiest place to add the
option. I'm attaching a suggested patch which adds a restore_sigpipe
option and documents it in what seem to be all the relevant places.

Note that nearly all the examples end up with restore_sigpipe enabled.
In some future release of Python, I think this should be the default.
I'm not entirely familiar with migration plan processes in Python; how
should this be done?
msg61300 - (view) Author: Georg Brandl (georg.brandl) * (Python committer) Date: 2008-01-20 14:27
Raising priority.

What incompatibilities could occur if SIGPIPE is restored by default?
msg61952 - (view) Author: Colin Watson (cjwatson) Date: 2008-02-01 09:41
To be quite honest I can't think of any incompatibilities that wouldn't
have the basic result of improving matters. I put the migration stuff in
my bug report in case somebody else could, because I don't want the bug
fix to stall on that.

My preference would be for Python to move to behaviour equivalent to
restore_sigpipe=True in the next release, but I would rather that it
gained restore_sigpipe with the wrong default than that it didn't gain
it at all.
msg61953 - (view) Author: Georg Brandl (georg.brandl) * (Python committer) Date: 2008-02-01 09:41
Martin, what do you think?
msg62327 - (view) Author: Martin v. Löwis (loewis) * (Python committer) Date: 2008-02-12 18:57
The patch as it stands (subprocess-sigpipe.patch) definitely can't go
into 2.5.x: it introduces a new feature.

It's not clear to me whether Colin intended to target it for 2.5.x, as
it is against the trunk. For the trunk, the patch is fine.

Regargeting for 2.6. Colin, if that wasn't your intention, please speak up.
msg65968 - (view) Author: Colin Watson (cjwatson) Date: 2008-04-29 14:53
2.6 is fine if that's what the release process dictates; I don't want it
to be lost, that's all.
msg65979 - (view) Author: Georg Brandl (georg.brandl) * (Python committer) Date: 2008-04-29 19:44
Martin, what do you think?
msg90004 - (view) Author: Colin Watson (cjwatson) Date: 2009-07-02 09:12
Is there anything more I can do to move this along? Thanks.
msg90040 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2009-07-03 01:37
Find someone to review the patch and post their evaluation here.  It
doesn't have to be a committer, though that would be even better.
msg90041 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2009-07-03 01:41
Hmm.  Looks like the patch is missing a unit test for the new feature.
msg99359 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2010-02-15 10:47
I think restore_sigpipe=True would be a reasonable default.
As RDM says, adding an unit test would be better, but it may be difficult to do so (we probably can't spawn Python itself since it will change the default SIGPIPE handler at startup).
Looking at the patch, you don't need to import `signal` again: it's already imported at the top level.
msg100047 - (view) Author: Gregory P. Smith (gregory.p.smith) * (Python committer) Date: 2010-02-24 17:09
Yes, we set SIG_IGN in Python/pythonrun.c initsigs() on SIGPIPE, SIGXFZ & SIGXFSZ.  

We should restore these to SIG_DFL prior to exec by default.  I'm in the middle of working on subprocess improvements that touch this code, I'll include this.
msg101040 - (view) Author: Gregory P. Smith (gregory.p.smith) * (Python committer) Date: 2010-03-14 06:56
A restore_signals parameter was added in py3k r78946.

I'm leaving this issue open as it needs backporting to 2.7.
msg101055 - (view) Author: Florent Xicluna (flox) * (Python committer) Date: 2010-03-14 12:35
Fixed an oversight on the switch None ==> -1 which prevents compilation on some buildbot. See r78961.
msg115364 - (view) Author: Mitar (mitar) Date: 2010-09-02 10:56
GHC Haskell compiler is currently opting for a different solution: installing an default empty handler which is cleared by exec automatically and signal handler is restored back to SIG_DFL:

http://hackage.haskell.org/trac/ghc/ticket/4274
msg115390 - (view) Author: Gregory P. Smith (gregory.p.smith) * (Python committer) Date: 2010-09-02 15:55
closing because it is too late to backport this to 2.7.  It is available as a backport in http://code.google.com/p/python-subprocess32/.

As for the idea of not using SIG_IGN and installing a default no-op handler, that is another approach.

signal.getsignal would need to be able to return it so that software wanting to temporarily handle its own sigpipes could restore that behavior afterwards.
History
Date User Action Args
2014-01-14 20:08:30jwilksetnosy: + jwilk
2012-05-30 11:53:56anacrolixsetnosy: + anacrolix
2010-09-27 15:33:19barrysetnosy: + barry
2010-09-02 15:55:45gregory.p.smithsetstatus: open -> closed
resolution: fixed
messages: + msg115390
2010-09-02 10:56:33mitarsetnosy: + mitar
messages: + msg115364
2010-07-22 18:27:18rcronksetnosy: + rcronk
2010-03-14 12:35:13floxsetnosy: + flox
messages: + msg101055
2010-03-14 06:56:18gregory.p.smithsetmessages: + msg101040
versions: - Python 3.2
2010-02-24 17:42:46gregory.p.smithlinkissue1736483 superseder
2010-02-24 17:09:51gregory.p.smithsetassignee: loewis -> gregory.p.smith

messages: + msg100047
nosy: + gregory.p.smith
2010-02-15 10:47:23pitrousetnosy: + pitrou
messages: + msg99359
2010-02-15 10:42:59pitroulinkissue7933 superseder
2009-07-03 01:41:17r.david.murraysetmessages: + msg90041
stage: patch review -> test needed
2009-07-03 01:37:58r.david.murraysetversions: + Python 2.7, Python 3.2, - Python 2.6
nosy: + r.david.murray

messages: + msg90040

stage: patch review
2009-07-02 12:20:04inducersetnosy: + inducer
2009-07-02 09:12:37cjwatsonsetmessages: + msg90004
2008-04-29 19:44:32georg.brandlsetassignee: loewis
messages: + msg65979
2008-04-29 14:53:35cjwatsonsetmessages: + msg65968
2008-02-12 18:57:18loewissetpriority: high -> normal
assignee: loewis -> (no value)
messages: + msg62327
versions: + Python 2.6, - Python 2.5
2008-02-01 09:41:56georg.brandlsetassignee: loewis
messages: + msg61953
nosy: + loewis
2008-02-01 09:41:06cjwatsonsetmessages: + msg61952
2008-01-20 14:27:52georg.brandlsetpriority: high
type: behavior
messages: + msg61300
severity: normal -> major
nosy: + georg.brandl
2007-12-18 15:41:06cjwatsoncreate