This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

Author cjwatson
Recipients cjwatson
Date 2007-12-18.15:41:03
SpamBayes Score 0.0005375431
Marked as misclassified No
Message-id <1197992466.47.0.0612439954222.issue1652@psf.upfronthosting.co.za>
In-reply-to
Content
On Unix, Python sets SIGPIPE to SIG_IGN on startup, because it prefers
to check every write and raise an IOError exception rather than taking
SIGPIPE. This is all well and good for Python itself. However,
non-Python Unix subprocesses generally expect to have SIGPIPE set to the
default action, since that's what will happen if they're started from a
normal shell. If this is not the case, then rather than taking SIGPIPE
when a write fails due to the reading end of a pipe having been closed,
write will return EPIPE and it's up to the process to check that. Many
programs are lazy and fail to check for write errors, but in the
specific case of pipe closure they are fine when invoked from a normal
shell because they rely on taking the signal. Even correctly written
programs that diligently check for write errors will typically produce
different (and confusing) output when SIGPIPE is ignored. For instance,
an example only very slightly adapted from one in the subprocess
documentation:

  $ dd if=/dev/zero of=bigfile bs=1024 seek=10000 count=1
  1+0 records in
  1+0 records out
  1024 bytes (1.0 kB) copied, 0.000176709 seconds, 5.8 MB/s
  $ cat bigfile | head -n0
  $ cat t.py
  from subprocess import Popen, PIPE
  p1 = Popen(["cat", "bigfile"], stdout=PIPE)
  p2 = Popen(["head", "-n0"], stdin=p1.stdout, stdout=PIPE)
  output = p2.communicate()[0]
  $ python t.py
  cat: write error: Broken pipe

In some cases this problem can be much more significant. For instance,
it is very common for shell scripts to rely on SIGPIPE's default action
in order to behave correctly. A year or two ago I ran into this in OS
installer code I was writing in Python, which called some underlying
utility code in shell and C to deal with disk partitioning. In the event
that the Python code failed to handle an exception, the shell script
being run in a subprocess would spiral out of control rather than
cleanly exiting at the first sign of trouble. This actually caused
massive data loss on several testers' systems and required a quick
release to fix
(https://bugs.launchpad.net/ubuntu/+source/ubiquity/+bug/40464 and
https://wiki.ubuntu.com/DapperBeta/PartitionTableCorruption). Now
obviously this was ultimately my fault for failing to catch the
exceptional condition in testing, but this misdesign in Python and the
lack of any documentation of this gotcha really didn't help at the time.
For the record, the fix was to call this in a preexec_fn:

  signal.signal(signal.SIGPIPE, signal.SIG_DFL)

This is an area of Unix that's very easy to get wrong. I've written my
own subprocess handling library in C for another project, and I still
found it non-trivial to track this down when it bit me. Since it
essentially arises due to an implementation detail of the Python
language, I think it should also be Python's responsibility to fix it up
when subprocesses are involved.

There are many ways to invoke subprocesses in Python, but the new,
all-singing, all-dancing one is of course the subprocess module. I think
it would be sufficient for that to do the right thing, or at least have
an option to do so, and it's certainly the easiest place to add the
option. I'm attaching a suggested patch which adds a restore_sigpipe
option and documents it in what seem to be all the relevant places.

Note that nearly all the examples end up with restore_sigpipe enabled.
In some future release of Python, I think this should be the default.
I'm not entirely familiar with migration plan processes in Python; how
should this be done?
Files
File name Uploaded
subprocess-sigpipe.patch cjwatson, 2007-12-18.15:41:03
History
Date User Action Args
2007-12-18 15:41:06cjwatsonsetspambayes_score: 0.000537543 -> 0.0005375431
recipients: + cjwatson
2007-12-18 15:41:06cjwatsonsetspambayes_score: 0.000537543 -> 0.000537543
messageid: <1197992466.47.0.0612439954222.issue1652@psf.upfronthosting.co.za>
2007-12-18 15:41:06cjwatsonlinkissue1652 messages
2007-12-18 15:41:03cjwatsoncreate