I have this example code to illustrate a problem I am having with python3. It works fine with python 2.6 and 2.7 but does not with python 3.1.

from __future__ import print_function
import os, subprocess, signal

def signal_handler( signum, frame ):

    print( "PID: %s" % pid )
    print( "Timed out! Process %s killed, max exec time (%ss) exceeded" % (pid, timeTol ) )
    os.kill( int( pid ), 15 )
    raise Exception( "Taking too long to finish... aborting!" )

if __name__ == '__main__':

    timeTol = 5

    cmd = 'find /'

    signal.signal(signal.SIGALRM, signal_handler)
    signal.alarm(timeTol)

    p = subprocess.Popen(cmd, shell=True, stderr = subprocess.STDOUT, stdout = subprocess.PIPE)
    pid = p.pid

    out = str( p.communicate()[0].decode() )
    print(out)


Executing it:

time python3.1 timout.py
PID: 27687
Timed out! Process 27687 killed, max exec time (5s) exceeded
Traceback (most recent call last):
  File "timout.py", line 23, in <module>
    out = str( p.communicate()[0].decode() )
  File "/sw/lib/python3.1/subprocess.py", line 719, in communicate
    stdout = self.stdout.read()
  File "timout.py", line 9, in signal_handler
    raise Exception( "Taking too long to finish... aborting!" )
Exception: Taking too long to finish... aborting!
python3.1 timout.py  0.52s user 3.88s system 19% cpu 22.841 total

#### It prints essentially the same thing with a *very* *big* difference it takes 22 seconds and actually the alarm only works when the whole task ('find /') is finished.

It's due to the way the python interpreter handles signals: when the signal is received, python runs a stub signal handler that just sets a flag indicating that the signal has been received: the actual handler is executed later, synchronously, mainly from the main evaluation loop.
The problem here is that since the main process is performing a read until EOF is reached (i.e. until the subprocess exits), the C code loops around the read(2) call without giving a chance for the handler to run.
A patch for is similar issue has been applied, see http://bugs.python.org/issue9617#

But it only fixed writes, not reads.
I think _bufferedreader_read_all and _bufferedreader_read_generic should be fixed too to call PyErr_CheckSignals().

Antoine?

Charles-François' analysis seems to be right. Note that the actual issue here is that read() always succeeds, returning a partial result (because you're executing a command, 'find /', which outputs a lot of data). If read() were interrupted before anything could be read, it would return EINTR and the handler would get executed immediately.

Anyone wants to propose a patch + tests?

> Antoine Pitrou <pitrou@free.fr> added the comment:
>
> Charles-François' analysis seems to be right. Note that the actual issue here is that read() always succeeds, returning a partial result (because you're executing a command, 'find /', which outputs a lot of data). If read() were interrupted before anything could be read, it would return EINTR and the handler would get executed immediately.
>
> Anyone wants to propose a patch + tests?
>

Attached is a tentative patch: it checks for pending signals inside
_bufferedreader_read_generic, fileio_readall, and rawiobase_readall.
I tested quickly on regular files and on pipes, and handlers are now
called on time. As a an obvious bonus, it's also easier to interrupt a
script doing I/O with a KeyboardInterrupt (SIGINT).
If nothing seems blatantly wrong about this patch, I will go ahead and
try to write some tests for this.

Charles

> ----------
> versions: +Python 3.2
>
> _______________________________________
> Python tracker <report@bugs.python.org>
> <http://bugs.python.org/issue9504>
> _______________________________________
>

> Attached is a tentative patch: it checks for pending signals inside
> _bufferedreader_read_generic, fileio_readall, and rawiobase_readall.
> I tested quickly on regular files and on pipes, and handlers are now
> called on time. As a an obvious bonus, it's also easier to interrupt a
> script doing I/O with a KeyboardInterrupt (SIGINT).
> If nothing seems blatantly wrong about this patch, I will go ahead and
> try to write some tests for this.

A potential downside is that the bytes which have been read simply get
lost. But since a signal can happen at any moment I guess nothing can be
done to protect from it in the general case, anyway.

As for the tests, the SignalsTest class in test_io.py is a natural
recipient.

It seems that my patch for issue10956 also fixes this issue (although it doesn't try to fix FileIO). It also has tests and fixes read() as well as write().
I'm sorry for not noticing that the two issues were so similar. Suggest closing this issue as duplicate.

Sorry, bad analysis. The patch in issue10956 only seemed to work because my system is too fast and "find /" in the script returned in less than 5 seconds. Replacing it with "yes" shows that the two issued are actually independent.