Strange blocking behaviour when reading sys.stdin in python 2 while having a custom signal handler in place

Question

Consider this small python script odd-read-blocking.py:

#!/usr/bin/python

import signal
import sys

sig = None


def handler(signum, frame):
    global sig
    sig = signum


signal.signal(signal.SIGINT, handler)
signal.signal(signal.SIGTERM, handler)

x = sys.stdin.read(3)

print 'signal', sig
print 'read bytes', len(x)

exit(0)

I run this and feed it with two bytes of standard input data ('a' + '\n'):

> echo a | ./odd-read-blocking.py 
signal None
read bytes 2
>

Fine.

Now I feed it with the same two bytes (by typing 'a' + '\n' into its standard input). Please note that standard input is then not at EOF yet and potentially has more data to come. So the read blocks, as it expects one more byte. I use Ctrl+C on the script.

> ./odd-read-blocking.py 
a
^Csignal 2
read bytes 2
>

Fine. We see that two bytes have been read and signal 2 was received.

Now I open a standard input stream, but do not send any byte on it. The read blocks as expected. If I now use Ctrl+C on the script, it will keep sitting there and wait. The read will not be interrupted. The SIGINT will not be processed.

> ./odd-read-blocking.py 
^C

Nothing here. Script still running (seemingly blocked at the read).

Now hitting return once, then Ctrl+C again:

^Csignal 2
read bytes 1
>

So, only after receiving at least some data (a single '\n' in this case) on its standard input will the script behave as I expect it and correctly interrupt the blocked read and tell me it has received signal 2 and read 1 byte.

Alternative 1: instead of using Ctrl+C as shown above, I have tried this same thing using kill pid from a separate terminal. The behaviour is the same.

Alternative 2: instead of using the shell standard input as described above, I have done this:

> sleep 2000 | ./odd-read-blocking.py

When using kill pid to send SIGTERM to the odd-read-blocking.py process I get the same behaviour. Here, the script process can only be killed using SIGKILL (9).

Why isn't the read interrupted, when it is blocking on an as yet empty but still active standard input stream?

I find this odd. Who doesn't? Who can explain?

Answer 1

The short version

If a Python signal handler throws an exception to abandon an ongoing file.read, any data already read is lost. (Any asynchronous exception, like the default KeyboardInterrupt, makes it basically impossible to prevent this sort of failure unless you have a way to mask it.)

To minimize the need for this, file.read returns early (i.e., with a shorter string than requested) when it is interrupted by a signal—note that this is in addition to the EOF and non-blocking I/O cases that are documented! However, it can't do this when it has no data yet, since it returns the empty string to indicate EOF.

Details

As always, the way to understand behavior like this is with strace.

read(2)

The actual read system call has a dilemma when a signal arrives while the process is blocked. First, the (C) signal handler gets invoked—but because that could happen between any two instructions, there's very little it can do beyond setting a flag (or writing to a self-pipe). Then what? If SA_RESTART is set, the call is resumed; otherwise…

If no data has been transferred yet, read can fail and the client can check its signal flag. It fails with the special EINTR to clarify that nothing actually went wrong with the I/O.

If some data has already been written into the (userspace) buffer, it can't just return "failure", because data would be lost—the client can't know how much (if any) data is in the buffer. So it just returns success (the number of bytes read so far)! Short reads like this are always a possibility: the client has to call read again to check that it has reached end of file. (Just like file.read, a short read of 0 bytes would be EOF.) The client therefore has to check their signal flag after every read, whether it succeeds or not. (Note that this is still not perfectly reliable, but it's good enough for many interactive use cases.)

file.read()

The system call isn't the whole story: after all, the normal configuration for a terminal has it return immediately after seeing a newline. Python 2's low-level file.read is a wrapper for fread, which will issue another read if one is short. But when a read fails with EINTR, fread returns early and file.read calls your (Python) signal handler. (If you add output to it, you'll see that it's called immediately for each signal you send, even if file.read doesn't return.)

Then it's faced with a dilemma similar to that for the system call: as discussed, a short read can't be empty because it means EOF. Unlike a C signal handler, however, a Python one can do arbitrary work (including raising an exception to abort the I/O immediately, at the cost of risking data loss as mentioned at the beginning), and it's considered a convenient simplification to the interface to hide the possibility EINTR. So the fread call is just silently repeated.

Python 3.5

The rules for retrying changed in 3.5. Now the io.IOBase.read resumes even if it has data in hand; this is more consistent, but it forces the use of exceptions to stop reading, which means that you can't opt to wait on some data in order not to risk losing any you already have. The very heavyweight solution is to switch to multiplexed I/O and use signal.set_wakeup_fd(); this has the added advantage of allowing SIGINT to affect the main thread without having to bother with masking it in all the others.