Synchronization issue with usage of pthread_kill() to terminate thread blocked for I/O

Question

Previously I had asked a question regarding how to terminate thread blocked for I/O. I have used pthread_kill() instead of pthread_cancel() or writing to pipes, considering few advantages.

I have implementing the code to send signal (SIGUSR2) to the target thread using pthread_kill(). Below is the skeleton code for this. Most of the times getTimeRemainedForNextEvent() returns a value that blocks poll() for several hours. Because of this large timeout value, even if Thread2 sets terminateFlag (to stop Thread1), Thread2 gets blocked till poll() of Thread1 returns (which might be after several hours if there are no events on sockets). So I'm sending signal to Thread1 using pthread_kill() to interrupt poll() system call (if it gets blocked).

static void signalHandler(int signum) {
    //Does nothing
}

// Thread 1 (Does I/O operations and handles scheduler events). 

void* Thread1(void* args) {
    terminateFlag = 0;
    while(!terminateFlag) {
        int millis = getTimeRemainedForNextEvent(); //calculate maximum number of milliseconds poll() can block.

        int ret = poll(fds,numOfFDs,millis);
        if(ret > 0) {
            //handle socket events.
        } else if (ret < 0) {
            if(errno == EINTR)
                perror("Poll Error");
            break;
        }

        handleEvent();  
    }
}

// Thread 2 (Terminates Thread 1 when Thread 1 needs to be terminated)

void* Thread2(void* args) {
    while(1) {

    /* Do other stuff */

    if(terminateThread1) {
            terminateFlag = 1;
            pthread_kill(ftid,SIGUSR2); //ftid is pthread_t variable of Thread1
            pthread_join( ftid, NULL );
        }
    }

    /* Do other stuff */
} 

Above code works fine if Thread2 sets terminateFlag and sends signal to Thread1 when it blocked in poll() system call. But, If context switch happens after getTimeRemainedForNextEvent() function of Thread1 and Thread2 sets terminateFlag and sends signal, poll() of Thread1 gets blocked for several hours as it lost the signal that interrupts the system call.

It seems I can not use mutex for synchronization as poll() will hold the lock till it gets unblocked. Is there any synchronization mechanism that I can apply to avoid the above mentioned issue ?

Answer 1

Consider having an additional file descriptor in the set of fds passed to poll whose sole job is to make poll return when you want to terminate the thread.

Thus, in thread 2 we would have something like:

if (terminateThread1) {
        terminateFlag = 1;
        send (terminate_fd, " ", 1, 0);
        pthread_join (ftid, NULL);
    }
}

And terminate_fd would be in the set of fds passed to poll by thread 1.

-- OR --

If the overhead of having an extra fd per thread is too much (as discussed in the comments) then send something to one of the existing fds that thread 1 ignores. This will cause poll to return and then thread 1 will terminate. You can even have this 'special' value act as the terminate flag, which makes the logic a little tidier.

Answer 2

In the first place, access to shared variable terminateFlag by multiple threads must be protected by a mutex or similar synchronization mechanism, else your program does not conform and all bets are off. That might, for instance, look like this:

void *Thread1(void *args) {
    pthread_mutex_lock(&a_mutex);
    terminateFlag = 0;
    while(!terminateFlag) {
        pthread_mutex_unlock(&a_mutex);

        // ...

        pthread_mutex_lock(&a_mutex);
    }
    pthread_mutex_unlock(&a_mutex);
}

void* Thread2(void* args) {
    // ...

    if (terminateThread1) {
        pthread_mutex_lock(&a_mutex);
        terminateFlag = 1;
        pthread_mutex_unlock(&a_mutex);
        pthread_kill(ftid,SIGUSR2); //ftid is pthread_t variable of Thread1
        pthread_join( ftid, NULL );
    }

    // ...
} 

But that does not solve the main problem, that a signal sent by thread 2 may be delivered to thread 1 after it tests terminateFlag but before it calls poll(), though it does narrow the window in which that could happen.

The cleanest solution is that suggested already by @PaulSanders' answer: have thread 2 wake thread 1 via a file descriptor that thread 1 is polling (i.e. by means of a pipe). Inasmuch as you seem to have a plausible reason to seek an alternative approach, however, it should also be possible to make your signaling approach work by appropriate use of signal masking. Expanding on @Shawn's comment, here's how it would work:

1. The parent thread blocks SIGUSR2 before starting thread 1, so that the latter, which inherits its signal mask from its parent, starts with that signal blocked.

2. Thread 1 uses ppoll() instead of poll(), so as to be able to specify that SIGUSR2 will be unblocked for the duration of that call. ppoll() does signal mask handling atomically, so that there is no opportunity for a signal to be lost when it is blocked before the call and unblocked within.

3. Thread 2 uses pthread_kill() to send SIGUSR2 to thread 1 to make it stop. Because that signal is only unblocked for that thread when it is performing a ppoll() call, it will not be lost (blocked signals remain pending until unblocked). This is precisely the kind of usage scenario for which ppoll() is designed.

4. You should even be able to do away with the terminateThread variable and associated synchronization, because you should be able to rely upon the signal being delivered during a ppoll() call and therefore causing the EINTR code path to be exercised. That path does not rely on terminateThread to make the thread stop.

Answer 3

As you say yourself, you could use thread cancellation to solve this. Outside of thread cancellation, I don't think there's a "right" way to solve this within POSIX (waking up the poll call with a write isn't exactly a generic method that would work for all situations in which a thread might get blocked), because POSIX's paradigm for making syscalls and handling signals simply doesn't allow you to close the gap between a flag check and a potentially long blocking call.

void handler() { dont_enter_a_long_blocking_call_flg=1; }
int main()
{  //...
    if(dont_enter_a_long_blocking_call_flg)
        //THE GAP; what if the signal arrives here ?
        potentially_long_blocking_call();
    //....
}

The musl libc library uses signals for thread cancellation (because signals can break long-blocking calls that are in kernel mode) and it uses them in conjunction with global assembly labels so that from the flag setting SIGCANCEL handler, it can do (conceptually, I'm not pasting their actual code):

void sigcancel_handler(int Sig, siginfo_t *Info, void *Uctx)
{
    thread_local_cancellation_flag=1;
    if_interrupted_the_gap_move_Program_Counter_to_start_cancellation(Uctx);
}

Now if you changed if_interrupted_the_gap_move_Program_Counter_to_start_cancellation(Uctx); to if_interrupted_the_gap_move_Program_Counter_to_make_the_syscall_fail(Uctx); and exported the if_interrupted_the_gap_move_Program_Counter_to_make_the_syscall_fail function along with the thread_local_cancellation_flag.

then you can use it to*:

• solve your problem robustly implement robust signal cancelation with any signal without having to put any of that pthread_cleanup_{push,pop} stuff into your already working thread-safe singel threaded code
• ensure assured normal-context reaction to a signal delivery in your target thread even if the signal is handled.

Basically without a libc extension like this, if you once kill()/pthread_kill() a process/thread with a signal it handles or if put a function on a signal-sending timer, you cannot be sure of an assured reaction to the signal delivery, as the target may well receive the signal in a gap like above and hang indefinitely instead of responding to it.

I've implemented such a libc extension on top of musl libc and published it now https://github.com/pskocik/musl. The SIGNAL_EXAMPLES directory also shows some kill(), pthread_kill, and setitimer() examples that under a demonstrated race condition hang with classical libcs but don't wit my extended musl. You can use that extended musl to solve your problem cleanly and I also use it in my personal project to do robust thread cancellation without having to litter my code with pthread_cleanup_{push,pop}

The obvious downside of this approach is that it's unportable and I only have it implemented for x86_64 musl. I've published it today in the hope that somebody (Cygwin, MacOSX?) copies it, because I think it's the right way to do cancellation in C.

In C++ and with glibc, you could utilize the fact that glibc uses exceptions to implement thread cancellation and simply use pthread_cancel (which uses a signal (SIGCANCEL) underneath) but catch it instead of letting it kill the thread.

---

Note:

I'm really using two thread-local flags -- a breaker flag that breaks the next syscall with ECANCELED if set before the syscall is entered (an EINTR returned from a potentially long-blocking syscall gets turned into ECANCELED in the modified libc-provided syscall wrapper iff the breaking flag is set) and a saved breaking flag -- the moment a breaking flag has been used it's saved in the saved breaking flag and zeroed so that the breaking flag doesn't break futher potentially long blocking syscalls.

The idea is that cancelling signals are handled one at a time (the signal handler can be left with all/most signals blocked; the handler code (if any) can then unblock them) and that correctly checking code starts unwinding, i.e., cleaning up while returning errors, the moment it sees an ECANCELED. Then, the next potentially long blocking syscall could be in the cleanup code (e.g., code that writes </html> to a socket) and that syscall must be enterrable (if the breaking flag stayed on, it wouldn't be). Of course with cleanup code having e.g., write(1,"</html>",...) in it, it could block indefinitely too, but you could write the cleanup code so that the potentially long-blocking syscall there runs under a timer when the cleanup is due to an error (ECANCELED is an error). As I've already mentioned, robust, race-condition free, signal driven timers is one of the things this extension allows.

The EINTR => ECANCELED translation happens so that code looping on EINTR knows when to stop looping (many EINTR (=signal interrupted a syscall) cannot be prevented and the code should simply handle them by retrying the syscall. I'm using ECANCELED as an "EINTR after which you shouldn't retry."