Does executing an int 3 interrupt stop the entire process on Linux or just the current thread?

Question

Suppose the architecture is x86. And the OS is Linux based. Given a multithreaded process in which a single thread executes an int 3 instruction, does the interrupt handler stop from executing the entire process or just the thread that executed the int 3 instruction?

Answer 1

Since the question is Linux specific, let's dive into kernel sources! We know int 3 will generate a SIGTRAP, as we can see in do_int3. The default behaviour of SIGTRAP is to terminate the process and dump core.

do_int3 calls do_trap which, after a lot of indirection, calls complete_signal, where most of the magic happens. Following the comments, it's quite clear to see what is happening without much need for explanation:

• A thread is found to deliver the signal to. The main thread is given first crack, but any thread can get it unless explicitly stated it doesn't want to.
• SIGTRAP is fatal (and we've assumed we want to establish what the default behaviour is) and must dump core, so it is fatal to the whole group
• The loop at line 1003 wakes up all threads and delivers the signal.

EDIT: To answer the comment:

When the process is being ptraced, the behaviour is pretty well documented in the manual page (see "Signal-delivery-stop"). Basically, after the kernel selects an arbitrary thread which handles the signal, if the selected thread is traced, it enters signal-delivery-stop -- this means the signal is not yet delivered to the process, and can be suppressed by the tracer process. This is the case with a debugger: a dead process is of no use to us when debugging (that's not entirely true, but let's consider the live-debugging scenario, which is the only one which makes sense in this context), so by default we block SIGTRAP unless the user specifies otherwise. In this case it is irrelevant how the traced process handles SIGTRAP (SIG_IGN or SIG_DFL or a custom handler) because it will never know it occurred.

Note that in the case of SIGTRAP, the tracer process must account for various scenarios other than the process being stopped, as also detailed in the man page under each ptrace action.

Answer 2

Easy enough to test:

#include <thread>
#include <vector>

void f(int v) {
    std::this_thread::sleep_for(std::chrono::seconds(2));
    if (v == 2) asm("int $3");
    std::this_thread::sleep_for(std::chrono::seconds(1));
    printf("%d\n", v); // no sync here to keep it simple
}

int main() {
    std::vector<std::thread> threads;
    for (int i = 0; i < 4; i++) threads.emplace_back(f, i);
    for (auto& thread : threads) thread.join();
    return 0;
}

If only thread was stopped it should still print the message from threads other then 2 but that is not the case and entire process stops before printing anything (or triggers a breakpoint when debugging). On Ubuntu the message you get is:

Trace/breakpoint trap (core dumped)

Answer 3

The answer is really neither. Int 3 is used to trigger a breakpoint. The interrupt handler is tiny, and neither the interrupt nor its handler stop any threads.

If there is no debugger loaded the handler will either ignore it or call the OS to take some kind of error action like raising a signal (perhaps SIGTRAP). No threads are harmed.

If there is an in-process debugger, the breakpoint ISR transfers control to it. The breakpoint does not stop any threads, except the one that breaks. The debugger may try to suspend others.

If there is a out-of-process debugger, the handler will invoke it, but this has to be mediated through the OS in order to do a suitable context switch. As part of that switch the OS will suspend the debuggee, which means all its threads will stop.

Answer 4

int 3 is a privileged instruction that userspace code is not allowed to run.

The kernel will then send a SIGTRAP signal to your process, and the default action for a SIGTRAP signal is to terminate the entire process.