what signal does GDB use to implement control transfer between tracee and tracer

Question

By control transfer, I mean, after the tracee executing a function and return, which signal is generated so that GDB can wait*() on it and seize control again? It is not SIGTRAP though many people claim that ...

Answer 1

after the tracee executing a function and return, which signal is generated so that GDB can wait*() on it and seize control again?

The tracee is stopped, and control is transferred back to GDB, only when one of "interesting" events happens.

The interesting events are:

1. A breakpoint fires,
2. The tracee encounters a signal (e.g. SIGSEGV or SIGFPE as a result of performing invalid memory access or invalid floating-point operation),
3. The tracee disappears altogether (such as getting SIGKILLed by an outside program),

[There might be other "interesting" events, but I can't think of anything else right now.]

Now, a technically correct answer to "what signal does GDB use ..." is: none at all. The control isn't transferred, unless one of above events happen.

Perhaps your question is: how does control get back to GDB after executing something like finish command (which steps out of the current function)?

The answer to that is: GDB sets a temporary breakpoint on the instruction immediately after the CALL instruction that got us into the current function.

Finally, what causes the kernel to stop tracee and make waitpid in GDB to return upon execution of the breakpoint instruction?

On x86, GDB uses the INT3 (opcode 0xCC) instruction to set breakpoints (there is an alternate mechanism using debug registers, but it is limited to 4 simultaneous breakpoints, and usually reserved for hardware watchpoints instead). When the tracee executes INT3 instruction, SIGTRAP is indeed the signal that the kernel generates (i.e. other answers you've found are correct).

Without knowing what led you to believe it isn't SIGTRAP, it's hard to guess how you convinced yourself that it isn't.

Update:

I try to manually send a SIGTRAP signal to the tracee, trying to causing a spuriously wake-up of GDB, but fail.

Fail in what way?

What I expect you observe is that GDB stops with Program received signal SIGTRAP .... That's because GDB knows where it has placed breakpoints.

When GDB receives SIGTRAP and the tracee instruction pointer matches one of its breakpoints, then GDB "knows" that is's the breakpoint that has fired, and acts accordingly.

But when GDB receives SIGTRAP and the tracee IP doesn't match any of the breakpoints, then GDB treats it as any other signal: prints a message and waits for you to tell it what to do next.

"GDB sets a temporary breakpoint ... that means GDB has to modify tracee's code area, which may be read-only. So, how does GDB cope with that?

You are correct: GDB needs to modify (typically non-writable) .text section to insert any breakpoint using INT3 method. Fortunately, that is one of the "superpowers" granted to it by the kernel via ptrace(POKE_TEXT, ...).

P.S. It's a fun exercise to white a program that checksums code bytes of one of its own functions. You can then perform the checksum before and after placing a breakpoint on the "to be checksummed" function, and observe that the checksum differs when a breakpoint is present.

P.P.S. If you are curious about what GDB is doing, setting maintenance debug inferior will provide a lot of clues.