Getting stack traces on Unix systems, automatically

Question

What methods are there for automatically getting a stack trace on Unix systems? I don't mean just getting a core file or attaching interactively with GDB, but having a SIGSEGV handler that dumps a backtrace to a text file.

Bonus points for the following optional features:

• Extra information gathering at crash time (eg. config files).
• Email a crash info bundle to the developers.
• Ability to add this in a dlopened shared library
• Not requiring a GUI

Answer 1

FYI,

the suggested solution (using backtrace_symbols in a signal handler) is dangerously broken. DO NOT USE IT -

Yes, backtrace and backtrace_symbols will produce a backtrace and a translate it to symbolic names, however:

1. backtrace_symbols allocates memory using malloc and you use free to free it - If you're crashing because of memory corruption your malloc arena is very likely to be corrupt and cause a double fault.

2. malloc and free protect the malloc arena with a lock internally. You might have faulted in the middle of a malloc/free with the lock taken, which will cause these function or anything that calls them to dead lock.

3. You use puts which uses the standard stream, which is also protected by a lock. If you faulted in the middle of a printf you once again have a deadlock.

4. On 32bit platforms (e.g. your normal PC of 2 year ago), the kernel will plant a return address to an internal glibc function instead of your faulting function in your stack, so the single most important piece of information you are interested in - in which function did the program fault, will actually be corrupted on those platform.

So, the code in the example is the worst kind of wrong - it LOOKS like it's working, but it will really fail you in unexpected ways in production.

BTW, interested in doing it right? check this out.

Cheers, Gilad.

Answer 2

If you are on systems with the BSD backtrace functionality available (Linux, OSX 1.5, BSD of course), you can do this programmatically in your signal handler.

For example (backtrace code derived from IBM example):

#include <execinfo.h>
#include <signal.h>
#include <stdio.h>
#include <stdlib.h>

void sig_handler(int sig)
{
    void * array[25];
    int nSize = backtrace(array, 25);
    char ** symbols = backtrace_symbols(array, nSize);

    for (int i = 0; i < nSize; i++)
    {
        puts(symbols[i]);;
    }

    free(symbols);

    signal(sig, &sig_handler);
}

void h()
{
    kill(0, SIGSEGV);
}

void g()
{
    h();
}

void f()
{
    g();
}

int main(int argc, char ** argv)
{
    signal(SIGSEGV, &sig_handler);
    f();
}

Output:

0   a.out                               0x00001f2d sig_handler + 35
1   libSystem.B.dylib                   0x95f8f09b _sigtramp + 43
2   ???                                 0xffffffff 0x0 + 4294967295
3   a.out                               0x00001fb1 h + 26
4   a.out                               0x00001fbe g + 11
5   a.out                               0x00001fcb f + 11
6   a.out                               0x00001ff5 main + 40
7   a.out                               0x00001ede start + 54

This doesn't get bonus points for the optional features (except not requiring a GUI), however, it does have the advantage of being very simple, and not requiring any additional libraries or programs.

Answer 3

Here is an example of how to get some more info using a demangler. As you can see this one also logs the stacktrace to file.

#include <iostream>
#include <sstream>
#include <string>
#include <fstream>
#include <cxxabi.h>

void sig_handler(int sig)
{
    std::stringstream stream;
    void * array[25];
    int nSize = backtrace(array, 25);
    char ** symbols = backtrace_symbols(array, nSize);
    for (unsigned int i = 0; i < size; i++) {
        int status;
        char *realname;
        std::string current = symbols[i];
        size_t start = current.find("(");
        size_t end = current.find("+");
        realname = NULL;
        if (start != std::string::npos && end != std::string::npos) {
            std::string symbol = current.substr(start+1, end-start-1);
            realname = abi::__cxa_demangle(symbol.c_str(), 0, 0, &status);
        }
        if (realname != NULL)
            stream << realname << std::endl;
        else
            stream << symbols[i] << std::endl;
        free(realname);
    }
    free(symbols);
    std::cerr << stream.str();
    std::ofstream file("/tmp/error.log");
    if (file.is_open()) {
        if (file.good())
            file << stream.str();
        file.close();
    }
    signal(sig, &sig_handler);
}

Answer 4

Dereks solution is probably the best, but here's an alternative anyway:

Recent Linux kernel version allow you to pipe core dumps to a script or program. You could write a script to catch the core dump, collect any extra information you need and mail everything back. This is a global setting though, so it'd apply to any crashing program on the system. It will also require root rights to set up. It can be configured through the /proc/sys/kernel/core_pattern file. Set that to something like ' | /home/myuser/bin/my-core-handler-script'.

The Ubuntu people use this feature as well.