Where to look when a daemonized version of a program crashes?

Question

I have a C++ program that receives streaming music and plays it. I can run the program via the shell and run with it attached and it runs completely fine. I can stream audio to it and send things to it, but it will not crash. However, when I daemonize it and run the fork code, it will crash unexpectedly after a little bit of streaming. I try to use gdb to debug it, but it doesn't give much output.

./bin/sonar -d ; sleep 1 ; gdb ./bin/sonar $(cat sonar.pid )
GNU gdb (GDB) 7.4.1-debian
Copyright (C) 2012 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "arm-linux-gnueabihf".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /var/sonar/bin/sonar...done.
Attaching to program: /var/sonar/bin/sonar, process 4050
Reading symbols from /usr/lib/arm-linux-gnueabihf/libcofi_rpi.so...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/arm-linux-gnueabihf/libcofi_rpi.so
Reading symbols from /var/sonar/bin/../lib/libboost_system.so.1.60.0...(no debugging symbols found)...done.
Loaded symbols for /var/sonar/bin/../lib/libboost_system.so.1.60.0
Reading symbols from /var/sonar/bin/../lib/libboost_chrono.so.1.60.0...(no debugging symbols found)...done.
Loaded symbols for /var/sonar/bin/../lib/libboost_chrono.so.1.60.0
Reading symbols from /var/sonar/bin/../lib/libboost_timer.so.1.60.0...(no debugging symbols found)...done.
Loaded symbols for /var/sonar/bin/../lib/libboost_timer.so.1.60.0
Reading symbols from /var/sonar/bin/../lib/libboost_iostreams.so.1.60.0...(no debugging symbols found)...done.
Loaded symbols for /var/sonar/bin/../lib/libboost_iostreams.so.1.60.0
Reading symbols from /var/sonar/bin/../lib/libboost_thread.so.1.60.0...(no debugging symbols found)...done.
Loaded symbols for /var/sonar/bin/../lib/libboost_thread.so.1.60.0
Reading symbols from /var/sonar/bin/../lib/libboost_log_setup.so.1.60.0...(no debugging symbols found)...done.
Loaded symbols for /var/sonar/bin/../lib/libboost_log_setup.so.1.60.0
Reading symbols from /var/sonar/bin/../lib/libboost_log.so.1.60.0...(no debugging symbols found)...done.
Loaded symbols for /var/sonar/bin/../lib/libboost_log.so.1.60.0
Reading symbols from /var/sonar/bin/../lib/libboost_filesystem.so.1.60.0...(no debugging symbols found)...done.
Loaded symbols for /var/sonar/bin/../lib/libboost_filesystem.so.1.60.0
Reading symbols from /var/sonar/bin/../lib/libboost_atomic.so.1.60.0...(no debugging symbols found)...done.
Loaded symbols for /var/sonar/bin/../lib/libboost_atomic.so.1.60.0
Reading symbols from /var/sonar/bin/../lib/libboost_date_time.so.1.60.0...(no debugging symbols found)...done.
Loaded symbols for /var/sonar/bin/../lib/libboost_date_time.so.1.60.0
Reading symbols from /var/sonar/bin/../lib/libboost_regex.so.1.60.0...(no debugging symbols found)...done.
Loaded symbols for /var/sonar/bin/../lib/libboost_regex.so.1.60.0
Reading symbols from /var/sonar/bin/../lib/libopus.so.0...(no debugging symbols found)...done.
Loaded symbols for /var/sonar/bin/../lib/libopus.so.0
Reading symbols from /var/sonar/bin/../lib/libmysql.so.16...(no debugging symbols found)...done.
Loaded symbols for /var/sonar/bin/../lib/libmysql.so.16
Reading symbols from /var/sonar/bin/../lib/libmysqlpp.so.3...(no debugging symbols found)...done.
Loaded symbols for /var/sonar/bin/../lib/libmysqlpp.so.3
Reading symbols from /usr/lib/arm-linux-gnueabihf/libcrypto.so.1.0.0...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/arm-linux-gnueabihf/libcrypto.so.1.0.0
Reading symbols from /usr/lib/arm-linux-gnueabihf/libssl.so.1.0.0...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/arm-linux-gnueabihf/libssl.so.1.0.0
Reading symbols from /usr/lib/arm-linux-gnueabihf/libasound.so.2...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/arm-linux-gnueabihf/libasound.so.2
Reading symbols from /usr/lib/arm-linux-gnueabihf/libstdc++.so.6...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/arm-linux-gnueabihf/libstdc++.so.6
Reading symbols from /lib/arm-linux-gnueabihf/libm.so.6...Reading symbols from /usr/lib/debug/lib/arm-linux-gnueabihf/libm-2.19.so...done.
done.
Loaded symbols for /lib/arm-linux-gnueabihf/libm.so.6
Reading symbols from /lib/arm-linux-gnueabihf/libgcc_s.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib/arm-linux-gnueabihf/libgcc_s.so.1
Reading symbols from /lib/arm-linux-gnueabihf/libpthread.so.0...Reading symbols from /usr/lib/debug/lib/arm-linux-gnueabihf/libpthread-2.19.so...done.
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/arm-linux-gnueabihf/libthread_db.so.1".
[New Thread 0x749ff450 (LWP 4054)]
[New Thread 0x75358450 (LWP 4053)]
[New Thread 0x75b58450 (LWP 4052)]
done.
Loaded symbols for /lib/arm-linux-gnueabihf/libpthread.so.0
Reading symbols from /lib/arm-linux-gnueabihf/libc.so.6...Reading symbols from /usr/lib/debug/lib/arm-linux-gnueabihf/libc-2.19.so...done.
done.
Loaded symbols for /lib/arm-linux-gnueabihf/libc.so.6
Reading symbols from /lib/arm-linux-gnueabihf/librt.so.1...Reading symbols from /usr/lib/debug/lib/arm-linux-gnueabihf/librt-2.19.so...done.
done.
Loaded symbols for /lib/arm-linux-gnueabihf/librt.so.1
Reading symbols from /lib/arm-linux-gnueabihf/libbz2.so.1.0...(no debugging symbols found)...done.
Loaded symbols for /lib/arm-linux-gnueabihf/libbz2.so.1.0
Reading symbols from /lib/arm-linux-gnueabihf/libz.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib/arm-linux-gnueabihf/libz.so.1
Reading symbols from /lib/arm-linux-gnueabihf/libdl.so.2...Reading symbols from /usr/lib/debug/lib/arm-linux-gnueabihf/libdl-2.19.so...done.
done.
Loaded symbols for /lib/arm-linux-gnueabihf/libdl.so.2
Reading symbols from /lib/ld-linux-armhf.so.3...(no debugging symbols found)...done.
Loaded symbols for /lib/ld-linux-armhf.so.3
Reading symbols from /lib/arm-linux-gnueabihf/libnss_files.so.2...Reading symbols from /usr/lib/debug/lib/arm-linux-gnueabihf/libnss_files-2.19.so...done.
done.
Loaded symbols for /lib/arm-linux-gnueabihf/libnss_files.so.2
0x763e1d14 in epoll_wait () at ../sysdeps/unix/syscall-template.S:81
81      ../sysdeps/unix/syscall-template.S: No such file or directory.
(gdb) c
Continuing.

Program received signal SIGABRT, Aborted.
[Switching to Thread 0x75358450 (LWP 4053)]
0x7633df70 in raise () from /lib/arm-linux-gnueabihf/libc.so.6
(gdb) bt
#0  0x7633df70 in raise () from /lib/arm-linux-gnueabihf/libc.so.6
#1  0x7633f324 in abort () from /lib/arm-linux-gnueabihf/libc.so.6
#2  0x7656eb5c in __gnu_cxx::__verbose_terminate_handler() () from /usr/lib/arm-linux-gnueabihf/libstdc++.so.6
#3  0x7656c9a0 in ?? () from /usr/lib/arm-linux-gnueabihf/libstdc++.so.6
#4  0x7656c9a0 in ?? () from /usr/lib/arm-linux-gnueabihf/libstdc++.so.6
Backtrace stopped: previous frame identical to this frame (corrupt stack?)
(gdb) c
Continuing.
[Thread 0x762ac000 (LWP 4050) exited]
[Thread 0x75b58450 (LWP 4052) exited]
[Thread 0x749ff450 (LWP 4054) exited]

Program terminated with signal SIGABRT, Aborted.
The program no longer exists.
(gdb)

The daemonizing code that I use:

// Daemonize
LOG_INFO( *logger ) << "Daemonizing.";

// Sig stuff
pid_t pid, sid;

// Start the forky pig
if( (pid = fork())<0 )
{
    LOG_ERROR( *logger ) << "Error forking: " << strerror( errno );
    return( EXIT_FAILURE );
}

// The parent can go home
if( pid>0 )
    return( EXIT_SUCCESS );

// Make ourselves emancipate
if( (sid = setsid())<0 )
{
    LOG_ERROR( *logger ) << "setsid error: " << strerror( errno );
    return( EXIT_FAILURE );
}

// Ignore sigpipes
signal( SIGPIPE, SIG_IGN );

// Close the file descriptors to detach
close( STDIN_FILENO );
close( STDOUT_FILENO );
close( STDERR_FILENO );

If the program works when it's not daemonized and crashes when it is, there is obviously some difference in the way that it runs. So my question really is, where can I look for this particular point of failure?

Here is the same error on a 64-bit intel arch processor:

Program received signal SIGABRT, Aborted.
[Switching to Thread 0x7f80777b8700 (LWP 10314)]
0x00007f807baa7297 in __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:55
55      ../sysdeps/unix/sysv/linux/raise.c: No such file or directory.
(gdb) bt
#0  0x00007f807baa7297 in __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:55
#1  0x00007f807baa862a in __GI_abort () at abort.c:89
#2  0x00007f807c5a700d in __gnu_cxx::__verbose_terminate_handler() () from /usr/lib/gcc/x86_64-pc-linux-gnu/4.9.3/libstdc++.so.6
#3  0x00007f807c5a4e96 in ?? () from /usr/lib/gcc/x86_64-pc-linux-gnu/4.9.3/libstdc++.so.6
#4  0x00007f807c5a4ee1 in std::terminate() () from /usr/lib/gcc/x86_64-pc-linux-gnu/4.9.3/libstdc++.so.6
#5  0x00007f807c600c21 in ?? () from /usr/lib/gcc/x86_64-pc-linux-gnu/4.9.3/libstdc++.so.6
#6  0x00007f807be16324 in start_thread (arg=0x7f80777b8700) at pthread_create.c:333
#7  0x00007f807bb5bf6d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109

You are not showing enough information to diagnose a problem. Please attempt to reproduce your symptom with an [MCVE](http://stackoverflow.com/help/mcve), and if that does not help you identify the problem, comeback here and post the minimal program and explain the problem again. — jxh, Apr 12 '16 at 17:44

score 1 · Answer 1 · answered Apr 15 '16 at 09:07

there is a difference in running a process in an interactive environment or as a daemon. When you run a process from the command line, then the process is inheriting all resources from the interactive environment like Paths, defined in environment PATH variables, or even the working directory, which is, in an interactive environment, the actual directory from where a process is started.

Without knowing your system or environment, one of the mistakes which are often done is trying to start background process, daemons, for example from a crontab, but to forget to give the information a process needs to work. When a process is running as a daemon, without knowing the circumstances of your work flow, for example, it i necessary to give the process the environment, to mention one, the working directory. I do it for example with "chdir("/");", so the forked child will know, on what working directory he is active.

Depending on your other resources, you should give the daemon all the information, it needs to fulfil its job, e.g. if it starts some process, for example the PATH to the process and so on.

regards

This is a very good answer, and I appreciate your answer. I have a problem with not being able to ask good questions, so I thank you for taking the time out to write the post. — Mikey A. Leonetti, Apr 15 '16 at 22:36

score 0 · Accepted Answer · edited May 23 '17 at 12:07

0

Turns out that it was a log file crash. I wasn't seeing the throw error because the error was with the logging system itself. Since all of the ties to stdout were cut, nothing was being displayed. It was exactly the issue in this post with Boost.Log:

Boost Log causes crash when trying first log statement (when not Administrator)

edited May 23 '17 at 12:07

Community

1
1

answered Apr 15 '16 at 22:37

Mikey A. Leonetti

2,834
3
22
36

Where to look when a daemonized version of a program crashes?

2 Answers2