Boost log destructor aborted

Question

I develop *nix CLI app with boost_log. Few days ago I moved all the code apart of main() to .so library and after that I get crash (please see below) every time my app exits. It's not critical but looks nasty so I'd love to get rid of it. Any advices would be appreciated.

I tried GDB but it didn't help (output below).

$ myapp > /dev/null
*** glibc detected *** myapp: double free or corruption (!prev):     0x0000000000b71f30 ***
======= Backtrace: =========
/lib64/libc.so.6(+0x75f4e)[0x7f51ffcd8f4e]
/lib64/libc.so.6(+0x78cf0)[0x7f51ffcdbcf0]
myapp(_ZN5boost3log12v2s_mt_posix7sources12basic_loggerIcNS2_15severity_loggerIN8my16logging_severity10value_typeEEENS2_19single_thread_modelEED1Ev+0x1c)[0x42e3a6]
myapp(_ZN5boost3log12v2s_mt_posix7sources21basic_severity_loggerINS2_12basic_loggerIcNS2_15severity_loggerIN8my16logging_severity10value_typeEEENS2_19single_thread_modelEEES8_ED2Ev+0x28)[0x42e3f6]
myapp(_ZN5boost3log12v2s_mt_posix7sources22basic_composite_loggerIcNS2_15severity_loggerIN8my16logging_severity10value_typeEEENS2_19single_thread_modelENS2_8featuresIJNS2_8severityIS7_EEEEEED1Ev+0x18)[0x42e410]
myapp(_ZN5boost3log12v2s_mt_posix7sources15severity_loggerIN8my16logging_severity10value_typeEED2Ev+0x18)[0x42e444]
/lib64/libc.so.6(__cxa_finalize+0x9d)[0x7f51ffc98ebd]
/home/v/lib64/libmy_app.so(+0x104de6)[0x7f5201eedde6]
======= Memory map: ========
00400000-0044c000 r-xp 00000000 fd:00 1046178                            /home/v/bin/myapp
0064b000-0064d000 rw-p 0004b000 fd:00 1046178                            /home/v/bin/myapp
00b59000-00b7a000 rw-p 00000000 00:00 0                                  [heap]
7f51f8000000-7f51f8021000 rw-p 00000000 00:00 0
7f51f8021000-7f51fc000000 ---p 00000000 00:00 0
7f51ffa4d000-7f51ffa63000 r-xp 00000000 fd:00 1566722                    /lib64/libgcc_s-4.4.7-20120601.so.1
7f51ffa63000-7f51ffc62000 ---p 00016000 fd:00 1566722                    /lib64/libgcc_s-4.4.7-20120601.so.1
7f51ffc62000-7f51ffc63000 rw-p 00015000 fd:00 1566722                    /lib64/libgcc_s-4.4.7-20120601.so.1
7f51ffc63000-7f51ffded000 r-xp 00000000 fd:00 1566736                    /lib64/libc-2.12.so
7f51ffded000-7f51fffed000 ---p 0018a000 fd:00 1566736                    /lib64/libc-2.12.so
7f51fffed000-7f51ffff1000 r--p 0018a000 fd:00 1566736                    /lib64/libc-2.12.so
7f51ffff1000-7f51ffff2000 rw-p 0018e000 fd:00 1566736                    /lib64/libc-2.12.so
7f51ffff2000-7f51ffff7000 rw-p 00000000 00:00 0
7f51ffff7000-7f520007a000 r-xp 00000000 fd:00 1566744                    /lib64/libm-2.12.so
7f520007a000-7f5200279000 ---p 00083000 fd:00 1566744                    /lib64/libm-2.12.so
7f5200279000-7f520027a000 r--p 00082000 fd:00 1566744                    /lib64/libm-2.12.so
7f520027a000-7f520027b000 rw-p 00083000 fd:00 1566744                    /lib64/libm-2.12.so
7f520027b000-7f5200282000 r-xp 00000000 fd:00 1566764                    /lib64/librt-2.12.so
7f5200282000-7f5200481000 ---p 00007000 fd:00 1566764                    /lib64/librt-2.12.so
7f5200481000-7f5200482000 r--p 00006000 fd:00 1566764                    /lib64/librt-2.12.so
7f5200482000-7f5200483000 rw-p 00007000 fd:00 1566764                    /lib64/librt-2.12.so
7f5200483000-7f520049a000 r-xp 00000000 fd:00 1566760                    /lib64/libpthread-2.12.so
7f520049a000-7f520069a000 ---p 00017000 fd:00 1566760                    /lib64/libpthread-2.12.so
7f520069a000-7f520069b000 r--p 00017000 fd:00 1566760                    /lib64/libpthread-2.12.so
7f520069b000-7f520069c000 rw-p 00018000 fd:00 1566760                    /lib64/libpthread-2.12.so
7f520069c000-7f52006a0000 rw-p 00000000 00:00 0
7f52006a0000-7f52006b0000 r-xp 00000000 fd:00 1566800                    /lib64/libbz2.so.1.0.4
7f52006b0000-7f52008af000 ---p 00010000 fd:00 1566800                    /lib64/libbz2.so.1.0.4
7f52008af000-7f52008b1000 rw-p 0000f000 fd:00 1566800                    /lib64/libbz2.so.1.0.4
7f52008b1000-7f5201b22000 r-xp 00000000 fd:00 1045440                    /home/v/lib64/libmy_common.so
7f5201b22000-7f5201d21000 ---p 01271000 fd:00 1045440                    /home/v/lib64/libmy_common.so
7f5201d21000-7f5201dd1000 rw-p 01270000 fd:00 1045440                    /home/v/lib64/libmy_common.so
7f5201dd1000-7f5201de9000 rw-p 00000000 00:00 0
7f5201de9000-7f5201faa000 r-xp 00000000 fd:00 1046177                    /home/v/lib64/libmy_app.so
7f5201faa000-7f52021aa000 ---p 001c1000 fd:00 1046177                    /home/v/lib64/libmy_app.so
7f52021aa000-7f52021b8000 rw-p 001c1000 fd:00 1046177                    /home/v/lib64/libmy_app.so
7f52021b8000-7f52021d8000 r-xp 00000000 fd:00 1569635                    /lib64/ld-2.12.so
7f52023bc000-7f52023cf000 rw-p 00000000 00:00 0
7f52023d4000-7f52023d7000 rw-p 00000000 00:00 0
7f52023d7000-7f52023d8000 r--p 0001f000 fd:00 1569635                    /lib64/ld-2.12.so
7f52023d8000-7f52023d9000 rw-p 00020000 fd:00 1569635                    /lib64/ld-2.12.so
7f52023d9000-7f52023da000 rw-p 00000000 00:00 0
7ffd9a0eb000-7ffd9a100000 rw-p 00000000 00:00 0                          [stack]
7ffd9a165000-7ffd9a166000 r-xp 00000000 00:00 0                          [vdso]
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0                  [vsyscall]
Aborted
✗ 1

GDB output:

$ gdb -x ~/gdb.myapp.gdb myapp
GNU gdb (GDB) 7.10
Copyright (C) 2015 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-unknown-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from myapp...done.
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Dump file(s) must be specified in read mode

*** glibc detected *** /home/v/bin/myapp: double free or corruption (!prev): 0x0000000000665f30 ***
======= Backtrace: =========
/lib64/libc.so.6(+0x75f4e)[0x7ffff58fdf4e]
/lib64/libc.so.6(+0x78cf0)[0x7ffff5900cf0]
/home/v/bin/myapp(_ZN5boost3log12v2s_mt_posix7sources12basic_loggerIcNS2_15severity_loggerIN8my16logging_severity10value_typeEEENS2_19single_thread_modelEED1Ev+0x1c)[0x42e3a6]
/home/v/bin/myapp(_ZN5boost3log12v2s_mt_posix7sources21basic_severity_loggerINS2_12basic_loggerIcNS2_15severity_loggerIN8my16logging_severity10value_typeEEENS2_19single_thread_modelEEES8_ED2Ev+0x28)[0x42e3f6]
/home/v/bin/myapp(_ZN5boost3log12v2s_mt_posix7sources22basic_composite_loggerIcNS2_15severity_loggerIN8my16logging_severity10value_typeEEENS2_19single_thread_modelENS2_8featuresIJNS2_8severityIS7_EEEEEED1Ev+0x18)[0x42e410]
/home/v/bin/myapp(_ZN5boost3log12v2s_mt_posix7sources15severity_loggerIN8my16logging_severity10value_typeEED2Ev+0x18)[0x42e444]
/lib64/libc.so.6(__cxa_finalize+0x9d)[0x7ffff58bdebd]
/home/v/_/lib64/libmy_app.so(+0x104de6)[0x7ffff7b12de6]
======= Memory map: ========
00400000-0044c000 r-xp 00000000 fd:00 1046178                            /home/v/bin/myapp
0064b000-0064d000 rw-p 0004b000 fd:00 1046178                            /home/v/bin/myapp
0064d000-0066e000 rw-p 00000000 00:00 0                                  [heap]
7ffff0000000-7ffff0021000 rw-p 00000000 00:00 0
7ffff0021000-7ffff4000000 ---p 00000000 00:00 0
7ffff5672000-7ffff5688000 r-xp 00000000 fd:00 1312927                    /work/lib64/libgcc_s.so.1
7ffff5688000-7ffff5887000 ---p 00016000 fd:00 1312927                    /work/lib64/libgcc_s.so.1
7ffff5887000-7ffff5888000 rw-p 00015000 fd:00 1312927                    /work/lib64/libgcc_s.so.1
7ffff5888000-7ffff5a12000 r-xp 00000000 fd:00 1566736                    /lib64/libc-2.12.so
7ffff5a12000-7ffff5c12000 ---p 0018a000 fd:00 1566736                    /lib64/libc-2.12.so
7ffff5c12000-7ffff5c16000 r--p 0018a000 fd:00 1566736                    /lib64/libc-2.12.so
7ffff5c16000-7ffff5c17000 rw-p 0018e000 fd:00 1566736                    /lib64/libc-2.12.so
7ffff5c17000-7ffff5c1c000 rw-p 00000000 00:00 0
7ffff5c1c000-7ffff5c9f000 r-xp 00000000 fd:00 1566744                    /lib64/libm-2.12.so
7ffff5c9f000-7ffff5e9e000 ---p 00083000 fd:00 1566744                    /lib64/libm-2.12.so
7ffff5e9e000-7ffff5e9f000 r--p 00082000 fd:00 1566744                    /lib64/libm-2.12.so
7ffff5e9f000-7ffff5ea0000 rw-p 00083000 fd:00 1566744                    /lib64/libm-2.12.so
7ffff5ea0000-7ffff5ea7000 r-xp 00000000 fd:00 1566764                    /lib64/librt-2.12.so
7ffff5ea7000-7ffff60a6000 ---p 00007000 fd:00 1566764                    /lib64/librt-2.12.so
7ffff60a6000-7ffff60a7000 r--p 00006000 fd:00 1566764                    /lib64/librt-2.12.so
7ffff60a7000-7ffff60a8000 rw-p 00007000 fd:00 1566764                    /lib64/librt-2.12.so
7ffff60a8000-7ffff60bf000 r-xp 00000000 fd:00 1566760                    /lib64/libpthread-2.12.so
7ffff60bf000-7ffff62bf000 ---p 00017000 fd:00 1566760                    /lib64/libpthread-2.12.so
7ffff62bf000-7ffff62c0000 r--p 00017000 fd:00 1566760                    /lib64/libpthread-2.12.so
7ffff62c0000-7ffff62c1000 rw-p 00018000 fd:00 1566760                    /lib64/libpthread-2.12.so
7ffff62c1000-7ffff62c5000 rw-p 00000000 00:00 0
7ffff62c5000-7ffff62d5000 r-xp 00000000 fd:00 1566800                    /lib64/libbz2.so.1.0.4
7ffff62d5000-7ffff64d4000 ---p 00010000 fd:00 1566800                    /lib64/libbz2.so.1.0.4
7ffff64d4000-7ffff64d6000 rw-p 0000f000 fd:00 1566800                    /lib64/libbz2.so.1.0.4
7ffff64d6000-7ffff7747000 r-xp 00000000 fd:00 1045440                    /home/v/lib64/libmy_common.so
7ffff7747000-7ffff7946000 ---p 01271000 fd:00 1045440                    /home/v/lib64/libmy_common.so
7ffff7946000-7ffff79f6000 rw-p 01270000 fd:00 1045440                    /home/v/lib64/libmy_common.so
7ffff79f6000-7ffff7a0e000 rw-p 00000000 00:00 0
7ffff7a0e000-7ffff7bcf000 r-xp 00000000 fd:00 1046177                    /home/v/lib64/libmy_app.so
7ffff7bcf000-7ffff7dcf000 ---p 001c1000 fd:00 1046177                    /home/v/lib64/libmy_app.so
7ffff7dcf000-7ffff7ddd000 rw-p 001c1000 fd:00 1046177                    /home/v/lib64/libmy_app.so
7ffff7ddd000-7ffff7dfd000 r-xp 00000000 fd:00 1569635                    /lib64/ld-2.12.so
7ffff7fe0000-7ffff7ff3000 rw-p 00000000 00:00 0
7ffff7ff8000-7ffff7ffb000 rw-p 00000000 00:00 0
7ffff7ffb000-7ffff7ffc000 r-xp 00000000 00:00 0                          [vdso]
7ffff7ffc000-7ffff7ffd000 r--p 0001f000 fd:00 1569635                    /lib64/ld-2.12.so
7ffff7ffd000-7ffff7ffe000 rw-p 00020000 fd:00 1569635                    /lib64/ld-2.12.so
7ffff7ffe000-7ffff7fff000 rw-p 00000000 00:00 0
7ffffffea000-7ffffffff000 rw-p 00000000 00:00 0                          [stack]
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0                  [vsyscall]

Program received signal SIGABRT, Aborted.
0x00007ffff58ba625 in raise () from /lib64/libc.so.6
(gdb) bt
#0  0x00007ffff58ba625 in raise () from /lib64/libc.so.6
#1  0x00007ffff58bbe05 in abort () from /lib64/libc.so.6
#2  0x00007ffff58f8537 in __libc_message () from /lib64/libc.so.6
#3  0x00007ffff58fdf4e in malloc_printerr () from /lib64/libc.so.6
#4  0x00007ffff5900cf0 in _int_free () from /lib64/libc.so.6
#5  0x000000000042e3a6 in boost::log::v2s_mt_posix::sources::basic_logger<char, boost::log::v2s_mt_posix::sources::severity_logger<my::logging_severity::value_type>, boost::log::v2s_mt_posix::sources::single_thread_model>::~basic_logger (this=0x64ca20 <xlog_logger>, __in_chrg=<optimized out>)
    at /work/include/boost/log/sources/basic_logger.hpp:79
#6  0x000000000042e3f6 in boost::log::v2s_mt_posix::sources::basic_severity_logger<boost::log::v2s_mt_posix::sources::basic_logger<char, boost::log::v2s_mt_posix::sources::severity_logger<my::logging_severity::value_type>, boost::log::v2s_mt_posix::sources::single_thread_model>, my::logging_severity::value_type>::~basic_severity_logger (this=0x64ca20 <xlog_logger>,
    __in_chrg=<optimized out>)
    at /work/include/boost/log/sources/severity_feature.hpp:145
#7  0x000000000042e410 in boost::log::v2s_mt_posix::sources::basic_composite_logger<char, boost::log::v2s_mt_posix::sources::severity_logger<my::logging_severity::value_type>, boost::log::v2s_mt_posix::sources::single_thread_model, boost::log::v2s_mt_posix::sources::features<boost::log::v2s_mt_posix::sources::severity<my::logging_severity::value_type> > >::~basic_composite_logger (
    this=0x64ca20 <xlog_logger>, __in_chrg=<optimized out>)
    at /work/include/boost/log/sources/basic_logger.hpp:506
#8  0x000000000042e444 in boost::log::v2s_mt_posix::sources::severity_logger<my::logging_severity::value_type>::~severity_logger (
    this=0x64ca20 <xlog_logger>, __in_chrg=<optimized out>)
    at /work/include/boost/log/sources/severity_logger.hpp:45
#9  0x00007ffff58bdebd in __cxa_finalize () from /lib64/libc.so.6
#10 0x00007ffff7b12de6 in __do_global_dtors_aux ()
   from /home/v/_/lib64/libmy_app.so
#11 0x000000000000000a in ?? ()
#12 0x0000000000000000 in ?? ()
(gdb) f 0
#0  0x00007ffff58ba625 in raise () from /lib64/libc.so.6
(gdb) f 10
#10 0x00007ffff7b12de6 in __do_global_dtors_aux ()
   from /home/v/_/lib64/libmy_app.so

without code nobody will be able to help you. The error is stated clearly: `double free or corruption` — Nidhoegger, Feb 09 '17 at 11:41
I don't think it's correct: (A) it worked nice before I moved most of classes to .so and (B) stack trace looks like double free happens in boost code which I find unlikely because of (A) — v.v, Feb 09 '17 at 11:48
A: Does not matter, does not work now. Maybe the error was there before but manifests now as you changed some timings or call order B: Of course it happens in boost code, but something triggers it. And as you have a lack of debug symbols in your .so (see: `#11 0x000000000000000a in ?? ()`) you do not know it! — Nidhoegger, Feb 09 '17 at 11:57
My binaries are debug ones, but they are linked vs. stripped libs. Still, I don't think that there's unknown double free problem in boost log, I rather think it's something about how I use it. — v.v, Feb 09 '17 at 12:04
and we cannot see how you use it, so we cannot help you. period. — Nidhoegger, Feb 09 '17 at 12:15
Do you have global or local statically initialized variables in your .so? If so you're possibly seeing the flip side of the [static initialization order fiasco](https://isocpp.org/wiki/faq/ctors#construct-on-first-use-v2) -- your variables are being initialized correctly but are being "destucted" in an indeterminate order causing the problems you see. — G.M., Feb 09 '17 at 12:16
@G.M. thank you, I do have static global logger initialised in main() [compiled to resulting exe] and class member loggers [compiled as .so] Got same idea 15 min ago, checking :) — v.v, Feb 09 '17 at 12:21
@Nidhoegger in fact we likely can. Many things require code. Things like this need spidey sense and experience. I'm pretty happy with the stacktrace (which, admittedly, I haven't looked at yet) — sehe, Feb 09 '17 at 12:46
@sehe: Experience is an important factor. But this could be anywhere caused by anything. Please look at the Stacktrace first. It could be the mentioned static initialization order fiasco, really a double free, a use after destruct, a never initialized pointer, etc. — Nidhoegger, Feb 09 '17 at 13:06
@Nidhoegger I knew the problem when I read half the title from the questions list. All the info in the question just confirmed it. — sehe, Feb 09 '17 at 13:12

score 1 · Answer 1 · answered Feb 09 '17 at 12:44

You're looking at Static Initialization Order Fiasco.

Despite the name, it also applies to the order in which global objects are destructed. Since the order likely changed in the reorganization of your library object files, you more than likely end up with use-after-destruction.

Look for logging inside destructors. Avoid global/namespace-scope statics.

Still, I don't think that there's unknown double free problem in boost log, I rather think it's something about how I use it.

Indeed.

+ @G.M. : thanks guys, the problem has gone once I get rid of global loggers. Cheers! — v.v, Feb 09 '17 at 15:11

Boost log destructor aborted

1 Answers1