1

everyone.

I'm trying to fix a problem in ceph using its core file and gdb. According to the following output, gdb should had successfully loaded debug symbols from ceph-debuginfo:

Reading symbols from /usr/bin/ceph-osd...Reading symbols from /usr/lib/debug/usr/bin/ceph-osd.debug...done.

However, it still couldn't find the symbol table when I used "bt" to trace the stack:

#0  0x000000393da0f65b in ?? ()
No symbol table info available.
#1  0x0000000000a51636 in install_standard_sighandlers () at global/signal_handler.cc:121
No locals.
#2  0x00007fc7a77f9ed0 in ?? ()
No symbol table info available.
#3  0x00007fc7a77f9e10 in ?? ()
No symbol table info available.
#4  0x00007fc7a77f9b90 in ?? ()
No symbol table info available.
#5  0x00007fc66d3142e0 in ?? ()
No symbol table info available.
#6  0x00007fc7fac64100 in ?? ()
No symbol table info available.
#7  0x0000003900000000 in ?? ()
No symbol table info available.
#8  0x0000000000a51155 in SignalHandler::unregister_handler (this=0x1105440, signum=<value optimized out>, handler=<value optimized out>) at global/signal_handler.cc:317
No locals.
#9  0x000000393eabcc33 in ?? ()
No symbol table info available.
#10 0x000000393eabcd2e in ?? ()
No symbol table info available.

Why is this happening?

PS: when gdb started running, it prompted the following warning:

BFD: Warning: /home/xuxuehan/online_problems.2016-11-19.7-01/core-ceph-osd-6-32337-32337-19906-1479510049 is truncated: expected core file size >= 8372899840, found: 7439335424

Could this be the cause of gdb not finding the symbol table?

Rainman1985
  • 435
  • 3
  • 6
  • 16

1 Answers1

1

This could be the symbols from shared libraries.

Launch:

 ldd your_binary

and see what shared objects it is using. Then download debug symbols for this so's.

woockashek
  • 1,588
  • 10
  • 25
  • Thanks for the quick reply:-) It seems that all the "so"s are third party libraries. However, most of the methods being invoked should be ceph's own methods which are all within the "ceph-osd" binary body. Is there any other possible causes? I found that the few methods whose symbols are found are all "C" methods, while ceph is mostly written in C++, is there any difference between loading symbols of C code and that of C++ code? Could that be a cause? – Rainman1985 Nov 19 '16 at 12:19
  • As far as I know this could be also caused by libstdc++ version mismatch between compilation and your environment. – woockashek Nov 19 '16 at 12:35
  • Is there any way to detect which version of libstdc++ I used to compile the ceph-osd that created the core file? Since the executable that created the core file was built a long time ago, I couldn't remenber which version of libstdc++ is built with it – Rainman1985 Nov 19 '16 at 14:35
  • if you are using the same environment for building and running it's not the point here – woockashek Nov 19 '16 at 14:38
  • I built it a long time ago, and I can't remember if it was built with the same version that I'm building with now. – Rainman1985 Nov 19 '16 at 14:39
  • still all libraries are rather backward compatible. You can check the output of `strings | grep GLIBC` and compare with the output from the same command for libstdc++ – woockashek Nov 19 '16 at 14:45
  • check also what debug symbols do you have in your binary by `nm --debug-syms ` – woockashek Nov 19 '16 at 14:47
  • Hi, I've checked that I did compile the source with -g, and the versions of libstdc++ are the same. And even there is no the "core file truncated" warning, gdb still can't find the symbol table, why? – Rainman1985 Nov 20 '16 at 10:44
  • have you checked the output on `nm`? Check if there are symbols that gdb couldn't find. Maybe not all of your files was compiled with -g. Maybe you set only CFLAGS but not CXXFLAGS. – woockashek Nov 20 '16 at 10:54
  • I checked the g++ commands in the output of make, all of them are with -g. – Rainman1985 Nov 20 '16 at 11:12
  • Well, I need to correct something. Previously, I said that the few methods whose symbols are found are all "C" methods. This is not true, some of them are C++ methods – Rainman1985 Nov 20 '16 at 11:16