9

I have my service running in EC2 (under systemd). It is a self-contained app built for .Net Core 2.1. From time to time (a few times a week) it crashes with SEGV.

Apr 30 21:20:51 ip-10-4-226-55 kernel: traps: App.Name[26176] general protection ip:7f22da3609da sp:7f1fedf11510 error:0 in libc-2.26.so[7f22da2e3000+1ad000]

Apr 30 21:20:51 ip-10-4-226-55 systemd: appname.service: main process exited, code=killed, status=11/SEGV

Apr 30 21:20:51 ip-10-4-226-55 systemd: Unit appname.service entered failed state.

Apr 30 21:20:51 ip-10-4-226-55 systemd: appname.service failed.

For some reason, crash dump is not created (even though I removed the size limit). How can I investigate the problem further? What can be the source of the issue?

Community
  • 1
  • 1
Artur Shamsutdinov
  • 3,127
  • 3
  • 21
  • 39

1 Answers1

3

How can I investigate the problem further?

I'm on ArchLinux so things may be different(even though systemd is present on both), but here's what I'd try:

  1. Does the system somehow create any cores?

Let's dump a harmless core to test: in bash shell:

sleep 200 & kill -11 "$!"

this shows the following in dmesg -w:

[17894.861369] systemd[1]: Started Process Core Dump (PID 31964/UID 0).
[17895.030166] systemd-coredump[31975]: Process 31963 (bash) of user 1000 dumped core.

               Stack trace of thread 31963:
               #0  0x00007c0aff6c642b kill (libc.so.6)
               #1  0x000056e836d6c56a termsig_handler.part.2 (bash)
               #2  0x000056e836d6c6d3 termsig_handler (bash)
               #3  0x000056e836d3a1b3 execute_simple_command (bash)
               #4  0x000056e836d3b20e execute_command_internal (bash)
               #5  0x000056e836d3b469 execute_command_internal (bash)
               #6  0x000056e836d3cf12 execute_command (bash)
               #7  0x000056e836d247f2 reader_loop (bash)
               #8  0x000056e836d2320d main (bash)
               #9  0x00007c0aff6b21bb __libc_start_main (libc.so.6)
               #10 0x000056e836d235ce _start (bash)

[17895.030324] systemd[1]: systemd-coredump@5-31964-0.service: Succeeded.

and is listed as latest with coredumpctl -r |head -2:

TIME                            PID   UID   GID SIG COREFILE  EXE
Sat 2019-05-18 21:48:22 CEST  31963  1000  1000  11 present   /usr/bin/bash

also:

$ ls -rlat /var/lib/systemd/coredump/|tail -n1
-rw-r-----+ 1 root root  3907584 18.05.2019 21:48 core.bash.1000.6d7dce73cd2342759a18d47914c16007.31963.1558208902000000

so, since it's latest I can just run coredumpctl gdb to start gdb on it, and then inside gdb see some info by typing thread apply all bt full:

$ coredumpctl gdb
           PID: 31963 (bash)
           UID: 1000 (user)
           GID: 1000 (user)
        Signal: 11 (SEGV)
     Timestamp: Sat 2019-05-18 21:48:22 CEST (3min 51s ago)
  Command Line: -bash
    Executable: /usr/bin/bash
 Control Group: /user.slice/user-1000.slice/session-1.scope
          Unit: session-1.scope
         Slice: user-1000.slice
       Session: 1
     Owner UID: 1000 (user)
       Boot ID: 6d7dce73cd2342759a18d47914c16007
    Machine ID: 5767ef25f523419aaa049f3d74481940
      Hostname: i87k
       Storage: /var/lib/systemd/coredump/core.bash.1000.6d7dce73cd2342759a18d47914c16007.31963.1558208902000000
       Message: Process 31963 (bash) of user 1000 dumped core.

                Stack trace of thread 31963:
                #0  0x00007c0aff6c642b kill (libc.so.6)
                #1  0x000056e836d6c56a termsig_handler.part.2 (bash)
                #2  0x000056e836d6c6d3 termsig_handler (bash)
                #3  0x000056e836d3a1b3 execute_simple_command (bash)
                #4  0x000056e836d3b20e execute_command_internal (bash)
                #5  0x000056e836d3b469 execute_command_internal (bash)
                #6  0x000056e836d3cf12 execute_command (bash)
                #7  0x000056e836d247f2 reader_loop (bash)
                #8  0x000056e836d2320d main (bash)
                #9  0x00007c0aff6b21bb __libc_start_main (libc.so.6)
                #10 0x000056e836d235ce _start (bash)

GNU gdb (GDB) 8.2.1
Copyright (C) 2018 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-pc-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /usr/bin/bash...done.
[New LWP 31963]
Core was generated by `-bash'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x00007c0aff6c642b in kill () at ../sysdeps/unix/syscall-template.S:78
78  ../sysdeps/unix/syscall-template.S: No such file or directory.
(gdb) thread apply all bt full

Thread 1 (LWP 31963):
#0  0x00007c0aff6c642b in kill () at ../sysdeps/unix/syscall-template.S:78
No locals.
#1  0x000056e836d6c56a in termsig_handler.part ()
No symbol table info available.
#2  0x000056e836d6c6d3 in termsig_handler ()
No symbol table info available.
#3  0x000056e836d3a1b3 in execute_simple_command ()
No symbol table info available.
#4  0x000056e836d3b20e in execute_command_internal ()
No symbol table info available.
#5  0x000056e836d3b469 in execute_command_internal ()
No symbol table info available.
#6  0x000056e836d3cf12 in execute_command ()
No symbol table info available.
#7  0x000056e836d247f2 in reader_loop ()
No symbol table info available.
#8  0x000056e836d2320d in main ()
No symbol table info available.
(gdb) 

not much to see because bash wasn't compiled with debug symbols or something stripped them. Recompiling bash with extra CFLAGS before doing its ./configure ... && make like:

export CFLAGS="${CFLAGS} -fstack-protector-strong -fno-omit-frame-pointer -ftrack-macro-expansion=2 -ggdb -fvar-tracking-assignments -O0"

(maybe you won't want -O0 if you want to keep your current program's behaviour, else it may not crash anymore)
and then re-running the above sleep to create a new coredump, yields the following much richer results:

$ coredumpctl gdb
           PID: 29241 (bash)
           UID: 1000 (user)
           GID: 1000 (user)
        Signal: 11 (SEGV)
     Timestamp: Sat 2019-05-18 22:01:41 CEST (13s ago)
  Command Line: -bash
    Executable: /usr/bin/bash
 Control Group: /user.slice/user-1000.slice/session-1.scope
          Unit: session-1.scope
         Slice: user-1000.slice
       Session: 1
     Owner UID: 1000 (user)
       Boot ID: 6d7dce73cd2342759a18d47914c16007
    Machine ID: 5767ef25f523419aaa049f3d74481940
      Hostname: i87k
       Storage: /var/lib/systemd/coredump/core.bash.1000.6d7dce73cd2342759a18d47914c16007.29241.1558209701000000
       Message: Process 29241 (bash) of user 1000 dumped core.

                Stack trace of thread 29241:
                #0  0x00007775d0d2642b kill (libc.so.6)
                #1  0x000060b781bce2c8 termsig_handler (bash)
                #2  0x000060b781b9107b execute_simple_command (bash)
                #3  0x000060b781b8aa1c execute_command_internal (bash)
                #4  0x000060b781b8dde0 execute_connection (bash)
                #5  0x000060b781b8ade5 execute_command_internal (bash)
                #6  0x000060b781b89f45 execute_command (bash)
                #7  0x000060b781b72e66 reader_loop (bash)
                #8  0x000060b781b70906 main (bash)
                #9  0x00007775d0d121bb __libc_start_main (libc.so.6)
                #10 0x000060b781b6fe2e _start (bash)

GNU gdb (GDB) 8.2.1
Copyright (C) 2018 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-pc-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /usr/bin/bash...done.
[New LWP 29241]
Core was generated by `-bash'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x00007775d0d2642b in kill () at ../sysdeps/unix/syscall-template.S:78
78  ../sysdeps/unix/syscall-template.S: No such file or directory.
(gdb) thread apply all bt full

Thread 1 (LWP 29241):
#0  0x00007775d0d2642b in kill () at ../sysdeps/unix/syscall-template.S:78
No locals.
#1  0x000060b781bce2c8 in termsig_handler (sig=11) at sig.c:597
        handling_termsig = 1
        i = -2097452368
        core = 24759
        mask = {__val = {140729597269152, 106341271890333, 106341294191024, 106341271640912, 106341291662592, 106341294178848, 
            140729597269200, 106341271910973, 140729597269200, 106341294191024, 106341271640912, 106341271911463, 
            106341272462763, 0, 140729597269232, 106341271911163}}
#2  0x000060b781b9107b in execute_simple_command (simple_command=0x60b78310a8c0, pipe_in=-1, pipe_out=-1, async=1, 
    fds_to_close=0x60b7831196b0) at execute_cmd.c:4394
        words = 0x60b78310b1b0
        lastword = 0x7ffe29a79910
        command_line = 0x0
        lastarg = 0x0
        temp = 0x0
        first_word_quoted = 0
        result = 0
        builtin_is_special = 0
        already_forked = 1
        dofork = 1
        old_last_async_pid = -1
        builtin = 0x0
        func = 0x0
        old_builtin = 0
        old_command_builtin = -2098586400
#3  0x000060b781b8aa1c in execute_command_internal (command=0x60b783107410, asynchronous=1, pipe_in=-1, pipe_out=-1, 
    fds_to_close=0x60b7831196b0) at execute_cmd.c:845
        exec_result = 0
        user_subshell = 0
        invert = 0
        ignore_return = 0
        was_error_trap = 0
        my_undo_list = 0x0
        exec_undo_list = 0x0
        tcmd = 0x0
        save_line_number = 1
        ofifo = 0
        nfifo = 0
        osize = 0
        saved_fifo = 0
        ofifo_list = 0x5b0000006e <error: Cannot access memory at address 0x5b0000006e>
#4  0x000060b781b8dde0 in execute_connection (command=0x60b783119680, asynchronous=0, pipe_in=-1, pipe_out=-1, 
    fds_to_close=0x60b7831196b0) at execute_cmd.c:2690
        tc = 0x60b783107410
--Type <RET> for more, q to quit, c to continue without paging--c
        second = 0x0
        ignore_return = 0
        exec_result = -2098586400
        was_error_trap = 0
        invert = 3
        save_line_number = 0
#5  0x000060b781b8ade5 in execute_command_internal (command=0x60b783119680, asynchronous=0, pipe_in=-1, pipe_out=-1, fds_to_close=0x60b7831196b0) at execute_cmd.c:1018
        exec_result = 0
        user_subshell = 0
        invert = 0
        ignore_return = 0
        was_error_trap = 32766
        my_undo_list = 0x0
        exec_undo_list = 0x0
        tcmd = 0x0
        save_line_number = -2117800288
        ofifo = 24759
        nfifo = -2096071056
        osize = 24759
        saved_fifo = 0
        ofifo_list = 0x60b781b89d9c <dispose_fd_bitmap> "UH\211\345H\203\354\020H\211}\370H\213E\370H\213@\bH\205\300t\020H\213E\370H\213@\bH\211\307\350\253R\376\377H\213E\370H\211\307\350\237R\376\377\220\311\303UH\211\345SH\203\354\030H\211}\350H\203", <incomplete sequence \350>
#6  0x000060b781b89f45 in execute_command (command=0x60b783119680) at execute_cmd.c:394
        bitmap = 0x60b7831196b0
        result = 0
#7  0x000060b781b72e66 in reader_loop () at eval.c:175
        code = 0
        our_indirection_level = 1
        current_command = 0x60b783119680
#8  0x000060b781b70906 in main (argc=1, argv=0x7ffe29a79918, env=0x7ffe29a79928) at shell.c:805
        i = 20
        code = 0
        old_errexit_flag = 0
        saverst = 0
        locally_skip_execution = 0
        arg_index = 1
        top_level_arg_index = 1
(gdb) 

However, coredump may be created but systemd may be cleaning/removing it after a while(eg. all my coredumps from earlier than 3 days ago are missing as reported by coredumpctl - unsure why, considering my settings - maybe you're seeing a similar issue?), or not even create it due to space contrains(see all /etc/systemd/coredump.conf mentioned below).
Let's see:
Is systemd-coredump even set to run to create the coredump?

$ sysctl -a |grep kernel.core
kernel.core_pattern = |/usr/lib/systemd/systemd-coredump %P %u %g %s %t %c %h %e
kernel.core_pipe_limit = 0
kernel.core_uses_pid = 1
$ ls -la /usr/lib/systemd/systemd-coredump
-rwxr-xr-x 1 root root 55296 13.05.2019 11:46 /usr/lib/systemd/systemd-coredump*

Does kernel support coredump-ing?

$ zcat /proc/config.gz |grep -i 'core.*dump'
CONFIG_CORE_DUMP_DEFAULT_ELF_HEADERS=y
CONFIG_COREDUMP=y
CONFIG_ALLOW_DEV_COREDUMP=y
# CONFIG_PROC_VMCORE_DEVICE_DUMP is not set

CONFIG_COREDUMP=y is probably enough.

Other things I'd look at:

$ systemctl|grep core
systemd-coredump.socket                                                                  loaded active listening Process Core Dump Socket 
$ cat /etc/systemd/coredump.conf
#  This file is part of systemd.
#
#  systemd is free software; you can redistribute it and/or modify it
#  under the terms of the GNU Lesser General Public License as published by
#  the Free Software Foundation; either version 2.1 of the License, or
#  (at your option) any later version.
#
# Entries in this file show the compile time defaults.
# You can change settings by editing this file.
# Defaults can be restored by simply deleting this file.
#
# See coredump.conf(5) for details.

[Coredump]
#Storage=external
#Compress=yes
Compress=no
#ProcessSizeMax=2G
ProcessSizeMax=10G
#ExternalSizeMax=2G
ExternalSizeMax=10G
#JournalSizeMax=767M
JournalSizeMax=10G
#MaxUse=
#KeepFree=

man 5 coredump.conf shows some info:

       All options are configured in the "[Coredump]" section:

       Storage=
           Controls where to store cores. One of "none", "external", and "journal". When "none", the core dumps may be
           logged (including the backtrace if possible), but not stored permanently. When "external" (the default), cores
           will be stored in /var/lib/systemd/coredump/. When "journal", cores will be stored in the journal and rotated
           following normal journal rotation patterns.

           When cores are stored in the journal, they might be compressed following journal compression settings, see
           journald.conf(5). When cores are stored externally, they will be compressed by default, see below.

       Compress=
           Controls compression for external storage. Takes a boolean argument, which defaults to "yes".

       ProcessSizeMax=
           The maximum size in bytes of a core which will be processed. Core dumps exceeding this size may be stored, but
           the backtrace will not be generated.

           Setting Storage=none and ProcessSizeMax=0 disables all coredump handling except for a log entry.

       ExternalSizeMax=, JournalSizeMax=
           The maximum (uncompressed) size in bytes of a core to be saved.

       MaxUse=, KeepFree=
           Enforce limits on the disk space taken up by externally stored core dumps.  MaxUse= makes sure that old core
           dumps are removed as soon as the total disk space taken up by core dumps grows beyond this limit (defaults to 10%
           of the total disk size).  KeepFree= controls how much disk space to keep free at least (defaults to 15% of the
           total disk size). Note that the disk space used by core dumps might temporarily exceed these limits while core
           dumps are processed. Note that old core dumps are also removed based on time via systemd-tmpfiles(8). Set either
           value to 0 to turn off size-based clean-up.

       The defaults for all values are listed as comments in the template /etc/systemd/coredump.conf file that is installed
       by default.
$ cd /etc/systemd && grep -nrIFi core
coredump.conf:12:# See coredump.conf(5) for details.
coredump.conf:14:[Coredump]
system.conf:19:DumpCore=yes
system.conf:20:#DefaultLimitCORE=
system.conf:21:#^ man 2 setrlimit:  RLIMIT_CORE
system.conf:22:#This is the maximum size of a core file (see core(5)) in bytes that the process may dump.  When 0 no core dump
user.conf:34:#DefaultLimitCORE=
user.conf:35:#^ man 2 setrlimit:  RLIMIT_CORE
user.conf:36:#This is the maximum size of a core file (see core(5)) in bytes that the process may dump.  When 0 no core dump

those seem to work for me as set. (if changed a sudo systemctl daemon-reload is required)

See also: man 8 systemd-coredump which says that coredumps are saved in /var/lib/systemd/coredump and you may even find other useful info (and a redirection to man 5 core)

Another thing that I have had changed:

$ colordiff -up /etc/security/limits.conf.ORIG /etc/security/limits.conf
--- /etc/security/limits.conf.ORIG  2017-12-29 21:26:09.000000000 +0100
+++ /etc/security/limits.conf   2017-12-29 21:26:09.000000000 +0100
@@ -47,4 +47,11 @@
 #ftp             hard    nproc           0
 #@student        -       maxlogins       4

+#*               soft    core            unlimited
+#^ this doesn't affect the root user!! what the!
+#@root               soft    core            unlimited
+0:               soft    core            unlimited
+#^ all uids from 0 upwards! so what I thought * was doing!
+#hmm works with su -, but not with ssh !
+
 # End of file

ie. I'm using this line:
0: soft core unlimited
instead of the typically recommended one:
* soft core unlimited
though I notice now that Arch Linux recommends:
* hard core 0

Another thing that I would do is recompile glibc with full debug and symbols so they would be available the next time your program crashes in libc-2.26.so. The way I do it is make sure strip (from PKGBUILD) doesn't run and I use:

CPPFLAGS="${CPPFLAGS} -fno-omit-frame-pointer -ftrack-macro-expansion=2 -ggdb -fvar-tracking-assignments -O2"
CXXFLAGS="${CXXFLAGS} -fno-omit-frame-pointer -ftrack-macro-expansion=2 -ggdb -fvar-tracking-assignments"
CFLAGS="${CFLAGS} -fno-omit-frame-pointer -ftrack-macro-expansion=2 -ggdb -fvar-tracking-assignments"

If you still don't get a coredump(for your program!), maybe look at /proc/<pid>/coredump_filter in kernel Documentation/filesystems/proc.txt

UPDATE: Because you only have that one dmesg line(and no coredump), maybe this answer would help you get some info. You might need the source code for that glibc 2.26 that your CentOS is using, unless you're happy with just reading assembler code ;)

UPDATE2: Try running coredumpctl 26176, even though it doesn't have a core, you should still see a stacktrace, for example:

$ coredumpctl -S '2019-05-04 23:37:56' -U '2019-05-05 23:37:56'
TIME                            PID   UID   GID SIG COREFILE  EXE
Sat 2019-05-04 23:37:56 CEST   3888     0     0   7 missing   /usr/bin/mc
Sat 2019-05-04 23:40:08 CEST   3916     0     0   7 missing   /usr/bin/mc
$ coredumpctl info 3888
           PID: 3888 (mc)
           UID: 0 (root)
           GID: 0 (root)
        Signal: 7 (BUS)
     Timestamp: Sat 2019-05-04 23:37:56 CEST (2 weeks 0 days ago)
  Command Line: mc
    Executable: /usr/bin/mc
 Control Group: /user.slice/user-0.slice/session-5.scope
          Unit: session-5.scope
         Slice: user-0.slice
       Session: 5
     Owner UID: 0 (root)
       Boot ID: ce932e7af1f04bc3af1c9573c70a912d
    Machine ID: 5767ef25f523419aaa049f3d74481940
      Hostname: i87k
       Storage: /var/lib/systemd/coredump/core.mc.0.ce932e7af1f04bc3af1c9573c70a912d.3888.1557005876000000 (inaccessible)
       Message: Process 3888 (mc) of user 0 dumped core.

                Stack trace of thread 3888:
                #0  0x00007f54782d427e __memcmp_avx2_movbe (libc.so.6)
                #1  0x000055db1382fdad n/a (mc)
                #2  0x000055db137cb126 n/a (mc)
                #3  0x000055db1380102d n/a (mc)
                #4  0x000055db13801bff n/a (mc)
                #5  0x000055db137b2d6c n/a (mc)
                #6  0x000055db137b2f65 n/a (mc)
                #7  0x000055db137cc8e2 n/a (mc)
                #8  0x000055db137a6782 n/a (mc)
                #9  0x00007f547819dce3 __libc_start_main (libc.so.6)
                #10 0x000055db137a68fe n/a (mc)

Then maybe you could use the tricks I mentioned above (in the UPDATE) to look into each address, assuming you didn't update your system since the crash occurred!