2

This happened on an Ubuntu 14.04 64-bit DigitalOcean Droplet:

Apr 22 02:17:02 localhost CRON[32514]: (root) CMD ( cd / && run-parts --report /etc/cron.hourly) 
Apr 22 03:16:59 localhost kernel: [211877.884132] show_signal_msg: 12 callbacks suppressed 
Apr 22 03:16:59 localhost kernel: [211877.884140] python[2376]: segfault at 24 ip 00000000004c278f sp 00007fbad163caa0 error 6 in python2.7[400000+2bd000]

Right around 3:16 AM, the CPU spiked up to 100%, and the machine became unresponsive to ping, SSH and (remote) console:

https://i.stack.imgur.com/wlNHA.png

How could the "callback suppressed" throttling appear before the first segfault?

http://lxr.free-electrons.com/source/arch/x86/mm/fault.c#L727

In general, how could a few segfaults cause the kernel to lock up? The offending process was in 'while :; do python script.py; sleep 1; done' (NOTE sleep), was running with 'nice 10' and had an increased oom_adj so that the kernel would kill it first:

open('/proc/self/oom_adj', 'w').write('10')
os.nice(10)
Catalin P
  • 39
  • 1
  • 1
  • 4
  • If 12 callbacks from `show_signal_msg` were suppressed that means that something segfaulted 13 times in a row. Do you have coredumps enabled on that box? – SaveTheRbtz Apr 23 '14 at 06:56
  • `ulimit -c` is 0. You would suggest setting it to `unlimited`? How would I use the core dumps to figure out why the machine locks up? – Catalin P Apr 23 '14 at 18:23
  • No, I just suggested that "unresponsiveness" can be connected to codedumps being written to disk. – SaveTheRbtz Apr 24 '14 at 08:22
  • Ah, I doubt it - the CPU was pegged at 100% for several hours: http://imgur.com/hMSH5SV.png – Catalin P Apr 24 '14 at 21:15

1 Answers1

-1

So far looks like this kernel bug:

http://blog.nitrous.io/2014/03/10/stability-and-a-linux-oom-killer-bug.html

Catalin P
  • 39
  • 1
  • 1
  • 4
  • 1
    That is the reason one should not answer questions with links. Now I have no idea what that killer bug is :( – douglaslps Mar 12 '18 at 12:47
  • 1
    @douglaslps: Thankfully, we have the Internet Archive on our side! https://web.archive.org/web/20160310202252/https://community.nitrous.io/posts/stability-and-a-linux-oom-killer-bug – tonysdg Mar 23 '18 at 23:37
  • Sorry about that @douglaslps. – Catalin P May 14 '18 at 03:56