mongodb mystery crashes with sigkill under strace

Question

I have a mongodb instance that seems to crash roughly once a day. I don't see any useful information in the mongo log file. All is well and then the process is just crashed with no additional information logged. I ran it under strace in hopes of getting some useful clues until it crashed and got this output:

Wed Apr 17 10:56:39.340 [conn172]               M/R: (1/3) Emit Progress: 2800/4351     64%
Wed Apr 17 10:57:16.696 [PeriodicTask::Runner] task: DBConnectionPool-cleaner took: 16ms
Wed Apr 17 10:57:17.035 [PeriodicTask::Runner] task: WriteBackManager::cleaner took: 17ms
Wed Apr 17 10:57:17.429 [PeriodicTask::Runner] task: DBConnectionPool-cleaner took: 79ms
+++ killed by SIGKILL +++

This is only happening on one specific VM and the rest are fine. Also checked hourly/daily cron jobs in case something was going haywire and killing mongod but no suspects there.

OS: Ubuntu 12.04.2

Mongo: 2.4.1

Anything else I can do to further troubleshoot?

Any chance that this is an Amazon VM? – Adam C Apr 30 '13 at 17:32 — Adam C, Apr 30 '13 at 17:32
It's a rackspace gen1 VM. – Peter Lyons Apr 30 '13 at 17:59 — Peter Lyons, Apr 30 '13 at 17:59

score 1 · Answer 1 · answered May 22 '13 at 17:40

You are running a Map Reduce job at the time it is killed, is that always the case?

I ask because it sounds like the kind of behavior you get when a watchdog type process decides you are using too much memory and kills the process. Map Reduce jobs (inline, or those that run on large data sets in particular) tend to push up RAM usage quickly.

SIGKILL is not what you would get if this was the kernel deciding you were Out Of Memory and invoking the OOM killer (that looks like a silent crash and is logged in dmesg). Hence, that would make me believe that there's something else doing the killing to prevent memory usage beyond a certain threshold.

If you wanted to verify, run db.collection.find().explain() on a large data set and see if that triggers a SIGKILL also. If so, I don't think this type of VM is suited to running a memory mapped database.

Hmm, sounds plausible I guess. I'll investigate. – Peter Lyons May 22 '13 at 17:45 — Peter Lyons, May 22 '13 at 17:45

mongodb mystery crashes with sigkill under strace

1 Answers1