0

I'm running a Debian 8 Jessie development server. 16GB mem.

I've looked at the other posts about oom-killer, but none are really pertinent to what I'm asking here

I'm trying to track down what process caused an oom-killer event. There's a memory dump, but no headers to the columns - so I don't know what the numbers mean. I'm not finding this info from Googling.

There's a couple of lines that look like potential problems, but I don't know until I can interpret the info.

I'm especially curious as to what the -999 docker-containe means ....

DUMP

Oct 26 10:41:09 Debian-89-jessie-64-minimal kernel: [6016369.996382] sshd invoked oom-killer: gfp_mask=0x2000d0, order=2, oom_score_adj=-1000
Oct 26 10:41:09 Debian-89-jessie-64-minimal kernel: [6016369.996458] sshd cpuset=/ mems_allowed=0
Oct 26 10:41:09 Debian-89-jessie-64-minimal kernel: [6016369.996589] CPU: 5 PID: 943 Comm: sshd Tainted: G         C    3.16.0-6-amd64 #1 Debian 3.16.56-1+deb8u1
Oct 26 10:41:09 Debian-89-jessie-64-minimal kernel: [6016369.996661] Hardware name: System manufacturer System Product Name/P8H67-M PRO, BIOS 3904 04/27/2013
Oct 26 10:41:09 Debian-89-jessie-64-minimal kernel: [6016369.997200]  0000000000000000 ffffffff81533b49 00000000002000d0 0000000000000000
Oct 26 10:41:09 Debian-89-jessie-64-minimal kernel: [6016369.997417]  ffffffff815326ab ffffffff81537f8e 0000000000000200 ffffffff8106b413
Oct 26 10:41:09 Debian-89-jessie-64-minimal kernel: [6016369.997634]  ffffffff810c91a4 ffffffff8115c7f3 ffff8800083ea210 ffffffff8172faca
Oct 26 10:41:09 Debian-89-jessie-64-minimal kernel: [6016369.997849] Call Trace:
Oct 26 10:41:09 Debian-89-jessie-64-minimal kernel: [6016369.997905]  [<ffffffff81533b49>] ? dump_stack+0x5d/0x78
Oct 26 10:41:09 Debian-89-jessie-64-minimal kernel: [6016369.997962]  [<ffffffff815326ab>] ? dump_header+0x95/0x1fd
Oct 26 10:41:09 Debian-89-jessie-64-minimal kernel: [6016369.998019]  [<ffffffff81537f8e>] ? mutex_lock+0xe/0x30
Oct 26 10:41:09 Debian-89-jessie-64-minimal kernel: [6016369.998078]  [<ffffffff8106b413>] ? put_online_cpus+0x23/0x90
Oct 26 10:41:09 Debian-89-jessie-64-minimal kernel: [6016369.998135]  [<ffffffff810c91a4>] ? rcu_oom_notify+0xc4/0xe0
Oct 26 10:41:09 Debian-89-jessie-64-minimal kernel: [6016369.998195]  [<ffffffff8115c7f3>] ? do_try_to_free_pages+0x483/0x520
Oct 26 10:41:09 Debian-89-jessie-64-minimal kernel: [6016369.998254]  [<ffffffff8114a5fd>] ? oom_kill_process+0x21d/0x370
Oct 26 10:41:09 Debian-89-jessie-64-minimal kernel: [6016369.998312]  [<ffffffff8114a19d>] ? find_lock_task_mm+0x3d/0xa0
Oct 26 10:41:09 Debian-89-jessie-64-minimal kernel: [6016369.998369]  [<ffffffff8114adbe>] ? out_of_memory+0x4be/0x4f0
Oct 26 10:41:09 Debian-89-jessie-64-minimal kernel: [6016369.998427]  [<ffffffff81151776>] ? __alloc_pages_nodemask+0xca6/0xcf0
Oct 26 10:41:09 Debian-89-jessie-64-minimal kernel: [6016369.998486]  [<ffffffff8106875a>] ? copy_process.part.25+0x11a/0x1ae0
Oct 26 10:41:09 Debian-89-jessie-64-minimal kernel: [6016369.998544]  [<ffffffff8106a2d3>] ? do_fork+0xc3/0x400
Oct 26 10:41:09 Debian-89-jessie-64-minimal kernel: [6016369.998601]  [<ffffffff8105a4cb>] ? __do_page_fault+0x1ab/0x470
Oct 26 10:41:09 Debian-89-jessie-64-minimal kernel: [6016369.998659]  [<ffffffff81539ef9>] ? stub_clone+0x69/0x90
Oct 26 10:41:09 Debian-89-jessie-64-minimal kernel: [6016369.998715]  [<ffffffff81539b5c>] ? system_call_fast_compare_end+0x1c/0x21
Oct 26 10:41:09 Debian-89-jessie-64-minimal kernel: [6016369.998783] Mem-Info:
Oct 26 10:41:09 Debian-89-jessie-64-minimal kernel: [6016370.079179] [20093]  1006 20093   306152     5057     107     1873             0 node
Oct 26 10:41:09 Debian-89-jessie-64-minimal kernel: [6016370.079247] [20120]     0 20120    13507       38      32       73             0 cron
Oct 26 10:41:09 Debian-89-jessie-64-minimal kernel: [6016370.079315] [20123]  1006 20123     1085        7       8       15             0 sh
Oct 26 10:41:09 Debian-89-jessie-64-minimal kernel: [6016370.079383] [20124]  1006 20124   306153     4858     110     2030             0 node
Oct 26 10:41:09 Debian-89-jessie-64-minimal kernel: [6016370.079452] [20557]     0 20557    13507       91      32       20             0 cron
Oct 26 10:41:09 Debian-89-jessie-64-minimal kernel: [6016370.079520] [20560]  1006 20560     1085       22       8        0             0 sh
Oct 26 10:41:09 Debian-89-jessie-64-minimal kernel: [6016370.079588] [20562]  1006 20562   306109     4393     110     2423             0 node
Oct 26 10:41:09 Debian-89-jessie-64-minimal kernel: [6016370.079656] [21447]     0 21447    13507       93      32       18             0 cron
Oct 26 10:41:09 Debian-89-jessie-64-minimal kernel: [6016370.079725] [21450]  1006 21450     1085       23       8        0             0 sh
Oct 26 10:41:09 Debian-89-jessie-64-minimal kernel: [6016370.079793] [21452]  1006 21452   305668     6278     108      261             0 node
Oct 26 10:41:09 Debian-89-jessie-64-minimal kernel: [6016370.079861] [21489]     0 21489    13507       93      32       18             0 cron
Oct 26 10:41:09 Debian-89-jessie-64-minimal kernel: [6016370.079929] [21492]  1006 21492     1085       22       8        0             0 sh

                                                                                         (snip ... a bazillion node, sh, cron)

Oct 26 10:41:09 Debian-89-jessie-64-minimal kernel: [6016370.176046] [23607]     0 23607    23318      237      49        0             0 sshd
Oct 26 10:41:09 Debian-89-jessie-64-minimal kernel: [6016370.176114] [ 2963]  1010  2963     8982      203      23        0             0 systemd
Oct 26 10:41:09 Debian-89-jessie-64-minimal kernel: [6016370.176183] [ 3198]  1010  3198    49655      573      31      154             0 (sd-pam)
Oct 26 10:41:09 Debian-89-jessie-64-minimal kernel: [6016370.176251] [ 3843]  1010  3843    23318      240      46        0             0 sshd
Oct 26 10:41:09 Debian-89-jessie-64-minimal kernel: [6016370.176319] [ 4188]  1010  4188     6510      867      18        0             0 bash
Oct 26 10:41:09 Debian-89-jessie-64-minimal kernel: [6016370.176387] [ 6721]     0  6721    13507       97      32       14             0 cron
Oct 26 10:41:09 Debian-89-jessie-64-minimal kernel: [6016370.176455] [28824]     0 28824    23319      237      50        0             0 sshd
Oct 26 10:41:09 Debian-89-jessie-64-minimal kernel: [6016370.176523] [ 9005]   110  9005     9560      142      21        0             0 pickup
Oct 26 10:41:09 Debian-89-jessie-64-minimal kernel: [6016370.176591] [ 9898]  1006  9898     1085       22       8        0             0 sh
Oct 26 10:41:09 Debian-89-jessie-64-minimal kernel: [6016370.176659] [ 9900]  1006  9900   300400     6495      94        0             0 node
Oct 26 10:41:09 Debian-89-jessie-64-minimal kernel: [6016370.176727] [21244]  1010 21244    23319      242      47        0             0 sshd
Oct 26 10:41:09 Debian-89-jessie-64-minimal kernel: [6016370.176795] [22039]  1010 22039     3177       42      12        0             0 sftp-server
Oct 26 10:41:09 Debian-89-jessie-64-minimal kernel: [6016370.176864] [31202]     0 31202    23318      238      48        0             0 sshd

                                                                                         (snip ... a bazillion node, sh, cron)

Oct 26 10:41:09 Debian-89-jessie-64-minimal kernel: [6016370.182092] [13192]     0 13192     1849      209       8        0          -999 docker-containe
Oct 26 10:41:09 Debian-89-jessie-64-minimal kernel: [6016370.182163] [13208]     0 13208   352255    84126     319        0             0 zcashd

                                                                                         (snip ... a bazillion node, sh, cron)

Oct 26 10:41:09 Debian-89-jessie-64-minimal kernel: [6016370.184048] [13334]     0 13334     1085       30       8        0             0 sessionclean
Oct 26 10:41:09 Debian-89-jessie-64-minimal kernel: [6016370.184118] [13335]  1006 13335    11718       53      19        0             0 node
Oct 26 10:41:09 Debian-89-jessie-64-minimal kernel: [6016370.184185] [13336]     0 13336     9884       49      18        0             0 php
Oct 26 10:41:09 Debian-89-jessie-64-minimal kernel: [6016370.184255] [13342]     0 13342     1085       30       8        0             0 sessionclean
Oct 26 10:41:09 Debian-89-jessie-64-minimal kernel: [6016370.184324] [13344]     0 13344     6447       29      11        0             0 sort
Oct 26 10:41:09 Debian-89-jessie-64-minimal kernel: [6016370.184392] [13345]     0 13345     6447       29      11        0             0 sort
Oct 26 10:41:09 Debian-89-jessie-64-minimal kernel: [6016370.184461] [13346]     0 13346     1085       28       8        0             0 sessionclean
Oct 26 10:41:09 Debian-89-jessie-64-minimal kernel: [6016370.184530] [13347]     0 13347    12216      139      28        0             0 sshd
Oct 26 10:41:09 Debian-89-jessie-64-minimal kernel: [6016370.184599] [13362]     0 13362    11068       74      19        0             0 docker
Oct 26 10:41:09 Debian-89-jessie-64-minimal kernel: [6016370.184668] [13363]     0 13363     1085       28       8        0             0 sh
Oct 26 10:41:09 Debian-89-jessie-64-minimal kernel: [6016370.184736] [13364]  1006 13364     1085       28       8        0             0 sh
Oct 26 10:41:09 Debian-89-jessie-64-minimal kernel: [6016370.184806] [13365]     0 13365     8808       55      23       25             0 cron
Oct 26 10:41:09 Debian-89-jessie-64-minimal kernel: [6016370.184875] [13366]     0 13366     8808       55      23       25             0 cron
Oct 26 10:41:09 Debian-89-jessie-64-minimal kernel: [6016370.184945] [13372]   111 13372     5874       67      16       76             0 nrpe
Oct 26 10:41:09 Debian-89-jessie-64-minimal kernel: [6016370.185015] [13373]     0 13373     9884       49      18        0             0 php7.2
Oct 26 10:41:09 Debian-89-jessie-64-minimal kernel: [6016370.185083] [13374]  1006 13374    11728      197      15        0             0 node
Oct 26 10:41:09 Debian-89-jessie-64-minimal kernel: [6016370.185153] [13375]     0 13375     9884       48      20        0             0 php
Oct 26 10:41:09 Debian-89-jessie-64-minimal kernel: [6016370.185221] [13378]     0 13378     7185       33      18       39             0 cron
Oct 26 10:41:09 Debian-89-jessie-64-minimal kernel: [6016370.185289] [13380]     0 13380     7185       33      18       39             0 cron
BurningKrome
  • 525
  • 2
  • 12
  • 22

1 Answers1

2

Use the source! Searching Debian source code for the phrase "invoked oom-killer" finds mm/oom_kill.c which has the 9 columns header it is supposed to print.

 * Dumps the current memory state of all eligible tasks.  Tasks not in the same
 * memcg, not in the same cpuset, or bound to a disjoint set of mempolicy nodes
 * are not shown.
 * State information includes task's pid, uid, tgid, vm size, rss, nr_ptes,
 * swapents, oom_score_adj value, and name.
 */
static void dump_tasks(const struct mem_cgroup *memcg, const nodemask_t *nodemask)
{
    struct task_struct *p;
    struct task_struct *task;

    pr_info("[ pid ]   uid  tgid total_vm      rss nr_ptes swapents oom_score_adj name\n");

This has been discussed on UNIX SE, see question Debug out-of-memory with /var/log/messages

-999 stood out to you because that process (docker something by the name) is the only one in this set of rows with its oom score adjusted way down. (Which seems like a greedy init script to me.) But its resident memory (rss) is tiny, so its possibly not a huge memory consumer.

Also look for a Killed process line and how big it was. OOM kills tasks that will free up the most memory resources.

Monitor system wide memory consumption, and determine whether the host has sufficient memory. The basics would be capturing /proc/meminfo over time and counting the number of tasks. Monitoring tools will help with collecting data. If you don't have tools like this, consider starting with netdata's available memory, memory per user, and process count graphs.

You have quite a grab bag of things running, cron and php and node and docker... Verify any scheduled tasks and apps that are supposed to be running, are performing tolerably.

John Mahowald
  • 32,050
  • 2
  • 19
  • 34