8

We have a debian linux webserver. It's just running apache2. Our mysql server is on another host. However we sometimes run cron tasks on the webserver to do regular tasks.

However recently one of the cron tasks had a bug and started to gobble up the memory. The Linux OOM killer killed apache. Which of course brought down our web site. The memory hungry cron kept running. However in this case, I'd like the OOM killer to kill that script, and not apache.

Is there some way to configure the kernel so that I can say don't kill processes called 'apache2' (or at least make apache2 be the last thing it kills)? Both apache and the regular crons are run as the same user (www-user).

Amandasaurus
  • 31,471
  • 65
  • 192
  • 253

5 Answers5

3

It doesn't sound like you are addressing the root cause of the issue by actually debugging why this cron job is using up so much memory.

You can try setting this option

echo 1 > /proc/sys/vm/oom_kill_allocating_task

which will tell the OOM killer to kill the process that triggered the OOM condition, but this is not guaranteed to be your cron job. You can also use "ulimit -m" in your script to set the max amount of resident memory to use. I think your best bet would be to evaluate why the cronjob is using up so much memory and if its perhaps best suited for another host or to be rewritten to consume less memory.

jlintz
  • 259
  • 1
  • 4
  • Yes I know the root cause should be addressed, but I also would like some way to 'prioritize' apache. E.g. I'd rather kill sshd than apache on this server. I'd like to tell the computer this. However I had forgotten about ulimit, that's a good suggestion. – Amandasaurus Jan 21 '10 at 21:47
3

The OOMKiller is configurable to an extent. After you have launched a process you can set the value of /proc/<pid>/oom_adj to a negative integer. This will affect the affinity of OOMKiller towards the process and its children . When your system hits an out of memory condition, other processes will be killed.

Kamil Kisiel
  • 12,184
  • 7
  • 48
  • 69
1

You can also change the Virtual Memory over commit behaviour. For example, you can change the value of /proc/sys/vm/overcommit_memory to '2' -- meaning do NOT overcommit memory. (Do not just change this value without understanding what it does.)

In 'no overcommit' mode, whatever new process asks for more ram, will receive an error when it tries to allocate. So rather than have the OOM killer go and wack your old, long running process(es), the new guy asking for RAM is told 'no'.

...and then you need to fix your memory issue. Find the leak, redesign the memory-consuming process, add more ram to the box, etc.

Craig Constantine
  • 759
  • 8
  • 22
0

Short answer: No, the OOM-killer is not configurable, and there is/has been resistance to changing that.

Just a few ideas off the top of my head:

  1. Increase swap space - if the scripts are 32-bit, then it should be easy to give them too much space to exhaust.

  2. Increase physical memory. At 1 pretty much.

  3. Use ulimit to restrict the amount of memory the scripts can take.

Douglas Leeder
  • 2,745
  • 18
  • 15
0

It says here that you can set a OOM_DISABLE "flag" on a process: http://linux-mm.org/OOM_Killer

Rolf
  • 1