Kill the most memory-hungry process when memory gets low

Question

Sometimes my Linux desktop, laptop, or server run out of RAM because the experimental software I work with occasionally tries to use more RAM than I have available. When this happens, the system and console hangs for several minutes while Linux thrashes memory between RAM and swap. It seems like Linux is trying very hard to prevent any program from being killed, but in my case, I'd just like Linux to go ahead and kill it so I can get back to work ASAP and rerun it without hard-rebooting or waiting several minutes.

Is there a third-party program I can run in the background that implements the following logic?

when RAM usage is >98%:
    kill the process using the most resident RAM

I've tried sudo sysctl vm.overcommit_memory=2, but it doesn't do what I think is best in my situation. I don't want to kill small processes that innocently allocate 1MB of RAM when RAM is low. I want to kill the single process that's consuming 31.9GB of RAM or something, since that process is obviously at fault, not the small process.

Also, vm.overcommit_memory is not what I want because vm.overcommit_ratio corresponds to ratio of virtual memory, not resident memory, divided by total physical memory. So an application that allocates 2GB of memory but doesn't touch any blocks (and therefore doesn't exist in physical RAM) shouldn't be counted toward in the overcommit_ratio.

score 2 · Answer 1 · answered Apr 28 '20 at 01:12

I wrote a quick implementation in Python 3.6+ with the psutil library. It seems to work great! It might have some issues, but I'll tweak it over time.

import time
import psutil

percent_threshold = 0.95

def check_memory():
    vm = psutil.virtual_memory()
    percent = vm.active / vm.total
    # Do we need to kill a process?
    if percent > percent_threshold:
        print(f"RAM usage is {percent}%")
        # Search for process using the most resident memory
        max_mem = 0
        max_ps = None
        for ps in psutil.process_iter():
            mem = ps.memory_info().rss
            if mem > max_mem:
                max_ps = ps
                max_mem = mem
        # Kill process
        cmd = " ".join(max_ps.cmdline())
        print(f"Killing {cmd}")
        print(f"using {max_mem / 2**20} MiB resident memory")
        max_ps.terminate()
        print("")

while True:
    check_memory()
    time.sleep(1.0)

terminate() could be replaced with kill() to more quickly kill a program, but terminating is quick enough if the threshold is set as low as 95%. During the time a program fills up the last 5%, the process chosen by this script can terminate cleanly in time.

Kill the most memory-hungry process when memory gets low

1 Answers1