Kill all R processes that hang for longer than a minute

Question

I use crontask to regularly run Rscript. Unfortunately, I need to do this on a small instance of aws and the process may hang, building more and more processes on top of each other until the whole system is lagging.

I would like to write a crontask to kill all R processes lasting longer than one minute. I found another answer on Stack Overflow that I've adapted that I think would solve the problem. I came up with;

if [[ "$(uname)" = "Linux" ]];then killall --older-than 1m "/usr/lib/R/bin/exec/R --slave --no-restore --file=/home/ubuntu/script.R";fi

I copied the task directly from htop, but it does not work as I expect. I get the No such file or directory error but I've checked it a few times.

I need to kill all R processes that have lasted longer than a minute. How can I do this?

I suspect you need "pkill" rather than "killall". http://superuser.com/questions/220517/how-to-kill-a-process-with-name-having-spaces — Jason, Jan 20 '16 at 02:19

jyvet · Accepted Answer · 2016-02-11T06:10:02.397

8

You may want to avoid killing processes from another user and try SIGKILL (kill -9) after SIGTERM (kill -15). Here is a script you could execute every minute with a CRON job:

#!/bin/bash

PROCESS="R"
MAXTIME=`date -d '00:01:00' +'%s'`

function killpids()
{
    PIDS=`pgrep -u "${USER}" -x "${PROCESS}"`

    # Loop over all matching PIDs
    for pid in ${PIDS}; do
        # Retrieve duration of the process
        TIME=`ps -o time:1= -p "${pid}" |
              egrep -o "[0-9]{0,2}:?[0-9]{0,2}:[0-9]{2}$"`

        # Convert TIME to timestamp
        TTIME=`date -d "${TIME}" +'%s'`

        # Check if the process should be killed
        if [ "${TTIME}" -gt "${MAXTIME}" ]; then
            kill ${1} "${pid}"
        fi
    done
}

# Leave a chance to kill processes properly (SIGTERM)
killpids "-15"
sleep 5

# Now kill remaining processes (SIGKILL)
killpids "-9"

edited Feb 11 '16 at 06:10

answered Feb 06 '16 at 12:42

jyvet

2,021
15
22

A few comments: `$USER` should already by a usable shell variable. piping `pgrep` to a plain `xargs` is redundant. You can save a bit on processing `ps` output if you use the correct options: `ps -o time:1= -p ${pid}` will give you just the time for that PID. – muru Feb 10 '16 at 22:36
Will this not select all processes that have `R` in their name? – Miserable Variable Feb 10 '16 at 23:39
1

@MiserableVariable the `-x` option of `pgrep` should be used, I guess. – muru Feb 10 '16 at 23:47
Yes it would be safer. Edited. – jyvet Feb 11 '16 at 06:11

score 8 · Answer 2 · answered Feb 06 '16 at 18:23

8

Why imply an additional process every minute with cron? Would it not be easier to start R with timeout from coreutils, the processes will then be killed automatically after the time you chose.

timeout [option] duration command [arg]…

answered Feb 06 '16 at 18:23

user1747036

524
4
12

so it would look like: ```timeout --kill-after=1m Rscript /home/ubuntu/script.R``` – cylondude Feb 09 '16 at 01:04
1

Almost, here it worked with: timeout 60 Rscript /home/ubuntu/script.R Alternatively, you can test with time timeout 60 Rscript /home/ubuntu/script.R – user1747036 Feb 10 '16 at 06:51

score 5 · Answer 3 · edited May 23 '17 at 11:51

I think the best option is to do this with R itself. I am no expert, but it seems the future package will allow executing a function in a separate thread. You could run the actual task in a separate thread, and in the main thread sleep for 60 seconds and then stop().

Previous Update user1747036's answer which recommends timeout is a better alternative.

My original answer

This question is more appropriate for superuser, but here are a few things wrong with

if [[ "$(uname)" = "Linux" ]];then 
  killall --older-than 1m \
    "/usr/lib/R/bin/exec/R --slave --no-restore --file=/home/ubuntu/script.R";
fi

The name argument is either the name of image or path to it. You have included parameters to it as well
If -s signal is not specified killall sends SIGTERM which your process may ignore. Are you able to kill a long running script with this on the command line? You may need SIGKILL / -9

More at http://linux.die.net/man/1/killall

Kill all R processes that hang for longer than a minute

3 Answers3