How to stop users from running CPU heavy programs on a server?

Question

I have a 24 core server, which users connect to through SSH.

It is not meant for them to run CPU heavy programs like MATLAB, R or their own scripts that perform simulations or things like that.

Are there ways to detect and kill -9 CPU heavy processes automatically?

`nice -n 19 matlab` or, if it is started: `renice -n 19 $(pgrep matlab)` — Mircea Vutcovici, Nov 04 '11 at 20:31
A bit more effective would be a small script that selects the top 5 CPU-consuming processes (`ps -eo pcpu,pid,user | sort -k 1 -r | head -5`) and renice those. You could do so from a cronjob. — MSalters, Nov 07 '11 at 08:45
@MSalters Won't that over time make all processes slow, as the the next 5 CPU heavy processes might just be legit processes? — Sandra, Nov 10 '11 at 11:16

score 6 · Answer 1 · answered Nov 04 '11 at 14:04

6

You can limit users resources usage with PAM.

I never tried it so I have no clue if it works properly or not...

answered Nov 04 '11 at 14:04

Alex

3,129
21
28

Very interesting! What does "Maximum CPU time" mean? I mean, what if the server is 50% utilized and a user starts a script that could max out one cpu core. Would this script then be detected? – Sandra Nov 04 '11 at 14:30
I don't want to claim that I'm an expert with this type of question but...my understanding is that it will prevent users from "hugging" the CPU. So after a total of x seconds of CPU usage the process will take the back seat. Therefore preventing 100% CPU usage from a single user. – Alex Nov 04 '11 at 14:40
4

Considering it's referencing ulimit, seems this is implemented as `RLIMIT_CPU` : "CPU time limit in seconds. When the process reaches the soft limit, it is sent a `SIGXCPU` signal. The default action for this signal is to terminate the process. However, the signal can be caught, and the handler can return control to the main program. If the process continues to consume CPU time, it will be sent `SIGXCPU` once per second until the hard limit is reached, at which time it is sent `SIGKILL`" – MSalters Nov 04 '11 at 15:15

score 3 · Accepted Answer · answered Nov 06 '11 at 16:52

You should use cgroups to do this.

See "man cgrules.conf" and "man cgconfig.conf".

Later versions of systemctl on fedora should support sticking users directly into a cgroup so you can do it better that way.

This wont limit CPU in the sense that if there is available CPU resources (which nobody is using) it will use all the CPU however it something else is also demanding CPU it will allocate a share of the CPU based off of the configured "cpu.shares" value.

Also as suggested sticking a ulimit on CPU time will ensure a running process is given a cumalative number of jiffies before being killed for using too much CPU. This might negatively impact long running processes a user is using which over a long period of time have accumulated a certain number of jiffies naturally.

You could also use cgroups to enforce that all a users processes live on one of your cores only, so that you can at least guarantee if one CPU is being overwhelmed it has no negative impact on the rest of the operating systems processes.

CGroups is also a awesome way to limit memory usage. You can combine it with pam_limits to prevent fork bombing.

Edit: I should also point out what I think your asking for is not necessarily relevant. Having 1 process use up 100% of the CPU is not necessarily bad, providing time is given for other processes to run. The completely fair scheduler on linux guarantees this behaviour anyway.

If the CPU is just idling theres nothing wrong with one process using up all the CPU. Your problem only comes where multiple processes are demanding CPU time and one of the processes is hogging the CPU. This is where cgroups should be of benefit as it permits you control how much cpu time you'll allocate different process in the event of CPU contention.

How to stop users from running CPU heavy programs on a server?

2 Answers2