I am hosting a computing service on Ubuntu 12.04 and I need a method to prevent users from forkbombing. I am currently using setrlimit(RLIMIT_NPROC)
in Linux. However, this actually sets a global limit on the number of processes for a given UID. Because my service has many parallel jobs, all running with the same UID, this means that one process can DOS the entire service.
Instead I am looking for a way to set a restriction on e.g. the recursive number of child procs for the current process. I have been told to look into linux cgroups
/ libcgroup
but I honestly don't really understand how this could solve my problem. Any tips?