I've looked into nice, cpulimit, and cggroups, but I'm not sure if any of them would do what I want to do. Maybe they do, but I'm doing it wrong.
My end goal is for CentOS to look into all the apache processes, and if they add up to a certain amount of CPU usage, kill the one with the biggest %. Is there such a tool out there that does that, or this is something I have to write a shell/python script and then schedule a cronjob?