1

On our web server, we perform an incremental backup of our files every 4 hours using rsnapshot. To perform the actual copying of files, I've configured it to use a custom script, cp-nice that has the following contents:

nice -n19 ionice -c3 cp "$@"

As you can see, this calls cp via nice and ionice to keep it from overloading the system. As far as I can tell, this has the desired effect (backups are made without bringing the system to a halt).

However, our NewRelic monitoring system freaks out every time a backup is made, since it detects that system IO is at 100%. Is there a way to perform these backups without causing NewRelic to think there's an issue on the server? Perhaps there's some way to "whitelist" a process in NewRelic so that it doesn't count against IO?

This seems like a pretty common use case - certainly, someone must have encountered and solved a situation like this before!

rinogo
  • 329
  • 4
  • 14
  • For others who happen upon this question, although I don't think it's right for our needs, you *might* find an acceptable solution by looking into how to tweak the CFQ scheduler. – rinogo Sep 11 '15 at 19:21

1 Answers1

1

nice and ionice only set CPU/IO priority. To clarify, if there are two competing IO requests and one of which has a higher priority, that one will be fulfilled first.

If there is no other competing IO/CPU, these processes will gladly consume all available resources.

The answer to this is to configure your monitoring checks to know when this type of activity happens and to either suppress alerts during that time frame or adjust alerting thresholds.

EEAA
  • 109,363
  • 18
  • 175
  • 245
  • Thansk, @EEAA. It sounds like we don't really have any good options to prevent this false alarm. Oh well; it was worth asking! I think the best I can do is to see if I can totally disable IO warnings in NewRelic. Thanks for your answer! – rinogo Sep 11 '15 at 19:15