0

We have over 100 servers that generate hundreds of emails every day using scheduled scripts (cron jobs) when the system/data is backed up or any other activity takes place. The email is sent to our network admin group, so a few people receive them every day. The issue is that we don't have enough time to go through each of them and verify if all the servers have been able to run their scheduled scripts and send the email. Is there an automated way to verify emails, using a client mail app or on the server side? I was thinking of generating an email only if the script fails, but what happens if that server is down?

ashvagan
  • 23
  • 1
  • 7
  • Hopefully your monitoring system would notice if the server's down. You do have one of those, don't you? – HopelessN00b Aug 15 '12 at 15:09
  • 1
    Do your emails contain identifiable tokens for success and failure? If so, you can use a categorisation, colourisation or even a simple move-to-folder rule in your email client, which searches for those tokens. If not, well that's your first issue: standardising all your emails so they can be machine-read. – jimbobmcgee Aug 15 '12 at 15:14
  • You are discovering that e-mail isn't a very scalable monitoring solution. Every small shop runs into this wall eventually. Rather than automating the reading of these e-mails, I'd suggest reevaluating your method. – Aaron Copley Aug 15 '12 at 16:21
  • yes, we do have monitoring system in place, but the scripts are mostly used to backup data or create some kind of new logs etc. – ashvagan Sep 07 '12 at 14:01

1 Answers1

1

Send email only if the scripts fail. And as HopelessNoob mentioned, for 100 servers, you should already have monitoring system in place to figure if a server is down. If not, it's high time you had one.

Chida
  • 2,491
  • 1
  • 17
  • 29
  • What happens when e-mail fails? Or networking stack all together..? – Aaron Copley Aug 15 '12 at 16:22
  • Thats why you engineer redundant systems and have a monitoring system to monitor every critical piece of your infrastructure. – Chida Aug 15 '12 at 16:38
  • You're missing my point. If a host only sends e-mail when there's a failure, but the failure is due to the host being offline... You're going to have a bad time. – Aaron Copley Aug 15 '12 at 16:41
  • Hundreds of emails are sent on success and nobody is looking at them. As your systems grow, even more emails will come in. this is setting up for failure. If a backup fails, send email. If the server running the cron itself fails, then your monitoring system should alert you. Either way, you will get emails of bad events, which will be less, and you can focus on fixing it than trying to find a needle in a haysack for a failed email. I'm not sure if I understood in any other way. – Chida Aug 15 '12 at 16:48
  • The problem is assuming that he has a monitoring system. It doesn't sound like you got all the way through the question because he specifically asks: `I was thinking of generating an email only if the script fails, but what happens if that server is down?` – Aaron Copley Aug 15 '12 at 16:53
  • If the server is down, a monitoring system such as nagios, zenoss,zabbix .. whatever blah that is monitoring all of his servers will send him or his group an email that one of the servers is down. – Chida Aug 15 '12 at 17:00
  • We do have monitoring system in place (Cacti and Nagios), but the scripts are more to do with backing up of data or syncing data between two servers. – ashvagan Sep 07 '12 at 14:03