1

On GCP we have created various logs-based metric where we basically are looking for log entries with some keywords like 'error', 'warning' etc...

We also created alerts based on those metrics.

Most of the logs come from cron like applications.

However when GCP creates incident it does not allow to manually close it. According to documentation the incident will close itself after 7 days or so. GCP sends alert only on the first occurrence of the incident. Therefore subsequent incidents, generated by cron (for example every hour) are treated as existing incident and alert is not being sent.

Is there any way to configure GCP Monitoring and Alerting to sent alert on every occurrence of the same incident?

bogumbiker
  • 87
  • 1
  • 7
  • I understand that you’re trying to receive an email every time your internal cron job returns an error or warning, etc., even if it is the same error as before. Could you tell us if you’re using App Engine or Compute Engine VMs? Perhaps a better approach for your question would be to make use of Cloud Functions, Cloud Scheduler and Pub/Sub, in a similar setup to this: https://edigleyssonsilva.medium.com/how-to-periodically-run-a-cloud-function-using-google-cloud-scheduler-a3e44b5d867c. – Jofre Mar 31 '21 at 15:45
  • Thanks @Jofre for the info. We are on GKE. Sorry but I do not understand how we can use Cloud Functions or Scheduler and Pub/Sub for this particular problem? – bogumbiker Apr 06 '21 at 03:13
  • Well, in the place where you're writing your error logs (i.e. `log.Error('This is the error we're monitoring')`), replace for sending a message to pubsub (i.e. `pubsub.Send(`This is the error we're monitoring')`). This will send an event that you can handle in any way you want. One of the ideas would be to have a pubsub triggered function that sends an email for every event. Messages to pubsub can be structured (to include as much detail as you want), so the resulting email could have as much detail as you need. – Jofre Apr 06 '21 at 07:42
  • In summary, pubsub is a messaging system, and it looks like you want to receive a message every time you get an error, which is why I thought about that product (rather than monitoring alerts): https://cloud.google.com/pubsub – Jofre Apr 06 '21 at 07:43
  • @Jofre thanks but we watch logs from several services and do not have own or custom errors. The beauty of GCP Alerting is that it has all email, slack, etc notification build in. – bogumbiker Apr 06 '21 at 23:21

1 Answers1

1

The Alerting tool has been thought to send emails only when the incident appears and when it finishes, it isn't able to send an email every time your internal cron reports the error, if the original incident wasn't closed first.

An alternative could be to use your custom log metrics, logs router and pub/sub.

Through logs router, create a sink with a filter (of whichever error you want to report), then export it to pub/sub and through pub/sub trigger a cloud function to send an email with whichever content you require.

You can find more information on exporting logs here, how to use pub/sub and cloud functions in here and in this link how to send an email.

verdier
  • 26
  • 4