How to exclude a certain hours in GCP Monitoring Query Language?

Question

I have this MQL script that will alert PICs if the metric is absent for 30 mins.

fetch pubsub_topic
| metric 'pubsub.googleapis.com/topic/send_request_count'
| filter
    resource.project_id == 'projectid'
    && (resource.topic_id == 'topic-id')
| align rate(1m)
| every 1m
| group_by [metric.response_code],
    [value_send_request_count_aggregate: aggregate(value.send_request_count)]
| absent_for 1800s

How do I exclude a certain hours from this query, for example do not alert from 10pm till 6am daily?

Answer 1

Currently there is no inbuilt feature to exclude non business hours or a few certain hours from alerting.

But below are the possible ways you have a try :

1. There is a work around which you can use to snooze notifications and alerts. This will not create the alerts within the time interval you set to snooze. The only problem is that we need to do this daily if the alert should be snoozed only for a particular interval of a day. This snooze can support 7 days or even more but snoozes the entire hour.

2. You can set a cloud scheduler job that enables/disables the alert in the time you prefer. Refer to this official doc for more information

3. There is a Feature Request raised under a public issue tracker about enabling the alerts only on certain hours and this is still under enhancement mode. You can Upvote this Feature Request to get future updates on this case. As now this FR is pending with the product engineering team, currently there is no ETA.

4. External customers can use Pager Duty to accomplish this but as of now it is not recommended but it is a way around. Some customers are working around this by having alert policies to send notifications to a -pager@google.com address which sends out notifications based on whatever schedule/escalation policy is configured in AlertManager.