I don't understand completely how stackdriver alerting should work. I have Custom Metrics and an Alert based on it. I got an alert triggered on the metrics. After that I turned off the alert. But in about 1 hour I got second notification from the alert.
It is very basic alert on threshold. Violates when: custom.googleapis.com/uptimecheck/external-api is above a threshold of 10 for greater than 5 minutes
Is it correct behaviour?
There is some latency involved in making changes to alerting policies, even changes as simple as disabling the policies (I'm assuming that by "turned off the alert" you mean that you clicked the big blue ON/OFF switch in the top right corner). An hour seems like a long time, but it's conceivable that it was just latency. Did it ever fire again after that?
If you think this isn't functioning correctly, you could file a support ticket and we'll investigate the policy in more detail.
Aaron Sher, Stackdriver engineer
No definitely is not the correct behavior.
I've just create a new alert policy based on cpu threshold. As soon as I turned it off no more alert will be sent.
