We're using Stackdriver services even before they were bought by Google in Legacy Stackdriver.
We're still using it and there are no significant issues with creating alerting policies for custom metrics or metrics coming from the agent.
If you don't see any of your EC2 instances in Stackdriver interfaces like dashboards or when creating alerting policies, you might have problem with linking your AWS account to Stackdriver.
Check that you have the proper IAM role that give to Stackdriver the right permissions to do its job.
Check trust relationship and conditions configuration in the IAM role you're using for linking your AWS account to Stackdriver.
In general;
You have to properly configure your AWS account in Stackdriver.
- There is couple of steps you have to take before you even think to install stackdriver agent on your AWS instances.
- You have to link your AWS account into Stackdriver by following the steps in Quickstart for AWS
- Note: Do not change GOOGLE_APPLICATION_CREDENTIALS file location. Just leave it as it is: "/etc/google/auth/application_default_credentials.json"
- I'm adding a note here because I've spend some time debugging this and apparently the agent does not care about this environment variable. (at least at the time I'm writing this)
- Synchronization between your AWS account and Stackdriver will take some time. Leave it for an hour to take all Cloudwatch metrics, s3 buckets size, etc...
- Install the agent on your instance and wait 5-10 minutes to start generating metrics like memory/cpu/disk/etc..
- Create new alerting policy.
- Click on Conditions
- Select Metric Absence
- Select RESOURCE TYPE: Instance (EC2)
- Select Single/Group or All. (define who this alert will monitor)
- Choice TRIGGER IF METRIC that you're sure you should always have.
- Example: "CPU User (Agent)"
- Choice "IS ABSENT FOR" - example 20 minutes.
- Click Save Condition
- Add Notification method (You can define multiple notification methods)
- Type Name of the policy and click Save Policy
You can have maximum Conditions in a single alerting policy.