I have an autoscaling group with lifecycle hooks for autoscaling:EC2_INSTANCE_LAUNCHING and autoscaling:EC2_INSTANCE_TERMINATING.
I have EventBridge configured to watch for those events and as I understand it they are supposed to go to CloudTrail. The problem is that even watching CloudTrail directly I don't seem to get near as many PutLifecycle events as it seems like I should.
Testing method:
- create ASG with hooks as above
- wait for it to stabilize
- bump up 'desired capacity'
- new member instance created
- wait for event to show up in CloudTrail. Sometimes it shows up, sometimes not.
- decrement 'desired capacity'.
- new member instance terminates.
- wait for event to show up in CloudTrail. Sometimes it shows up, sometimes not.
This almost feels like an IAM problem, but even doing all the above as full admin the results are spotty.
I've also tried the ASG -> SNS -> SQS route and gotten similar results.
Is there something in the guts of ASG events I'm not understanding? Is there somewhere else I should be looking?