0

I have working AWS cloudwatch alarm form lambda configured in the way:
Namespace: AWS/Lambda
Metric name: Errors
Statistic: Sum
Error: 5min
For conditions with static threshold >= 1

Got the alarm triggered and I cannot find what caused it to rise in the logs as they are flooded with messages that contain an "Error" phrase (one of the properties that are logged is has "error" in the name).

Does anybody know what is considered an "Error" in the AWS Metric? or how can I search for that specific event that caused the metric to rise?

I have tried quering Logs Insights for that period of time with filtering

| sort @timestamp desc
| limit 20
| filter @message like /(?i)(error)/

but only got lots of messages that looks the same and if they would be counted in the metric then it would be thousands of errors not just 1

pope_yol
  • 1

1 Answers1

0

Does anybody know what is considered an "Error" in the AWS Metric?

An "Error" in Lambda context is a function that raises an exception before completing. If you catch an exception, and handle it yourself it doesn't count. If nothing catches the exception in your code or you raise the exception manually yourself, then it counts as an error.

how can I search for that specific event that caused the metric to rise?

The best way is to filter by time. You know when your alarm was triggered. Use the time range in Cloudwatch insights to limit the search within a few minutes of the alarm trigger time.

The default logger puts [ERROR] prefix if the Lambda throws an exception and you can filter it by this query: filter @message like [ERROR]

If you have a custom logger, then you need to update the query.

Additionally, you can use AWS X-Ray which helps with tracing specific requests but it has additional cost.

Brian
  • 1,056
  • 9
  • 15
  • could be that this "error" is not appearing in the Logs Inshights for log group related to the lambda? I have tried the search for [ERROR] etc but nothing is returned. I cannot filter by time as there are thousands of entries. I have tried to search for the exact definition of the AWS ERROR, but found nothing – pope_yol Dec 22 '22 at 10:50
  • ""error" is not appearing in the Logs Inshights for log group related to the lambda?" -> If you are searching the correct log group, then the logs will appear in Log Insights – Brian Dec 22 '22 at 10:56
  • logs are present for the log group of this lambda, but there are no entries with the query that had your filter applied – pope_yol Dec 23 '22 at 15:53
  • Like Brian mentioned, it all depends on how your function is handling those errors and what logs your function is emitting in response to those errors. A good practice is to catch all exceptions and report them in a standard way. I also recommend looking at using the embedded metric format to co-locate metrics with structured log data so it’s easy to run queries like filter Error > 0 | fields ExceptionMessage – jaredcnance Dec 23 '22 at 17:01