-2

I am looking to setup some CloudFormation stuff that is able to find any email addresses in CloudWatch logs and let us know that one slipped through the cracks. I thought this would be a simple process of using a RegEx pattern that catches all the possible variations and email address can have, and using that as a filter. Having discovered that CloudWatch filtering does not support RegEx I've become a bit stumped as to how to write a filter that can be relied upon to catch any email address.

Has anyone done something similar to this, or know where a good place to start would be?

Jay Cork
  • 344
  • 1
  • 6
  • 15
  • If you have un-redacted logs that you want to send to a third party, you can use Sublime Text to retrospectively redact them - see https://codingrob.medium.com/how-to-redact-text-from-hundreds-of-text-files-in-less-than-60-seconds-511b4b91c2d8 – RobbiewOnline Mar 15 '22 at 12:13

3 Answers3

1

Amazon has launched a service called CloudWatch insights and it allows to filter messages logs. In the previous link you have examples of queries.

You need to select the CloudWatch Log Group and the period of time in which search.

Example:

fields @message
| sort @timestamp desc
| filter @message like /.*47768.*/
WaterKnight
  • 197
  • 7
0

If you're exporting the logs somewhere (Like Sumologic, Datadog etc) thats a better place to do that alerting.

If not and you're exporting them into S3 then a triggered lambda function that runs the check might do the trick. Could be expensive long term though.

Jaboy
  • 212
  • 1
  • 6
0

The solution that we landed upon was to pass stings through a RegEx pattern that recognises email addresses before they logged into AWS. Replacing any matches with [REDACTED]. Which is simple enough to do in a lambda.

Jay Cork
  • 344
  • 1
  • 6
  • 15