6

I looked through the documentation of CloudWatch Logs, but I could not find any method to manually remove partial data from a log stream.

The only way to remove data is to:

  • Remove the log stream, but that will remove all data (not just the data we want to remove)
  • Set the retention policy to a smaller time window, so that the log events older than X time will be deleted. This also removes more than we'd like to remove.

Does anyone have experience with removing specific log records?

Titulum
  • 9,928
  • 11
  • 41
  • 79
  • Are you looking for someone to tell you there's a secret undocumented API call? – kdgregory Jun 25 '19 at 12:46
  • Hah, that would be nice too. But no, I'm wondering if someone already found a work-around (exporting logs, removing what needs to be removed and then reuploading them as new logs, etc. etc.) – Titulum Jun 25 '19 at 13:22
  • 1
    Unfortunately, as of today, you cannot delete a single log event from CloudWatch log stream, the alternative will be using Lambda functions: set a Lambda function trigger, filter all logs, then write the remaining logs to a new log group/stream, then delete the original log stream. This document uses the CloudTrail event as an example: https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/SubscriptionFilters.html#LambdaFunctionExample – ronald8192 Jun 25 '19 at 17:11
  • You'll get better (or any) answers if you explain _why_ you want to delete log records. Copying will work, but it's not very optimal, and it works best if you do it on an ongoing basis (versus "oops, we wrote a password into the log, let's delete those rows"). If your use case is the latter, you'd be better off using Elasticsearch as your log repository, because it will let you select and individually delete documents.(and you might be interested in [this article](https://www.kdgregory.com/index.php?page=aws.loggingPipeline) that I wrote on how to set up an ES-based logging pipeline). – kdgregory Jun 26 '19 at 11:20

1 Answers1

2

The comment of user ronald8192 is indeed the reality:

Unfortunately, as of today, you cannot delete a single log event from CloudWatch log stream, the alternative will be using Lambda functions: set a Lambda function trigger, filter all logs, then write the remaining logs to a new log group/stream, then delete the original log stream.

Titulum
  • 9,928
  • 11
  • 41
  • 79