17

I have Exported the aws cloudwatch Log Data to Amazon S3 Using the AWS CLI. Followed kb: https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/S3ExportTasks.html.

Any idea how we can import the logs saved in s3 for analysing the data?

manu thankachan
  • 433
  • 3
  • 9
  • 19
  • 3
    And why exactly do you need to move your logs from S3 to CloudWatch? For analysis? What is wrong with Athena or Redshift Spectrum which can operate directly on data stored on S3? Or why don't you move your data to Redshift or ES or RDS or any other DB suitable for your needs? Honestly, I don't see any value in moving data from S3 to CloudWatch, this is usually done the other way around, mostly because CloudWatch is not suitable for complex analysis (even though we have Insights now). – Matus Dubrava Jun 18 '19 at 20:54
  • Actually, I am creating a DR(Disaster recovery),so I need to move the cloudwatch logs which are generated in newly cluster to original cluster when it is up. – manu thankachan Jun 20 '19 at 08:21
  • 1
    @manuthankachan Not so sure you need the answer still, but it seems using S3 event notification to send its logs to CW by Lambda function could be an option for you. https://aws.amazon.com/blogs/mt/sending-cloudfront-standard-logs-to-cloudwatch-logs-for-analysis/ – Haeyoon J. Aug 29 '22 at 12:46

2 Answers2

2

Well, for data analysing I would recommend using Athena,

Regarding your comments a quick search was not able to provide an answer on how to import logs.

Logs are attached to resources that you spin up, so it sounds kind of impossible to attach logs from a newly generated cluster to the old cluster when it is up, since they are two separate resources with different unique id's etc.

Logs are stored in cloudwatch don't have an expiry date even if the resource is deleted. So you would still be able to check the logs of the newly generated cluster when it is no longer online (destroyed) export the logs to s3 and analyse them using Athena.

kldnz
  • 76
  • 2
1

The only way to do this is to use a script of some form to pull the files from S3, and put them into CloudWatch Logs. Lambda would be a good candidate, using an S3 trigger if you wanted to automate it.

But, the better way would be to archive the logs. In a DR scenario as long as your log entries are safe, you're ok. Trying to replay those logs into a new cluster sounds like a bad idea to me.

andybkay
  • 41
  • 3