1

I am trying to push EMR log files stored in S3 bucket to AWS Cloudwatch for further analysis.

My intended approach is to use SNS and trigger a Lambda function to push any new log files in the bucket to Cloudwatch. Is this a doable appraoch?

And is there a boto3 service for pushing logs into Cloudwatch?

Thank you all for your help.

Deepak Gupta
  • 387
  • 2
  • 17
DJASTI
  • 11
  • 1
  • 3

1 Answers1

0

One approach could be like that.

You can create lambda function and add s3 bucket as a source for that lambda function. so whenever EMR logs file written to S3 bucket. S3 bucket will invoke lambda function then write code in boto3 in lambda function to push the file to cloud watch. Below is example of boto3 to write code to upload log file to cloud watch logs.

https://boto3.amazonaws.com/v1/documentation/api/1.9.42/reference/services/logs.html#CloudWatchLogs.Client.put_log_events

Deepak Gupta
  • 387
  • 2
  • 17
  • You can also use another approach mentioned here https://stackoverflow.com/questions/59147344/sending-emr-logs-to-cloudwatch – Deepak Gupta Apr 08 '21 at 11:06
  • Thank you for the response. Will try the boto3 services to push logs to cloudwatch. – DJASTI Apr 08 '21 at 20:02
  • Yeah I have coded a Lamda function to push logs from S3 to Cloudwatch. The log streams are created during the function execution and are named after the EMR cluster. Will refine and share the code in next couple of days. – DJASTI Apr 15 '21 at 16:57