I'm trying to parse a log group and filter some information I need and then upload it to s3. I did some research and I have the following two approaches:
By using subscription filter which means I can create another lambda function to subscribe the log group I'm interested --> the lambda will parse the data in a streaming fashion --> upload the stream data to s3 piece by piece --> maybe later create another job to combine multiple objects in s3 into one bigger object (My goal is to use some of the log data to do analysis)
By using the logs insight query. Hopefully using a lambda to periodically trigger the insight query and write into s3 -- in this way I do not need to worry about small uploads to s3. however, I do not know if there's a programmatic way of doing that. How can I use a lambda function to trigger logs insight? So far I only know how to do this in the CloudWatch console.
More ways to do that I don't know yet?
Please give me some suggestions. Thank you in advance!
