0

There are various ways to transfer logs from S3 to Opensearch:

What should be used in what situation? What is the cheapest? I would imagine that Kinesis and/or Event Handler method would be the quickest, but that might also put a big load on your cluster given that many calls would be made very often and there is not as much opportunity for bulk uploads. But with Glue you could for example do this operation say every 10 minutes, and then have a lot of bulk uploads, or schedule this operation in low usage periods for logs you do not need to be inserted into opensearch so quickly. I'd be interested to hear under what situation what strategy is used. I want to minimize the load on my cluster as I feel at the end of the day, putting a higher load on opensearch will cost the most.

Derrops
  • 7,651
  • 5
  • 30
  • 60
  • I don't have a say on this but just to understand the situation, who puts the logs on S3? Is it a one time operation or S3 will be updated with logs and these logs will be transfered to opensearch on a daily/hourly... basis? – Riz May 26 '22 at 09:08
  • 1
    Continuously updated, could be vpc flow logs, could be cloudtrail. Usually these are logs from many accounts in an AWS org – Derrops May 26 '22 at 10:11
  • 1
    Logs are in s3 for compliance reasons with an object lock. Some logs are streamed from cloud watch to s3. But they all need to go to s3. – Derrops May 26 '22 at 10:39

0 Answers0