Getting AWS logs from S3 on kibana using EFK stack

Question

I have an EFK stack running on EKS. How can I collect logs from an S3 bucket and see them through Kibana?

Answer 1

you can use this plugin to add a new source for the s3 https://github.com/tomohisaota/fluent-plugin-forward-aws

first, you need to add the source configs

<source>
  type forward_aws
  aws_access_key_id     XXXXXXXXXXXXXXXXXXXX
  aws_secret_access_key XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

  aws_s3_endpoint       s3-ap-northeast-1.amazonaws.com
  aws_s3_bucketname     XXXXXXXXXXXXXXXXXXXX

  aws_sqs_endpoint      sqs.ap-northeast-1.amazonaws.com
  aws_sqs_queue_url     https://sqs.ap-northeast-1.amazonaws.com/XXXXXXXXXXXXXXXXXXXX
</source>

then you need to parse them with a match and forward them to elasticsearch