0

I am trying to ingest cloudtrail logs to logstash to detect a certain event. I have installed logstash-codec-cloudtrail plugin and was able to ingest cloudtrail logs if i point to a specific folder. Currently i am ingesting it from

s3://cloudtrail/AWSLogs/123456789/CloudTrail/ap-southeast-1/2021/08/04/

Logstash Version: logstash 7.13.4

input {
    s3 {
        bucket => "com.abc.cloudtrail-nonprod"
        delete => false
        interval => 60 # seconds
        prefix => "AWSLogs/123456789/CloudTrail/ap-southeast-1/2021/08/04/"
        type => "cloudtrail"
        codec => "cloudtrail"
        region => "ap-southeast-1"
    } 
}

However, I wish to ingest cloudtrail logs from S3 bucket from a region and not specific to a date.

I was trying to do this but i do not know if this is the right way to do it or such feature has not been developed yet (based on my google search)

input {
    s3 {
        bucket => "com.abc.cloudtrail-nonprod"
        delete => false
        interval => 60 # seconds
        prefix => "AWSLogs/123456789/CloudTrail/ap-southeast-1/*/*/*/"
        type => "cloudtrail"
        codec => "cloudtrail"
        region => "ap-southeast-1"
    }
}

My end goal is to be able to read cloudtrail event from now (not the past) and create an audit log to watch who assumed admin role in AWS.

Tsu Wei Quan
  • 335
  • 1
  • 5
  • 19

1 Answers1

0

The prefix option on the s3 input explicitly does not support wildcards, but it is a prefix, so the trailing wildcard is implicit. Use

prefix => "AWSLogs/123456789/CloudTrail/ap-southeast-1/"
Badger
  • 3,943
  • 2
  • 6
  • 17
  • Thanks for the quick reply. After setting this prefix & when i did a stdout {} without any mutation, I do not see any output. I am not sure if this is working or it is processing/reading all the cloudtrail logs. [2021-08-04T02:51:01,228][INFO ][logstash.inputs.s3 ][cloudtrail][65f44a4daf14845169699aa81d06c5750e1e5779cc408a4c593a188361831040] Using default generated file for the sincedb {:filename=>"/var/lib/logstash/plugins/inputs/s3/sincedb_016786087050d06758f0b2ef4eb4d3bf"} – Tsu Wei Quan Aug 04 '21 at 02:56