0

Is there any better solution implement to get aws cloudtrail logs to kibana, here I am using ElasticSearch Service from AWS

user60679
  • 709
  • 14
  • 28
  • what do you mean by "better solution"? Do you have an input to elasticsearch already working? – tedder42 Nov 21 '15 at 01:00
  • I have written a python code which downloads the cloudtrail logs, and appends to a file. i am running the code in cron for every 5 mins – user60679 Nov 21 '15 at 04:42

2 Answers2

1

Heres the logstash input that I use with 1.4.2. It works well, though I suspect it is noisy (it requires a lot of S3 GET/HEAD/LIST requests).

input {
  s3 {
    bucket => "bucketname"
    delete => false
    interval => 60 # seconds
    prefix => "cloudtrail/"
    type => "cloudtrail"
    codec => "cloudtrail"
    credentials => "/etc/logstash/s3_credentials.ini"
    sincedb_path => "/opt/logstash_cloudtrail/sincedb"
  }
}

filter {
  if [type] == "cloudtrail" {
    mutate {
      gsub => [ "eventSource", "\.amazonaws\.com$", "" ]
      add_field => {
        "document_id" => "%{eventID}"
      }
    }
    if ! [ingest_time] {
      ruby {
        code => "event['ingest_time'] = Time.now.utc.strftime '%FT%TZ'"
      }
    }
    ruby {
      code => "event.cancel if (Time.now.to_f - event['@timestamp'].to_f) > (60 * 60 * 24 * 1)"
    }
    ruby { 
      code => "event['ingest_delay_hours'] = (Time.now.to_f - event['@timestamp'].to_f) / 3600" 
    }

    # drop events more than a day old, we're probably catching up very poorly
    if [ingest_delay_hours] > 24 {
      drop {}
    }

    # example of an event that is noisy and I don't care about
    if [eventSource] == "elasticloadbalancing" and [eventName] == "describeInstanceHealth" and [userIdentity.userName] == "deploy-s3" {
      drop {}
    }
  }
}

The credentials.ini format is explained on the s3 input page; it's just this:

AWS_ACCESS_KEY_ID=
AWS_SECRET_ACCESS_KEY=

I also have a search that sends results to our #chatops but I'm not posting that here.

tedder42
  • 23,519
  • 13
  • 86
  • 102
1

If you haven't tried it already, you can use cloudtrail and cloudwatch logs together. Then use cloudwatch logs to create a subscription to send the cloudtrail data to elasticsearch.

Once that is done you should be able to define a kibana index that starts with cwl* that is time based.

Cheers-

chadneal
  • 886
  • 7
  • 8