1

I want to import aws cloudtrail eventTime through logstash. Works well but fails to get eventTime.

my logstash.conf

input {
  s3 {
    bucket => "xxxxx"
    prefix => "xxxxx"
    sincedb_path => "/etc/logstash/sincedb/cloudtrail"
    temporary_directory => "/etc/logstash/tmp"
    region => "xxxxx"
    type => "cloudtrail"
    codec => "cloudtrail"
  }
}

filter {
  if [type] == "cloudtrail" {
    mutate {
      gsub => [ "eventSource", "\.amazonaws\.com$", "" ]
    }

    if [eventSource] == "elasticloadbalancing" and [eventName] == "describeInstanceHealth" and [userIdentity.userName] == "secret_username" {
      drop {}
    }
  }

  date {
      match => ["eventTime", "ISO8601"]
  }
}

In Kibana, other tables can be checked, but eventTime cannot be found.

loanshark
  • 105
  • 2
  • 8
  • Was the index pattern already in place when yoou changed stuff? did you refresh it? have you tried without the date filter? – YouryDW Apr 28 '21 at 09:46
  • 1
    By default, the `date` plugin will populate the `@timestamp` field with the parsed date. Does the `@timestamp` field contain the ingest time, or the parsed `eventTime` value? If you want to store it somewhere else, use the [target](https://www.elastic.co/guide/en/logstash/current/plugins-filters-date.html#plugins-filters-date-target) setting. – tomr Apr 29 '21 at 02:42
  • I indeed forgot that the date filter targetted the @timestamp event, it is indeed needed to target the field itself if you want to hold the value there – YouryDW Apr 29 '21 at 11:21

1 Answers1

0

The cloudtrail codec removes eventTime from the event and assigns this to the events @timestamp. See: https://github.com/logstash-plugins/logstash-codec-cloudtrail/blob/4486ce2f986bc4778562060bbaf9d6dfd99ab84e/lib/logstash/codecs/cloudtrail.rb#L22

McP
  • 1