0

TL;DR; Map missing logs from LogStash in Kibana dashboard to their correct date and time

I have an Amazon ElasticSearch domain(which includes both ElasticSearch service and Kibana dashboard) configured. My source of logs is a Beanstalk environment. I have installed FileBeat inside that environment, which sends the logs to an EC2, which have configured with LogStash. The LogStash server will then send the logs to ES domain endpoint.

This happened for a while without an issue, but when I checked yesterday, I saw logs were not sent to the ES for like 4 days. I got it fixed and the current logs are being transferred alright. The missing logs are still stacked in the LogStash server.

I modified my logstash.conf file to include the missing log files, but they all appear as a one single bar in the current date in my Kibana graph. What I want to do is that, make sure each missing set of logs is shown in Kibana in their respective date and time.

Example date and time part:

2021-05-20 19:44:34.700+0000

The following is my logstash.conf configuration. (Please let me know if I should post my FileBeat config too).

input {
        beats {
            port => 5044
        }
}

filter {
  date {
    match => [ "logdate", "yyyy-MM-dd HH:mm:ss.SSS+Z" ]
  }
  mutate {
    split => { "message" => "|" }
    add_field =>  { "messageDate" => "%{[message][0]}" }
    add_field =>  { "messageLevel" => "%{[message][1]}" }
    add_field =>  { "messageContextName" => "%{[message][2]}" }
    add_field =>  { "messagePID" => "%{[message][3]}" }
    add_field =>  { "messageThread" => "%{[message][4]}" }
    add_field =>  { "messageLogger" => "%{[message][5]}" }
    add_field =>  { "messageMessage" => "%{[message][6]}" }
  }
}
output {
  amazon_es {
    hosts => ["hostname"]
    index => "dev-%{+YYYY.MM.dd}"
    region => "region"
    aws_access_key_id => "ackey_id"
    aws_secret_access_key => "ackey"
  }
}

enter image description here

Sandun
  • 395
  • 2
  • 10
  • 25

1 Answers1

1

Use a date filter to parse the date contained in the message. This will set [@timestamp] and then the event will appear in the right bucket in kibana.

Badger
  • 3,943
  • 2
  • 6
  • 17
  • Hi thanks, I tried the method mentioned in the given article and added a 'date' section inside 'filter' but I still cannot see the missing logs in their correct date. I updated my question with that new configuration. Did you suggest this method in order to work for future logs only? – Sandun Jun 02 '21 at 14:57
  • Changing the logstash configuration will not change any documents already indexed into elasticsearch, it will only affect documents that logstash indexes in the future. – Badger Jun 02 '21 at 15:47
  • So there's no way to set already missed logs in to their correct date? – Sandun Jun 03 '21 at 06:18
  • You would need to pass them through logstash again and re-index them. – Badger Jun 03 '21 at 15:35