3

I have an application that receives logs remotely from IoT devices. Those logs have timestamps from when they really happened - I process and log these logs and therefore have a specific time field original-log-time in my JSON logs.

So every one of my log files has the known @timestamp field of when this line was written to the file and an original-log-time field that contains the original log time.

Now I would like to only forward logs to elastic search if the original log time is less than 30 days in the past. The reason for this is, that logs from each day get their own index and only the last 30 indices are warm - the others are actually closed and I don't want them to be reopened.

What I am trying:

if [original_log_time].compareTo(ctx._source.time.plusDays(30)) <= 0 {
  elasticsearch {
     hosts => ["${ELASTICSEARCH_HOSTS:?ELASTICSEARCH_HOSTS environment variable is required}"]
     index => "my-logs-" + original_log_time
     user => "${ELASTICSEARCH_USERNAME:}"
     password => "${ELASTICSEARCH_PASSWORD:}"
  }
}

But this leads to the following error

Expected one of [, #, in, not , ==, !=, <=, >=, <, >, =~, !~, and, or, xor, nand, { at line 28, column 24*(byte 512) after output if [log_index_date]"

I read about Logstash ignore_older but it looks like I can't specify which timestamp it should take into account for that check? Any smarter solution?

UPDATE

As I got some errors that there is neither a compareTonor a plusDaysin logstash I tried another approach I read here: https://discuss.elastic.co/t/adding-1-day-to-the-date/129168

which was a filter

filter {
  date {
    match => [ "log_index_date", "dd.MM.yyyy" ]
    target => "log_index_plus_thirty"
  }

  ruby {
    code => 'event.set("log_index_plus_thirty", LogStash::Timestamp.new(event.get("log_index_plus_thirty")+86400*30))'
  }
}

with the following if condition:

   if [@timestamp] <= [log_index_plus_thirty] {
      elasticsearch {
        hosts => ["${ELASTICSEARCH_HOSTS:?ELASTICSEARCH_HOSTS environment variable is required}"]
        index => "device-logs-" + log_index_date
        user => "${ELASTICSEARCH_USERNAME:}"
        password => "${ELASTICSEARCH_PASSWORD:}"
      }
    }

but this complains at the _plus_thirty part of the if condition as if that variable would not exist.

Also log_index_date is an optional field so not sure if that leads to a problem as well?

peach
  • 657
  • 5
  • 26
  • It looks like the error is just coming from your `if` statement. Can you try `if [original_log_time] < ctx._source.time.plusDays(30) ` and see if this gets rid of the error? I don't believe there is a `compareTo()` in logstash. – SevvyP Jul 22 '20 at 13:50
  • yeah it seems there is also no plusDays in logstash, I tried something else and will update my question – peach Jul 22 '20 at 14:37

0 Answers0