I have an application that receives logs remotely from IoT devices. Those logs have timestamps from when they really happened - I process and log these logs and therefore have a specific time field original-log-time
in my JSON logs.
So every one of my log files has the known @timestamp
field of when this line was written to the file and an original-log-time
field that contains the original log time.
Now I would like to only forward logs to elastic search if the original log time is less than 30 days in the past. The reason for this is, that logs from each day get their own index and only the last 30 indices are warm - the others are actually closed and I don't want them to be reopened.
What I am trying:
if [original_log_time].compareTo(ctx._source.time.plusDays(30)) <= 0 {
elasticsearch {
hosts => ["${ELASTICSEARCH_HOSTS:?ELASTICSEARCH_HOSTS environment variable is required}"]
index => "my-logs-" + original_log_time
user => "${ELASTICSEARCH_USERNAME:}"
password => "${ELASTICSEARCH_PASSWORD:}"
}
}
But this leads to the following error
Expected one of [, #, in, not , ==, !=, <=, >=, <, >, =~, !~, and, or, xor, nand, { at line 28, column 24*(byte 512) after output if [log_index_date]"
I read about Logstash ignore_older
but it looks like I can't specify which timestamp it should take into account for that check? Any smarter solution?
UPDATE
As I got some errors that there is neither a compareTo
nor a plusDays
in logstash I tried another approach I read here: https://discuss.elastic.co/t/adding-1-day-to-the-date/129168
which was a filter
filter {
date {
match => [ "log_index_date", "dd.MM.yyyy" ]
target => "log_index_plus_thirty"
}
ruby {
code => 'event.set("log_index_plus_thirty", LogStash::Timestamp.new(event.get("log_index_plus_thirty")+86400*30))'
}
}
with the following if condition:
if [@timestamp] <= [log_index_plus_thirty] {
elasticsearch {
hosts => ["${ELASTICSEARCH_HOSTS:?ELASTICSEARCH_HOSTS environment variable is required}"]
index => "device-logs-" + log_index_date
user => "${ELASTICSEARCH_USERNAME:}"
password => "${ELASTICSEARCH_PASSWORD:}"
}
}
but this complains at the _plus_thirty part of the if condition as if that variable would not exist.
Also log_index_date
is an optional field so not sure if that leads to a problem as well?