Logstash -> Elasticsearch : update document @timestamp if newer, discard if older

Question

Using the elasticsearch output in logstash, how can i update only the @timestamp for a log message if newer?

I don't want to reindex the whole document, nor have the same log message indexed twice.

Also, if the @timestamp is older, it must not update/replace the current version.

Currently, i'm doing this:

filter {
    if ("cloned" in [tags]) {
        fingerprint {
            add_tag => [ "lastlogin" ]
            key     => "lastlogin"
            method  => "SHA1"
        }
    }
}

output {
    if ("cloned" in [tags]) {
        elasticsearch {
            action              => "update"
            doc_as_upsert       => true
            document_id         => "%{fingerprint}"
            index               => "lastlogin-%{+YYYY.MM}"
            sniffing            => true
            template_overwrite  => true
        }
    }
}

---

It is similar to How to deduplicate documents while indexing into elasticsearch from logstash but i do not want to always update the message field; only if the @timestamp field is more recent.

Answer 1

You can't decide from Logstash level if a document needs to be updated or nothing should be done, this needs to be decided at Elasticsearch level. Which means that you need to experiment and test with _update API.

I suggest looking at https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-update.html#upserts. Meaning, if the document exists the script is executed (where you can check, if you want, the @timestamp), otherwise the content of upsert is considered as a new document.