update elastic-search document with the same ID

Question

everyone. I'm new in elk and I have a question about logstash. I have some services and each one has 4 or 6 logs; it means a doc in elastic may has 4 or 6 logs. I want to read these logs and if they have the same id, put them in one elastic doc. I must specify that all of the logs have a unique "id" and each request and every log that refers to that request has the same id. each log has a specific type. I want to put together every log that has the same id and type; like this:

    {
      "_id":"123",
      "Type1":{},
      "Type2":[{},{}],
      "Type3":[{},{}],
      "Type4":{}
    }

Every log for the same requset: Some of them must be in the same group. because their type are the same. look example above. Type2 is Json Array and has 2 jsons. I want to use logstash to read every log and have them classified. Imagine that our doc is like bellow JSON at the moment:

    {
      "_id": "123",
      "Type1":{},
      "Type2":[{},{}],
      "Type3":{}
    }

now a new log arrives, with id 123 and it's type is Type4. The doc must update like this:

    {
      "_id": "123",
      "Type1":{},
      "Type2":[{},{}],
      "Type3":{},
      "Type4":{}
    }

again, I have new log with id, 123 and type, Type3. the doc update like this:

    {
      "_id": "123",
      "Type1":{},
      "Type2":[{},{}],
      "Type3":[{},{}],
      "Type4":{}
    }

I tried with script, but I didn't succeed. :

    {
      "id": 1,
        "Type2": {}
    }

The script is:

input {
    stdin {
        codec => json_lines
    }
}
output {
    elasticsearch {
      hosts => ["XXX.XXX.XXX.XXX:9200"]
      index => "ss"
      document_id => "%{requestId}"
      action => "update" # update if possible instead of overwriting 
      document_type => "_doc"
      script_lang => "painless"
      scripted_upsert => true
      script_type => "inline"
      script => 'if (ctx._source.Type3 == null) { ctx._source.Type3 = new ArrayList() } if(!ctx._source.Type3.contains("%{Type3}")) { ctx._source.Type3.add("%{Type3}")}'
    }
}

now my problem is this script format just one type; if it works for multiple types, what would it look like? there is one more problem. I have some logs that they don't have an id, or they have an id, but don't have a type. I want to have these logs in the elastic, what should I do?

score 0 · Answer 1 · answered Jul 29 '19 at 08:51

You can have a look on aggregate filter plugin for logstash. Or as you mentioned if some of the logs don't have an id, then you can use fingerprint filter plugin to create an id, which you can use to update document in elasticsearch. E.g:

input {
    stdin {
        codec => json_lines
    }
}
filter {
  fingerprint {
     source => "message"
     target => "[@metadata][id]"
     method => "MURMUR3"
  }
}
output {
   elasticsearch {
      hosts => ["XXX.XXX.XXX.XXX:9200"]
      index => "ss"
      document_id => "%{[@metadata][id]}"
      action => "update" # update if possible instead of overwriting
    }
}

update elastic-search document with the same ID

1 Answers1