Delete data or document from elastic search using logstash

Question

I am trying to delete elastic search data or document using logstash configuration but delete seems to be not working. I am using logstash 5.6.8 version Below is the logstash configuration file:

```input {
   jdbc {
    #db configuration
      '''
      statement => " select * from table "

  }
output {
    elasticsearch {
        action => "delete"
        hosts => "localhost"
        index => "myindex"
        document_type => "doctype"
        document_id => "%{id}"
    }
    stdout { codec => json_lines } 
}```

But the above configuration are deleting the id's present in my db table and not deleting the id's that are not present. when i sync from db to elastic search using logstash, i expect that deleted rows in db also synched and it should be consistent.

I also tried below configuration but getting some error:

```input {
   jdbc {
    #db configuration
      '''
      statement => " select * from table "

  }
output {
    elasticsearch {
        action => "delete"
        hosts => "localhost"
        index => "myindex"
        document_type => "doctype"
    }
    stdout { codec => json_lines } 
}```

Error in logstash console: "current_call"=>"[...]/vendor/bundle/jruby/1.9/gems/stud-0.0.23/lib/stud/interval.rb:89:in sleep'"}]}} [2019-12-27T16:30:16,087][WARN ][logstash.shutdownwatcher ] {"inflight_count"=>9, "stalling_thread_info"=>{"other"=>[{"thread_id"=>22, "name"=>"[main]>worker0", "current_call"=>"[...]/vendor/bundle/jruby/1.9/gems/stud-0.0.23/lib/stud/interval.rb:89:insleep'"}]}} [2019-12-27T16:30:18,623][ERROR][logstash.outputs.elasticsearch] Encountered a retryable error. Will Retry with exponential backoff {:code=>400, :url=>"http://localhost:9200/_bulk"} [2019-12-27T16:30:21,086][WARN ][logstash.shutdownwatcher ] {"inflight_count"=>9, "stalling_thread_info"=>{"other"=>[{"thread_id"=>22, "name"=>"[main]>worker0", "current_call"=>"[...]/vendor/bundle/jruby/1.9/gems/stud-0.0.23/lib/stud/interval.rb:89:in `sleep'"}]}}

Can someone tell me how to delete document and sync db data or how to handle deleted records in elastic search?

Delete data or document from elastic search using logstash

0 Answers0