Delete records not updating in elastic search river plugin

Question

In elastic search river , if i deleted a record in mysql , its still showing in index . I have enabled auto-commit also . How make mysql and elastic search in sync and also how to make delta-imports in elastic ?

   {
"type" : "jdbc",
"jdbc" : {
    "driver" : "com.mysql.jdbc.Driver",
    "url" : "jdbc:mysql://localhost:3306/testrivet",
    "user" : "root",
    "password" : "Gemini*123",
    "sql" : [
                        { 
                             "statement" : "select *,empid as _id from empdata"

                        }
       ],
    "strategy" : "simple",
 "schedule" : "0 0-59 0-23 ? * *",
 "autocommit" : true,
 "metrics": {enabled:true}
    },
    "index" : {
   "autocommit":true
    }
}

score 2 · Answer 1 · answered Aug 23 '15 at 04:13

Indeed, if a record is deleted from your database, there's no way your JDBC river will be able to retrieve it anymore in order to delete the corresponding record in ES.

An alternative is to "soft-delete" records from your database by setting a flag (i.e. a new boolean column). The flag would be true when the record is active and false when the record is deleted. That way when your import process runs, you'd get all records and based on that flag you know you have to delete the documents from Elasticsearch.

There are other ways but they involve adding another component to the mix, so if this would do the job I'd suggest doing like that.

Delete records not updating in elastic search river plugin

1 Answers1