Is there a way to remove old data from azure search index

Question

I have created a search service on azure. Now there is some data change on the db and some values are removed. When i reset and run the indexer after completion the search explorer is still showing old values which are not required. For example:this the response before change for cycle:202007

        "ID":"ABC_202007",
        "EntityID": "ABC",
        "PartnerOneName": "ABC",
        "RiskScore": 47.04,
        "POCount": 2,
        "POAmount": 591925.54,
        "Category": "ABC",
        "Cycle": "202007",
        "Country": "ABC",
        "Area": "ABC",
        "POAmountScore": 100,
        "POCountScore": 96,

now, after some time the cycle is changed to 202019 and when i reset and run the indexer the response is :

{
            "ID":"ABC_202007"
            "EntityID": "ABC",
            "PartnerOneName": "ABC",
            "RiskScore": 47.04,
            "POCount": 2,
            "POAmount": 591925.54,
            "Category": "ABC",
            "Cycle": "202007",
            "Country": "ABC",
            "Area": "ABC",
            "POAmountScore": 100,
            "POCountScore": 96,
},

{
              "ID":"ABC_202019"
            "EntityID": "ABC",
            "PartnerOneName": "ABC",
            "RiskScore": 47.04,
            "POCount": 2,
            "POAmount": 591925.54,
            "Category": "ABC",
            "Cycle": "202019",
            "Country": "ABC",
            "Area": "ABC",
            "POAmountScore": 100,
            "POCountScore": 96
}

I want only the second result set. Please help.

to update document instead of inserting new one - our index request should contain same _id. I suppose now you are using autogeneration for ids, but maybe you can specify your EntityID as ES document id — Danila Zharenkov, Oct 15 '20 at 07:18
Yes, that could be a approach but in my case the id is the concat of EntityID and Cycle . I have poste the edited reponse. Can you tell me if we can empty the index ? — Shrajan Tiwari, Oct 15 '20 at 07:29
you can remove ALL docs from index using delete_by_query request. See here https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-delete-by-query.html or, you can write your own syncronization. Like when item (cycle field) in DB changed - remove old document by old id and index new one — Danila Zharenkov, Oct 15 '20 at 07:50

score 1 · Answer 1 · answered Oct 15 '20 at 07:51

Unfortunately, you can only delete single documents by id, which is described here. Iterating over mutliple documents and deleting them one by one is currently the only way. Thanks Microsoft!

Deleting mutliple documents is a long wanted feature, but nothing happened so far. Please vote for the suggestion, if you also consider this to be essential.

In your case it would make sense to make EntityID as id, so the documents get overwritten. Or you just create queries that filter out the outdated versions... who cares about storage space.

Is there a way to remove old data from azure search index

1 Answers1