Elasticsearch reindex items in existing type mapping

Question

I added a new property to a type mapping and I need to reindex all existing items of that type in order to use the new property.

Which API should I use to do this?

Answer 1

If you are adding a new field, which it never existed before in your index you don't need to reindex, you only have to add the new field by using PUT Mapping API http://nocf-www.elastic.co/guide/en/elasticsearch/reference/current/indices-put-mapping.html

Documents you created before updating the mapping with the new field will not contain this new field, so searches or aggregations won't take into account this field, it will work as a missing field.

If you need that this new field is considered in searches on old documents using the default value of the type of the new field, then you need to reindex. For instance, if your new field is type integer and you explicitly need that this field be included in the old documents with zero value (default value) because you want to count how many documents have this new field = 0 then you need reindex, but most of the use cases we can consider missing fields as default value, so no need to reindex.

There is no way in ElastiSearch (ES) to add a new field in the mapping and old indexes be updated automatically, even with the default value for that new index due to the nature of how ES store data. ES uses immutable segments to store the indexes so when you update a document, ES doesn't update physically the fields which changed, but it creates a new copy of the old document updated with the new data and mark as deleted the old one, so even when you update a simple field in a document, you get a new version of the document and the old one is marked as deleted

Answer 2

You need to execute those commands. Replace my_index by the name of your index.

# First of all: enable blocks write to enable clonage
PUT /my_index/_settings
{
  "settings": {
    "index.blocks.write": true
  }
}

# clone index into a temporary index
POST /my_index/_clone/my_index-000001  

# Disable blocks write
PUT /my_index/_settings
{
  "settings": {
    "index.blocks.write": false
  }
}

# Copy back all documents in the original index to force their reindexetion
POST /_reindex?wait_for_completion=false
{
  "source": {
    "index": "my_index-000001"
  },
  "dest": {
    "index": "my_index"
  }
}

# Copy the task id from the previous result and check the progression:
GET /_tasks/K1IOaNo8R26gRwc55yXJLQ:1165945865

# Finaly delete the temporary index
DELETE my_index-000001

What it does:

• clone your index into a temporary index
• then reindex all the documents from temporary to the original index. This will overwrite existing documents and regenerate the missing fields.

Caveats: This method assumes that nothing insert new data in the index. New data would be overwrite during the reindex process.

Answer 3

Another option is to create an index alias in Elasticsearch that your code will reference. If you need to make mapping changes, you can do the following that will allow little to no downtime.

• create new index with updated ES mapping config
• use the reindex api to copy the data to this new index
• delete old index alias and recreate it with the same name.

Answer 4

You have to use the reindex api: first you have to create the new index, and then you can use the reidex api to transfer data from the source index into the new index.