how to rename an index in a cluster?

Question

I need to rename several indexes in a cluster (their name must be changed, I cannot use aliases).

I saw that there are no supported ways to do that, the closest I found is to rename the directory of the index, I tried this in a cluster.

The cluster has 3 machines A, B and C and the shards are replicated on each of them. I shut down elasticsearch on A, renamed /var/lib/elasticsearch/security/nodes/0/indices/oldindexname to /var/lib/elasticsearch/security/nodes/0/indices/newindexname and restarted A.

The state of the cluster was yellow and elasticsearch was doing some magic to restore a correct state. After some time I ended up with

oldindexname being available and fully replicated (recovered from B and C I guess)
newindexname being available (I can search it) but the head plugin shows that its shards are in an "Unassigned" state and that they are grayed out (not replicated)

During the recovery security.log showed the following message:

[2015-02-20 11:02:33,461][INFO ][gateway.local.state.meta ] [A.example.com] dangled index directory name is [newindexname], state name is [oldindexname], renaming to directory name

While newindexname is searchable, it is certainly not in a normal state.

I rolled back to the previous state by deleting newindexname. The cluster is back to green without any "Unassigned" entries.

Given that, how can I rename oldindexname to newindexname in a cluster?

Note: the ultimate solution I have in mind is to scroll-copy oldindex into newindex and delete oldindex afterwards. This is going to take time so if there is a more direct solution it would be great.

score 206 · Answer 1 · edited Feb 01 '17 at 19:37

206

You can use REINDEX to do that.

Reindex does not attempt to set up the destination index. It does not copy the settings of the source index. You should set up the destination index prior to running a _reindex action, including setting up mappings, shard counts, replicas, etc.

First copy the index to a new name

POST /_reindex
{
  "source": {
    "index": "twitter"
  },
  "dest": {
    "index": "new_twitter"
  }
}

Now delete the Index

DELETE /twitter

edited Feb 01 '17 at 19:37

phlegx

2,618
3
35
39

answered Jun 28 '16 at 13:49

reto

16,189
7
53
67

3

Does this work if the mapping has `_source: {enabled: false}`? – Harald Jul 05 '16 at 13:35
3

@Harald No, `_reindex` uses `_source` as the original document data. – Agop Sep 19 '16 at 21:54
9

This won't copy the mapping of `twitter` to `new_twitter` as far as I know. – Nick Dec 08 '16 at 20:55
Note that `REINDEX` is considered experimental in Elastic 5.1 – mthmulders Dec 09 '16 at 10:28
@mthmulders Are you sure it's considered experimental in 5.1? I see a note saying that it's experimental in 2.4, but nothing in any of the 5.x versions: [2.4](https://www.elastic.co/guide/en/elasticsearch/reference/2.4/docs-reindex.html) vs. [5.1](https://www.elastic.co/guide/en/elasticsearch/reference/5.1/docs-reindex.html) – M. Justin May 30 '17 at 18:30
@M.Justin Well, I'm pretty sure it had the "Experimental" mark when I posted that remark. But indeed, it's not there (anymore). So I suppose it can be used without precautions. Thanks for the check! – mthmulders May 31 '17 at 05:52
6

I agree on the solution of _reindex, but the question should be changed. Reindexing is not simply a renaming. It can even change how data is indexed. – lucabelluccini Jun 26 '17 at 09:06
1

Reindexing can take a long time, so I highly recommend setting timeouts appropriately, as per the documentation: https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-reindex.html – cbp Sep 22 '17 at 01:30
I'd also point out that `reindex` doesn't carry the previous index configuration, so you should properly set the new index with the old configuration before performing the operation (you can get that by making a request to `:9200//_settings,_mappings`. I followed the example of this answer and wrote about my experience at https://ops.tips/blog/change-name-of-index-elasticsearch (it's just me going through what has been answered here). – Ciro Costa Dec 02 '17 at 11:21
It gave me an 'index_not_found_exception' error when I did DELETE /twitter. – raga Sep 03 '18 at 23:03

score 110 · Accepted Answer · answered Dec 18 '19 at 18:00

Starting with ElasticSearch 7.4, the best method to rename an index is to copy the index using the newly introduced Clone Index API, then to delete the original index using the Delete Index API.

The main advantage of the Clone Index API over the use of the Snapshot API or the Reindex API for the same purpose is speed, since the Clone Index API hardlinks segments from the source index to the target index, without reprocessing any of its content (on filesystems that support hardlinks, obviously; otherwise, files are copied at the file system level, which is still much more efficient that the alternatives). Clone Index also guarantee that the target index is identical in every point to the source index (that is, there is no need to manually copy settings and mappings, contrary to the Reindex approach), and doesn't require a local snapshot directory be configured.

Side note: even though this procedure is much faster than previous solutions, it still implies down time. There are real use cases that justify renaming indices (for example, as a step in a split, shrink or backup workflow), but renaming indices should not be part of day-to-day operations. If your workflow requires frequent index renaming, then you should consider using Indices Aliases instead.

Here is an example of a complete sequence of operations to rename index source_index to target_index. It can be executed using some ElasticSearch specific console, such as the one integrated in Kibana. See this gist for an alternative version of this example, using curl instead of an Elastic Search console.

# Make sure the source index is actually open
POST /source_index/_open

# Put the source index in read-only mode
PUT /source_index/_settings
{
  "settings": {
    "index.blocks.write": "true"
  }
}

# Clone the source index to the target name, and set the target to read-write mode
POST /source_index/_clone/target_index
{
  "settings": {
    "index.blocks.write": null 
  }
}

# Wait until the target index is green;
# it should usually be fast (assuming your filesystem supports hard links).
GET /_cluster/health/target_index?wait_for_status=green&timeout=30s

# If it appears to be taking too much time for the cluster to get back to green,
# the following requests might help you identify eventual outstanding issues (if any)
GET /_cat/indices/target_index
GET /_cat/recovery/target_index
GET /_cluster/allocation/explain

# Delete the source index
DELETE /source_index

In "Put the source index in read-only mode" you say `"index.blocks.write": "true"` which reads a bit like there should be `false` instead.. is that a typo or can you explain why `true` is correct here? — leberknecht, Oct 14 '21 at 07:39
side note: for me it was `PUT /settings {"index": {"blocks": {"write": false}}}` ..with the leading `"settings"` key it didnt gave an error, but the index was still read-only afterwards oO — leberknecht, Oct 26 '21 at 14:23
This is really helpful specially step by steps on performing cloning. Almost similar steps applies to reindex. — khichar.anil, Sep 06 '22 at 14:27
I would like to note that cloning seems to re-set `index.number_of_replicas` to 1, so I also had to pass `index.number_of_replicas` with value I had before as POST payload. — Mitar, Oct 29 '22 at 09:05

score 67 · Answer 3 · answered Feb 24 '16 at 05:00

67

For renaming your index you can use Elasticsearch Snapshot module.

First you have to take snapshot of your index.while restoring it you can rename your index.

    POST /_snapshot/my_backup/snapshot_1/_restore
    {
     "indices": "jal",
     "ignore_unavailable": "true",
     "include_global_state": false,
     "rename_pattern": "jal",
     "rename_replacement": "jal1"
     }

rename_replacement :-New indexname in which you want backup your data.

answered Feb 24 '16 at 05:00

krishna kumar

1,190
12
14

1

Much better than the accepted solution! This is actually a binary copy of the index, so no risk of losing anything and doesn't require `_source` to be enabled in the index. I have renamed some multi-TB indices this way without any issues. – Jacket Mar 18 '18 at 07:39
1

does it retain the mapping? – Amogh Mishra Mar 08 '19 at 12:09
@krishnakumar What is `my_backup` and `snapshot_1` here? I suppose, the current index name is `jal` and new index name is `jal1`. – Vivek Vardhan Aug 08 '19 at 07:41
@AmoghMishra - No you need to put mapping again. – krishna kumar Jul 16 '20 at 05:22

Leo · Answer 4 · 2017-05-19T11:34:54.233

10

If you can't REINDEX a workaround is to use aliases. From the official documentation:

APIs in elasticsearch accept an index name when working against a specific index, and several indices when applicable. The index aliases API allow to alias an index with a name, with all APIs automatically converting the alias name to the actual index name. An alias can also be mapped to more than one index, and when specifying it, the alias will automatically expand to the aliases indices. An alias can also be associated with a filter that will automatically be applied when searching, and routing values. An alias cannot have the same name as an index.

Be aware that this solution does not work if you're using More Like This feature. https://github.com/elastic/elasticsearch/issues/16560

edited May 19 '17 at 11:34

answered Sep 19 '16 at 08:38

Leo

1,102
2
11
18

2

`I need to rename several indexes in a cluster (their name must be changed I cannot use aliases).` By @WoJ – Thales P May 18 '17 at 23:20
*Reason alias won't work*: Using `alias` _requires_ you to plan ahead and create the original index name `as an alias` to a real index. Then you can create a new alias name and re-use the old alias name for something else. *But* you lose access to old_data if you have only have a real_index, make an alias to it, delete the old real_index. The alias now points to nothing. – Jesse Chisholm Feb 12 '19 at 17:58
@JesseChrisholm I think one cannot have an "alias that points to nothing". Try removing real_index, you'll see that the alias "an alias to it" will also be removed. – mgaert Dec 11 '19 at 11:37

score 5 · Answer 5 · answered Jun 30 '15 at 19:59

5

As such there is no direct method to copy or rename index in ES (I did search extensively for my own project)

However a very easy option is to use a popular migration tool [Elastic-Exporter].

http://www.retailmenot.com/corp/eng/posts/2014/12/02/elasticsearch-cluster-migration/

[PS: this is not my blog, just stumbled upon and found it good]

Thereby you can copy index/type and then delete the old one.

answered Jun 30 '15 at 19:59

lazywiz

1,091
2
13
26

The link does not work anymore. Any other place we could find it or have this info? – elachell Jul 06 '17 at 15:47

score 5 · Answer 6 · edited Sep 18 '18 at 02:21

5

Another different way to achieve the renaming or change the mappings for an index is to reindex using logstash. Here is a sample of the logstash 2.1 configuration:

input {
  elasticsearch {
   hosts => ["es01.example.com", "es02.example.com"]
   index => "old-index-name"
   size => 500
   scroll => "5m"
  }
}
filter {

 mutate {
  remove_field => [ "@version" ]
 }

 date {
   "match" => [ "custom_timestamp", "MM/dd/YYYY HH:mm:ss" ]
   target => "@timestamp"
 }

}
output {
 elasticsearch {
   hosts => ["es01.example.com", "es02.example.com" ]
   manage_template => false
   index => "new-index-name"
 }
}

edited Sep 18 '18 at 02:21

Drubio

1,157
3
13
30

answered Dec 16 '15 at 17:20

Gabriel Rosca

83
1
1

4

So you're saying the best way to reindex an Elasticsearch index is to install Logstash, and then use that to reindex? Seems a bit overkill, especially if you don't actually want/use Logstash... – M. Justin May 30 '17 at 18:27
The only problem in the answer is the "The best" part. I would say "Another way'. Other than that, it is a good answer. – Robert Sep 04 '18 at 13:21

score 4 · Answer 7 · answered Jan 12 '16 at 13:10

4

As indicated in Elasticsearch reference for snapshot module,

The rename_pattern and rename_replacement options can be also used to rename index on restore using regular expression

answered Jan 12 '16 at 13:10

oldbam

2,397
1
16
24

score 0 · Answer 8 · answered Feb 01 '23 at 11:17

0

For those landing here based on OpenSearch, here's how you do it...

PUT /<source-index>/_block/write

DELETE <target-index> (if one already exists)

PUT /<source-index>/_clone/<target-index>

answered Feb 01 '23 at 11:17

Tirinoarim

624
7
14

score -11 · Answer 9 · answered Jul 21 '16 at 20:27

-11

Just in case someone still needs it. The successful, not official, way to rename indexes are:

Close indexes that need to be renamed
Rename indexes' folders in all data directories of master and data nodes.
Reopen old closed indexes (I use kofp plugin). Old indexes will be reopened but stay unassigned. New indexes will appear in closed state
Reopen new indexes
Delete old indexes

If you happen to get this error "dangled index directory name is", remove index folder in all master nodes (not data nodes), and restart one of the data nodes.

answered Jul 21 '16 at 20:27

Anh Le

19
3

2

Strongly discouraged by Elastic. Ensure to have backups if you do so. – lucabelluccini Jun 26 '17 at 09:04
I don't understand where the name of the index appears in the data directory. When I look in /var/lib/elasticsearch/nodes/0/indices/ the names of the directories are random-generated, like "1aS4RusHSYWLdt-Wx7NnBw" (Elasticsearch version 5.6.3) – Johan Boulé Nov 04 '17 at 15:02
1

@JohanBoulé, this method is no longer valid since version 5 of Elasticsearch. – Anh Le Nov 07 '17 at 06:45
@lucabelluccini, agreed. Better to just leave them as is and use aliases. – Anh Le Nov 07 '17 at 06:46

how to rename an index in a cluster?

9 Answers9

Linked