0

What do I have:

  • Elasticsearch (7.7.0) cluster (amazon/opendistro-for-elasticsearch:1.8.0 Docker image)
    • one master node
    • one coordinating node
    • two data nodes with node.attr.data=hot
    • one data node with node.attr.data=warm

What do I want: prevent shard allocation and relocation from hot data nodes to warm (and cold in future) data nodes.

What have I tried:

I've put "index.routing.allocation.require.data": "hot" for all index templates, so newly created indices won't be allocated to any but hot data nodes. This works fine.

Anyway, I can't restrict shards relocation from hot nodes to warm. At the moment I'm using "cluster.routing.allocation.exclude._ip" : "warm data node ip" to prevent relocation from hot data nodes to the warm data node. But will I be able to use ILM with this filter?

I've also tried to

PUT _cluster/settings
{
  "transient": {
    "cluster.routing.allocation.awareness.attributes": ["data"]
  }
}

PUT _cluster/settings
{
  "transient": {
    "cluster.routing.allocation.awareness.force.data.values": ["hot"]
  }
}

and then remove the "cluster.routing.allocation.exclude._ip" filter. Shards were relocating from hot data nodes to the warm data node anyway. What am I missing?

Tarasovych
  • 2,228
  • 3
  • 19
  • 51

2 Answers2

0

I think you miss the allocation awareness in your ILM Policy. Your warm phase definition should have

"allocate": {
   "require": {
        "data": "warm"
    }
}

Remove this from your definition and it should resolve your issue.

The best article to fully inderstand the ILM for me is https://www.elastic.co/blog/implementing-hot-warm-cold-in-elasticsearch-with-index-lifecycle-management You will find a full example in "Optimizing your ILM policy for hot-warm-cold"

Jaycreation
  • 2,029
  • 1
  • 15
  • 30
  • But ILM policy will not affect shard relocation. I don't have ILM yet, and shards relocate from hot to warm anyway – Tarasovych Oct 29 '20 at 08:54
  • Let me explain a bit more. I have a running cluster. I want to add new node(s) - warm and cold. I don't want to allow cluster to rebalance shards from hot nodes to warm and cold. Only the ILM I'll add in future might be able to move indices, shards though, to the warm and cold nodes – Tarasovych Oct 29 '20 at 08:55
  • It's more clear. Everything looks find to me. It should already work for you with index.routing.allocation.require.data. Or with cluster setting Did you execute https://www.elastic.co/guide/en/elasticsearch/reference/current/cat-nodeattrs.html to verify that your configuration are ok ? – Jaycreation Oct 29 '20 at 09:19
  • Yes, I checked `GET /_cat/nodeattrs` and my nodes have correct `data` attributes – Tarasovych Oct 29 '20 at 10:36
  • Should I assign `"index.routing.allocation.require.data": "hot"` for my old indices? – Tarasovych Oct 29 '20 at 10:39
  • Oh yes, I miss this point Or old Shards / Replicas can be allocated to your new nodes – Jaycreation Oct 29 '20 at 10:52
0

I had to update settings for my old incides:

PUT my-index/_settings
{
  "index.routing.allocation.require.data": "hot"
}

Indices which are under certain index template won't be updated if the index template is updated.

Tarasovych
  • 2,228
  • 3
  • 19
  • 51
  • I am facing the same issue currently. I have added warm node of 8TB, but I am unable to assign the old index to this warm node. When I try to run the command to move one of the index to old , it gives me the error "resource_already_exists_exception", ```curl -XPUT localhost:9200/index_name/_settings { "settings": { "index.routing.allocation.require.box_type": "warm" } }``` – user11549576 Jan 20 '21 at 06:38