My index is in 'yellow' state because one shard is unassigned. My three data nodes have plenty of disk space overall but they are partitioned. Each data node has three disk partitions called /data1, /data2 and /data3. Elasticsearch is only storing shards on /data1. Following this page from the manual, I modified elasticsearch.yml on the head node to include the following:
path.data: /data1/elasticsearch, /data2/elasticsearch_2, /data3/elasticsearch_3
And then ran:
curl -XPOST "localhost:9200/_cluster/reroute?retry_failed"
To try and re-allocate the shards. However, I am getting the warning the node is above the low watermark cluster setting [cluster.routing.allocation.disk.watermark.low=85% from to of the nodes and the shard cannot be allocated to the same node on which a copy of the shard already exists from the third data node.
There is a second, much smaller index also on this cluster, which in green status.
My question is, how can I get Elasticsearch to store shards on the additional partitions of the data nodes when the first partition is full? Each partition by itself is large enough to store one shard but I think because I have data for multiple indices in one partition it is triggering the warning and preventing the distribution of the replica shards.
