2

I have an elasticsearch cluster managed by AWS Elasticsearch Service. I am setting up a hot-warm policy to move indices older than 8 days into ultrawarm storage. I want to reduce the replicas to 0 as the ultrawarm nodes are backed by S3 and say that you do not need replicas when using ultrawarm nodes.

I've set up an index state management policy as this:



{
    "policy_id": "hot_warm_workflow",
    "description": "hot-warm workflow - Indices stay hot storage for 8 days and then move to warm storage",
    "last_updated_time": 1632847870167,
    "schema_version": 1,
    "error_notification": null,
    "default_state": "hot",
    "states": [
        {
            "name": "hot",
            "actions": [],
            "transitions": [
                {
                    "state_name": "warm",
                    "conditions": {
                        "min_index_age": "8d"
                    }
                }
            ]
        },
        {
            "name": "warm",
            "actions": [
                {
                    "retry": {
                        "count": 5,
                        "backoff": "exponential",
                        "delay": "1h"
                    },
                    "warm_migration": {}
                },
                {
                    "retry": {
                        "count": 5,
                        "backoff": "exponential",
                        "delay": "1h"
                    },
                    "replica_count": {
                        "number_of_replicas": 0
                    }
                }
            ],
            "transitions": []
        }
    ],
    "ism_template": null
}

The index successfully moves to warm storage but when the next action is taken to reduce the replicas to 0, the migration errors. The error message is as follows:


{
    "cause": "index [log-1-9-2021] blocked by: [TOO_MANY_REQUESTS/12/index read-only / allow delete (api)];",
    "message": "Failed to set number_of_replicas to 0 [index=log-1-9-2021]"
}

I don't think I've hit a rate limit as I only tried to do this on one index. Is there an issue with changing replica size on ultrawarm nodes? Do I need to reduce the replicas on the hot nodes and then migrate to warm?

Dónal Tuohy
  • 31
  • 2

1 Answers1

1

Figured it out:

As ultrawarm storage is read only, I believe it errored trying to reduce the replicas in the warm nodes. I changed the order of the warm state actions so the replicas were reduced and then migrated to warm storage.

The migration then worked for me.

Dónal Tuohy
  • 31
  • 2