0

I'm a complete newbie with AWS and I'm having issues with AWS Elasticsearch.

We have a large Magento 2 catalog with hundreds of thousands of products (although we only index product name and SKU data). Elasticsearch constantly reboots when indexing is done and kills indexing process.

We have:

Instance type: c5.large.elasticsearch

Number of nodes: 2

EBS volume size: 25GB

From what I understand, it runs into the Minimum Free storage space limit (18GB) and as soon as it hits 18GB per node it reboots the instance.

We have increased the EBS volume size to 30GB, but the Minimum Free Storage space limit also increased to 22GB.

What's the point of increasing the EBS volume size if minimum increases together with it?

Am I missing something? Why did the limit go up?

Here are some graphs with failure points:

Kibana health status

Total free storage space (FreeStorageSpace)

Master instance connection status (MasterReachableFromNode)

UPDATE:

here is the error I'm getting when indexing is done and instance resets:

MasterReachableFromNode-RED

Elasticsearch engine returned an error response. item id: 445689. Error type: "cluster_block_exception", reason "blocked by: [SERVICE_UNAVAILABLE/2/no master];".
item id: 445699. Error type: "cluster_block_exception", reason "blocked by: [SERVICE_UNAVAILABLE/2/no master];".
item id: 445709. Error type: "cluster_block_exception", reason "blocked by: [SERVICE_UNAVAILABLE/2/no master];".
item id: 445719. Error type: "cluster_block_exception", reason "blocked by: [SERVICE_UNAVAILABLE/2/no master];".
item id: 445727. Error type: "cluster_block_exception", reason "blocked by: [SERVICE_UNAVAILABLE/2/no master];".
item id: 445730. Error type: "cluster_block_exception", reason "blocked by: [SERVICE_UNAVAILABLE/2/no master];".
item id: 445732. Error type: "cluster_block_exception", reason "blocked by: [SERVICE_UNAVAILABLE/2/no master];".
item id: 445739. Error type: "cluster_block_exception", reason "blocked by: [SERVICE_UNAVAILABLE/2/no master];".
Kalvin Klien
  • 911
  • 1
  • 12
  • 32
  • 1
    Minimum free storage increasing means you have more available storage. It's how much free storage is available on the node with the least available. – jordanm Apr 21 '21 at 19:25
  • How come once the storage hits the Minimum limit, it resets the instance? Before the increase the Minimum limit was at 18. You can see this in the screenshots - as soon as it hit 18GB and it bricked the instance. So I'm very confused as to why the limit increased after the increase in EBS - is it going to do the same thing and brick the instance once minimum hits 22GB? – Kalvin Klien Apr 21 '21 at 19:34
  • 1
    Correlation != causation. the problem is more likely caused by an increase in your indexing rate (which would reduce free space more quickly). Those EBS volume are *extremely* small for an elasticsearch cluster. With EBS gp2 your disk throughput and access rate is based on the disk size, so those disks you have attached only provide a guaranteed 90 IOPs. I generally don't recommend running elasticsearch on EBS at all. i3 instances with their local attached nvme work significantly better for ES workloads. – jordanm Apr 21 '21 at 19:51

1 Answers1

2

Your interpretation of the metric is not right.

FreeStorageSpace: The free space for data nodes in the cluster. Sum shows total free space for the cluster. Minimum and Maximum show the nodes with the least and most free space, respectively. This metric is also available for individual nodes.

So when you increase the EBS size, the ability to store more data (FreeStorageSpace) also increases.

The "cluster_block_exception" error you are getting is due to a different reason - "SERVICE_UNAVAILABLE" - no master. It is recommended to have 3 master eligible nodes for better cluster availability.

Ankit Garg
  • 540
  • 2
  • 9
  • Yes you might be right. Its a bit confusing. We have added more EBS storage and this morning the logs are clear from the errors. Master instance and Kibana also did not crash like they did every morning previously. Indexing is still getting stuck somewhere though, but I think thats a Magento 2 issue that we need to resolve. – Kalvin Klien Apr 22 '21 at 16:50