I have a separate machine for elasticsearch. It is 500GB, but logs are consuming full memory in 24 hours. How do I compress it and free memory?

Question

[2019-08-01T13:20:48,015][INFO ][logstash.outputs.elasticsearch] retrying failed action with response code: 403 ({“type”=>“cluster_block_exception”, “reason”=>“index [metricbea...delete (api)];“})

Answer 1

Your log message is cut off. Is it by any chance actually this one or close to it?

[logstash.outputs.elasticsearch] retrying failed action with response code: 403 ({"type"=>"cluster_block_exception", "reason"=>"blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];"})

That would mean your disk is full (hit the floodstage watermark, which is at 95% by default). I can't really see anything related to memory in your log message.

To clear the floodstage: Add disk space (or delete old data) and then you will need unlock all affected indices with something like this:

PUT /_all/_settings
{
  "index.blocks.read_only_allow_delete": null
}