We have a ELK (ElasticSearch-Logstash-Kibana) deployment in which we ship logs via logstash to Elasticsearch Cluster. Indices are created daily. We close indices which are more than 3 days old and take a snapshot of indices which are more than 7 days old and push them to Amazon S3 via curator.
We have about 10 different daily indices where average size of each index about 1GB. Replication factor of 1. Each index is having 2 shards. Logstash pushes data to ES Cluster at the rate 2000 log_events per second
Our topology
- 3 Dedicated Master + Data
- 1 Dedicated Client Node + Kibana
Hardware Configs
- 12 Core
- 64 GB RAM
- 2 TB Spinning Disk
- Debian 7
- ElasticSearch Version - 1.7.1
- Logstash - 1.5.3
All standard config has been followed like unicast mode in discovery , 30 GB RAM has been allotted.
Right now the snapshot job is run via the curator from the Client machine with the requests sent locally to the ES instance running on the Client machine. Logstash sends logs directly to the Client Node
The curator command that is being used:-
curator --timeout 21600 --host es-client --port 9200 snapshot --name $snapshot_name_$project-$date --repository walle_elk_archive indices --older-than 3 --time-unit days --timestring %Y-%m-%d --prefix $prefix
Can someone help me in the following:-
- Is it ok to run the curator job on the client machine as we have done?
- Is it ok to take the snapshot of all the indices from a single machine?
- Since the logs are pushed continuously will it make the cluster unstable when snapshot creation and pushing to Amazon S3 is going on?
- What are the best practices people generally follow for backing up old indices from Elasticsearch?