-1

Im facing the problem of setting up a production ready elasticsearch cluster. At the moment im storing only the testing logfiles in elasticsearch.

So far so good, but since we have Production Logs of 1TB per Day i was wondering how to setup an elasticsearch index properly for this use case.

We want to save these logs for 30Days. The Cluster Setup has 100TB Disk Space. I would like to choose a Replica Count of 3, so the used disk space should be around 90TB.

But how many shards should i allocate?

Is there a difference between the Shards in Elastic and the Lucene Segments?

Raynigon
  • 660
  • 1
  • 9
  • 21
  • 1
    Each Elasticsearch shard is a Lucene index, but there's no such thing as a Lucene shard. Then you should consider reading this article that explains how to size your shards: https://www.elastic.co/blog/how-many-shards-should-i-have-in-my-elasticsearch-cluster – Val Jun 18 '18 at 15:24

1 Answers1

1

You should read article, that was sent by Val. But in case of logs you can create one index per day, this strategy can give you an ability to try different configurations.

Count of replicas should depend on count of your elasticsearch nodes.

You can also read this short article: https://www.elastic.co/guide/en/elasticsearch/guide/current/_how_primary_and_replica_shards_interact.html

And if you have doubts about count of replicas, this one can also help you: https://codingexplained.com/coding/elasticsearch/understanding-replication-in-elasticsearch

Vadim Beskrovnov
  • 941
  • 6
  • 18