0

I'm very new to ELK stack and trying to learn about it. I want to understand if how to plan the capacity for 100GB/day data to be kept for 30 days , how to plan for it. The data is mainly access logs, application logs, events. I will need to run few regular expression to populate data fields ( unless there is any default delimiter support for it ) . I'm planning to keep 3 replicas of it .

Can please someone guide me if how many servers and in what type of configuration would be required for this ?

Thanks very much in advance .

Learner
  • 1,544
  • 8
  • 29
  • 55

1 Answers1

1

We use the "hot-warm architecture" (docs) of clustered elasticsearch instances (you can run multiple on one server!). Each with 31g memory.

Our "hot" instances configured with an retention time of 3 days and an ssd raid. After this three days data is moved to an "warm" HD Raid with an retention time of (30 - 90 days). Its nice because latest data to search in kibana is quite fast processed because of the ssds.

Sometimes we started 1 Elastic Master (4g memory) and 2 Elastic Clients (31g memory) on a single server.

In my opinion such a machine should have at least about 128g memory, 1tb ssd capacity, 4-6tb hd capacity.

HauLuk
  • 155
  • 1
  • 11