2

Hey folks can anyone explain to me why does i need to use fluentd for aggregator? Below is the infrastructure i have been working on

Fluentbit(Log Forwarder) --> Fluentd(Data Processing and Aggregator) --> Elasticsearch --> Kibana

Sorry for not show picture, but you can find the topologhy what i mean in here : https://medium.com/redbox-techblog/building-an-open-data-platform-logging-with-fluentd-and-elasticsearch-4582de868398

Okay maybe this is a weird a question, but i still dont understand why i need it? when i just can use the fluentbit to forward the log straight to elasticsearch, why we must need the aggregator?

For more information i will be using a lot of fluentbit for couple of node(around 50 nodes i think)

I appriciate if you help me, and sorry for my bad english

Thank you

for gag
  • 23
  • 2

2 Answers2

0

You don't need it, just make sure your Elasticsearch cluster is properly setup so it can handle the load with several incoming connections and traffic.

edsiper
  • 398
  • 1
  • 4
  • Okay thanks for the answer ed really appriciate it, but will it be good enough for production ready? And if we dont need aggregator for it, why someone suggest it we have to? Are those kind of tophology for something big? Or something else? Thank you if you reply again, because i really neet it for my essay to graduate. – for gag Apr 11 '20 at 03:29
0

A log collector like fluentd or logstash makes sure that there is no risk of "bad data", which may cause problems while in production. Log collectors provides a unified logging layer, decoupling data sources from back-end systems.

Hope this helps.

Ayush Pallav
  • 919
  • 9
  • 18