0

I have to build a log-cluster and monitoring cluster ( For high-availability ) like this topology. I'm wondering to know how to config those log-shippers clusters. ( I have 2 Topo in the Image)

  1. If I use Kafka with FileBeat in Kafka Cluster, Will Elastic Search receive duplication data because Kafka has replicas in data?

  2. If I use Logstash (In Elastic Search Cluster) for getting logs from Kafka Cluster, how the config should be because I think that Logstash will not know where to read the log efficiency on Kafka Cluster.

Cluster topology

Thanks for reading. If you have any idea, please discuss with me ^^!

Akatsuki Pain
  • 1
  • 1

1 Answers1

1

As i see both configurations are compatible with Kafka, you can use filebeat, logstash or mixed them in consumer and producer stages!

IMHO all depends about your needs, ie: sometimes we use some filters to rich the data before ingest to kafka (producer stage), or before index the data to elastic (consumer stage), in this case is better work with logsatsh, because is easier using filters than in filebeat

But if you want to play with raw data, maybe filebeat is betther, because the agent is lighter.

About your questions:

  1. Kafka has the data replicted, but for HA propouses, you only read one time the data with the same consumer group
  2. For read the log from kafka with logstash, you can use the logstash input plugin for kafka, is easy and works fine!

https://www.elastic.co/guide/en/logstash/current/plugins-inputs-kafka.html

Víctor Oriol
  • 492
  • 4
  • 15
  • 1. So I have to do a good configuration in Kafka Cluster to suite with Elastic Cluster, Right? 2. Ye, I think so too, thanks. – Akatsuki Pain Aug 26 '20 at 10:18
  • For sure! for a "basic kafka production config" review this options: https://docs.confluent.io/current/kafka/deployment.html#production-configuration-options and for high avaliability, put your efforts in the "replication configs". – Víctor Oriol Aug 26 '20 at 11:50