Server Fault Stack Exchange
Server Fault Stack Exchange

Questions tagged [kafka]

Kafka is a distributed messaging queue application written in java.

Kafka website: http://kafka.apache.org/

When asking a question about provides relevant information like the kafka version and the version. The design of your cluster should be mentioned with the number of brokers, the topic configuration.

When you have a bug on a particular topic, perform the same test on another topic with different parameters and observe if you can reproduce.

97 questions
0
votes
1 answer

hadoop + can we install zookeeper servers on kafka hosts

we want to dedicated the zookeeper servers only for kafka machines so each kafka machine include the zookeeper server and zookeeper server will serve only the kafka host and not other application in that case is it ok?
shalom
  • 461
  • 13
  • 29
0
votes
1 answer

Kafka Producer & Consumer hosts

Would Kafka Producer & Consumer hosts can be same. Suppose if I've producers like host1,host2 & host3. Can I use same host1, host2 & host3 as consumers? What are the consequences/challenges if I use producers & consumers as same hosts.
KKE
  • 135
  • 1
  • 4
  • 11
0
votes
0 answers

reducing tps in kafka disks on linux

we're using kafka brokers with: 24 HT cores 256GB RAM 16 X 7.2K RPM NLSAS disks configured in RAID-10 and replication factor 3. the sar -q command the runq-sz reached ~40 on average, load average reached 10 and the CPU was only 10% us and 5% sys.…
Elad Eldor
  • 61
  • 3
0
votes
1 answer

Dose zookeeper servers must use SSD disks?

we want to install kafka cluster and 3 zookeeper servers kafka should use the zookeeper servers in order to save the metadata on the zookeeper servers ZK Data and Log files should be on disks, which have least contention from other I/O activities.…
shalom
  • 461
  • 13
  • 29
0
votes
1 answer

Kafka multiple listeners

Initial apologies for the long post (this is also on superuser as i wasn't sure the best place for this https://superuser.com/questions/1404421/kafka-multiple-listeners so let me know if one needs closing)... I have setup a Kafka cluster in AWS with…
Simon Thorley
  • 1
  • 1
  • 2
0
votes
1 answer

kafka machines in the cluster and kafka comunications

We have kafka cluster with 3 kafka brokers nodes and 3 zookperes servers kafka version - 10.1 ( hortonworks ) from my understanding since all meta data is located on the zookeeper servers , and kafka brokers are using this data ( kafka talk with…
shalom
  • 461
  • 13
  • 29
0
votes
1 answer

How to monitor Kafka Consumer lag in Stackdriver?

I can view all the Kafka broker metrics by using the JMX exporter -> stackdriver monitoring agent and is displayed in the Stackdriver but i cant see any consumer metrics, i am particularly interested in consumer lag, is there a way to monitor…
Dchan
  • 1
0
votes
1 answer

stackdriver not monitoring kafka metrics

I am trying to setup kafka metrics on stackdriver. I followed all steps as mentioned in https://cloud.google.com/monitoring/agent/plugins/kafka , but still cant see "kafka" metrics. Although I can see metrics of Host, Agent, Zookeeper and JVM I can…
himanshu
  • 1
0
votes
0 answers

Avoid kafka disk to became 100% used by Cron job

We want to suggest the following based on our issues on kafka disks We have many HDP clusters ( based on ambari , and all machines are redhat version 7.2 ) Each cluster include 3 kafka machines , while each kafka include disk with ~15 T Because we…
shalom
  • 461
  • 13
  • 29
0
votes
1 answer

What are the minimum permissions needed by Kafka brokers for them to operate?

I'm securing a Kafka cluster. I read and followed the Confluent Kafka Security Tutorial, and it works like a charm. Now, I would like to add ACLs to the brokers in order to remove them from the super users list (super.users=User:kafkabroker). What…
Eric Citaire
  • 181
  • 1
  • 9
0
votes
1 answer

kafka with ACL fails to connect zk and stops

I have a kafka installation (with ssl listener and ssl client authentication). It works fine until I try to add ACL. this is the error (full error in the bottom): [edit-1] the first error on kafka startup is this: [2018-07-25 14:15:10,498] ERROR…
ozma
  • 121
  • 1
  • 5
0
votes
2 answers

Kafka cluster: Withstanding a network outage

We have a kafka cluster. And a network. Yay. The network will be unavailable across all racks in our data center for 5-10 minutes (!) because maintenance requires it. I'm concerned that is too long an outage for kafka to handle gracefully and that…
Eric Horne
  • 1
0
votes
1 answer

Is it possible to get Apache Kafka to present an SSL intermediate certificate?

After reading this documentation: http://kafka.apache.org/documentation.html#security_ssl I was able to set up Kafka using SSL with a self-signed certificate. I tried setting it up with a Digicert certificate, and it can present the certificate, but…
danznz
  • 3
  • 2
0
votes
2 answers

Mongodb primary node not Resolved in Debezium Kafka Connect — Docker Compose

i have this docker compose file, that contains bitnami/mongodb containers for creating a replica set. and I've used zookeeper, kafka & debezium-connector for monitoring my mongodb replica set. version: '2.2' services: mongodb-primary: …
hossein bakhtiari
  • 1
  • 2
0
votes
1 answer

How to return the original log retention after purge kafka topics

We have 3 kafka machines version - 0.10.0.2.6 We want to purge all data files from all kafka's topics ( purge completely all data ) The CLI for this action should be ( should be run from the zoo server ) kafka-topics.sh --zookeeper localhost:2181…
shalom
  • 461
  • 13
  • 29
1 2 3 4
5
6 7