Stack Overflow
Stack Overflow

Questions tagged [kafka-cluster]

39 questions
1
vote
0 answers

kafka listening on multiple interfaces

I have a requirement as below: Kafka needs to listen to multiple interfaces, one external and one internal interface. All other components within the system will connect kafka to internal interfaces. At installation time internal ips on other host…
u_peerless
  • 644
  • 2
  • 9
  • 23
1
vote
1 answer

why should i use docker image "confluentinc/kafka" to kafka cluster?

I did docker-compose up for kafka clustering in my local. and whenever i want to make topic, retrieve topics i have or search data stored in the topic i use docker run --net=host --rm confluentinc/cp-kafka:latest kafka-topics --describe --topic…
Jaeho Lee
  • 463
  • 1
  • 5
  • 14
1
vote
1 answer

Filebeat not pushing events to multiple Kafka broker in a single Kafka cluster

The Filebeat in my setup pushes the events to a Kafka cluster with 2 brokers. I have added only one node in the host list but both the brokers in the cluster were discovered. I understood this from the Filebeat logs. But, even though both the…
Hariprasad S
  • 21
  • 5
1
vote
1 answer

Two (Kafka) S3 Connectors not working simultaneously

I have a Kafka Connect working in a cluster (3 nodes) with 1 connector (topic -> S3), everything is fine: root@dev-kafka1 ~]# curl localhost:8083/connectors/s3-postgres/status | jq -r % Total % Received % Xferd Average Speed Time Time …
Paulo Singaretti
  • 13
  • 2
1
vote
0 answers

Configuring 5 node Kafka cluster on Linux RHEL7.5 Virtul Machine

I am configuring 5 node Kafka cluster on Linux RHEL7.5 Virtual Machine. Can anybody suggest which Storage Type I have to use SAN or NAS ?
Priyanka Marihal
  • 33
  • 1
  • 5
1
vote
2 answers

Kafka cluster loses messages after zookeeper restart

I'm starting a cluster of kafka brokers using Docker (5 brokers for example, one broker per container). Kafka version 2.12-0.11.0.0, Zookeeper 3.4.10. The scenario: Starting 1st broker with config…
usbulat
  • 11
  • 3
0
votes
1 answer

Consumen multiple network interfaces of single machine for kafka cluster

I have a Linux machine with 3 network interfaces, let's say IPs are 192.168.1.101,192.168.1.102,192.168.1.103 I want to consume all 3 IPs of this single node to create a Kafka cluster with other nodes, Should all 3 IPs have their separate…
Rohit Dwivedi
  • 94
  • 8
0
votes
0 answers

Springboot kafka streams Application failed if one kafka broker went down

we are using springboot application to develop kafka streams application. Till these days we are using single broker only so we are not facing any issues But a week ago we created cluster mode with 3 zookeepers and 3 kafka broker for higher…
RandomUser
  • 31
  • 1
  • 8
0
votes
1 answer

Running Spring Cloud Data Flow in Cluster mode

Is there a possibility to run Spring Cloud Data Flow in cluster mode ? I have many apps running on SCDF, I wanted to ensure that SCDF is always up and running just like a kafka cluster.
Jibit George
  • 76
  • 4
0
votes
1 answer

Setting up a kafka cluster on a single machine and configuration

Trying to set a kafka cluster on a single machine following some online tutorials and edited the config/server.properties to choose the port 9091 for one broker and for another broker using 9092 and the respective zookeepers for kafka brokers are…
anon
0
votes
1 answer

What happens when min.compaction.lag.ms is greater than log.cleaner.max.compaction.lag.ms?

I'm using Kafka 2.2.0, so the option to configure max.compaction.lag.ms per topic is not yet available. Globally I have log.cleaner.max.compaction.lag.ms=86400000 (one day): For the topic __consumer_offsets, I'd like to delay the compaction to at…
Tobias Hermann
  • 9,936
  • 6
  • 61
  • 134
0
votes
2 answers

Not able to read messages from kafka consumer in kafka cluster setup

I have created two kafka brokers in a kafka cluster. When one broker is down I am not able to get any data to kafka consumer. I am using this command to read messages from consumer: bin/kafka-console-consumer.sh --topic test_kafka_cluster \ …
lakshmiravali rimmalapudi
  • 355
  • 6
  • 19
0
votes
1 answer

Remote Kafka Consumer

I have a 3-node Kafka cluster on another machine in which I cant connect and consume messages with Spring boot application remotely. I have changed my config file like this: advertised.listeners=PLAINTEXT://:9092 and here is my…
Yasmean Mohammadi
  • 5
  • 1
0
votes
1 answer

Kafka-Connect : Getting an unrecognised error when starting a S3 Sink Connector

I am trying to set up my 3rd worker for a Kafka Connect cluster with 3 nodes. The worker runs properly on this 3rd node and I'm able to make REST calls to get existing connectors (which for now I have 2, one on each node). However, when I try to…
DLSMatias
  • 11
  • 3
0
votes
1 answer

How to configure the Kafka Cluster to work with Elastic Search Cluster?

I have to build a log-cluster and monitoring cluster ( For high-availability ) like this topology. I'm wondering to know how to config those log-shippers clusters. ( I have 2 Topo in the Image) If I use Kafka with FileBeat in Kafka Cluster, Will…
Akatsuki Pain
  • 1
  • 1
1
2
3