1

I've a Cloudera cluster with a clusterized Kafka service. I've two instances of Kafka controllers, lets say C1 and C2.

When C1 is the active controller everything seems to work fine. When for some reason, C2 becomes the active controller. Some of the sent messages sent via kafka-console-producer are not received by a kafka-console-consumer (exactly a half of the messages sent are not received by the consumer, one every two messages). I'm not sure if this is due to the change of active controller and some partition cannot be read.

Also, I've another issue if I shutdown C1 and only keep C2 up. If I try to start a previously working Streamsets flow reading from kafka, I get the error message "Cannot retrieve metadata for topic XXXX", It seems the topic metadata is only present in C1, which is offline in this scenario.

If I open a kafka-console-consumer after shutting down the first broker, I get the following exception:

WARN [console-consumer-16627_node10.agatha-cluster-1515508696963-2e45e6d8-leader-finder-thread]: 
Failed to find leader for Set(testD-1, testD-0) 
(kafka.consumer.ConsumerFetcherManager$LeaderFinderThread)
kafka.common.KafkaException: fetching topic metadata for topics [Set(testD)] 
from broker [ArrayBuffer(BrokerEndPoint(183,110.250.17.242,9092))] failed

What I'm doing wrong when trying to use kafka with several brokers?

dhalfageme
  • 1,444
  • 4
  • 21
  • 42

2 Answers2

1

I had a typo in the IP address of the following configuration property for one of the brokers: advertised.listeners This was causing the second broker to be not accessible from other places. This was causing the two issues: the messages that were lost and the fetch topic metadata error

metadaddy
  • 4,234
  • 1
  • 22
  • 46
dhalfageme
  • 1,444
  • 4
  • 21
  • 42
0

First ensure two controllers up and then Create a new topic with partition 2 and replication factor 2

Ehasanul Hoque
  • 578
  • 8
  • 14
  • I tested it and I got the same result. These are the steps I've done: 1. Crate kafka topic with 2 partitions and replication factor 2 with 2 brokers up 2. Check that the streamsets flow works, can connect and receive data from topic 3. Turn down the active controller broker. Now only the second broker is up 4. When I try to start the streamsets flow it fails with error "Cannot find topic metadata on the second broker" – dhalfageme Jan 09 '18 at 13:57
  • I get the following exception in a kafka-console-consumer after shutting down the first broker: Fetching topic metadata with correlation id 0 for topics [Set(testD)] from broker [BrokerEndPoint(183,110.250.17.242,9092)] failed (kafka.client.ClientUtils$) java.nio.channels.ClosedChannelException – dhalfageme Jan 09 '18 at 14:39
  • Please check how many zookeeper is up in your cluster. if you have 2 zookeeper in your cluster, it is mandatory to up 2 zookeeper, If you have 3 zookeeper in your cluster at least 2 zookeeper must up. – Ehasanul Hoque Jan 10 '18 at 04:28
  • I already found the reason, see my answer please. Thanks for your help – dhalfageme Jan 10 '18 at 08:09