2

Looking through the instructions -- https://www.cloudera.com/documentation/kafka/latest/topics/kafka_command_line.html

I'm running these test command lines and one set works, but the other set doesn't.

Following instructions, it works, but noticed it has "zookeeper" as a parameter and I thought it was discontinued.

Producer:

/usr/bin/kafka-console-producer --broker-list local-ip:9092 --topic test

Consumer:

/usr/bin/kafka-console-consumer --bootstrap-server local-ip:9092 --topic test --from-beginning

the above doesn't work on the Cloudera version, but works on my standalone Kafka installs.

This works on Cloudera:

/usr/bin/kafka-console-consumer --zookeeper local-ip:2181 --topic test --from-beginning

Trying to understand what the difference between the Cloudera's Kakfa version (3.0.0-1.3.0.0.p0.40?) and mine (2.11-0.11.0.1) or there has to be something turned on or off.

I see some similar topic, and tried following them to no avail. I think it's something to do with Cloudera.

Rob Koch
  • 1,523
  • 3
  • 18
  • 26
  • 1
    The kafka version that you are using (2.11-0.11.0.1) supports both the options. --bootstrap-server as well as --zookeeper. --zookeeper was the older method and is not recommended in Kafka 0.11 but I think Cloudera has not upgraded its distribution. – afsd Dec 06 '17 at 17:53
  • Did you get some advance? I have the same issue: The configuration 'offsets.storage' was supplied but isn't a known config. – Daniel Argüelles Feb 26 '18 at 17:07
  • How many kafka brokers are you using? I've just reinstall zookeeper and kafka (deleteing all related data from disks) and set offsets.topic.replication.factor=2 in kafka's setup because I only have 2 brokers. Now it works like magic – Daniel Argüelles Feb 28 '18 at 08:37

1 Answers1

2

Updated answer:

In my case, I have two brokers configured and the kafka's config value for offsets.topic.replication.factor set to 3. So, when Kafka tries to build a topic with more replicas than available brokers, an exception is thrown and the topic is not created.

The solution is to set offsets.topic.replication.factor = 2 and try again. Maybe you need to remove and deploy the brokers again.

I don't know why, maybe is a bug in Cloudera's Kafka release, but I solved it with a local kafka test.

I've downloaded latest version of Kafka from https://kafka.apache.org/downloads and updated the broker config file config\server.properties to use remote zookeeper server. With this, I had a mix configuration broker cluster:

  • brokers in my laptop
  • zookeeper in the cloudera cluster

With this configuration, I created a topic and run the kafka-console-consumer and kafka-console-producer from my laptop but against remote zookeeper:

$ kafka-topics --create --zookeeper zookeeper.cloudera-cluster:2181 --replication-factor 1 --partitions 1 --topic test
$ kafka-console-consumer --broker-list localhost:9092 --topic test
$ kafka-console-producer --broker-list localhost:9092 --topic test

this works propertly. Furthermore, using this the topic __consumer_offsets has been created automatically and now the new-consumer version works perfectly. At this point, you can remove the topic created and stop the local brokers and start to use kafka cluster normally.

Is this a bug from Cloudera's release? Maybe the version of Cloudera is not able to build __consumer_offsets automatically?

  • Kafka version downloaded: kafka_2.11-1.0.0.tgz
  • Cloudera's kafka version: 3.0.0-1.3.0.0.p0.40
Daniel Argüelles
  • 2,229
  • 1
  • 33
  • 56