3

I'm running rdkafka_simple_producer.c to produce messages to a Kafka cluster. I have one topic and 30 partitions. Using the default round-robin partitioner. While the producer is working and generating messages to Kafka, I add more partitions to Kafka 

kafka/bin/kafka-topics.sh --alter --zookeeper server2:2181 --topic demotest --partitions 40

I'd expect the producer to realize about the change and eventually begin to produce to all 40 topics. However, at the end I only see data was produced to the original 30 partitions.

In the test the producer ran for 2 minutes.

Do I need to add any function call in the simple_producer or is it a Kafka parameter I need to consider?

Thanks in advance!

rodolk
  • 5,606
  • 3
  • 28
  • 34

1 Answers1

5

I finally have a response for this question. Actually the producer is refreshed periodically about metadata. The interval is defined by configuration parameter topic.metadata.refresh.interval.ms

The value of topic.metadata.refresh.interval.ms is 300000 (in ms) by default. This is 5 minutes and test lasted only 2 minutes. The producer will recognize the new added partitions automatically after the metadata refresh.

So, for anybody with the same problem, you can configure this parameter to have faster metadata refresh. No coding needed more than setting this value with:

rd_kafka_conf_set(conf, "topic.metadata.refresh.interval.ms", "15000",
                          errstr, sizeof(errstr)) != RD_KAFKA_CONF_OK)

Of course, you need to consider this implies more messaging between a broker and a producer. More on librdkafka configuration: librdkafka configuration

rodolk
  • 5,606
  • 3
  • 28
  • 34