1

I subscribed a Kafka topic with only one partition. I want to read the data, but failed, is there something wrong?

My script is as follows:

brokerList = "....."
groupId = "Test001"
topics=["TEST1"]
consumerCfg = dict(string, any);
consumerCfg["metadata.broker.list"] = brokerList;
consumerCfg["group.id"] = groupId;
consumer = kafka::consumer(consumerCfg);
kafka::subscribe(consumer, topics);

I called function kafka::pollByteStream(consumer) but still could not get the data.

winnie
  • 183
  • 1
  • 6

1 Answers1

0

DolphinDB Kafka plugin reads data from the latest record by default. Data already published in topic cannot be consumed on Kafka with the above script.

You can set offset to get the previous records as follows.

consumerCfg["auto.offset.reset"] = "earliest"

For more parameters, you can refer to https://github.com/confluentinc/librdkafka/blob/master/CONFIGURATION.md

jinwandalaohu
  • 226
  • 1
  • 7