0

I have a code similar to the one below:

KafkaConsumer<Long, String> kafkaConsumer = new KafkaConsumer<>(properties);
partitions = Collections.singletonList(new TopicPartition(topic, partition));
kafkaConsumer.assign(partitions);
kafkaConsumer.seekToBeginning(List.of(partition));

ConsumerRecords<Long, String> records = kafkaConsumer.poll(Duration.ofMillis(pollDuration));

While running the code, I am not able to get all the messages in this particular topic. For example, if I have 24 records in this partition, I am able to get only 20 records.

This issue does not happen always. When we make Kafka (wurstmeister/kafka:1.1.0) up using docker in the same machine in which my application runs, I am able to get all records from that particular partition.

But when I make the same docker-compose up in another machine and connect to it, this issue happens.

Sandeep G
  • 13
  • 4

1 Answers1

0

Try increasing fetch.min.bytes and, possibly fetch.max.wait.ms too.

fetch.min.bytes The minimum amount of data the server should return for a fetch request. If insufficient data is available the request will wait for that much data to accumulate before answering the request. The default setting of 1 byte means that fetch requests are answered as soon as a single byte of data is available or the fetch request times out waiting for data to arrive. Setting this to something greater than 1 will cause the server to wait for larger amounts of data to accumulate which can improve server throughput a bit at the cost of some additional latency.

fetch.max.wait.ms The maximum amount of time the server will block before answering the fetch request if there isn't sufficient data to immediately satisfy the requirement given by fetch.min.bytes.

Community
  • 1
  • 1
Gary Russell
  • 166,535
  • 14
  • 146
  • 179
  • I tried setting the above mentioned values and after that added few messages into the topic. But after 2 days, when I checked, the problem still exists. Please note that i am using default values for log retention and offset retention – Sandeep G Nov 18 '19 at 04:22
  • `>the problem still exists` What "problem"? There is no guarantee that all records in a topic/partition will be returned on a poll. You need to keep polling. – Gary Russell Nov 18 '19 at 13:57
  • Sorry for replying late. But do you mean to say it is ok to keep polling in a loop till we get all the messages from the topic? – Sandeep G Dec 11 '19 at 03:59