1

We are facing an issue while running a Kafka consumer (Java). The poll returns fewer messages when the messages are larger. We have tried increasing different config parameters while constructing the consumer such as fetch.min.bytes, fetch.max.bytes, max.partition.fetch.bytes and fetch.max.wait.ms but the output is the same. (5-8 messages)

We found that the messages in a single poll amount to roughly 10KB in size. This was found when we tried to get the sum of byte length (message.value().getBytes().length) of the ConsumerRecords returned on a poll.

Vihit Shah
  • 314
  • 1
  • 5
  • I don't think I understand the problem. To me, it sounds like all the messages eventually arrive, but they don't necessarily arrive batched together in as large a batch when messages are bigger. Is that correct? – Ted Dunning Jan 10 '20 at 21:14
  • 1
    yes @TedDunning For eg, there are 10,000 records in the topic, the poll fetches only 5-6 messages at a time. I want to increase this batch. – Vihit Shah Jan 17 '20 at 12:47
  • @TedDunning any solutions? – Vihit Shah Jan 22 '20 at 07:09
  • Feedback from others is that changing these parameters has a substantial effect. What I am hearing from you is that there is no effect. In my own experience, that often means that I am not actually changing the parameters I think I am changing either due to changing the wrong config file or because another config is over-riding my changes. Is that possible in your case? – Ted Dunning Jan 23 '20 at 18:07
  • I have changed these parameters in my code and I see these changed values in the log as well which kafka generates (producer and consumer config). – Vihit Shah Jan 24 '20 at 06:50
  • I see that some parameters in MapR are ignored. such as receive buffer bytes and send buffer bytes, can these parameters have some impact on what i want to achieve? – Vihit Shah Jan 24 '20 at 06:51

0 Answers0