0

I have a consumer application which I would like to speed up. Most of the time consumer is waiting for http response from the third party system hence I cannot proceed with other messages in queue and lag increases. What would be a recommended way to improve performance in such cases. My ideas:

  1. Create multiple consumers on separate threads
  2. Consume a batch of messages, run Parallel.ForEach to process them and then commit. But then what to do if one the messages in batch fails during the processing?
user2412672
  • 1,459
  • 3
  • 20
  • 36
  • How many consumers are you currently using? and how many partitions does your topic have set? You could increase the number of consumers, yes. However, if you have performance problems upstream you would also need to resolve there. In terms of improving consumer performance, it depends on what measurement you are focussed on, throughput or latency? – Jessica Vasey Jan 23 '23 at 17:53
  • 16 partitions and 2 consumers on different instances. I see that my instance resources are not fully utilized because of the time I'm just waiting for http response. I'm focused on higher throughput. – user2412672 Jan 23 '23 at 20:00

2 Answers2

0

The greater the number of partitions, the greater the throughput (although there are some downfalls such as increased unavailability and some providers limit the number of partitions per broker & cluster).

You mentioned you have 16 partitions, I'm assuming you calculated this based on target throughput.

To optimise your consumer for throughput, you could increase fetch.min.bytes. See Confluent guide here.

This will increase the data the consumers get for each fetch request. This configuration sets the min number of bytes expected for each response and reduces the number of fetch requests to the leader. The leader won't send the consumer messages until this is met or the wait time exceeds fetch.max.wait.ms.

You should also use consumer groups with multiple consumers (I'm not sure if earlier you meant 2 consumer groups with multiple consumers but if not you can increase this so the partitions are evenly distributed).

Hope this helps.

Jessica Vasey
  • 382
  • 1
  • 7
0

Since you are using .NET, you may want to take a look at KafkaFlow (https://github.com/farfetch/kafkaflow). KafkaFlow let you build multi-threaded consumers with message order guarantee, so it's a good tool for what you are looking for.

Gui Ferreira
  • 4,367
  • 6
  • 28
  • 41