1

I have a route in camel to consumer from kafka. It is consuming and producing with a TPS of 2000 if the incoming message is 18000 TPS. so the consumer topic has consumer lag. If I keep max.poll.recors = 500 i'm able to achieve 2000 TPS. If I keep producer settings requestRequiredAcks=0 I can achieve 4000 TPS. but still with consumer lag. We know that camel route is complete when from->to is complete. a consumer which is consuming from 2 partitions with a consumer count 2, is busy until the route is complete.

is there a way to make camel-kafka consumer asynchronous . any code example?


        from("kafka:{{consumer.topic}}?brokers={{kafka_dev.host}}"
                    + "&maxPollRecords={{consumer.maxPollRecords}}" + "&consumersCount=2"
                    + "&seekTo=latest" + "&groupId={{consumer.group}}" + "&keyDeserializer="
                    + KEYDESERIALIZER + "&valueDeserializer=" + VALUEDESERIALIZER + SSL).doTry()
                            .routeId("route1")
                            .process(new CamelProcessor())
                            .to("kafka:{{producer.topic}}?brokers={{kafka_dev.host}}" +"&requestRequiredAcks=1" )

                            .doCatch(Exception.class));

Also we have observed that introducing threads in this route re-reads the same processed and sent messages. is this link saying camel-kafka https://stackoverflow.com/questions/56716812/how-to-commit-offsets-thread-safe-using-camel-kafka

OneCricketeer
  • 179,855
  • 19
  • 132
  • 245
driven_spider
  • 495
  • 4
  • 16
  • Have you done an actual Kafka consumer benchmark on the same machine? And `acks=0` is fire-and-forget... Do you care about dropped messages? – OneCricketeer Jan 24 '20 at 19:36
  • Hi @OneCricketeer, Did you find any solution for this problem ? Can you tell me how to configure Camel kafka multithreaded consumer to increase the consumer throughput and reduce the consumer lag ? I understand that we can make number of consumers as same as number of kafka partitions . But I would like to know multiple threads processing the messages within one Kafka consumer. – shijin raj Jul 14 '22 at 07:42
  • @shijinraj This is not my post. Please post your own issue if you'd like to ask a new one – OneCricketeer Jul 14 '22 at 19:18

0 Answers0