0

As asked in question How to manually control the offset commit with camel-kafka? I want to commit offsets manually using camel-kafka. My route:

.from(kafka:topic1)
 .aggregate(new GroupByExchangeStrategy())
.to(kafka:topic2)
 .process(new ManualCommitProcessor())

, where ManualCommitProcessor will do the commitment after sending the message to another topic.

Problem is that aggregator and kafka producer are working in separated threads to the kafka consumer which is responsible for offset commitment. Hence, I am ending in 

java.util.ConcurrentModificationException: KafkaConsumer is not safe for multi-threaded access

Is there a possibility to call the consumer thread again after aggregation and dispatching to commit offsets?

R_FF92
  • 552
  • 1
  • 5
  • 19

1 Answers1

0

No this is not possible, the consumer thread runs independently of the output of the aggregator.

Claus Ibsen
  • 56,060
  • 7
  • 50
  • 65
  • Ok.@Claus Ibsen: I need the aggregator because I found out that the producer will not batch records without it. However, with auto commit there is risk to lose data (which is a real problem for our usecase) because commitment is done before the aggregate is really sent out to the other topic. Is there an alternative way to deal with this? – R_FF92 Jun 24 '19 at 08:00
  • @Claus Ibsen : camel-kafka threads are unsafe ? I observed when used threads in my route. my route consumed teh same message again after couple hours. I have autocommit to true & producer requestRequiredAcks=0 – driven_spider Jan 24 '20 at 13:17