2

I am trying to implement a Camel Spring Boot application that is using FileComponent to poll on a directory. I also want to support clustering meaning multiple instances of this Camel-spring boot application could be started and consume from the directory.

I am trying to implement the IdempotentRepository on the File consumer with KafkaIdempotentRepository. However, when I start two instances at the same time, both of them consume a file coming into the directory and both instances broadcasts action:add for key my_file_name.

The configuration for the file component is the following:

file:incoming?readLock=idempotent&idempotentRepository=#myKafkaRepo&readLockLoggingLevel=WARN&shuffle=true

All the examples on clustered Idempotent Repository were with Hazelcast and for me it is difficult to impose on my users from operational reasons.

My question: does KafkaIdempotentRepository support clustered IdempotentRepository? If not which implementation would you suggest to use?

Tanzhu Ozen
  • 33
  • 4

1 Answers1

0

Kafka:: Apache Camel - IdempotentRepository Documentation

On startup, the instance subscribes to the topic and rewinds the offset to the beginning, rebuilding the cache to the latest state. The cache will not be considered warmed up until one poll of pollDurationMs in length returns 0 records. Startup will not be completed until either the cache has warmed up, or 30 seconds go by; if the latter happens the idempotent repository may be in an inconsistent state until its consumer catches up to the end of the topic.

My opinion

  • It depends how many recently processed records you need to remember and what the retention period of the topic will be.

  • If you can set the retention time of the topic is big enough that it satisfies your number of records to remember requirement but small enough for cache warm up can complete in much less than 30 seconds, go for it.

Kavithakaran Kanapathippillai
  • 7,883
  • 1
  • 11
  • 29