Cache Kafka Records using Caffeine Cache Springboot

Question

I am trying to cache Kafka Records within 3 minutes of interval post that it will get expired and removed from the cache.

Each incoming records which is fetched using kafka consumer written in springboot needs to be updated in cache first then if it is present i need to discard the next duplicate records if it matches the cache record.

I have tried using Caffeine cache as below,

@EnableCaching
public class AppCacheManagerConfig {

    @Bean
    public CacheManager cacheManager(Ticker ticker) {
        CaffeineCache bookCache = buildCache("declineRecords", ticker, 3);
        SimpleCacheManager cacheManager = new SimpleCacheManager();
        cacheManager.setCaches(Collections.singletonList(bookCache));
        return cacheManager;
    }

    private CaffeineCache buildCache(String name, Ticker ticker, int minutesToExpire) {
        return new CaffeineCache(name, Caffeine.newBuilder().expireAfterWrite(minutesToExpire, TimeUnit.MINUTES)
                .maximumSize(100).ticker(ticker).build());
    }

    @Bean
    public Ticker ticker() {
        return Ticker.systemTicker();
    }

}

and my Kafka Consumer is as below,

@Autowired
    CachingServiceImpl cachingService;

@KafkaListener(topics = "#{'${spring.kafka.consumer.topic}'}", concurrency = "#{'${spring.kafka.consumer.concurrentConsumers}'}", errorHandler = "#{'${spring.kafka.consumer.errorHandler}'}")
    public void consume(Message<?> message, Acknowledgment acknowledgment,
            @Header(KafkaHeaders.RECEIVED_TIMESTAMP) long createTime) {
        logger.info("Recieved Message: " + message.getPayload());
        try {
            boolean approveTopic = false;
            boolean duplicateRecord = false;
if (cachingService.isDuplicateCheck(declineRecord)) {
//do something with records
}
else
{
//do something with records
}
    cachingService.putInCache(xmlJSONObj, declineRecord, time);

and my caching service is as below,

@Component
public class CachingServiceImpl {
    private static final Logger logger = LoggerFactory.getLogger(CachingServiceImpl.class);
    @Autowired
    CacheManager cacheManager;

    @Cacheable(value = "declineRecords", key = "#declineRecord", sync = true)
    public String putInCache(JSONObject xmlJSONObj, String declineRecord, String time) {
        logger.info("Record is Cached for 3 minutes interval check", declineRecord);
        cacheManager.getCache("declineRecords").put(declineRecord, time);
        return declineRecord;

    }

    public boolean isDuplicateCheck(String declineRecord) {
        if (null != cacheManager.getCache("declineRecords").get(declineRecord)) {
            return true;
        }
        return false;
    }
}

But Each time a record comes in consumer my cache is always empty. Its not holding the records.

Modifications Done:

I have added Configuration file as below after going through the suggestions and more kind of R&D removed some of the earlier logic and now the caching is working as expected but duplicate check is failing when all the three consumers are sending the same records.

`

  @Configuration
  public class AppCacheManagerConfig {
  public static Cache<String, Object> jsonCache = 
  Caffeine.newBuilder().expireAfterWrite(3, TimeUnit.MINUTES)
            .maximumSize(10000).recordStats().build();
    @Bean
    public CacheLoader<Object, Object> cacheLoader() {
        CacheLoader<Object, Object> cacheLoader = new CacheLoader<Object, Object>() {
            @Override
            public Object load(Object key) throws Exception {
                return null;
            }

            @Override
            public Object reload(Object key, Object oldValue) throws Exception {
                return oldValue;
            }
        };
        return cacheLoader;
    }

`

Now i am using the above cache as manual put and get.

Answer 1

I guess you're trying to implement records deduplication for Kafka.

Here is the similar discussion:

https://github.com/spring-projects/spring-kafka/issues/80

Here is the current abstract class which you may extend to achieve the necessary result:

https://github.com/spring-projects/spring-kafka/blob/master/spring-kafka/src/main/java/org/springframework/kafka/listener/adapter/AbstractFilteringMessageListener.java

Your caching service is definitely incorrect: Cacheable annotation allows marking the data getters and setters, to add caching through AOP. While in the code you clearly implement some low-level cache updating logic of your own.

At least next possible changes may help you:

1. Remove @Cacheable. You don't need it because you work with cache manually, so it may be the source of conflicts (especially as soon as you use sync = true). If it helps, remove @EnableCaching as well - it enables support for cache-related Spring annotations which you don't need here.

2. Try removing Ticker bean with the appropriate parameters for other beans. It should not be harmful as per your configuration, but usually it's helpful only for tests, no need to define it otherwise.

3. Double-check what is declineRecord. If it's a serialized object, ensure that serialization works properly.

4. Add recordStats() for cache and output stats() to log for further analysis.