How to cache millions of records in cache of CacheManager when using JdbcPagingItemReader?

Question

I want to take opportunity to ask the question further from: getting data from DB in spring batch and store in memory. In my case, I am loading data from different tables and create the Hierarchy of Data like Person deatils, Customer Address, Bank Identification details etc., so it create tree and load the data into elastic.

To create hierarchy, I would need to keep all data in cache so that one type could search for other type and create final data out of it.

I've almost 5 tables each of them has around 1 million records which I want to cache in order to create final object.

Almost all batch jobs has JdbcPagingItemReader implemented so how will cache the data using Cache where ? What will be the appropriate process ?

I am thinking to use ConcurrentMapCacheManager here. Could you please share example ?

Does this answer your question? [Spring Batch With Annotation and Caching](https://stackoverflow.com/questions/52642796/spring-batch-with-annotation-and-caching) — Mahmoud Ben Hassine, Jun 25 '20 at 20:34
I shared a complete example here: https://stackoverflow.com/a/52644962/5019386. 5 tables with 1M records each is too much for a cache IMO, but if you give it enough memory that should be fine. — Mahmoud Ben Hassine, Jun 25 '20 at 20:35
Can you please confirm if ConcurrentHashMap capable of holding 10 million records in cache ? — Jeff Cook, Jun 26 '20 at 05:51
If you give enough memory to your JVM, I don't see why it couldn't. — Mahmoud Ben Hassine, Jun 26 '20 at 12:25
Is there any way if we can read data from cache using Chunk based ? — PAA, Feb 24 '22 at 08:11

How to cache millions of records in cache of CacheManager when using JdbcPagingItemReader?

0 Answers0