1

In the spring batch program, I am reading the records from a file and comparing with the DB if the data say column1 from file is already exists in table1.

Table1 is fairly small and static. Is there a way I can get all the data from table1 and store it in memory in the spring batch code? Right now for every record in the file, the select query is hitting the DB.

The file is having 3 columns delimited with "|".

The file I am reading is having on an average 12 million records and it is taking around 5 hours to complete the job.

user3379502
  • 223
  • 4
  • 22

3 Answers3

1

I'd use caching like a standard web app. Add service caching using Spring's caching abstractions and that should take care of it IMHO.

Michael Minella
  • 20,843
  • 4
  • 55
  • 67
1

Preload in memory using a StepExecutionListener.beforeStep (or @BeforeStep).
Using this trick data will be loaded once before step execution.
This also works for step restarting.

Luca Basso Ricci
  • 17,829
  • 2
  • 47
  • 69
0

Load static table in JobExecutionListener.beforeJob(-) and keep this in jobContext and you can access through multiple steps using 'Late Binding of Job and Step Attributes'. You may refer 5.4 section of this link http://docs.spring.io/spring-batch/reference/html/configureStep.html

Srinivas Kasarla
  • 131
  • 5