0

I have a method which uses JdbcPagingItemReader to read 1M rows from a dynamic query.

JdbcPagingItemReader itemReader = new JdbcPagingItemReader();
itemReader.setDataSource(dataSource);

String hardsql = "Dynamic SQL string";
SqlPagingQueryProviderFactoryBean qu = new SqlPagingQueryProviderFactoryBean();
qu.setDatabaseType("Oracle");
qu.setSelectClause(StringUtils.substringBefore(hardsql, "FROM"));
qu.setFromClause(StringUtils.substringAfter(hardsql, "FROM"));

itemReader.setQueryProvider((PagingQueryProvider) qu.getObject());
itemReader.setRowMapper(new CustomDataMapper());
itemReader.setPageSize(500);
itemReader.setFetchSize(500);

ExecutionContext executionContext = new ExecutionContext();
itemReader.open(executionContext);

// Do further processing and create a list of objects from itemReader.read();

itemReader.close();

Is there any way/working example available, by which I can parallelize the process(e.g. each thread works on 10K rows parallely and finally resulting the list of objects which can be returned). I am using oracle 11g, spring batch version 2.2.4.Release.

Thanks in advance..

Soutrick
  • 77
  • 1
  • 9
  • Why are you doing it yourself instead of letting Spring Batch do that for you? that way your requirement would be a matter of configuration... – M. Deinum Dec 22 '14 at 14:08
  • Hi @M.Deinum, I am not using spring batch configuration, because I am not sure how spring batch would behave without a processor or item writer. Moreover this is a dao method and the object list will be returned to a business method for further processing, and I am not sure how to have a object list returned to a java method from spring batch job. – Soutrick Dec 22 '14 at 15:30
  • hi @Soutrick in spring-batch the processor is not mandatory, the reader and writer are though. You can use task-executor tasklet setting the throttle-limit for running it in multi-thread and use a reader to fetch the rows in batch of 500 and write them to the db. If for some reason you don't need to transform rows one by one but in batch before writing it in the db you can do that in the Writer. Check this SO for more details https://stackoverflow.com/questions/20386642/spring-batch-which-itemreader-implementation-to-use-for-high-volume-low-laten – gzp___ Jun 09 '20 at 14:59

0 Answers0