I am using R2DBC Springboot framework to perform my data extraction/insertion. I noticed that when i tried to perform the query below, it is not saving the records in parallel but sequentially. Saving record takes more than 25 minutes to save 2 millions of records.
I would like to check how can i go about introducing parallelism so that records can be inserted simultaneously?
Thanks in advance
List<DataRecord> recordsToSave = convertMapToList(keyMapToDataRecords);
Integer MAX_BATCH_SIZE = isNull(jobParams.get(BATCH_SIZE.getValue()))
? parseInt("5000")
: (Integer) jobParams.get(BATCH_SIZE.getValue());
return ParallelFlux.from(Flux.fromIterable(
partition(new ArrayList<>(recordsToSave), MAX_BATCH_SIZE)
.parallelStream()
.map((List<DataRecord> data) -> saveInBatch(data).collectList())
.collect(Collectors.toList())
)).runOn(Schedulers.parallel())
.sequential()
.flatMap(d -> d)
.collectList()
.map(List::size);