0

I am using R2DBC Springboot framework to perform my data extraction/insertion. I noticed that when i tried to perform the query below, it is not saving the records in parallel but sequentially. Saving record takes more than 25 minutes to save 2 millions of records.

I would like to check how can i go about introducing parallelism so that records can be inserted simultaneously?

Thanks in advance

  List<DataRecord> recordsToSave = convertMapToList(keyMapToDataRecords);
Integer MAX_BATCH_SIZE = isNull(jobParams.get(BATCH_SIZE.getValue()))
    ? parseInt("5000")
    : (Integer) jobParams.get(BATCH_SIZE.getValue());
return ParallelFlux.from(Flux.fromIterable(
        partition(new ArrayList<>(recordsToSave), MAX_BATCH_SIZE)
            .parallelStream()
            .map((List<DataRecord> data) -> saveInBatch(data).collectList())
            .collect(Collectors.toList())
    )).runOn(Schedulers.parallel())
    .sequential()
    .flatMap(d -> d)
    .collectList()
    .map(List::size);
Ugine
  • 143
  • 8

0 Answers0