0

I have scenario where my Spring batch job is running every 3 mins.

Steps should be

  1. Each user's record should get executed parallel. Each user can have maximum of 150k records.
  2. Every user can have update and delete records. Update records should run before delete.
  3. Update/delete sets should run parallel on their own. But strictly all updates should complete before delete.

Can anyone suggest the best approach to achieve the parallelism at multiple levels and follow the order at update and delete level. I am looking something around Spring Async Executor Service, Parallel Streams and other Spring libraries. Rx, only if it gives some glaring performance which the above specified can't provide.

James Z
  • 12,209
  • 10
  • 24
  • 44
Shankar Narayana
  • 1
  • 1

1 Answers1

0

Glaring performance is based on the design of spring batch implementation and we are sure you will get with spring batch as we are processing millions of records with select delete and update.

  1. Each user's record should get executed parllely. Each user can have maximum of 1.5 lakh records. "Partition the selection based on User and Each user will run as parallel steps."
  2. Every user can have update and delete records. Update records should run before delete. " Create a Composite Writer and delegates added for update Ist writer and delete 2nd writer "
  3. Update/delete sets should run parallel on their own. But strictly all updates should complete before delete. "Each writer step update and delete manages the transaction and make sure update executes first ".

Please refer below

Spring Batch multiple process for heavy load with multiple thread under every process

Composite Writer Example

Spring Batch - Read a byte stream, process, write to 2 different csv files convert them to Input stream and store it to ECS and then write to Database

Rakesh
  • 658
  • 6
  • 15
  • How to order writers in composite writer such a way that update executes first before delete at user level? How can run a specific Writer records in parallel instead of sequential for loop – Shankar Narayana Feb 12 '21 at 16:39
  • CompositeWriter is a sequential execution to manage complete transaction . Ordering is based on adding order. Parallel execution you already achieved in step level and you can extend based on your system resources. In summary each parallel step is a combination of reader,processor and writer in single boundary transaction. This helps to rollback complete transaction. – Rakesh Feb 12 '21 at 17:17
  • I have to read from DB and write to kafka. Can you give me any link on KafkaWriter. – Shankar Narayana Feb 13 '21 at 05:06
  • Could you open a new question about KafkaWriter and mark this as answered so that the specific experts will answer for that. This question is related to parallel processing and Spring batch transaction steps – Rakesh Feb 13 '21 at 06:56