0

I have a Spring Batch application. My Spring batch application consists of 2 steps.

  1. Extract csv data, add into Records tbl
  2. Extract Records tbl rows, parse into Food tbl based on data validation.

Step 2 is done using tasklets.
I need to process Records row tbl and no matter whether it is validated or not, the current DateTime would be added back into the Records row.
a. Fails validation, DateTime + error code would also be added into the Records row.
b. Passes validation, DateTime added into Records row. Row would also be added into Food tbl.

In step 2, I have a databaseReader and a databaseWriter. Where do I implement my business logic to check whether validation is pass/fail? It would be calling from my service classes.

I was thinking that my databaseWriter could call add/update/delete methods from my service class. If there's any error, my databaseWriter would catch it. Based on the error thrown by my service class, my databaseWriter would decide what error code + DateTime to parse back into the Record row.

What are your thoughts? Is this a good implementation?

technoob
  • 11
  • 3
  • Please let me know why we are separating step1 and step2 Could we do it in one step to avoid IO. – Rakesh Jan 02 '21 at 10:46
  • 1 reader --> 1 Processor-->CompositeWriter .Validation can be done in processor/listener . If your business scenarios are compatible this will be best solution. – Rakesh Jan 02 '21 at 14:13
  • Processors are for data transformation. Is it right to do validation there? Step 1 and step 2 is an requirement as I have to read my csv records into Record tbl first, before retrieving it for validation into Food tbl.. – technoob Jan 02 '21 at 16:16
  • You can do it in processor or listener. Based on validation success or failure modify your input object and handle . Check this helps for you https://stackoverflow.com/questions/65515206/spring-batch-read-a-byte-stream-process-write-to-2-different-csv-files-conve/65517381?noredirect=1#comment115832702_65517381 – Rakesh Jan 02 '21 at 17:57

2 Answers2

0

Step 2 is done using tasklets.

In step 2, I have a databaseReader and a databaseWriter. Where do I implement my business logic to check whether validation is pass/fail?

I don't know if you use those reader/writer in a simple tasklet or a chunk-oriented tasklet. But I would implement step 2 with a chunk-oriented tasklet and use an item processor for the validation logic. Data validation is a typical use case for an item processor: https://docs.spring.io/spring-batch/docs/4.3.x/reference/html/processor.html#validatingInput. You can decide to reject items or filter them in your processor.

Mahmoud Ben Hassine
  • 28,519
  • 3
  • 32
  • 50
0

I was thinking that my databaseWriter could call add/update/delete methods from my service class. If there's any error, my databaseWriter would catch it. Based on the error thrown by my service class, my databaseWriter would decide what error code + DateTime to parse back into the Record row.

A ItemWriter by its name should only update your storage data (e.g database). You can build a chunk based job with item reader - processor - writer with the processor as the business logic where you can filter records that you will write on the writer based on your validation.

https://docs-stage.spring.io/spring-batch/docs/current/reference/html/processor.html#filteringRecords

Plus a chunk base job should handle splitting data in partitions in case an exception occurs only a chunk of data will be affected.

jcccoquia
  • 1
  • 3