How to handle Web Service error responses in ItemWriter of Spring Batch?

Question

I have a spring batch job where the ItemWriter posts to a web service. I have an input file that I must transform and post to a web service, I do not have direct access to the database.

My question is, how should I handle failure responses from the web service? For example, if i am processing a record and I send it to the web service, but the web service responds, 'Failure - can't find that id' How should I record that this record failed (in a business sense) in spring batch? I was hoping to throw a skippable exception, so that in Spring Batch Admin a write skip would be recorded, but I see when I throw an error from the item writer, the chunk is automatically rolled back. And then when the chunk is reprocesses, records get sent to the web service for a second time.

What's the recommendation on this kind of scenario? Something like send the bad record to another item writer that records it in a separate place? Or is there a way to record it as write skip without rolling back? I realize this may be a larger architecture problem in our application and am open to suggestions.

thanks!

Answer 1

When an error occurs within an ItemWriter in Spring Batch, we roll back the entire chunk, set the commit-interval to 1 for the items in that chunk, and retry them one by one so we can identify the single item that caused the issue (we don't know the first time around since we are passing a list of items to the write method). While this works well for transactional ItemWriter implementations (and those we can work around like flat files), it obviously has some issues with non-transactional options. Another thing to note is that you can turn off rollback, but that really isn't your issue. It's not that you can't turn off rollback, it's that rollback doesn't do anything. Turning off rollback will still retry the previously successfully processed items. It just indicates that it's ok to write them again.

Not knowing your full use case (as well as volume), there are a couple options that come to mind:

• Set the chunk size to 1 - One of the primary drivers for chunking items is the optimization you can get on the write. Batching JDBC writes. Writing all lines to a file in a single block. However, calling a web service doesn't get that benefit if you have to call it once for each item. While you'll obviously have a higher load on the job repository's data store, if that isn't a concern, this option would allow you to use regular skip/retry logic that Spring Batch provides without the resending of previously successful messages.
• Handle the exception on your own - Since you've obviously written your own ItemWriter implementation, you can handle the exception on your own. As long as you don't throw it out of the writer, Spring Batch won't do anything.