9

I created a job which splits my file into small chunks and all these chunks are read in the separated steps. For ex. 3 steps are finished without any errors and the records are commited to the database, but if 4th step fails I need to rollback all records from previous step. Is it possible to rollback them somehow?

Or perhaps there is a possibility to commit all records only when the last step was finished correctly? (But here is problem with large files)

Shendor
  • 749
  • 2
  • 11
  • 18
  • 1
    you can also disable the auto-commit property of the datasource by default. then, commit when all your steps are successfully completed. – Keerthivasan Nov 15 '13 at 14:23
  • 2
    Without committing/flushing to the database periodically, you will always be at risk of running out of memory with large files since the application needs to hold all of the objects in the persistence context until they are flushed/cleared. Why do you need to rollback all steps? What kind of errors are you expecting that would cause the 4th step to fail? – FGreg Nov 15 '13 at 14:29
  • I wrote that I load the large file and split it to small chunks (10000 records for each batch step) and I need to rollback all steps because these steps save records of the file and I need to commit all these records of the file only if all of them were read without any errors (for ex. parse errors) – Shendor Nov 15 '13 at 14:45
  • Yes, but why are ALL of the records invalid if only 1 is formatted incorrectly? Would it not make sense to skip the bad record and allow the others to complete? If it really is all or nothing, I think you need to consider a different design where the records go into some sort of 'pending' state before being transferred to the 'active' table. – FGreg Nov 15 '13 at 14:49
  • It's just a business rule - all records must be valid and decimal values can be parsed – Shendor Nov 15 '13 at 14:54
  • If that's the case, I think you need to validate all records before inserting them instead of validating and inserting at the same time. – FGreg Nov 15 '13 at 14:58
  • Yeah, looks like I have to validate file before inserting the records. Thought that it was possible to apply transaction on the entire job =( – Shendor Nov 15 '13 at 15:01
  • I also want similar behavior, in my case, I need to import XML with XK users and results must be ALL or NOTHING, can I achieve this with Spring Batch? – Ketan Dec 02 '14 at 13:04

2 Answers2

6

Don't play with transaction while using spring batch; due to its transactional nature is a really bad idea manually manage transaction.
See Transaction Management in Spring batch or Spring batch - One transaction over whole Job for further explanation

Community
  • 1
  • 1
Luca Basso Ricci
  • 17,829
  • 2
  • 47
  • 69
  • 3
    Agreed, Spring Batch is designed so that steps incrementally commit their work. This is why it's possible to restart failed jobs. Messing with this would be forcing Spring Batch to do something it is not designed to do and should be avoided if at all possible. – FGreg Nov 15 '13 at 14:54
  • So, in other words if one of the steps fails I have to remove the previos records from database manually? – Shendor Nov 15 '13 at 14:59
  • Ok, looks like transaction cannot be applied to the entire job. Thanks for your suggestions. – Shendor Nov 15 '13 at 15:02
  • if you can't re-think about your process flow, yes. – Luca Basso Ricci Nov 15 '13 at 15:02
0

Not just spring, for any framework if you need to perform atomic operation across multiple read/write to data source, generally all those calls need to be wrapped in a transaction and then either committed or rolled back at the end. Understanding how JTA works goes a long way in identifying how to use framework that handle transactions, more information on JTA can be found here

user1339772
  • 783
  • 5
  • 19
  • Yeah, I know it, but the problem here, that if my file has 1kk records I can catch OutOfMemoryError. I need somehow to perform commit by small chunks, not all records in one commit operation. And before I do these commits, I need to make shure that all records of the file from all steps were read correctly. And here is a problem. – Shendor Nov 15 '13 at 14:46