I'm fairly new to Spring Batch and I would like to know if it is possible to pass data saved by writer of one job to next job or save in process and pass to next step.
I've found similar question here where it says you can use a JobStep to launch the second job from within the first job but I'm not clear how data can be passed by looking at the examples.
Consider the following scenario:
- First, from MongoDB I need to query list of contact details and save it to PostgresSQL. I've achieved this by writing a contact job with steps reader->processor->writer.
- Now, based on the contacts that were saved earlier, I need to query MongoDB to get list of accounts for each contacts and set the new contact ID as FK to the account and then save the processed accounts to PostgresSQL. Note: I can't do cascade save to save accounts automatically while saving contacts. So need to set contact_id in account manually.
- The saving of contact and its respective accounts should be transactional and able to re-process any failed rows.
Currently, I'm able to achieve 1&2 by writing a combined reader->processor->writer which would do the same thing by mapping both contact and account to a different Input/Output class as below:
public class AllReader implements ItemReader<InputRecord> {
@BeforeStep
public void before(StepExecution stepExecution) {
List<InputRecord> inputRecords = Lists.newArrayList();
List<Contact> contacts = mongoTemplate.findAll(Contact.class);
contacts.forEach(crmContact -> {
InputRecord inputRecord = new InputRecord();
inputRecord.setContact(crmContact);
List<Account> accounts = mongoTemplate.find(new Query().addCriteria(Criteria.where("refContactId").is(contact.getId())), Account.class);
inputRecord.setAccounts(accounts);
inputRecords.add(inputRecord);
});
inputRecordIterator = inputRecords.iterator();
}
@Override
public InputRecord read() throws Exception, UnexpectedInputException, ParseException, NonTransientResourceException {
if (inputRecordIterator != null && inputRecordIterator.hasNext()) {
return inputRecordIterator.next();
} else {
return null;
}
}
}
public class AllWriter implements ItemWriter<OutputRecord> {
@Override
public void write(List<? extends OutputRecord> outputRecords) throws Exception {
List<Account> accounts = Lists.newArrayList();
for (OutputRecord outputRecord : outputRecords) {
Person contact = contactRepository.save(outputRecord.getContact());
accounts = outputRecords.stream()
.map(outputRecord -> {
Account account = outputRecord.getAccount();
account.setContactId(contact.getId());
return account;
})
.collect(Collectors.toList());
}
accountRepository.saveAll(accounts);
}
}
But I want to accomplish something similar by chaining the jobs or steps as there are many other similar jobs/steps that needs similar type of chaining. Since the write method of ItemWriter returns void, I'm assuming if there could be way of adding multiple processors instead, where contactPorcessor would save the contacts and then passed to next step for account reader? Or any ways to chain jobs/steps/flows by passing data something like below:
@Override
public void write(List<? extends Contact> contacts) throws Exception {
contactRepository.saveAll(contacts);
}
@Bean
public Job job() {
return this.jobBuilderFactory.get("job")
.start(contactSteps())
.next(accountSteps())
.end()
.build();
}
@Bean
public Step contactSteps(ContactReader reader,
ContactWriter writer,
ContactProcessor processor) {
return stepBuilderFactory.get("step1")
.<MContact, Contact>chunk(100)
.reader(reader)
.processor(processor)
.writer(writer)
.build();
}
@Bean
public Step accountSteps(AccountReader reader,
AccountWriter writer,
AccountProcessor processor) {
return stepBuilderFactory.get("step1")
.<MAccount, Account>chunk(100)
.reader(reader(contacts))
.processor(processor)
.writer(writer)
.build();
}
Some examples with code samples would be helpful. What I want to achieve is like below:
Job:
Step1: ContactMigrationStep
-> ContactReader (reads all the contacts from MongoDB)
-> ContactProcessor
-> ContactWriter
Step 2: AccountMigrationStep
-> AccountReader (should read respective accounts for each contact from Step1 contactWriter)
-> AccountProcessor
-> AccountWriter