1

I have an application on Spring batch, I have used Chunk Oriented programming, in that I have extended FlatFileItemWriter class to generate the output file.

The problem is I will be having more than 1000 records as input and the chunk value is 100. So 10 times the customFlatFileItemWriter will get called. But the problem is every record is interlinked, I should check previous id with the next record id.

Input to the batch

123, sample1
123, sample2 // Here I have to check 123 is same as 123 which is previous id
124, sample3 // Here I have to check 124 is same as 123 which is previous id
125, sample4 // Here I have to check 125 is same as 124 which is previous id

To do that if chunk completes at id - 124, then we can't get the record id '124' while doing the manipulation of record id '125'.

In stepExecution we can keep the object in Execution and carry to the next task, but here in this scenario we don't get the stepExecution, so is there anyway to get the object to the next chunk manipulation.

Code of my custom FlatFileItemWriter

public class FileIW<T> extends FlatFileItemWriter<T> {

    @Override
    public String doWrite(List<? extends T> items) {
        StringBuilder lines = new StringBuilder();
        for (T item : items) {
            User user = (User)item;
            System.out.println(user.getId());
            lines.append(this.lineAggregator.aggregate(item)).append(this.lineSeparator);
        }
        return lines.toString();
    }

}
Yaswanth
  • 483
  • 1
  • 9
  • 25
  • `To do that if chunk completes at id - 124, then we can't get the record id '124' while doing the manipulation of record id '125'.`: Where are you doing this manipulation? Do you have an item processor? How are you dealing with this situation inside the *same* chunk (and not between chunks)? – Mahmoud Ben Hassine Nov 16 '20 at 11:49
  • Hi Mahmoud Ben Hassine, Actually I don't have item processor, basically we can do this operation either in processor or else in writer. But not sure how to store the I'd temporarily when the execution is happening between the chunks. Please let me know if you have any ideas. – Yaswanth Nov 17 '20 at 17:53
  • You should do it in an item processor, data manipulation/transformation is a typical use case for an item processor. What you can try to do is store items (or only their IDs) in the execution context so you can get access to them in subsequent chunks. – Mahmoud Ben Hassine Nov 18 '20 at 08:56
  • @MahmoudBenHassine Can we store bulk data in execution context? Is it recommended to do so? – DK93 Feb 25 '22 at 02:24

0 Answers0