0

I have the following ItemProcessor, I need to know if the item is the last item the reader sends. Is this possible?

The flow is:

  • ItemReader - read lines from database (expected millions of lines)
  • ItemProcessor - perform some validations on each line, send to webservice and perform more validations with webservice response.
  • ItemWriter - write in database errors from validations, and change line status.

Another possible solution would be the ItemReader to send a list of Line with size NUMBER_OF_LINES_TO_CALL_WEBSERVICE, but I could not find examples on how to modify the JpaPagingItemReader.


    @Override
    public List<Line> process(Line line) {
        

        lineList.add(line);

        if (lineList.size() == NUMBER_OF_LINES_TO_CALL_WEBSERVICE) {

            callWs(lineList);
            
            return lineList;    // goes to ItemWriter to write the response from ws in database 
        }
        
        if( /* TODO if last line */) {      
            
            callWs(lineList);

            return lineList;
        }
        

        return null;
    }
    
    private void callWs(List<Line> lineList){
        ...
        //Get response from webservice and add info to lines
    }

    @Override
    public void afterChunk(ChunkContext chunkContext) {

        if (lineList.size() == NUMBER_OF_LINES_TO_CALL_WEBSERVICE) {
            lineList.clear();
        }

    }

The reader is a JpaPagingItemReader:


    @Bean
    @StepScope
    public JpaPagingItemReader<Line> itemReader(@Qualifier("EntityManagerFactory") EntityManagerFactory entityManagerFactory) {


        String queryStr = "select l from Line l" ;

        return new JpaPagingItemReaderBuilder<Linha>()
                .name("lineReader")
                .entityManagerFactory(entityManagerFactory)
                .queryString(queryStr)
                .pageSize(PAGE_SIZE)
                .build();
    }
rMonteiro
  • 1,371
  • 1
  • 14
  • 37
  • Why do you need that? Can you describe what you are trying to achieve without referring to Spring Batch? What is the input/output of your step/job? There is a way to do that but this depends on the answers to those questions. – Mahmoud Ben Hassine May 26 '21 at 07:34
  • I want to group lines in a list of 100 elements to send to a webservice, and when the last item is sent to processor the remaining lines are sent to the webservice. – rMonteiro May 26 '21 at 08:46

1 Answers1

1

I want to group lines in a list of 100 elements to send to a webservice, and when the last item is sent to processor the remaining lines are sent to the webservice.

You can use a chunk-oriented step with a chunk size of 100. The step will do the buffering for you, and the first point in the process where you have access to the list of items is in the ItemWriteListener#beforeWrite(List items). So you can either use that listener to perform you web service call, or do it in the ItemWriter itself (if this is the only write operation you need to perform).

Once you have the list of items, you can do whatever you want with it (ie check the last item, filter the first 99 items and send post them to the web service, etc).

Mahmoud Ben Hassine
  • 28,519
  • 3
  • 32
  • 50
  • It is not the only `write` operation. The `ItemWriter` will write errors in database. Maybe the best approach would be to change the `ItemReader` to send a list of 100 elements, would you agree? But with this approach I need help to modify the `ItemReader`, maybe I can create a specific question for this. – rMonteiro May 26 '21 at 09:18
  • 1
    In this case, you can use the `ItemWriteListener#beforeWrite` to perform your web service call and an `ItemWriter` to write to the database. I see no need to modify the reader, you should leverage the chunking/buffering of items provided by the step implementation. – Mahmoud Ben Hassine May 26 '21 at 09:38
  • Thanks, I didn't know about `ItemWriteListener`. This approach looks good – rMonteiro May 26 '21 at 11:29