How to know which chunk has failed in Spring Batch and assign Counter value?

Question

I am extending this question here - Identify which chunk has failed in chunk based step in Spring Batch.

Can you show me code to have below?

How to get to know which chunk has failed ?
How to make a counter and assign autoincremented values to one field which is not PK and save to DB?

A chunk won't fail just 1 or more items of that chunk. To determine which items failed you can use a `SkipListener`. As already answered in that other question. — M. Deinum, Feb 11 '21 at 08:41
@M.Deinum - I've implemented SkipListener but I need to get which chunk has failed ... how can we get that? I dont need item here — PAA, Feb 11 '21 at 08:47

score 0 · Accepted Answer · answered Feb 11 '21 at 10:55

In a non fault-tolerant step, the first error in any chunk will fail the step and you could get the chunk number with a counter in your ChunkListener implementation as mentioned previously. Here is a quick example:

import java.util.Arrays;

import org.springframework.batch.core.Job;
import org.springframework.batch.core.JobParameters;
import org.springframework.batch.core.configuration.annotation.EnableBatchProcessing;
import org.springframework.batch.core.configuration.annotation.JobBuilderFactory;
import org.springframework.batch.core.configuration.annotation.StepBuilderFactory;
import org.springframework.batch.core.launch.JobLauncher;
import org.springframework.batch.core.listener.ChunkListenerSupport;
import org.springframework.batch.core.scope.context.ChunkContext;
import org.springframework.batch.item.support.ListItemReader;
import org.springframework.context.ApplicationContext;
import org.springframework.context.annotation.AnnotationConfigApplicationContext;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;

@Configuration
@EnableBatchProcessing
public class MyJobConfiguration {

    @Bean
    public Job job(JobBuilderFactory jobs, StepBuilderFactory steps) {
        return jobs.get("job")
                .start(steps.get("step")
                        .<Integer, Integer>chunk(5)
                        .reader(new ListItemReader<>(Arrays.asList(1, 2, 3, 4, 5, 6, 7, 8, 9, 10)))
                        .writer(items -> {
                            System.out.println("About to write items " + items);
                            if (items.contains(8)) {
                                throw new Exception("No 8 here!");
                            }
                        })
                        .listener(new MyChunkListener())
                        .build())
                .build();
    }

    static class MyChunkListener extends ChunkListenerSupport {
        private int counter;

        @Override
        public void beforeChunk(ChunkContext context) {
            counter++;
        }

        @Override
        public void afterChunkError(ChunkContext context) {
            System.out.println("Chunk number " + counter + " failed");
        }
    }

    public static void main(String[] args) throws Exception {
        ApplicationContext context = new AnnotationConfigApplicationContext(MyJobConfiguration.class);
        JobLauncher jobLauncher = context.getBean(JobLauncher.class);
        Job job = context.getBean(Job.class);
        jobLauncher.run(job, new JobParameters());
    }

}

This prints:

About to write items [1, 2, 3, 4, 5]
About to write items [6, 7, 8, 9, 10]
Chunk number 2 failed

However, in a fault-tolerant step, items will be retried one by one and Spring Batch will create single-item chunks. In this case, the ChunkListener will be called for each one of those single-item chunks, so the counter should be correctly interpreted. Here is a fault-tolerant version of the previous example:

import java.util.Arrays;

import org.springframework.batch.core.Job;
import org.springframework.batch.core.JobParameters;
import org.springframework.batch.core.configuration.annotation.EnableBatchProcessing;
import org.springframework.batch.core.configuration.annotation.JobBuilderFactory;
import org.springframework.batch.core.configuration.annotation.StepBuilderFactory;
import org.springframework.batch.core.launch.JobLauncher;
import org.springframework.batch.core.listener.ChunkListenerSupport;
import org.springframework.batch.core.scope.context.ChunkContext;
import org.springframework.batch.item.support.ListItemReader;
import org.springframework.context.ApplicationContext;
import org.springframework.context.annotation.AnnotationConfigApplicationContext;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;

@Configuration
@EnableBatchProcessing
public class MyJobConfiguration {

    @Bean
    public Job job(JobBuilderFactory jobs, StepBuilderFactory steps) {
        return jobs.get("job")
                .start(steps.get("step")
                        .<Integer, Integer>chunk(5)
                        .faultTolerant()
                        .skip(Exception.class)
                        .skipLimit(10)
                        .reader(new ListItemReader<>(Arrays.asList(1, 2, 3, 4, 5, 6, 7, 8, 9, 10)))
                        .writer(items -> {
                            System.out.println("About to write items " + items);
                            if (items.contains(8)) {
                                throw new Exception("No 8 here!");
                            }
                        })
                        .listener(new MyChunkListener())
                        .build())
                .build();
    }

    static class MyChunkListener extends ChunkListenerSupport {
        private int counter;

        @Override
        public void beforeChunk(ChunkContext context) {
            counter++;
        }

        @Override
        public void afterChunkError(ChunkContext context) {
            System.out.println("Chunk number " + counter + " failed");
        }
    }

    public static void main(String[] args) throws Exception {
        ApplicationContext context = new AnnotationConfigApplicationContext(MyJobConfiguration.class);
        JobLauncher jobLauncher = context.getBean(JobLauncher.class);
        Job job = context.getBean(Job.class);
        jobLauncher.run(job, new JobParameters());
    }

}

which prints:

About to write items [1, 2, 3, 4, 5]
About to write items [6, 7, 8, 9, 10]
Chunk number 2 failed
About to write items [6]
About to write items [7]
About to write items [8]
Chunk number 5 failed
About to write items [9]
About to write items [10]

Thanks, Can you help for this as well - "How to make a counter and assign autoincremented values to one field which is not PK and save to DB?" — PAA, Feb 11 '21 at 11:08
What do you mean by that? Do you want to save the counter value in DB? Please elaborate on what you want exactly. — Mahmoud Ben Hassine, Feb 11 '21 at 11:38
If I am reading 1000 records from DB , then I need to set autoincremented value to src_row_num field which is field from say Employee, Employee1 should have src_row_num=1, Employee2 should have src_row_num=2, Employee3 should have src_row_num=3 and so on. — PAA, Feb 11 '21 at 13:48
How is this related to your initial question about how to identify which chunk has failed? What you are describing here is assigning the ROW_ID for the current item. That's a different story and can be done with a `ItemReadListener#afterRead` in which you would assign an auto-incremented value to the current item. — Mahmoud Ben Hassine, Feb 11 '21 at 14:47
Can you please guide here: https://stackoverflow.com/questions/60925289/hikaripool-1-connection-marked-as-broken-because-of-sqlstate08006-errorcode? — PAA, Feb 02 '22 at 17:50

How to know which chunk has failed in Spring Batch and assign Counter value?

1 Answers1