Spring batch IncorrectTokenCountException reader accept multi colomn in csv file

Question

I've never used Spring Batch before, I have a csv file which I need to read all columns with a specific model but I have many exceptions.

The csv file has 8 fields. if all the five fields are there, the file processing works well. but if by mistake, if 8 fields are not entered, like below:

c1;c2;C3;C4;C5;C6;C7;C8 // IT WORKS c1;C3;C7 // NOT WORKS : Program crashes

The error is: Caused by: org.springframework.batch.item.file.transform.IncorrectTokenCountException: Incorrect number of tokens found in record: expected 3 actual 8.

My code :

@Bean
public Step Step2(StepBuilderFactory stepBuilders) throws IOException {
    System.out.println("cecStep2");
    return stepBuilders.get("fileReject")
            .<CSCivique, String>chunk(100)
            .reader(reader())
            .processor(processor2FileReject())
            .writer(writer2())
            .build();
}

@Bean
public FlatFileItemReader<CSCivique> reader() throws IOException{

    try {


    return new FlatFileItemReaderBuilder<CSCivique>().name("personItemReader")
                                .resource(new ClassPathResource(confFile ()))
                                .delimited()
                                .delimiter(";")
                                .names(new String[] { "c1", "c2", "c3", "c4", "c5", "c6", "c7", "c8" })
                                .targetType(CSCivique.class)
                                .strict(false)
                                .build();

    }catch(Exception e) {
        System.out.println("----------- Exception reader() --------------");
        return null ;
    }


}

I want to save lines in file reject like : c1;C3;C7: please check the number of fields

Thank you.

score 3 · Answer 1 · answered Apr 13 '21 at 11:10

understand that this answer is coming 2 years late, recently faced this issue in which I wanted to support partial rows to be accepted and processed.

In Spring batch the DelimitedLineTokenizer tokenizer (And most other standard tokenizers in spring batch) extends the AbstractLineTokenizer class which has a property strict which if set too true will fail if the number of tokens dont match. Conversely if the property is set too false then it will ignore extra/missing tokens.

https://docs.spring.io/spring-batch/docs/current/api/org/springframework/batch/item/file/transform/AbstractLineTokenizer.html

This is how I configure my flat file reader in XML

<bean class="org.springframework.batch.item.file.FlatFileItemReader" id="add-item-reader" scope="step">
<property name="resource" value="${file.path}"/>
<property name="lineMapper">
    <bean class="org.springframework.batch.item.file.mapping.DefaultLineMapper">
        <property name="fieldSetMapper" ref="fset-mapper"/>
        <property name="lineTokenizer">
            <bean class="org.springframework.batch.item.file.transform.DelimitedLineTokenizer">
                <property name="delimiter" value="|"/>
                <property name="names" value="tokens...."/>
                <property name="strict" value="false"/>
            </bean>
        </property>
    </bean>
</property>

score 2 · Accepted Answer · answered Feb 25 '19 at 08:47

2

That's the normal behaviour as documented here. You configured the reader to expect 8 columns with:

.names(new String[] { "c1", "c2", "c3", "c4", "c5", "c6", "c7", "c8" })

So any line having a different number of tokens will be rejected.

What you can do is use a fault tolerant step and skip those lines with:

@Bean
public Step Step2(StepBuilderFactory stepBuilders) throws IOException {
    System.out.println("cecStep2");
    return stepBuilders.get("fileReject")
            .<CSCivique, String>chunk(100)
            .reader(reader())
            .processor(processor2FileReject())
            .writer(writer2())
            .faultTolerant()
            .skip(FlatFileParseException.class)
            .skipLimit(10)
            .build();
}

With this configuration, those lines will be skipped and your job will not fail. You can afterwards get the skip count with StepExecution#getSkipCount.

Otherwise, you can create your own LineTokenizer and tokenize the line as you need.

answered Feb 25 '19 at 08:47

Mahmoud Ben Hassine

28,519
3
32
50

Thank you Mahmoud, can I save those skipped lines in a reject file csv? have you example of code? – Monoem Youneb Feb 25 '19 at 17:07
Yes, you can do this using a `SkipListener`. Here is an example: https://stackoverflow.com/a/54648302/5019386. If this helps, please accept the answer. Hope it helps :-) – Mahmoud Ben Hassine Feb 26 '19 at 07:53
Hello Mahmoud, How can I read csv file in ArryLis and save lines skipped in csv file? Thanks. – Monoem Youneb Feb 27 '19 at 10:22
hello, put each column in a box to check its contents (null or not) and write this line in a rejection file. – Monoem Youneb Feb 27 '19 at 14:03
ok I see. You can use the [PassThroughFieldSetMapper](https://docs.spring.io/spring-batch/4.1.x/api/org/springframework/batch/item/file/mapping/PassThroughFieldSetMapper.html) which will give you the `FieldSet` (similar to JDBC `ResultSet`). The field set allows you to get the array of values with `getValues` method. To write skipped lines, you need a custom `SkipListener`, you can find an example here: https://stackoverflow.com/a/51985616/5019386 (Just replace `System.out` with writing to a file). – Mahmoud Ben Hassine Feb 27 '19 at 14:37
Okay Thaks Mahmoud :) – Monoem Youneb Feb 27 '19 at 15:01

Spring batch IncorrectTokenCountException reader accept multi colomn in csv file

2 Answers2