spring batch : IncorrectTokenCountException

Question

I have flatfile, .csv file It has 5 five fields. if all the five fields are there, the file processing works well. but if by mistake, if 5 fields are not entered, like below

d,e,f,g,h a,b,c d,e,f,g,h

the program crashes. I want to continue reading to the next line.

The error is org.springframework.batch.item.file.transform.IncorrectTokenCountException: Incorrect number of tokens found in record

follow is the code.

How do I continue to read to the third line? I would expect first line read, second line skipped, third line read.

FlatFileItemReader<CustomerBatch> reader = new FlatFileItemReader<>();
DelimitedLineTokenizer delimitedTokeniser = new DelimitedLineTokenizer();
delimitedTokeniser.setNames(new String[]{ "a", "b","c","d","e" });
DefaultLineMapper<CustomerBatch> lineMapper = new DefaultLineMapper<CustomerBatch>();
lineMapper.setLineTokenizer(delimitedTokeniser);
lineMapper.setFieldSetMapper(new BeanWrapperFieldSetMapper<CustomerBatch>() {{setTargetType(CustomerBatch.class);}});
ByteArrayResource y= new ByteArrayResource(batchFile.getBytes());
reader.setResource(y);
reader.setLineMapper(lineMapper);
reader.setLinesToSkip(1); //do not read the first line of the csv file.
reader.open(new ExecutionContext());

try {
        while ((newCustomerBatch = reader.read()) != null) {
        System.out.println("name:"+newCustomerBatch.getName());
        //some logic.
        }
    }catch(Exception e) {

    }
}

Check https://stackoverflow.com/questions/34435817/how-to-handle-the-exceptions-thrown-from-item-reader?noredirect=1&lq=1 — Luca Basso Ricci, May 07 '18 at 06:44

score 0 · Answer 1 · answered May 07 '18 at 05:15

You have a feature called skipPolicy which handles how many ParseException you are willing to tolerate until you fail the batch.

You should implement the SkipPloicy interface:

@Override
    public boolean shouldSkip(Throwable exception, int skipCount) throws SkipLimitExceededException {
        if (exception instanceof FileNotFoundException) {
            return false;
        } else if (exception instanceof FlatFileParseException && skipCount <= 5) {
            FlatFileParseException ffpe = (FlatFileParseException) exception;
            StringBuilder errorMessage = new StringBuilder();
            errorMessage.append("An error occured while processing the " + ffpe.getLineNumber()
                    + " line of the file. Below was the faulty " + "input.\n");
            errorMessage.append(ffpe.getInput() + "\n");
            logger.error("{}", errorMessage.toString());
            return true;
        } else {
            return false;
        }
    }

More about skipping exceptions can be found here

thanks! how would i integrate this into my above code? – Jacel May 07 '18 at 05:23 — Jacel, May 07 '18 at 05:23

spring batch : IncorrectTokenCountException

1 Answers1