How do you handle blank non String values with a FieldSet in FieldSetMapper?

Question

I am using common FieldSetMapper logic found through searches and in examples on StackOverflow and I have run into a situation which surprised me. Either it is a feature or a bug, but I thought I would present it here for review to see how others handle it.

Using Spring Batch, I have a pipe delimited file which has string and number values which may by optional depending on position. For example:

string|string|number|number|string
string||number||string

In your field set mapper class which implements FieldSetMapper, you usually do some mapping such as:

newThingy.setString1(fieldSet.readString("string1"));
newThingy.setString2(fieldSet.readString("string2"));
newThingy.setValue1(fieldSet.readInt("value1"));
newThingy.setValue2(fieldSet.readInt("value2"));
newThingy.setString3(fieldSet.readString("string3"));

During testing the code for line 1 above worked fine. For line 2 with the blank values for string2 and value, a Java exception was thrown for the number but not the string:

Caused by: java.lang.NumberFormatException: Unparseable number: 
    at org.springframework.batch.item.file.transform.DefaultFieldSet.parseNumber(DefaultFieldSet.java:754)
    at org.springframework.batch.item.file.transform.DefaultFieldSet.readInt(DefaultFieldSet.java:323)
    at org.springframework.batch.item.file.transform.DefaultFieldSet.readInt(DefaultFieldSet.java:335)
    at com.healthcloud.batch.mapper.MemberFieldSetMapper.mapFieldSet(MemberFieldSetMapper.java:31)
    at com.healthcloud.batch.mapper.MemberFieldSetMapper.mapFieldSet(MemberFieldSetMapper.java:1)

I did some research in the DefaultFieldSetMapper.java class provided by Spring Batch which implements the FieldSet class to try and understand what is going on.

What I found is that the readAndTrim function called by readString returns null if the value read is blank

protected String readAndTrim(int index) {
    String value = tokens[index];

    if (value != null) {
        return value.trim();
    }
    else {
        return null;
    }
}

... but when using readInt (and maybe others) we are returning an exception.

private Number parseNumber(String candidate) {
    try {
        return numberFormat.parse(candidate);
    }
    catch (ParseException e) {
        throw new NumberFormatException("Unparseable number: " + candidate);
    }
}

I do see where you can return a default value in some of the methods, but null is obviously not allowed. What I would expect is consistent behavior between all methods in FieldSet implementations which allow one to match the file to my database as the data is read. Blank values in delimited and fixed length files are fairly common.

If number based values cannot be properly handled, I will probably have to convert everything over to String as it is read and then go through the trouble to manual handle the conversion to the database, which obviously defeats the purpose of using Spring Batch.

Am I missing something that I should handle better? I can add more code if needed, I just felt this is commonly used and I could keep this short. Will edit as needed.

Edit: Add info on Unit Tests found for Spring Batch class

The comments in the test case state a default should be set instead, but why? I don't want a default. My database allows a null value in the Integer column. I would have to set the default to some arbitrary number which hopefully no one EVER sends, check for it before insert and then switch to null on insert. I still don't like this "feature."

@Test
public void testReadBlankInt() {

    // Trying to parse a blank field as an integer, but without a default
    // value should throw a NumberFormatException
    try {
        fieldSet.readInt(13);
        fail();
    }
    catch (NumberFormatException ex) {
        // expected
    }

    try {
        fieldSet.readInt("BlankInput");
        fail();
    }
    catch (NumberFormatException ex) {
        // expected
    }

}

Please using NumberUtils.toInt from Apache Commons Lang. Signature is public static int NumberUtils.toInt(java.lang.String str, int defaultValue) — Nghia Do, Oct 31 '16 at 21:16
did you find a solution for the problem , I have the same issue. — Omar B., May 08 '19 at 15:49

score 0 · Answer 1 · edited Jun 15 '21 at 11:26

0

Always sanity check your input/data. I'll usually throw together a Util class with all the parse/read/verification I need. Bare bones version below...

public static Integer getInteger(FieldSet fs, String key, Integer default) {
    if(StringUtils.isNumeric(fs.readString(key))) {
        return fs.readInt(key);
    } else {
        return default;
    }
}

edited Jun 15 '21 at 11:26

chillworld

4,207
3
23
50

answered Nov 02 '16 at 02:07

Andy Sampson

271
4
7

My real goal of this post was to draw attention to the fact that readInt is handling null differently than readString and we shouldn't have to do this level of work ourselves. Also, couldn't one just use newThingy.setValue1(fieldSet.readInt("value1", DEFAULT_VALUE)); instead? – Tony Edwards Nov 03 '16 at 18:45
If readInt returned Integer instead of int, they could certainly go ahead and just return null. But it returns an int, which mean they can't return null. Hence the exception being thrown. Further, I use the method above because most of the time my default is null. Which the readInt(String, int) method can't do. – Andy Sampson Dec 06 '17 at 01:38
Looks like there are a million implementations of StringUtils. Can you share your import statement? – TheJeff Jun 03 '20 at 19:41
@TheJeff been a minute, but probably Apache Commons – Andy Sampson Jun 05 '20 at 02:48

How do you handle blank non String values with a FieldSet in FieldSetMapper?

1 Answers1