jackson-dataformat-csv does not ignore unknown properties

Question

Trying to parse .csv file with jackson-dataformat-csv. File contains a lot of columns not relevant for my program.

Tried to use @JsonIgnoreProperties(ignoreUnknown = true) on my data class, and csvMapper.disable(DeserializationFeature.FAIL_ON_UNKNOWN_PROPERTIES), but neither works, and application throws exception:

com.fasterxml.jackson.databind.RuntimeJsonMappingException: Too many entries: expected at most 2 (value #2 (17 chars) "policy_issue_date")
 at [Source: (com.fasterxml.jackson.dataformat.csv.impl.UTF8Reader); line: 1, column: 37]

    at com.fasterxml.jackson.databind.MappingIterator.next(MappingIterator.java:194)
    at pl.polins.readers.oc.OcPolicyCsvReader.readNext(OcPolicyCsvReader.kt:25)
    at pl.polins.readers.oc.OcPolicyCsvReaderTest.should read PolicyCsv from .csv file(OcPolicyCsvReaderTest.groovy:19)
Caused by: com.fasterxml.jackson.dataformat.csv.CsvMappingException: Too many entries: expected at most 2 (value #2 (17 chars) "policy_issue_date")
 at [Source: (com.fasterxml.jackson.dataformat.csv.impl.UTF8Reader); line: 1, column: 37]
    at com.fasterxml.jackson.dataformat.csv.CsvMappingException.from(CsvMappingException.java:23)
    at com.fasterxml.jackson.dataformat.csv.CsvParser._reportCsvMappingError(CsvParser.java:1210)
    at com.fasterxml.jackson.dataformat.csv.CsvParser._handleExtraColumn(CsvParser.java:965)
    at com.fasterxml.jackson.dataformat.csv.CsvParser._handleNextEntry(CsvParser.java:826)
    at com.fasterxml.jackson.dataformat.csv.CsvParser.nextToken(CsvParser.java:580)
    at com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeUsingPropertyBased(BeanDeserializer.java:418)
    at com.fasterxml.jackson.databind.deser.BeanDeserializerBase.deserializeFromObjectUsingNonDefault(BeanDeserializerBase.java:1266)
    at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserializeFromObject(BeanDeserializer.java:325)
    at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:159)
    at com.fasterxml.jackson.databind.MappingIterator.nextValue(MappingIterator.java:277)
    at com.fasterxml.jackson.databind.MappingIterator.next(MappingIterator.java:192)
    ... 2 more

Is there any solution to ignore unwanted columns in csv?

score 12 · Accepted Answer · answered May 03 '17 at 14:43

12

Found solution:

csvMapper.enable(CsvParser.Feature.IGNORE_TRAILING_UNMAPPABLE)

answered May 03 '17 at 14:43

mbaranauskas

435
1
4
8

1

`.enable(JsonGenerator.Feature.IGNORE_UNKNOWN)` helped me with similar issues when trying to write from a class that had more fields than the schema I was using – Tom Power Jan 25 '18 at 11:15
1

As the constant name already states this would ignore only **trailing** columns. But unknown columns can appear at any position. `csvMapper.disable(DeserializationFeature.FAIL_ON_UNKNOWN_PROPERTIES);` works for me using jackson-csv 2.11.4. Please verify if your solution is correct. – bjmi Aug 21 '21 at 05:26

score 6 · Answer 2 · answered Jun 12 '19 at 03:30

6

This worked for me:

csvMapper.disable(DeserializationFeature.FAIL_ON_UNKNOWN_PROPERTIES);

answered Jun 12 '19 at 03:30

Caesar

1,092
12
19

KeyMaker00 · Answer 3 · 2020-05-20T13:46:39.907

Introduction

For the sake of comprehensibility, here is a simple (Java) example that:

read a CSV-file from an InputStream (jackson-dataformat-csv dependency)
map its content to a list of objects (jackson-core dependency)

CSV file content

Let be data.csv a CSV file with the following data:

a;b;c
1;2;0.5
3;4;

Java Class Data with a missing attributes

The class MyModel represents a data class with the following attributes:

private Long a;
private Integer b;

Note the attribute c is missing, hence the parser will have to ignore it.

Read the CSV content and map into a list of objects

So the CsvMapper can be coupled with the Jackson object mapper to read records from a CSV file to a list of objects, the whole in a convenient method:

<U> List<U> mapRecordsToObjects(InputStream inputStream, Class<U> encodingType) {
    CsvMapper csvMapper = new CsvMapper();
    CsvSchema bootstrapSchema = CsvSchema.emptySchema() //
                                         .withHeader() //
                                         .withColumnSeparator(";");
    ObjectReader reader = csvMapper.disable(DeserializationFeature.FAIL_ON_UNKNOWN_PROPERTIES) //
                                   .readerFor(encodingType) //
                                   .with(bootstrapSchema);
    MappingIterator<U> iterator;
    try {
        iterator = reader.readValues(inputStream);
    } catch (IOException e) {
        throw new IllegalStateException(String.format("could not access file [%s]", this.source), e);
    }
    List<U> results = new ArrayList<>();
    iterator.forEachRemaining(results::add);
    return results;
}

Finally, let's call this method:

List<MyModel> result = mapRecordsToObjects(fileInputStream, MyModel.class);

In order to read the file, you will just need to initialise the InputStream.

Deserialization feature

In the documentation, the class DeserializationFeature has the following description:

Enumeration that defines simple on/off features that affect the way Java objects are deserialized from JSON

In this enumeration class, there are many feature with a default state (sometimes per default enabled, sometimes disabled). The feature FAIL_ON_UNKOWN_PROPERTIES is disable per default can can be enabled as shown in the example. In its description, one can read:

Feature that determines whether encountering of unknown properties (ones that do not map to a property, and there is no "any setter" or handler that can handle it) should result in a failure (by throwing a {@link JsonMappingException}) or not.

Just on a side note, there is a also `List MappingIterator#readAll()`, so you could'have amended `results::add` and simply write `List results = iterator.readAll()`. Not sure if there are any complications, though. — Xobotun, Nov 28 '20 at 10:40

jackson-dataformat-csv does not ignore unknown properties

3 Answers3