0

I have a kafka consumer which does the deserialization of JSON.

I get big messages like :

{
   data1: [],
   data2: ["content"],
   data3: null,
   data4: [],
   data5: "foo",
   data6: "",
   data7: [""],
   data8: [null],
   ...
   dataX: []
}

with a lot of keys, where the values are null respective empty lists. I want handle only the keys, which does have values or lists with senseful content like data2 or data5 and ignore the others. Even with data6 (empty string) and data7 (list with empty string) I want stay. This shall pass.

I have this kafka consumer configuration.

    private void configureAndInitConsumer() {
        final Properties props = new Properties();
        props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, bootstrapServers);
        props.put(ConsumerConfig.GROUP_ID_CONFIG, consumerGroupId);
        props.put(ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG, autoCommit);
        props.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class.getName());
        props.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, org.apache.kafka.connect.json.JsonDeserializer.class.getName());
        props.put(ConsumerConfig.MAX_POLL_RECORDS_CONFIG, maxPollRecords);
        consumer = new KafkaConsumer<>(props);
   }

I see, that here the org.apache.kafka.connect.json.JsonDeserializer is configured. But the config is for kafka consumer.

I can configure the deserializer by replacing the KafkaConsumer constructor.

JsonDeserializer<JsonNode> valueDeserializer = createCustomizedJsonDeserializer();
consumer = new KafkaConsumer<>(props, new StringDeserializer(), valueDeserializer);

and then I add the method

private JsonDeserializer<JsonNode> createCustomizedJsonDeserializer() {
        ObjectMapper objectMapper = new ObjectMapper();
        objectMapper.configure(MapperFeature.DEFAULT_VIEW_INCLUSION, false);
        objectMapper.configure(DeserializationFeature.FAIL_ON_UNKNOWN_PROPERTIES, false);
        objectMapper.configure(DeserializationFeature.ACCEPT_SINGLE_VALUE_AS_ARRAY, false);
        return new JsonDeserializer<>(JsonNode.class, objectMapper);
    }

The hint for this I got from https://github.com/spring-projects/spring-kafka/issues/915

Is is possible that the Deserializer ignores the keys without values? (empty lists or null values)

I realized that the Deserializer intern has an object mapper. So I testest this:

public class ObjectMapperTest {

    public static void main(String[] args) throws JsonMappingException, JsonProcessingException {
        System.exit(new ObjectMapperTest().start());

    }

    private int start() throws JsonMappingException, JsonProcessingException {
        String testJson = "{\"date1\":[],\"date2\":null,\"date3\":[null],\"date4\":[\"\"],\"date5\":[],\"date6\":\"\",\"date8\":\"foobar\"}";

        ObjectMapper objectMapper = new ObjectMapper();
        objectMapper.disable(DeserializationFeature.ACCEPT_EMPTY_STRING_AS_NULL_OBJECT);

        JsonNode jn = objectMapper.readTree(testJson);

        System.out.println(jn);
        return 0;
    }
}

I get this result:

{"date1":[],"date2":null,"date3":[null],"date4":[""],"date5":[],"date6":"","date8":"foobar"}
  • I see there, that date2 is still there.
Maik
  • 310
  • 1
  • 11
  • You don't need a custom deserializer. You can use the StringDeserializer along with the ObjectMapper, which has an ignore null configuration, I think... That being said, Kafka is an implementation detail, and you should refer the Jackson documentation. Btw, the only key "without a value" is data3 – OneCricketeer May 25 '21 at 12:10
  • You are right. The only key wiothout a value is data3. But I want also ignore empty Arrays. I made some tests with objectMapper. Even the objectmapper I can't configure so that it ignores keys without values. The null configuration only works for serialization. Sadly not for deserialization. – Maik May 25 '21 at 12:25
  • Have you tried storing into a Hashmap/Treemap instead of JsonNode? Or will that fail since can't add null values into some maps? – OneCricketeer May 25 '21 at 12:38
  • final HashMap> map = objectMapper.treeToValue(jsonDocNode, HashMap.class); This was my origin. This made the initial error: He maps keys with null and empty strings to the same. I need to differentiate between null and empty string (""). I asked therefore this: https://stackoverflow.com/questions/67639381/jackson-objectmapper-treetovalue-maps-empty-string-to-null-instead-of-empty-string What do you mean in your former comment with "ignore null configuration". Maybe we mean different things. – Maik May 25 '21 at 12:45
  • 1
    Not all your values are lists. You'd only be able to use ``. Null and empty string **are** different – OneCricketeer May 25 '21 at 12:47
  • Right. Before there was only lists accepted. This was working. I added now the possibility also to accept leafs only with my node traverse solution. Yes, my example coveres more. But even we I look only to the arrays I had the issue. – Maik May 25 '21 at 12:49
  • Maybe I am only thinking of `Include.NON_NULL` serializer features. AFAIK, deserializer needs to read every field, then can ignore certain ones when converting to a POJO, but not a generic container like a map or JsonNode/Object/Tree – OneCricketeer May 25 '21 at 13:03
  • 1
    @OneCricketeer: You are right. My issue was the value type inside HashMap. Changing to ```final HashMap map = objectMapper.treeToValue(jn, HashMap.class);``` at least lets ```treeValue``` work as expected. – Maik Jun 08 '21 at 06:27

0 Answers0