Kafka stream append to JSON as event enrichment

Question

I have a producer that writes a json file to the topic to be read by a kafka consumer stream. Its simple key-value pair. I want to stream the topic and enrich the event by adding/concatenating more JSON key-value rows and publish to another topic. None of the values or keys have anything in common by the way. I am probably overthinking this, but how would I get around this logic?

Answer 1

I suppose you want to decode JSON message at the consumer side.

If you are not concerned about schema and but just want to deal with JSON as a Map, you can use Jackson library to read the JSON string as a Map<String,Object>. For this you can add the fields that you want, convert it back to a JSON string and push it to the new topic.

If you want to have a schema, you need to store the information as to which class it is mapping to or the JSON schema or some id that maps to this, then the following could work.

Store the schema info in headers

For example, you can store the JSON schema or Java class name in the headers of the message while producing and write a deserializer to extract that information from the headers and decode it.

The Deserializer#deserialize() has the Headers argument.

default T deserialize(java.lang.String topic,
                      Headers headers,
                      byte[] data)

and you can do something like..

objectMapper.readValue(data,
                new Class.forName(
                new String(headers.lastHeader("classname").value()
                ))

Use schema registry

Apart from these, there is also a schema registry from Confluent which can maintain different versions of the schema. You would need to run another process for that, though. If you are going to use this, you may want to look at the subject naming strategy and set it to RecordNameStrategy since you have multiple schemas in the same topic.