How to achieve handling multiple versions of UserDetailDto while processing it from Topic-A to Topic-B with Kafka stream using processor API.
Existing instance/replica of aggregation service should not impacted and Kubernetes upgrade scenario should also not hamper (means old version of aggregation replica service are able to handle the modified/new versioned of UserDetailDto).
For Example, modify the UserId datatype from Integer to String and remove UserPhone field from the below User detail dto
class UserDetailDto{
@JsonProperty("userID)
@NotNull(message = "UserId can not be null")
private int userID;
@JsonProperty("userPhone")
@NotNull(message = "User Phone number can not be null")
private int userPhone;
Now after update UserDetailDto, old replica/instance of aggregation service should able to handle both new or old UserdetailDto and also new replica/instance of aggregation service should able to new or old UserdetailDto.
My Processor as given below with Custom Serde UserDetailDto
public class AggregationProcessor implements Processor<String, UserDetailDto, String, UserDetailDto> {
private ProcessorContext<String, UserDetailDto> processorContext;
public AggregationProcessor() {
super();
}
@Override
public void init(ProcessorContext<String, UserDetailDto> processorContext) {
System.out.println("Inside Aggregation Processor init method.");
Objects.requireNonNull(processorContext, "Processor context should not be null or empty.");
this.processorContext = processorContext;
}
@Override
public void process(Record<String, UserDetailDto> message) {
System.out.println("Inside AggregationProcessor init method - to initialize all the resources for the data processing.");
Objects.requireNonNull(processorContext, "Processor context should not be null or empty.");
// Forwarding the message as is it without doing any modification
processorContext.forward(message);
}
@Override
public void close() {
System.out.println("Inside AggregationProcessor close method.");
}
Topology given below
Topology topology = new Topology();
// Adding the sourceNode of the application
topology = topology.addSource(Topology.AutoOffsetReset.EARLIEST,
sourceName,
new UsePartitionTimeOnInvalidTimestamp(),
KEY_SERDE.deserializer(),
USER_DETAIL_DTO.deserializer(),
sourceTopic);
// Adding the processorNode of the application
topology = topology.addProcessor(
processorName,
AggregationProcessor::new,
parentNames);
// Adding sinkNode of the application
topology = topology.addSink(sinkName,
destinationTopic,
KEY_SERDE.serializer(),
USER_DETAIL_DTO.serializer(),
parentNames);
Please provide all possible suggestions.Thanks!
