How do I access header information in kafka materialized views?

Question

how do I access custom headers in a materialized view? I'm trying to build some custom dlq logic in my application and want to build a retry mechanism based on header information. The actual retry is triggered by a scheduler which should look up these header information in a materialized view.

Here are some code snippets:

Create Materialized View:

@Slf4j
@EnableBinding(DlqBinding.class)
public class DlqRetryService {

    @StreamListener
    public void readTable(@Input(DlqBinding.DLQ_TOPIC) KTable<String, String> table) {
    }
}

public interface DlqBinding {

    String DLQ_TOPIC = "dlq";

    @Input(DLQ_TOPIC)
    KTable<?, ?> dlqInput();
}

spring:
  cloud:
    stream:
      kafka:
        streams:
          binder:
            brokers: localhost:29092
            configuration:
              default:
                key.serde: org.apache.kafka.common.serialization.Serdes$StringSerde
                value.serde: org.apache.kafka.common.serialization.Serdes$StringSerde
          bindings:
            dlq:
              consumer:
                materializedAs: currentDL
                keySerde: org.apache.kafka.common.serialization.Serdes$StringSerde
                valueSerde: org.apache.kafka.common.serialization.Serdes$StringSerde

Scheduler:

    public void processDL() {
        ReadOnlyKeyValueStore<Object, Object> currentDL = interactiveQueryService.getQueryableStore("currentDL", QueryableStoreTypes.keyValueStore());

        KeyValueIterator<Object, Object> all = currentDL.all();

        while (all.hasNext()) {
            KeyValue<Object, Object> next = all.next();
            log.info("Found Entry in currentDL: {}", next);
            // some retry logic would be here
        }
    }```

score 0 · Answer 1 · answered Mar 30 '20 at 21:45

I don't think it is possible to access headers that way from the materialized view using interactive query.

Its not clear what you are retrying. Are you trying to re-process the records? You can encapsulate this kind of logic by using a KStream and the transformer/processor API. Here is a blueprint of such a pattern. Keep in mind that this is a general way of accessing headers. Your application might need to adjust based on your particular use case.

@StreamListener
public void processStream(@Input(DlqBinding.DLQ_TOPIC) KStream<String, String> stream) {

stream.process(() -> new Processor() {

            ProcessorContext context;

            @Override
            public void init(ProcessorContext context) {
                this.context = context;
            }

            @Override
            public void process(Object key, Object value) {
                final Headers headers = this.context.headers();
                final Iterable<Header> headerXyz = headers
                        .headers("HEADER_KEY");
                // iterate on the header information returned and perform 
                // your application specific logic here.
            }

            @Override
            public void close() {

            }
        });

}

I'm trying to re-process Records which were sent to the dlq topic. My Goal is to implement something like what is already present with the RetryTemplate, but with larger ever increasing retry intervals (1min, 2 min, 5 min, ..., 1day). I had hoped this would already be present in the framework. — Fabian Lang, Mar 31 '20 at 07:46
Ya, unfortunately, materialized views won't give you access to the original headers. If you can share a small contrived application with the problem you are facing, we can potentially suggest some workarounds maybe. — sobychacko, Mar 31 '20 at 14:21

How do I access header information in kafka materialized views?

1 Answers1