0

Following on from this (apologies, had a different user): Kafka Key access on Ingress of a Python Flink Stateful function

Our use case is that we make use of the Kafka headers as a means of tracing and lineage as well as required metadata. Looking at this: https://github.com/apache/flink-statefun/blob/master/statefun-flink/statefun-flink-io-bundle/src/main/java/org/apache/flink/statefun/flink/io/kafka/binders/ingress/v1/RoutableKafkaIngressDeserializer.java#L45-L61 It looks like using the standard deserializer, the headers are dropped.

Effectively what I'd want, is a way to inject my own deserializer that would return a message containing this and any other metadata from the record. I'd want to add something like the UniversalKafkaIngress so that I could configure it using a remote module.

Looking at the code, I can see that I could register a new ExtensionModule, and replace the deserializer (and create a custom kind). Is this recommended? If so - are there any docs on this (if not, how could I configure statefun to pick this up)?

Or, is there another preferred method?

Thanks again...

Richard Noble
  • 213
  • 3
  • 6

1 Answers1

0

Ah - found out where I was going wrong.

You can load a ExtensionModule using the standard module SPI process - and therefore register it as a new 'universal' ingress, so that it can be loaded remotely. I had a typo - which is why I battled.

There are a few gotchas - and I'll post a gist a little later to show how it can be done.

Richard Noble
  • 213
  • 3
  • 6