0

How to get Kafka header fields (which were introduced in Kafka 0.11+) in Spark Structured Streaming?

I see the headers implementation is added in Spark 3.0 but not in 2.4.5. And I see by default spark-sql-kafka-0-10 is using kafka-client 2.0.

If it is not possible to read Kafka headers using Spark then can you suggest any alternative?

mazaneicha
  • 8,794
  • 4
  • 33
  • 52
Kishorekumar Yakkala
  • 311
  • 7
  • 14
  • Looks, like spark 2.x structured streaming doesn't support kafka headers features. work around should be to use the dstreams and fetch the headers using kafkarecord. – Kishorekumar Yakkala Mar 20 '20 at 04:05

1 Answers1

0

I don't found the way to do it in spark 2.X. can use Kafka connect SMT if the use case is simple

Kishorekumar Yakkala
  • 311
  • 7
  • 14