How to read DF Column of struct type and add they key value pairs to kafka headers?

Question

I have a new dataframe with 2 columns one is headers and other is a payload, facing issues in reading the headers column and assigning the values to kafka headers while publishing.

earlier the dataframe had 4 columns as

Old Df Schema :
    Id - String,
    key - String,
    ver - String,
    Payload - String

here i was able to parse this and add to headers as

    df.foreachPartition {
      partitionData: Iterator[Row] =>
      partitionData.foreach { row =>
        val kafkaProps = getKafkaConfig()
        val producer = JsonObjectProducer.getOrCreateProducer(kafkaProps)
        val data = new ProducerRecord[String, String](kafkaTopic, row.getAs[String]("Payload"))
        if(headerFlag=="True"){
          data.headers().add("Id",row.getAs[String]("Id").getBytes(StandardCharsets.UTF_8))
          data.headers().add("key",row.getAs[String]("key").getBytes(StandardCharsets.UTF_8))
          data.headers().add("ver",row.getAs[String]("ver").getBytes(StandardCharsets.UTF_8))
        }
        producer.send(data, new Callback() {
          def onCompletion(rm: RecordMetadata, e: Exception) {
            if (e != null) {
              e.printStackTrace()
            }
          }
        })
      }
    }
  }

but with this new schema i am not sure on how to access Header fields and add it to headers()

new DF Schema:
Headers (struct type)
    Id - String
    key - String
    ver - String
Payload - String

how can i add the headers from struct Headers while publishing to kafka?

Why not use Structured Streaming, which does support headers natively? — OneCricketeer, Feb 09 '23 at 21:41
Also, what exactly doesn't work with this code using `data.headers().add`? — OneCricketeer, Feb 09 '23 at 21:43
how do i access the fields and add to headers()? not able to access the struct with row — abhimanyu, Feb 10 '23 at 00:24
I dont understand your question. If `row.getAs[String]("Payload")` works, then the other fields should work too... Or, like I said, don't use RDD's, and use Structured Streaming instead where you don't use `ProducerRecord` at all — OneCricketeer, Feb 10 '23 at 20:43

How to read DF Column of struct type and add they key value pairs to kafka headers?

0 Answers0