How to read from specific Kafka partition in Spark structured streaming

Question

I have three partitions for my Kafka topic and I was wondering if I could read from just one partition out of three. My consumer is spark structured streaming application.

Below is my existing kafka settings in spark.

  val inputDf = spark.readStream
  .format("kafka")
  .option("kafka.bootstrap.servers", brokers)
  .option("subscribe", topic)
  .option("startingOffsets", "latest")
  .load()

score 5 · Accepted Answer · answered Feb 15 '19 at 07:06

Here is how you can read from specific partition.

 val inputDf = spark.readStream
  .format("kafka")
  .option("kafka.bootstrap.servers", brokers)
  .option("assign", """{"topic":[0]}""") 
  .option("startingOffsets", "latest")
  .load()

PS: To read from multiple partitions instead of 1--> """{"topic":[0,1,2..n]}"""

score 1 · Answer 2 · answered Jun 15 '20 at 23:26

Similarly, How do you write to a specific partition. I tried this and it doesn't work.

        someDF
          .selectExpr("key", "value")
          .writeStream
          .format("kafka")
          .option("kafka.bootstrap.servers", kafkaServers)
          .option("topic", "someTopic")
          .option("partition", partIdx)
          .start()

How to read from specific Kafka partition in Spark structured streaming

2 Answers2