0

The below code throws below exception. Topic present in the kafka cluster. No network connectivity relate issue.

Exception:

Job aborted due to stage failure: Topic spark_poc_topic not present in metadata after 60000 ms. Caused by: TimeoutException: Topic spark_poc_topic not present in metadata after 60000 ms.

 df2.selectExpr("key", "value")
    .write
    .format("kafka")
    .option("kafka.bootstrap.servers", "xxxx.azure.confluent.cloud:9092")
    .option("kafka.security.protocol", "SASL_PLAINTEXT")
    .option("kafka.sasl.jaas.config", "kafkashaded.org.apache.kafka.common.security.plain.PlainLoginModule required username='{}' password='{}';".format("INxxxxxxxI", "n/LeO+aEJbxxxxx"))
    .option("kafka.ssl.endpoint.identification.algorithm", "https")
    .option("kafka.sasl.mechanism", "PLAIN")
    .option("topic", "topic123")
    .save()

Problem: .option("kafka.sasl.jaas.config", "kafkashaded.org.apache.kafka.common.security.plain.PlainLoginModule required username='{}' password='{}';".format("INxxxxxxxI", "n/LeO+aEJbxxxxx"))

Solution: .option("kafka.sasl.jaas.config", "kafkashaded.org.apache.kafka.common.security.plain.PlainLoginModule required username="" + confluentApiKey + ""password="" + confluentSecret + "";")

191180rk
  • 735
  • 2
  • 12
  • 37
  • Code says `topic123`... Error says `spark_poc_topic`... Which is it? Please show us that `spark_poc_topic` actually does exist in the cluster using `kafka-topics --list` – OneCricketeer Jun 15 '23 at 20:31
  • 1
    Btw, I haven't seen SASL config accepting single quotes and scala strings can be templated like `s"username=$username"` https://docs.scala-lang.org/overviews/core/string-interpolation.html – OneCricketeer Jun 15 '23 at 20:32
  • First check if the server is reachable: `%sh nc -vz xxxx.azure.confluent.cloud 9092` - you may have firewall in-place that limits a traffic – Alex Ott Jun 16 '23 at 06:16
  • Issue resolved, updated the post with problem & solution – 191180rk Jun 16 '23 at 07:12

0 Answers0