1

I am using scala, spark, and Kafka. I have 2 questions.

1.how can I confirm the topic exists in Kafka broker(server)?

2.how can I confirm the Kafka server (bootstrap server) is running or not?

object kafkaProducer extends App {

  def sendMessages(): Unit = {


//define topic
val topic = "spark-topic"       // how can i confirm this topic is exist in kafka server ? 

//define producer properties
val props = new java.util.Properties()
props.put("bootstrap.servers", "localhost:9092")
props.put("client.id", "KafkaProducer")
props.put("key.serializer", "org.apache.kafka.common.serialization.StringSerializer")
props.put("value.serializer", "org.apache.kafka.connect.json.JsonSerializer")

//create producer instance
val kafkaProducer = new KafkaProducer[String, JsonNode](props)

//create object mapper
val mapper = new ObjectMapper with ScalaObjectMapper
mapper.registerModule(DefaultScalaModule)
mapper.configure(DeserializationFeature.FAIL_ON_UNKNOWN_PROPERTIES, false)

//mapper Json object to string
      def toJson(value: Any): String = {
        mapper.writeValueAsString(value)
      }


//send producer message

    val jsonstring =
      s"""{
         | "id": "0001",
         | "name": "Peter"
         |}
      """.stripMargin

    val jsonNode: JsonNode = mapper.readTree(jsonstring)
    val rec = new ProducerRecord[String, JsonNode](topic, jsonNode)
    kafkaProducer.send(rec)
    //println(rec)

  }

}
OneCricketeer
  • 179,855
  • 19
  • 132
  • 245
stack0114104
  • 87
  • 1
  • 11
  • Use this util class to know if topic exists https://github.com/apache/kafka/blob/0.8/core/src/main/scala/kafka/admin/ListTopicCommand.scala To know health of the brokers https://stackoverflow.com/a/29521307/3595618 – Puneeth Reddy V Dec 10 '18 at 10:26

1 Answers1

0

1) The recommended way to check if a topic exists is to use the AdminClient API.

You can use listTopics() or describeTopics().

2) Assuming you don't have any privilege access to the cluster (to check metrics or liveness probes), the only way to check the cluster is running is to try connecting to/use it.

With the AdminClient, you can use describeCluster(), for example, to attempt to retrieve the state of the cluster.

Mickael Maison
  • 25,067
  • 7
  • 71
  • 68
  • Is there any particular method is available to check whether a particular topic exists or not .. Like this ---> spark.catalog.tableExists(tablename) – stack0114104 Dec 13 '18 at 09:26
  • Unfortunately not with the public APIs, you have to use either `listTopics()` or `describeTopics()` and interpret their result and that's pretty easy. – Mickael Maison Dec 13 '18 at 09:55