0

I'm implementing a service which sends messages to a downstream service via a Kafka topic. This will only happen when my service's API is called, which is likely to be infrequently, at least at first.

I've found that the reactive Kafka producer API connects to Kafka lazily, which I'm sure is great for most use-cases, but I'd like to know that my Kafka connection configuration is correct at the point I deploy my service. I don't want to have to wait for the first API call only to find out that something somewhere is wrong.

The solution I've got for this at the moment is simply to send an initialisation message to the topic at start-up, but this feels clunky. Is there anything better I can be doing to force an initial connection to the topic, or at least to validate the connection configuration?

val response = kafkaSender.send(Mono.just(initialise))
        .next()
        .block();

if (response == null) {
    throw new RuntimeException("empty Mono from Kafka initialisation");
}

if (response.exception() != null) {
    throw propagate(response.exception());
}
AndyB
  • 453
  • 1
  • 7
  • 16

1 Answers1

1

You can use AdminClient API to describe the cluster or the topic you're going to use if you prefer not to be sending heartbeats via a producer

OneCricketeer
  • 179,855
  • 19
  • 132
  • 245
  • Thank you, I don't know how I didn't find that API when I was looking initially. – AndyB Nov 06 '20 at 10:04
  • does this work if i send a message on a different topic than the one it needs the connection on? happens underneath the hood - do AdminClients and KafkaTemplates share connections? I guess to sum up - how can i implement this and still use KafkaTemplate (already use it everywhere) – Dave Ankin Mar 11 '21 at 14:37
  • @Dave I'm not very familiar with Spring, but I believe the AdminClient establishes a separate connection (although some of the properties can be refactored to overlap) and using another topic doesn't really help establish health/existence of the topic you really care about – OneCricketeer Mar 13 '21 at 01:19
  • I've confirmed that it does :) – Dave Ankin Mar 13 '21 at 15:00
  • it also appears there is no way to "eagerly" connect to kafka as a producer, this can check the configuration but not avoid the delay on the first sent message (which is what i wanted to do) – Dave Ankin Mar 13 '21 at 15:01
  • @AndyB , it will be really helpful if you could share the AdminClient API reactive code snippet for reference or if you find a solution to avoid lazy initializattion for the first call – shijin raj Mar 03 '22 at 09:19
  • @shijin The Vertx kafka documentation has examples, if you're using that library – OneCricketeer Mar 03 '22 at 14:25