0

My kafka consumer is connected to a cluster of brokers. There is a domain name server in between. Each broker has a domain name associated with an IP address. Problem is, the IPs are not static and for some reason, I have to restart the broker. The consumers are configured to reconnect. My question is when it will try to reconnect, will the consumer resolve the domain name to the new IP address or it will use the previously resolved IP address?

sovon
  • 877
  • 2
  • 12
  • 28

2 Answers2

1

Domain name resolution is performed by the underlying operating system's DNS resolver, which caches the resolved IP address for a certain period of time (known as the "time to live" or TTL). When your Kafka consumer is configured to reconnect to a Kafka cluster after a broker failure, it will attempt to resolve the domain name associated with the broker to an IP address every time it attempts to establish a new connection.

If you restart a broker and its IP address changes, the consumer will eventually attempt to connect to the new IP address once the TTL for the domain name has expired and a new lookup is performed. A similar issue has been fixed on the latest release of Kafka

GitHub Issue: https://github.com/akka/alpakka-kafka/issues/734

Kafka Ticket: https://issues.apache.org/jira/browse/KAFKA-7755

But it's a good idea to monitor your Kafka consumer's logs and behavior after a broker restart to ensure that it is successfully reconnecting to the cluster.

A l w a y s S u n n y
  • 36,497
  • 8
  • 60
  • 103
0

When the underlying IP of the domain name is changed after the initial connection, the next time the consumer tries to connect to that broker, it should use the updated IP address. However, this would greatly depend upon your DNS server and the time-to-live (TTL) for the DNS records. If shorter, it would propagate sooner

mad_max_
  • 1
  • Your answer could be improved with additional supporting information. Please [edit] to add further details, such as citations or documentation, so that others can confirm that your answer is correct. You can find more information on how to write good answers [in the help center](/help/how-to-answer). – xlmaster Feb 25 '23 at 11:59