0

We have Kafka data source which has 7 brokers (nodes) lets say node1, node2 to node7. When Upsolver Kafka Data source is created, the connection only specified node1 to node5. node6 and node7 were not added.

When Upsolver is trying to ingest data, it's getting a timeout exception, and looks like it is hitting node6 and is getting a timeout. The question is why is it that Upsolver attempts to connect to node6 even though node6 is not included in the Data Source's host list?

Jason Fine
  • 32
  • 6
Ajay C
  • 66
  • 5

1 Answers1

0

The broker list provided to Kafka clients is typically only used to initiate the connection to Kafka.

Once connected, the Kafka cluster can and will redirect the client to specific nodes in order to:

  • Send the client to the node containing the data for the topic and partition it wants to read from
  • Perform load balancing and error recovery

In Upsolver specifically, the client list is passed on to the Kafka client library which behaves in the manner described above.

Access to all broker nodes is usually required to read from Kafka.

See here for more information

Jason Fine
  • 32
  • 6