0

Has Kafka cluster and Kafka broker the same meaning?

I know cluster has multiple brokers (Is this wrong?).

But when I write code to produce messages, I find awkward option.

props.put("bootstrap.servers", "kafka001:9092, kafka002:9092, kafka003:9092");

Is this broker address or cluster address? If this is broker address, I think it is not good because we have to modify above address when brokers count changes.
(But it seems like broker address..)

Additionally, I saw in MSK in amazon, we can add broker to each AZ.
It means, we cannot have many broker. (Three or four at most?)

And they guided we should write this broker addresses to bootstrap.serveroption as a,` seperated list.

Why they don't guide us to use clusters address or ARN?

Michael Heil
  • 16,250
  • 3
  • 42
  • 77
JoonT
  • 1,106
  • 1
  • 13
  • 29

1 Answers1

7

A Kafka cluster is a group of Kafka brokers.

When using the Producer API it is not required to mention all brokers within the cluster in the bootstrap.servers properties. The Producer configuration documentation on bootstrap.servers gives the full details:

A list of host/port pairs to use for establishing the initial connection to the Kafka cluster. The client will make use of all servers irrespective of which servers are specified here for bootstrapping—this list only impacts the initial hosts used to discover the full set of servers. This list should be in the form host1:port1,host2:port2,.... Since these servers are just used for the initial connection to discover the full cluster membership (which may change dynamically), this list need not contain the full set of servers (you may want more than one, though, in case a server is down).

All brokers within a cluster share meta information of other brokers in the same cluster. Therefore, it is sufficient to mention even only one broker in the bootstrap-servers properties. However, you should still mention more than one in case of the one broker being not available for whatever reason.

Michael Heil
  • 16,250
  • 3
  • 42
  • 77