0

Deployed a 3 node kafka cluster in kubernetes using a statefulset using Kraft.

Within the cluster, the configuration is as follows (pseudo-code):

  • controller quorum: 0@kafka-0,1@kafka-1,2@kafka-2
  • listeners: broker://:9092,controller://:9093,LOCAL://localhost:9094
  • advertised_listeners: broker://:9092,controller://:9093,LOCAL://localhost:9094

All nodes are broker/controller

Within the kubernetes cluster, this kraft configuration is working flawlessly. The cluster stands up and services pointing to all three brokers are able to produce and consume messages as expected.

However, to access the kafka cluster from outside Kubernetes, I need to port-forward to gain access (or otherwise ingress).

Since Kraft utilizes all controllers/brokers as "leader" nodes as per the documentation, why am I unable to connect to a single node and get full access to the partitions.

For example, if I port forward kafka-0 and connect to it, I have access to create and list topics.

However, if I create a topic with 3 partitions, they might be configured as follows:

  • partition 0 : leader 1 : replicas 3
  • partition 1 : leader 2 : replicas 3
  • partition 2 : leader 0 : replicas 3

If I port forward kafka-0 (which maps to leader 0 in the above example), I am able to push a message ONLY to partition 2. If I attempt to push a message to partition 0 or partition 1, I receive the following error NOT_ENOUGH_REPLICAS.

I would've expected the broker/controller node I port-forwarded to forward the produced message to the correct broker/controller that owned the partition, in a proxy-like pattern.

However, this was the not the case.

What actually happens when you specify a subset of the kafka nodes in your connection string (bootstrap-servers for example) under Kraft?

Gary Russell
  • 166,535
  • 14
  • 146
  • 179
StevenPG
  • 311
  • 4
  • 16
  • You seem to be asking two different questions. The quorum config is separate protocol from bootstrapping. Bootstrap servers can only be brokers, not controllers. Controller quorum must include no machines with only broker role – OneCricketeer May 18 '23 at 22:53

2 Answers2

1

That's why it's called BOOTSTRAP_SERVERS_CONFIG since the initial connection is for bootstrapping only, it is made to one of the servers specified there; once the connection is established, that broker returns information about the leader for each topic/partition.

When needed, the client then connects to the leader for a particular partition. You must have access to all broker instances.

Kraft Vs. Zookeeper makes no difference; it works the same.

Gary Russell
  • 166,535
  • 14
  • 146
  • 179
  • I'm seeing some inconsistency compared to the Zookeeper implementation that may just be due to a gap in my knowledge. For example, I have a producer configured to point to each node. However, when I send a message, only the leader accepts the message into the topic, the others report the following error: `Closing connection due to error during produce request with correlation id 3 from client id producer-45 with ack=0` `Topic and partition to exceptions: test3-0 -> org.apache.kafka.common.errors.NotLeaderOrFollowerException (kafka.server.KafkaApis)` – StevenPG May 18 '23 at 16:15
  • If the two implementations functioned the same, I wouldn't expect an error like NotLeaderOrFollower, since all nodes are configured as broker,controller. I imagine it's talking about the partition leader as you mentioned, but this partition has all nodes set as replica nodes and all are listed as in-sync. – StevenPG May 18 '23 at 16:22
0

I am able to push a message ONLY to partition 2

Because you've only forwarded ports of one broker.

Kafka clients are required to write to the leader partitions, so each broker in your cluster. One port will not proxy the entire cluster

This is a separate issue from bootstrap protocol, but you should always give more than one broker (or a DNS name that resolves to many, which is what an Ingress would be, but Ingress are only for HTTP traffic, according to k8s docs) in case networking errors when communicating to that one

OneCricketeer
  • 179,855
  • 19
  • 132
  • 245