0

I am new to KAFKA and would like help

I have a topic XXXX and I have some applications consuming this topic, all listening to the same group

spring.cloud.stream.bindings.aaa_bbb.destination=XXXX
spring.cloud.stream.bindings.aaa_bbb.group=XXXX_group

topic XXXX has only one partition

sending 1000 messages to topic XXXX only one application consumes all messages.

but when I add a new partition to topic XXXX and the messages are divided into 2 applications and I still have applications without receiving anything.

I repeat the process and add a new partition to topic XXXX now the topic has 3 partitions and the messages are divided into 3 applications.

it looks like it's a partition for each consumer. which doesn't make much sense to me or I don't understand.

is there a way to make this load balance work, without having to create a partition for each consumer?

Can someone explain to me how this relationship works?

Peter Parker
  • 91
  • 1
  • 10

2 Answers2

0

That is a fundamental of Kafka - only one consumer in each group can consume from a partition. A topic/partition is a simple log, it is not like a queue in JMS or RabbitMQ.

Kafka maintains only a simple current committed offset for each group/topic/partition.

The only way to add concurrency is to increase the partitions.

Gary Russell
  • 166,535
  • 14
  • 146
  • 179
0

To increase the throughput you have to have more than one partition. When events are written to the log the ID of the event will determine which partition the message will be delivered to.

Kafka only guarantees the ordering over an ID, not over the entire log.

I normally recommend having more than one partition even if you have a single node as this allows the cluster to scaled in the future for improved performance.

You can't change the number of partitions after the topic has been created as that would have an impact on the partitioning target.

In your case I'd start with 3 per node up to a maximum of 9 if you had 3 nodes in the cluster - Please test this yourself.

There's a limit of 1 consumer per partition which is the behaviour you're seeing.

RubbleFord
  • 7,456
  • 9
  • 50
  • 80