1

I'm trying to increase the number of consumers to match the number of partitions for a Kafka topic that we are reading from. There are three partitions so I configured the partitions for incoming messages to three as shown below:

mp:
  messaging:
    incoming:
      topicA:
        auto:
          offset:
            reset: earliest
        topics: TOPIC-NAME
        connector: smallrye-kafka
        value:
          deserializer: org.apache.kafka.common.serialization.StringDeserializer
        group:
          id: consumer-group
        partitions: 3

However, I've been running the app for a while and it seems that the app is only processing messages in partition 0 and not in partitions 1 and 2. I see in the log that it creates three consumers.

2021-06-16 23:35:59,826 INFO  [org.apa.kaf.cli.con.int.AbstractCoordinator] (vert.x-kafka-consumer-thread-0) [Consumer clientId=kafka-consumer-topicA-0, groupId=consumer-group] Successfully joined group with generation 15
2021-06-16 23:35:59,826 INFO  [org.apa.kaf.cli.con.int.AbstractCoordinator] (vert.x-kafka-consumer-thread-2) [Consumer clientId=kafka-consumer-topicA-2, groupId=consumer-group] Successfully joined group with generation 15
2021-06-16 23:35:59,826 INFO  [org.apa.kaf.cli.con.int.AbstractCoordinator] (vert.x-kafka-consumer-thread-1) [Consumer clientId=kafka-consumer-topicA-1, groupId=consumer-group] Successfully joined group with generation 15
2021-06-16 23:35:59,831 INFO  [org.apa.kaf.cli.con.int.ConsumerCoordinator] (vert.x-kafka-consumer-thread-1) [Consumer clientId=kafka-consumer-topicA-1, groupId=consumer-group] Adding newly assigned partitions: TOPIC-NAME-1
2021-06-16 23:35:59,831 INFO  [org.apa.kaf.cli.con.int.ConsumerCoordinator] (vert.x-kafka-consumer-thread-0) [Consumer clientId=kafka-consumer-topicA-0, groupId=consumer-group] Adding newly assigned partitions: TOPIC-NAME-0
2021-06-16 23:35:59,831 INFO  [org.apa.kaf.cli.con.int.ConsumerCoordinator] (vert.x-kafka-consumer-thread-2) [Consumer clientId=kafka-consumer-topicA-2, groupId=consumer-group] Adding newly assigned partitions: TOPIC-NAME-2

But it seems to process messages in partition 0:

2021-06-16 23:38:00,141 INFO  [MessageListener] (vert.x-worker-thread-2) Partition number:0; offset: 1593011
2021-06-16 23:38:00,282 INFO  [MessageListener] (vert.x-worker-thread-1) Partition number:0; offset: 1593012
2021-06-16 23:38:00,412 INFO  [MessageListener] (vert.x-worker-thread-4) Partition number:0; offset: 1593013
2021-06-16 23:38:00,543 INFO  [MessageListener] (vert.x-worker-thread-6) Partition number:0; offset: 1593014
2021-06-16 23:38:00,692 INFO  [MessageListener] (vert.x-worker-thread-8) Partition number:0; offset: 1593015
2021-06-16 23:38:00,838 INFO  [MessageListener] (vert.x-worker-thread-10) Partition number:0; offset: 1593016
2021-06-16 23:38:00,977 INFO  [MessageListener] (vert.x-worker-thread-12) Partition number:0; offset: 1593017
2021-06-16 23:38:01,131 INFO  [MessageListener] (vert.x-worker-thread-14) Partition number:0; offset: 1593018
2021-06-16 23:38:01,272 INFO  [MessageListener] (vert.x-worker-thread-16) Partition number:0; offset: 1593019
2021-06-16 23:38:01,406 INFO  [MessageListener] (vert.x-worker-thread-18) Partition number:0; offset: 1593020
2021-06-16 23:38:01,535 INFO  [MessageListener] (vert.x-worker-thread-0) Partition number:0; offset: 1593021
2021-06-16 23:38:01,670 INFO  [MessageListener] (vert.x-worker-thread-3) Partition number:0; offset: 1593022
2021-06-16 23:38:01,799 INFO  [MessageListener] (vert.x-worker-thread-5) Partition number:0; offset: 1593023

Here's the code snippet of the listener class:

    @Incoming("topicA")
    @Blocking
    public CompletionStage<Void> consume(final IncomingKafkaRecord<String, String> message) {

        log.info("Partition number:" + message.getPartition() + "; offset: " + message.getOffset());
    
        return message.ack();
    }

Is this a bug with small-rye kafka?

Delta
  • 11
  • 3
  • Is the producer producing to all partitions? Likely it only writes to partition 0. – didiz Jun 17 '21 at 10:33
  • It is. All the messages are in all three partitions of the topic that I am reading. Another app is publishing the messages to the topic I am consuming from. – Delta Jun 17 '21 at 13:49
  • You need to create a unique consumer group id, that should get you reading from all partitions – didiz Jun 17 '21 at 13:52
  • If my understanding is correct, when setting the partitions config to 3, the Small-Rye Reactive Messaging library creates three kafka consumers on seperate threads under the hood using the same group id. Since they are using the same group id, the consumers will read each partition from that topic. – Delta Jun 17 '21 at 14:22
  • Does not appear to be happening:) – didiz Jun 17 '21 at 14:23

1 Answers1

0

It appears these settings will use multiple partitions: this will consume all messages.

mp.messaging.incoming.your-events.auto.offset.reset=earliest
mp.messaging.incoming.your-events.group.id=${quarkus.uuid}

If you are using an emitter this will work without the above settings;

int partition = 0;
Message<Integer> message = Message.of(value)
            .addMetadata(OutgoingKafkaRecordMetadata.<String>builder()
                .withKey(key)
                .withPartition(partition) // change for each partition, 0, 1, 2..                
                .withTopic("your-events")
                .build());
Dave O
  • 11
  • 2