1

I have a question about Alpakka Kafka behaviour by handling partitions for consumers.

For ex, how Alpakka Kafka will act in the following configuration.

Let's say, I have a topicA with 4 partitions and I have 6 Akka Nodes.

Normal behaviour of Kafka Consumers when they started under same groupId, they scale as number of partitions, so if I have 6 Akka Nodes, that would mean I would have 6 Kafka Consumers but while I have 4 partition, 2 Kafka Consumer would be always idle (let's say those would be Akka Node 5, Akka Node 6), these Akka Nodes that have idle Kafka Consumers will only get messages if ShardId logic from another Akka Node points to Akka Node 5 or 6 but not over their Kafka Consumer?

Is the assumption above is correct?

Then the second part of the question, if I increase the partition number to 12 for TopicA, now 6 Kafka Consumers from 6 Akka Nodes would share the partitions because they are under the same groupId. So every Akka Node would receive messages from 2 partitions.

Is the assumption above is correct?

Now if i want to have 6 Akka Nodes but 12 Kafka Consumers for 12 partitions, is it possible to configure that? Is the parallelism parameter at mapAsync controls this? (which you can see below code snippet).

val control: akka.kafka.scaladsl.Consumer.DrainingControl[Done] =
  Consumer
    .sourceWithOffsetContext(
      creditSMConsumerConfig.consumerSettings,
      Subscriptions.topics(topicProperties.getCreditSM)
    )
    .mapAsync(Runtime.getRuntime.availableProcessors() * 2) { consumerRecord =>
      {
        val creditSMCommand = consumerRecord.value()
        actorService.actorSystem().log.debug("Received Command: {}", creditSMCommand)
        askFacade.askCreditSMCommand(creditSMCommand)
      }
    
}
.via(Committer.flowWithOffsetContext(CommitterSettings(actorService.actorSystem())))
.toMat(Sink.ignore)(Consumer.DrainingControl.apply)
.run()

Thx for answers

posthumecaver
  • 1,584
  • 4
  • 16
  • 29

0 Answers0