0

I want to understand if Consumer.committablePartitionedSource uses separate kafka consumer per partition(all my partitions say(8) are assigned to single node)

Rohit Thakur
  • 244
  • 2
  • 12

1 Answers1

0

You can run your application and separately usekafka-consumer-groups --describe CLI command to inspect which client ids are mapped to which partitions. If they're all the same, then it's a single process / single node

OneCricketeer
  • 179,855
  • 19
  • 132
  • 245
  • I have already checked, client id(127.0.1.1-1684083288052) is same for all the partitions in the kafka-consumer-groups command. Does Consumer.committablePartitionedSource internals manages the partitions assignment for each partition and create consumer for each, to make all partition run in isolation and parallel? I want to know how it works internally, and how it is different from Consumer.committableSource. – Rohit Thakur May 14 '23 at 17:02
  • I don't use Alpakka. Kafka consumer API itself uses a PartitionAssignor interface. If all clients are the same, then you then need to check if you have 8 active threads in your one instance consuming in parallel (use JMX monitoring tools) – OneCricketeer May 15 '23 at 11:48