0

I am trying to consume messages from a Kafka topic in Amazon MSK using Apache NiFi. NiFi ConsumeKafka processor requires group.id property of Kafka.Where or How can I find the group.id property for the MSK cluster?

Tris
  • 63
  • 1
  • 1
  • 8

1 Answers1

1

You can set the group.id property to whatever you want - this simply allows you to group multiple consumers (i.e. multiple NiFi nodes) together to consume from a single topic concurrently.

Read up on Kafka Consumer Groups

E.g. https://blog.cloudera.com/scalability-of-kafka-messaging-using-consumer-groups/

Sdairs
  • 1,912
  • 1
  • 13
  • 13
  • Can I define group.id while creating a topic? – Tris Jan 05 '21 at 10:48
  • No, groups have no relation to topics. The `group.id` is set on the consumer, and can be used on any topics. – Sdairs Jan 05 '21 at 11:12
  • I tried to list the groups using kafka-console-consumer.sh and obtained a default consumer group console-consumer-34567. When I configured this group id in NiFi it gave the error "org.apache.kafka.clients.NetworkClient [Consumer clientId=console-consumer-34567-5, groupId=console-consumer-34567] Bootstrap broker ip:port (id: -1 rack: null) disconnected". Is it issue with configuring wrong group id.? – Tris Jan 05 '21 at 11:28
  • You could set your `group.id` to `turnips` it doesn't matter what the value is - this just groups multiple consumers together for synchronous consumption. Your error suggests you can't connect to the Kafka brokers. There is a tutorial from Amazon about using MSK here https://docs.aws.amazon.com/msk/latest/developerguide/getting-started.html – Sdairs Jan 05 '21 at 12:10