3

I use https://github.com/Shopify/sarama for interaction with Kafka. I have a topic with, for example, 100 partitions. I have application, which is deployed on 1 host. So, I want to consume from this topic in multiple goroutines.

I see this example - https://github.com/Shopify/sarama/blob/master/examples/consumergroup/main.go , in which we can see, how to create consumer in specific consumer group.

So, my question is, should I create multiply such consumers, or there is some setting in Sarama, where I can set up needed number of consumer goroutines.

P.S. I see this question - https://github.com/Shopify/sarama/issues/140 - but there is no answer, how to create MultiConsumer.

Max
  • 1,803
  • 3
  • 25
  • 39
  • I think you can consume with 100 consumers running in 100 goroutines belonging to the same consumergroup. But if you do this from a single host you might not have much speedup to consuming with a single consumer. And yes, you need to set up N consumers if you wish to use N consumers. – Volker Jul 04 '19 at 19:00

1 Answers1

3

This example shows a fully working console application which can consume for all partitions in a topic creating one goroutine per partition:

https://github.com/Shopify/sarama/blob/master/tools/kafka-console-consumer/kafka-console-consumer.go

It is linked at the end of the thread you posted in your question.

It basically creates one consumer:

c, err := sarama.NewConsumer(strings.Split(*brokerList, ","), config)

Then gets all the partitions for the desired topic:

func getPartitions(c sarama.Consumer) ([]int32, error) {
    if *partitions == "all" {
        return c.Partitions(*topic)
    }
...

Then for each partition it creates a PartitionConsumer and consumes from each partition in a different goroutine:

for _, partition := range partitionList {
    pc, err := c.ConsumePartition(*topic, partition, initialOffset)
    ....

    wg.Add(1)
    go func(pc sarama.PartitionConsumer) {
        defer wg.Done()
        for message := range pc.Messages() {
            messages <- message
        }
    }(pc)

}
eugenioy
  • 11,825
  • 28
  • 35
  • thank you, this example works pretty good for 1 node of application. Interesting, is there anything like this for consumer group API? – Max Jul 04 '19 at 19:07
  • 1
    Typically consumer groups are used when you need to split the consumers in different processes (same group name would indicate that all those separate processes are actually part of the same group and not independent consumers). In that case, you'd probably want to implement something along these lines: https://godoc.org/github.com/Shopify/sarama#ConsumerGroup and then run that same code in different processes / hosts. – eugenioy Jul 05 '19 at 03:47