0

I'm using sarama-cluster lib to create a kafka group consumer, in a backend service. This example code from godoc works:

for {
    if msg, ok := <-consumer.Messages(); ok {
        fmt.Fprintf(os.Stdout, "%s/%d/%d\t%s\t%s\n", msg.Topic, msg.Partition, msg.Offset, msg.Key, msg.Value)
        consumer.MarkOffset(msg, "") // mark message as processed
    }
}

since it's a dead loop, I put it in a goroutine to avoid blocking other activities, then it can no longer consume any message:

go func() {
    for msg := range consumer.Messages() {
        fmt.Fprintf(os.Stdout, "%s/%d/%d\t%s\t%s\n", msg.Topic, msg.Partition, msg.Offset, msg.Key, msg.Value)
        consumer.MarkOffset(msg, "") // mark message as processed
    }
}()

(service is running, so this goroutine is not terminated. It just fails to consume)

Any idea how to solve this?

Patryk
  • 22,602
  • 44
  • 128
  • 244
June
  • 13
  • 1
  • 6
  • Do you close the channel after firing up the for loop? – rsp Apr 16 '20 at 14:13
  • @rsp I think no, the channel is owned by the lib, I just call `Messages()` to read from it. – June Apr 16 '20 at 15:52
  • You're not meant to close this channel, in fact you can't because it's a read only channel - you can't send on it and you can't close it. What @rsp is probably referring to is if you've closed it then the loop (and the goroutine) would exit immediately. Double check if this goroutine and loop is running. – Patryk Apr 16 '20 at 16:18
  • @Patryk Yes. Above the message reading, there is a `defer consumer.Close()` statement. So wrapping consuming action in goroutine makes the function exit quickly. My carelessness! – June Apr 17 '20 at 06:49

1 Answers1

0

sarama-cluster is in unmaintained for quite some time already with the following notice:

Please note that since https://github.com/Shopify/sarama/pull/1099 was merged and released (>= v1.19.0) this library is officially deprecated. The native implementation supports a variety of use cases that are not available through this library.

I suggesty you to use github.com/Shopify/sarama instead. It has all the functionalities of sarama-cluster and it's actively maintained.

You can follow a simple consumer group example from their repository.

Patryk
  • 22,602
  • 44
  • 128
  • 244
  • Helpful reference! Yet I still haven't figured out why the two versions in my question behave differently~ – June Apr 16 '20 at 15:50
  • Solved. As comments under the question suggest, channel closed (by a `defer consumer.Close()`) after firing up the for loop... – June Apr 17 '20 at 06:53
  • @June Glad you figured it out! – Patryk Apr 17 '20 at 08:51