1

I have the following component in k8s:

apiVersion: dapr.io/v1alpha1
kind: Component
metadata:
  name: kafka-pubsub
  namespace: default
spec:
  type: pubsub.kafka
  version: v1
  metadata:
      # Kafka broker connection setting
    - name: brokers
      value: "kafka-0.kafka-headless.default.svc.cluster.local:9092, 
kafka.default.svc.cluster.local:9092"
    - name: authRequired
      value: "false"
    - name: saslUsername
      value: ""
    - name: saslPassword
      value: ""
scopes:
  - signalrconnectorinterface
  - cardhandler
  - eventtrigger

I have a number of microservices which all subscribe properly, however the ones which subscribe to two topics (and more presumably) get the following error message upon subscription in the daprd logs:

time="2023-08-04T13:41:54.24563974Z" level=info msg="initializing Dapr workflow component" app_id=cardhandler component="dapr (workflow.dapr/v1)" instance=cardhandler-deployment-ff8d86465-xt85c scope=dapr.contrib type=log ver=1.11.2
time="2023-08-04T13:41:54.263847422Z" level=info msg="app is subscribed to the following topics: [send-card receive-command-message] through pubsub=kafka-pubsub" app_id=cardhandler instance=cardhandler-deployment-ff8d86465-xt85c scope=dapr.runtime type=log ver=1.11.2
time="2023-08-04T13:41:54.267674781Z" level=info msg="placement tables updated, version: 0" app_id=cardhandler instance=cardhandler-deployment-ff8d86465-xt85c scope=dapr.runtime.actor.internal.placement type=log ver=1.11.2
time="2023-08-04T13:41:55.200932919Z" level=error msg="Error consuming [receive-command-message]. Retrying...: kafka: tried to use a consumer group that was closed" app_id=cardhandler component="kafka-pubsub (pubsub.kafka/v1)" instance=cardhandler-deployment-ff8d86465-xt85c scope=dapr.contrib type=log ver=1.11.2
time="2023-08-04T13:41:58.812244394Z" level=info msg="dapr initialized. Status: Running. Init Elapsed 4843ms" app_id=cardhandler instance=cardhandler-deployment-ff8d86465-xt85c scope=dapr.runtime type=log ver=1.11.2

It notices there are two subscriptions, tries to allocate them then fails. Any advice?

OneCricketeer
  • 179,855
  • 19
  • 132
  • 245
Matt
  • 97
  • 9
  • 1
    Kafka doesn't have a concept of "closed" consumer groups, only "inactive"... This message seems specific to [sarama client](https://github.com/IBM/sarama/blob/main/consumer_group.go#L154), so if you have one group with more than one topic, but one app fails/disconnects/stops, then that's effectively "closing" that group resource, and should then rebalance and connect again, not be a fatal error, hence `Retrying...` log message followed by `Status: Running`. In other words - you can't "stop it" and this is expected behavior – OneCricketeer Aug 04 '23 at 17:12

1 Answers1

0

As per OneCricketeer, there is actually no issue here, this is intended behaviour. For some reason this just started happening when I was using a new helm install of Apache Kafka.

Ended up being a codebase issue so reset and all fine :)

Matt
  • 97
  • 9