What happens to a Kafka consumer group when all consumers are removed

Question

I have a service with an autoscaler and each instance needs to be in a separate consumer group, I've achieved it by making a random consumer group name group-id: my-service-${random.uuid}. I want to know what happens to a consumer group if all consumers are gone. I've noticed in Confluent platform that I have many consumer groups without any consumers.

For how long consumer groups without any consumers can exist?

How can I configure consumer groups to be removed after for example 5 minutes after the service is removed (I want the group to be removed not only the offset removed)?

Michael Heil · Accepted Answer · 2020-10-22T09:43:13.920

11

In Kafka you have the configuration offsets.retention.minutes at broker level:

offsets.retention.minutes: After a consumer group loses all its consumers (i.e. becomes empty) its offsets will be kept for this retention period before getting discarded. For standalone consumers (using manual assignment), offsets will be expired after the time of last commit plus this retention period."

This configuration defaults to 7 days. See all details in the Broker Configurations overview.

There is also an API in the AdminClient that allows you to remove specific ConsumerGroups. I have explained this procedure with some code in another answer.

Since Confluent 6.x you can also use the confluent control Center (using REST API) to easily remove outdated ConsumerGroups.

edited Oct 22 '20 at 09:43

answered Oct 22 '20 at 09:36

Michael Heil

16,250
3
42
77

I don't suppose you know how to do this pre-2.5.0 do you? – geedoubleya Jun 16 '23 at 11:53
unfortunately, I don't know how to do this pre-2.5.0. – Michael Heil Jun 16 '23 at 12:54

What happens to a Kafka consumer group when all consumers are removed

1 Answers1