understanding consumer group id

Question

I did fresh installation of Apache Kafka 0.10.1.0.

I was able to send / receive messages on command prompt.

While using Producer / Consumer Java Example, I am not able to know group.id parameter on Consumer Example.

Let me know on how to fix this issue.

Below is Consumer Example I had used:

public static void main(String[] args) {
             Properties props = new Properties();
             props.put("bootstrap.servers", "localhost:9092");
             props.put("group.id", "my-topic");
             props.put("enable.auto.commit", "true");
             props.put("auto.commit.interval.ms", "1000");
             props.put("key.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");
             props.put("value.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");
             KafkaConsumer<String, String> consumer = new KafkaConsumer<>(props);
             try {
                 consumer.subscribe(Arrays.asList("my-topic"));

                     ConsumerRecords<String, String> records = consumer.poll(100);
                     System.err.println("records size=>"+records.count());
                     for (ConsumerRecord<String, String> record : records) 
                         System.out.printf("offset = %d, key = %s, value = %s%n", record.offset(), record.key(), record.value());



              }
             catch (Exception ex){
                 ex.printStackTrace();
             }
            finally {
                 consumer.close();
            }
        }

After running the command for consumer, I can see the messages (on the console) posted by producer. But unable to see the messages from java program

bin\windows\kafka-console-consumer.bat --bootstrap-server localhost:9092 --topic my-topic --from-beginning

If you run your java consumer and produce some messages AFTER launching it, you still don't see any message being consumed? — Luciano Afranllie, Dec 29 '16 at 20:08
You can get the value of `group.id` for your kafka cluster by looking into `$KAFKA_HOME/config/consumer.properties`. There you can see the line `#consumer group id`. Use this value and your code will work. You can group multiple consumers to same group by giving same value of `group.id` in this file. — vindev, Apr 05 '18 at 11:50
I am not sure whether you have got the answer. But, my assumption is that you may be running the code in eclipse and in one window you are running producer. But without stopping producer, you are trying to start consumer hence you may not be able to see the records in IDE. But you can see those reocrds on console. Though this is a trivial thing still needs to discuss it. This is my assumption. Please correct me if I am incorrect. — anshuman sharma, Dec 01 '18 at 17:55
I am getting error : java.lang.NoSuchMethodError: org.apache.kafka.clients.consumer.Consumer.subscribe(Ljava/util/Collection;)V — ShrJoshi, Jun 28 '19 at 11:45

score 63 · Answer 1 · answered Dec 29 '16 at 10:18

63

Consumers label themselves with a consumer group name, and each record published to a topic is delivered to one consumer instance within each subscribing consumer group. Consumer instances can be in separate processes or on separate machines.

If all the consumer instances have the same consumer group, then the records will effectively be load balanced over the consumer instances.

If all the consumer instances have different consumer groups, then each record will be broadcast to all the consumer processes.

The group.id is a string that uniquely identifies the group of consumer processes to which this consumer belongs.

(Kafka intro)

answered Dec 29 '16 at 10:18

Raz Omessi

1,812
1
14
13

1

But how I would know this string since i didn't mention any group.id when sending the message ? – Ankit Dec 29 '16 at 10:21
6

you can use any string, if you will run two consumers that have the same string they will be at the same group. – Raz Omessi Dec 29 '16 at 10:26
i could not able to see the messages after putting group.id on above java program. However, i can see message on executing command (mentioned above) on the console. Ironically, there is no group.id required to see the messages for consumer. – Ankit Dec 29 '16 at 10:29
1

What if not specify consumer group id? – Akmal Salikhov Mar 31 '20 at 12:42
@AkmalSalikhov , it will be created automatically – Ahmed Abdelhak Aug 31 '21 at 14:29

score 5 · Answer 2 · answered Apr 05 '18 at 11:43

Here are some test results on partition and consumer property group.id

 Properties props = new Properties();
  //set all other properties as required
  props.put("group.id", "ConsumerGroup1");
  props.put("max.poll.records", "1");
  KafkaConsumer<String, String> consumer = new KafkaConsumer<String, String>(props);

consumer.group id is to load balance the produced data (if the group.id is different for each consumer, each consumer will get the copy of data)

if partition=1 and total consumers count = 2, only one out of two active consumer will get data

if partition=2 and total consumers count = 2, each of the two active consumers evenly get data

if partition=3 and total consumers count = 2, each of the two active consumers will get data. one consumer gets data from 2 partitions and other gets data from 1 partition.

if partition=3 and total consumers count = 3, each of the three active consumers evenly gets data.

score 2 · Answer 3 · edited Apr 30 '19 at 10:50

2

The consumer group id the consumer group which should be defined in the Kafka consumer.properties file.

Do add "my-topic" to consumer group and it should work as below:

# consumer group id
group.id=my-topic-consumer-group

edited Apr 30 '19 at 10:50

karel

5,489
46
45
50

answered Apr 30 '19 at 10:24

MANISH PARGANIHA

142
1
3

score 0 · Answer 4 · answered May 24 '17 at 10:41

In the code you provided you just wait for data once for 100ms. You should receive the data in a loop or wait for longer period of time (you will only get one portion of data in this case). As for 'group.id' it the case you run consumer from console it gets random 'group.id'.

score 0 · Answer 5 · answered Oct 07 '17 at 23:02

Since no offset was provided, the java client will wait for new messages but will not show existing messages - this is as expected. If one intends to read all the messages already in the topic one can use this piece of code:

if (READ_FROM_BEGINNING) {
    //consume all the messages from the topic from the beginning.
    //this doesn't work reliably if it consumer.poll(..) is not called first 
    //probably because of lazy-loading issues            
    consumer.poll(10);
    consumer.seekToBeginning(consumer.assignment()); //if intending to 
    //read from the beginning or call below to read from a predefined offset.
    //consumer.seek(consumer.assignment().iterator().next(), READ_FROM_OFFSET);
}

score -14 · Answer 6 · edited May 08 '18 at 11:46

-14

Give any random value to group id. It doesn't matter.

props.put("group.id", "Any Random Value");

edited May 08 '18 at 11:46

Giorgos Myrianthous

36,235
20
134
156

answered Nov 05 '17 at 09:24

Harpreet Varma

490
5
4

1

Following the accepted answer, it really matters and it will affect the behavior on consumers – OlegI Jun 11 '19 at 14:59
2

group.id is one of the most important properties. if you give a random value to it, your consummer will alway use your auto.offset.reset properties all the time. this can be a problem if you want to restart your consumer from a specific offset – maxime G Jul 02 '19 at 13:17
@maxime G: It's more significant than that: a consumer group combines multiple consumers into a logical group so they correctly share consumption of each topic. It's the fundamental building block of parallelism in Kafka. – boycy Nov 02 '21 at 10:40
This is such a bad answer it should actually be deleted. You can cause outages doing this. – John Fantastico Feb 11 '23 at 16:57

understanding consumer group id

6 Answers6