How to set group name when consuming messages in kafka using command line?

Question

Any idea how to set group name when consuming messages in kafka using command line.

I tried with the following command :

bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic nil_RF2_P2 --from-beginning --config group.id=test1
'config' is not a recognized option

The goal is to find the offset of consumed messages with the following command:

bin/kafka-run-class.sh kafka.tools.ConsumerOffsetChecker --zookeeper localhost:2181 --group test1

Can somebody help in this regards!!

Thanks in advance !!

score 45 · Accepted Answer · answered Jul 27 '17 at 13:50

45

The simplest solution is:

bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic nil_RF2_P2 --from-beginning --consumer-property group.id=test1

In case you specify the flag --from-beginning just remember that the consumer group should not have consumed any records in the past or else your consumer will start consuming from the earliest not consumed record by the specified group (and not from the actual beginning, as you may incorrectly assume).

answered Jul 27 '17 at 13:50

vtsamis

1,384
1
12
9

2

It works only if one replaces `--zookeeper localhost:2181` with `--bootstrap-server localhost:9092`. – Amit Dash Jun 12 '21 at 09:32

score 20 · Answer 2 · edited Aug 28 '18 at 15:19

20

Got the answer to change the groupname from command prompt!!

steps:

create a new consumer.properties file, say consumer1.properties.
change group.id=<Give a new group name> in the consumer1.properties.
bin/kafka-console-consumer.sh --new-consumer --bootstrap-server localhost:9092 --topic topicname --from-beginning --consumer.config config/consumer1.properties --delete-consumer-offsets

edited Aug 28 '18 at 15:19

OneCricketeer

179,855
19
132
245

answered Jul 24 '16 at 15:19

Nilotpal

3,237
4
34
56

3

Is there a list of all possible options (and their formats) accepted in this properties file? – patricknelson Mar 07 '18 at 19:50
@chunk_split they are available here: https://kafka.apache.org/documentation/#consumerconfigs – Ivan G. Dec 05 '18 at 16:11
2

Or you can directly use `--consumer-property group.id=XXXX` instead of creating a conflig file (or `--group` since the 1.x). – Idriss Neumann May 10 '19 at 14:53
@Nilotpal are you trying to delete messages from the given topic here ?? with --delete-consumer-offsets – Shasu Nov 03 '21 at 17:16

score 4 · Answer 3 · edited Aug 28 '18 at 15:18

4

if you want change group id without lost offset of record you have get offset manually of current Group.id and set to new run consumer that have new id. if don't have any control to get offset in consumer instance you can run this command.

/bin/kafka-consumer-groups.sh --new-consumer --bootstrap-server <ip_address>:<Broker_port>  --group Group_name --describe

and then you can seek data from specific offset. pay attention you should call seek after call poll، Assign command not work. also you can see my sample of code in github

Example here

edited Aug 28 '18 at 15:18

OneCricketeer

179,855
19
132
245

answered Jul 25 '16 at 05:02

Jeus

486
3
10
26

i guess kafka-consumer-groups.sh utility is only present from 0.10.0.0 onwards and not in 0.8.2.x versions of kafka!! – Nilotpal Jul 25 '16 at 10:13
@Nilotpal ooh yes i forget this. :) – Jeus Jul 25 '16 at 10:16

score 4 · Answer 4 · answered Aug 02 '18 at 11:47

4

You can use the --group option like this (tested with Kafka 2.0.0):

bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --group test-consumer --topic test --from-beginning

answered Aug 02 '18 at 11:47

Eno Thereska

234
2
6

1

Work since the 1.x version I think. Not with the 0.10.x version (you can use `--consumer-property group.id=XXXX` instead). – Idriss Neumann May 10 '19 at 14:52

score 0 · Answer 5 · answered Feb 13 '18 at 15:46

0

If you are using bash then you can use its process substitution capability.

bin/kafka-console-consumer.sh --zookeeper localhost:2181 \
--topic nil_RF2_P2 --from-beginning \
--consumer.config <(echo group.id=test1)

answered Feb 13 '18 at 15:46

slavik

1,223
15
17

How to set group name when consuming messages in kafka using command line?

5 Answers5

Linked