1

I have been using kafka AdminClient (below snippet) for resetting offset :

Map<TopicPartition, OffsetAndMetadata> newOffsets = getOffsetMap();
AlterConsumerGroupOffsetsResult resultst = wrapper.getAdminClient()
.alterConsumerGroupOffsets("somegroup", newOffsets);

private Map<TopicPartition, OffsetAndMetadata> getOffsetMap() {
Map<TopicPartition, OffsetAndMetadata> a = new HashMap<>();
String topic = "SOME_TOPIC";
int partition = 0;
long offset = 2L;
a.put(new TopicPartition(topic, partition), new OffsetAndMetadata(offset));
return a;
}

This works as expected but I came across some info which points out that perfoming these kind of offset reset actions from adminClient may create race conditions if the consumers from same group are committing offsets to kafka at the exact same time of offset reset action.

To avoid this; I am thinking of stopping the specific consumer group first from adminClient side and then initiate offset reset action.

I want to specifically know below pointers :

  1. Is this really needed to be done, or there are no race conditions in kafka as explained above.
  2. How to "STOP A CONSUMER GROUP" using AdminClient java code (regardless of point #1 above). I looked up at below method but not sure if that achieves stopping a CG
@Override public RemoveMembersFromConsumerGroupResult removeMembersFromConsumerGroup(String groupId, RemoveMembersFromConsumerGroupOptions options){}

Edit1:

String inputCG = "somegroup";
Map<String, Collection<MemberToRemove>> toBeRemoved = new HashMap<>();
wrapper.getAdminClient().describeConsumerGroups(Arrays.asList(inputCG)).all().get()
        .forEach((cg, consumerGroupDescription) -> {
            Collection<MemberToRemove> memberToRemoves = consumerGroupDescription.members().stream()
                    .map(memberDescription -> new MemberToRemove(memberDescription.groupInstanceId().get()))
                    .collect(Collectors.toList());
            if (!CollectionUtils.isEmpty(memberToRemoves))
                toBeRemoved.put(cg, memberToRemoves);
        });

if (toBeRemoved.containsKey(inputCG)) {
    KafkaFuture<Void> removed = wrapper.getAdminClient().removeMembersFromConsumerGroup(inputCG,
            new RemoveMembersFromConsumerGroupOptions(toBeRemoved.get(inputCG))).all();
    removed.get();
    System.out.println("removal completed; proceeding for offset reset");
} else {
    System.out.println("no members found for removal; proceeding for offset reset");
}
Saurabh Bhoomkar
  • 595
  • 1
  • 9
  • 29
  • AFAIK, you cannot remotely stop a consumer – OneCricketeer Aug 03 '21 at 18:09
  • Wondering what impact that code snippet will have, added as edit 1 @OneCricketeer – Saurabh Bhoomkar Aug 03 '21 at 18:31
  • Based on my experience with the CLI tools, consumer groups must be inactive to be modified at all – OneCricketeer Aug 03 '21 at 18:38
  • @OneCricketeer So there's no way to achieve this fully remotely by staying transparent to consumers ? – Saurabh Bhoomkar Aug 03 '21 at 19:05
  • 1
    There are two options changing positions of consumer, one requires clients down is the adminClient / cli tools , the other , as far as I understand, did not get the chance to fully test it, is changing without deactivate the clients , consumer.seek – Ran Lupovich Aug 03 '21 at 19:07
  • 2
    You effectively need to halt the consumer heartbeat threads, which need direct access to the consumer processes, which cannot be done remotely without some interface for doing so (for example, adding a REST endpoint on a Spring app that closes/pauses the consumer instances) – OneCricketeer Aug 03 '21 at 19:34

0 Answers0