Consumers within same group ids are consuming same message

Question

I am using Kafka Consumer API. The scenario is I have a topic with 7 partitions and each partition will be having a dedicated separate consumer for it. The code for consumer is as follows:

 String groupID = "topic_"+partitionID;

    Properties props = new Properties();
    props.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG,BOOTSTRAP_SERVERS);
    props.put(ConsumerConfig.GROUP_ID_CONFIG,groupID);
    props.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, IntegerDeserializer.class.getName());
    props.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, JsonDeserializer.class.getName());
    props.put(JsonDeserializer.TRUSTED_PACKAGES,"*");

    props.put(ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG, true);
    props.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "latest");

    KafkaConsumer<Integer,Student> consumer = new KafkaConsumer<>(props);
    TopicPartition topicPartition = new TopicPartition("request_topic",partitionID);

    consumer.assign(Collections.singletonList(topicPartition));

Now If I run only one single instance of this application, one partition will be having only one consumer listening to it. But if I start one more instance, there will be one more consumer listening to the same partition. Now both the consumers will be having same group id. If they are listening to partition zero, group id will be 'topic_0'. Now as per the rules, if a message is consumed by a consumer, the other consumers from the same group cannot consume it. But in my case, all the consumer within the same group are consuming the same message.

What could be the problem?

Please show a [mcve]. How are you setting partitionID to unique values for each deployment? Otherwise, it's the same value, therefore, yes, you'll consume the same records, because your group id is also different (even though, it's not used when you use assign function) — OneCricketeer, Jun 02 '23 at 13:25
Well, all I want is consumers within same consumer group listens to the message from same partition. How do I do it using assign or subscribe whatever way possible? The partitionID is passed through a function and it keeps on varying from 0-6. — Glitch07, Jun 02 '23 at 16:19
Two consumers cannot read from the same partition when using the same consumer group (with subscribe API). `group.id` is not used when using `assign` — OneCricketeer, Jun 02 '23 at 16:21
You've gotten two answers below that describe how Kafka consumer groups w/ subscribe API or non-grouped assign API works. — OneCricketeer, Jun 02 '23 at 20:42

score 1 · Answer 1 · answered Jun 02 '23 at 13:10

You are missing the fact that there is a difference between subscribe and assign in Apache Kafka. The consumer group logic works only with subscribe. Therefore it is not a surprise that after assigning the same partition to several consumers you get the same message consumed: there is just no exclusive consumer involved just because those consumers are out of consumer group scope.

See more info here:

KafkaConsumer Java API subscribe() vs assign()

And those KafkaConsumer methods Javadocs:

/**
 * Manually assign a list of partitions to this consumer. This interface does not allow for incremental assignment
 * and will replace the previous assignment (if there is one).
 * <p>
 * If the given list of topic partitions is empty, it is treated the same as {@link #unsubscribe()}.
 * <p>
 * Manual topic assignment through this method does not use the consumer's group management
 * functionality. As such, there will be no rebalance operation triggered when group membership or cluster and topic
 * metadata change. Note that it is not possible to use both manual partition assignment with {@link #assign(Collection)}
 * and group assignment with {@link #subscribe(Collection, ConsumerRebalanceListener)}.
 * <p>
 * If auto-commit is enabled, an async commit (based on the old assignment) will be triggered before the new
 * assignment replaces the old one.
 *
 * @param partitions The list of partitions to assign this consumer
 * @throws IllegalArgumentException If partitions is null or contains null or empty topics
 * @throws IllegalStateException If {@code subscribe()} is called previously with topics or pattern
 *                               (without a subsequent call to {@link #unsubscribe()})
 */
@Override
public void assign(Collection<TopicPartition> partitions) {

/**
 * Subscribe to the given list of topics to get dynamically assigned partitions.
 * <b>Topic subscriptions are not incremental. This list will replace the current
 * assignment (if there is one).</b> It is not possible to combine topic subscription with group management
 * with manual partition assignment through {@link #assign(Collection)}.
 *
 * If the given list of topics is empty, it is treated the same as {@link #unsubscribe()}.
 *
 * <p>
 * This is a short-hand for {@link #subscribe(Collection, ConsumerRebalanceListener)}, which
 * uses a no-op listener. If you need the ability to seek to particular offsets, you should prefer
 * {@link #subscribe(Collection, ConsumerRebalanceListener)}, since group rebalances will cause partition offsets
 * to be reset. You should also provide your own listener if you are doing your own offset
 * management since the listener gives you an opportunity to commit offsets before a rebalance finishes.
 *
 * @param topics The list of topics to subscribe to
 * @throws IllegalArgumentException If topics is null or contains null or empty elements
 * @throws IllegalStateException If {@code subscribe()} is called previously with pattern, or assign is called
 *                               previously (without a subsequent call to {@link #unsubscribe()}), or if not
 *                               configured at-least one partition assignment strategy
 */
@Override
public void subscribe(Collection<String> topics) {

NOTE: question like this has nothing to do with Spring. Please, be reasonable next time you choose tags for your question.

OneCricketeer · Answer 2 · 2023-06-02T20:44:11.867

0

If you deploy 7 instances that use consumer.subscribe("request_topic"), then partitions will automatically be distributed amongst them (assuming a static group.id); you shouldn't try to manually assign them.

If you deploy one instance, then all 7 partitions will be consumed. Any number of instances in between 1 and 6 will consume from multiple partitions, in parallel.

edited Jun 02 '23 at 20:44

answered Jun 02 '23 at 13:27

OneCricketeer

179,855
19
132
245

Consumers within same group ids are consuming same message

2 Answers2