0

I would like to share the properties before i clearly explain the issue.

Below are the properties of producer that i have used:

bootstrap.servers=xyz:9092
acks=all
retries=0
batch.size=16384
auto.commit.interval.ms=1000
linger.ms=0 key.serializer=org.apache.kafka.common.serialization.StringSerializer
value.serializer=org.apache.kafka.common.serialization.StringSerializer
block.on.buffer.full=true

Consumer proprties:

bootstrap.servers=ec2-54-218-85-12.us-west-2.compute.amazonaws.com:9092
group.id=test
enable.auto.commit=true
key.deserializer=org.apache.kafka.common.serialization.StringDeserializer
value.deserializer=org.apache.kafka.common.serialization.StringDeserializer
session.timeout.ms=10000
fetch.min.bytes=50000
receive.buffer.bytes=262144
max.partition.fetch.bytes=2097152

Producer.java

    int numOfmessages = Integer.valueOf(args[1]);
    // set up the producer
    KafkaProducer<String, String> producer;
    try (InputStream props = Resources.getResource("producer.props").openStream()) {
        Properties properties = new Properties();
        properties.load(props);
        producer = new KafkaProducer<>(properties);
    }

    try {
        for (int i = 0; i < numOfmessages; i++) {
            String message = "message number " +i;
            // send lots of messages
            producer.send(new ProducerRecord<String, String>("fast-messages", message ));
            logger.info("sent message "+message);
        }
    } catch (Throwable throwable) {
        System.out.printf("%s", throwable.getStackTrace());
    } finally {
        producer.close();
    }

Consumer.java

    KafkaConsumer<String, String> consumer;
    try (InputStream props = Resources.getResource("consumer.props").openStream()) {
        Properties properties = new Properties();
        properties.load(props);
        consumer = new KafkaConsumer<>(properties);
    }

    try {
        consumer.subscribe(Arrays.asList("fast-messages"));
        int timeouts = 0;
        //noinspection InfiniteLoopStatement
        while (true) {
            // read records with a short timeout. If we time out, we don't really care.
            ConsumerRecords<String, String> records = consumer.poll(200);
            if (records.count() == 0) {
                timeouts++;
            } else {
                logger.info("Got %d records after %d timeouts\n", records.count(), timeouts);
                timeouts = 0;
            }
            for (ConsumerRecord<String, String> record : records) {

                logger.info("consumed "+record.value());

                logger.info("doing some complex operation in consumer with "+record.value());

                for(int i= 0;i <999999999;i++) {

                    for(int j= 0;j <999999999;j++) {

                    }

                }

            }
        }
    }finally {
        consumer.close();
    }

With the above properties and code, when i run producer, its safely sending all the messages. On the consumer side, i am able to consume all the messages but when the offset commit happens, its failing with the below error.

2016-11-04 09:55:08 INFO AbstractCoordinator:540 - Marking the coordinator 2147483647 dead. 2016-11-04 09:55:08 ERROR ConsumerCoordinator:544 - Error UNKNOWN_MEMBER_ID occurred while committing offsets for group test 2016-11-04 09:55:08 WARN ConsumerCoordinator:418 - Auto offset commit failed: Commit cannot be completed due to group rebalance 2016-11-04 09:55:09 ERROR ConsumerCoordinator:544 - Error UNKNOWN_MEMBER_ID occurred while committing offsets for group test 2016-11-04 09:55:09 WARN ConsumerCoordinator:439 - Auto offset commit failed: 2016-11-04 09:55:09 INFO AbstractCoordinator:361 - Attempt to join group test failed due to unknown member id, resetting and retrying.

I kind of understand the issue that, its failing because of the complex operation we are having inside consumer. Any suggestions on how to handle this? This must be a very common scenrario, just wanted to understand if we need to change any configs, etc...

Andrey
  • 485
  • 3
  • 17
user3329002
  • 156
  • 2
  • 9
  • could you paste the entire stack trace and kafka version? – amethystic Nov 04 '16 at 07:01
  • Which Kafka Version are you using? The above error happens when the time taken to process the messages is higher than the configured `min.session.timeout.ms`. This issue has been resolved in the latest Kafka v-0.10.1.0. You might be hit the Bug - https://issues.apache.org/jira/browse/KAFKA-3888 – Kamal Chandraprakash Nov 05 '16 at 08:39

0 Answers0