I would like to share the properties before i clearly explain the issue.
Below are the properties of producer that i have used:
bootstrap.servers=xyz:9092
acks=all
retries=0
batch.size=16384
auto.commit.interval.ms=1000
linger.ms=0 key.serializer=org.apache.kafka.common.serialization.StringSerializer
value.serializer=org.apache.kafka.common.serialization.StringSerializer
block.on.buffer.full=true
Consumer proprties:
bootstrap.servers=ec2-54-218-85-12.us-west-2.compute.amazonaws.com:9092
group.id=test
enable.auto.commit=true
key.deserializer=org.apache.kafka.common.serialization.StringDeserializer
value.deserializer=org.apache.kafka.common.serialization.StringDeserializer
session.timeout.ms=10000
fetch.min.bytes=50000
receive.buffer.bytes=262144
max.partition.fetch.bytes=2097152
Producer.java
int numOfmessages = Integer.valueOf(args[1]);
// set up the producer
KafkaProducer<String, String> producer;
try (InputStream props = Resources.getResource("producer.props").openStream()) {
Properties properties = new Properties();
properties.load(props);
producer = new KafkaProducer<>(properties);
}
try {
for (int i = 0; i < numOfmessages; i++) {
String message = "message number " +i;
// send lots of messages
producer.send(new ProducerRecord<String, String>("fast-messages", message ));
logger.info("sent message "+message);
}
} catch (Throwable throwable) {
System.out.printf("%s", throwable.getStackTrace());
} finally {
producer.close();
}
Consumer.java
KafkaConsumer<String, String> consumer;
try (InputStream props = Resources.getResource("consumer.props").openStream()) {
Properties properties = new Properties();
properties.load(props);
consumer = new KafkaConsumer<>(properties);
}
try {
consumer.subscribe(Arrays.asList("fast-messages"));
int timeouts = 0;
//noinspection InfiniteLoopStatement
while (true) {
// read records with a short timeout. If we time out, we don't really care.
ConsumerRecords<String, String> records = consumer.poll(200);
if (records.count() == 0) {
timeouts++;
} else {
logger.info("Got %d records after %d timeouts\n", records.count(), timeouts);
timeouts = 0;
}
for (ConsumerRecord<String, String> record : records) {
logger.info("consumed "+record.value());
logger.info("doing some complex operation in consumer with "+record.value());
for(int i= 0;i <999999999;i++) {
for(int j= 0;j <999999999;j++) {
}
}
}
}
}finally {
consumer.close();
}
With the above properties and code, when i run producer, its safely sending all the messages. On the consumer side, i am able to consume all the messages but when the offset commit happens, its failing with the below error.
2016-11-04 09:55:08 INFO AbstractCoordinator:540 - Marking the coordinator 2147483647 dead. 2016-11-04 09:55:08 ERROR ConsumerCoordinator:544 - Error UNKNOWN_MEMBER_ID occurred while committing offsets for group test 2016-11-04 09:55:08 WARN ConsumerCoordinator:418 - Auto offset commit failed: Commit cannot be completed due to group rebalance 2016-11-04 09:55:09 ERROR ConsumerCoordinator:544 - Error UNKNOWN_MEMBER_ID occurred while committing offsets for group test 2016-11-04 09:55:09 WARN ConsumerCoordinator:439 - Auto offset commit failed: 2016-11-04 09:55:09 INFO AbstractCoordinator:361 - Attempt to join group test failed due to unknown member id, resetting and retrying.
I kind of understand the issue that, its failing because of the complex operation we are having inside consumer. Any suggestions on how to handle this? This must be a very common scenrario, just wanted to understand if we need to change any configs, etc...