Kafka Consumer receiving same message

Question

I am new to Kafka and I have a Kafka consumer implemented using Java Apache Camel library. The issue I found out is - Consumer takes a long time(>15 mins) to process for few messages- which is fine for our use case.

Need some config help because the same message is re-sent after 15 mins if not processed within 15 mins(thread control does not return I believe). I think this might be the default interval, not sure which property is this.

So, where can I have to fix the config

Producer level, so that it does not resend
Or Producer is not involved in re-sending, it's the Broker - Kafka server so Consumer must acknowledge the message - in my case just before processing.

My producer has these properties:

<entry key="bootstrap.servers" value="${kafka.bootstrap.servers}" />
       <entry key="key.serializer" value="org.apache.kafka.common.serialization.StringSerializer" />
       <entry key="value.serializer" value="org.apache.kafka.common.serialization.StringSerializer" />

My consumer configs look like

<endpoint id="apolloKafkaJanitorEventListenerURI"
            uri="kafka:${kafka.bootstrap.servers}?topic=${apollo.janitor.event.topic}&amp;
                                                        groupId=${apollo.janitor.event.group.id}&amp;
                                                        consumersCount=${apollo.janitor.event.consumer.count}&amp;
                                                        consumerRequestTimeoutMs=${eventConsumerRequestTimeoutMs}&amp;
                                                        sessionTimeoutMs=${eventConsumerSessionTimeoutMs}&amp;
                                                        maxPartitionFetchBytes=${eventConsumerMaxPartitionFetchBytes}" />

I have googled didn't find any relevant issues. Found "acks=0" property on Producer and for Consumer the following. Have not tested but want to see if I am on right track first

KafkaManualCommit manual =
       exchange.getIn().getHeader(KafkaConstants.MANUAL_COMMIT, KafkaManualCommit.class);
   manual.commitSync();

https://github.com/apache/camel/blob/master/components/camel-kafka/src/main/docs/kafka-component.adoc

What are the actual values in your consumer configuration? Currently I only see placeholders like `${eventConsumerRequestTimeoutMs}`. — Mickael Maison, Dec 13 '18 at 15:28

score 1 · Answer 1 · answered Dec 13 '18 at 15:00

The problem might be at the producer end. You may nee need to check if producer is resending the message. you can use logging statements for the same. or you can use exactly once semantics for kafka producer. you only need to add an extra property for the same.

Other might be that your consumer is not committing the offsets. you might need to do some brainstorming at this end too

score 1 · Answer 2 · answered Dec 14 '18 at 11:27

You try to avoid multiple deliveries of the same message. This is the wrong approach.

In a messaging system you have to deal with messages that can be delivered multiple times, simply because they are needed to guarantee message delivery in some situations (see here for a short explanation).

You can't totally avoid multiple deliveries without sacrifice other aspects of your system.

If you instead build your consumers idempotent, they don't care if a message is delivered multiple times by the broker. That way you don't need to restrict your broker.

Kafka Consumer receiving same message

2 Answers2