0

Background :

Setting consumer interceptor to StreamsConfig will ensure that the interceptor(s) are called when messages are consumed/committed. Snippet from org.apache.kafka.clients.consumer.internals.ConsumerCoordinator#commitOffsetsSync

       if (future.succeeded()) {
            if (interceptors != null)
                interceptors.onCommit(offsets);
            return true;
        }

But the consumerInterceptor.onCommit() was never called even though I saw the offsets being committed at the source topic.

Issue:

I figured that it was because I was using kstreams with Exactly once processing guarantee enabled.

This was the logic at org.apache.kafka.streams.processor.internals.StreamTask#commit

        if (this.eosEnabled) {
            this.producer.sendOffsetsToTransaction(consumedOffsetsAndMetadata, this.applicationId);
            this.producer.commitTransaction();
            if (startNewTransaction) {
                this.producer.beginTransaction();
            }
        } else {
            this.consumer.commitSync(consumedOffsetsAndMetadata);
        }

As you can see, consumer.commitSync which in turns calls the consumerCoordinator.commit which calls the interceptor.onCommit, never gets called because with eos enabled, it is the transaction api that gets invoked.

Question: Is there a way I can hook a callback to kstream when offsets are committed at the source topic with eos enabled?

Vinodhini Chockalingam
  • 306
  • 2
  • 17
  • I don't think there is. Did you try producer interceptors? Might be a nice feature to add to the producer. Maybe create a ticket for it? – Matthias J. Sax Jul 04 '20 at 20:00
  • Sure Matthias. I will create one ticket. I did check the ProducerInterceptor. It has only onAcknowledgement. I have a processor which does not forward the events right away. i.e. offsets might be committed to the source first and after the window has passed, the events from state store are forwarded to the sink. So using a producerInterceptor.onAck does not help. – Vinodhini Chockalingam Jul 06 '20 at 09:19
  • 1
    Ticket created at https://issues.apache.org/jira/browse/KAFKA-10236 – Vinodhini Chockalingam Jul 06 '20 at 09:30

0 Answers0