Duplicate check for spring cloud stream consumers

Question

I want to check for duplicate messages in my Spring Cloud Stream Kafka consumers. This is what I have now:

@Component
public class MessageSubscriber {


    @Autowired
    MessageConsumer messageConsumer;

    public Consumer<Message<PlaneEvent>> planeEventConsumer() {
        return event -> messageConsumer.consumePlaneEvent(event);
    }

    @Component
    public static class MessageConsumer {
        
        @Transactional
        @DuplicateCheck(key = "id")
        public void consumePlaneEvent(Message<PlaneEvent> msg) {
            // do something here
        }

    }

}

The custom @DuplicateCheck annotation uses Spring AOP to intercept the consumer method and checks for duplicates in the database using the specified key, and the kafka_receivedTopic header from the Message.

Is there a recommended pattern or better way to:

• Intercept functional consumers. (I am aware about ChannelInterceptor, but would not prefer to use it because of multiple reasons. ChannelInterceptor intercepts all channels [input and output], and the regex to filter channels will need me to enforce a channel naming scheme. ChannelInterceptor will also complicate skipping the processing of duplicate messages. I'd like something less intrusive.)

• Get the kafka topic or binding destination in the functional consumer. Relying on the kafka_receivedTopic header seems brittle and inflexible.

• Handle transactions with functional consumers. Since the @Transactional annotation creates dynamic proxies, I'm having to wrap the consumer logic in another class and create its bean.

These things were a little simpler with the older annotation-based consumers (@StreamListener).

Answer 1

You can add a ListenerContainerCustomizer bean and add a RecordInterceptor to the listener container.

/**
 * An interceptor for {@link ConsumerRecord} invoked by the listener
 * container before and after invoking the listener.
 *
 * @param <K> the key type.
 * @param <V> the value type.
 *
 * @author Gary Russell
 * @since 2.2.7
 *
 */
@FunctionalInterface
public interface RecordInterceptor<K, V> extends ThreadStateProcessor {

    /**
     * Perform some action on the record or return a different one. If null is returned
     * the record will be skipped. Invoked before the listener. IMPORTANT; if this method
     * returns a different record, the topic, partition and offset must not be changed
     * to avoid undesirable side-effects.
     * @param record the record.
     * @param consumer the consumer.
     * @return the record or null.
     * @since 2.7
     */
    @Nullable
    ConsumerRecord<K, V> intercept(ConsumerRecord<K, V> record, Consumer<K, V> consumer);

    /**
     * Called after the listener exits normally.
     * @param record the record.
     * @param consumer the consumer.
     * @since 2.7
     */
    default void success(ConsumerRecord<K, V> record, Consumer<K, V> consumer) {
    }

    /**
     * Called after the listener throws an exception.
     * @param record the record.
     * @param exception the exception.
     * @param consumer the consumer.
     * @since 2.7
     */
    default void failure(ConsumerRecord<K, V> record, Exception exception, Consumer<K, V> consumer) {
    }

    /**
     * Called when processing the record is complete either
     * {@link #success(ConsumerRecord, Consumer)} or
     * {@link #failure(ConsumerRecord, Exception, Consumer)}.
     * @param record the record.
     * @param consumer the consumer.
     * @since 2.8
     */
    default void afterRecord(ConsumerRecord<K, V> record, Consumer<K, V> consumer) {
    }

}