Creating kafka consumers dynamically (without @Incoming annotation)

Question

(Using Quarkus 2)

I have a common process/function to execute for multiple topics, and I would like to make sure that the process on each channel is completely separated (If I send nack or the process is long, other channels should not be affected)

Right now, I'm declaring multiple channels on microprofile configuration, and making a consumer for each group of topics explicitely using @Incoming annotation, e.g.

mp:
  messaging:
    incoming:
      specificA:
        connector: smallrye-kafka
        topics: topicSpecificA1, topicSpecificA2
        ...
      specificB:
        connector: smallrye-kafka
        topics: topicSpecificB1, topicSpecificB2
        ...
      default:
        connector: smallrye-kafka
        topics: topicDefault1, topicDefault2

    @Incoming("default")
    @Blocking
    public CompletionStage<Void> consumeDefaultTopic(Message<OhpEntry> message) {
        return process(message);
    }

    @Incoming("specificA")
    @Blocking
    public CompletionStage<Void> consumeDefaultTopic(Message<OhpEntry> message) {
        return process(message);
    }


    @Incoming("specificB")
    @Blocking
    public CompletionStage<Void> consumeDefaultTopic(Message<OhpEntry> message) {
        return process(message);
    }

Since I know I will have a lot of "specificX" topics in the future, I'd like to avoid the need to change the code when a new channel needs to be read, and only manage the addition in the configuration.

My idea was to read the configuration and instantiate a consumer for each channel dynamically, but I don't really see how to do this with the existing API.

The org.apache.kafka.clients.admin.KafkaAdminClient seems to be used for topic creation and management on the cluster, not for consumer creations, and the io.smallrye.reactive.messaging.kafka.KafkaClientService can list consumers or channels, but not instantiate them.

How would you go about creating the consumers and linking them to a common consumption function ? I found answers for Spring but not for Quarkus/smallrye

Or is there another way to ensure that each channel consumption really stays independant (ignoring the shared thread pool) ?

Answer 1

You can't dynamically create channels with Microprofile Reactive Messaging. It works declaratively by reading the configuration and your injection points to set up the channels and wire them together.

Note that each channel subscribed to Kafka topic(s) creates a Kafka consumer underneath, with a dedicated polling thread. If the processing is common for all those channels you could use topics or pattern to specify topics to subscribe to. But that won't give you independent consumption for every topic.

Also from the code snippet you provided, you are using @Blocking which will use the shared worker thread pool for processing. You can read more in https://smallrye.io/smallrye-reactive-messaging/4.7.0/concepts/blocking/