0

I am using the Quarkus Kafka extension, but not in the intended way, as I need to connect to multiple different Kafka brokers to transfer messages between them - which the extension doesn't support. As such I am using the plain Kafka API to connect to these brokers. For this, I have set up multiple configuration properties which have the "bootstrap.servers" for each broker I need to connect to:

my-app.broker-one-properties."bootstrap.servers"=kafkaone.example.com:9092
my-app.broker-two-properties."bootstrap.servers"=kafkatwo.example.com:9092

This all works fine, however for my automated tests, I would actually like to the Quarkus Kafka devservices to start up a RedPanda container before running my tests (https://quarkus.io/guides/kafka#kafka-dev-services). The extension starts a RedPanda at a random port which is what I want. For tests it is ok to use the same instance for all brokers, so I would like to use "bootstrap.servers" from the devservices during test.

I tried something like:

%test.my-app.broker-one-properties."bootstrap.servers"=${kafka.bootstrap.servers}
%test.my-app.broker-two-properties."bootstrap.servers"=${kafka.bootstrap.servers}

in the hope that I could get the correct configuration into my own service this way but this doesn't seem to work this way. So how could I find out which port the devservices are using to feed this to my own configuration?

Jan Thomä
  • 13,296
  • 6
  • 55
  • 83

1 Answers1

2

The Quarkus Reactive Messaging Kafka connector supports having multiple brokers. You need to configure the bootstrap servers per channel:

mp.messaging.incoming.channel-in.bootstrap.servers=broker1:9092
mp.messaging.outgoing.channel-out.bootstrap.servers=broker2:9092

A simple method like this will read from one and write to the other one:

@Incoming("channel-in")
@outgoing("channel-out")
public X transfer(X x) {
   return x;
}

Now, about the dev service, you are right; it only starts with a single broker. However, for your tests (and only tests, not for the dev mode), you can write a custom QuarkusTestResource, starting 2 brokers and passing the configuration to the application.

%test.mp.messaging.incoming.channel-in.bootstrap.servers=${broker1}
%test.mp.messaging.outgoing.channel-out.bootstrap.servers=${broker2}

Here is an example of Quarkus Test Resources starting a Kafka broker: https://github.com/quarkusio/quarkus/blob/main/integration-tests/kafka-snappy/src/test/java/io/quarkus/it/kafka/KafkaTestResource.java.

Create your own, starting 2 brokers and passing broker1 and broker2 as passed configuration (instead of kafka.bootstrap.servers).

Finally, your test needs to be annotated with @QuarkusTestResource(MyKafkaTestResource.class)

Clement
  • 2,817
  • 1
  • 12
  • 11
  • I have the additional complication that one incoming message needs to be written to two different outgoing topics and that the incoming message only may be acknowledged once it is confirmed written by both outgoing targets. As far as I understand this is not supported by the reactive messaging, hence the custom implementation. – Jan Thomä Jul 20 '23 at 14:36
  • Even in that case, you can use reactive messaging. You just need to use "Message" and (at least until the next version) Emitters to write to the two output channels. Then, coordinate the acknowledgment and you are done. – Clement Jul 24 '23 at 05:56
  • That is good to know, I will give it a spin. Thank you very much! – Jan Thomä Jul 25 '23 at 08:03