1

Currently I have a Quarkus app which consumes from a Kafka topic and produces on another Kafka topic. It uses SmallRye Reactive Messaging. It works well. Due to external changes the topic to produce on and the topic to consume from will be on Kafka servers which are on a different cluster (and should not/cannot be combined in one cluster).

In the application configuration (yaml) we set the Kafka server (broker):

kafka:
  bootstrap:
    servers: localhost:9092

Adding a server here does not help, it then tries to spread data over the brokers which is not my intention.

Is it possible to connect to multiple clusters (maybe set a server per topic)? Can't find anything on that on internet not in the Quarkus docs nor in the SmallRye docs.

Triple S
  • 96
  • 1
  • 4

2 Answers2

5

It's not crystal clear but it's explained on the documentation for exemple on the kafka inbound documentation it is said that you can configure your broker gloablly (using kafka.<props>) or per channel.

You can pass to each channel all supported Kafka properties :

mp.messaging.incoming.chanel1.bootstrap.servers=kafka1:9092
mp.messaging.incoming.chanel2.bootstrap.servers=kafka2:9092
loicmathieu
  • 5,181
  • 26
  • 31
  • That would be great, I see you could read the text like that. When doing this and skipping the bootstrap server at the main level I am getting DeploymentException: `No config value of type [java.lang.String] exists for: kafka.bootstrap.servers`. Though this would be what the docs appear to be saying one should do. When I keep also specifying the server at the mail level (ie. both kafka1:9092, kafka2:9092), I am getting errors stating `Error while fetching metadata with correlation id <725> : {=LEADER_NOT_AVAILABLE` – Triple S Jun 03 '21 at 09:15
  • the bootstrap server can be configured on every channel as shown in the answer. "kafka.bootstrap.servers" defines the default one if you don't configure it explicitly on the channel. – Clement Jun 03 '21 at 09:20
  • If you have the error `No config value of type [java.lang.String] exists for: kafka.bootstrap.servers` this means something else needs the configuration. Do you inject a KafkaClient somewhere or enable Kafka health check ? For `Error while fetching metadata with correlation id <725> : {=LEADER_NOT_AVAILABLE` there seems to be a connection issue somewhere, without a reproducer it's hard to tell. – loicmathieu Jun 03 '21 at 14:37
0

Remove the below property from property file

kafka.bootstrap.servers

Next add the below property . What we are doing here is instead of setting bootstrap servers globally for all your channel in your application, we are setting for individual channel .

mp.messaging.incoming.channel1_name.bootstrap.servers=kafka1:9092 mp.messaging.incoming.channel2_name.bootstrap.servers=kafka1:9092

Once you set the above property in your application.properties and run it will show docker container failed to start ... something like that and a warn message like No config value of type [java.lang.String] exists for: kafka.bootstrap.servers.

This is due to dev services that is active. For this to go away we need to add one more property.

quarkus.devservices.enabled = false

now run the application and it should resolve the issue .

Praveen Gowthaman
  • 67
  • 5