1

In one of our spring-boot based services, we intended to connect to 2 different kafka clusters simultaneously. These clusters each have their own set of bootstrap-servers, topic configurations etc. They are nowhere related to each other as was the case in this question.

I will have different types messages to be read from each cluster on different topic names. There may or may not be multiple producers connecting to both the clusters from this service but we will surely have at least one consumer per cluster.

I would like to know how can I define properties in application.yml to cater to this setup so that I can just use 2 different KafkaProperties objects to create 4 container factories (2 consumer, 2 producer). The rest, I believe, should be pretty straight forward as I would need to use the relevant factory to create a particular container/listener/kafkaTemplate as per the business requirements.

Mukund Jalan
  • 1,145
  • 20
  • 39

1 Answers1

2

You cannot; you need to disable Boot's auto configuration and configure the infrastructure beans for each cluster yourself.

Boot's auto configuration only supports one cluster.

Gary Russell
  • 166,535
  • 14
  • 146
  • 179
  • In that case I would like to let spring boot auto-configure one cluster and do the other one manually. To achieve this, I was thinking of an approach where I could create another bean of `KafkaProperties` using `@Bean` & `@ConfigurationProperties("my-app.kafka")`, follow all the rules of `KafkaProperties` under `my-app.kafka` and use this property store to configure the 2nd cluster. Is it doable? If yes, are there any cons/cautions I need to know? – Mukund Jalan Jul 28 '20 at 06:35
  • 2
    It won't work; Boot will only configure, for example, a `ConsumerFactory` if there is not already a `ConsumerFactory` bean in the application context `@ConditionalOnMissingBean(ConsumerFactory.class)`. You have to declare both sets of infrastructure beans yourself. FYI, the boot team does not consider `KafkaProperties` a public API and it can change at any time so there is some (minimal) risk of using it directly. – Gary Russell Jul 28 '20 at 13:16
  • I also facing the same issue. @MukundJalan Have you able to connect two cluster from springboot? – user6044627 May 06 '22 at 07:10
  • @user6044627 It has been long since I was working on this requirement and do not have access to the code as well. As far as I can remember, our use case required connecting to a cluster most of times so we had it auto configured and for the other cluster we had manually specified config & beans which we used only where it was relevant. We had to ensure that all the beans for the 2nd cluster are marked to not be the primary ones or something similar (I don't recall fully) so that they don't end up being autowired in any scenario unless specified explicitly. – Mukund Jalan May 06 '22 at 07:52
  • If the configuration is mostly the same, you can now override the `bootstrap.servers` property (and others if necessary) on each `@KafkaListener` via the `properties` attribute. – Gary Russell Aug 20 '22 at 12:16