2

I have two cluster setup cluster-1 at site 1(3 brokers) and cluster-2 at site 2(3 brokers) . Using spring kafka(1.3.6) consumer( one machine) and listening messages through @KafkaListener annotation . How do we instantiate multiple KafkaListenerContainerFactory for each cluster (c1 and c2) and listening data from both the cluster at the same time.

My listener should consume messages from both the cluster at the same time .

Selva
  • 45
  • 1
  • 6

1 Answers1

2

Kafka will automatically connect to both instances, there is nothing more you need to do.

Simply add both addresses to the bootstrap.servers config so the client can connect even if one of the brokers is down (but partitions must be sufficiently replicated).

Gary Russell
  • 166,535
  • 14
  • 146
  • 179
  • both the clusters are having same topics in different site and ip address will not be known initially and site will get added dynamically up to 7 sites.However one client m/c consume message from all the cluster with same set of topic & partition think we must have to instantiate diff ConcurrentKafkaListenerContainerFactory for each site dynamically . could you pls provide detail how do we achieve it. – Selva Sep 24 '18 at 17:14
  • That is not correct; you don't need multiple consumer or producer factories; Kafka will take care of it all. – Gary Russell Sep 24 '18 at 17:50
  • How do we pass cluster (brokers ips) to consumer config dynamically?for e.g. If have consumer then will it read messages from partition (p0) in cluster 1 ,2..7. – Selva Sep 24 '18 at 18:00
  • Kafka does it automatically; you need to know the address of at least one broker that is running. – Gary Russell Sep 24 '18 at 18:44
  • I think there is a misunderstanding on the question. we need to consume messages across cluster. initially cluster1(3 brokers,3 topic,3 partiotion each) will be created at city A and same set up will be created at City B. Consumer has to read messages from both the cluster. We have created consumer config with city-A cluster . How do I create for other Cluster city -B likewise have upto 7 cluster.If I have one cluster and multiple broker then no issue kafka does it automatically . Here not sure How do we do it across cluster**s** . – Selva Sep 25 '18 at 02:03
  • 1
    Sorry - I though you were talking about multiple instances within a cluster. A listener can't communicate with multiple clusters - you need a separate consumer factory and listener container factory (and listener) for each. – Gary Russell Sep 25 '18 at 12:25
  • NP . could you please provide an example it will help me a lot. not sure how to instantiate ConcurrentKafkaListenerContainerFactory dynamically and set message listener . I have 3 topics and 2 partition each and 1 cluster initially and it will grow dynamically . – Selva Sep 25 '18 at 14:04
  • `@KafkaListener` and supporting infrastructure are not designed to be dynamically added at runtime; probably the easiest way would be to create a child application context for each instance. – Gary Russell Sep 25 '18 at 15:42
  • What if I create multiple messagelistnercintainer with diff consumer config(ip is different) and new instance of messagelistener. So I have 7 instance of messagelistnercintainer and messagelistener. Will it work?? I am not sure on child application context any reference pls – Selva Sep 25 '18 at 16:01
  • it worked with multiple messagelistnercintainer with diff consumer config. – Selva Oct 01 '18 at 16:18
  • @Selva have you been able to resolve this issue? If yes how? – Alchemist Apr 23 '20 at 07:59
  • This is a very old answer; simply add 2 `@KafkaListener`s to your method, with each one pointing to a different container factory. If you need more help with this; ask a new question. – Gary Russell Apr 23 '20 at 14:04