9

I have a listener that need to read from multiple kafka servers with the same topic which are all configured under one zookeeper. How do I read from those multiple servers. Can you please help with this.

Instead of Kafka servers, can I point to zookeeper instead?

daemon54
  • 1,057
  • 3
  • 16
  • 36

2 Answers2

10

The @KafkaListener requires KafkaListenerContainerFactory @Bean, which, in turn, is based on the ConsumerFactory. And the DefaultKafkaConsumerFactory accepts a Map<String, Object> of consumer configs:

@Configuration
@EnableKafka
public class KafkaConfig {

    @Bean
    KafkaListenerContainerFactory<ConcurrentMessageListenerContainer<Integer, String>>
                        kafkaListenerContainerFactory() {
        ConcurrentKafkaListenerContainerFactory<Integer, String> factory =
                                new ConcurrentKafkaListenerContainerFactory<>();
        factory.setConsumerFactory(consumerFactory());
        return factory;
    }

    @Bean
    public ConsumerFactory<Integer, String> consumerFactory() {
        return new DefaultKafkaConsumerFactory<>(consumerConfigs());
    }

    @Bean
    public Map<String, Object> consumerConfigs() {
        Map<String, Object> props = new HashMap<>();
        props.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, ...);
        ...
        return props;
    }
}

https://docs.spring.io/spring-kafka/docs/1.2.2.RELEASE/reference/html/_reference.html#__kafkalistener_annotation

Where that ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG is exactly standard Apache Kafka bootstrap.servers property:

A list of host/port pairs to use for establishing the initial connection to the Kafka cluster. The client will make use of all servers irrespective of which servers are specified here for bootstrapping—this list only impacts the initial hosts used to discover the full set of servers. This list should be in the form host1:port1,host2:port2,.... Since these servers are just used for the initial connection to discover the full cluster membership (which may change dynamically), this list need not contain the full set of servers (you may want more than one, though, in case a server is down).

No, you can't point Zookeeper address. That isn't supported by Kafka any more.

Artem Bilan
  • 113,505
  • 11
  • 91
  • 118
  • 1
    The new (0.9+) client objective is to avoid client code having to talk to zookeeper. You could, however, (I presume) query zookeeper yourself to get the broker list and populate the bootstrap servers config property. – Gary Russell Sep 19 '17 at 13:48
  • Thanks Artem. Also I believe Kafka has a topic called _consumer-offsets using which offset management between server and client is handled. So if I use multiple servers corresponding to multiple zookeepers, Will spring-kafka handle particularly the offset management. – daemon54 Sep 19 '17 at 13:52
  • 1
    Spring Kafka doesn't manage offsets. Everything is done on the Kafka Broker. We only provide `consumer.group` and rely on the offset provided for the Consumer instance. – Artem Bilan Sep 19 '17 at 14:19
  • Will I be able to use multiple servers from different Kafka clusters with same topic name? – daemon54 Sep 19 '17 at 16:45
  • 1
    I don't think so. Those `bootstrap.servers` are really must be in the same Kafka cluster. Even if your topic name is the same, physically they are different objects because they are in different clusters. – Artem Bilan Sep 19 '17 at 16:47
  • Are you sure about that Artem? Did you try/read it somewhere or is that just what you think? – tomer.z Jan 16 '18 at 21:22
  • What do you mean? The Kafka doc clearly states with the: `A list of host/port pairs to use for establishing the initial connection to the Kafka cluster.`. So, all the brokers must be in the same cluster. Am I missing anything? And right, I didn't try that because my task not to find bugs in Apache Kafka – Artem Bilan Jan 16 '18 at 21:31
3

The map has key: String and Value: Object.

The Object of key:ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG should be series of host-port pair seperated by ',' like: host1:port1,host2:port2,host3:port3 ......

e.g for listening three servers: localhost:9092,192.168.22.12:9088,localhost:7898

Karthikumar
  • 71
  • 5