How to load Kafka Consumer lazily in Spring boot?

Question

I want to provide group Id through command line argument but when I tried this I got following error.

Failed to start bean 'org.springframework.kafka.config.internalKafkaListenerEndpointRegistry'; nested exception is java.lang.IllegalStateException: No group.id found in consumer config, container properties, or @KafkaListener annotation; a group.id is required when group management is used.

That means while loading kafkalistener it required group_id. If I gave groupId in consumerConfig file then Its working properly.

So is there any way so that I can give group Id through command line and kafka listener loads lazily So that I will not require while program starting.

My ConsumerConfig :

@Configuration
class KafkaConsumerConfig {

    @Value("${kafka.bootstrap-servers}")
    private String bootstrapServers;

    @Autowired
    private ArgumentModel argumentModel;

    private Logger logger = LoggerFactory.getLogger(KafkaConsumerConfig.class);

    @Bean
    public Map<String, Object> consumerConfigs() {
        Map<String, Object> props = new HashMap<>();
        logger.info("bootstrapServers : {}", bootstrapServers);
        props.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, bootstrapServers);
        props.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class);
        props.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class);
        props.put(ConsumerConfig.GROUP_ID_CONFIG, argumentModel.getKafkaGroupId());
        return props;
    }

    @Bean
    public ConsumerFactory<String, String> consumerFactory() {
        return new DefaultKafkaConsumerFactory<>(consumerConfigs());
    }

    @Bean
    public KafkaListenerContainerFactory<ConcurrentMessageListenerContainer<String, String>> kafkaListenerContainerFactory() {
        ConcurrentKafkaListenerContainerFactory<String, String> factory = new ConcurrentKafkaListenerContainerFactory<>();
        factory.setConsumerFactory(consumerFactory());
        return factory;
    }
}

What is ArgumentModel? Is it your custom code? If so, Why don't use use ConfigurationProperties that can be injected into the consumerConfigs bean? — Mark Bramnik, Aug 11 '20 at 10:44
In that scenario its working, But I want to pass through command line. So when program starts I read the value of kafkaGroupId and save it into model and then will use it here. But when I start spring program it directly gave me error. So is there any way so that I can load lazily my listener so that I will not give me error. — vijayk, Aug 11 '20 at 10:54
Passing stuff through command line doesn't mean that you can't use configuration properties and in general spring's configuration properties mechanism. You can run like this: `java -jar myapp.jar --kafka.groupId=someGroupHere` — Mark Bramnik, Aug 11 '20 at 10:56
Use a property placeholder in the `groupId` KafkaListener property. — Gary Russell, Aug 11 '20 at 14:25

score 2 · Accepted Answer · answered Aug 11 '20 at 14:25

2

@KafkaListener(... groupId = "${group.id}")

Then pass -Dgroup.id=myGroup on the command line.

answered Aug 11 '20 at 14:25

Gary Russell

166,535
14
146
179

It solved my group Id issue. If I want to provide consumer topic name from cmd then I should use : @KafkaListener(topics = "${consumer_topic}", groupId = "${group.id}") and pass -Dconsumer_topic=topic_name . Is this correct ? – vijayk Aug 12 '20 at 13:27
how can I pass kafka.bootstrap-servers values from command line ? right now i am passing through application.properties. – vijayk Jan 18 '21 at 12:05
1

You can pass any boot properties on the command-line with `--` (two dashes) - e.g. `java -jar target/ktest26-0.0.1-SNAPSHOT.jar --spring.kafka.bootstrap-servers=localhost:9092` – Gary Russell Jan 18 '21 at 17:44

How to load Kafka Consumer lazily in Spring boot?

1 Answers1