1

How can I read a message from Kafka topic on demand. I have the topic name, offsetId, PartitionID, using these three params, how can i retrieve a specific message from Kafka Topic. Is it possible using Spring Kafka ? I am using spring boot 2.2.4.RELEASE

AJK1305
  • 109
  • 2
  • 15

1 Answers1

4
  • create consumer
  • assign the topic/partition
  • seek
  • poll for one record
  • close consumer
@SpringBootApplication
public class So64759726Application {

    public static void main(String[] args) {
        SpringApplication.run(So64759726Application.class, args);
    }

    @Bean
    ApplicationRunner runner(ConsumerFactory<String, String> cf) {
        return args -> {
            try (Consumer<String, String> consumer = cf.createConsumer()) {
                TopicPartition tp = new TopicPartition("so64759726", 0);
                consumer.assign(Collections.singleton(tp));
                consumer.seek(tp, 2);
                ConsumerRecords<String, String> records = consumer.poll(Duration.ofSeconds(5));
                System.out.println(records.iterator().next().value());
            }
        };
    }

}

application.properties

spring.kafka.consumer.max-poll-records=1

UPDATE

Since this answer was posted, the KafkaTemplate now has receive() methods for on-demand consumption.

https://docs.spring.io/spring-kafka/docs/current/reference/html/#kafka-template-receive

ConsumerRecord<K, V> receive(String topic, int partition, long offset);

ConsumerRecord<K, V> receive(String topic, int partition, long offset, Duration pollTimeout);

ConsumerRecords<K, V> receive(Collection<TopicPartitionOffset> requested);

ConsumerRecords<K, V> receive(Collection<TopicPartitionOffset> requested, Duration pollTimeout);
Gary Russell
  • 166,535
  • 14
  • 146
  • 179
  • Thanks Gary. I assume i do not need to do any of those container creation configurations which we do for a generic KafkaListener. Will this be sufficient to pull a particular message. If i need to configure SSL params, where do i do that – AJK1305 Nov 10 '20 at 14:30
  • 1
    Correct; what I show above is the complete application. SSL settings go in `application.properties/application.yml` as normal. https://docs.spring.io/spring-boot/docs/current/reference/html/appendix-application-properties.html#integration-properties - scroll down to the `spring.kafka.ssl...` properties. – Gary Russell Nov 10 '20 at 14:49
  • 1
    This is exactly what I was looking for. Is it possible to use ConcurrentKafkaListenerContainerFactory instead of consumerFactory to create the consumer instances? – perplexedDev Feb 22 '22 at 20:11
  • 1
    That doesn't make any sense; the container factory creates containers; those containers subsequently use the consumer factory to create consumers for message-driven consumption. By the way, after this answer was posted, the `KafkaTemplate` now has `receive()` operations for on-demand consumption. https://docs.spring.io/spring-kafka/docs/current/reference/html/#kafka-template-receive – Gary Russell Feb 22 '22 at 20:27
  • This seems bit counterintuitive to me. Kafkatemplate needs a "producerfactory" in constructor but helps in message "consumption" with receive() API ? Am i missing or misunderstanding something here ? – Chetan Gowda May 04 '22 at 05:30
  • For the vast majority of users, the template is used for publishing only. To use the receive methods you need to also add a consumer factory (`setConsumerFactory()`). This functionality was added here for consistency with other templates in the Spring portfolio that also support send and receive operations; `JmsTemplate`, `RabbitTemplate`, etc. – Gary Russell May 04 '22 at 13:15