1

I am a bit new to Kafka and reading through documentation. The Kafka office site has an example on KStream. Where the application is bound to a topic and as soon as the message arrives its processed. The results are posted back to topic or databases.

Spring Kafka annotation @KafkaListener does the same functionality. For example, I tried my hands on KafaListner application. In here as well, we listen to a topic and process it when something is posted.

So I was curious to know 1. How these 2 are different? 2. Which one to prefer in which scenario?

Shankar
  • 2,625
  • 3
  • 25
  • 49
Pavan_opencodez
  • 41
  • 1
  • 2
  • 1
    Possible duplicate of [Kafka: Consumer API vs Stream API](https://stackoverflow.com/questions/44014975/kafka-consumer-api-vs-stream-api) – donm Jan 18 '18 at 22:05

2 Answers2

5

Please note that this is is a very limited explanation. Refer the docs.

To answer your question 1 "How these 2 are different?" - Both KafkaListener and KStream consume messages from Kafka topics. However they differ in the way they maintain state. The KafkaListener does not maintain state. It consumes messages as it comes. KStream reads the topic as a continuous Stream of messages.

Lets assume that a topic sends lines and we maintain a count of the number of each word. So after we send the topic these 2 lines,

Hello good morning, Hello thanks

We will have the word counts - Hello 2, good 1, morning 1 & thanks 1.

KakfaListener can be used to keep this word count manually. The developer can store the words in a static Hashmap and keep the count. KStream will do it naturally because it reads the topic as a stream -

it is designed to operate on an infinite, unbounded stream of data

The KStream example explains this in good detail.

To answer your question 2 "Which one to prefer in which scenario?", Use KafkaListener if you need to consume messages without maintaining state..like a pipeline, to take info from source to sink. Use KStream if your messages are related to each other - like find total number of a particular word in all the messages (roughly similar to a GROUP BY in SQL).

Shankar
  • 2,625
  • 3
  • 25
  • 49
2

@KafkaListener is not using KStream (Stream API). @KafkaListener is an annotation from spring-kafka which uses Consumer API internally. KStream is not available in Consumer API, it's available in Stream API.

For differences between Stream and Consumer APIs, check out question linked in comment to your question. Just remember one thing, spring-kafka library wraps Kafka libraries, so you have available four APIs: Stream API wrapped by spring-kafka, Consumer API wrapped by spring-kafka, Stream API and Consumer API. Two examples you mentioned are: Stream API and Consumer API wrapped by spring-kafka.

ctomek
  • 1,696
  • 16
  • 35
  • 1
    _KStream is not available in Consumer API, it's available in Stream API._ True, because KStream builds on top of Consumer and Producer API. Please also see here to understand the difference between consumer API and streams API: https://stackoverflow.com/questions/44014975/kafka-consumer-api-vs-streams-api In that case the KafkaListener of Spring would go into the same direction Streams API does and would build upon Consumer API a convention over configuration library to process Kafka messages – hasan Aug 21 '19 at 11:47