Stack Overflow
Stack Overflow

Questions tagged [spring-kafka]

The Spring for Apache Kafka (spring-kafka) project applies core Spring concepts to the development of Apache Kafka-based messaging solutions.

The Spring for Apache Kafka (spring-kafka) project applies core Spring concepts to the development of Apache Kafka-based messaging solutions. It provides a "template" as a high-level abstraction for sending messages. It also provides support for Message-driven POJOs with @KafkaListener annotations and a "listener container". These libraries promote the use of dependency injection and declarative. In all of these cases, you will see similarities to the JMS support in the Spring Framework and RabbitMQ support in Spring AMQP.

has a module spring-integration-kafka providing channel adapters and gateways based on spring-kafka.

Resources

See Also

4185 questions
20
votes
6 answers

Error to serialize message when sending to kafka topic

i need to test a message, which contains headers, so i need to use MessageBuilder, but I can not serialize. I tried adding the serialization settings on the producer props but it did not work. Can someone help me? this…
Tiago Costa
  • 1,004
  • 4
  • 17
  • 31
19
votes
5 answers

How to write Unit test for @KafkaListener?

Trying to figure out if I can write unit test for @KafkaListener using spring-kafka and spring-kafka-test. My Listener class. public class MyKafkaListener { @Autowired private MyMessageProcessor myMessageProcessor; …
Abbin Varghese
  • 2,422
  • 5
  • 28
  • 42
19
votes
3 answers

Kafka Streams with Spring Boot

I want to work with Kafka Streams real time processing in my spring boot project. So I need Kafka Streams configuration or I want to use KStreams or KTable, but I could not find example on the internet. I did producer and consumer now I want to…
Alpcan Yıldız
  • 671
  • 3
  • 13
  • 33
18
votes
2 answers

Spring-Kafka Concurrency Property

I am progressing on writing my first Kafka Consumer by using Spring-Kafka. Had a look at the different options provided by framework, and have few doubts on the same. Can someone please clarify below if you have already worked on it. Question - 1 :…
Akhil
  • 498
  • 2
  • 6
  • 22
18
votes
5 answers

Spring Kafka - How to reset offset to latest with a group id?

I am currently using Spring Integration Kafka to make real-time statistics. Though, the group name makes Kafka search all the previous values the listener didn't read. @Value("${kafka.consumer.group.id}") private String…
bachrc
  • 1,106
  • 1
  • 12
  • 20
18
votes
4 answers

How to use Spring Kafka's Acknowledgement.acknowledge() method for manual commit

I am using Spring Kafka first time and I am not able to use Acknowledgement.acknowledge() method for manual commit in my consumer code as mentioned here https://docs.spring.io/spring-kafka/reference/html/_reference.html#committing-offsets. Mine is…
kumarhimanshu449
  • 799
  • 3
  • 7
  • 19
18
votes
1 answer

Kafka error deserializing key/value for partition

I've got an integration tests that passes when I send to a Kafka topic without a key. However, when I add a key I start to get serialization errors. org.apache.kafka.common.errors.SerializationException: Error deserializing key/value for partition…
Lewis Watson
  • 543
  • 1
  • 4
  • 15
17
votes
1 answer

What is a listener container in Spring for Apache Kafka?

I understood that to make a method to be the target of Kafka message listener, I have to mark this method with the @KafkaListener annotation. This annotation lets specify by containerFactory element the KafkaListenerContainerFactory. Below there are…
Vin
  • 701
  • 1
  • 9
  • 30
17
votes
1 answer

Kafka consumer exception and offset commits

I've been trying to do some POC work for Spring Kafka. Specifically, I wanted to experiment with what are the best practices in terms of dealing with errors while consuming messages within Kafka. I am wondering if anyone is able to help…
yfl
  • 347
  • 2
  • 3
  • 10
16
votes
3 answers

Spring Kafka integration test Error while writing to highwatermark file

I'm writing integration test using spring-kaka-2.2.0 in spring boot application, I'm nearly succeeded still my test case return true but still I see multiple error after that. 2019-02-21 11:12:35.434 ERROR 5717 --- [ Thread-7]…
app
  • 733
  • 5
  • 15
  • 27
16
votes
1 answer

Spring Kafka-Difference between configuring KafkaTemplate with Producer Listener and registering a callback with Listenable Future

So I was going through the Spring kafka documentation and came across Producer Listener. This is what the Spring Kafka Documentation says- "Optionally, you can configure the KafkaTemplate with a ProducerListener to get an async callback with the…
Indraneel Bende
  • 3,196
  • 4
  • 24
  • 36
16
votes
1 answer

Why does a Kafka consumer take a long time to start consuming?

We start a Kafka consumer, listening on a topic which may not yet be created (topic auto creation is enabled though). Not long thereafter a producer is publishing messages on that topic. However, it takes some time for the consumer to notice this:…
Raf
  • 842
  • 1
  • 9
  • 25
16
votes
2 answers

Can a single Spring's KafkaConsumer listener listens to multiple topic?

Anyone know if a single listener can listens to multiple topic like below? I know just "topic1" works, what if I want to add additional topics? Can you please show example for both below? Thanks for the help! @KafkaListener(topics =…
R.C
  • 573
  • 3
  • 7
  • 19
15
votes
2 answers

Receiving Kafka Key in spring boot kafka listener

I am new in Spring Kafka. I have an microservice which sends message with a kafka key which is an user defined object. 1) First microservice sends message to Kafka with a key which is instance of MyKey object. 2) What I need to do is, to listen…
user1474111
  • 1,356
  • 3
  • 23
  • 47
15
votes
2 answers

How to write Unit test case for adding callback for ListenableFuture

I am trying to write the unit test case for ListenableFuture adding Callback but I am not sure how to do it. Didn`t get anything useful on internet. @Test public void can_publish_data_to_kafka() { String topic = someString(10); …
David
  • 507
  • 2
  • 6
  • 14
1
2
3
…
99 100