Stack Overflow
Stack Overflow

Questions tagged [spring-kafka]

The Spring for Apache Kafka (spring-kafka) project applies core Spring concepts to the development of Apache Kafka-based messaging solutions.

The Spring for Apache Kafka (spring-kafka) project applies core Spring concepts to the development of Apache Kafka-based messaging solutions. It provides a "template" as a high-level abstraction for sending messages. It also provides support for Message-driven POJOs with @KafkaListener annotations and a "listener container". These libraries promote the use of dependency injection and declarative. In all of these cases, you will see similarities to the JMS support in the Spring Framework and RabbitMQ support in Spring AMQP.

has a module spring-integration-kafka providing channel adapters and gateways based on spring-kafka.

Resources

See Also

4185 questions
10
votes
1 answer

Kafka - Deserializing the object in Consumer

We are considering to use Kafka in our for messaging and our applications are developed using Spring. So, we have planned to use spring-kafka. The producer puts the message as HashMap object into the queue. We have JSON serializer and we assumed…
Thiru
  • 2,541
  • 4
  • 25
  • 39
10
votes
2 answers

Tombstone messages not removing record from KTable state store?

I am creating KTable processing data from KStream. But when I trigger a tombstone messages with key and null payload, it is not removing message from KTable. sample - public KStream processRecord(@Input(Channel.TEST)…
R K
  • 382
  • 5
  • 25
10
votes
3 answers

Spring @KafkaListener execute and poll records after certain interval

We wanted to consume the records after a certain interval (e.g. every 5 minutes). Consumer properties are standard: @Bean public KafkaListenerContainerFactory> kafkaListenerContainerFactory() { …
Sandeep B
  • 765
  • 1
  • 6
  • 19
10
votes
2 answers

Spring Boot Kafka: Unable to start consumer due to NoSuchBeanDefinitionException

With in my spring boot service while trying to start a kafka consumer seeing NoSuchBeanDefinitionException and unable to start the service itself. Below is my bean class which has all the required beans created for Kafka configuration Spring Boot…
shoaib1992
  • 410
  • 1
  • 8
  • 26
10
votes
3 answers

Inject ObjectMapper into Spring Kafka serialiser/deserialiser

I'm using Spring Kafka 1.1.2-RELEASE with Spring Boot 1.5.0 RC and I have configured a custom value serialiser/deserialiser class extending…
wjans
  • 10,009
  • 5
  • 32
  • 43
9
votes
1 answer

Implement Reactive Kafka Listener in Spring Boot application

I'm trying to implement reactive kafka consumer in my Spring boot application and I'm looking at these examples: https://github.com/reactor/reactor-kafka/blob/master/reactor-kafka-samples/src/main/java/reactor/kafka/samples/SampleScenarios.java and…
bobryash
  • 129
  • 1
  • 8
9
votes
1 answer

Kafka: what is the point of using "acknowledgment.nack" if I can simply "not acknowledgment.acknowledge"

Regard new feature of Kafka aimed for negative acknowledgement and now supported by Spring-Kafka, according to /spring-kafka/docs/2.4.4.RELEASE/ "... Starting with version 2.3, the Acknowledgment interface has two additional methods nack(long sleep)…
Jim C
  • 3,957
  • 25
  • 85
  • 162
9
votes
1 answer

What is the difference between MANUAL and MANUAL_IMMEDIATE in spring-kafka AckMode

From spring-docs, I can see MANUAL - the message listener is responsible to acknowledge() the Acknowledgment; after which, the same semantics as BATCH are applied. MANUAL_IMMEDIATE - commit the offset immediately when the…
Abbin Varghese
  • 2,422
  • 5
  • 28
  • 42
9
votes
2 answers

AWS Kafka (MSK) - How to generate Keystore and truststore and use the same in my Spring Cloud Stream application?

Is there any info as to how can I use the AWS MSK details in my Spring Cloud Stream application ? I believe we need to generate a keystore and truststore and then incorporate the same in our application ? I went through the "Client Authentication"…
Kenny Weeler
  • 153
  • 1
  • 3
  • 15
9
votes
1 answer

Spring Kafka : Record listener vs Batch listener

With spring-kafka, there is two types of Kafka listeners. Record Listeners : @KafkaListener(groupId = "group1", topics = {"my.topic"}) public void listenSingle(String message, @Header(KafkaHeaders.RECEIVED_TOPIC) String topic) { /* Process my…
Dynamite
  • 389
  • 5
  • 17
9
votes
2 answers

Spring @KafkaListener and concurrency

I am working with spring boot + spring @KafkaListener. And the behavior I expect is: my kafka listener reads messages in 10 threads. So that, if one of threads hangs, other messages are would continue reading and handling messages. I defined bean…
Sviatlana
  • 1,728
  • 6
  • 27
  • 55
9
votes
2 answers

how to send batched data with Spring Kafka producer

Currently I have code like this: KafkaTemplate kafkaTemplate; List myData; for(Pet p: myData) { String json = objectWriter.writeValueAsString(p) kafkaTemplate.send(topic, json) } so each list item is sent one by one. How…
pdeva
  • 43,605
  • 46
  • 133
  • 171
9
votes
1 answer

how to pause and resume @KafkaListener using spring-kafka

I have implemented the Kafka consumer, now I have a scenario. Read data from the Kafka stream 2.2.5.Release via Srpingboot load in the database table1 copy the data from table1 to table2 clear the table1 To do the above things, I need to…
vicky
  • 890
  • 4
  • 23
  • 54
9
votes
2 answers

How to fix kafka.common.errors.TimeoutException: Expiring 1 record(s) xxx ms has passed since batch creation plus linger time

I am using kafka_2.11-2.1.1 and Producer using spring 2.1.0.RELEASE. I am using spring while I am sending the messages to Kafka topic my producer generates a lot of TimeoutExceptions org.apache.kafka.common.errors.TimeoutException: Expiring 1…
BdEngineer
  • 2,929
  • 4
  • 49
  • 85
9
votes
1 answer

How to set groupId to null in @KafkaListeners

In relation with : this question I'm trying to read a compacted topic via a @KafkaListener. I'd like every consumers to read the whole topic each time. I can not generate an unique groupId each for each consumers. So I wanted to use a null…
JohnD
  • 395
  • 1
  • 6
  • 18
1 2 3
…
99 100