Stack Overflow
Stack Overflow

Questions tagged [spring-kafka]

The Spring for Apache Kafka (spring-kafka) project applies core Spring concepts to the development of Apache Kafka-based messaging solutions.

The Spring for Apache Kafka (spring-kafka) project applies core Spring concepts to the development of Apache Kafka-based messaging solutions. It provides a "template" as a high-level abstraction for sending messages. It also provides support for Message-driven POJOs with @KafkaListener annotations and a "listener container". These libraries promote the use of dependency injection and declarative. In all of these cases, you will see similarities to the JMS support in the Spring Framework and RabbitMQ support in Spring AMQP.

has a module spring-integration-kafka providing channel adapters and gateways based on spring-kafka.

Resources

See Also

4185 questions
1
vote
1 answer

Get underlying low-level Kafka consumers and Producers in Spring Cloud Stream

I have a usecase where I want to get the underlying Kafka producer (KafkaTemplate) in a Spring Cloud Stream application. While navigating the code I stumbled upon KafkaProducerMessageHandler which has a getKafkaTemplate method. However, it fails to…
cannot_mutably_borrow
  • 679
  • 8
  • 25
1
vote
2 answers

Spring boot Kafka Stream Listener message offset lost between restarts

We are using Spring boot kafka listener the prblem is whenever the pod stops or restarts it looses the offset and then stop processing the messages where it left, this happens with a single pod(if it goes down for some reason) as well as all the…
vaibhav
  • 3,929
  • 8
  • 45
  • 81
1
vote
1 answer

How to know when consumer.poll() is called while using @KafkaListener annotation?

I understand that I cannot control when to poll if I use @KafkaListener, and I read from this answer that the next poll() is performed after the last message from the previous poll has been processed by the listener. So I'm wondering how to know…
yifei
  • 561
  • 5
  • 18
1
vote
1 answer

Spring-Kafka with Micrometer throws NullPointerException during app shutdown

We have spring-boot app with multiple nodes, and use KafkaTemplate for producing kafka messages. On classpath we have both org.springframework.kafka:spring-kafka and io.micrometer:micrometer-core. In our case, messages produced continuously, around…
Vasyl Sarzhynskyi
  • 3,689
  • 2
  • 22
  • 55
1
vote
1 answer

How do I start Spring Cloud Stream Binders manually if autoStartup is set to false

I'm using spring-cloud-starter-stream-kafka to create a Kafka consumer binding and I've configured spring.cloud.stream.bindings.input.consumer.autoStartup to…
Chr3is
  • 387
  • 5
  • 21
1
vote
2 answers

Spring Kafka - Display Custom Error Message with @Retry

I am currently using Spring Kafka to consume messages from topic along with @Retry of Spring. So basically, I am retrying to process the consumer message in case of an error. But while doing so, I want to avoid the exception message thrown by…
Jestino Sam
  • 526
  • 2
  • 12
  • 31
1
vote
1 answer

Kafka + Spring locally broker may not be available. Windows 10

I'm having trouble configuring kafka and spring on Windows 10 machine. I did according to the guide, which I found on YouTube https://www.youtube.com/watch?v=IncG0_XSSBg&t=538s. I can't connect locally in any way. The spring application is very…
Grafik Krystian
  • 179
  • 2
  • 13
1
vote
1 answer

Does spring Kafka batch listener commits db transaction in batch mode and in case of failure is the complete transaction rolled back?

I have a simple requirement to read kafka messages and store in database. I am using spring kafka in batch listener mode. I have gone through spring kafka docs but still its not clear that when using spring kafka in batch listener mode, does it…
Ritesh
  • 25
  • 5
1
vote
1 answer

Handling commits for errors with @KafkaListener in batch consumers

We have a Kafka Consumer setup like below @Bean public ConsumerFactory consumerFactory() { final Map props = kafkaProperties.buildConsumerProperties(); return new DefaultKafkaConsumerFactory<>(props); …
Mukund Jalan
  • 1,145
  • 20
  • 39
1
vote
1 answer

Cannot publish to dead letter topic on transaction failed

In my application, I set up Kafka as following. @Configuration public class KafkaConfiguration { @Bean public ConcurrentKafkaListenerContainerFactory…
Mr.J4mes
  • 9,168
  • 9
  • 48
  • 90
1
vote
2 answers

Multiple StreamListeners to same Topic with Spring Cloud Stream connected to Kafka

I have Spring Boot app I am using Spring Cloud Stream connecting to Kafka. I'm trying to set up two separate stream listener methods to same kafka topic. @StreamListener("countries") @SendTo("aggregated-statistic") public KStream
Mykhailo Yablon
  • 25
  • 1
  • 6
1
vote
1 answer

ReplyingKafkaTemplate with KafkaTransactionManager raise Exception

I am developing a microservice that needs to make use of synchronous call under transaction with Spring Boot and Kafka.To do this, I use ReplyKafkaTemplate with KafkaTransactionManager. However ReplyKafkaTemplate apparently doesn't work with…
Jorge Avelar
  • 11
  • 3
1
vote
1 answer

Spring Kafka - deserialise Pojos without type information

I am working on a distributed microservices application that uses Kafka for internal communication. The applications exchange POJOs over topics. When a producer sends a message to consumer, a header is added by default indicating the package name…
greenhorn
  • 15
  • 4
1
vote
0 answers

mongoDB inserting twice when called on different Threads

Basically I am consuming Messages from spring cloud stream kafka and inserting it into the MongoDB My code works fine if my mongo cluster is up I have 2 problems In case My Mongo Instance is down auto commit of cloud stream is disabled…
Himanshu Parwani
  • 57
  • 4
1
vote
1 answer

Kafka Spring consumer offsets are not committed with ConsumerRecordRecoverer

Tech details: Versions: spring-boot : 2.2.2.RELEASE spring-kafka : 2.3.7.RELEASE kafka broker : 2.3.1 (via amazon MSK) Props: auto.offset.reset: earliest enable.auto.commit: false isolation.level: read_committed Issue and Behavior: I have a…
Madmartigan
  • 13
  • 1
  • 3
1 2 3
…
99 100