Stack Overflow
Stack Overflow

Questions tagged [spring-kafka]

The Spring for Apache Kafka (spring-kafka) project applies core Spring concepts to the development of Apache Kafka-based messaging solutions.

The Spring for Apache Kafka (spring-kafka) project applies core Spring concepts to the development of Apache Kafka-based messaging solutions. It provides a "template" as a high-level abstraction for sending messages. It also provides support for Message-driven POJOs with @KafkaListener annotations and a "listener container". These libraries promote the use of dependency injection and declarative. In all of these cases, you will see similarities to the JMS support in the Spring Framework and RabbitMQ support in Spring AMQP.

has a module spring-integration-kafka providing channel adapters and gateways based on spring-kafka.

Resources

See Also

4185 questions
1
vote
1 answer

Kafka No Acknowledgment available as an argument

we are trying to implement Kafka Acknowledgment in java spring project. Without the Acknowledgment, we successfully receive and read the message but when we add the Acknowledge in the method we receive this…
Liverpool
  • 265
  • 7
  • 21
1
vote
1 answer

Spring cloud stream unexpected shutdown is not covered by DLQ

We are using Spring Cloud Stream 2.2 with the Kafka binder. Something we have noticed is if the pod is killed in the middle of doing the job for whatever reason, then we will miss the message to be sent to DLQ. We are managing exceptions by catching…
Ali
  • 1,759
  • 2
  • 32
  • 69
1
vote
1 answer

Spring Integration Kafka - Exception Handling by using SeekToCurrentErrorHandler

Problem Statement : Need to handle exceptions occur while consuming messages in kafka Commit failed offset Seek to the next unprocessed offset, so that next polling start from this offset. Seems all these are handled as part of…
vinod kumar
  • 11
  • 2
1
vote
1 answer

KafkaProducer InterruptedException during gracefull shutdown on spring boot application

For a project we are sending some events to kafka. We use spring-kafka 2.6.2. Due to usage of spring-vault we have to restart/kill the application before the end of credentials lease (application is automatically restarted by kubernetes). Our…
fercam
  • 21
  • 1
  • 6
1
vote
1 answer

How can I disable KafkaAdmin when using spring boot

Currently I'm using Spring boot 2.4.0 with spring-kafka. I want to use consumers and producers but not kafka admin. I tried to override KafkaAdmin class by settings "bootstrap.servers" to null, but doesn't work. It still 'try' to connect to make…
Roeniss
  • 386
  • 5
  • 16
1
vote
0 answers

Cannot consume message

I have an API which takes a Message object as input produces a message to Kafka. I wrote a consumer aswell, which consumes the message and saves it to the db. My producer works as intended but the consumer does not work, For now I wanted to log a…
Yasin Eraslan
  • 171
  • 2
  • 13
1
vote
1 answer

spring kafka message processing telemetry

I'm trying to create some kind of "kafka message processing graph" - which service is consuming which topics and what messages - with some additional metadata(processing duration, whether it was processed OK or it ended with exception,...). I could…
mawek
  • 679
  • 10
  • 19
1
vote
3 answers

How to control the number of messages that being emitted by Apache Kafka per a specific time?

I am new to Apache Kafka and I am trying to configure Apache Kafka that it receives messages from the producer as much as possible but it only sends to the consumer configured number of messages per specific time. In other words How to configure…
Ahmed Naguib
  • 19
  • 5
1
vote
1 answer

KafkaListener not resolving containerFactory with "#{__listener.getContainerFactory()}"

I am using the following methods to get the KafkaListener properties. But it does not resolve the containerFactory value. But it works for topics and groupId. @KafkaListener(topics = "#{__listener.getTopics()}", groupId =…
Prakash Gopinathan
  • 11
  • 2
1
vote
1 answer

How to reuse Spring Kafka code to setup multiple listeners?

If my application has one listener, the case is dead simple: I will configure spring.kafka.*, Spring Boot will parse the configuration to KafkaPropeties and initialize all required beans. But I have two listeners, let say one saves records to file…
michaldo
  • 4,195
  • 1
  • 39
  • 65
1
vote
1 answer

Conditional (content-based) routing on Kafka headers in Spring Cloud Stream

I'm using Spring Cloud Stream 3.x in Spring Boot 2.x application to consume messages from a Kafka topic. I want to have a listener that consumes messages conditionally on some custom header value, as per doc: @StreamListener(value = "someTopic",…
Oleg Efimov
  • 632
  • 1
  • 7
  • 17
1
vote
2 answers

Overriding KStreams default serializer (ByteArraySerializer)

I can't seem to override the serializer of a topic to Serdes.String(). I'm trying a simple use case of reading from a topic (stream), and writing to a KTable. What I have so far: @Component class Processor { @Autowired public void…
Tiberiu
  • 990
  • 2
  • 18
  • 36
1
vote
1 answer

KafkaListener mulitple topics in yaml file

I want to read from multiple topics, so i declared them in yaml file with comma separated but getting below error: java.lang.IllegalStateException: Topic(s) [ topic-1 , topic-2, topic-3, topic-4, topic-5, topic-6, topic-7] is/are not present…
VKR
  • 195
  • 4
  • 18
1
vote
0 answers

What is the right configuration for Kafka with long processing messages and horizontally scaling

My application consumes messages that their processing duration average is between 5-15 min for 1 message with max.poll.records: 1. When no messages there is 1 pod (K8S) and for each incoming message I'm scaling the pod adding 1 pod up to a maximum…
Udi
  • 598
  • 8
  • 19
1
vote
1 answer

Spring Kafka Consumer Configs - default values and at least once semantics

I am writing kafka consumer using spring-kafka template. When I am instantiating consumers, Spring kafka takes in parameters like the following. props.put(ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG,…
Raj V
  • 87
  • 12
1 2 3
…
99 100