Stack Overflow
Stack Overflow

Questions tagged [spring-kafka]

The Spring for Apache Kafka (spring-kafka) project applies core Spring concepts to the development of Apache Kafka-based messaging solutions.

The Spring for Apache Kafka (spring-kafka) project applies core Spring concepts to the development of Apache Kafka-based messaging solutions. It provides a "template" as a high-level abstraction for sending messages. It also provides support for Message-driven POJOs with @KafkaListener annotations and a "listener container". These libraries promote the use of dependency injection and declarative. In all of these cases, you will see similarities to the JMS support in the Spring Framework and RabbitMQ support in Spring AMQP.

has a module spring-integration-kafka providing channel adapters and gateways based on spring-kafka.

Resources

See Also

4185 questions
1
vote
2 answers

Getting PKIX path building failed Exception with Apache Camel while connecting to the Kafka cluster with SSL

while I'm connecting to the Kafka cluster with apache camel with SSL then I'm facing the below problem, can anyone please help to resolve the issue javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building…
Srikanth Janapati
  • 393
  • 3
  • 16
1
vote
1 answer

Spring kafka integration properties for batch listener using application.yml/properties

I am trying to use kafka consumer batching in spring boot app. I could see examples, where we have a kafka config class in which, KafkaListenerContainerFactory is configured and ConcurrentKafkaListenerContainerFactory.setBatchListener(true); is…
Adil Khan
  • 144
  • 9
1
vote
1 answer

Pause and Resume KafkaConsumer

What I've to do is pause the KafkaConsumer if during message consuming an error is thrown. This is what I wrote @KafkaListener(...) public void consume( @Header(KafkaHeaders.CONSUMER) KafkaConsumer consumer, @Payload String message)…
Vin
  • 701
  • 1
  • 9
  • 30
1
vote
1 answer

spring-kafka: Use Interactive queries in Spring Boot

What's an «idiomatic» way of configuring (providing host and port info) and accessing Interactive Queries in Kafka Streams + Spring Boot application? What's the correct way to access KafkaStreams instance to access the state stores? I'm aware of…
Vik Gamov
  • 5,446
  • 1
  • 26
  • 46
1
vote
0 answers

Reactor Kafka consumer continues to consume messages after the partitions are revoked

When the node where I am running io.projectreactor.kafka:reactor-kafka:1.1.0.RELEASE gets overloaded sometimes it happens that the consumer loses partitions. I am guessing that it can't manage to send heartbeats. This is not the problem I need help…
Chobicus
  • 2,024
  • 2
  • 17
  • 26
1
vote
0 answers

Trace and Span information lost by default SeekToCurrentErrorHandler

When a consumer fails to process a record, the SeekToCurrentErrorHandler kicks in and does its job. After 10 retries, I can see a log line that the backoff was exhausted for the erroring record, however the logged record now has no headers. On…
Rhys Saldanha
  • 21
  • 3
1
vote
1 answer

Kafka consumer health check and recreate

We use Spring Kafka Client on a project. Recently we noticed that if a Kafka Consumer dies due to OutOfMemoryError service continues operating normally and no new consumers get created. The only way to fix this is to monitor OOM in logs and restart…
Kiryl Klachkou
  • 23
  • 7
1
vote
0 answers

DefaultKafkaHeaderMapper Could not decode json type

I am using spring-kafka, my producer is org.apache.kafka:kafka-clients:2.3.12 and consumer is org.springframework.kafka:spring-kafka:2.1.7.RELEASE I am getting the following error on the consumer…
CruncherBigData
  • 1,112
  • 3
  • 14
  • 34
1
vote
1 answer

How to Store failed data on sending to topic in spring cloud stream kafka

I have configured 3 brokers in Kafka running on different ports .I am using spring cloud stream kafka brokers: localhost:9092,localhost:9093,localhost:9094. I am creating a data pipeline that gets continuous stream of data .I am storing stream…
abishkar bhattarai
  • 7,371
  • 8
  • 49
  • 66
1
vote
0 answers

Kafka streams send to topic in transactional manner

Post reading messages one by one into the KStreams, we apply some filtering and transformation and then produce it to the SINK topic. Now, our scenario transformation is little complex task. Is there a way to make this portion of work (i.e.…
Aditya Goel
  • 201
  • 1
  • 15
1
vote
0 answers

Spring boot Kafka stream listener same message consumed multiple times by seperate PODs/Consumers in the same consumer group

Using Kafka and Spring boot kafka stream listener in my application. The configurations are: 160 partitions and the HPA is set to 6-40 so under peak load the pods scale to 40. The problem is that a single message on a single offset is getting…
vaibhav
  • 3,929
  • 8
  • 45
  • 81
1
vote
1 answer

Replies are not always delivered to the desired reply listener container

We are applying the request/reply semantics using spring-kafka with the ReplyingKafkaTemplate. However, we noticed that sometimes the reply does not end up where it should. Here is a rough description of our setup: Service A 2 instances consuming…
BitfulByte
  • 4,117
  • 1
  • 30
  • 40
1
vote
0 answers

Database Transaction in @KafkaListener method

I'm reading a ton of questions and answers about this topic, but I can't solve my problem. I initialized a Springboot project with Kafka and spring-data-jdbc. What I'm trying to do is Configure a Kafka JDBC Connector in order to push record changes…
Vin
  • 701
  • 1
  • 9
  • 30
1
vote
1 answer

How do I Restart a shutdown embeddedKafkaServer in a Spring Unit Test?

I have a Spring-boot Unit Test that is testing Switch Back capabilities of my application when the primary Kafka Cluster comes online. The application successfully switches to secondary when the primary goes offline. Now we're adding the ability to…
Raystorm
  • 6,180
  • 4
  • 35
  • 62
1
vote
1 answer

Spring Kafka Listening to all partitions -NumberFormatException

'Am trying to listen to all kafka partitions , in a Spring Boot app, and use the below code as documented in assign-all-parts, however i get the exception as below java.lang.NumberFormatException: For input string: "*" Is there any otherway to…
codevalid
  • 191
  • 1
  • 3
  • 8
1 2 3
…
99 100