Stack Overflow
Stack Overflow

Questions tagged [spring-kafka]

The Spring for Apache Kafka (spring-kafka) project applies core Spring concepts to the development of Apache Kafka-based messaging solutions.

The Spring for Apache Kafka (spring-kafka) project applies core Spring concepts to the development of Apache Kafka-based messaging solutions. It provides a "template" as a high-level abstraction for sending messages. It also provides support for Message-driven POJOs with @KafkaListener annotations and a "listener container". These libraries promote the use of dependency injection and declarative. In all of these cases, you will see similarities to the JMS support in the Spring Framework and RabbitMQ support in Spring AMQP.

has a module spring-integration-kafka providing channel adapters and gateways based on spring-kafka.

Resources

See Also

4185 questions
1
vote
1 answer

How to expose Kafka metrics from SpringBoot application through JMX to Prometheus?

I have a springboot app from which i would like to expose the kafka.consumer metrics to Prometheus via JMX. I can see the metrics here but i just don't know where to set those mBeans(i.e. kafka.consumer:type=consumer-metrics,client-id=([-.\w]+)). I…
Charles
  • 570
  • 11
  • 29
1
vote
1 answer

kafkaendpointlistenerregistry.start() throws null pointer exception

I have a requirement where I want to start Kakfa consumer manually. Code : class Dummy implements ConsumerSeekAware { @Autowired KafkaListenerEndpointRegistry registry; CountDownLatch latch; @Autowired …
victor perfect
  • 31
  • 2
  • 6
1
vote
1 answer

spring cloud stream kafka - Functional approach: Consumer is never called when producing tombstone records

The issue is pretty much the same as https://github.com/spring-cloud/spring-cloud-stream-binder-kafka/issues/455 but for the Functional approach @Bean public Consumer> fooConsumer() { return message -> { …
Sankarganesh Eswaran
  • 10,076
  • 3
  • 23
  • 24
1
vote
1 answer

Why KafkaTemplate does not close tansactional producers?

I have written a simple Kafka app with spring integration kafka 3.2.1.RELEASE and kafka-clients 2.5 to learn kafka transactions. It recieves the messages from a topic and sends them to another topic. The beans.xml file is as follows …
simsini2
  • 13
  • 4
1
vote
0 answers

Kafka consumer stopped consuming message after producer ack mode change

I have a producer and consumer application which works perfectly. My three producers are used ack mode '1' and today I changed one producer ack mode to 'ALL'. Soon after all consumers stopped consuming messages. I can see producers are producing…
era
  • 391
  • 4
  • 24
1
vote
2 answers

Stop KafkaListener ( Spring Kafka Consumer) after it has read all messages till some specific time

I am trying to schedule my consumption process from a single partition topic. I can start it using endpointlistenerregistry.start() but I want to stop it after I have consumed all the messages in current partition i.e. when I reach to last offset in…
victor perfect
  • 31
  • 2
  • 6
1
vote
0 answers

Kafka Too many files open

When I am pushing more number of records into the queue, I am getting a below exception and my total instance has been stopped. Environment OS: AIX Server: Jboss 7.2 kafka producer code private Map producerConfig(String…
Akilan R
  • 13
  • 5
1
vote
1 answer

SAGA and local transactions with Kafka and Postgres in Spring Boot

Haven't worked with SAGAs and spring-kafka (and spring-cloud-stream-kafka-binder) for a while. Context: there are several (3+) Spring-boot microservices that have to span business transaction in order to keep data in eventually consistent state.…
Igor Petrov
  • 166
  • 2
  • 15
1
vote
1 answer

Micrometer throws exception when creating a second kafka-consumer

Exception occurred when upgrading to spring-boot 2.3.0. Exception is as follows: java.lang.IllegalArgumentException: Prometheus requires that all meters with the same name have the same set of tag keys. There is already an existing meter named…
ODDminus1
  • 13
  • 3
1
vote
1 answer

How to perform a flatTransform in a Spring Cloud Kafka Streams app?

I'm trying to perform a flatTransform in a Spring Cloud Kafka Streams app. But I'm not sure where exactly to put the KafkaStreamsStateStore annotation. At the moment I'm getting the error: Invalid topology: StateStore activeInstruments is not added…
user6429576
1
vote
1 answer

How to delete old keys from a KTable

I have a producer which publishes records with a key of either A or B to a Kafka topic. In a streams app, I'm flat-mapping each record with key A to records with key U, V or W, and each record with key B to records with key X, Y or Z. The number of…
user6429576
1
vote
1 answer

Abort mongo transaction on kafka write failure

I have an example Spring Boot application saving a document to MongoDB using Spring's MongoRepository. After that, I write a message to a Kafka topic. What I want is to rollback the MongoDB transaction when the Kafka write fails. Do I have other…
JointEffort
  • 583
  • 7
  • 21
1
vote
3 answers

Spring Kafka can not deserialize JSON

My consumer, which is a Spring Boot application, uses Spring Kafka. It is supposed to read messages from a topic in JSON format. When the messages are "small", everything goes without problems. However, when the message is "big" (which is not…
scarface
  • 574
  • 6
  • 20
1
vote
1 answer

How long a rollbacked message is kept in a Kafka topic

I came across with this scenario when implementing a chained transaction manager inside our spring boot application interacting with consuming messages from JMS then publishing to a Kafka topic. My testing strategy was explained on here: Unable to…
Julian
  • 3,678
  • 7
  • 40
  • 72
1
vote
1 answer

How can setConsumerRebalanceListener get the calling ConsumerAwareRebalanceListener instance?

Maybe this is a naive question but it somhow stuck me there for some time. Please bear with me. I have a class DataConsumer.java that implements ConsumerAwareRebalanceListener: @Component public class DataConsumer implements…
yifei
  • 561
  • 5
  • 18
1 2 3
…
99
100