Questions tagged [reactor-kafka]
106 questions
9
votes
1 answer
Implement Reactive Kafka Listener in Spring Boot application
I'm trying to implement reactive kafka consumer in my Spring boot application and I'm looking at these examples:
https://github.com/reactor/reactor-kafka/blob/master/reactor-kafka-samples/src/main/java/reactor/kafka/samples/SampleScenarios.java
and…
bobryash
- 129
- 1
- 8
5
votes
1 answer
Does Reactor onErrorContinue operator let the original sequence continue?
The Reactor error handling documentation (https://projectreactor.io/docs/core/3.4.10/reference/index.html#error.handling) states that error-handling operators do not let the original sequence continue.
Before you learn about error-handling…
ankush
- 167
- 1
- 11
5
votes
2 answers
Using onErrorResume to handle problematic payloads posted to Kafka using Reactor Kafka
I am using reactor kafka to send in kafka messages and receive and process them.
While receiving the kakfa payload, I do some deserialization, and if there is an exception, I want to just log that payload ( by saving to mongo ), and then continue…
use1803
- 153
- 10
4
votes
1 answer
How to retry failed ConsumerRecord in reactor-kafka
I am trying on reactor-kafka for consuming messages. Everything else work fine, but I want to add a retry(2) for failing messages. spring-kafka already retries failed record 3 times by default, I want to achieve the same using reactor-kafka.
I am…
Aniket Singla
- 555
- 1
- 5
- 20
4
votes
3 answers
Spring Boot Kafka - Kafka metrics not available in /actuator/prometheus
I would like to monitor Kafka metrics but unfortunately nothing related to Kafka is present under /actuator/prometheus endpoint. Is anything missing in my setup?
Application dependencies: Kotlin 1.4.31, Spring Boot 2.3.9, Spring Kafka 2.6.7, Reactor…
Pawel
- 466
- 1
- 7
- 20
3
votes
0 answers
Reactor Kafka how to gracefully shutdown?
We use reactiveKafkaConsumerTemplate to receive messages,then process the message. We try to graceful shutdown the application.
Stop poll new messages
Wait for the current message finish process
Shutdown application
We tried…
Gin
- 31
- 2
3
votes
1 answer
Using DeadLetterPublishingRecoverer with ReactiveKafkaConsumerTemplate
I started looking into using a DeadLetterPublishingRecoverer for my kafka consumer factory common error handler. However, it seems that in order to instantiate an instance of this class, 1 option is to pass in a KafkaOperations
jimbob542
- 121
- 1
- 1
- 7
3
votes
1 answer
Polling behavior when using ReactiveKafkaConsumerTemplate
I have a Spring Boot application using ReactiveKafkaConsumerTemplate for consuming messages from Kafka.
I've consume messages using kafkaConsumerTemplate.receive() therefore I'm manually acknowledging each message. Since I'm working in an…
Eyal Ringort
- 601
- 6
- 19
3
votes
0 answers
Reactive Kafka Receiver and paraller records processing flow
I'm trying to process batches of records from Kafka, using reactor-kafka like this:
KafkaReceiver.create(
kafkaProperties.createReceiverOptions(String::class.java).subscription(listOf("test_topic1"))
)
…
avtogen-burn
- 31
- 3
3
votes
1 answer
Reactor Kafka health check in a Spring webflux app
I have a Reactor Kafka application that consumes messages from a topic indefinitely. I need to expose a health check REST endpoint that can indicate the health of this process - Essentially interested in knowing if the Kafka receiver flux sequence…
ankush
- 167
- 1
- 11
3
votes
0 answers
Azure Kafka Bootstrap broker disconnected
I try to create an Apache Kafka connection to Azure event hubs with reactor kafka in a spring boot application. At first I followed the official Azure tutorial to set up Azure event hubs and the spring backend:…
Rooky
- 810
- 3
- 12
- 20
3
votes
2 answers
Continue consuming subsequent records in reactor kafka after deserialization exception
I am using reactor kafka and have a custom AvroDeserializer class for deserialization of messages.
Now I have a case where for certain payloads the deserialization class throws an exception.
My Kafka listener dies as soon as it tries to read such…
use1803
- 153
- 10
3
votes
0 answers
Spring Cloud Sleuth with Reactor Kafka
I'm using Reactor Kafka in a Spring Boot Reactive app, with Spring Cloud Sleuth for distributed tracing.
I've setup Sleuth to use a custom propagation key from a header named "traceId".
I've also customized the log format to print the header in my…
Rajesh J Advani
- 5,585
- 2
- 23
- 35
3
votes
0 answers
Sending events with Reactor Kafka seems very slow
My project uses Reactor Kafka 1.2.2.RELEASE to send events serialized with Avro to my Kafka broker. This works well, however sending events seems to be quite slow.
We plugged a custom metric to the lifecycle of the Mono sending the event through…
Sancho
- 417
- 4
- 20
2
votes
0 answers
Events Lost When Consuming using reactor kafka
I am using Reactive kafka for consuming events.
Problem: I pushed 7 events to the queue, but the consumer only consumed 5 of them. (Only happening when deployed on a server, working fine in local environment). This has happened a lot of times, and…
Aashish Goyal
- 21
- 1