Stack Overflow
Stack Overflow

Questions tagged [reactive-kafka]

Akka Streams Kafka, also known as Reactive Kafka, is an Akka Streams connector for Apache Kafka.

User guide: http://doc.akka.io/docs/akka-stream-kafka/current/home.html

Source code: https://github.com/akka/reactive-kafka

62 questions
2
votes
0 answers

Async and mapAsync - Achieving parallelism in akka streams - reactive kafka

I am currently working with akka streams and building a reactive application. My understanding so far is when async boundary is set, Each stage(Flow) runs in an actor. Is that true? Calling mapAsync is similar to applying async to the…
druuu
  • 1,676
  • 6
  • 19
  • 36
2
votes
2 answers

Reactive-Kafka : How to pause the consumer on exception and retry on demand

I have asked this question already on Google Groups, but have not received any response yet. So posting this here for a different audience. We are using Reactive-Kafka for our application. We have a scenario as below, where we want to stop sending…
Yadu Krishnan
  • 3,492
  • 5
  • 41
  • 80
2
votes
1 answer

How to transform Bytestring to Source[Bytestring, Any]

I'm receiving a file as a Bytestring from Kafka Reactive Streams consumer; I want to construct an akka-http request with this Bytestring as an entity HttpEntity.Default. HttpEntity.Default requires Source[Bytestring, Any] as one of its…
Rabzu
  • 52
  • 5
  • 26
1
vote
0 answers

Wanted to know if there is any way we can modify existing flux to receive messages at slower rate

Trying to modify kafka flux which is listening on a kafka topic to accept messages at slow rate. But the blow code is giving exception. public class SampleConsumer { private static final Logger log =…
shiva kumar
  • 21
  • 1
  • 10
1
vote
0 answers

How to transform reactive kafka receiver flux to kotlin coroutines?

tldr: Using spring, reactive, kafka, webflux, coroutines, need to somehow connect receiver Flux to coroutine Flow, but are GlobalScope and Dispatcher.Unconfined the right tools for the job? The context of the question: I am using spring, reactive…
Dknot
  • 63
  • 1
  • 8
1
vote
0 answers

Spring Reactive Kafka - Consumer Subscription not Working

I create a Reactive Kafka producer that works in Spring. See code below: KafkaPayload payload = new KafkaPayload(1, "value 1"); String topic = "a.kafka.topic"; Map props =…
Michela Dennis
  • 11
  • 2
1
vote
0 answers

Creating a new consumer when cosumer stops due to an error in Reactor Kafka

I am working on an application where I have multiple consumers for each Topic partition so there is concurrency in reading from the topic. I followed this link to ensure that the consumer gets created again if the existing consumer stops. .repeat…
perplexedDev
  • 857
  • 4
  • 17
  • 49
1
vote
1 answer

Multiple Kafka configurations in a reactive Kafka project

I am working on a project that has 2 services: both reading, transforming the messages and then writing to another Kafka. The Kafka configuration for both these services are different. Here is my application.yml spring: kafka: …
perplexedDev
  • 857
  • 4
  • 17
  • 49
1
vote
2 answers

Test a Reactive-Kafka Consumer and Producer Template using embedded kafka + custom serialised

We need an example on how to test ReactiveKafkaConsumerTemplate and ReactiveKafkaProducerTemplate with an embedded-kafka-broker. Thanks. CORRECT CODE IS HERE AFTR DISCUSSION You can have your custom de-serializer accordingly to use custom…
nihar
  • 135
  • 1
  • 17
1
vote
1 answer

Stop consuming from KafkaReceiver after a timeout

I have a common rest controller: private final KafkaReceiver receiver; @GetMapping(produces = MediaType.APPLICATION_STREAM_JSON_VALUE) public Flux produceFluxMessages() { return…
Kristina
  • 55
  • 5
1
vote
1 answer

Reactive program exiting early before sending all messages to Kafka

This is a subsequent question to a previous reactive kafka issue (Issue while sending the Flux of data to the reactive kafka). I am trying to send some log records to the kafka using the reactive approach. Here is the reactive code sending messages…
Nishit Jain
  • 1,549
  • 8
  • 21
  • 33
1
vote
1 answer

Transaction synchronisation: how to create ChainedKafkaTransactionManager bean with Reactor Kafka and R2DBC

I have a following consumer in my Spring Boot (WebFlux/R2DBC/Reactor Kafka) application @EventListener(ApplicationStartedEvent::class) fun onMyEvent() { kafkaReceiver .receive() .doOnNext { record -> …
Pawel
  • 466
  • 1
  • 7
  • 20
1
vote
1 answer

Reactor - Delay Flux elements in case of processing errors

I have a similar problem to this question and I do not see an accepted answer. I have researched through and did not get a satisfactory answer. I have a reactive Kafka consumer (Spring Reactor) with a poll amount 'x' and the application pushes the…
Pavan Kumar
  • 157
  • 2
  • 4
  • 17
1
vote
1 answer

Issue with Kafka Broker - UnknownServerException

Our application uses springBootVersion = 2.0.4.RELEASE along with compile('io.projectreactor.kafka:reactor-kafka:1.0.1.RELEASE') dependency. The Kafka Broker that we have is at version 1.0.1. Intermittently when we send the messages onto Kafka by…
Santosh Sindham
  • 879
  • 8
  • 18
1
vote
1 answer

Incompatible equality constraint while using Akka Kafka Streams

I am trying to use Akka Kafka Streams following the Akka Kafka Streams documentation. Here is the code I have: ConsumerSettings consumerSettings = ConsumerSettings .create(actorSystem, new…
Prasanth
  • 1,005
  • 5
  • 19
  • 39
1
2
3 4 5