Stack Overflow
Stack Overflow

Questions tagged [reactive-kafka]

Akka Streams Kafka, also known as Reactive Kafka, is an Akka Streams connector for Apache Kafka.

User guide: http://doc.akka.io/docs/akka-stream-kafka/current/home.html

Source code: https://github.com/akka/reactive-kafka

62 questions
1
vote
0 answers

Need predictable constant latency of <5ms in kafka

We are planning to use 3 kafka-brokers and a zookeeper node. We have kept one partition for each topic. But we have a stringent requirement of latency under 5ms. We are using reactive-kafka client library https://github.com/akka/reactive-kafka. e.g.…
Saloni Vithalani
  • 323
  • 1
  • 4
  • 13
1
vote
1 answer

How to configure offsets paramter while constructing ProducerMessage in Akka Streams Kafka?

I am using Scala 2.11 and Akka Streams Kafka 0.17. I have a stream where: A Source is created using Source.actorRef. Here, the actor is scheduled to run at some regular interval and generate messages continuously, which are emitted to the…
oblivion
  • 5,928
  • 3
  • 34
  • 55
1
vote
1 answer

Using Akka Streams Kafka Producer in parallel

I'd like to parallelize the write to kafka, that is having multiple producer sending data to kafka, although from within an akka-stream. In other my stream would have several initial stage from the source, and then when arriving at sending the data,…
MaatDeamon
  • 9,532
  • 9
  • 60
  • 127
1
vote
1 answer

Kafka - patterns for handling consumer unable to process messages

I am trying to implement simple service that pulls messages from kafka wraps them in some data and sends to external service. What is the common pattern for handling external service unavailabilty when processing message? So far I am manually…
user3364192
  • 3,783
  • 2
  • 21
  • 30
1
vote
0 answers

Chain Akka Streams Kafka with Akka-http by Source[Bytestring]

I'm receiving a file as a Bytestring from Kafka Reactive Streams consumer, which I want to send to a service. Is there a away to extract Source[Bytestring, Any] from Kafka Reactive Stream Consumer, so that I can chain the stream from Kafka to…
Rabzu
  • 52
  • 5
  • 26
0
votes
0 answers

Retry logic for Reactive Kafka is not working

im using spring reactive kafka to create a consumer but retry logic is not working when there is an exception. @EventListener(ApplicationReadyEvnt.class) public void…
kumar
  • 1
0
votes
0 answers

Reactive Kafka - how to correctly perform manual commit?

I am trying to do manual offset commit in following way: val receiver: KafkaReceiver val sender: KafkaSender val flux = receiver.receive() // from one message, I am creating Flux of items which are transformed and then send to another topic …
mike
  • 1
  • 1
0
votes
0 answers

I want to write junit for my kafka consumer but not sure how can I provide mock data when calling .subscription

protected void startConsumer() { props.put(StreamsConfig.APPLICATION_ID_CONFIG, applicationIdConfig); props.put(SCHEMAREG_URL, schemaregUrl); props.put(SSL_KEYSTORE_ALIAS, sslKeystoreAlias); props.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG,…
manu
  • 1
0
votes
0 answers

How to implement dead letter queue in WebFlux react kafka?

May I know how to implement dead letter queue example for the below code? The same input record should be published to some dlq topic. reactiveKafkaConsumerTemplate .receiveAutoAck() .map(ConsumerRecord::value) …
Rambabu Kokkiligadda
  • 185
  • 1
  • 3
  • 15
0
votes
1 answer

Producer as non-reactive kafka and consumer as reactive kafka

What happens if I use a non-reactive Kafka producer in service A and a reactive consumer in service B?
Shubham Kumar
  • 13
  • 4
0
votes
1 answer

Schedule running a reactive stream for every 1 min

I have a reactive stream that gets some data, loops through the data, processes the data, finally writes the data to Kafka public Flux sendData(){ Flux.fromIterable(o.getC()).publishOn(Schedulers.boundedElastic()) …
perplexedDev
  • 857
  • 4
  • 17
  • 49
0
votes
1 answer

Null point exception on embedded Kafka

I am using embedded Kafka for unit testing a spring boot application. I am using the code -…
perplexedDev
  • 857
  • 4
  • 17
  • 49
0
votes
1 answer

Reactive kafka producer

Am exploring reactive kafka and just wanted to confirm if reactive kafka is equivalent to sync producer. With sync producer, We get message delivery guarantee with ACK all and producer sequence is maintained. But, delivery and sequencing is not…
user1578872
  • 7,808
  • 29
  • 108
  • 206
0
votes
1 answer

Stream output of one flux to another

I am getting the data from messageHandler(kafka) and processing it via servicehandler to give me some result serviceHandler.process(message).getMessage().toString() Now I need to output this stream of results so that whenever a new record comes…
Parikshit
  • 43
  • 6
0
votes
0 answers

Kafka Consumer Thread Subscribing Indefinitely - ReactiveKafkaConsumerTemplate

I am seeing issue whereby after some time, randomly, consumers stop consuming message from a topic and there's no error that was thrown. There's not many message in the topic because this is a test environment. The consumer lag for that topic would…
user1955934
  • 3,185
  • 5
  • 42
  • 68
1 2
3
4 5