Stack Overflow
Stack Overflow

Questions tagged [reactive-kafka]

Akka Streams Kafka, also known as Reactive Kafka, is an Akka Streams connector for Apache Kafka.

User guide: http://doc.akka.io/docs/akka-stream-kafka/current/home.html

Source code: https://github.com/akka/reactive-kafka

62 questions
0
votes
0 answers

Issue while sending the Flux of data to the reactive kafka

I am using reactor library to fetch the large stream of data from the network and sending it to the kafka broker using the reactive kafka approach. Below is the Kafka Producer I am using public class LogProducer { private final…
Nishit Jain
  • 1,549
  • 8
  • 21
  • 33
0
votes
0 answers

Creating Reactive Kafka SenderRecord from Flux

I am doing a prototype integrating Spring Cloud Gateway with Reactive Kafka. My goal is to push the post body to kafka. I have access to Flux which could potentially give me the body in chunks. Need expert advice how to convert this Flux of…
anaray
  • 309
  • 1
  • 8
0
votes
0 answers

How to send String data to kafka producer using Reactive Kafka?

I have a data in string format. I need to send it to kafka producer to a specific partition but using reactive kafka ie making async call. How to achieve this using code ? I checked reactive kafka docs but unfortunately I dont have much idea about…
aarish_codev
  • 43
  • 1
  • 5
0
votes
1 answer

Kafka Producer Idempotence - Exactly Once or Just Producer Transaction is Enough?

From this article https://www.confluent.io/blog/transactions-apache-kafka/ Using vanilla Kafka producers and consumers configured for at-least-once delivery semantics, a stream processing application could lose exactly once processing semantics in…
user1955934
  • 3,185
  • 5
  • 42
  • 68
0
votes
1 answer

Akka Stream Kafka, complete stream when reached end of log

I am using Akka Streams Kafka and I am looking for a way to do the following: Initiation stream from offset x Consume sequentially items x, x+1, x+2 .. until the last item Once the last item has been consumed, complete the stream. Code would look…
ticofab
  • 7,551
  • 13
  • 49
  • 90
0
votes
1 answer

Equivalent of kafkaConsumer.partitionsFor using reactive kafka (akka kafka streams)

I'm using reactive-kafka (akka kafka streams): https://doc.akka.io/docs/akka-stream-kafka/current/consumer.html What is the equivalent of the below code using reactive-kafka? import org.apache.kafka.clients.consumer.KafkaConsumer ... val properties…
Rory
  • 798
  • 2
  • 12
  • 37
0
votes
2 answers

Found java.util.concurrent.Future Required scala.concurrent.Future

Related: scala.concurrent.Future wrapper for java.util.concurrent.Future This came from my other question: How to integrate akka streams kafka (reactive-kafka) into akka http application? I have a AKKA HTTP application, and I'd like to send a…
Rory
  • 798
  • 2
  • 12
  • 37
0
votes
1 answer

How to integrate akka streams kafka (reactive-kafka) into akka http application?

I have a basic scala akka http CRUD application. See below for the relevant classes. I'd simply like to write an entity id and some data (as json) to a Kafka topic whenever, for example, an entity is created/updated. I'm looking at…
Rory
  • 798
  • 2
  • 12
  • 37
0
votes
1 answer

connect producer flow to graph

I am new using akka streams kafka (and akka streams in general) . I am trying to construct a graph in order to publish a message to different topics. How can I connect the producer as flow in order to commit the processed messages ? I tried using…
igx
  • 4,101
  • 11
  • 43
  • 88
0
votes
0 answers

Adding a Flow for writing to a Kafka subscriber

I need to build the following graph: val graph = getFromTopic1 ~> doSomeWork ~> writeToTopic2 ~> commitOffsetForTopic1 but trying to implement it in Reactive Kafka has me down a rabbit hole. And that seems wrong because this strikes me as a…
Keith Nordstrom
  • 354
  • 2
  • 9
0
votes
1 answer

Reactive Kafka: No error while creating stream for a non-existent topic

We recently switched to Reactive Kafka. It seems kafka does not throw any error if the topic exists while creating stream for the topic. Why is it so? I know that existence of a topic can explicitly be checked with AdminUtils.topicExists(zkClient,…
user87407
  • 647
  • 3
  • 7
  • 24
0
votes
1 answer

adding newly create topic to subscription in reactive kafka

iam developing reactive kafka in our play scala project, in the project we created 5 topic which are subscribed by consumer group and working good, now the problem is i created a new topic, how can i add this topic to the existing consumer group…
Kiran Sunkari
  • 301
  • 1
  • 3
  • 15
0
votes
1 answer

How to 'Chunk and Re-assmble' large messages in Reactive Kafka using Akka-Stream

When sending a large file using Kafka, is it possible to distribute it across partitions and then re-assemble it using Akka-Stream? as described in this…
Rabzu
  • 52
  • 5
  • 26
0
votes
1 answer

reactive-kafka: why does publisher "go with" consumer?

I looked at the following documentation: https://github.com/akka/reactive-kafka and I saw the following code snippet: implicit val actorSystem = ActorSystem("ReactiveKafka") implicit val materializer = ActorMaterializer() val kafka = new…
CrazySynthax
  • 13,662
  • 34
  • 99
  • 183
0
votes
1 answer

How to produce a message using a Publisher(in reactive-kafka)?

I have to work with a code from reactive-kafka module: https://github.com/akka/reactive-kafka/blob/master/README.md My code starts with: val kafka = new ReactiveKafka() val kafkaIdpsMsgs: Publisher[StringKafkaMessage] = kafka.consume( …
CrazySynthax
  • 13,662
  • 34
  • 99
  • 183
1 2 3
4
5