Stack Overflow
Stack Overflow

Questions tagged [akka-kafka]

akka-kafka is an actor based kafka consumer built on top of the high-level kafka consumer that allows for asynchronous/concurrent processing of messages from kafka while keeping an upper bound on the number of in-flight messages, and explicitly managing the commit of offsets.

27 questions
0
votes
0 answers

What happens with the Akka cluster when the Kafka cluster goes down?

Does Akka cluster also goes down when the Kafka cluster is down?
Akshay Jain
  • 73
  • 2
  • 8
0
votes
1 answer

Fast Processing Topic and Slow Processing Topic - Akka Kafka

I have a problem where i need to prioritize some events to be processes earlier and some events lets say after the high priority events. Those events come from one source and i need to prioritize the streams depending on their event type priority to…
rollercoaster
  • 15
  • 1
  • 2
0
votes
1 answer

Lagom topic subscriber - how to retry in Future Exception?

I have a topic subscriber in lagom as bellow fooService.fooTopic().subscribe .atLeastOnce( Flow[fooMsg].map { case fooMsg(_) => foo() case a => println(a) }.async.map{ _ => Done } ) to subscribe…
Mark
  • 2,041
  • 2
  • 18
  • 35
0
votes
1 answer

Akka streams Kafka consumer process parallel

I’m working on a Kafka consumer application using Akka Kafka connector. I would like the consumer to process messages parallelly. which consumer group should I choose Choosing a consumer? how can I configure the parallelism on the consumer side?
vkt
  • 1,401
  • 2
  • 20
  • 46
0
votes
2 answers

Is there functionality in the akka-Kafka (scala framework) to list all available topics

I am trying to check if a topic passed to my "start stream" method is a valid/already existing topic in the Kafka that my program is connected to. I know Java has the KafkaConsumer.ListTopics, but I'm working with the akka-kafka library and it seems…
person personson
  • 155
  • 1
  • 10
0
votes
1 answer

avro4s can not deserialize AnyRef

i have a simple case class case class KafkaContainer(key: String, payload: AnyRef) then i want to send this to kafka topic via producer i do this val byteArrayStream = new ByteArrayOutputStream() val output =…
HoTicE
  • 573
  • 2
  • 13
0
votes
1 answer

How to sessionize / group the events in Akka Streams?

The requirement is that I want to write an Akka streaming application that listens to continuous events from Kafka, then sessionizes the event data in a time frame, based on some id value embedded inside each event. For example, let's say that my…
dks551
  • 1,113
  • 1
  • 15
  • 39
0
votes
1 answer

Using reactive-kafka to conditionally process messages

I've been trying to use reactive-kafka, and I am having a problem with conditional processing, to which I didn't find a satisfying answer. Basically I'm trying to consume one kafka topic which contains a huge number of messages (around 10 billion…
Thundzz
  • 675
  • 1
  • 11
  • 15
0
votes
1 answer

Akka Kafka stream supervison strategy not working

I am running an Akka Streams Kafka application and I want to incorporate the supervision strategy on the stream consumer such that if the broker goes down, and the stream consumer dies after a stop timeout, the supervisor can restart the…
dks551
  • 1,113
  • 1
  • 15
  • 39
0
votes
1 answer

How do you return a future containing a list of messages after all available messages have been consumed from a Kafka topic?

I am probably missing the point of the Kafka Consumer but what I want to do is: Consumer subscribes to a topic, grabs all messages within the topic and returns a Future with a list of all of those messages The code I have written to try and…
goodOldFashioned
  • 117
  • 10
0
votes
1 answer

NoSuchMethodError: KafkaConsumer.subscribe

I use the following dependencies: val akkaVersion = "2.4.9" val kafkaVersion = "0.10.0.1" dependencies = ... "com.typesafe.akka" %% "akka-actor" % akkaVersion, "com.typesafe.akka" %% "akka-cluster" % akkaVersion, …
John Mullins
  • 1,061
  • 4
  • 11
  • 38
0
votes
1 answer

Kafka producer with adjustable amount of messages per second

What is the best way to write an Apache Kafka producer with a steady but adjustable output. Example: The producer should send constant 1000 messages/sec to the broker. During runtime the output should be adjustable to either 10 or 10000…
benny.la
  • 1,212
  • 12
  • 26
1
2