0

I need to model a series of LINEAR Akka Streams processing streams that model a 1 producer N consumers system.

1 Producer N consumers

As a quick reference, you can imagine having a single producer that produces messages in a Kafka topic, and N consumers that consume from that topic (with their own different consumer groups). In my specific case, however, this has all to be handled in memory, within the same process.

Since each linear stream is handled independently, I cannot use standard tools provided by Akka streams, such as the GraphDSL Broadcast operator. Each linear stream has to have its own Source and Sink.

Moreover, those linear streams are dynamically constructed at runtime, meaning I have to provide some reusable Source and Sink implementation.

I tried to use Actor interoperator (ActorSink.actorRefWithBackpressure() for the producer and N ActorSource.actorRef() for the consumers, but it doesn't model my case, as I cannot access the full stream materialized value, i.e. the source actor ref).

What I would need is something with the same semantics of the Kafka Source and Sink, but backed by a fully in-memory data structure. Is there anything (maybe in Alpakka) that I could use for this, otherwise what would be the correct approach?

Alessandro D'Armiento
  • 135
  • 11

1 Answers1

1

BroadcastHub is probably the closest thing to what you're describing. It includes a built-in buffer to allow attached subscribers to fall behind by up to the size of the buffer.

Note that any "time travel" semantics (consumers receiving messages produced before the consuming stream materialized) are going to be limited or non-existent (I'm not sure which) compared to Kafka.

Levi Ramsey
  • 18,884
  • 1
  • 16
  • 30
  • Hello and thank you. The BroadcastHub seems like being able to provide the functionality that I need, however, in order to obtain the sources, I first need to materialize the producer, which is impossible, since I can't have the representation of the producer ready before the others. – Alessandro D'Armiento Sep 23 '22 at 12:35
  • I solved the issue of the materialization using the `preMaterialed()` API. Here's an example `class Broadcast(implicit materializer: Materializer) { private val broadcastSink: Sink[String, Source[String, NotUsed]] = BroadcastHub.sink(bufferSize = 256) private val tuple: (Source[String, NotUsed], Sink[String, NotUsed]) = broadcastSink.preMaterialize() val ingress: Sink[String, NotUsed] = tuple._2 val egress: Source[String, NotUsed] = tuple._1 }` – Alessandro D'Armiento Sep 23 '22 at 13:48
  • It's however imperfect, as any message created before the materialization gets lost. – Alessandro D'Armiento Sep 23 '22 at 13:56