2

I am currently working with akka streams and building a reactive application. My understanding so far is when async boundary is set,

Each stage(Flow) runs in an actor. Is that true?

Calling mapAsync is similar to applying async to the flow?

Ideally, I would have bunch of graphs and would want each to run parallelly in an actor, which helps to scale.

Additional information:

I am implementing something with reactive kafka, Would like to know how kafka topic consumption is load balanced. ie: If I have 20 partitions, if I am using plainPartitionSource and build a graph, does it create that many consumer actors?

If there are going to be 20 consumer actors(equal to 20 partitions for a specific topic it is consuming to), then there will ideally be 20 graphs. Materializing the graphs, I would want it to run in different actors.

In short, I want full parallelism depending on the number of partitions, from consumption to processing it.

Hope my question is clear.

EDIT:

val consumerGroup =
  Consumer.committablePartitionedSource(consumerSettings, Subscriptions.topics("topic1"))
//Process each assigned partition separately
consumerGroup.map {
  case (topicPartition, source) =>
    source
      .via(business)
      .toMat(Sink.ignore)(Keep.both)
      .run()
}

.mapAsyncUnordered(maxPartitions)(_._2) 
//maxPartitions create that 
//many actors here to materialize each of the internal graph in 
//separate actors?

.runWith(Sink.ignore)
druuu
  • 1,676
  • 6
  • 19
  • 36
  • Usually, each time you call `run()`a new actor for the stream will be created so you should be fine if you let enough elements come through. So you are right that `maxPartitions` will actually limit the amount of streams running in parallel because it provides the demand that will pull elements into the `map` stage above. It would be even clearer if you would replace the `map(...).mapAsyncUnordered` combo with a single `mapAsyncUnordered`. – jrudolph Aug 22 '17 at 15:25

0 Answers0