1

I'd like to parallelize the write to kafka, that is having multiple producer sending data to kafka, although from within an akka-stream. In other my stream would have several initial stage from the source, and then when arriving at sending the data, i would like to have about 16 worker sending the data at the same time.

I wonder if i need to embed Akka Streams Kafka in an akka-stream Graph DSL and use a balancer for that, or if there is an easier solution. Also, simply, if someone has done anything like that in general that would be great.

Stefano Bonetti
  • 8,973
  • 1
  • 25
  • 44
MaatDeamon
  • 9,532
  • 9
  • 60
  • 127

1 Answers1

2

Akka Streams Kafka (aka Reactive Kafka) has got a specific setting for the producer parallelism (see the docs)

akka.kafka.producer {
  # Tuning parameter of how many sends that can run in parallel.
  parallelism = 100

  ...
}

Have you tried tuning it to solve your problem?

Stefano Bonetti
  • 8,973
  • 1
  • 25
  • 44
  • I did not even know that parameter. Thank you. – MaatDeamon Sep 13 '17 at 16:47
  • 1
    Quick stretch on this, Does it mean that for instance, be it a situation where i need to preserve the order of my content, then i would need to bring that parallelism down to 1 ? – MaatDeamon Sep 13 '17 at 16:57
  • 1
    I would think this is the case, by looking at this code https://github.com/akka/reactive-kafka/blob/master/core/src/main/scala/akka/kafka/scaladsl/Producer.scala#L79 – Stefano Bonetti Sep 13 '17 at 17:18