Akka Streams filter & group by on a collection of keys

Question

I have a stream of

case class Msg(keys: Seq[Char], value: String)

Now I want to filter for a subset of keys e.g. val filterKeys = Set[Char]('k','f','c') and Filter(k.exists(filterKeys.contains))) And then split these so certain keys are processed by different flows and then merged back together at the end;

                                 /-key=k-> f1 --\
Source[Msg] ~> Filter ~> router |--key=f-> f2 ----> Merge --> f4
                                 \-key=c-> f3 --/

How should I go about doing this?

FlexiRoute in the old way seemed like a good way to go but in the new API I'm guessing I want to either make a custom GraphStage or create my own graph from the DSL as I see no way to do this through the built-in stages..?

Ramón J Romero y Vigil · Answer 1 · 2016-10-06T12:56:09.693

Small Key Set Solution

If your key set is small, and immutable, then a combination of broadcast and filter would probably be the easiest implementation to understand. You first need to define the filter that you described:

def goodKeys(keySet : Set[Char]) = Flow[Msg] filter (_.keys exists keySet.contains)

This can then feed a broadcaster as described in the documentation. All Msg values with good keys will be broadcasted to each of three filters, and each filter will only allow a particular key:

val g = RunnableGraph.fromGraph(GraphDSL.create() { implicit builder: GraphDSL.Builder[NotUsed] =>
  import GraphDSL.Implicits._

  val source : Source[Msg] = ???

  val goodKeyFilter = goodKeys(Set('k','f','c'))

  val bcast = builder.add(BroadCast[Msg](3))
  val merge = builder.add(Merge[Msg](3))

  val kKey = goodKeys(Set('k'))
  val fKey = goodKeys(Set('f'))
  val cKey = goodKeys(Set('c'))

  //as described in the question
  val f1 : Flow[Msg, Msg, _] = ???
  val f2 : Flow[Msg, Msg, _] = ???
  val f3 : Flow[Msg, Msg, _] = ???

  val f4 : Sink[Msg,_] = ???

  source ~> goodKeyFilter ~> bcast ~> kKey ~> f1 ~> merge ~> f4
                             bcast ~> fKey ~> f2 ~> merge
                             bcast ~> cKey ~> f3 ~> merge

Large Key Set Solution

If you key set is large, then groupBy is better. Suppose you have a Map of keys to functions:

//e.g. 'k' -> f1
val keyFuncs : Map[Set[Char], (Msg) => Msg]

This map can be used with the groupBy function:

source
  .via(goodKeys(Set('k','f','c'))
  .groupBy(keyFuncs.size, _.keys)
  .map(keyFuncs(_.keys)) //apply one of f1,f2,f3 to the Msg
  .mergeSubstreams

Akka Streams filter & group by on a collection of keys

1 Answers1