Idiomatic way to handle multiple concurrent streams in Scala

Question

I have a list of streams that, upon calling their next() will sleep random amount of time and then read one char from a different source.

I am trying to write a consumer(s) that will keep calling these streams until EOF and build a common dictionary of these streams at runtime.

So far, I am using a ConcurrentHashMap for the dictionary and simply creating a new thread for each of the stream consumers.

while my solution works, it seems very naiive and I wonder if there's a better use for a streaming library such as monix or fs2

Using a streaming library of any sort means you don't have to re-learn all the hard-won lessons of handling concurrency. Unless that is your aim. — Bob Dalgleish, May 28 '18 at 13:41
What is being mapped in your `ConcurrentHashMap`, from what to what? Do you have a "stream" or an `Iterator`? I ask because you are calling `next` which is one of the methods of an Iterator, not a stream... — Ramón J Romero y Vigil, May 29 '18 at 11:21
@RamonJRomeroyVigil Indeed it's an Iterator. the concurrentHashmap simply maps from strings (tokenized strings from the different readers to number of occurences). — smohamed, May 29 '18 at 14:36

score 1 · Accepted Answer · answered May 29 '18 at 15:25

Based on the description of the question, and subsequent comments, I am assuming there exists multiple Iterator[Char] sources:

val allSources : Iterable[Iterator[Char]] = ???

And the question is: how to concurrently collect String values from these Iterators to form a mapping of String to count.

Stream Based Solution

First we need to convert each of the Iterators to an Iterator of String values based on a separator:

trait Word {
  val data : String
}

object EmptyWord extends Word {
  override val data = ""
}

case class PartialWord(val data : String) extends Word

case class WholeWord(val data : String) extends Word

val appendToWord : Char => (Word, Char) => Word = 
  (separator) => (originalWord, appendChar) => originalWord match {
    case PartialWord(d) => 
      if(appendChar == separator)
        WholeWord(d)
      else
        PartialWord(d + appendChar)
    case _ => PartialWord(appendChar.toString)
  }

val isWholeWord : Word => Boolean = (_ : Word) match {
  case _ : WholeWord => true
  case _             => false
}

//using space as separator
val convertCharIterator : Iterator[Char] => Iterator[String] = 
  (_ : Iterator[Char])
    .scanLeft(EmptyWord)(appendToWord(' '))
    .filter(isWholeWord)
    .map(_.data)

We can now convert all of the Iterators to generate Strings and we can combine all of the Iterators into a single Iterator:

val allWordSource : Iterator[String] = 
  allSources.map(convertCharIterator)
            .reduceOption( _ ++ _)
            .getOrElse(Iterator.empty[String])

This Iterator can now be the source of an akka stream which will calculate your count:

val addToCounter : (Map[String, Int], String) => Map[String, Int] = 
  (counter, word) => 
    counter.updated(word, counter.getOrElse(word, 0) + 1)

val counter : Future[Map[String, Int]] = 
  Source
    .fromIterator( () => allWordSource)
    .runFold(Map.empty[String, Int])(addToCounter)

very impressive functional solution. it solves most of my problems — smohamed, Jun 01 '18 at 10:11
@SKandeel Let me know if there was any aspect of the question that I missed and I'll update accordingly. — Ramón J Romero y Vigil, Jun 01 '18 at 10:55
it was not part of the question, but the iterator actually calls `thread.sleep()` for random amount of milliseconds, and I need to write the value to the map as soon as it's ready from each iterator. I tried to use akka streams to put each iterator as a Source, and merge them but it still waits for all of them to be ready before producing answer. I am guessing it's about thread handling in akka. — smohamed, Jun 01 '18 at 12:08

Idiomatic way to handle multiple concurrent streams in Scala

1 Answers1