2

The following code will read a file line by line, create a task for each line and then queue it up to the executor. If the executor's queue is full, the reading from the file stops until there is space again.

I looked at a few suggestions in SO, but they all either require the entire content of the file be read into memory, or suboptimal scheduling (e.g. read 100 lines, process them in parallel, only after that finishes, read the next 100 lines). I also don't want to use libraries like Akka for this.

What is the Scala way of achieving this, without those drawbacks?

  val exec = executorWithBoundedQueue()
  val lines = Source.fromFile(sourceFile, cs).getLines()
  lines.map {
    l => exec.submit(new Callable[String] {
      override def call(): String = doStuff(l)
    })
  }.foreach {
    s => consume(s.get())
  }
  exec.shutdown()

Illustrative definition of executorWithBoundedQueue

def executorWithBoundedQueue(): ExecutorService = {
    val boundedQueue = new LinkedBlockingQueue[Runnable](1000)
    new ThreadPoolExecutor(boundedQueue)
}
Paul Sweatte
  • 24,148
  • 7
  • 127
  • 265
BasilTomato
  • 1,071
  • 1
  • 8
  • 14

0 Answers0