Processing Akka stream in Slick transaction

Question

Software versions:

Akka 2.4.4
Slick 3.1.0

I want to process elements from an Akka stream in a Slick transaction. Here is some simplified code to illustrate one possible approach:

def insert(d: AnimalFields): DBIO[Long] =
  animals returning animals.map(_.id) += d

val source: Source[AnimalFields, _]
val sourceAsTraversable = ???

db.run((for {
  ids <- DBIO.sequence(sourceAsTraversable.map(insert))
} yield { ids }).transactionally)

One solution I could come up with so far is blocking each future to traverse the elements:

class TraversableQueue[T](sinkQueue: SinkQueue[T]) extends Traversable[T] {

  @tailrec private def next[U](f: T => U): Unit = {
    val nextElem = Await.result(sinkQueue.pull(), Duration.Inf)
    if (nextElem.isDefined) {
      f(nextElem.get)
      next(f)
    }
  }

  def foreach[U](f: T => U): Unit = next(f)
}

val sinkQueue = source.runWith(Sink.queue())
val queue = new TraversableQueue(sinkQueue)

Now I can pass the traversable queue to DBIO.sequence(). This defeats the purpose of streamed processing, though.

Another approach I found is this:

def toDbioAction[T](queue: SinkQueue[DBIOAction[S, NoStream, Effect.All]]):
        DBIOAction[Queue[T], NoStream, Effect.All] =
  DBIO.from(queue.pull() map { tOption =>
    tOption match {
      case Some(action) =>
          action.flatMap(t => toDbioAction(queue).map(_ :+ t))
      case None => DBIO.successful(Queue())
    }
  }).flatMap(r => r)

With this method, a sequence of DBIOActions can be generated without blocking:

toDbioAction(source.runWith(Sink.queue()))

Is there any better / more idiomatic way to achieve the desired result?

I think this is about as good of a solution as you're going to get. You need to block at the end in order to run in a single transaction; and furthermore, Slick won't actually send anything to the database until you call `db.run()`. — jkinkead, May 06 '16 at 20:22

score 0 · Answer 1 · answered Nov 15 '19 at 17:31

Here is my implementation of sourceAsTraversable:

import scala.collection.JavaConverters._
def sourceAsTraversable[A](source: Source[A, _])(implicit mat: Materializer): Traversable[A] =
    source.runWith(StreamConverters.asJavaStream()).iterator().asScala.toIterable

The issue with TraversableQueue was that the forEach had to finish processing the stream fully - it did not support the "break" concept, so methods like "drop"/"take", etc. would still have to process whole source. This could be important from error handling point of view and failing fast.

Processing Akka stream in Slick transaction

1 Answers1