How to copy some records from TableA to TableB with slick-streaming and akka-streaming

Question

There are two tables TableA and TableB.

I need to copy some records from TableA to TableB. I use slick-3.0 and use the following way:

import akka.stream._
import akka.stream.scaladsl._
...

//{{ READ DATA FROM TABLE A
val q = TableA.filter(somePredicate).result
val source = Source.fromPublisher {
      db.stream(q.result).mapResult { r => 
        val record: RecordA = someTransformation(r) 
        record
      }
    }.grouped(50) // grouping because I want to write records in batch mode
//}}

//{{ WRITE DATA TO TABLE B
val f:Future[Done] = source.runWith(Sink.foreach { batch: Seq[RecordA] =>
      //TODO how to write batch to TableB asynchronously?
      val insertAction = TableB ++= batch  // insert batch to table
      val fInsert: Future[_] = db.run(insertAction)
      Await.result(fInsert, ...)           // #1 this works only with blocking
})
//}}

But I've faced with an issue - how to write batch to TableB asynchronously (see TODO). Now the above code works with blocking to inner future only (see #1 comment). Is there a right way for implementing that task asynchronously?

@thwiegan, If I don't block on the inner future and return it then it will not completed — John Mullins, Jun 08 '17 at 08:03
This seems to be your use case: https://stackoverflow.com/questions/36400152/how-are-reactive-streams-used-in-slick-for-inserting-data I don't see anything too different from your example — thwiegan, Jun 08 '17 at 08:12
@thwiegan, thanks for the answer. But unfortunately this example doesn't work too. It works only if inner future has been blocked. Otherwise the data is not saved to `TableB`. — John Mullins, Jun 08 '17 at 08:37

score 2 · Accepted Answer · answered Jun 08 '17 at 09:52

use mapAsync it expects a future to be returned and exposes the "unwrapped" result in the next stage.

source.mapAsync(4){batch: Seq[RecordA] =>
      val insertAction = TableB ++= batch  // insert batch to table
      db.run(insertAction)
}).to(Sink.ignore).run

How to copy some records from TableA to TableB with slick-streaming and akka-streaming

1 Answers1