Scala Slick: Never ending stream

Question

With Slick you can do the following to produce a stream of results from a table:

val q = for (e <- events) yield e.name
val p: DatabasePublisher[String] = db.stream(q.result)

p.foreach { s => println(s"Event: $s") }

That will print all the events in the events table and terminate after the last row.

Assuming you can be notified in some way of when new rows are entered into the events table, is it possible to write a stream that would continuously output events as they were inserted? A sort of tail -f for a DB table.

I think Slick won't support this natively, but I think it should be possible to use Akka streaming to help. So if you could have something that took from the Slick Source until it was empty, then waited for an event to indicate more data in the table, then streamed the new data. Possibly by using an ActorPublisher to bind this logic?

Just wondering if someone has any experience in this area or any advice?

This feels a bit like Spark's Dstream: http://spark.apache.org/docs/latest/api/scala/index.html#org.apache.spark.streaming.dstream.DStream — The Archetypal Paul, Dec 14 '15 at 18:51

Anna Zubenko · Accepted Answer · 2015-12-15T22:22:06.523

You were correct about ActorPublisher :) Here a is simple example using PostgreSQL, async DB driver and LISTEN/NOTIFY mechanism:

Actor:

class PostgresListener extends ActorPublisher[String] {

  override def receive = {
    case _ ⇒
      val configuration = URLParser.parse(s"jdbc://postgresql://$host:$port/$db?user=$user&password=$password")
      val connection = new PostgreSQLConnection(configuration)
      Await.result(connection.connect, 5.seconds)

      connection.sendQuery(s"LISTEN $channel")
      connection.registerNotifyListener { message ⇒ onNext(message.payload) }
  }
}

Service:

def stream: Source[ServerSentEvent, Unit] = {
  val dataPublisherRef = Props[PostgresListener]
  val dataPublisher = ActorPublisher[String](dataPublisherRef)

  dataPublisherRef ! "go"

  Source(dataPublisher)
    .map(ServerSentEvent(_))
    .via(WithHeartbeats(10.second))
}

build.sbt in libraryDependencies:

"com.github.mauricio"  %% "postgresql-async"         % "0.2.18"

Postgres trigger should call select pg_notify('foo', 'payload')

As far as I know, Slick does not support LISTEN.

I've now got something working, a little different to your answer, but using ActorPublisher so thanks for the confirmation. Also I think your answer is using code from the akka-sse project (ServerSentEvent/WithHeartbeats)? Which isn't necessary for this question but happens to be something I was looking for. So thanks for a pointer to that too! — David, Dec 17 '15 at 18:14
Yeah, you're right, that's Akka and I'm streaming stuff from DB to UI via SSE. Forgot about that when copypasted code :) You're welcome! — Anna Zubenko, Dec 20 '15 at 10:45

Jeffrey Chung · Answer 2 · 2018-07-25T10:26:39.950

ActorPublisher has been deprecated since Akka 2.5.0. Here is an alternative that uses the postgresql-async library and creates a SourceQueue inside an actor:

import akka.actor._
import akka.stream._
import akka.stream.scaladsl._

import com.github.mauricio.async.db.postgresql.PostgreSQLConnection
import com.github.mauricio.async.db.postgresql.util.URLParser

import scala.concurrent.duration._
import scala.concurrent.Await

class DbActor(implicit materializer: ActorMaterializer) extends Actor with ActorLogging {
  private implicit val ec = context.system.dispatcher

  val queue =  
    Source.queue[String](Int.MaxValue, OverflowStrategy.backpressure)
      .to(Sink.foreach(println))
      .run()

  val configuration = URLParser.parse("jdbc:postgresql://localhost:5233/my_db?user=dbuser&password=pwd")
  val connection = new PostgreSQLConnection(configuration)
  Await.result(connection.connect, 5 seconds)

  connection.sendQuery("LISTEN my_channel")
  connection.registerNotifyListener { message =>
    val msg = message.payload
    log.debug("Sending the payload: {}", msg)
    self ! msg
  }

  def receive = {
    case payload: String =>
      queue.offer(payload).pipeTo(self)
    case QueueOfferResult.Dropped =>
      log.warning("Dropped a message.")
    case QueueOfferResult.Enqueued =>
      log.debug("Enqueued a message.")
    case QueueOfferResult.Failure(t) =>
      log.error("Stream failed: {}", t.getMessage)
    case QueueOfferResult.QueueClosed =>
      log.debug("Stream closed.")
  }
}

The code above simply prints notifications from PostgreSQL as they occur; you can replace the Sink.foreach(println) with another Sink. To run it:

import akka.actor._
import akka.stream.ActorMaterializer

object Example extends App {
  implicit val system = ActorSystem()
  implicit val materializer = ActorMaterializer()
  system.actorOf(Props(classOf[DbActor], materializer))
}

Scala Slick: Never ending stream

2 Answers2