0

I have a very large relational database dataset which I would like to index in elastic search. The query which retrieves the data consists of multiple joins and all other SQL goodies. The data is grouped/processed (in-memory) in order to create meaningful json representation and bulk update is created from the results and send to elastic search with elastic4s scala client.

I would like to introduce streaming to this process as both slick and elastic support it.

The problem I have is that the in-memory grouping and converting to json makes only sense if all the results (for given relation) are loaded into memory (due to several joins/left joins, I need to group by id and map the results in memory). How would it be handled with streaming?

Mon Calamari
  • 4,403
  • 3
  • 26
  • 44
  • What specific stream implementation are you looking at? [scalaz-stream](https://github.com/functional-streams-for-scala/fs2), [spark-streaming](http://spark.apache.org/streaming/), [akka-streams](http://doc.akka.io/docs/akka-stream-and-http-experimental/1.0-M2/scala/stream-index.html), maybe [play-iteratees](https://www.playframework.com/documentation/2.0.4/Iteratees)? – Odomontois Nov 21 '15 at 14:20
  • https://github.com/sksamuel/elastic4s#elastic-reactive-streams – Mon Calamari Nov 21 '15 at 14:22
  • Basically, slick creates a publisher which elastic4s can subscribe to. – Mon Calamari Nov 21 '15 at 14:22

1 Answers1

1

I have solved streaming of Multiple joins using slick Reactive stream like below code. This will create stream using Publisher, Elastic need to subscribe for this publisher to start retrieve stream data.

def listTransactions(): DatabasePublisher[Transaction] = {
  val join = for {
    (t, a) <- transactions joinLeft (account) on (_.accountid === _.id)
  } yield (t, a)
 db.stream(join.result.withStatementParameters(rsType = ResultSetType.ForwardOnly,rsConcurrency = ResultSetConcurrency.ReadOnly, fetchSize = 1000).transactionally).mapResult {
  result => result._1
  }
}

Note:- _1 in code represent transaction table.

Manjunath A
  • 81
  • 1
  • 2