Play + ReactiveMongo: capped collection and tailable cursor

Question

I'm using Play Framework with Scala, Akka and ReactiveMongo. I want to use a collection in MongoDB as a circular queue. Several actors can insert documents into it; one actor retrieves these documents as soon as they're available (a sort of publish-subscribe system). I'm using capped collections and tailable cursor. Everytime I retrieve some documents I have to run the command EmptyCapped to flush the capped collection (it's not possible to REMOVE elements from it) otherwise I retrieve always the same document. is there an alternative solution? for example is there a way to slide a cursor without removing elements? or it's better not to use capped collection in my case?

object MexDB {

def db: reactivemongo.api.DB = ReactiveMongoPlugin.db
val size: Int = 10000

// creating capped collection
val collection: JSONCollection = {

    val c = db.collection[JSONCollection]("messages")

    val isCapped = coll.convertToCapped(size, None)

    Await.ready(isCapped, Duration.Inf)

    c
}

def insert(mex: Mex) = {

    val inserted = collection.insert(mex)

    inserted onComplete {
      case Failure(e) =>
        Logger.info("Error while inserting task: " + e.getMessage())
        throw e

      case Success(i) =>
        Logger.info("Successfully inserted task")
    }

}


def find(): Enumerator[Mex] = {

  val cursor: Cursor[Mex] = collection
    .find(Json.obj())
    .options(QueryOpts().tailable.awaitData)
    .cursor[Mex]

    // meaning of maxDocs ???
    val maxDocs = 1
    cursor.enumerate(maxDocs)
}


def removeAll() = {
    db.command(new EmptyCapped("messages"))
}

}

/*** part of receiver actor code ***/

// inside preStart
val it = Iteratee.fold[Mex, List[Mex]](Nil) {
    (partialList, mex) => partialList ::: List(mex)
}

// Inside "receive" method
case Data =>

  val e: Enumerator[Mex] = MexDB.find()

  val future = e.run(it)

  future onComplete {
    case Success(list) =>
      list foreach { mex =>
        Logger.info("Mex: " + mex.id)
      }
      MexDB.removeAll()
      self ! Data

    case Failure(e) => Logger.info("Error:  "+ e.getMessage())
  }

cchantep · Accepted Answer · 2015-05-19T20:32:55.413

0

Your tailable cursor is closed after each found doc as the maxDocs = 1. To keep it open indefinitely you should omit this limit.

With awaitData, .onComplete will only be called if you explicitly shutdown RM.

You need to use some streaming function from the cursor, such as .enumerate and process each new step/result. See https://github.com/sgodbillon/reactivemongo-tailablecursor-demo/

edited May 19 '15 at 20:32

answered May 19 '15 at 07:29

cchantep

9,118
3
30
41

Thanks. So if I omit that limit I can create the enumerator just once and re-use it right? Is there an alternative way to retrieve documents from capped collection like a circular queue without running EmptyCapped every time? – Mattia Micomonaco May 19 '15 at 07:42
Max 1 only read one doc. Remove the limit and the cursor is kept open at its current 'position', not back to the first doc. – cchantep May 19 '15 at 08:03
I removed the limit, I created just once the cursor and the enumerator. I periodically execute: val future = enum.run(it) future onComplete { *** } I'm inserting periodically mex in the collection but enum never retrieves some mex... what am I doing wrong? – Mattia Micomonaco May 19 '15 at 08:17
Do i need to inject Input.Done into my enumerator? – Mattia Micomonaco May 19 '15 at 09:51
With `awaitData`, `onComplete` will only be called if you explicitly shutdown RM. You need to use some streaming function from the cursor, such as `enumerate` and process each new step/result. See https://github.com/sgodbillon/reactivemongo-tailablecursor-demo/blob/master/app/controllers/Application.scala – cchantep May 19 '15 at 11:01
Nope. Approach waiting for completion won't work as the cursor is not supposed to end, but to return results in a streaming way. – cchantep May 19 '15 at 11:53

Play + ReactiveMongo: capped collection and tailable cursor

1 Answers1

Linked