How to get a result from Enumerator/Iteratee?

Question

I am using play2 and reactivemongo to fetch a result from mongodb. Each item of the result needs to be transformed to add some metadata. Afterwards I need to apply some sorting to it.

To deal with the transformation step I use enumerate():

def ideasEnumerator = collection.find(query)
    .options(QueryOpts(skipN = page))
    .sort(Json.obj(sortField -> -1))
    .cursor[Idea]
    .enumerate()

Then I create an Iteratee as follows:

val processIdeas: Iteratee[Idea, Unit] =
  Iteratee.foreach[Idea] { idea =>
    resolveCrossLinks(idea) flatMap { idea =>
      addMetaInfo(idea.copy(history = None))
    }
  }

Finally I feed the Iteratee:

ideasEnumerator(processIdeas)

And now I'm stuck. Every example I saw does some println inside foreach, but seems not to care about a final result.

So when all documents are returned and transformed how do I get a Sequence, a List or some other datatype I can further deal with?

`Iteratee.foreach` is purely side effect oriented (it actually is the convention of `foreach` on most collection types) and returns `Unit`. What you want is probably a variation of `fold` instead. — vptheron, Jul 09 '14 at 17:45
If you're fine with a `List`, you can just use `enum.run(Iteratee.getChunks)`. — Travis Brown, Jul 09 '14 at 17:54

score 2 · Accepted Answer · answered Jul 09 '14 at 17:49

Change the signature of your Iteratee from Iteratee[Idea, Unit] to Iteratee[Idea, Seq[A]] where A is the type. Basically the first param of Iteratee is Input type and second param is Output type. In your case you gave the Output type as Unit.

Take a look at the below code. It may not compile but it gives you the basic usage.

ideasEnumerator.run(
  Iteratee.fold(List.empty[MyObject]) { (accumulator, next) => 
    accumulator + resolveCrossLinks(next) flatMap { next => 
      addMetaInfo(next.copy(history = None))
    } 
  }
) // returns Future[List[MyObject]]

As you can see, Iteratee is a simply a state machine. Just extract that Iteratee part and assign it to a val:

val iteratee = Iteratee.fold(List.empty[MyObject]) { (accumulator, next) => 
        accumulator + resolveCrossLinks(next) flatMap { next => 
          addMetaInfo(next.copy(history = None))
        } 
      }

and feel free to use it where ever you need to convert from your Idea to List[MyObject]

If an exception is thrown from the Iteratee, it doesn't get propogated to the Enumerator and hence the subsequent recoverWith in the Future.map {}.recoverWith{} doesn't work. How do we resolve this issue? — Anand, May 03 '17 at 15:41
Hey @Anand! The errors do get propagated. Just tried it :) Here's a gist: https://gist.github.com/vaedama/ec4019d50ef09dea1f554c3a856007d4 — Sudheer Aedama, May 03 '17 at 18:08

DanielKhan · Answer 2 · 2014-07-10T09:16:25.750

With the help of your answers I ended up with

val processIdeas: Iteratee[Idea, Future[Vector[Idea]]] = 
    Iteratee.fold(Future(Vector.empty[Idea])) { (accumulator: Future[Vector[Idea]], next:Idea) =>
  resolveCrossLinks(next) flatMap { next =>
    addMetaInfo(next.copy(history = None))
  } flatMap (ideaWithMeta => accumulator map (acc => acc :+ ideaWithMeta))
}


val ideas = collection.find(query)
  .options(QueryOpts(page, perPage))
  .sort(Json.obj(sortField -> -1))
  .cursor[Idea]
  .enumerate(perPage).run(processIdeas)

This later needs a ideas.flatMap(identity) to remove the returning Future of Futures but I'm fine with it and everything looks idiomatic and elegant I think.

The performance gained compared to creating a list and iterate over it afterwards is negligible though.

How to get a result from Enumerator/Iteratee?

2 Answers2