mongo-scala-driver: returning an iterator of results

Question

I am using Mongo Scala Driver to read documents from MongoDb. I have following method as follow which returns Future[List[T]]

def findAll(): List[Future[List[T]] = {
   db.getCollection(collectionName)
      .find(queryObject)
      .toFuture()
      .map{
        results => results.map(/* converting result to T */).toList
      }
}

Question:

Instead of a List, is there some way, this method can return Iterator[T] or Future[Iterator[T]] ?

The doc mentions about iterating the results via the subscribe method, but I cannot find any concrete implementation or examples of using this subscribe method to achieve my use case.

There is documentation and example for `subscribe` at http://mongodb.github.io/mongo-java-driver/4.3/driver-scala/reference/observables/ — Thilo, May 03 '21 at 09:58
@Thilo Thanks for the reference. The explained implementation would be returning Unit. So, could I transform it in such a way that it would return `Iterator[T]` ? I can't put all the logic inline within this method and therefore I need this method to return the Iterator. And, I want to achieve that functionally. Any suggestions ? Thanks — oblivion, May 03 '21 at 10:23
Do you really need to stream this? If it is not terribly large data, using `toFuture` will be easier to use. If you really want to use streaming, best use a library for that such as Monix Observables or ZIO Stream. They will have adapters for the callbacks that the Mongo driver uses. But that's a bit of a learning curve if you haven't used those before. — Thilo, May 03 '21 at 10:54
" I can't put all the logic inline within this method " You could pass a callback (of the form `Document => Unit`) into the method. Your logic would then be called for each document. — Thilo, May 03 '21 at 10:57
@Thilo "Do you really need to stream this?" . Yes, I have to feed the result as a Source to a streaming process. Therefore, I need an iterator. — oblivion, May 03 '21 at 12:18
What streaming process is that? Cannot the `Source` be driven by the document-consuming callback that the Mongo driver wants? — Thilo, May 03 '21 at 12:29
@Thilo The method signature to build a source is `Source.fromIterator` where signature is `def fromIterator[T](f: () => Iterator[T])` — oblivion, May 03 '21 at 12:35
There is an adapter for MongoDB to Akka Stream: https://doc.akka.io/docs/alpakka/current/mongodb.html It uses the Java driver instead of the Scala one, but it has things like `val source: Source[Number, NotUsed] = MongoSource(numbersColl.find(classOf[Number]))` This is built on `Source.fromPublisher` — Thilo, May 03 '21 at 12:37
The latest **Mongo** **Scala** driver is based on **reactive streams** which can be easily converted to a `Source` - An advice, never focus on a particular solution but rather on the real problem you want to solve a `Future[Iterator]` would not really be streaming. — Luis Miguel Mejía Suárez, May 03 '21 at 12:54
@LuisMiguelMejíaSuárez Can you please be more specific ? How can it be easily converted ? We are here to learn so sharing helps :) And, another question, how `Future[Iterator]` would not really be streaming ? What purpose would an Iterator solve other than streaming behavior and not loading everything up in memory at once ? May be I am missing something — oblivion, May 03 '21 at 13:48
@oblivion By just calling `Source.fromPublisher` check the [docs](https://doc.akka.io/api/akka/current/akka/stream/scaladsl/Source$.html#fromPublisher[T](publisher:org.reactivestreams.Publisher[T]):akka.stream.scaladsl.Source[T,akka.NotUsed]). A `Future[Iterator]` means that you have an async process that produced an **Iterator** since such async process is pulling the DB that means that by the time the **Future** completes all the data is in memory, as such the iterator is not really streaming; unless the **Iterator** will pull the DB again, but then it would be blocking the async process. — Luis Miguel Mejía Suárez, May 03 '21 at 14:11

mongo-scala-driver: returning an iterator of results

0 Answers0