How to stream mongo data with alpakka without any memory issue in lesser time

Question

I'm new to alpakka. I used following code in Alpakka using MongoDB connector to fetch and loop through 100K records

// Using Stream
def getAllContacts(user_id: Int, list_id: Int): Source[ListContact, NotUsed] = {
        MongoSource[ListContact](mongoDB.getCollection(getCollectionName(user_id, list_id)).find())
    }
// Mongo Observable
    def getAllContacts2(user_id: Int, list_id: Int): FindObservable[ListContact] = {
        mongoDB.getCollection(getCollectionName(user_id, list_id)).find()
    }

Implementation:

def validateUserList2(user_id: Int, list_id: Int, prefix: String, filter: Option[String]): Future[Seq[PhoneNumber]] = {
    val prefixTrim = prefix.trim
    val listContact = new ListContactRepository
    listContact.getAllContacts2(user_id, list_id).map{ line =>
        validateNumber(line.phone, prefixTrim)
    }.toFuture()
}

def validateUserList(user_id: Int, list_id: Int, prefix: String, filter: Option[String]): Future[immutable.Seq[PhoneNumber]] = {
    val prefixTrim = prefix.trim
    val listContact = new ListContactRepository
    listContact.getAllContacts(user_id, list_id).runWith(Sink.seq).map(lines =>
        lines map { line =>
            validateNumber(line.phone, prefixTrim)
        }
    )
}

And Route:

// Taking around 2.5 seconds to fetch 100K data
private def mapUserListNetwork: Route = {
    get {
        path("validate" / Segment / Segment / Segment) { (userId, listId, prefix) =>
            parameters('filter.?) { filter =>
                complete((phoneValidationActor ? ValidateUserList(userId.toInt, listId.toInt, prefix.toUpperCase, filter)).mapTo[Seq[PhoneNumber]])
            }
        }
    }
}
// Taking around 10 seconds to fetch 100K data
private def mapUserListNetwork2: Route = {
    get {
        path("validate2" / Segment / Segment / Segment) { (userId, listId, prefix) =>
            parameters('filter.?) { filter =>
                complete(PhoneNumberActor.validateUserList2(userId.toInt, listId.toInt, prefix, filter))
            }
        }
    }
}

I wanted to stream the mongodb collection to akka-http without any Out of Memory issue/Heap Space. Please suggest the better approach.

You could also have a look at http://reactivemongo.org/releases/0.1x/documentation/tutorial/streaming.html (i'm contributor of) — cchantep, Jan 01 '19 at 19:22

score 0 · Answer 1 · answered Jan 02 '19 at 12:36

In the implementation of validateUserList instead of running (.runWith(Sink.seq)) the source, return it from that method. You can still apply the map operation there on the source.

Then complete the request with the Source. Akka Http can complete requests with sources, in which case it will take care of materialization and streaming the results to the client as a chunked response.

How to stream mongo data with alpakka without any memory issue in lesser time

1 Answers1