Recursive HTTP-requests in Scala

Question

I need to do recursive requests and then collect all models into one List, but not understand how to do it. Please tell me am I thinking right way?

package kindSir.main

import dispatch.Defaults._
import dispatch._
import kindSir.models._
import org.json4s._
import org.json4s.jackson.JsonMethods._

object ApplicationMain extends App {

  def fetchMergeRequests(startPage: Int = 1): Future[List[MergeRequest]] = {
    val requestsUrl = url(s"https://gitlab.com/api/v4/projects/gitlab-org%2Fgitlab-ce/merge_requests?state=opened&per_page=3&page=${startPage}")
    Http(requestsUrl).map { res =>
      (parse(res.getResponseBody), res.getHeader("X-Next-Page").toInt) match {
        case (list@JArray(_), nextPage: Int) =>
          val currentList: List[MergeRequest] = MergeRequest.parseList(list).get
          val nextPageListFuture: Future[List[MergeRequest]] = fetchMergeRequests(nextPage)
          // And how to merge these two lists?
        case (list@JArray(_), _) => MergeRequest.parseList(list).get
        case _ => throw new RuntimeException(s"No merge requests for project found")
      }
    }
  }

}

Interesting. Were you explicitly asked to generate recursive http requests, like in a classroom setting? Or is there some business problem you're solving with this mechanism? — Don Branson, Oct 22 '18 at 18:54

score 1 · Accepted Answer · answered Oct 23 '18 at 04:44

The main problem you're dealing with here is that you're trying to combine data you already have (List[MergeRequest]) with the data you'll retrieve in future (Future[List[MergeRequest]]). There are a few things you need to do to handle this scenario:

Use flatMap instead of map on result of the HTTP request. This allows you to make further HTTP requests inside the recursion but map them back to a single Future.
Call map on the result of the recursion fetchMergeRequests(nextPage) to combine the data you already have with the future data from the recursion.
Wrap the other list in Future.successful() because flatMap requires all the pattern matches to return a Future — except for the exception.

I'm not familiar with the libraries you are using so I haven't tested it, but I think your code should work like this:

def fetchMergeRequests(startPage: Int = 1): Future[List[MergeRequest]] = {
  val requestsUrl = url(s"https://gitlab.com/api/v4/projects/gitlab-org%2Fgitlab-ce/merge_requests?state=opened&per_page=3&page=${startPage}")
  Http(requestsUrl).flatMap { res =>
    (parse(res.getResponseBody), res.getHeader("X-Next-Page").toInt) match {
      case (list@JArray(_), nextPage: Int) =>
        val currentList: List[MergeRequest] = MergeRequest.parseList(list).get
        val nextPageListFuture: Future[List[MergeRequest]] = fetchMergeRequests(nextPage)
        nextPageListFuture.map(nextPageList => currentList ++ nextPageList)
      case (list@JArray(_), _) =>
        Future.successful(MergeRequest.parseList(list).get)
      case _ => throw new RuntimeException(s"No merge requests for project found")
    }
  }
}

Thank you very much @gutch! I used your code to [propose request to open source project kind-sir](https://github.com/answr42/kind-sir/pull/6/files). — Vladimir Babin, Oct 23 '18 at 07:48

Recursive HTTP-requests in Scala

1 Answers1