Spray http client and thousands requests

Question

I want to configure spray http client in a way that controls max number of request that were sent to the server. I need this because server that i'm sending requests to blocks me if there are more then 2 request were sent. I get

akka.pattern.AskTimeoutException: Ask timed out on [Actor[akka://smallTasks/user/IO-HTTP#151444590]] after [15000 ms]
akka.pattern.AskTimeoutException: Ask timed out on [Actor[akka://smallTasks/user/IO-HTTP#151444590]] after [15000 ms]
akka.pattern.AskTimeoutException: Ask timed out on [Actor[akka://smallTasks/user/IO-HTTP#151444590]] after [15000 ms]
akka.pattern.AskTimeoutException: Ask timed out on

I need to send thousands of requests but i get blocked after i got responses from ~ 100 requests.

I have this method:

  implicit val system = ActorSystem("smallTasks")
  implicit val timeout = new Timeout(15.seconds)

  import system.dispatcher

  def doHttpRequest(url: String): Future[HttpResponse] = {
    (IO(Http) ? HttpRequest(GET, Uri(url))).mapTo[HttpResponse]
  }

And here i catch responses and retry if it fails(recursively):

def getOnlineOffers(modelId: Int, count: Int = 0): Future[Any] = {

    val result = Promise[Any]()

    AkkaSys.doHttpRequest(Market.modelOffersUrl(modelId)).map(response => {
      val responseCode = response.status.intValue
      if (List(400, 404).contains(responseCode)) {
        result.success("Bad request")
      } else if (responseCode == 200) {
        Try {
          Json.parse(response.entity.asString).asOpt[JsObject]
        } match {
          case Success(Some(obj)) =>
            Try {
              (obj \\ "onlineOffers").head.as[Int]
            } match {
              case Success(offers) => result.success(offers)
              case _ => result.success("Can't find property")
            }

          case _ => result.success("Wrong body")
        }
      } else {
        result.success("Unexpected error")
      }
    }).recover { case err =>
      if (count > 5) {
        result.success("Too many tries")
      } else {
        println(err.toString)
        Thread.sleep(200)
        getOnlineOffers(modelId, count + 1).map(r => result.success(r))
      }
    }

    result.future

  }

How to do this properly? May be i need to configure akka dispatcher somehow?

What do you want to do if max concurrent connections has reached? Reject request? Wait with given timeout? — Denis Borovikov, Jul 01 '15 at 12:23
I need to collect information for each model, so if request fails i want to retry that failed request with timeout. I already did that (see Thread.sleep(200)). It works fine within 10 - 15 seconds , after that time i get a lot of Akka ask timeout exceptions in console. I think it happens because i have 10000 models and they all are launched concurrently. I'm looking for a way that force http client to send only 2 or 3 request per time and not to send other request while old request are without response — Alexander Kondaurov, Jul 01 '15 at 14:41

Александр Третьяк · Answer 1 · 2016-03-10T20:55:04.650

you can use http://spray.io/documentation/1.2.2/spray-client/ and write you personal pipeline

val pipeline: Future[SendReceive] =
      for (
        Http.HostConnectorInfo(connector, _) <-
          IO(Http) ? Http.HostConnectorSetup("www.spray.io", port = 80)
      ) yield sendReceive(connector)

    val request = Get("/segment1/segment2/...")
    val responseFuture: Future[HttpResponse] = pipeline.flatMap(_(request))

to get HttpResponse

import scala.concurrent.Await
import scala.concurrent.duration._
val response: HttpResponse = Aweit(responseFuture, ...)

to convert

import spray.json._
response.entity.asString.parseJson.convertTo[T]

to check

Try(response.entity.asString.parseJson).isSuccess

too many brackets. In scala you can write it shorter

Spray http client and thousands requests

1 Answers1