Handle Error elegantly with Scala sttp client

Question

I'm using the scala sttp client and in particular, using the Monix backend and I have the following code:

def httpTask(href: String): Task[HttpBinaryResponse] = {
  val request = basicRequest
    .get(uri"$href")

  AsyncHttpClientMonixBackend
    .resource()
    .use { backend =>
      request.send(backend).map {
        response: Response[Either[String, String]] =>
          println(s"For URL: $href >> Got response code: ${response.code}")
          HttpBinaryResponse(
            href,
            response.code.isSuccess,
            response.headers.map(elem => (elem.name, elem.value)))
      }
    }
}

I have a set of URL's that I run through this method:

hrefs.toSet.flatten.filter(_.startsWith("http")).map(httpTask) // Set[Task[HttpBinaryResponse]]

In my caller function, I then execute the task:

val taskSeq = appBindings.httpService.parse(parserFilter)
val chunks = taskSeq.sliding(30, 30).toSeq
val batchedTasks = chunks.map(chunk => Task.parSequence(chunk))
val allBatches = Task.sequence(batchedTasks).map(_.flatten)

I then pattern match on the materialized type and return, bit what I want here is not just a Success and a Failure, but I would like to have both the values at once:

allBatches.runToFuture.materialize.map {
          case Success(elem) =>
            Ok(
              Json
                .obj("status" -> "Ok", "URL size" -> s"${elem.size}")
            ).enableCors
          case Failure(err) =>
            Ok(
              Json
                .obj("status" -> "error", "message" -> s"${err.getMessage}")
            ).enableCors
        }

I need to get a List of URL's with a corresponding Success or a Failure message. With this approach of what I have, I get only either one of those, which is obvious as I pattern match. How can I collect both Success and Failure?

What's the type of `elem` at the end? Isn't it a list of all your HttpBinaryResponse, and you can look at them to count errors and successes? — Thilo, Mar 21 '21 at 03:59
`Set[Task[HttpBinaryResponse]]` is weird, because there is no meaningful `equals` for `Task`. I think you want to `.distinct` on your input urls, but still keep it a `Seq`, ending up with `Seq[Task[HttpBinaryResponse]]` — Thilo, Mar 21 '21 at 04:01
Set to Seq is not my concern! When the backend fails for a specific url, I end up in the Failure case and I could not do what I want which is to aggregate both failure and Success. — joesan, Mar 21 '21 at 04:03
I thought when BE fails, you get a `Response(Left(errormessage))`. But if it actually fails, you can recover the failed Task and change it to return an HttpBinaryResponse with an error status (or wrap the whole thing into another `Either`). You definitely have to prevent the Task to fail, otherwise the whole batch will fail and you end up with just a single Failure at the end. You need to end up with `Success(Seq(goodOrBad))` — Thilo, Mar 21 '21 at 04:05

Handle Error elegantly with Scala sttp client

0 Answers0