Decode a streaming GZIP response in Scala Dispatch?

Question

Receiving a Gzipped response from an API, but Dispatch 0.9.5 doesn't appear to have any methods to decode the response. Any ideas?

Here's my current implementation, the println only prints out string representations of bytes.

   Http(
      host("stream.gnip.com")
      .secure
      .addHeader("Accept-Encoding", "gzip")
       / gnipUrl
      > as.stream.Lines(println))()

Tried to look at implementing my own handler, but not sure where to begin. Here's the relevant file for Lines: https://github.com/dispatch/reboot/blob/master/core/src/main/scala/as/stream/lines.scala

Thanks!

I see you opened on issue on this at https://github.com/dispatch/reboot/issues/52 — Seth Tisue, Mar 24 '14 at 23:29

score 3 · Accepted Answer · answered Apr 04 '13 at 17:42

Simply abandoned Dispatch and used Java APIs directly. Disappointing, but it got the job done.

  val GNIP_URL = isDev match {
    case true => "https://url/apath/track/dev.json"
    case false => "https://url/path/track/prod.json"
  }
  val GNIP_CHARSET = "UTF-8"

  override def preStart() = {
    log.info("[tracker] Starting new Twitter PowerTrack connection to %s" format GNIP_URL)

    val connection = getConnection(GNIP_URL, GNIP_USER, GNIP_PASSWORD)
    val inputStream = connection.getInputStream()
    val reader = new BufferedReader(new InputStreamReader(new StreamingGZIPInputStream(inputStream), GNIP_CHARSET))
    var line = reader.readLine()
    while(line != null){
        println(line)
        line = reader.readLine()
    }
  }

  private def getConnection(urlString: String, user: String, password: String): HttpURLConnection = {
    val url = new URL(urlString)

    val connection = url.openConnection().asInstanceOf[HttpURLConnection]
    connection.setReadTimeout(1000 * 60 * 60)
    connection.setConnectTimeout(1000 * 10)

    connection.setRequestProperty("Authorization", createAuthHeader(user, password));
    connection.setRequestProperty("Accept-Encoding", "gzip")
    connection
  }

  private def createAuthHeader(username: String, password: String) = {
    val encoder = new BASE64Encoder()
    val authToken = username+":"+password
   "Basic "+encoder.encode(authToken.getBytes())
  }

Used GNIP's example: https://github.com/gnip/support/blob/master/Premium%20Stream%20Connection/Java/StreamingConnection.java

spray might meet your needs. Gzip example on this page http://spray.io/documentation/spray-client/#spray-client. — Carnell, Apr 05 '13 at 01:47
interesting, looks like i'll need to implement the raw client as the high-level client "doesn't yet support HTTP streaming". nonetheless, could be a big help :) — crockpotveggies, Apr 05 '13 at 16:59

score 3 · Answer 2 · answered Mar 24 '14 at 23:29

This isn't so much a solution as a workaround, but I ended up resorting to bypassing the Future-based stuff and doing:

val stream = Http(req OK as.Response(_.getResponseBodyAsStream)).apply val result = JsonParser.parse( new java.io.InputStreamReader( new java.util.zip.GZIPInputStream(stream)))

I'm using JsonParser here because in my case the data I'm receiving happens to be JSON; substitute with something else in your use case, if needed.

score 2 · Answer 3 · answered Jul 27 '15 at 10:46

My solution just defined a response parser and also adopted json4s parser:

    object GzipJson extends (Response => JValue) {
      def apply(r: Response) = {
        if(r.getHeader("content-encoding")!=null && r.getHeader("content-encoding").equals("gzip")){
          (parse(new GZIPInputStream(r.getResponseBodyAsStream), true))
        }else
          (dispatch.as.String andThen (s => parse(StringInput(s), true)))(r)
      }
    }

So that I can use it to extract Gzip Json response as the following code:

    import GzipJson._

    Http(req OK GzipJson).apply

score -1 · Answer 4 · answered Apr 04 '13 at 02:20

-1

Try >> instead of >

See https://github.com/dispatch/dispatch/blob/master/core/src/main/scala/handlers.scala#L58

answered Apr 04 '13 at 02:20

Carnell

749
6
10

I thought that myself, too, but that is actually only valid in dispatch `0.8` not `0.9`. If you check the package it was renamed to `dispatch.classic`. :( – crockpotveggies Apr 04 '13 at 04:09
p.s. here's the new handlers file: https://github.com/dispatch/reboot/blob/master/core/src/main/scala/handlers.scala – crockpotveggies Apr 04 '13 at 04:10
Ah I missed that. What do you get when you try as.string? Maybe the concern of dealing with the type doesn't matter any longer. Looks like someone else is trying to find the answer in the forums as well. https://groups.google.com/forum/#!searchin/dispatch-scala/gzip/dispatch-scala/xesmA7NasCM/dEFq04fdfVwJ – Carnell Apr 04 '13 at 15:24
That's me in the mailing list ;) Looks like `as.String` doesn't have an "iterable" way to continually deal with incoming chunks as it gives a blank output. I'm getting Mujibake when I use the `as.stream.Lines` handler even though I fixed the character encoding :( – crockpotveggies Apr 04 '13 at 16:48

Decode a streaming GZIP response in Scala Dispatch?

4 Answers4