Akka HTTP Streaming JSON Deserialization

Question

Is it possible to dynamically deserialize an external, of unknown length, ByteString stream from Akka HTTP into domain objects?

Context

I call an infinitely long HTTP endpoint that outputs a JSON Array that keeps growing:

[
    { "prop": true, "prop2": false, "prop3": 97, "prop4": "sample" },
    { "prop": true, "prop2": false, "prop3": 97, "prop4": "sample" },
    { "prop": true, "prop2": false, "prop3": 97, "prop4": "sample" },
    { "prop": true, "prop2": false, "prop3": 97, "prop4": "sample" },
    { "prop": true, "prop2": false, "prop3": 97, "prop4": "sample" },
    ...
] <- Never sees the daylight

For clarification, are you trying to receive this JSON stream or broadcast this stream? If broadcasting, what is your internal representation (e.g. Iterator, scala Stream, ...)? Also, does the communcation have to be an Array or can it be a stream of individual domain objects? — Ramón J Romero y Vigil, Dec 14 '15 at 19:42
In your particular case you could wait for closing `}` and call a deserializer of your choice for text in between. This requires some operations and probably buffering on ByteString, but they are quite basic. — Pavel Kudinov, Dec 15 '15 at 06:19
Do you really need a JSON array? Using a different separator (e.g. new line) for JSON documents will make this task easier. See http://doc.akka.io/docs/akka-stream-and-http-experimental/2.0-M2/scala/stream-cookbook.html#Parsing_lines_from_a_stream_of_ByteStrings — kostya, Dec 15 '15 at 06:59
@kostya I don't have control over the output of the endpoint — Martijn, Dec 15 '15 at 11:15

score 5 · Accepted Answer · answered Feb 05 '19 at 16:01

I guess that JsonFraming.objectScanner(Int.MaxValue) should be used in this case. As docs state:

Returns a Flow that implements a "brace counting" based framing operator for emitting valid JSON chunks. It scans the incoming data stream for valid JSON objects and returns chunks of ByteStrings containing only those valid chunks. Typical examples of data that one may want to frame using this operator include: Very large arrays

So you can end up with something like this:

val response: Future[HttpResponse] = Http().singleRequest(HttpRequest(uri = serviceUrl))

response.onComplete {
  case Success(value) =>
    value.entity.dataBytes
      .via(JsonFraming.objectScanner(Int.MaxValue))
      .map(_.utf8String)         // In case you have ByteString
      .map(decode[MyEntity](_))  // Use any Unmarshaller here
      .grouped(20)
      .runWith(Sink.ignore)      // Do whatever you need here 
  case Failure(exception) => log.error(exception, "Api call failed")
}

score 0 · Answer 2 · answered Jan 08 '16 at 18:54

I had a very similar problem trying to parse the Twitter Stream (an infinite string) into a domain object. I solved it using Json4s, like this:

case class Tweet(username: String, geolocation: Option[Geo])
case class Geo(latitude: Float, longitude: Float)
object Tweet{
    def apply(s: String): Tweet = {
        parse(StringInput(s), useBigDecimalForDouble = false, useBigIntForLong = false).extract[Tweet]
    }
}

Then I just buffer the stream and mapped it to a Tweet:

val reader = new BufferedReader(new InputStreamReader(new GZIPInputStream(inputStream), "UTF-8"))
var line = reader.readLine()
while(line != null){
    store(Tweet.apply(line))
    line = reader.readLine()
}

Json4s has full support over Option (or custom objects inside the object, like Geo in the example). Therefore, you can put an Option like I did, and if the field doesn't come in the Json, it will be set to None.

Hope it helps!

MrRontgen · Answer 3 · 2015-12-15T15:39:03.157

I think that play-iteratees-extras must help you. This library allow to parse Json via Enumerator/Iteratee pattern and, of course, don't waiting for receiving all data.

For example, lest build 'infinite' stream of bytes that represents 'infinite' Json array.

import play.api.libs.iteratee.{Enumeratee, Enumerator, Iteratee}

var i = 0
var isFirstWas = false

val max = 10000

val stream = Enumerator("[".getBytes) andThen Enumerator.generateM {
  Future {
    i += 1
    if (i < max) {
      val json = Json.stringify(Json.obj(
        "prop" -> Random.nextBoolean(),
        "prop2" -> Random.nextBoolean(),
        "prop3" -> Random.nextInt(),
        "prop4" -> Random.alphanumeric.take(5).mkString("")
      ))

      val string = if (isFirstWas) {
        "," + json
      } else {
        isFirstWas = true
        json
      }


      Some(Codec.utf_8.encode(string))
    } else if (i == max) Some("]".getBytes) // <------ this is the last jsArray closing tag
    else None

  }
}

Ok, this value contains jsArray of 10000 (or more) objects. Lets define case class that will be contain data of each object in our array.

case class Props(prop: Boolean, prop2: Boolean, prop3: Int, prop4: String)

Now write parser, that will be parse each item

import play.extras.iteratees._    
import JsonBodyParser._
import JsonIteratees._
import JsonEnumeratees._

val parser = jsArray(jsValues(jsSimpleObject)) ><> Enumeratee.map { json =>
  for {
    prop <- json.\("prop").asOpt[Boolean]
    prop2 <- json.\("prop2").asOpt[Boolean]
    prop3 <- json.\("prop3").asOpt[Int]
    prop4 <- json.\("prop4").asOpt[String]
  } yield Props(prop, prop2, prop3, prop4)
}

Please, see doc for jsArray, jsValues and jsSimpleObject. To build result producer:

val result = stream &> Encoding.decode() ><> parser

Encoding.decode() from JsonIteratees package will decode bytes as CharString. result value has type Enumerator[Option[Item]] and you can apply some iteratee to this enumerator to start parsing process.

In total, I don't know how you receive bytes (the solution depends heavily on this), but I think that show one of the possible solutions of your problem.

Akka HTTP Streaming JSON Deserialization

Context

3 Answers3