parse recursively nested Json structure with Play framework

Question

we are using Play framework 2.3.4. From one of the APIs we make a web service call to a third party service - structure of the returned response is dynamic and may change. Only sub-structure that's static within the response JSON is a particular element and nesting inside it. for e.g.

{
 "response": 
  {
   "someElement1": "",
   "element2": true,
   "buckets": [
     {
       "key": "keyvalue",
       "docCount": 10,
       "somethingElse": {
         "buckets": [
           {
             "key": "keyvalue1",
             "docCount": 5,
             "somethingElseChild": {
               "buckets": [
                 {
                   "key": "Tan",
                   "docCount": 1
                 }
               ]
             }
           },
           {
             "key": "keyvalue2",
             "docCount": 3,
             "somethingElseChild": {
               "buckets": [
                 {
                   "key": "Ban",
                   "docCount": 6
                 }
               ]
             }
           }
         ]
       }
     }
   ]
  }
}

we don't know how the response structure is going to look like but ONLY thing we know is that there will be "buckets" nested elements somewhere in the response and as you can see there are other nested "buckets" inside a top level "buckets" element. also please note that structure inside buckets array is also not clear and if there will be another sub bucket it's definite that sub bucket must be somewhere inside parent bucket - so that pattern is consistent.

what's the best way to parse such recursive structure and populate following Bucket class recursively?

case class Bucket(key:String,docCount, subBuckets: List[Bucket] )

First I was thinking to

val json = Json.parse(serviveResponse)
val buckets = (json \ "response" \\ "buckets")

but that will not bring bring buckets recursively and not right way to traverse.

Any ideas?

Would you mind depending on http://argonaut.io/ + https://github.com/alexarchambault/argonaut-shapeless ? — Reactormonk, Feb 23 '16 at 16:02
@Reactormonk : although I would have preferred to stay with play json given we already have play json heavily used elsewhere in the project. I am open to use argnaut for this particular requirement if argonaut is going to offer elegant/efficient solution to the problem at hand. I had only glanced over argonaut but never used it. would you mind sharing more detail on how can this be achieved using argonaut? — user2066049, Feb 23 '16 at 16:11
@Reactormonk : just added argonaut tag to the question - if you can share your answer. — user2066049, Feb 23 '16 at 16:18
Using argonaut is a side-tangent. You just want to recursively build your data structure, right? Write a recursive method. — Rex Kerr, Feb 23 '16 at 16:37
@RexKerr : sure. my question is not around how to write a recursive method; but how to look for "buckets" recursively using JsPath (https://www.playframework.com/documentation/2.2.x/api/scala/index.html#play.api.libs.json.JsPath )- where I don't have knowledge of path upfront. — user2066049, Feb 23 '16 at 16:43
@user2066049 You use `buckets` again and call `buckets \ "buckets"` etc. — Reactormonk, Feb 23 '16 at 16:50

score 1 · Answer 1 · answered Feb 23 '16 at 18:56

To make a Reads[T] for a recursive type T, you have to

define it as a lazy val,
use lazyRead where you need recursive parsing,
and manually pass to it the Reads[T] object or its derivative.

Of course you have to know what paths exactly the buckets element may appear at, and also account for it missing from any of those. You can use orElse to try several paths.

For your definition of Bucket, the Reads may look like this:

import play.api.libs.json._
import play.api.libs.functional.syntax._

implicit lazy val readBucket: Reads[Bucket] = (
  (__ \ "key").read[String] and
  (__ \ "docCount").read[Int] and
  (
    (__ \ "somethingElse" \ "buckets").lazyRead(Reads.list(readBucket)) orElse
    (__ \ "somethingElseChild" \ "buckets").lazyRead(Reads.list(readBucket)) orElse
    Reads.pure(List.empty[Bucket])
  )
) (Bucket.apply _)

You can simplify it a bit by extracting the common part to a function, e.g.:

def readsBucketsAt(path: JsPath): Reads[List[Bucket]] =
  (path \ "buckets").lazyRead(Reads.list(readBucket))

/* ... 
  readsBucketsAt(__ \ "somethingElse") orElse
  readsBucketsAt(__ \ "somethingElseChild") orElse
  Reads.pure(List.empty[Bucket])
... */

This example doesn't account for possible merging of several buckets arrays at different paths inside a single bucket. So if you need that functionality, I believe you'd have to define and use a play.api.libs.functional.Monoid instance for Reads[List[T]], or somehow combine the existing monoid instances for JsArray.

score 0 · Answer 2 · answered Feb 23 '16 at 16:54

Parse recursively. Something like this (not tested):

case class Bucket(key: String, count: Int, sub: List[Bucket])

def FAIL = throw new Exception  // Put better error-handling here

def getBucket(js: JsValue): Bucket = js match {
  case o: JsObject =>
    val key = (o \ "key") match {
      case JsString(s) => s
      case _ => FAIL
    }
    val count = (o \ "docCount") match {
      case JsNumber(n) => n.toInt
      case _ => FAIL
    }
    val sub = (o \ "buckets") match {
      case a: JsArray => a.value.toList.map(getBucket)
      case _ => Nil
    }
    Bucket(key, count, sub)
  case _ => throw new Exception
}

parse recursively nested Json structure with Play framework

2 Answers2