Parse Complex JSON in SCALA using spray-json

Question

I am trying to parse a JSON string to a case class in Scala (So I can do the filtering data processing etc). After some research I am going with spray-json as there are several examples on the link. Unfortunately the link does not show how to parse a JSON with nested fields that have arrays.

I am testing my code on a Scala notebook with the code below and it works.

// Dependencies
io.spray spray-json_2.10 1.3.2
import spray.json._
import DefaultJsonProtocol._ // if you don't supply your own Protocol (see below)

// simple source
val source = """{
  "EventId": "29ca61f3-b8b6-41e7-8236-802fa232e7cf",
  "Timestamp": "2016-03-09T20:14:07.5535193Z",
  "StartTime": "2016-03-09T02:51:04.397",
  "EndTime": "2016-03-09T02:51:04.397",
  "ActiveStates": "{\"No Motion\":1,\"Motion Detected\":1,\"Face Detected\":1}",
  "Created": "2016-03-09T02:51:04.397",
  "Modified": "2016-03-09T02:51:04.397"
}"""

// simple case class
case class claX(  EventId: String,
  Timestamp: String,
  StartTime: String,
  EndTime: String,
  ActiveStates: String,
  Created: String,
  Modified: String)


object MyJsonProtocol extends DefaultJsonProtocol {
  implicit val claXFormat = jsonFormat7(claX)
}
import MyJsonProtocol._
import spray.json._


val json = source.parseJson // parse string to json
val cx0 = json.convertTo[claX] // convert to class claX

My problem is when the JSON string has nested array in data that has a nested 'Product' class inside it. This is the sample JSON:

{
      "EventId": "29ca61f3-b8b6-41e7-8236-802fa232e7cf",
      "Timestamp": "2016-03-09T20:14:07.5535193Z",
      "StartTime": "2016-03-09T02:51:04.397",
      "EndTime": "2016-03-09T02:51:04.397",
      "ActiveStates": "{\"No Motion\":1,\"Motion Detected\":1,\"Face Detected\":1}",
      "Created": "2016-03-09T02:51:04.397",
      "Modified": "2016-03-09T02:51:04.397",
      "Data": {
        "AgeRange": {
          "Name": "30 - 35"
        },
        "Company": {
          "Id": "f3ad1744-0ead-458a-9416-852c43ccde24"
        },
        "CompanyType": {
          "Name": "Retailer"
        },
        "ConnectorType": {
          "Name": "Camera Capturing"
        },
        "Content": {
          "Ids": [
            "0c0f0a9a-fece-4b3e-abb4-0f508d357220"
          ]
        },
        "Customer": {
          "LoyaltyId": 0
        },
        "DeviceRegistries": [
          {
            "Id": "f19f5daa-e9b9-43d0-91a7-51da4fdd0e31",
            "DeviceName": "Company 3 Cooler",
            "DeviceType": "CCU"
          }
        ],
        "Emotion": {
          "Name": "Happy"
        },
        "Gender": {
          "Name": "Male"
        },
        "Products": [
          {
            "Name": "Molson Canadian",
            "ProductCategory": "Beverage",
            "InventoryTrackingNumberType": "SKU",
            "InventoryTrackingNumber": "438654935776",
            "ProductPrice": {
              "RetailPrice": 2.1,
              "RetailPriceSymbol": "?",
              "PromotionPrice": 1.8,
              "PromotionPriceSymbol": "?"
            }
          },
          {
            "Name": "Coors Original",
            "ProductCategory": "Beverage",
            "InventoryTrackingNumberType": "SKU",
            "InventoryTrackingNumber": "438654935775",
            "ProductPrice": {
              "RetailPrice": 1.1,
              "RetailPriceSymbol": "?",
              "PromotionPrice": 0.8,
              "PromotionPriceSymbol": "?"
            }
          },
          {
            "Name": "Coors Light",
            "ProductCategory": "Beverage",
            "InventoryTrackingNumberType": "SKU",
            "InventoryTrackingNumber": "438654935778",
            "ProductPrice": {
              "RetailPrice": 6.1,
              "RetailPriceSymbol": "?",
              "PromotionPrice": 5.8,
              "PromotionPriceSymbol": "?"
            }
          },
          {
            "Name": "Blue Moon",
            "ProductCategory": "Beverage",
            "InventoryTrackingNumberType": "SKU",
            "InventoryTrackingNumber": "438654935777",
            "ProductPrice": {
              "RetailPrice": 4.1,
              "RetailPriceSymbol": "?",
              "PromotionPrice": 3.8,
              "PromotionPriceSymbol": "?"
            }
          }
        ],
        "Race": {
          "Name": "Latin"
        },
        "Region": {
          "Name": "Region 01"
        },
        "SensorRegistry": {
          "Name": "Company 3 Camera 01"
        },
        "SensorType": {
          "Name": "Proximity"
        },
        "SfuRegistries": {
          "Ids": [
            "7effea8c-56dd-4905-bbc3-2158d14cd7cc",
            "24a7253d-174a-44f0-8145-483cc0f45adb",
            "bc970c8e-7e41-4889-859b-55c6a3f8ba5d",
            "46e599f5-8082-499f-b5d0-9d611409a652"
          ]
        },
        "Shelves": {
          "Ids": [
            "ea442504-7d64-4c01-bdde-1eb46e53b81c",
            "d6fe9c78-e21b-4a57-b620-99a7d94d46f9"
          ]
        },
        "State": {
          "Name": "Face Detected"
        },
        "StockLevel": {
          "OnHand": 0
        },
        "Store": {
          "Id": "268c852d-86b8-4b7c-b865-2f29a3e2307e"
        },
        "Unit": {
          "Id": "52c58781-b2bf-46ea-81ad-b9d9fbacb471"
        },
        "UnitType": {
          "Name": "5-Shelf Cooler"
        }
      },
      "id": "54bfd971-0fec-4e0e-87cc-851a697705e9"
    }

I've made two more case classes to manage the 'Product' and the 'Prices'

case class ProductPrice(RetailPrice: Double,
          RetailPriceSymbol: Double,
          PromotionPrice: Double,
          PromotionPriceSymbol: Double)

case class Product(Name: String,
        ProductCategory: String,
        InventoryTrackingNumberType: String,
        InventoryTrackingNumber: String,
        ProductPrice: ProductPrice)

What I don't know is how do you combine this so that the Data node in the JSON is parsed properly into a claXBig (where everything from the JSON string is correctly parsed. This is where I am tripping up:

case class claX2(  EventId: String,
  Timestamp: String,
  StartTime: String,
  EndTime: String,
  ActiveStates: String,
  Created: String,
  Modified: String,
  Data: Map[String, Any]) // <- how do I parse this and the nested products

object MyJsonProtocol2 extends DefaultJsonProtocol {
  implicit val claXFormat2 = jsonFormat8(claX2)
}

I'm also trying to load a larger JSON (collection of these 'events') using code outlined here

So I added a new case class below to handle the array of 'event's or claX2

case class claX2Collection(clax2s: Array[claX2])
    extends IndexedSeq[claX2] {
  def apply(index: Int) = clax2s(index) //<- not sure what this mean 
  def length = clax2s.length // or whether index is doing anything
}

I assume claX2Collection is correct as its compiling. But the code below is definitely wrong, but is needed to load the event collection from the JSON array

implicit object claX2JsonFormat extends RootJsonFormat[claX2]{
def write(f: claX2) = {
val buf = scala.collection.mutable.ArrayBuffer(
"events" -> JsString("claX2"), // <- error
"Timestamp" -> JsObject(f.Timestamp), // error
"StartTime" -> JsObject(f.StartTime), // error
"EndTime" -> JsObject(f.EndTime), // error
"ActiveStates" -> JsObject(f.ActiveStates), // error
"Created" -> JsObject(f.Created), // errors
"Modified" -> JsObject(f.Modified), // errors
"Data" -> JsObject(f.Data) // errors
)
}
def read(value:JsValue) = {
val jso = value.asJsObject
// not sure what to do here but
// assuming I have to pick out
val EventId = jso.fields.get("EventId")
Timestamp = jso.fields.get("Timestamp")
StartTime = jso.fields.get("StartTime")
EndTime = jso.fields.get("EndTime")
ActiveStates = jso.fields.get("ActiveStates")
Created = jso.fields.get("Created")
Modified = jso.fields.get("Modified")
Data = jso.fields.get("Data")
claX2(EventId,Timestamp,StartTime,EndTime,ActiveStates,Created,
Modified,Data)
}
}

When that is fixed it should be able to read this JSON:

    {
  "type": "EventCollection",

"events": [
{
  "EventId": "29ca61f3-b8b6-41e7-8236-802fa232e7cf",
  "Timestamp": "2016-03-09T20:14:07.5535193Z",
  "StartTime": "2016-03-09T02:51:04.397",
  "EndTime": "2016-03-09T02:51:04.397",
  "ActiveStates": "{\"No Motion\":1,\"Motion Detected\":1,\"Face Detected\":1}",
  "Created": "2016-03-09T02:51:04.397",
  "Modified": "2016-03-09T02:51:04.397",
  "Data": {
    "AgeRange": {
      "Name": "30 - 35"
    },
    "Company": {
      "Id": "f3ad1744-0ead-458a-9416-852c43ccde24"
    },
    "CompanyType": {
      "Name": "Retailer"
    },
    "ConnectorType": {
      "Name": "Camera Capturing"
    },
    "Content": {
      "Ids": [
        "0c0f0a9a-fece-4b3e-abb4-0f508d357220"
      ]
    },
    "Customer": {
      "LoyaltyId": 0
    },
    "DeviceRegistries": [
      {
        "Id": "f19f5daa-e9b9-43d0-91a7-51da4fdd0e31",
        "DeviceName": "Company 3 Cooler",
        "DeviceType": "CCU"
      }
    ],
    "Emotion": {
      "Name": "Happy"
    },
    "Gender": {
      "Name": "Male"
    },
    "Products": [
      {
        "Name": "Molson Canadian",
        "ProductCategory": "Beverage",
        "InventoryTrackingNumberType": "SKU",
        "InventoryTrackingNumber": "438654935776",
        "ProductPrice": {
          "RetailPrice": 2.1,
          "RetailPriceSymbol": "?",
          "PromotionPrice": 1.8,
          "PromotionPriceSymbol": "?"
        }
      },
      {
        "Name": "Coors Original",
        "ProductCategory": "Beverage",
        "InventoryTrackingNumberType": "SKU",
        "InventoryTrackingNumber": "438654935775",
        "ProductPrice": {
          "RetailPrice": 1.1,
          "RetailPriceSymbol": "?",
          "PromotionPrice": 0.8,
          "PromotionPriceSymbol": "?"
        }
      },
      {
        "Name": "Coors Light",
        "ProductCategory": "Beverage",
        "InventoryTrackingNumberType": "SKU",
        "InventoryTrackingNumber": "438654935778",
        "ProductPrice": {
          "RetailPrice": 6.1,
          "RetailPriceSymbol": "?",
          "PromotionPrice": 5.8,
          "PromotionPriceSymbol": "?"
        }
      },
      {
        "Name": "Blue Moon",
        "ProductCategory": "Beverage",
        "InventoryTrackingNumberType": "SKU",
        "InventoryTrackingNumber": "438654935777",
        "ProductPrice": {
          "RetailPrice": 4.1,
          "RetailPriceSymbol": "?",
          "PromotionPrice": 3.8,
          "PromotionPriceSymbol": "?"
        }
      }
    ],
    "Race": {
      "Name": "Latin"
    },
    "Region": {
      "Name": "Region 01"
    },
    "SensorRegistry": {
      "Name": "Company 3 Camera 01"
    },
    "SensorType": {
      "Name": "Proximity"
    },
    "SfuRegistries": {
      "Ids": [
        "7effea8c-56dd-4905-bbc3-2158d14cd7cc",
        "24a7253d-174a-44f0-8145-483cc0f45adb",
        "bc970c8e-7e41-4889-859b-55c6a3f8ba5d",
        "46e599f5-8082-499f-b5d0-9d611409a652"
      ]
    },
    "Shelves": {
      "Ids": [
        "ea442504-7d64-4c01-bdde-1eb46e53b81c",
        "d6fe9c78-e21b-4a57-b620-99a7d94d46f9"
      ]
    },
    "State": {
      "Name": "Face Detected"
    },
    "StockLevel": {
      "OnHand": 0
    },
    "Store": {
      "Id": "268c852d-86b8-4b7c-b865-2f29a3e2307e"
    },
    "Unit": {
      "Id": "52c58781-b2bf-46ea-81ad-b9d9fbacb471"
    },
    "UnitType": {
      "Name": "5-Shelf Cooler"
    }
  },
  "id": "54bfd971-0fec-4e0e-87cc-851a697705e9"
},
{
  "EventId": "29ca61f3-b8b6-41e7-8236-802fa232e7cf",
  "Timestamp": "2016-03-09T20:14:07.5535193Z",
  "StartTime": "2016-03-09T02:51:04.397",
  "EndTime": "2016-03-09T02:51:04.397",
  "ActiveStates": "{\"No Motion\":1,\"Motion Detected\":1,\"Face Detected\":1}",
  "Created": "2016-03-09T02:51:04.397",
  "Modified": "2016-03-09T02:51:04.397",
  "Data": {
    "AgeRange": {
      "Name": "30 - 35"
    },
    "Company": {
      "Id": "f3ad1744-0ead-458a-9416-852c43ccde24"
    },
    "CompanyType": {
      "Name": "Retailer"
    },
    "ConnectorType": {
      "Name": "Camera Capturing"
    },
    "Content": {
      "Ids": [
        "0c0f0a9a-fece-4b3e-abb4-0f508d357220"
      ]
    },
    "Customer": {
      "LoyaltyId": 0
    },
    "DeviceRegistries": [
      {
        "Id": "f19f5daa-e9b9-43d0-91a7-51da4fdd0e31",
        "DeviceName": "Company 3 Cooler",
        "DeviceType": "CCU"
      }
    ],
    "Emotion": {
      "Name": "Happy"
    },
    "Gender": {
      "Name": "Male"
    },
    "Products": [
      {
        "Name": "Molson Canadian",
        "ProductCategory": "Beverage",
        "InventoryTrackingNumberType": "SKU",
        "InventoryTrackingNumber": "438654935776",
        "ProductPrice": {
          "RetailPrice": 2.1,
          "RetailPriceSymbol": "?",
          "PromotionPrice": 1.8,
          "PromotionPriceSymbol": "?"
        }
      },
      {
        "Name": "Coors Original",
        "ProductCategory": "Beverage",
        "InventoryTrackingNumberType": "SKU",
        "InventoryTrackingNumber": "438654935775",
        "ProductPrice": {
          "RetailPrice": 1.1,
          "RetailPriceSymbol": "?",
          "PromotionPrice": 0.8,
          "PromotionPriceSymbol": "?"
        }
      },
      {
        "Name": "Coors Light",
        "ProductCategory": "Beverage",
        "InventoryTrackingNumberType": "SKU",
        "InventoryTrackingNumber": "438654935778",
        "ProductPrice": {
          "RetailPrice": 6.1,
          "RetailPriceSymbol": "?",
          "PromotionPrice": 5.8,
          "PromotionPriceSymbol": "?"
        }
      },
      {
        "Name": "Blue Moon",
        "ProductCategory": "Beverage",
        "InventoryTrackingNumberType": "SKU",
        "InventoryTrackingNumber": "438654935777",
        "ProductPrice": {
          "RetailPrice": 4.1,
          "RetailPriceSymbol": "?",
          "PromotionPrice": 3.8,
          "PromotionPriceSymbol": "?"
        }
      }
    ],
    "Race": {
      "Name": "Latin"
    },
    "Region": {
      "Name": "Region 01"
    },
    "SensorRegistry": {
      "Name": "Company 3 Camera 01"
    },
    "SensorType": {
      "Name": "Proximity"
    },
    "SfuRegistries": {
      "Ids": [
        "7effea8c-56dd-4905-bbc3-2158d14cd7cc",
        "24a7253d-174a-44f0-8145-483cc0f45adb",
        "bc970c8e-7e41-4889-859b-55c6a3f8ba5d",
        "46e599f5-8082-499f-b5d0-9d611409a652"
      ]
    },
    "Shelves": {
      "Ids": [
        "ea442504-7d64-4c01-bdde-1eb46e53b81c",
        "d6fe9c78-e21b-4a57-b620-99a7d94d46f9"
      ]
    },
    "State": {
      "Name": "Face Detected"
    },
    "StockLevel": {
      "OnHand": 0
    },
    "Store": {
      "Id": "268c852d-86b8-4b7c-b865-2f29a3e2307e"
    },
    "Unit": {
      "Id": "52c58781-b2bf-46ea-81ad-b9d9fbacb471"
    },
    "UnitType": {
      "Name": "5-Shelf Cooler"
    }
  },
  "id": "54bfd971-0fec-4e0e-87cc-851a697705e9"
}
]
}

never used spray-json, usually I go with play-json. If you are willing to give it a try, look at http://pedrorijo.com/blog/scala-json/ http://pedrorijo.com/blog/scala-json-part2/ — pedrorijo91, Mar 30 '16 at 16:30
The fastest solution - change Map[String, Any] to JsObject, next create next case class for it. — Andrzej Jozwik, Mar 30 '16 at 18:25
@pedrorijo91 thanks for both links, reading them and will add it to the toolbox. — Rocket Surgeon, Mar 31 '16 at 07:41

score 3 · Answer 1 · answered Apr 19 '16 at 14:34

This is the complete solution The git repo for the solution and data files is here

    import spray.json._
import DefaultJsonProtocol._

object parseJson {
    def main(args: Array[String]){

    // case classes for all the nested information in the 
    case class Ids(Ids: Seq[String])
    case class Id(Id: String)
    case class Name(Name: String)
    case class OnHand(OnHand: Int)
    case class LoyaltyId(LoyaltyId: Int)

    case class ProductPrice(RetailPrice: Double,
                            RetailPriceSymbol: String,
                            PromotionPrice: Double,
                            PromotionPriceSymbol: String)

    case class Product(Name: String,
                       ProductCategory: String,
                       InventoryTrackingNumberType: String,
                       InventoryTrackingNumber: String,
                       ProductPrice: ProductPrice)

    case class Data(Content: Ids,
                    SfuRegistries: Ids,
                    AgeRange: Name,
                    Company: Id,
                    SensorType: Name,
                    StockLevel: OnHand,
                    Region: Name,
                    UnitType: Name,
                    Emotion: Name,
                    Shelves: Ids,
                    Customer: LoyaltyId,
                    DeviceRegistries: Seq[Map[String, String]],
                    ConnectorType: Name,
                    CompanyType: Name,
                    State: Name,
                    Gender: Name,
                    SensorRegistry: Name,
                    Race: Name,
                    Store: Id,
                    Products: Seq[Product])

    case class Element(EventId: String,
                     Timestamp: String,
                     StartTime: String,
                     EndTime: String,
                     ActiveStates: String,
                     Created: String,
                     Modified: String,
                     Data: Data)

 // This is the code that is blowing up
 case class RootCollection(items: Array[Element]) extends IndexedSeq[Element]{
    def apply(index: Int) = items(index)
    def length = items.length
}

object MyJsonProtocol extends DefaultJsonProtocol {
  implicit val nameFormat = jsonFormat1(Name)
  implicit val productPriceFormat = jsonFormat4(ProductPrice)
  implicit val productFormat = jsonFormat5(Product)
  implicit val loyaltyIdFormat = jsonFormat1(LoyaltyId)
  implicit val onHandFormat = jsonFormat1(OnHand)
  implicit val idFormat = jsonFormat1(Id)
  implicit val idsFormat = jsonFormat1(Ids)
  implicit val dateFormat = jsonFormat20(Data)
  implicit val ElementFormat = jsonFormat8(Element)  
  implicit object RootCollectionFormat extends RootJsonFormat[RootCollection] {
    def read(value: JsValue) = RootCollection(value.convertTo[Array[Element]])
    def write(f: RootCollection) = JsArray(f.toJson)
  }
 }

 import MyJsonProtocol._

        println("Running Parse JSON")
        val input = scala.io.Source.fromFile("sample2.json")("UTF-8").mkString.parseJson
        //println("JSON string read:")
        //println(input)

        val jsonCollection = input.convertTo[RootCollection]

        // print some items
        jsonCollection.map(y => y.Data.Products.map(x => println(x)))
        println(jsonCollection.length)

    }
}

score 0 · Answer 2 · answered Mar 30 '16 at 19:35

More less like :-)

  case class claX2(
    EventId: String,
    Timestamp: String,
    StartTime: String,
    EndTime: String,
    ActiveStates: String,
    Created: String,
    Modified: String,
    Data: Data)

  case class Data(Content: Ids,
    SfuRegistries: Ids,
    AgeRange: Name,
    Company: Id,
    SensorType: Name,
    StockLevel: OnHand,
    Region: Name,
    UnitType: Name,
    Emotion: Name,
    Shelves: Ids,
    Customer: LoyaltyId,
    DeviceRegistries: Seq[Map[String, String]],
    ConnectorType: Name,
    CompanyType: Name,
    State: Name,
    Gender: Name,
    SensorRegistry: Name,
    Race: Name,
    Store: Id,
    Products: Seq[Product])

  case class Ids(Ids: Seq[String])

  case class Id(Id: String)

  case class Name(Name: String)

  case class OnHand(OnHand: Int)

  case class LoyaltyId(LoyaltyId: Int)

  case class ProductPrice(RetailPrice: Double, 
   RetailPriceSymbol: String,
    PromotionPrice: Double,
    PromotionPriceSymbol: String)

  case class Product(Name: String, 
                    ProductCategory: String, 
                    InventoryTrackingNumberType: String,
                    InventoryTrackingNumber: String, 
                    ProductPrice: ProductPrice)

  object MyJsonProtocol2 extends DefaultJsonProtocol {
    implicit val nameFormat = jsonFormat1(Name)
    implicit val productPriceFormat = jsonFormat4(ProductPrice)
    implicit val productFormat = jsonFormat5(Product)
    implicit val loyaltyIdFormat = jsonFormat1(LoyaltyId)
    implicit val onHandFormat = jsonFormat1(OnHand)
    implicit val idFormat = jsonFormat1(Id)
    implicit val idsFormat = jsonFormat1(Ids)
    implicit val dateFormat = jsonFormat20(Data)
    implicit val claXFormat2 = jsonFormat8(claX2)
  }

@AndrzeyJoswik the solution worked for the sample (thanks), will have to make it work for the array of claX2 in the massive json file I will soon have. — Rocket Surgeon, Mar 31 '16 at 07:36
If you have generic mapping you can use only JsObject instead of Map[String, Any] or Map[String,JsObject] and then parse next part. — Andrzej Jozwik, Mar 31 '16 at 07:43
@AndrzeyJoswick I've been trying to adapt the solution of loading the JSON event collection by adapting the code frame work here [link](https://github.com/sryza/aas/blob/master/ch08-geotime/src/main/scala/com/cloudera/datascience/geotime/GeoJson.scala) so I can use .filter .map .foreach etc on each of the event elements. Can't parse this though — Rocket Surgeon, Apr 01 '16 at 11:35
Added the code that I am trying to adapt to read the elements array in the JSON — Rocket Surgeon, Apr 01 '16 at 12:25

Parse Complex JSON in SCALA using spray-json

2 Answers2