Cannot created object walking though JSON string using Circe JSON parser

Question

So i am trying to create a custom decoder for a JSON string to be converted into a domain object. I am using Scala/Circe to walk through the JSON and create the object. I am unable to get this to run. I am sure i am not clear on how to walk through the JSON. Can someone advise please ?

This is the JSON in question ,

   {
  "args": {}, 
  "headers": {
    "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3", 
    "Accept-Encoding": "gzip, deflate", 
    "Accept-Language": "en-US,en;q=0.9", 
    "Host": "httpbin.org", 
    "Upgrade-Insecure-Requests": "1", 
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.108 Safari/537.36"
  }, 
  "origin": "61.16.136.118, 61.16.136.118", 
  "url": "https://httpbin.org/get"
}

Here is what my code looks like,

    import com.softwaremill.sttp._
import com.softwaremill.sttp.circe._
import io.circe._, io.circe.parser._

case class Headers(Accept: String, Accept_Encoding: String, Accept_Language: String, Host: String, Upgrade_Insecure_Requests: String, User_Agent: String)
case class RootInterface(args: String, headers: Headers, origin: String, url: String)

object SttpClientGetPost extends App {
  implicit val backend = HttpURLConnectionBackend()

  implicit val rootDecoder: Decoder[RootInterface] =
    (hCursor: HCursor) => {
      val tcursor = hCursor.downField("headers")
      val argsCursor = hCursor.downField("args")
        for{
          args <- for{
          testString <-  argsCursor.get[String]("args")
        }yield testString

        headers <- for {
          Accept <- tcursor.downField("Accept").as[String]
          Accept_Encoding <- tcursor.downField("Accept-Encoding").as[String]
          Accept_Language <- tcursor.downField("Accept-Language").as[String]
          Host <- tcursor.downField("Host").as[String]
          Upgrade_Insecure_Requests <- tcursor.downField("Upgrade-Insecure-Requests").as[String]
          User_Agent <- tcursor.downField("User-Agent").as[String]
        } yield Headers(Accept, Accept_Encoding, Accept_Language, Host, Upgrade_Insecure_Requests, User_Agent)
        origin <- hCursor.downField("Origin").as[String]
        url <- hCursor.downField("url").as[String]
      } yield RootInterface("", headers, origin, url)
    }
  val secondRequest = sttp //.headers(("userId", USER_ID),("password","testpassword"))
    .get(uri"http://httpbin.org/get")

  secondRequest.send().body match {
    case Left(fail) => System.out.println("The Get request was unsuccessful.   " + fail)
    case Right(rawJSONResponse) =>
      parser.decode(rawJSONResponse) match {
        case Left(fail) => System.out.println("The JSON response could not be parsed by Circe.  " + fail)
        case Right(rootInterfaceObject) =>
          System.out.println("Origin :" + rootInterfaceObject.origin)

      }
  }

}

Update : I am trying with this to no avail now,

case class Headers(Accept: String, Accept_Encoding: String, Accept_Language: String, Host: String, Upgrade_Insecure_Requests: String, User_Agent: String)
case class RootInterface(args: String, headers: Headers, origin: String, url: String)


val doc = """{
  "args": {}, 
  "headers": {
    "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3", 
    "Accept-Encoding": "gzip, deflate", 
    "Accept-Language": "en-US,en;q=0.9", 
    "Host": "httpbin.org", 
    "Upgrade-Insecure-Requests": "1", 
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.108 Safari/537.36"
  }, 
  "origin": "61.16.136.118, 61.16.136.118", 
  "url": "https://httpbin.org/get"
}"""


  implicit val decodeHeaders: Decoder[Headers] = Decoder.forProduct6(
  "Accept",
  "Accept-Encoding",
  "Accept-Language",
  "Host",
  "Upgrade-Insecure-Requests",
  "User-Agent"
 )(Headers(_, _, _, _, _, _))

implicit val decodeRootInterface: Decoder[RootInterface] = Decoder.forProduct4(
  "args",  
  "headers",
  "origin",
  "url"
)(RootInterface(_, _, _, _))

val t = decode[RootInterface](doc)

I am still getting a decoding failure. It looks like its because of the args field.

score 4 · Accepted Answer · answered Apr 24 '19 at 15:29

I'd strongly recommend building up your instances compositionally—e.g. instead of baking all of the Headers decoding logic into the RootInterface decoder, you can define a separate Decoder[Headers] instance and then use that in your Decoder[RootInterface].

I'd also recommend avoiding working with cursors directly unless you're doing something particularly complicated that you know requires that level of definition.

So given this document:

val doc = """{
  "args": {}, 
  "headers": {
    "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3", 
    "Accept-Encoding": "gzip, deflate", 
    "Accept-Language": "en-US,en;q=0.9", 
    "Host": "httpbin.org", 
    "Upgrade-Insecure-Requests": "1", 
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.108 Safari/537.36"
  }, 
  "origin": "61.16.136.118, 61.16.136.118", 
  "url": "https://httpbin.org/get"
}"""

…and these case classes:

case class Headers(Accept: String, Accept_Encoding: String, Accept_Language: String, Host: String, Upgrade_Insecure_Requests: String, User_Agent: String)
case class RootInterface(args: String, headers: Headers, origin: String, url: String)

…the following is a complete working example you can use in a REPL:

import io.circe.Decoder, io.circe.jawn.decode

implicit val decodeHeaders: Decoder[Headers] = Decoder.forProduct6(
  "Accept",
  "Accept-Encoding",
  "Accept-Language",
  "Host",
  "Upgrade-Insecure-Requests",
  "User-Agent"
 )(Headers(_, _, _, _, _, _))

implicit val decodeRootInterface: Decoder[RootInterface] = Decoder.forProduct3(
  "headers",
  "origin",
  "url"
)(RootInterface("", _, _, _))

…like this:

scala> decode[RootInterface](doc)
res0: Either[io.circe.Error,RootInterface] = Right(RootInterface(,Headers(text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3,gzip, deflate,en-US,en;q=0.9,httpbin.org,1,Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.108 Safari/537.36),61.16.136.118, 61.16.136.118,https://httpbin.org/get))

(I'm not sure what the intention is with the args decoding, so I've just followed your implementation in using the empty string.)

If you control the Headers case class definition, though, I'd strongly suggest using idiomatic Scala member names (i.e. camel case, not upper snake). Combining that with circe-derivation makes a pretty clear solution in my view:

import io.circe.Decoder, io.circe.derivation.deriveDecoder

case class Headers(
  accept: String,
  acceptEncoding: String,
  acceptLanguage: String,
  host: String,
  upgradeInsecureRequests: String,
  userAgent: String
)

object Headers {
  implicit val decodeHeaders: Decoder[Headers] = deriveDecoder(
    _.replaceAll("([A-Z])", "-$1").capitalize
  )
}

case class RootInterface(
  args: Map[String, String],
  headers: Headers,
  origin: String,
  url: String
)

object RootInterface {
  implicit val decodeRootInterface: Decoder[RootInterface] = deriveDecoder
}

And then:

scala> io.circe.jawn.decode[RootInterface](doc)
res0: Either[io.circe.Error,RootInterface] = Right(RootInterface(Map(),Headers(text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3,gzip, deflate,en-US,en;q=0.9,httpbin.org,1,Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.108 Safari/537.36),61.16.136.118, 61.16.136.118,https://httpbin.org/get))

Much thanks for the details. I will try both these approaches that you mentioned and get back to you. For starters i do not understand what is this `implicit val decodeHeaders: Decoder[Headers] = deriveDecoder( _.replaceAll("([A-Z])", "-$1").capitalize` ? ) doing ? — Som Bhattacharyya, Apr 24 '19 at 18:02
@SomBhattacharyya It transforms the member names from the Scala-idiomatic camel case (`acceptHeader`) to the incoming `Accept-Header` format. — Travis Brown, Apr 24 '19 at 18:51
I am unable to make the first way work. Updated my question with my code. — Som Bhattacharyya, May 06 '19 at 14:16
It looks like you're trying to decode a JSON object into a `String`. How do you want this to work? — Travis Brown, May 06 '19 at 19:09
Yeah so i am trying to make a decoder that can decode the given JSON string. What i have changed in this code is i am trying to decode the `args` field too. Is that causing this ?Are you saying i need to have a decoder for args separately ? — Som Bhattacharyya, May 07 '19 at 03:47
You're going to need some kind of mapping from JSON objects to strings for the value of the `args` field. What do you want that to look like? — Travis Brown, May 07 '19 at 07:19
Yeah i think i just got it working with a `Map[String,String]` mapping for args. I was using `String` which is not correct. Cant thank you enough for the eager help. — Som Bhattacharyya, May 07 '19 at 16:30

Cannot created object walking though JSON string using Circe JSON parser

1 Answers1