Deserialize JSON distinguising missing and null values

Question

I have a requirement to parse a JSON object, using play-json and distinguish between a missing value, a string value and a null value.

So for example I might want to deserialize into the following case class:

case class MyCaseClass(
  a: Option[Option[String]]
)

Where the values of 'a' mean:

• None - "a" was missing - normal play-json behavipr
• Some(Some(String)) - "a" had a string value
• Some(None) - "a" had a null value

So examples of the expected behavior are:

{}

should deserialize to myCaseClass(None)

{
  "a": null
} 

should deserialize as myCaseClass(Some(None))

{
  "a": "a"
}

should deserialize as myCaseClass(Some(Some("a"))

I've tried writing custom formatters, but the formatNullable and formatNullableWithDefault methods don't distinguish between a missing and null value, so the code I've written below cannot generate the Some(None) result

object myCaseClass {
  implicit val aFormat: Format[Option[String]] = new Format[Option[String]] {
    override def reads(json: JsValue): JsResult[Option[String]] = {
      json match {
        case JsNull => JsSuccess(None) // this is never reached
        case JsString(value) => JsSuccess(Some(value))
        case _ => throw new RuntimeException("unexpected type")
      }
    }
    override def writes(codename: Option[String]): JsValue = {
      codename match {
        case None => JsNull
        case Some(value) =>  JsString(value)
      }
    }
  }

  implicit val format = (
      (__ \ "a").formatNullableWithDefault[Option[String]](None)
  )(MyCaseClass.apply, unlift(MyCaseClass.unapply))
}

Am I missing a trick here? How should I go about this? I am very much willing to encode the final value in some other way than an Option[Option[Sting]] for example some sort of case class that encapsulates this:

case class MyContainer(newValue: Option[String], wasProvided: Boolean)

Answer 1

I recently found a reasonable way to do this. I'm using Play 2.6.11 but I'm guessing the approach will transfer to other recent versions.

The following snippet adds three extension methods to JsPath, to read/write/format fields of type Option[Option[A]]. In each case a missing field maps to a None, a null to a Some(None), and a non-null value to a Some(Some(a)) as the original poster requested:

import play.api.libs.json._

object tristate {
  implicit class TriStateNullableJsPathOps(path: JsPath) {
    def readTriStateNullable[A: Reads]: Reads[Option[Option[A]]] =
      Reads[Option[Option[A]]] { value =>
        value.validate[JsObject].flatMap { obj =>
          path.asSingleJsResult(obj) match {
            case JsError(_)           => JsSuccess(Option.empty[Option[A]])
            case JsSuccess(JsNull, _) => JsSuccess(Option(Option.empty[A]))
            case JsSuccess(json, _)   => json.validate[A]
                                             .repath(path)
                                             .map(a => Option(Option(a)))
          }
        }
      }

    def writeTriStateNullable[A: Writes]: OWrites[Option[Option[A]]] =
      path.writeNullable(Writes.optionWithNull[A])

    def formatTriStateNullable[A: Format]: OFormat[Option[Option[A]]] =
      OFormat(readTriStateNullable[A], writeTriStateNullable[A])
  }
}

Like previous suggestions in this thread, this method requires you to write out a JSON format in full using the applicative DSL. It's unfortunately incompatible with the Json.format macro, but it gets you close to what you want. Here's a use case:

import play.api.libs.json._
import play.api.libs.functional.syntax._
import tristate._

case class Coord(col: Option[Option[String]], row: Option[Option[Int]])

implicit val format: OFormat[Coord] = (
  (__ \ "col").formatTriStateNullable[String] ~
  (__ \ "row").formatTriStateNullable[Int]
)(Coord.apply, unlift(Coord.unapply))

Some examples of writing:

format.writes(Coord(None, None))
// => {}

format.writes(Coord(Some(None), Some(None)))
// => { "col": null, "row": null }

format.writes(Coord(Some(Some("A")), Some(Some(1))))
// => { "col": "A", "row": 1 }

And some examples of reading:

Json.obj().as[Coord]
// => Coord(None, None)

Json.obj(
  "col" -> JsNull, 
  "row" -> JsNull
).as[Coord]
// => Coord(Some(None), Some(None))

Json.obj(
  "col" -> "A", 
  "row" -> 1
).as[Coord]
// => Coord(Some(Some("A")), Some(Some(1)))

As a bonus exercise for the reader, you could probably combine this with a little shapeless to automatically derive codecs and replace the Json.format macro with a different one-liner (albeit one that takes longer to compile).

Answer 2

Following @kflorence suggestion about OptionHandler I was able to get the desired behavior.

implicit def optionFormat[T](implicit tf: Format[T]): Format[Option[T]] = Format(
    tf.reads(_).map(r => Some(r)),
    Writes(v => v.map(tf.writes).getOrElse(JsNull))
  )

object InvertedDefaultHandler extends OptionHandlers {
  def readHandler[T](jsPath: JsPath)(implicit r: Reads[T]): Reads[Option[T]] = jsPath.readNullable

  override def readHandlerWithDefault[T](jsPath: JsPath, defaultValue: => Option[T])(implicit r: Reads[T]): Reads[Option[T]] = Reads[Option[T]] { json =>
    jsPath.asSingleJson(json) match {
      case JsDefined(JsNull) => JsSuccess(defaultValue)
      case JsDefined(value)  => r.reads(value).repath(jsPath).map(Some(_))
      case JsUndefined()     => JsSuccess(None)
    }
  }

  def writeHandler[T](jsPath: JsPath)(implicit writes: Writes[T]): OWrites[Option[T]] = jsPath.writeNullable
}

val configuration = JsonConfiguration[Json.WithDefaultValues](optionHandlers = InvertedDefaultHandler)

case class RequestObject(payload: Option[Option[String]] = Some(None))

implicit val requestObjectFormat: OFormat[RequestObject] = Json.configured(configuration).format[RequestObject]

Json.parse(""" {} """).as[RequestObject] // RequestObject(None)
Json.parse(""" {"payload": null } """).as[RequestObject] // RequestObject(Some(None))
Json.parse(""" {"payload": "hello" } """).as[RequestObject] // RequestObject(Some(Some(hello)))

So the important parts are:

• The readHandlerWithDefault basically flipping how JsDefined(JsNull) and JsUndefined are handling absent and explicit nulls compared to the original implementation in OptionHandlers.Default
• The JsonConfiguration taking both Json.WithDefaultValues and optionHandlers
• How the default value is being set. Note the RequestObject.payload's default value

Answer 3

Unfortunately I don't know how to achieve what you want automatically. For now it seems to me that you can't do that with the standard macro. However surprisingly you might achieve a similar result if you are OK with swapping the null and "absent" cases (which I agree is a bit confusing).

Assume class Xxx is defined as (default value is important - this will be the result for the null case)

case class Xxx(a: Option[Option[String]] = Some(None))

and you provide following implicit Reads:

implicit val optionStringReads:Reads[Option[String]] = new Reads[Option[String]] {
  override def reads(json: JsValue) = json match {
    case JsNull => JsSuccess(None) // this is never reached
    case JsString(value) => JsSuccess(Some(value))
    case _ => throw new RuntimeException("unexpected type")
  }
}

implicit val xxxReads = Json.using[Json.WithDefaultValues].reads[Xxx]

Then for a test data:

val jsonNone = "{}"
val jsonNull = """{"a":null}"""
val jsonVal = """{"a":"abc"}"""
val jsonValues = List(jsonNone, jsonNull, jsonVal)

jsonValues.foreach(jsonString => {
  val jsonAst = Json.parse(jsonString)
  val obj = Json.fromJson[Xxx](jsonAst)
  println(s"'$jsonString' => $obj")
})

the output is

'{}' => JsSuccess(Xxx(Some(None)),)
'{"a":null}' => JsSuccess(Xxx(None),)
'{"a":"abc"}' => JsSuccess(Xxx(Some(Some(abc))),)

So

• absent attribute is mapped onto Some(None)
• null is mapped onto None
• Value is mapped onto Some(Some(value))

This is clumsy and a bit unexpected by a developer, but at least this distinguishes all 3 choices. The reason why null and "absent" choices are swapped is that the only way I found to distinguish those cases is to have the value in the target class to be declared as Option and with default value at the same time and in that case the default value is what the "absent" case is mapped to; and unfortunately you can't control the value that null is mapped onto - it is always None.