Flattening nested JSON objects with Circe

Question

Suppose I have a JSON object like this:

{
   "foo": true,
   "bar": {
      "baz": 1,
      "qux": {
        "msg": "hello world",
        "wow": [null]
      }
   }
}

And I want to flatten it recursively to a single layer, with the keys merged with an underscore:

{
   "foo": true,
   "bar_baz": 1,
   "baz_qux_msg": "hello world",
   "baz_qux_wow": [null]
}

How can I do this with Circe?

(Note: this is another FAQ from the Circe Gitter channel.)

score 6 · Accepted Answer · answered Sep 20 '19 at 10:04

You can do this without too much pain in Circe with a recursive method:

import io.circe.Json

def flatten(combineKeys: (String, String) => String)(value: Json): Json = {
  def flattenToFields(value: Json): Option[Iterable[(String, Json)]] =
    value.asObject.map(
      _.toIterable.flatMap {
        case (k, v) => flattenToFields(v) match {
          case None => List(k -> v)
          case Some(fields) => fields.map {
            case (innerK, innerV) => combineKeys(k, innerK) -> innerV
          }
        }
      }
    )

  flattenToFields(value).fold(value)(Json.fromFields)
}

Here our internal flattenToFields method takes each JSON value and either returns None if it's a non-JSON object value, as a signal that that field doesn't need flattening, or a Some containing a sequence of flattened fields in the case of a JSON object.

If we have a JSON value like this:

val Right(doc) = io.circe.jawn.parse("""{
   "foo": true,
   "bar": {
      "baz": 1,
      "qux": {
        "msg": "hello world",
        "wow": [null]
      }
   }
}""")

We can verify that flatten does what we want like this:

scala> flatten(_ + "_" + _)(doc)
res1: io.circe.Json =
{
  "foo" : true,
  "bar_baz" : 1,
  "bar_qux_msg" : "hello world",
  "bar_qux_wow" : [
    null
  ]
}

Note that flattenToFields is not tail recursive, and will overflow the stack for deeply-nested JSON objects, but probably not until you're several thousand levels deep, so it's unlikely to be an issue in practice. You could make it tail recursive without too much trouble, but at the expense of additional overhead for the common cases where you only have a few layers of nesting.

score 1 · Answer 2 · answered Aug 04 '21 at 11:04

I propose a variation of the solution by Travis Brown. The variation concerns objects in the JSON lists, i.e. how to handle

{
  "foo": true,
  "bar": {
    "baz": 1,
    "qux": {
      "msg": "hello world",
      "wow": [{"x": 1, "y": 2}, {"x": 3, "y": 4}]
    }
  }
}

One possible solution for recursively handling objects in lists is the following implementation, where the position in the list is taken as an additional key part

def flattenDeep(combineKeys: (String, String) => String)(value: Json): Json = {
  def flattenToFields(value: Json): Option[Iterable[(String, Json)]] = {
    value.fold(
      jsonNull = None,
      jsonNumber = _ => None,
      jsonString = _ => None,
      jsonBoolean = _ => None,
      jsonObject = { obj =>
        val fields = obj.toIterable.flatMap {
          case (field, json) =>
            flattenToFields(json).fold(Iterable(field -> json)) {
              _.map {
                case (innerField, innerJson) =>
                  combineKeys(field, innerField) -> innerJson
              }
            }
        }
        Some(fields)
      },
      jsonArray = { array =>
        val fields = array.zipWithIndex.flatMap {
          case (json, index) =>
            flattenToFields(json).fold(Iterable(index.toString -> json)) {
              _.map {
                case (innerField, innerJson) =>
                  combineKeys(index.toString, innerField) -> innerJson
              }
            }
        }
        Some(fields)
      }
    )
  }
  flattenToFields(value).fold(value)(Json.fromFields)
}

With this implementation the above example is flattened to:

{
  "foo" : true,
  "bar_baz" : 1,
  "bar_qux_msg" : "hello world",
  "bar_qux_wow_0_x" : 1,
  "bar_qux_wow_0_y" : 2,
  "bar_qux_wow_1_x" : 3,
  "bar_qux_wow_1_y" : 4
}

For even deeper nested structures one still gets a flat representation, e.g.

{
  "foo": true,
  "bar": {
    "baz": 1,
    "qux": {
      "msg": "hello world",
      "wow": [
        {
          "x": 1,
          "y": 2
        },
        {
          "x": 3,
          "y": 4
        }
      ],
      "deeper": [
        {
          "alpha": {
            "h": 12,
            "m": 1
          },
          "beta": [ "a", "b", "c" ]
        },
        {
          "alpha": {
            "h": 21,
            "m": 0
          },
          "beta": [ "z" ]
        }
      ]
    }
  }
}

will be flattened into

{
  "foo" : true,
  "bar_baz" : 1,
  "bar_qux_msg" : "hello world",
  "bar_qux_wow_0_x" : 1,
  "bar_qux_wow_0_y" : 2,
  "bar_qux_wow_1_x" : 3,
  "bar_qux_wow_1_y" : 4,
  "bar_qux_deeper_0_alpha_h" : 12,
  "bar_qux_deeper_0_alpha_m" : 1,
  "bar_qux_deeper_0_beta_0" : "a",
  "bar_qux_deeper_0_beta_1" : "b",
  "bar_qux_deeper_0_beta_2" : "c",
  "bar_qux_deeper_1_alpha_h" : 21,
  "bar_qux_deeper_1_alpha_m" : 0,
  "bar_qux_deeper_1_beta_0" : "z"
}

Flattening nested JSON objects with Circe

2 Answers2