Conditional parsing of Nested Jsons with jq

Question

Need help parsing the below nested Json. The end goal is to get the quota limits from GCP that have non-default values. Below is the sample data for one of the services (compute APIs for a given CPU type)

I am struggling to be able to compare I came up with the below but it doesn't work and returns an error .[]|select (.quota[].defaultLimit|tonumber != .quota[].effectiveLimit|tonumber)

[
  {
    "metric": "compute.googleapis.com/a2_cpus",
    "quota": [
      {
        "defaultLimit": "-1",
        "effectiveLimit": "-1"
      }
    ]
  },
  {
    "metric": "compute.googleapis.com/a2_cpus",
    "quota": [
      {
        "defaultLimit": "12",
        "effectiveLimit": "12"
      },
      {
        "defaultLimit": "12",
        "dimensions": {
          "region": "asia-east1"
        },
        "effectiveLimit": "13",
        "producerOverride": {
          "dimensions": {
            "region": "asia-east1"
          }
        }
      }
    ]
  }
]

the expected output would be something like this

[
  {
    "metric": "compute.googleapis.com/a2_cpus",
    "quota": [    
      {
        "defaultLimit": "12",
        "dimensions": {
          "region": "asia-east1"
        },
        "effectiveLimit": "13"
      }
    ]
  }
]

Any help is appreciated.

Your query triggers an error because some parentheses are missing. — peak, Mar 15 '23 at 11:07

score 1 · Answer 1 · answered Mar 15 '23 at 07:38

You could first select all quotas with a non-default limit and then drop any metric without remaining quotas/keep only metrics with at least one quota:

map(.quota |= map(select(.defaultLimit != .effectiveLimit)) | select(.quota | length > 0))

Output:

[
  {
    "metric": "compute.googleapis.com/a2_cpus",
    "quota": [
      {
        "defaultLimit": "12",
        "dimensions": {
          "region": "asia-east1"
        },
        "effectiveLimit": "13",
        "producerOverride": {
          "dimensions": {
            "region": "asia-east1"
          }
        }
      }
    ]
  }
]

To keep only the limits and dimensions (dropping producerOverride and any other additional properties):

map(.quota |= map(select(.defaultLimit != .effectiveLimit) | {defaultLimit, effectiveLimit, dimensions}) | select(.quota | length > 0))

Output:

[
  {
    "metric": "compute.googleapis.com/a2_cpus",
    "quota": [
      {
        "defaultLimit": "12",
        "effectiveLimit": "13",
        "dimensions": {
          "region": "asia-east1"
        }
      }
    ]
  }
]

score 0 · Answer 2 · answered Mar 15 '23 at 05:13

Based on your basic description of the problem, it looks like this would suffice:

walk(if type == "object" and has("defaultLimit") 
     then select(.defaultLimit != .effectiveLimit) else . end)

However, based on your example, you might want to extend the pipeline by adding:

| walk(if type == "object" then select(.quota != []) else . end)

score 0 · Answer 3 · answered Mar 15 '23 at 08:15

A possible solution:

map(
  .quota |=
    map(
      select(.defaultLimit != .effectiveLimit)
      | del(.producerOverride)
    )
  | select(.quota | length > 0)
)

How it works

map(...) - because the input is an array, map(...) applies the expression passed as argument to each item of the array and yields a new array that contains the results of the expression applied to each argument; all subsequent steps described below operate on one object (each item of the array).
.quota |= map(...) - the value of property .quota (which is an array) is passed to map() (which yields a new array) and the resulting array is stored back in .quota; this is the same thing as .quota = (.quota | map(...)) but it's shorter and easier to read.
map(select(...) | del(...)) - for each item of the .quota array run select(.defaultLimit != .effectiveLimit) then remove (del) the property .producerOverride from the returned object (if it exists). The result is an array that contains only the quota entries that non-default values.
select(.defaultLimit != .effectiveLimit) - if the condition is met then this yields the input object, otherwise it does not yield anything. This is how the quota items that have non-default values are filtered.
del(.producerOverride) - this filter is here because the property .producerOverride is not displayed in your expected output. If the input object may contain other properties, not listed here, and you want to keep only a known set of properties in the output then use this filter instead:
```
| { defaultLimit, dimensions, effectiveLimit }
```
But be aware that this one produces objects that always have these three properties, even when not all of them are in the input (dimensions is not present in all quota items in the input). In this case, the value of those properties is null.
| select(.quota | length > 0) - this filter gets at input the objects of the input array after their .quota property is processed as described above. It keeps from its input only the objects that have non-empty .quota property. The previous filters produce from the first input item an item that looks like this:
```
{
   "metric": "compute.googleapis.com/a2_cpus",
   "quota": []
}
```
This filter removes such items.

`jq` documentation

Read the jq tutorial and the jq manual for more information.

Thanks a ton axiac for the wonderful explanation. Why do we have two maps, that is the only part not clear right now. If I run just the inner map for my understanding, I get an error ```.quota |= map( select(.defaultLimit != .effectiveLimit) )``` — Python Beginner, Mar 15 '23 at 16:06
The first, outer map, applies a filter to each object from the input array. It makes sure that all objects from the input array are processed and the final outcome is also an array. The second, inner map, operates on each item of the array stored in property `.quota` (of each object processed by the outer map). The processing on the items of `.quota` keeps only the entries having non-default limit. — axiac, Mar 15 '23 at 16:37

Conditional parsing of Nested Jsons with jq

3 Answers3

How it works

jq documentation

`jq` documentation