jq: group and key by property

Question

I have a list of objects that look like this:

[
  {
    "ip": "1.1.1.1",
    "component": "name1"
  },
  {
    "ip": "1.1.1.2",
    "component": "name1"
  },
  {
    "ip": "1.1.1.3",
    "component": "name2"
  },
  {
    "ip": "1.1.1.4",
    "component": "name2"
  }
]

Now I'd like to group and key that by the component and assign a list of ips to each of the components:

{
  "name1": [
    "1.1.1.1",
    "1.1.1.2"
  ]
},{
  "name2": [
    "1.1.1.3",
    "1.1.1.4"
  ]
}

score 67 · Accepted Answer · answered Apr 05 '17 at 03:55

67

I figured it out myself. I first group by .component and then just create new lists of ips that are indexed by the component of the first object of each group:

jq ' group_by(.component)[] | {(.[0].component): [.[] | .ip]}'

answered Apr 05 '17 at 03:55

replay

3,569
3
21
30

1

And what if key is of type number? – branquito May 24 '18 at 15:28
3

@branquito It can’t. Object keys are always strings in JSON. If you have a JSON like `{"0": 1}` you can get the `"0"` key using `."0"`. – bfontaine Sep 28 '18 at 09:47
The output produced by this query isn't valid json. You can find further explanations in this answer I posted recently: https://stackoverflow.com/a/69346071/2508466 – yaccob Sep 27 '21 at 11:53
Pipe the output to `| jq -s .` to create valid json. So Final Answer: `jq ' group_by(.component)[] | {(.[0].component): [.[] | .ip]}' | jq -s .` – Aditya Pednekar Apr 18 '23 at 14:29

yaccob · Answer 2 · 2021-09-27T12:18:13.380

The accepted answer doesn't produce valid json, but:

{
  "name1": [
    "1.1.1.1",
    "1.1.1.2"
  ]
}
{
  "name2": [
    "1.1.1.3",
    "1.1.1.4"
  ]
}

name1 as well as name2 are valid json objects, but the output as a whole isn't.

The following jq statement results in the desired output as specified in the question:

group_by(.component) | map({ key: (.[0].component), value: [.[] | .ip] }) | from_entries

Output:

{
  "name1": [
    "1.1.1.1",
    "1.1.1.2"
  ],
  "name2": [
    "1.1.1.3",
    "1.1.1.4"
  ]
}

Suggestions for simpler approaches are welcome.

If human readability is preferred over valid json, I'd suggest something like ...

jq -r 'group_by(.component)[] | "IPs for " + .[0].component + ": " + (map(.ip) | tostring)'

... which results in ...

IPs for name1: ["1.1.1.1","1.1.1.2"]
IPs for name2: ["1.1.1.3","1.1.1.4"]

score 1 · Answer 3 · answered Aug 11 '20 at 21:16

As a further example of @replay's technique, after many failures using other methods, I finally built a filter that condenses this Wazuh report (excerpted for brevity):

{
  "took" : 228,
  "timed_out" : false,
  "hits" : {
    "total" : {
      "value" : 2806,
      "relation" : "eq"
    },
    "hits" : [
      {
        "_source" : {
          "agent" : {
            "name" : "100360xx"
          },
          "data" : {
            "vulnerability" : {
              "severity" : "High",
              "package" : {
                "condition" : "less than 78.0",
                "name" : "Mozilla Firefox 68.11.0 ESR (x64 en-US)"
              }
            }
          }
        }
      },
      {
        "_source" : {
          "agent" : {
            "name" : "100360xx"
          },
          "data" : {
            "vulnerability" : {
              "severity" : "High",
              "package" : {
                "condition" : "less than 78.0",
                "name" : "Mozilla Firefox 68.11.0 ESR (x64 en-US)"
              }
            }
          }
        }
      },
      ...

Here is the jq filter I use to provide an array of objects, each consisting of an agent name followed by an array of names of the agent's vulnerable packages:

jq ' .hits.hits |= unique_by(._source.agent.name, ._source.data.vulnerability.package.name) | .hits.hits | group_by(._source.agent.name)[] | { (.[0]._source.agent.name): [.[]._source.data.vulnerability.package | .name ]}'

Here is an excerpt of the output produced by the filter:

{
  "100360xx": [
    "Mozilla Firefox 68.11.0 ESR (x64 en-US)",
    "VLC media player",
    "Windows 10"
  ]
}
{
  "WIN-KD5C4xxx": [
    "Windows Server 2019"
  ]
}
{
  "fridxxx": [
    "java-1.8.0-openjdk",
    "kernel",
    "kernel-headers",
    "kernel-tools",
    "kernel-tools-libs",
    "python-perf"
  ]
}
{
  "mcd-xxx-xxx": [
    "dbus",
    "fribidi",
    "gnupg2",
    "graphite2",
    ...

jq: group and key by property

3 Answers3

Linked