Use jq to collect recursive-descent results into a single array

Question

Is it possible to collect recursive-descent results into a single array with jq?

Would flatten help? Looks so to me, but I just cannot get it working. Take a look how far I am now at https://jqplay.org/s/6bxD-Wq0QE, anyone can make it working?

BTW,

.data.search.edges[].node | {name, topics: ..|.topics?} works, but I want all topics from the same node to be in one array, instead of having same name in all different returned results.
flatten alone will give me Cannot iterate over null, and
that's why I'm trying to use map(select(.? != null)) to filter the nulls out. However, I'd get Cannot iterate over null as well for my map-select.

So now it all comes down to how to filter out those nulls?

UPDATE:, by "collect into a single array" I meant to get something like this:

[
  {
    "name": "leumi-leumicard-bank-data-scraper",
    "topics": ["banking", "leumi", "api", "puppeteer", "scraper", "open-api"]
  }
]

instead of having same name duplicated in all different returned results. Thus recursively descends seems to me to be the option, but I'm open to any solution as long as I can get result like above. Is that possible? Thx.

post the final expected result – RomanPerekhrest Mar 21 '18 at 16:09 — RomanPerekhrest, Mar 21 '18 at 16:09
Please follow the guidelines at stackoverflow.com/help/mcve – peak Mar 21 '18 at 19:08 — peak, Mar 21 '18 at 19:08

Jeff Mercado · Answer 1 · 2018-03-21T17:18:58.203

Not sure what you're expecting to get in your results... but it seems like you're trying to get all the repositories and their topics in a flat array. I don't see any reason why you should use recurse here, you're only selecting from one class of objects. Just reference them directly.

[.data.search.edges[].node | {name,topic:(.repositoryTopics.nodes[].topic.topics)}]

For your particular input produces:

[
  {
    "name": "leumi-leumicard-bank-data-scraper",
    "topic": "banking"
  },
  {
    "name": "leumi-leumicard-bank-data-scraper",
    "topic": "leumi"
  },
  {
    "name": "leumi-leumicard-bank-data-scraper",
    "topic": "api"
  },
  {
    "name": "leumi-leumicard-bank-data-scraper",
    "topic": "puppeteer"
  },
  {
    "name": "leumi-leumicard-bank-data-scraper",
    "topic": "scraper"
  },
  {
    "name": "leumi-leumicard-bank-data-scraper",
    "topic": "open-api"
  }
]

https://jqplay.org/s/G2inYAJNLS

If you wanted to have an array of topics within the nodes instead, just collect them in an array by putting the filter that selects the topics within [].

[.data.search.edges[].node | {name,topic:[.repositoryTopics.nodes[].topic.topics]}]

[
  {
    "name": "leumi-leumicard-bank-data-scraper",
    "topic": [
      "banking",
      "leumi",
      "api",
      "puppeteer",
      "scraper",
      "open-api"
    ]
  },
  {
    "name": "echarts-scrappeteer",
    "topic": []
  }
]

https://jqplay.org/s/0AFneNK89i

Thanks a lot for your understanding and help Jeff! I edited my OP to make it clear what I want, but noticed afterward that @peak's answer is already giving the correct solution. Thanks again for your help Jeff! — xpt, Mar 21 '18 at 16:41

peak · Accepted Answer · 2018-03-21T16:42:28.780

2

One way to collect the non-falsey values:

.data.search.edges[].node 
| {name, topics: [.. | .topics? | select(.)]}

The result would be:

{
  "name": "leumi-leumicard-bank-data-scraper",
  "topics": [
    "banking",
    "leumi",
    "api",
    "puppeteer",
    "scraper",
    "open-api"
  ]
}
{
  "name": "echarts-scrappeteer",
  "topics": []
}

edited Mar 21 '18 at 16:42

answered Mar 21 '18 at 16:33

peak

105,803
17
152
177

Use jq to collect recursive-descent results into a single array

2 Answers2