Using jq with bash to run command for each object in array

Question

How can I run a Bash command for every JSON object in a JSON array using jq? So far I have this:

cat credentials.json | jq -r '.[] | .user, .date, .email' | mycommand -u {user} -d {date} -e {email}

This doesn't seem to work. How can I take the parameters out of the JSON array into my command?

My JSON file looks something like this:

[
   "user": "danielrvt",
   "date": "11/10/1988",
   "email": "myemail@domain.com",
   ...
]

score 71 · Accepted Answer · answered Apr 03 '17 at 19:25

71

Your best bet is probably to output each record in something like TSV format, then read that from a shell loop.

jq -r '.[]|[.user, .date, .email] | @tsv' |
  while IFS=$'\t' read -r user date email; do
    mycommand -u "$user" -d "$date" -e "$email"
  done

jq itself doesn't have anything like a system call to run an external command from within a filter, although it seems that they are working on it.

answered Apr 03 '17 at 19:25

chepner

497,756
71
530
681

`IFS=$'\t'` is not working when I use this in Makefile. Can you suggest any other way? – shaik moeed Sep 22 '22 at 14:38
`make` is probably using `/bin/sh`, not `bash`, and it doesn't support `$'...'`-style quoting. – chepner Sep 22 '22 at 15:25

score 44 · Answer 2 · answered Jun 13 '18 at 20:09

44

With xargs:

curl localhost:8082/connectors | jq .[] | xargs -L1 -I'{}' curl -XDELETE 'localhost:8082/connectors/{}'

Or equivalently, to show the output of that first curl:

echo '["quickstart-file-sink4","quickstart-file-source","quickstart-file-sink","quickstart-file-sink2","quickstart-file-sink3","quickstart-file-source2"]' | jq .[] | xargs -L1 -I'{}' curl -XDELETE 'localhost:8082/connectors/{}'

jq .[] strips off one level of containment, so that a list becomes output as one line per item.

xargs -L1 processes one line at a time

xargs -I'{}' specifies that the string {} be replaced with the input line when invoking the following command.

xargs is essentially a map operator for the shell.

answered Jun 13 '18 at 20:09

Marcin

48,559
18
128
201

1

This is an excellent option for iterating over an array of values and executing a command for each value. It does not require a multi-line script nor does it require a step to generate commands and a step to execute them. `xargs` is a great command. – Adam Link Nov 06 '19 at 22:20
I copied it letter for letter and it does not work: curl: (3) empty string within braces in URL position 28: localhost:8082/connectors/{} When doing a simple 'xargs echo' output of jq will print, but never with replacement token :( – anydoby May 05 '21 at 09:01
I had issues using the piped `xargs` command as shown in the answer, but this answer definitely got me headed down the right path! For those interested, I ended up replacing the arguments with the more verbose config flag versions, with the equivalent for this question being `curl localhost:8082/connectors | jq .[] | xargs --max-lines=1 --replace={} curl -XDELETE 'localhost:8082/connectors/{}'` (I also used jq's `-r` flag before chaining). – Brad Lucas Sep 21 '21 at 17:51

score 31 · Answer 3 · answered Apr 03 '17 at 19:46

31

You could have jq output the commands to execute, something like

.[] | "mycommand \(.user|@sh) \(.date|@sh) \(.email|@sh)"

Then execute it. Something like

bash <(jq -r '.[] | "mycommand \(.user|@sh) \(.date|@sh) \(.email|@sh)"' foo)

answered Apr 03 '17 at 19:46

kojiro

74,557
19
143
201

\(.propname) was all I needed. Is the |@sh piping each individual property? How is that different from \(.user) – DeadlyChambers Jun 08 '21 at 19:43
1

The @sh guarantees each parameter is properly quoted so the shell doesn't wordsplit or expand globs if that's what gets output. Suppose someone manages to drop a file like `{"email": "daniel;rm -rf *", ...}` – kojiro Jun 08 '21 at 23:24

score 20 · Answer 4 · answered Sep 19 '19 at 10:36

Here is another variation which I based on the answer from @chepner.

echo "$config" | jq -c '.[]' |
while IFS=$"\n" read -r c; do
    echo "start"
    host=$(echo "$c" | jq -r '.host')
    echo $host
    echo "end"
done

I used jq's -c option to output "compact" jsons, so they are all on one line.

In combination with IFS=$"\n", I was able to loop over each item in the input json's array and do what I wanted to do.

So, with an input of

[
 {
  "host": "host1",
  "settings": {}
 },
 {
  "host": "host1",
  "settings": {}
 }
]

the output is

start
host1
end
start
host2
end

score 4 · Answer 5 · answered Nov 13 '18 at 04:04

4

I came across the same problem recently where xargs doesn't help that much due to the relatively complicated set of arguments I wanted to pass around. Thus I implemented an sh filter (and its friends) to jq. I haven't yet had enough time to write documentation and tests for it so not creating a PR for it to become a part of the official codebase yet. So now it's only for the ones who are willing to compile this version themselves:

https://github.com/haochenx/jq/tree/sh-support

answered Nov 13 '18 at 04:04

Haochen Xie

360
2
9

I looked at your code. I like the idea. Some of the code you added to `src/builtin.c` is repetitive and needs to be simplified, but I think you should open a pull request against stedolan/jq. – Mr. Lance E Sloan Jan 22 '19 at 16:07

Using jq with bash to run command for each object in array

5 Answers5

Linked