Pipe multiple jq values to "while read -r" loop

Question

I am trying to find a clean way to assign 2 json string values to 2 variables within a while loop. The loop and input I am working with are as follows.

INPUT:

foo='
[
   {
      "name":"name string (more info)",
      "nested_name":{
         "name":"my name",
         "conclusion":"failure",
         "number":11
      }
   },
   {
      "name":"name string (more info)",
      "nested_name":{
         "name":"my other name",
         "conclusion":"failure",
         "number":13
      }
   }
]'

Currently I have the following:

echo "$foo" | jq ".[].name,.[].nested_name.name" | while read -r foo bar; do
    # do stuff with foo and bar
done

In iteration 1, I want:

foo = "name string (more info)" 
bar = "my name"

Iteration 2:

foo = "name string (more info)" 
bar = "my other name"

However this generates the following incorrect output:

foo = "name 
bar = string (more info)"
foo = "name 
bar = string (more info)"
foo = "my 
bar = name"
foo = "my 
bar = other name"

I've been at this all day, any input or suggestions would be very much appreciated!

Answer 1

Use two reads, one for each line:

#!/usr/bin/env bash

json='
[
   {
      "name":"name string (more info)",
      "nested_name":{
         "name":"my name",
         "conclusion":"failure",
         "number":11
      }
   },
   {
      "name":"name string (more info)",
      "nested_name":{
         "name":"my other name",
         "conclusion":"failure",
         "number":13
      }
   }
]'


while read -r foo && read -r bar; do
    printf "foo=%s\nbar=%s\n" "$foo" "$bar"
done < <(jq ".[] | .name, .nested_name.name" <<<"$json")

---

I had to change your jq expression to get your desired output.

Also note using different names for your original JSON text and the loop variable, using a here string instead of echo, and using file redirection instead of a pipeline; useful if the body of the while sets any variables needed later.

---

An alternative that reads all the jq output into an array and then iterates it:

readarray -t lines < <(jq ".[] | .name, .nested_name.name" <<<"$json")
for (( i = 0; i < ${#lines[@]}; i += 2 )); do
    foo=${lines[i]}
    bar=${lines[i+1]}
    printf "foo=%s\nbar=%s\n" "$foo" "$bar"
done

Answer 2

If I simulate some "CSV input" I can do this: (Note the IFS set to ,)

echo 'aaa,bbb\nccc,ddd' | while IFS=, read -r x y; do
  echo "x=$x, y=$y"
done

Output:

x=aaa, y=bbb
x=ccc, y=ddd

You can generate a similar "CSV output" with the following jq command:

jq --raw-output '.[] | "\(.name),\(.nested_name.name)"'

So I think it'd look something like this:

echo "$foo" | jq --raw-output '.[] | "\(.name),\(.nested_name.name)"' | while IFS=, read -r foo bar; do
    # do stuff with foo and bar
done

Answer 3

how about this: https://jqplay.org/s/4MPWmMjFmM

assuming foobar.json
    jq -r '.[] | "foo=\"" + .name + "\"","bar=\"" + .nested_name.name + "\""' foobar.json
    foo="name string (more info)"
    bar="my name"
    foo="name string (more info)"
    bar="my other name"

Answer 4

I prefer to do it like this:

jq --argjson foo "$foo" --null-input --compact-output '$foo[]' | while IFS= read -r item
do
    name="$(jq --argjson item "$item" --null-input --raw-output '$item.name')"
    nested_name="$(jq --argjson item "$item" --null-input --raw-output '$item.nested_name.name')"
    printf 'name is "%s" and nested_name is "%s"\n' "$name" "$nested_name"
done

While this method involves extra calls to jq, it has several benefits:

• It's easy to extend if you need to iterate over something with more complex structure.
• It's safe against spaces, commas, quotes, tabs and newlines.
• It's easy to understand and have confidence in. It doesn't involve any special algorithm, parsing or escaping.

Here's the same thing using abbreviated options if you want it a little bit more terse.

jq --argjson foo "$foo" -nc '$foo[]' | while IFS= read -r item
do
    name="$(jq --argjson item "$item" -nr '$item.name')"
    nested_name="$(jq --argjson item "$item" -nr '$item.nested_name.name')"
    printf 'name is "%s" and nested_name is "%s"\n' "$name" "$nested_name"
done

In a little more detail:

• We use --argjson to pass our JSON string in as a single argument and --null-input to ignore stdin. This way we don't have to worry about escaping or delimiters on the way in.
• We use --compact-output (or -c) to output one JSON object per line.
• We use IFS= to make sure that spaces and tabs inside our JSON string don't get picked up as delimiters. Only newlines will act as delimiters, and because all newlines inside JSON strings must be escaped, we can be confident that we will loop exactly once with for each item in the original JSON array.
• We use --raw-output (or -r) to output the value of the JSON string directly into the shell variable. (We can choose to omit this if we instead want the JSON-encoded string.)