jq script with back tick in JSON value

Question

I'm using jq inside a bash script, to parse a JSON returned by an API REST service

This is my code (getting .count value)

# getAPIId
# Parse the answer of query API to get the API ID (checking version name)
# return (echo to stdout) the APIID
function getAPIId() {
    local apiId
    local version
    local -i count          

    if [ -z $1 ]
    then
        echo "Use: getAPIId \$apiQuery" >&2
        exit 1
    else    
        # Parse answer to API info
        count=$(echo "$1"|jq -r '.count')
        if [ "$count" == "0" ]
        then
            echo $1 >&2
            echo "ERROR: Cannot find an API called ${APINAME}" >&2
            return 1
        else
            for ((i=0; i<count; i++))
            do
                version=$(echo $1|jq -r '.list['$i'].version')
                if [ "$APIVERSION" == "$version" ] && [ "$APINAME" == "$name" ]
                then
                    apiId=$(echo $1|jq -r '.list['$i'].id')
                    echo "$apiId"
                    return 0
                fi
            done
        fi
    fi
    return 1

}

This is the JSON file I try to parse

{"count":1,"next":"","previous":"","list":[{"id":"6d2822e5-a90d-4738-b7b7-ef7d7848cb48","name":"SwaggerPetstore",
"description":"`special key`","context":"/v2","version":"1.0.0","provider":"admin","status":"CREATED","thumbnailUri":null}],"pagination":{"total":1,"offset":0,"limit":25}}

As you see, the field description contains back tick value. In this case, jq fails and return empty string (as not found value for count)

How can avoid the problem with backtick?

As an aside, you should be doing more of your logic in `jq` and less in bash; looping in bash calling `jq` over and over is pretty much always a code smell to the effect that you're not using it effectively. — Charles Duffy, Jun 27 '18 at 21:34
...reading your code, I don't see why you're trying to use `count` *at all*. Why not just have `jq` return **all** the IDs for which the name and version are correct? — Charles Duffy, Jun 27 '18 at 21:36
btw, regarding my comment about all-caps variable names -- see http://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1_chap08.html, fourth paragraph; all-caps names are used for variables that change behavior of the shell and operating system, whereas lowercase names are reserved for application use and guaranteed not to have side effects on POSIX-defined tools. — Charles Duffy, Jun 27 '18 at 21:42
Also, `==` is bad form in `[ ]`; the only POSIX-defined test operator is `=`; so it isn't portable. If you want to use extended syntax, use `[[ ]]` to make that explicit -- you get a larger set of benefits anyhow. — Charles Duffy, Jun 27 '18 at 21:43
...and with respect to the `function` keyword, see http://wiki.bash-hackers.org/scripting/obsolete — Charles Duffy, Jun 27 '18 at 21:44
I'll take in consideration your suggestion, but that does not solve the issue. If there are back ticks, the JSONpath fails. — Sourcerer, Jun 27 '18 at 21:45
(...and btw, with respect to declaring locals -- you missed `i`, the counter used in the `for` loop; that's a tricky one, because if you call this function from another one that *also* uses `i` as a counter variable, they can interact nastily). — Charles Duffy, Jun 27 '18 at 21:46
See the edited answer, showing full (copy/paste-ready) test procedure. As you can see, the backticks are genuinely present in the data being tested against. — Charles Duffy, Jun 27 '18 at 21:49
(Another thing is that `echo $1` -- as being done at several places in this code -- is itself buggy; see [BashPitfalls #14](http://mywiki.wooledge.org/BashPitfalls#echo_.24foo)). — Charles Duffy, Jun 27 '18 at 21:52
I wonder if the problem is not a string ```"`special key`"``` but something **inside** that "special key" that isn't actually valid JSON. Obviously, if all you provide here is a sanitized/obfuscated string that *is* valid JSON (and backticks are perfectly valid!), nobody but you can reproduce the problem. (This is why our [mcve] guidelines instruct you to test that a question's code sample is **verifiable** -- that you've actually tested before posting that someone else can actually reproduce a specifically-identified problem with the exact, unmodified content given in the question). — Charles Duffy, Jun 27 '18 at 22:36
HI, thank you very much for your code! I confirm that the problem was in `echo $1|jq .` when the $1 variable contains back ticks. I don't fully understand the expansion sequence, but it was not solved either with echo "$1"|jq . But, of course, your approach is fully elegant. Thanks again. — Sourcerer, Jun 28 '18 at 10:42
Even without the quotes, `echo $1 | jq .` isn't inherently broken by backticks. ```set -- '"`special key`"'; echo $1 | jq .``` doesn't trigger any of `echo`'s (or unquoted expansion's) faults, assuming a default `IFS`. Now, if there are wildcards inside the backticks and you have `nullglob` or `failglob` set, that would explain it, but again, that's something other than / additional to the backticks themselves. — Charles Duffy, Jun 28 '18 at 10:52

Charles Duffy · Accepted Answer · 2018-06-27T21:57:25.970

jq does not have any problem at all with backticks in literal values. You can see that as follows:

$ jq '.value' <<<'{"otherKey": "`otherValue`", "value": "desired value `with backticks`"}'
"desired value `with backticks`"

With respect to the code given in the question, though, it's vastly overcomplicated. Rather than muddle through what could be wrong, consider the following replacement (which doesn't need to read count at all, and calls jq only once):

# Aside: Using all-caps names is bad practice. Don't do this.
APINAME=SwaggerPetstore
APIVERSION=1.0.0

getAPIId() {
  [[ $1 ]] || { echo 'Usage: getAPIId "$json"' >&2; return 1; }
  jq -er \
       --arg target_name "$APINAME" \
       --arg target_version "$APIVERSION" '
    .list[] |
    select(.name == $target_name) |
    select(.version == $target_version) |
    .id' <<<"$1"
}

...returns, when called as follows:

s='{
  "count": 1,
  "next": "",
  "previous": "",
  "list": [
    {
      "id": "6d2822e5-a90d-4738-b7b7-ef7d7848cb48",
      "name": "SwaggerPetstore",
      "description": "`special key`",
      "context": "/v2",
      "version": "1.0.0",
      "provider": "admin",
      "status": "CREATED",
      "thumbnailUri": null
    }
  ],
  "pagination": {
    "total": 1,
    "offset": 0,
    "limit": 25
  }
}'

getAPIId "$s"; echo "Exit status is $?" >&2

...the correct result of:

6d2822e5-a90d-4738-b7b7-ef7d7848cb48
Exit status is 0

...whereas if we run it again with an invalid name or version to search for:

APINAME=NotFound getAPIId "$s"; echo "Exit status is $?" >&2

...correctly reflects that:

Exit status is 4

jq script with back tick in JSON value

1 Answers1