bash function input $1 is only processing first word of string & formatting issue

Question

My BASH function:

json_format () {
        echo '{
                "question": "';echo "$1";echo '",' 
}

for:

json_format ${questions[$Q_counter]}

Is returning:

{
                "question": "
VM
",

Instead of expected json format and string:

{
    "question": "VM CPU is at 100% and you'r jumpbox creds be broken! What do you do?",

The string appears to be cut off at the space after the first word "VM", and the formatting is a little off with those echo commands. How can i correct my function? Thanks!

First read [the `echo` manual page](http://man7.org/linux/man-pages/man1/echo.1.html). Then read [the Bash manual page](http://man7.org/linux/man-pages/man1/bash.1.html) and think about the argumentS you pass to the function. — Some programmer dude, Aug 06 '16 at 17:04
BTW, what's your real goal here? If it's to generate valid JSON containing arbitrary strings, there's a **much** better approach. — Charles Duffy, Aug 06 '16 at 17:59
...one of the things that's a little unfortunate about this question is that it folds two separate questions into one. We already have duplicate knowledgebase entries for "how do I generate JSON from bash reliably?", and we already have duplicate entries for "why is only the first word of my function/command argument in $1?", but we don't have a single question that answers **both**; if this had been further debugged before asking the question, to isolate only one or the other of the two issues, it'd be easier to handle. — Charles Duffy, Aug 06 '16 at 18:16

Charles Duffy · Accepted Answer · 2016-08-06T19:57:24.393

1

The ideal way to generate JSON from shell is to use a tool such as jq that actually understands the format:

json_format() {
  jq -n --arg q "$1" '{"question": $q}'
}

...or, if you have a Python interpreter, the built-in json module can also be used:

json_format() {
  python -c 'import sys, json; print json.dumps({"question": sys.argv[1]})' "$1"
}

If you don't have any of those tools, however, at least make a best-effort attempt at escaping:

json_format() {
  local s=$1
  s=${s//'\'/'\\'}   # replace \ with \\
  s=${s//'"'/'\"'}   # replace " with \"
  s=${s//$'\n'/'\n'} # replace newline literals with \n
  printf '{\n\t"question": "%s"\n}\n' "$s"
}

...or, to process one value at a time:

json_format() {
  local s=$1
  s=${s//'\'/'\\'}   # replace \ with \\
  s=${s//'"'/'\"'}   # replace " with \"
  s=${s//$'\n'/'\n'} # replace newline literals with \n
  printf '%s\n' "$s"
}

...invoked individually for each string to be formatted, as in:

cat <<EOF
{
  "question": "$(json_format "$question")",
  "image": "$(json_format "$image_url")",
  "choices": [ ],
  "correct": [ "$(json_format "$answer")" ],
  "explanation": "$(json_format "$explanation")"
}
EOF

This will work correctly with cases for which naive approaches will produce valid that is not in fact valid JSON. Consider, for instance:

# naive string substitution will fail here because it won't escape double quotes
json_format 'How do you spell "hello" in Spanish?'

or

# naive string substitution will fail here because it won't escape the newline
json_format $'This question has\ntwo lines'

or

# naive string substitution will fail here because it won't escape the backslash
json_format 'This question ends in a backslash: \'

Note, in all of the above, the quoting -- which ensures that the string is passed as a single argument.

edited Aug 06 '16 at 19:57

answered Aug 06 '16 at 18:02

Charles Duffy

280,126
43
390
441

jq is a great suggestion however the requirements are too high for my situation. [ flex, bison, gcc, make, autotools, ruby, oniguruma and git ] I have none of these in my deployment. – Matt Aug 06 '16 at 18:54
Those are tools you need to *compile* `jq`, not to *use* a copy of `jq` already compiled (well, depending on whether doing static or dynamic linking to oniguruma). That said, do you have a Python interpreter? A Python one-liner can also be used for the purpose. – Charles Duffy Aug 06 '16 at 18:56
(BTW, if -- as I assume, given the minimal tool availability -- you're on an embedded system, what's your build tooling? `buildroot` is a great tool for being able to generate an environment with pretty much any arbitrary build chain cross-compiled). – Charles Duffy Aug 06 '16 at 19:03
@user6672948, ...also added a pure-bash approach. It's not going to be as good as a real JSON generation library will -- I'm sure there are non-printable characters, multibyte characters, or other cases for which it fails to comply with the standard -- but it's better than what you have now. – Charles Duffy Aug 06 '16 at 19:09
Your pure bash example function works really well except I am not ready to append the last " } " since i have more tags to add beneath "question" such as: `{ "question": "what is the greatest command in the universe? ( all lower case )", "image": "https://website/reboot.gif", "choices": [ ], "correct": [ "reboot" ], "explanation": "The reboot command is the most great command!" },` – Matt Aug 06 '16 at 19:39
Shouldn't be hard to modify. I'd suggest building a function that JSON-ifies a single string, using the same transforms I'm doing in my function, and calling that once per value (perhaps from a command substitution inside a heredoc). – Charles Duffy Aug 06 '16 at 19:53
Edited again -- and for the last time; I've already humored considerably more scope creep than is generally considered acceptable. – Charles Duffy Aug 06 '16 at 19:57
I'm able to get it working, your response is most complete. Thanks for the lesson. – Matt Aug 06 '16 at 19:58

bash function input $1 is only processing first word of string & formatting issue

1 Answers1