Bash does not remove leading whitespace

Question

I seem to have a very strange problem. I am trying to retrieve JSON field values from a CURL command using jsawk, however jsawk requires its JSON pretty-printed (which can be easily achieved with a properly formatted JSON file thanks to "python -mjson.tool").

The trouble is I have a whitespace at the start of the JSON file/string (which is illegal), however I can't seem to remove it.

 {"response": {"status": {"version": "4.2", "code": 5, "message": "The Identifier specified does not exist"}}}

Several options work independently of my script (eg echo ~/m.json | sed -e 's/^[ \t]*//')

{"response": {"status": {"version": "4.2", "code": 5, "message": "The Identifier specified does not exist"}}}

See the difference? But all the following methods do not achieve the desired result. I've even tried piping the string to sed to emulate command line behaviour with no luck. Can anyone point me in the right direction please?

thisjson=$(curl -F "api_key=$apikey" -F "format=json" -F "md5=$thismd5" -F... );
echo $thisjson > $tempjson; #Req'd else bash re-evals curl command
temp=$(cat $tempjson);      #Read string back to variable
echo $temp;                 #Try several methods to strip ws
temp="${temp##+([[:space:]])}";
echo $temp;
temp=$(sed -e 's/^[[:space:]]*//' <<<"$temp")
echo "|${temp}|";
temp=$(echo ${temp/ /} );
temp="${temp#"${temp%%[![:space:]]*}"}"
echo $temp;                #Try piping string directly
thisprettyjson=$(echo $temp | sed -e 's/^[ \t]*//' |python -mjson.tool);
echo $thisprettyjson;

Which spits out several lines (one for each echo) until "No JSON...decoded"

 {"response": {"status": {"version": "4.2", "code": 5, "message": "The Identifier specified does not exist"}}}
...
 {"response": {"status": {"version": "4.2", "code": 5, "message": "The Identifier specified does not exist"}}}
No JSON object could be decoded

I'm sure there is something silly I've missed. Perhaps the only other thing to mention would be I have changed the IFS, from Space/Tab/NL to simply NL.

Anyone got any ideas? Or an alternative simple method for parsing JSON? Thanks!

Try `printf "%q\n" "$temp"` -- that'll at least tell you what the whitespace character really is (since it's acting like it isn't a normal space). BTW, the `echo $thisjson > $tempjson; temp=$(cat $tempjson)` part should *not* be necessary -- curl is run when you use `$(curl ...)`, and not re-run when you expand $thisjson. Also, you don't need semicolons at the ends of lines in bash. — Gordon Davisson, Nov 09 '11 at 16:59

score 1 · Answer 1 · answered Nov 09 '11 at 11:59

I did the test using this:

tempjson='tempjson.txt'
thisjson=' {"response": {"status": {"version": "4.2", "code": 5, "message": "The Identifier specified does not exist"}}}'
echo $thisjson > $tempjson; #Req'd else bash re-evals curl command
...

It works for me:

{"response": {"status": {"version": "4.2", "code": 5, "message": "The Identifier specified does not exist"}}}
|{"response": {"status": {"version": "4.2", "code": 5, "message": "The Identifier specified does not exist"}}}|

So I would say that you have a special space character that is generated at the beginning of the string by your $(curl -F "api_key=$apikey" -F "format=json" -F "md5=$thismd5" -F... ); command

So try to remove the first char instead of a whitespace char if it is not a '{'

Yes, that's what I think I've tracked down- anything on the shell would work perfectly, however this command always seemed to prepend a whitespace. Good shout, though- I didn't think of removing the char from the string. — , Nov 09 '11 at 12:41

Bash does not remove leading whitespace

1 Answers1