0

I want to create scipt to faciliate producing local text file extracts from Hive. This is to basically execute commands like below:

hive -e "SET hive.cli.print.header=true;SELECT * FROM dropme"|perl -pe 's/(?:\t|^)\KNULL(?=\t|$)//g'>extract/outbound/dropme.txt

While the above works like a charm I find it quite problematic to implement through the parametrized following script (much simplified): 

#!/bin/sh

TNAME=dropme
SQL="SELECT * FROM $TNAME"
echo $SQL
echo "SQL: $SQL"
EXTRACMD="hive -e \"SET hive.cli.print.header=true;$SQL\"|perl -pe 'BEGIN{if(defined(\$_=<ARGV>)){s/\b\w+\.//g;print}}s/(?:\t|^)\KNULL(?=\t|$)//g'>extract/outbound/$TNAME.txt"
echo "CMD: $EXTRACMD";
${EXTRACMD}

When run I get: Exception in thread "main" java.lang.NumberFormatException: For input string: "e"

I know there may be many flavours you can print the text or execute command. For instance the line echo $SQL prints me list of files in the directory instead:

SELECT file1.txt file2.txt file3.txt file4.txt FROM dropme

while the next one: echo "SQL: $SQL" gives just what I want: SQL: SELECT * FROM dropme

echo "CMD: $EXTRACMD" prints the (almost) the command to be executed. Almost, as I see \t in perl code being expanded:

CMD: hive -e "SET hive.cli.print.header=true;SELECT * FROM dropme"|perl -pe 'BEGIN{if(defined($_=<ARGV>)){s\w+\.//g;print}}s/(?: |^)\KNULL(?= |$)//g'>extract/outbound/dropme.txt

Maybe that's still ok, but what I want is to be able to copy&paste this command into (other) terminal and execute as the command I put at the top. Ideally I would like that command to be exactly the same (so with \t there)

Biggest problem I have comes when I try to execute it (${EXTRACMD} line). I'm getting the error:

Exception in thread "main" java.lang.NumberFormatException: For input string: "e" …and so on, irrelevant as bash treats every 'word' as single command here. I assume as I don't even know what is really tries to run (prior print attempt obviously doesn't help)

I'm aware that I have multiple options, like:

  • escaping special characters in the command definition string (like I did with doublequotes)
  • experimenting with echo and $VAR, '$VAR' or "$VAR"
  • experimenting with "${EXTRACMD}" or evaluating through eval "${EXTRACMD}"
  • experimenting with shopt -s extglob or set -f

but as number of combinations is quite large and with my little bash experience I feel it's better to ask for good practice here so my question is:

Is there a way to print a (complex/compound shell) command first and subsequently be able to execute it (exactly as per printed output)? In this case it would be printing the exact command from the top, then executing it the same way as by manually copying that output into terminal prompt and pressing Enter.

Cyrus
  • 84,225
  • 14
  • 89
  • 153
msciwoj
  • 772
  • 7
  • 23

1 Answers1

2

Do not construct commands as strings. See http://mywiki.wooledge.org/BashFAQ/050 for details.

That page also talks about a built-in way of getting the shell to tell you what it is running (section 6).

If that doesn't do what you want you can also, with bash, try using printf %q\\n "${arr[*]}".

Etan Reisner
  • 77,877
  • 8
  • 106
  • 148
  • Very well! What if I want the SQL variable to be passed by parameter? So executing the script as `hivedump.sh SELECT * FROM dropme` through the construct `SQL=$@` later gets expanded to `hive -e 'SET hive.cli.print.header=true;SELECT file1.txt file2.txt file3.txt FROM dropme…'` ? – msciwoj Oct 31 '14 at 11:40
  • @msciwoj You need to quote `*` when you use it as an argument to keep the shell from expanding it as a glob. And don't use `SQL=$@` and then use `$SQL` later that just re-creates the problem. Use `$@` *exactly* where you need it expanded. – Etan Reisner Oct 31 '14 at 12:56