How to escape single quote characters in bash command substitution

Question

I want to generate a SQOOP command which is appended by some variable like CUSTOM_PARAMS. I have defined the variable in a file : say hi.cfg

The variable have some single quotes as well like 'orc'.

cat hi.cfg
CUSTOM_PARAMS="--query select * from blah..blah where \$CONDITIONS --split-by blah --create-hcatalog-table --hcatalog-external-table --hcatalog-storage-stanza stored as 'orc' --compress --compression-codec 'snappy' --fields-terminated-by '|' -m 5 --target-dir /any/dir"

When I source to the file on command prompt and do echo it gives me exactly whats written over there which looks correct.

source hi.cfg
echo "$CUSTOM_PARAMS"
--query select * from blah..blah where $CONDITIONS --split-by blah --create-hcatalog-table --hcatalog-external-table --hcatalog-storage-stanza stored as 'orc' --compress --compression-codec 'snappy' --fields-terminated-by '|' -m 5 --target-dir /any/dir

But when I call this from a shell script as below :

cat hi.sh

echo "Generating Sqoop Command"
source $HOME/hi.cfg
echo "${CUSTOM_PARAMS}"
SQOOP_COMMAND="SQOOP statement : sqoop import blah blah "$CUSTOM_PARAMS""
echo $SQOOP_COMMAND

The * character in the variable is been treated as a command:

sh hi.sh

Generating Sqoop Command
--query select * from blah..blah where $CONDITIONS --split-by blah --create-hcatalog-table --hcatalog-external-table --hcatalog-storage-stanza stored as 'orc' --compress --compression-codec 'snappy' --fields-terminated-by '|' -m 5 --target-dir /any/dir
SQOOP statement : sqoop import blah blah --query select 0 00 000000_0 000073_0 000103_0 02 09.txt 1  from blah..blah where $CONDITIONS --split-by blah --create-hcatalog-table --hcatalog-external-table --hcatalog-storage-stanza stored as 'orc' --compress --compression-codec 'snappy' --fields-terminated-by '|' -m 5 --target-dir /any/dir

I need to run the SQOOP Statement later on in the script and have tried couple of options but didn't help. I also tried \* but didn't help, it outputs:

Generating Sqoop Command
--query select \* from blah..blah where $CONDITIONS --split-by blah --create-hcatalog-table --hcatalog-external-table --hcatalog-storage-stanza stored as 'orc' --compress --compression-codec 'snappy' --fields-terminated-by '|' -m 5 --target-dir /any/dir
SQOOP statement : sqoop import blah blah --query select \* from blah..blah where $CONDITIONS --split-by blah --create-hcatalog-table --hcatalog-external-table --hcatalog-storage-stanza stored as 'orc' --compress --compression-codec 'snappy' --fields-terminated-by '|' -m 5 --target-dir /any/dir`

janos · Accepted Answer · 2016-11-13T12:13:35.680

Change this line:

SQOOP_COMMAND="SQOOP statement : sqoop import blah blah "$CUSTOM_PARAMS""

To this:

SQOOP_COMMAND="SQOOP statement : sqoop import blah blah $CUSTOM_PARAMS"

Also change this line:

echo $SQOOP_COMMAND

To this:

echo "$SQOOP_COMMAND"

Or if you want to embed the extra double-quotes, then to this:

SQOOP_COMMAND="SQOOP statement : sqoop import blah blah \"$CUSTOM_PARAMS\""

What's happening is that the way you wrote it, the first embedded double-quote closes the quoted expression. Effectively you have there SQOOP statement : sqoop import blah blah quoted, followed by $CUSTOM_PARAMS unquoted, followed by an empty string (""). If you want to embed double-quotes within double-quotes, then you need to escape them with \.

But it seems to me you don't want to embed double-quotes at all.

How to escape single quote characters in bash command substitution

1 Answers1