10

I seem to be running into an issue that's specific to ksh88 that's changing single quotes to double quotes, but only under certain situations involving heredocs and command substitution.

Here's an example:

#!/bin/ksh

# This example works correctly
echo "Example 1:"
cat <<EOF
The 'quick' brown fox "jumped" over the lazy dog.
EOF
echo


# This example is broken
echo "Example 2:"
var=$(cat <<EOF
The 'quick' brown fox "jumped" over the lazy dog.
EOF)
echo "${var}"
echo


# This example works correctly
echo "Example 3:"
var=`cat <<EOF
The 'quick' brown fox "jumped" over the lazy dog.
EOF`
echo "${var}"
echo

And here's the output (note how Example 2 is different):

Example 1:
The 'quick' brown fox "jumped" over the lazy dog.

Example 2:
The "quick" brown fox "jumped" over the lazy dog.

Example 3:
The 'quick' brown fox "jumped" over the lazy dog.

The ' to " substitution seems to occur before the command runs. In actual context, the heredoc is passing SQL to Oracle. By changing ' to ", strings are being converted to identifiers, thus breaking the SQL. This can also be observed by enabling xtrace during execution of the above code.

How can I prevent the ' to " conversion in the above code snippet without using backticks?

Edit: The plot thickens. Replacing the command substituion $( ... ) with backtick notation doesn't replace the single quotes with double quotes. So (optional) question two: why?

Mr. Llama
  • 20,202
  • 2
  • 62
  • 115
  • 3
    The behaviour described sounds like a bug. Changing back-ticks to `$(…)` should not change the content of the output. Can you not upgrade to [`ksh93`](http://www.kornshell.com/)? – Jonathan Leffler Aug 29 '14 at 14:13
  • @JonathanLeffler - If it is a bug, it looks pretty dang deliberate. As for switching to ksh93, that's not necessarily an option. The actual script needs to run on multiple AIX and Solaris servers of varying versions. A few have ksh93, but most only have ksh88. – Mr. Llama Aug 29 '14 at 14:27
  • @AdrianFrühwirth - `echo ${var}` and `echo "${var}"` produce the same results in all of the above cases. If you enable xtrace `#!/bin/ksh -x` you can see the value of `var` even before it gets displayed. – Mr. Llama Aug 29 '14 at 14:34
  • +1 for model question. If only! You seem to know your kshs, but my experience with Solaris and AIX is that you might have to use an alternate path or even alternate name, like `dtksh` to get access to ksh93. (Can't remember what I had on AIX, but it was there). Good luck. – shellter Aug 29 '14 at 17:06
  • what level of AIX? Do: lslpp -w /usr/bin/ksh and then also give the level of that fileset. – pedz Aug 30 '14 at 04:17
  • As I recall, back ticks are not precisely the same as $( .. ) but I can't recall the differences. Note, another test that would be interesting is to do var="$( .... )". You can also try quoted here docs with "EOF" rather than just vanilla EOF. – pedz Aug 30 '14 at 04:18
  • 4
    Others have seen this, too. From http://www.in-ulm.de/~mascheck/various/cmd-subst/ : "In ksh88, at least from release a to i, you have to be aware of a subtle quoting issue inside $( ). Single quotes in embedded here-documents are converted to double quotes.". – Mark Plotnick Sep 02 '14 at 14:42
  • [Some further discussion on it at Google Groups](https://groups.google.com/forum/#!topic/gnu.bash.bug/lZYHvCtDu3M). Seems like the fix is to use back ticks, and the why has something to do with the parsing rules inside $() and backwards compatibility. – bishop Sep 02 '14 at 15:05
  • @MarkPlotnick: well done for tracking that down! Backwards compatibility can make fixing bugs a real pain. I'm not sure it is really justifiable even so; it is such a weird change to make, but it must have been coded deliberately — and yet it is nigh-on impossible to think of a good reason for wanting to change the quotes ever. It complicates life so much, as Mr Llama has discovered. – Jonathan Leffler Sep 02 '14 at 15:15

1 Answers1

7

Here are my notes when I discovered this same bug several years ago.

Test script:

#!/bin/ksh
cat <<EOF
  $PWD "$PWD" '$PWD'
EOF
echo `cat <<EOF
  $PWD "$PWD" '$PWD'
EOF
`
echo $(cat <<EOF
  $PWD "$PWD" '$PWD'
EOF
)

Output for different shells:

  • Linux KSH Version M 1993-12-28 q
  • Linux Bash 3.00.15(1)

(NOTE: works as expected)

 /home/jrw32982 "/home/jrw32982" '/home/jrw32982'
 /home/jrw32982 "/home/jrw32982" '/home/jrw32982'
 /home/jrw32982 "/home/jrw32982" '/home/jrw32982'
  • AIX Version M-11/16/88f
  • Solaris Version M-11/16/88i

(NOTE: single quotes replaced with double quotes and variable not substituted)

 /home/jrw32982 "/home/jrw32982" '/home/jrw32982'
 /home/jrw32982 "/home/jrw32982" '/home/jrw32982'
 /home/jrw32982 "/home/jrw32982" "$PWD"

Work-around:

  1. Compute the single-quoted string externally from the here-file

    abc=xyz
STR="'$abc'"
x=$( cat <<EOF
  $abc "$abc" $STR
EOF
)

  2. Use the here-file in a function instead of directly

    fn() {
  cat <<EOF
    $abc "$abc" '$abc'
EOF
}
abc=xyz
x=$(fn)
jrw32982
  • 608
  • 5
  • 11