Escaping the close-bracket character in a bash v3 heredoc

Question

I have a need to embed a fragment of a shell script in a heredoc as part of the creation of a cloud-init script to provision an Ubuntu 14.04 LTE machine. A simplified version of the script demonstrating the problem is as follows:

#!/bin/bash

cloudconfig=$(cat <<EOF
    if host \$NAMESERVER 1>/dev/null 2>&1; then
    case \$reason in
    BOUND|RENEW|REBIND|REBOOT) nsupdate -k /var/lib/dhcp/nsupdate.key << EOX
    server \$NAMESERVER
    update delete \$HOST_NAME A
    update add \$HOST_NAME \$TTL A \$HOST_ADDR
    send
    EOX
    ;;
    esac
    fi
EOF
)

echo "${cloudconfig}"

Running the above script fails as follows:

Little-Net:orchestration minfrin$ bash /tmp/test.sh
could not read key from /var/lib/dhcp/nsupdate.{private,key}: file not found
couldn't get address for '$NAMESERVER': not found

The problematic character is the closing bracket to the right of "REBOOT", and the obvious solution is to escape the character:

BOUND|RENEW|REBIND|REBOOT\) nsupdate -k /var/lib/dhcp/nsupdate.key << EOX

This backslash character however ends up in the final cloudconfig variable, which in turn breaks the output:

Little-Net:orchestration minfrin$ bash /tmp/test.sh
    if host $NAMESERVER 1>/dev/null 2>&1; then
    case $reason in
    BOUND|RENEW|REBIND|REBOOT\) nsupdate -k /var/lib/dhcp/nsupdate.key << EOX
    server $NAMESERVER
    update delete $HOST_NAME A
    update add $HOST_NAME $TTL A $HOST_ADDR
    send
    EOX
    ;;
    esac
    fi

This particular fragment above is part of a larger file that is being written that relies on variable interpolation, so quoting there heredoc with >>"EOF" is going to break the rest of our script.

How do I escape the ")" character without the escape character leaking through the heredoc?

The the problem is obviously **not** *the closing bracket to the right of "REBOOT"*. Note if you think the `)` is the problem, please create a most basic example which shows **only that** problem and don't post your whole mess here. Let me add that in this case, it would have shown you that the `)` is not the problem. — hek2mgl, Aug 19 '15 at 12:43
@hek2mgl The `)` is definitely not the problem. /var/lib/dhcp/nsupdate.key obviously doesn't exist or can't be seen, as the error clearly states, no idea why OP would think it was the bracket.@OP why are you trying to do it in cat< — 123, Aug 19 '15 at 12:57
No, the `)` is the problem. The shell sees it as closing the `$(...)` and tries to run the `nsupdate` command locally which is incorrect. — Etan Reisner, Aug 19 '15 at 13:08
@hek2mgl I did. It failed. This seems to be a bash version issue. It works [here](http://ideone.com/sJfeOB) but fails for me with bash 3. — Etan Reisner, Aug 19 '15 at 13:10
@EtanReisner You are right! :) Verified this. IMHO [bash] is not an appropriate tag here. Right would be [macos-ancient-bash] plus the question should go to http://history.stackexchange.com — hek2mgl, Aug 19 '15 at 13:37
@hek2mgl CentOS 5 also (though that's also ancient at this point even if it is what I do all of my day-to-day work on still at the moment). — Etan Reisner, Aug 19 '15 at 13:50
Just to clarify, this is the bash that comes by default with Ubuntu 14.04, released on 17 April 2014. The test case to demonstrate the problem was developed on MacOSX 10.10.4. No ancient versions of operating systems are being used here. — Graham Leggett, Aug 20 '15 at 09:15
It's not the OS that they're calling ancient, it's the version of `bash` that ships with Mac OS X. `bash` 4 was released over 6 years ago, but Apple does not ship it (possibly because of the change in license between verisions 3 and 4). — chepner, Aug 20 '15 at 11:30

score 3 · Accepted Answer · answered Aug 19 '15 at 13:16

As this seems to be a bash 3.x parsing issue (as it works in bash 4.x as can be seen here) you would need to avoid the parser getting confused. This seems to work for me:

#!/bin/bash

rp=")"
cloudconfig=$(cat <<EOF
    if host \$NAMESERVER 1>/dev/null 2>&1; then
    case \$reason in
    BOUND|RENEW|REBIND|REBOOT${rp} nsupdate -k /var/lib/dhcp/nsupdate.key << EOX
    server \$NAMESERVER
    update delete \$HOST_NAME A
    update add \$HOST_NAME \$TTL A \$HOST_ADDR
    send
    EOX
    ;;
    esac
    fi
EOF
)

echo "${cloudconfig}"

chepner · Answer 2 · 2015-08-19T14:12:30.567

1

Since you don't have any parameters you actually want to expand inside the here document, I would just quote the entire thing (which saves you a lot of explicit backslashes):

#!/bin/bash

cloudconfig=$(cat <<'EOF'
    if host $NAMESERVER 1>/dev/null 2>&1; then
    case $reason in
    BOUND|RENEW|REBIND|REBOOT) nsupdate -k /var/lib/dhcp/nsupdate.key << EOX
    server $NAMESERVER
    update delete $HOST_NAME A
    update add $HOST_NAME $TTL A $HOST_ADDR
    send
EOX
    ;;
    esac
    fi
EOF
)

echo "${cloudconfig}"

Note that you cannot indent EOX either, or else that here document will not be correctly terminated when you go to use it.

Even easier, though, is to not use a here document at all; just use embedded newlines in the parameter assignment.

#!/bin/bash

cloudconfig='
if host $NAMESERVER 1>/dev/null 2>&1; then
  case $reason in
    BOUND|RENEW|REBIND|REBOOT) nsupdate -k /var/lib/dhcp/nsupdate.key << EOX
    server $NAMESERVER
    update delete $HOST_NAME A
    update add $HOST_NAME $TTL A $HOST_ADDR
    send
EOX
    ;;
  esac
fi'

echo "${cloudconfig}"

If you need to allow some parameter expansion, you can still use the multiline string with some modifications. Whenever you need an interpolation, close the single quote and immediately open the double quote. After the expansion is complete, close the double and reopen the single.

cloudconfig='
    if [[ $SOMEVARIABLE =='"$value"' ]]; then
    ...
'

edited Aug 19 '15 at 14:12

answered Aug 19 '15 at 13:23

chepner

497,756
71
530
681

OP indicated that quoted heredoc is not an option: "This particular fragment above is part of a larger file that is being written that relies on variable interpolation, so quoting there heredoc with >>"EOF" is going to break the rest of our script." – Etan Reisner Aug 19 '15 at 13:49
The version without the here-document could be adapted to allow some parameter expansions; `cloudconfig='$foo='"$foo"', etc.'` – chepner Aug 19 '15 at 14:08
Using `cat` with a here-document to create a multi-line string is clumsy and inefficient. – chepner Aug 19 '15 at 14:10
I'm not arguing that just pointing out that your solution as-written was directly counter-indicated by the OP. And I'd argue that manually jumping in and out of single quotes like that is much clumsier, uglier and more prone to certain failures (which isn't to say that the unquoted heredoc isn't prone to its own failures, it absolutely is). – Etan Reisner Aug 19 '15 at 14:48
The test case indicated here is a small test to indicate the problem, and is not our complete solution. We make extensive use of variable interpolation, just not in this test case, and therefore your solution is not workable for us. – Graham Leggett Aug 20 '15 at 09:12

phyatt · Answer 3 · 2020-10-27T18:11:09.860

0

I solved it by escaping the parenths manually inside the HEREDOC and then follow it with a sed statement to remove the backslashes.

long_string=$(cat << 'HEREDOC'
This is a string with \( escaped \) parenthesis.
HEREDOC
)

long_string=$(echo $long_string | sed -e 's:\\(:(:g' -e 's:\\):):g')

Using bash 4+ would be better, but then I have to get everyone on my team using mac's to also get bash 4+.

edited Oct 27 '20 at 18:11

answered Oct 27 '20 at 17:29

phyatt

18,472
5
61
80

Escaping the close-bracket character in a bash v3 heredoc

3 Answers3