Clean way to write complex multi-line string to a variable

Question

I need to write some complex xml to a variable inside a bash script. The xml needs to be readable inside the bash script as this is where the xml fragment will live, it's not being read from another file or source.

So my question is this if I have a long string which I want to be human readable inside my bash script what is the best way to go about it?

Ideally I want:

to not have to escape any of the characters
have it break across multiple lines making it human readable
keep it's indentation

Can this be done with EOF or something, could anyone give me an example?

e.g.

String = <<EOF
 <?xml version="1.0" encoding='UTF-8'?>
 <painting>
   <img src="madonna.jpg" alt='Foligno Madonna, by Raphael'/>
   <caption>This is Raphael's "Foligno" Madonna, painted in
   <date>1511</date>-<date>1512</date>.</caption>
 </painting>
EOF

I'm willing to bet that you're just going to dump that data into a stream again. Why store it in a variable when you could make things more complex and use streams? — Zenexer, Sep 26 '13 at 22:57
see this too: [Multi-line string with extra space (preserved indentation)](https://stackoverflow.com/questions/23929235/multi-line-string-with-extra-space-preserved-indentation) — Yibo Yang, Jun 10 '17 at 16:51

Dennis Williamson · Accepted Answer · 2018-05-06T15:46:50.780

167

This will put your text into your variable without needing to escape the quotes. It will also handle unbalanced quotes (apostrophes, i.e. '). Putting quotes around the sentinel (EOF) prevents the text from undergoing parameter expansion. The -d'' causes it to read multiple lines (ignore newlines). read is a Bash built-in so it doesn't require calling an external command such as cat.

IFS='' read -r -d '' String <<"EOF"
<?xml version="1.0" encoding='UTF-8'?>
 <painting>
   <img src="madonna.jpg" alt='Foligno Madonna, by Raphael'/>
   <caption>This is Raphael's "Foligno" Madonna, painted in
   <date>1511</date>-<date>1512</date>.</caption>
 </painting>
EOF

edited May 06 '18 at 15:46

answered Oct 08 '09 at 12:37

Dennis Williamson

62,149
16
116
151

I think avoiding `cat` is not an advantage here. At least you save some characters by using it. ;) – joschi Oct 08 '09 at 21:14
6

`cat` is an external command. Not using it saves doing that. Plus, some have the philosophy that if you're using cat with fewer than two arguments "Ur doin' it wrong" (which is distinct from "useless use of `cat`"). – Dennis Williamson Oct 09 '09 at 00:03
This doesn't work for me unless I put a space between `-d` and `''` -- presumably because it doesn't realize there's an empty string after `-d` unless it's technically a separate parameter. This may also depend on what version of bash you have (I'm using 3.2.17 under Mac OS X v10.5.8). – Gordon Davisson Oct 09 '09 at 02:54
You're right about the `-d` needing a space after it. I may have typo'd that. I've fixed it now. – Dennis Williamson Oct 09 '09 at 06:04
12

and never ever indent second EOF.... (multiple table to head bangs involved) – IljaBek May 18 '12 at 19:40
1

Halas, this solution doesn't allow variable substitution like ${foo}... – Offirmo Sep 27 '12 at 16:53
@Offirmo: Hence my third sentence. – Dennis Williamson Sep 27 '12 at 17:09
12

I tried to use the above statement while `set -e`. It seems `read` always returns non-zero. You can thick this behaviour by using `! read -d .......` – krissi Nov 08 '12 at 10:56
1

@krissi: You should [never use](http://mywiki.wooledge.org/BashFAQ/105) `set -e` - it's an archaic relic. Use proper error handling. – Dennis Williamson Nov 08 '12 at 14:14
@krissi: read returns non-zero in this case because it encounters end-of-file before the (non-existent) delimiter. – Andrew Nov 19 '12 at 22:36
3

@DennisWilliamson: "Proper" error handling in shell is prohibitively tedious. `set -e` is imperfect and lays the occasional trap, but makes scripts far more reliable. – Andrew Nov 19 '12 at 22:42
2

@Andrew: reliable until it isn't – Dennis Williamson Nov 19 '12 at 23:09
this solution doesn't preserve leading spaces in the first line of the text. For example, try adding a few spaces before ` – dogbane Dec 04 '12 at 15:11
You could use `IFS= read -d'' String << "EOF"` to preserve leading space in the first line of the text. – dogbane Dec 04 '12 at 15:20
2

@krissi: You could use `read -d '' String <<"EOF" ||:` to avoid the error trap. – VladV Jun 04 '13 at 08:18
1

But I <3 cat :( – Zenexer Sep 26 '13 at 22:53
13

And if you are using this multi-line `String` variable to write to a file, put the variable around "QUOTES" like `echo "${String}" > /tmp/multiline_file.txt` or `echo "${String}" | tee /tmp/multiline_file.txt`. Took me more than an hour to find that. – Aditya Apr 20 '14 at 16:16
Can I still reference bash variables inside of this? – AAM111 Aug 06 '17 at 15:07
@OldBunny2800: "Putting quotes around the sentinal (EOF) prevents the text from undergoing parameter expansion." – Dennis Williamson Aug 23 '17 at 13:04
Even though the desire to avoid running external commands is understandable `cat` is still much safer option. I can see at least two issues in the provided examples: losing leading spaces and backslashes. To solve the first one IFS should be changed by `IFS=''`. For the second one `-r` should be used: `read -r -d '' ...` – Stanislav German-Evtushenko May 06 '18 at 13:36
Be sure to `echo "$String"` in quotes to inspect your work or newlines won't print. – rjurney Sep 24 '20 at 03:04

score 41 · Answer 2 · answered Oct 08 '09 at 11:14

41

You've been almost there. Either you use cat for the assembly of your string or you quote the whole string (in which case you'd have to escape the quotes inside your string):

#!/bin/sh
VAR1=$(cat <<EOF
<?xml version="1.0" encoding='UTF-8'?>
<painting>
  <img src="madonna.jpg" alt='Foligno Madonna, by Raphael'/>
  <caption>This is Raphael's "Foligno" Madonna, painted in
  <date>1511</date>-<date>1512</date>.</caption>
</painting>
EOF
)

VAR2="<?xml version=\"1.0\" encoding='UTF-8'?>
<painting>
  <img src=\"madonna.jpg\" alt='Foligno Madonna, by Raphael'/>
  <caption>This is Raphael's \"Foligno\" Madonna, painted in
  <date>1511</date>-<date>1512</date>.</caption>
</painting>"

echo "${VAR1}"
echo "${VAR2}"

answered Oct 08 '09 at 11:14

joschi

21,387
3
47
50

1

Unfortunately, the apostrophe in "Raphael's" makes the first one not work. – Dennis Williamson Oct 08 '09 at 12:37
Both assignments work for me eventually. The single quote in VAR1 should not be a problem (at least not for bash). Maybe you have been misled by the syntax highlighting? – joschi Oct 08 '09 at 21:13
2

It works in a script, but not at a Bash prompt. Sorry for not being clearer. – Dennis Williamson Oct 09 '09 at 06:08
2

It is better to quote EOF as `'EOF'` or `"EOF"`, otherwise shell variables will be parsed. – Stanislav German-Evtushenko May 06 '18 at 13:43
Be sure to `echo "$VAR1"` with quotes to inspect your work or newlines will not print. – rjurney Sep 24 '20 at 03:04

score 21 · Answer 3 · answered Sep 05 '12 at 15:54

21

#!/bin/sh

VAR1=`cat <<EOF
<?xml version="1.0" encoding='UTF-8'?>
<painting>
  <img src="madonna.jpg" alt='Foligno Madonna, by Raphael'/>
  <caption>This is Raphael's "Foligno" Madonna, painted in
  <date>1511</date>-<date>1512</date>.</caption>
</painting>
EOF
`
echo "VAR1: ${VAR1}"

This should work fine within Bourne shell environment

answered Sep 05 '12 at 15:54

schweiz

219
2
2

3

+1 this solution allow variable substitution like ${foo} – Offirmo Sep 27 '12 at 16:54
Upside: sh-compatible. Downside: backticks are deprecated/discouraged in bash. Now if I had to choose between sh and bash... – Zenexer Sep 26 '13 at 22:55
2

since when are backticks deprecated/discouraged? just curious – Alexander Mills Apr 21 '18 at 01:50

F. Hauri - Give Up GitHub · Answer 4 · 2013-07-23T02:26:09.130

Yet another way to do the same...

I like to use variables and special <<- who drop tabulation at begin of each lines to permit script indentation:

#!/bin/bash

mapfile Pattern <<-eof
        <?xml version="1.0" encoding='UTF-8'?>
        <painting>
          <img src="%s" alt='%s'/>
          <caption>%s, painted in
          <date>%s</date>-<date>%s</date>.</caption>
        </painting>
        eof

while IFS=";" read file alt caption start end ;do
    printf "${Pattern[*]}" "$file" "$alt" "$caption" "$start" "$end"
  done <<-eof
        madonna.jpg;Foligno Madonna, by Raphael;This is Raphael's "Foligno" Madonna;1511;1512
        eof

warning: there is no blank space before eof but only tabulation.

<?xml version="1.0" encoding='UTF-8'?>
 <painting>
   <img src="madonna.jpg" alt='Foligno Madonna, by Raphael'/>
   <caption>This is Raphael's "Foligno" Madonna, painted in
   <date>1511</date>-<date>1512</date>.</caption>
 </painting>

Some explanations:

mapfile read entire here document in an array.
the syntaxe "${Pattern[*]}" do cast this array into a string.
I use IFS=";" because there is no ; in required strings
The syntaxe while IFS=";" read file ... prevent IFS to be modified for the rest of the script. In this, only read do use the modified IFS.
no fork.

Note that `mapfile` requires Bash 4 or higher. And the syntax `"${Pattern[*]}"` casts the array into a string *when in quotes* (as shown in the example code). — Dennis Williamson, Jul 22 '13 at 23:54

score 2 · Answer 5 · edited May 23 '17 at 12:41

2

There are too many corner cases in many of the other answers.

To be absolutely sure there are no issues with spaces, tabs, IFS etc., a better approach is to use the "heredoc" construct, but encode the contents of the heredoc using uuencode as explained here:

https://stackoverflow.com/questions/6896025/#11379627.

edited May 23 '17 at 12:41

Community

1

answered May 17 '14 at 09:10

Chris Johnson

805
6
6

Clean way to write complex multi-line string to a variable

5 Answers5

Yet another way to do the same...