How to execute curl command stored in heredoc in bash script?

Question

When write bash scripts, I want to store my whole curl command in heredoc to get a better layout. The following works fine:

#/bin/bash

read -r -d '' command1 <<- MULTI_STRING_SCOPE
  curl -v www.stackoverflow.com
MULTI_STRING_SCOPE

But when add some json data with the -d option, the command is executed weirdly. For example:

#/bin/bash

read -r -d '' command2 <<- MULTI_STRING_SCOPE
  curl -v www.stackoverflow.com
    -d '{
      "hello":"world"
    }'
MULTI_STRING_SCOPE
response2=$(${command2})

Wrong logs from terminal:

curl: (3) URL using bad/illegal format or missing URL
curl: (3) unmatched close brace/bracket in URL position 1:
}'

And it seems that the curl take line }' as a seperated url, and thus the json data not sent as a unit.

How to solve the problem? Any suggestions will be highly appreciated.

Storing commands as strings is a bad idea; it may make the layout better, but the parsing is a confusing mess. Basically, variables are for storing data, not executable code. See [BashFAQ #50: I'm trying to put a command in a variable, but the complex cases always fail!](http://mywiki.wooledge.org/BashFAQ/050) In cases like this, I'd strongly recommend using functions instead of variables. (Note: some people recommend `eval` for things like this, but that just opens up new sorts of parsing confusion. That way lies madness and really weird bugs.) — Gordon Davisson, Oct 30 '21 at 03:01
@GordonDavisson That's really something. I once learned things about the story of `data and command`, but I just did not recognize myself got engaged in this situation. Thx for your tips. — Lebecca, Oct 30 '21 at 03:09

score 1 · Accepted Answer · answered Oct 30 '21 at 07:36

1

I learned from this post to make heredoc work with curl command.

As the comment made by @Gordon Davisson in current post, we should not mix command with data. Since the json data set to -d option is only data and other parts is command, so I decide to use heredoc to store only the json data and remain other parts to be command itself, rather than store them in string by heredoc.

The result bash script should be something like this:

#/bin/bash

response3=$(curl -v www.stackoverflow.com \
  -d @- <<- MULTI_STRING_SCOPE
        {
            "hello":"world"
        }
    MULTI_STRING_SCOPE
  )

Notice: heredoc indent only works with tab, not with blanks. Be careful, especially when you are working with editors like Visual Studio Code, which may have already set indent as blanks for you.

answered Oct 30 '21 at 07:36

Lebecca

2,406
15
32

What actual value does the here document provide here, though? Wouldn't it be simpler and clearer to just pass in a multi-line string? – tripleee Oct 30 '21 at 08:03
@tripleee To the case I gave above, the here document is not necessary. In my fact situation, I need to send data in json format with '-d' option, with variable substitution to that json data at the same time. Because there are so many double quotes in json, I choose `-d '{"json":"data"}'` to work with single quote, and in single quote the variable substitution seems not work, so I turn my way to `here document`. Is this make sense, or it can also be achieved by multi-line string in a simple way? – Lebecca Oct 30 '21 at 09:22
You can't get variable substitution inside single quotes. A here document is actually not a bad solution for this scenario; other solutions include "seesaw quoting", like `'{"key" :"'"$value"'"}'` (which is just as messy as escaping all literal double quotes with a backslash) or using an external utility like `sed` to replace placeholder tokens in an otherwise static string or document. – tripleee Oct 30 '21 at 10:05
I learned a lot, thank you @tripleee – Lebecca Oct 30 '21 at 11:46

How to execute curl command stored in heredoc in bash script?

1 Answers1