1

I want to assemble a curl invocation with two variables (one for common arguments and one for the URL) and capture the output via command substitution. This breaks when I add parameters that need enclosing quotes, like say -H "Accept: text/html".

Here's a small script to demonstrate the problem, which calls a public echo service to show the sent request headers:

#!/bin/bash
COMMON_ARGS="--http1.1 -H \"Accept: text/html\""
URL="http://scooterlabs.com/echo"
RETVAL=$(curl -s $COMMON_ARGS $URL | grep Accept)
echo "1 $RETVAL"   
RETVAL=$(curl -s --http1.1 -H "Accept: text/html" $URL | grep Accept)
echo "2 $RETVAL"

Output:

1           [Accept] => */*
2           [Accept] => text/html

So the first attempt where I try to supply the header via a variable, is not working (curl sends the default Accept: */* ), whereas the second attempt, where I manually added the header, works. Neither using single quotes nor quote escaping did help.

Update: I learnt at least a part of my lesson here when I stumbled upon a BashFaq/050 just before submitting the question. The problem is word splitting performed by the shell's parser before it invokes a given command. The correct way to do it my case is by supplying the parameters as an array (older bashes may not support this):

COMMON_ARGS=(--http1.1 -H "Accept: text/html")
RETVAL=$(curl -s "${COMMON_ARGS[@]}" $URL | grep Accept)
echo "3 $RETVAL"

Output:

3             [Accept] => text/html

But there's still one thing I couldn't figure out: If word splitting by default acts on a space, a tab, and a newline. then why doesn't this variant work - there's no space to wrongly split?

COMMON_ARGS="--http1.1 -H 'Accept:text/html'"
RETVAL=$(curl -s $COMMON_ARGS $URL | grep Accept)
echo "4 $RETVAL"

Output:

4             [Accept] => */*
zb226
  • 9,586
  • 6
  • 49
  • 79
  • 1
    But there are the single quotes which would be normally removed in quote removal before curl saw the header. By including them in double quotes, they're preserved, but curl doesn't know what to do with them. – choroba Dec 15 '20 at 21:09
  • @choroba: that makes a lot of sense. thanks, please post that comment as an answer! – zb226 Dec 15 '20 at 21:16
  • Hmm. Since you spotted BashFAQ #50 and amended the question, that makes it not so duplicative of the many other questions we have about the same problem. Perhaps the title should be edited to distinguish? – Charles Duffy Dec 15 '20 at 21:38
  • @CharlesDuffy i read not just one of these questions, but the missing bit of information wasn't conveyed by the answers ... thought this might well help others! – zb226 Dec 15 '20 at 21:58

1 Answers1

1

there's no space to wrongly split?

But there are the single quotes which would be normally removed in quote removal before curl saw the header. By including them in double quotes, they're preserved, but curl doesn't know what to do with them.

choroba
  • 231,213
  • 25
  • 204
  • 289