133

I'm trying to use curl in bash to download a webpage, but the & symbol in the URL isn't interpreted as a character as I would like. Any ideas on how I can convince bash that the symbol & is just a boring character and nothing special?

Chernoff
  • 2,494
  • 2
  • 20
  • 24
  • Have a look at the following topic this should help you - http://stackoverflow.com/questions/2067969/encoding-an-ampersand-for-twitter-w-curl – morgents Nov 12 '12 at 09:18
  • 2
    I've tried replacing the `&` symbol with `\&`, `'&'`, `"&"`, `^&` but haven't been able to get the URL to evaluate correctly. Actually, the `'&'` works when I run the curl command from Terminal.app, but fails when I place it in a bash script, wrap it in a loop and run it. – Chernoff Nov 12 '12 at 14:53

7 Answers7

173

Putting the entire URL inside double quotes should take care of your problem.

mjuarez
  • 16,372
  • 11
  • 56
  • 73
  • 11
    single quotes might be better, unless there's a need to resolve variables. – jpm Nov 12 '12 at 07:01
  • 1
    When I use quotes around the whole URL the expression `${i}` doesn't return a value. – Chernoff Nov 12 '12 at 14:44
  • 2
    In my case, double quotes failed while single quotes worked just fine. – abeboparebop May 27 '16 at 17:13
  • 4
    Please use caution what program is used to type the quotes. While in a hurry to edit a shell script, I used a text editing program. It put slanted quotes instead of straight double quotes. This really messes up the script and makes the problem difficult to find! – David Jan 05 '17 at 11:57
  • Single quotes do not work in my .cmd file, but double quotes work – Michael Freidgeim Mar 02 '21 at 03:55
  • Confirmed windows needs double quotes and also make sure there are no spaces, leading or trailing, inside the quotes. – dbagnara Aug 13 '23 at 23:53
58
curl "http://www.example.com?m=method&args=1"

Are you using the & as a delimiter for a GET URL? Or is in a piece of data?

If it is in data you must encode it to an HTML character, if not, surround with quotes.

The encoding for & should become %26 in the URL.

curl "http://www.example.com?m=this%26that
Matt Clark
  • 27,671
  • 19
  • 68
  • 123
  • I'm using the `&` symbol as a piece of data, almost exactly as you've shown. The catch is that, instead of numerical values I'm using `${i}`. So I want to get `http://www.example.com?m=method&args=${i}` to evaluate as a URL. – Chernoff Nov 12 '12 at 14:46
  • 3
    The way I just showed the URL uses the '&' as a delimeter for the GET variables, so now I am confused, in $[i] you have a piece of data that contains '&'? If that is the case you will need to URL encode it. & = %26 – Matt Clark Nov 12 '12 at 17:47
  • 2
    Searched the whole internet couldn't find the solution. @MattClark you are the hero. I just url encoded & which was a piece of data and my script is working fine. – Haris ur Rehman Aug 12 '15 at 18:53
  • 4
    Is there a missing " at the end of the second code snippet? – David Jan 05 '17 at 11:58
31

Putting single quotes around the & symbol seems to work. That is, using a URL like http://www.example.com/page.asp?arg1=${i}'&'arg2=${j} with curl returns the requested webpage.

Chernoff
  • 2,494
  • 2
  • 20
  • 24
24

Instead of trying the escape "&" characters, you can specify http url parameters in POST requests with -d parameter as shown below:

curl -X POST http://www.example.com \
-d arg1=this \
-d arg2=that

If the request is a GET, then you should also add -G option, which tells curl to send the data with GET request.

curl -X GET -G http://www.example.com \
-d arg1=this \
-d arg2=that
koders
  • 5,654
  • 1
  • 25
  • 20
13

You can use %26 instead &. The entity code will work fine.

Seralto
  • 1,016
  • 1
  • 16
  • 30
4

Use the parameter --url with double quotes worked for me

Shehan Weerasooriya
  • 768
  • 2
  • 6
  • 23
3

easiest way :

curl -X GET -G http://example.com -d var1=$1 -d var2=$2
Saurav Kumar
  • 891
  • 8
  • 14