54

So I have a very simple bash script that is curl'ing to an auth server for a header. The header url is written to a var and then used in the next curl call. When using the var set in the first curl call I am getting "curl: (3) Illegal characters found in URL". I am able to echo the var and all looks good, I am even able to reset the var (in my example below) and it works.

The Bash script

URL=$(curl -i -X GET -H "X-Auth-User: MyUserna,e" -H "X-Auth-Key: MyAPIKey" "https://urlToAuthServer.tld/auth/v1.0/" | grep "X-Storage-Url:" | awk '{print $2}')

curl -X GET -H "X-Auth-Token: MyAuthTok" "${URL}/folder/myfile.txt" -o ./myfile.txt

When running the above example I get:

curl: (3) Illegal characters found in URL

The URL var looks like this (no illegal chars)

https://somesecureurl.com/auth/AUTH_67383834-45245453-g34g34t5-34534

When I do this in terminal it works:

$ URL=$(curl -i -X GET -H "X-Auth-User: MyUserna,e" -H "X-Auth-Key: MyAPIKey" "https://urlToAuthServer.tld/auth/v1.0/" | grep "X-Storage-Url:" | awk '{print $2}')

$ echo $URL

https://somesecureurl.com/auth/AUTH_67383834-45245453-g34g34t5-34534

Now I copy and paste the string and reasign it to URL like so (again all in terminal):

>$ URL="https://somesecureurl.com/auth/AUTH_67383834-45245453-g34g34t5-34534"
>$ curl -X GET -H "X-Auth-Token: MyAuthTok" "${URL}/folder/myfile.txt" -o ./myfile.txt

It works.

So why am I getting the "curl: (3) Illegal characters found in URL" error in the first example?

Update I ran this: printf %s "$URL" | xxd

Here is the output (addressed changed up you get the idea)

0000000: 6874 7470 733a 2f2f 6461 6c30 352e 6f62  https://server.ob
0000010: 6a65 6374 7374 6f72 6167 652e 736f 6674  jectstorage.lite
0000020: 6c61 7965 722e 6e65 742f 7631 2f41 5554  sabers.com/v1/AUT
0000030: 485f 6665 3235 3339 3434 2d38 6433 322d  H_aE2563981-7d32-
0000040: 3432 3138 2d61 6566 632d 6665 6638 3465  4201-bdoi-fef94a
0000050: 6166 3331 6232 0d                        ag11c8.
gregwinn
  • 941
  • 1
  • 9
  • 16
  • Any whitespace, newlines, hidden characters in the url? Try with `printf %s "$URL" | xxd`. – choroba Jan 26 '16 at 16:37
  • @choroba, I am getting a "." at the end like so: 0000050: 6166 3331 6232 0d af31b2. thats the last line in the output. – gregwinn Jan 26 '16 at 16:45

4 Answers4

126

The $URL contains a \r (CR) at the end (0d). Remove it with

URL=${URL%$'\r'}

before using it with curl.

choroba
  • 231,213
  • 25
  • 204
  • 289
  • I understand that parameter expansion with % expands `$'\r'` into a pattern and then deletes that pattern from the URL. But why is the $ needed before the '\r'? – Noumenon Feb 03 '17 at 04:35
  • 6
    @Noumenon: Because `'\r'` is two characters, backslash and `r`, while `$'\r'` is one character, the CR. `$'...'` are different quotes than `'...'`. – choroba Feb 03 '17 at 16:51
  • Worked for me as well, I am wondering why curl cannot do it on its own. – Adam Siemion Nov 16 '18 at 15:08
  • 1
    @AdamSiemion: Because it would break calls to urls actually containing the character (even if there are probably not many of them). – choroba Nov 16 '18 at 15:38
  • @choroba if there are such URLs then they are not supported by curl, curl then just fails and does not make any request – Adam Siemion Nov 16 '18 at 16:05
10

I had the same problem, but it was caused by me having created the script through Windows, which meant that the end of line had an unnecessary \r.

A simple dos to unix conversion fixed it.

dos2unix <scriptname>
ianiver9
  • 101
  • 1
  • 2
3

stripping away trailing \r

tr -d '\r' < test.sh > testWithoutR.sh
amalik2205
  • 3,962
  • 1
  • 15
  • 21
0

It could be an enter at the end of the file. Try to remove it or delete it, note that the enter character is not something view-able....

jeff porter
  • 6,560
  • 13
  • 65
  • 123
ichak khoury
  • 217
  • 2
  • 11