WGET 401 Unauthorized

Question

I'm trying to use a batch file with WGET to download the public FCC file from here

http://wireless.fcc.gov/uls/data/complete/l_micro.zip

When I intially run the batch file with parameters

wget --server-response -owget.log http://wireless.fcc.gov/uls/data/complete/l_micro.zip

It fails with an HTTP 401 unauthorized error. I can retry at this point and it keeps failing. However I noticed if I open up IE, start a download and cancel when prompted to save, I can rerun the batch file and it executes perfectly!

Here is my detailed server response from the log

--2012-02-06 14:32:24--  http://wireless.fcc.gov/uls/data/complete/l_micro.zip
Resolving wireless.fcc.gov (wireless.fcc.gov)... 192.104.54.158
Connecting to wireless.fcc.gov (wireless.fcc.gov)|192.104.54.158|:80... connected.
HTTP request sent, awaiting response... 
  HTTP/1.1 302 Found
  Location: REMOVED - appears to have my IP
  Cache-Control: no-cache
  Pragma: no-cache
  Content-Type: text/html; charset=utf-8
  Connection: close
  Content-Length: 513
Location: REMOVED [following]
--2012-02-06 14:32:24--  REMOVED
Resolving REMOVED... 192.168.2.11
Connecting to REMOVED|192.168.2.11|:80... connected.
HTTP request sent, awaiting response... 
  HTTP/1.1 401 Unauthorized
  Cache-Control: no-cache
  Pragma: no-cache
  WWW-Authenticate: NTLM
  WWW-Authenticate: BASIC realm="AD_BCAAA"
  Content-Type: text/html; charset=utf-8
  Proxy-Connection: close
  Set-Cookie: BCSI-CS-8ECFB6B4AA642EF0=2; Path=/
  Connection: close
  Content-Length: 575
Authorization failed.

Here is the log after doing my little IE procedure and getting it to work

--2012-02-08 15:52:43--  http://wireless.fcc.gov/uls/data/complete/l_micro.zip
Resolving wireless.fcc.gov (wireless.fcc.gov)... 192.104.54.158
Connecting to wireless.fcc.gov (wireless.fcc.gov)|192.104.54.158|:80... connected.
HTTP request sent, awaiting response... 
  HTTP/1.1 200 OK
  Server: Sun-Java-System-Web-Server/7.0
  Date: Fri, 27 Jan 2012 18:37:51 GMT
  Content-type: application/zip
  Last-modified: Sun, 22 Jan 2012 11:18:09 GMT
  Etag: "46fa95c-4f1bf071"
  Accept-ranges: bytes
  Content-length: 74426716
  Connection: Keep-Alive
  Age: 1045014
Length: 74426716 (71M) [application/zip]
Saving to: `l_micro.zip'

Any help is appreciated!

I attempted `wget http://wireless.fcc.gov/uls/data/complete/l_micro.zip` and got a file `l_micro.zip` of `74693506` bytes with sha1sum of `1f8393bf6231bfa9689ec91cab6a660244f35605`. — Dan D., Feb 08 '12 at 21:13
Do you have a transparent proxy on your network? Can you run wireshark or fiddler on your IE session to see what it's doing differently? — Edward Thomson, Feb 09 '12 at 17:56

score 39 · Answer 1 · answered Mar 20 '13 at 04:10

39

If the website has simply a htpassword setup, you can try:

wget --user=admin --ask-password https://www.yourwebsite.com/file.zip

answered Mar 20 '13 at 04:10

Federico Giorgi

10,495
9
42
56

score 12 · Answer 2 · answered Jul 30 '15 at 11:41

12

I used --auth-no-challenge and the exact error get solved .

answered Jul 30 '15 at 11:41

Omid Ataollahi

368
3
8

Perfect... The API I was calling wasn't sending the auth challenge. [This option](https://www.gnu.org/software/wget/manual/html_node/HTTP-Options.html) forces wget to always send the Authorization header. – mlathe Mar 09 '18 at 16:28

score 10 · Answer 3 · answered Feb 09 '12 at 18:05

You have a Blue Coat secure web gateway on your network, as evidenced by the line in the response:

Set-Cookie: BCSI-CS-8ECFB6B4AA642EF0=2; Path=/

It looks like it wants you to authenticate, presumably with your domain credentials. Try passing them with --http-user and --http-passwd.

score 6 · Answer 4 · answered Apr 19 '14 at 23:58

I had a similar issue with the xwiki based site. after several attempts I found some combination that worked for me just fine

wget --no-check-certificate --auth-no-challenge -k -nc -p -l 1 -r https://user:password@host.domain

I think the key was --auth-no-challenge

score 2 · Answer 5 · answered Jul 17 '18 at 11:26

2

Try using this extension for firefox. It generates a wget or a curl command that can be copied and run from bash.

answered Jul 17 '18 at 11:26

Vasantha Ganesh

4,570
3
25
33

2

Brilliant! Thanks for this, helped me out greatly! – rbennell Sep 12 '18 at 10:23

score 1 · Answer 6 · answered Mar 13 '17 at 18:32

I came here trying to find out why wget was giving a 401 unauthorized message when on another system the problem did not occur.

After installing a later version of wget from source (binary was not available in my distro) it worked. I can't explain why, except that it must be some kind of bug so if none of the above fixes your problem, consider upgrading wget.

robf · Answer 7 · 2012-02-09T01:10:45.867

0

Try setting a user-agent string with wget - e.g.

--user-agent=Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; .NET CLR 1.1.4322; .NET CLR 2.0.50727)

it's entirely feasible for a site to reject requests from certain user agents, particularly if they look to be circumventing the "usual" routes to information (i.e. through webpages).

Although this doesn't explain your problem, it's a good idea anyway. Perhaps the site implements a mechanism whereby when you browse with a "known" browser (e.g. IE) it then caches your IP as "safe" then allows any user agent from your IP to download anything :)

edited Feb 09 '12 at 01:10

answered Feb 09 '12 at 00:35

robf

11
2

I tried your suggestion and also used my own UA from my browser and it did not help. It seems to be on and off, when I got on my computer this morning it worked fine regardless of the settigns and recently stopped working. Could this somehow be a firewall issue or is it likely something else? – user1192537 Feb 09 '12 at 13:48

WGET 401 Unauthorized

7 Answers7