0

What should I be looking for, when wget returns 500 but the same URL works fine in my web browser? I don't see any access_log entries that seem to be related to the error.

DEBUG output created by Wget 1.14 on linux-gnu.

<SSL negotiation info stripped out>

---request begin---
GET /survey/de/tools/clear-caches/password/<some-token> HTTP/1.1
User-Agent: Wget/1.14 (linux-gnu)
Accept: */*
Host: testing.thesurveylab.net
Connection: Keep-Alive

---request end---
HTTP request sent, awaiting response...
---response begin---
HTTP/1.0 500 Internal Server Error
Date: Wed, 12 Dec 2012 14:53:07 GMT
Server: Apache/2.2.3 (CentOS)
Set-Cookie: blueprint2-staging=8jnbmkqapl30hjkgo0u6956pd1; path=/
Expires: Thu, 19 Nov 1981 08:52:00 GMT
Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0
Pragma: no-cache
Strict-Transport-Security: max-age=8640000;includeSubdomains
X-UA-Compatible: IE=Edge,chrome=1
Content-Length: 5
Connection: close
Content-Type: text/html; charset=UTF-8

---response end---
500 Internal Server Error

Stored cookie testing.thesurveylab.net -1 (ANY) / <session> <insecure> 
[expiry none] blueprint2-staging 8jnbmkqapl30hjkgo0u6956pd1
Closed 3/SSL 0x0000000001f33430
2012-12-12 15:53:07 ERROR 500: Internal Server Error.
markus
  • 602
  • 2
  • 9
  • 18
  • 1
    Please share the full output of your `wget` command. It could be that the server is checking for a matching user agent string or performing a redirect `wget` interprets differently. – gertvdijk Dec 12 '12 at 14:51
  • You should add --debug to wget command and provide more information to us – Valentin Bajrami Dec 12 '12 at 14:52
  • My suggestion would be to try and use curl. It probably will throw a 500 identical arror but might fetch the page. Let me know of the progress – Valentin Bajrami Dec 12 '12 at 15:02
  • I will try that but I want to understand and fix the problem, not just switch to curl and let it be. – markus Dec 12 '12 at 15:03
  • 2
    I am almost certain that this problem is a result of a buggy webpage. You should contact the maintainer of the webpage and report this. wget gives a faulty 500 report. – Valentin Bajrami Dec 12 '12 at 15:06
  • Well, no, it's not a 'buggy webpage' as such. This is all I can post at the moment. I'm pretty sure it has something to do with a redirect, I read, wget is pretty bad at handling those. – markus Dec 12 '12 at 15:07
  • A similair problem described in http://stackoverflow.com/questions/6083102/http-500-error-in-wget – Valentin Bajrami Dec 12 '12 at 15:13
  • Yep, I've seen this, of course. – markus Dec 12 '12 at 15:13

2 Answers2

1

This question had been already asked and answered into this thread.

It's a bug in the webpage. 
The HTTP status is indeed seemingly incorrectly set to HTTP 500. Firefox/Firebug also confirms this. 
Basically, you're facing a HTTP 500 error page with "normal" content.

    Report it to the site admin.

https://stackoverflow.com/questions/6083102/http-500-error-in-wget

Community
  • 1
Napster_X
  • 3,373
  • 18
  • 20
  • 2
    "It's a buggy webpage" is NOT an answer as much as "doesn't work" is not a valid error message. – markus Dec 12 '12 at 15:37
  • 1
    We really do prefer that answers contain content not pointers to content. Whilst this may theoretically answer the question, [it would be preferable](http://meta.stackexchange.com/q/8259) to include the essential parts of the answer here, and provide the link for reference. – user9517 Dec 14 '12 at 20:19
1

There are probably a gazillion possible reasons for this but whenever I've struck the same issue, whether it be using wget or some other downloader, it's usually because there is code on the site which is designed to respond in certain ways to "normal" web browsers. Along comes wget, or similar, and the code breaks because it doesn't have code for it, nor a proper fallback. Such site usually break for very old browsers as well. In such cases you have to get the downloader to mimic a common browser to get it to work correctly.

John Gardeniers
  • 27,458
  • 12
  • 55
  • 109
  • Yes, after all I realize, everyone here was quite right. Once I tried the call with curl I finally got the feedback I needed to realize there was an early error from which the site was able to recover when viewed in a 'normal' browser. wget wasn't able to handle the error but curl output the stack trace. – markus Dec 15 '12 at 12:26