wget: download a website which is password protected using a form

Question

I have a problem with downloading a website using wget.

First I need to login to the website with a username and password:

wget --save-cookies cookies.txt --post-data --cookies=on --keep-session-cookies 
     --post-data="strUser=username&strPW=password" 
     http://hostname/wwwawelle/member/nixlog.asp

Then I try to download the complete website http://hostname/wwwawelle/member/ :

wget -k -r -m -np --load-cookies cookies.txt 
     http://hostname/wwwawelle/member/80_01_member.asp

I get a response saying that I am not logged in then it redirects me to:

http://hostname/wwwawelle/member/nixlog.asp?grund=notloggedin

Any ideas?

HttpFox

00:00:07.712    0.193   597 301 POST    302 Redirect to: 80_01_member.asp   http://hostname/wwwawelle/member/login.asp
00:00:07.943    0.255   481 11957   GET 200 text/html   http://hostname/wwwawelle/member/80_01_member.asp
00:00:08.232    0.060   428 (6054)  GET (Cache) text/css    http://hostname/wwwawelle/style.css
00:00:08.260    0.051   420 (589)   GET (Cache) application/x-javascript    http://hostname/wwwawelle/news/jscr_inc.js

Cookies

ASPSESSIONIDCSATRCQQ    APPIOGADBMLHOMFHCALPBFNL    /   hostname    End Of Session

POST Data

strUser username
strPW   password
Abschicken  Anmelden

wget cookies.txt

# HTTP cookie file.
# Generated by Wget on 2011-03-28 14:23:17.
# Edit at your own risk.

hostname    FALSE   /   FALSE   0   ASPSESSIONIDCSATRCQQ    EBAJOGADDHGHCANLKJCMDCPP

Is there any javascript on the site that might be getting in the way? Also could the site be checking the REFERER? — Kev, Mar 28 '11 at 09:29
After logging-in with wget, do the contents of `cookies.txt` look sensible? — nickgrim, Mar 28 '11 at 09:43

score 2 · Answer 1 · answered Mar 28 '11 at 11:08

2

If the site is saving the session with cookies you can log on using a browser, then have wget use the appropriate cookie to continue accessing the site. However, if the site uses other means, such as PHP $_SESSION variables you're out of luck.

If this is a site to which you have legitimate access surely you can configure other access means. On the other hand, if you don't have legitimate access and are merely trying to leach someone else's site, which is the impression I get from your question, I hope you fail.

answered Mar 28 '11 at 11:08

John Gardeniers

27,458
12
55
109

Re PHP session variables, I think that any data in $_SESSION will rely on a session-id stored in a cookie (or optionally in the URL) so that might not be a problem – grojo Mar 28 '11 at 11:51
@grojo, that cookie is in memory, not in a file, so it's kind of hard to pass on to wget. – John Gardeniers Mar 28 '11 at 11:59
the session data is on the server, however the session ID is stored in the cookie, and AFAIK wget _will_ save it to disk if told to. One possible tip is to GET the page with the form first to aquire/initialize any session data, before attempting to POST login data to the form URL. – grojo Mar 28 '11 at 12:10

score 1 · Answer 2 · edited Jul 23 '18 at 16:16

1

It could be that the site is checking referer, user-agent, etc.

Check out the HttpFox and Tamper Data Firefox plugins. They let you see/manipulate data going to/from any server. Make sure you note any redirects and cookies coming from the server.

edited Jul 23 '18 at 16:16

Peter Mortensen

2,318
5
23
24

answered Mar 28 '11 at 10:43

grojo

429
1
7
18

score -1 · Answer 3 · edited Mar 28 '11 at 11:22

-1

Try this before the URL:

wget username:password@http://me.com/me.html

edited Mar 28 '11 at 11:22

jscott

24,484
8
79
100

answered Mar 28 '11 at 09:24

Jonathan Ross

2,183
11
14

3

Looks like the site is password protected using a form, not basic auth. – Kev Mar 28 '11 at 09:27
1

FYI: The form for basic auth is http://username:password@example.com/me.html – Sean Reifschneider Mar 28 '11 at 09:40

wget: download a website which is password protected using a form

3 Answers3