log in with browser and then ruby/mechanize takes it over?

Question

Is that even possible? what I need to pass to mechanize? With what url I can start then?

I cannot manage (so far) to log into one website using mechanize so I was thinking if I can do this little workaround. I believe I can capture all cookies and everything else and then pass it to ruby/mechanize to do the rest ...

screenshots below are made using firebug ( Firebug logs the POST or GET request, the response headers )

login that works = just one line

and html for login that works

        <script type="text/javascript" src="clientscript/vbulletin_md5.js?v=3612"></script>
                    <table cellpadding="0" cellspacing="1" border="0">
                    <tr>
                        <td class="smallfont" align="left"><label for="navbar_username">User Name</label></td>
                        <td class="smallfont" align="left" colspan="2"><label for="navbar_password">Password</label></td>
                    </tr>
                    <tr>

                        <td><input type="text" class="button" name="vb_login_username" id="navbar_username" size="10" accesskey="u" tabindex="101" value="User Name" onfocus="if (this.value == 'User Name') this.value = '';" /></td>        
                        <td><input type="password" class="button" name="vb_login_password" id="navbar_password" size="10" accesskey="p" tabindex="102" /></td>
                        <td class="smallfont" align="left" valign="middle"><input type="submit" class="button" value="Log in" tabindex="103" title="Enter your username and password in the boxes provided to login, or click the 'register' button to create a profile for yourself." accesskey="s" />
                            <label for="cb_cookieuser_navbar">
                            <input type="checkbox" name="cookieuser" value="1" tabindex="103" id="cb_cookieuser_navbar" accesskey="c" />Save?</label>
                            <input type="hidden" name="s" value="" />
<input type="hidden" name="securitytoken" value="1cbc0286417d97b4eb43ee0b0c2b54e7c615e5b8" />
                            <input type="hidden" name="do" value="login" />
                            <input type="hidden" name="vb_login_md5password" />

                            <input type="hidden" name="vb_login_md5password_utf" /></td>
                    </tr>
                    </table>
      </form>
      <!-- / login form -->

login that doesn't work for me

and it's html

<form class="login" method="post"> <fieldset>
<legend>Members Login</legend> 

<div>
<label for="auth_username">Username</label> <input id="auth_username" name="auth_username">
</div>

<div>
<label for="auth_password">Password</label> <input id="auth_password" name="auth_password" type="password">
</div>

</fieldset>
<div class="buttons">
<input name="auth_login" type="submit" value="Login"><p class="note"><a href="/forgotpassword">Forgot your password?</a></p>

</div>

</form>

my script is almost the same in both cases.

require 'rubygems'
require 'mechanize'
#agent = WWW::Mechanize.new

agent = WWW::Mechanize.new 

page = agent.get("http://www.vbulletin.org/forum/index.php")

login_form = page.form_with(:action => 'login.php?do=login')


puts 
login_form.fields.each { |f| puts "#{f.name} : #{f.value}" }    
login_form['vb_login_username'] = 'user name'
login_form['vb_login_password'] = ''

page = agent.submit login_form

output = File.open("login.html", "w") {|f| f.write(page.parser.to_html) }

mechanize log for the login that doesn't work

 INFO -- : Net::HTTP::Get: /login?auth_successurl=http://www.somedomain.com/forum/yota?baz_r=1
DEBUG -- : request-header: accept-language => en-us,en;q=0.5
DEBUG -- : request-header: connection => keep-alive
DEBUG -- : request-header: accept => */*
DEBUG -- : request-header: accept-encoding => gzip,identity
DEBUG -- : request-header: user-agent => WWW-Mechanize/0.9.3 (http://rubyforge.org/projects/mechanize/)
DEBUG -- : request-header: accept-charset => ISO-8859-1,utf-8;q=0.7,*;q=0.7
DEBUG -- : request-header: host => www.somedomain.com
DEBUG -- : request-header: keep-alive => 300
DEBUG -- : Read 400 bytes
DEBUG -- : Read 1424 bytes
DEBUG -- : Read 2448 bytes
DEBUG -- : Read 3211 bytes
DEBUG -- : response-header: vary => Accept-Encoding
DEBUG -- : response-header: cache-control => no-store, no-cache, must-revalidate, post-check=0, pre-check=0
DEBUG -- : response-header: connection => close
DEBUG -- : response-header: expires => Thu, 19 Nov 1981 08:52:00 GMT
DEBUG -- : response-header: content-type => text/html; charset=utf-8
DEBUG -- : response-header: date => Fri, 29 Jan 2010 23:43:12 GMT
DEBUG -- : response-header: content-encoding => gzip
DEBUG -- : response-header: server => Apache/2.2.3 (CentOS)
DEBUG -- : response-header: content-length => 3211
DEBUG -- : response-header: set-cookie => PHPSESSID=7cfilg86ju2ldcgso22246hpu4; path=/, WebStats:visitorId=lSMkcwuSWEE%3D; expires=Mon, 27-Jan-2020 23:43:12 GMT; path=/, WebStats:sessionId=%2B2HHK296t%2BQ%3D; expires=Mon, 27-Jan-2020 23:43:12 GMT; path=/
DEBUG -- : response-header: accept-ranges => bytes
DEBUG -- : response-header: pragma => no-cache
DEBUG -- : gunzip body
DEBUG -- : saved cookie: PHPSESSID=7cfilg86ju2ldcgso22246hpu4
DEBUG -- : saved cookie: WebStats:visitorId=lSMkcwuSWEE%3D
DEBUG -- : saved cookie: WebStats:sessionId=%2B2HHK296t%2BQ%3D
 INFO -- : status: 200
DEBUG -- : query: "auth_username=radek&auth_password=mypassword"
 INFO -- : Net::HTTP::Post: /login?auth_successurl=http://www.somedomain.com/forum/yota?baz_r=1
DEBUG -- : request-header: accept-language => en-us,en;q=0.5
DEBUG -- : request-header: connection => keep-alive
DEBUG -- : request-header: accept => */*
DEBUG -- : request-header: accept-encoding => gzip,identity
DEBUG -- : request-header: content-type => application/x-www-form-urlencoded
DEBUG -- : request-header: user-agent => WWW-Mechanize/0.9.3 (http://rubyforge.org/projects/mechanize/)
DEBUG -- : request-header: cookie => WebStats:sessionId=%2B2HHK296t%2BQ%3D; WebStats:visitorId=lSMkcwuSWEE%3D; PHPSESSID=7cfilg86ju2ldcgso22246hpu4
DEBUG -- : request-header: referer => http://www.somedomain.com/login?auth_successurl=http://www.somedomain.com/forum/yota?baz_r=1
DEBUG -- : request-header: accept-charset => ISO-8859-1,utf-8;q=0.7,*;q=0.7
DEBUG -- : request-header: content-length => 43
DEBUG -- : request-header: host => www.somedomain.com
DEBUG -- : request-header: keep-alive => 300
DEBUG -- : Read 650 bytes
DEBUG -- : Read 1674 bytes
DEBUG -- : Read 2698 bytes
DEBUG -- : Read 3211 bytes
DEBUG -- : response-header: vary => Accept-Encoding
DEBUG -- : response-header: cache-control => no-store, no-cache, must-revalidate, post-check=0, pre-check=0
DEBUG -- : response-header: connection => close
DEBUG -- : response-header: expires => Thu, 19 Nov 1981 08:52:00 GMT
DEBUG -- : response-header: content-type => text/html; charset=utf-8
DEBUG -- : response-header: date => Fri, 29 Jan 2010 23:43:13 GMT
DEBUG -- : response-header: content-encoding => gzip
DEBUG -- : response-header: server => Apache/2.2.3 (CentOS)
DEBUG -- : response-header: content-length => 3211
DEBUG -- : response-header: accept-ranges => bytes
DEBUG -- : response-header: pragma => no-cache
DEBUG -- : gunzip body
 INFO -- : status: 200

@Radek, is this a network flow? And what did you do different between the one that worked and the one that didn't? — Wayne Conrad, Jan 30 '10 at 06:03
@Wayne Conrad: I updated my question. I did not do anything different except that I use password field in the case when my script doesn't work. In the other one I use md5 hash of the password. I think the trouble is that the communication webserver <=> browser is more then one line and mechanize doesnt handle it. — Radek, Jan 30 '10 at 06:36

score 1 · Answer 1 · answered Jul 04 '11 at 17:37

1

Your problem is very likely from the fact that Mechanize only tracks cookies created with the Set-Cookie HTTP header. It cannot handle cookies created by JavaScript.

answered Jul 04 '11 at 17:37

iconoclast

21,213
15
102
138

score 0 · Accepted Answer · answered Jan 30 '10 at 04:13

0

Yes, capturing cookies (eg, through FireCookies Plugin in Firefox), and manually passing it to mechanize could work for most cases.

answered Jan 30 '10 at 04:13

YOU

120,166
34
186
219

@S.Mark: the question is WHAT I have to pass and how :-) see the picture above. – Radek Jan 30 '10 at 05:54
Hi, there is FireCookie extension for Firebug, https://addons.mozilla.org/ja/firefox/addon/6683, and here is screen shot about cookie tab https://addons.mozilla.org/en-US/firefox/images/p/18997/1204922922 , you need to pass those cookies into mechanize – YOU Jan 30 '10 at 06:24
@S.Mark:thank you for that. Now I am thinking what is the next step. I call new url from mechanize? But which one? Or I save the page after the successful login within my web browser and then load in my ruby scrip into `page object`? – Radek Jan 30 '10 at 06:48
sorry, please new question for `How to pass cookies to ruby/mechanize`, I am not good in that. – YOU Jan 30 '10 at 07:13
As an update, FireCookie is now included with the Firebug addon. – CodeBiker Aug 19 '13 at 18:50

log in with browser and then ruby/mechanize takes it over?

2 Answers2

Linked