How to programmatically log into website in Python

Question

I have searched all over the Internet, looking at many examples and have tried every one I've found, yet none of them are working for me, so please don't think this is a duplicate - I need help with my specific case.

I'm trying to log into a website using Python (in this instance I'm trying with v2.7 but am not opposed to using a more recent version, it's just I've been able to find the most info on 2.7).

I need to fill out a short form, consisting simply of a username and password. The form of the webpage I need to fill out and log in to is as follows (it's messy, I know):

<form method="post" action="login.aspx?ReturnUrl=..%2fwebclient%2fstorepages%2fviewshifts.aspx" id="Form1">
<div class="aspNetHidden">
<input type="hidden" name="__VIEWSTATE" id="__VIEWSTATE" value="/wEPDwUKMTU4MTgwOTM1NWRkBffWXYjjifsi875vSMg9OVkhxOQYYstGTNcN9/PFb+M=" />
</div>

<div class="aspNetHidden">

    <input type="hidden" name="__EVENTVALIDATION" id="__EVENTVALIDATION" value="/wEdAAVrmuRkG3j6RStt7rezNSLKVK7BrRAtEiqu9nGFEI+jB3Y2+Mc6SrnAqio3oCKbxYY85pbWlDO2hADfoPXD/5td+Ot37oCEEXP3EjBFcbJhKJGott7i4PNQkjYd3HFozLgRvbhbY2j+lPBkCGQJXOEe" />
</div>
            <div><span></span>
                <table style="BORDER-COLLAPSE: collapse" borderColor="#000000" cellSpacing="0" cellPadding="0"
                    width="600" align="center" border="1">
                    <tr>
                        <td>
                            <table cellSpacing="0" cellPadding="0" width="100%" align="center" border="0">
                                <tr>
                                    <td width="76%"><span id="centercontentTitle"></span>
                                        <H1 align="center"><br>
                                            <span>
                                                <IMG height="52" src="../images/logo-GMR.jpg" width="260"></span><span><br>
                                            </span></H1>
                                        <div id="centercontentbody">
                                            <div align="center">
                                                <TABLE width="350">
                                                    <TR>
                                                        <TD class="style7">Username:</TD>
                                                        <TD>
                                                            <div align="right"><input name="txtUsername" type="text" id="txtUsername" style="width:250px;" /></div>
                                                        </TD>
                                                    </TR>
                                                    <TR>
                                                        <TD class="style7">Password:</TD>
                                                        <TD>
                                                            <div align="right"><input name="txtPassword" type="password" id="txtPassword" style="width:250px;" /></div>
                                                        </TD>
                                                    </TR>
                                                    <TR>
                                                        <TD></TD>
                                                        <TD align="right"><input type="submit" name="btnSubmit" value="Submit" id="btnSubmit" /><input type="submit" name="btnCancel" value="Cancel" id="btnCancel" /></TD>
                                                    </TR>
                                                    <TR>
                                                        <TD colspan="2" align="center"></TD>
                                                    </TR>
                                                </TABLE>
                                            </div>
                                        </div>
                                    </td>
                                    <td>
                                        <div align="center" style='height:250px'></div>
                                    </td>
                                </tr>
                            </table>
                        </td>
                    </tr>
                </table>
                <br>
                <br>
                <p>&nbsp;</p>
        </form>

From searching around online, the best Python code I have found to fill out this form and log into the website is as follows:

Note: This is not my code, I got it from this question/example, where many people have said they've found it to work well.

import cookielib
import urllib
import urllib2


# Store the cookies and create an opener that will hold them
cj = cookielib.CookieJar()
opener = urllib2.build_opener(urllib2.HTTPCookieProcessor(cj))

# Add our headers
opener.addheaders = [('User-agent', 'LoginTesting')]

# Install our opener (note that this changes the global opener to the one
# we just made, but you can also just call opener.open() if you want)
urllib2.install_opener(opener)

# The action/ target from the form
authentication_url = '<URL I am trying to log into>'

# Input parameters we are going to send
payload = {
  '__EVENTVALIDATION': '/wEdAAVrmuRkG3j6RStt7rezNSLKVK7BrRAtEiqu9nGFEI+jB3Y2+Mc6SrnAqio3oCKbxYY85pbWlDO2hADfoPXD/5td+Ot37oCEEXP3EjBFcbJhKJGott7i4PNQkjYd3HFozLgRvbhbY2j+lPBkCGQJXOEe"',
  'txtUsername': '<USERNAME>',
  'txtPassword': '<PASSWORD>',
  }

# Use urllib to encode the payload
data = urllib.urlencode(payload)

# Build our Request object (supplying 'data' makes it a POST)
req = urllib2.Request(authentication_url, data)

# Make the request and read the response
resp = urllib2.urlopen(req)
contents = resp.read()

Unfortunately, this is not working for me and I'm unable to figure out why. If someone could please please please look over the code and tell me how I could improve it so as it works as it should. It would be so greatly appreciated!

Thanks in advance for all help I receive :)

I am not getting an error, the code is just not logging into the website. When I fetch data from the site after running the code, data from the login page is given- not from after the login thus how I know the login has failed. — zachs, May 07 '14 at 11:50

scandinavian_ · Answer 1 · 2014-05-08T12:19:44.703

0

__EVENTVALIDATION is probably not static, you need to load the login page in python, get the __EVENTVALIDATION field and then do the login.

Something like this should work:

import requests
from bs4 import BeautifulSoup

s = requests.session()

def get_eventvalidation():
    r = s.get("http://url.to.login.page")
    bs = BeautifulSoup(r.text)

    return bs.find("input", {"name":"__EVENTVALIDATION"}).attrs['value']

authentication_url = '<URL I am trying to log into>'

payload = {
  '__EVENTVALIDATION': get_eventvalidation(),
  'txtUsername': '<USERNAME>',
  'txtPassword': '<PASSWORD>',
  }

login = s.post(authentication_url, data=payload)

print login.text

You need the requests module and beautifulsoup4. Or you can just rewrite it to not use libraries.

Edit: You probably need __VIEWSTATE as a POST value.

edited May 08 '14 at 12:19

answered May 07 '14 at 11:30

scandinavian_

2,496
1
17
19

Thank you for your response! Sorry, but what do you mean by this and how would I go about it? Pardon my ignorance. – zachs May 07 '14 at 11:31
Thank you I will take a look at the updated answer and give that a go – zachs May 07 '14 at 11:51
Hi, I have tried implementing your solution, however have unfortunately had no luck :( I'm wondering if I am required to programmatically push the "submit" button when submitting the form, or is that done automatically? Thanks! – zachs May 08 '14 at 11:29
Also, I am able to confirm that __EVENTVALIDATION is indeed static, so that is not what is causing the issue here. Any further help would be greatly appreciated :-) – zachs May 08 '14 at 11:48
You probably need __VIEWSTATE as a POST value, I missed that the first time I looked at the HTML. – scandinavian_ May 08 '14 at 12:20

How to programmatically log into website in Python

1 Answers1

Linked