5


I would like to access a web page from a python program. I have to set up cookies to load the page.
I used the httplib2 library, but I didn't find how add my own cookie

resp_headers, content = h.request("http://www.theURL.com", "GET")

How can I create cookies with the right name and value, add it to the function and then load the page?
Thanks

Mac Fly
  • 359
  • 1
  • 7
  • 22

3 Answers3

14

From http://code.google.com/p/httplib2/wiki/Examples hope will help )

Cookies

When automating something, you often need to "login" to maintain some sort of session/state with the server. Sometimes this is achieved with form-based authentication and cookies. You post a form to the server, and it responds with a cookie in the incoming HTTP header. You need to pass this cookie back to the server in subsequent requests to maintain state or to keep a session alive.

Here is an example of how to deal with cookies when doing your HTTP Post.

First, lets import the modules we will use:

import urllib
import httplib2

Now, lets define the data we will need. In this case, we are doing a form post with 2 fields representing a username and a password.

url = 'http://www.example.com/login'   
body = {'USERNAME': 'foo', 'PASSWORD': 'bar'}
headers = {'Content-type': 'application/x-www-form-urlencoded'}

Now we can send the HTTP request:

http = httplib2.Http()
response, content = http.request(url, 'POST', headers=headers, body=urllib.urlencode(body))

At this point, our "response" variable contains a dictionary of HTTP header fields that were returned by the server. If a cookie was returned, you would see a "set-cookie" field containing the cookie value. We want to take this value and put it into the outgoing HTTP header for our subsequent requests:

headers['Cookie'] = response['set-cookie']

Now we can send a request using this header and it will contain the cookie, so the server can recognize us.

So... here is the whole thing in a script. We login to a site and then make another request using the cookie we received:

#!/usr/bin/env python

import urllib
import httplib2

http = httplib2.Http()

url = 'http://www.example.com/login'   
body = {'USERNAME': 'foo', 'PASSWORD': 'bar'}
headers = {'Content-type': 'application/x-www-form-urlencoded'}
response, content = http.request(url, 'POST', headers=headers, body=urllib.urlencode(body))

headers = {'Cookie': response['set-cookie']}

url = 'http://www.example.com/home'   
response, content = http.request(url, 'GET', headers=headers)
sultan
  • 5,978
  • 14
  • 59
  • 103
7
http = httplib2.Http()
# get cookie_value here
headers = {'Cookie':cookie_value}
response, content = http.request("http://www.theURL.com", 'GET', headers=headers)

You may want to add another header parameters to specify another HTTP request parameters.

khachik
  • 28,112
  • 9
  • 59
  • 94
0

You can also use just urllib2 library

        import urllib2

        opener = urllib2.build_opener()
        opener.addheaders.append(('Cookie', 'cookie1=value1;cookie2=value2'))
        f = opener.open("http://www.example.com/")
        the_page_html = f.read()
Alberto Perez
  • 1,019
  • 15
  • 17