not getting all cookie info using python requests module

Question

I'm learning how to login to an example website using python requests module. This Video Tutorial got me started. From all the cookies that I see in GoogleChrome>Inspect Element>NetworkTab, I'm not able to retrieve all of them using the following code:

import requests
with requests.Session() as s:
    url = 'http://www.noobmovies.com/accounts/login/?next=/'
    s.get(url)
    allcookies = s.cookies.get_dict()
    print allcookies

Using this I only get csrftoken like below:

{'csrftoken': 'ePE8zGxV4yHJ5j1NoGbXnhLK1FQ4jwqO'}

But in google chrome, I see all these other cookies apart from csrftoken (sessionid, _gat, _ga etc): Google Chrome ScreenShot

I even tried the following code from here, but the result was the same:

from urllib2 import Request, build_opener, HTTPCookieProcessor, HTTPHandler
import cookielib

#Create a CookieJar object to hold the cookies
cj = cookielib.CookieJar()
#Create an opener to open pages using the http protocol and to process cookies.
opener = build_opener(HTTPCookieProcessor(cj), HTTPHandler())

#create a request object to be used to get the page.
req = Request("http://www.noobmovies.com/accounts/login/?next=/")
f = opener.open(req)

#see the first few lines of the page
html = f.read()
print html[:50]

#Check out the cookies
print "the cookies are: "
for cookie in cj:
    print cookie

Output:

<!DOCTYPE html>
<html xmlns="http://www.w3.org
the cookies are: 
<Cookie csrftoken=ePE8zGxV4yHJ5j1NoGbXnhLK1FQ4jwqO for www.noobmovies.com/>

So, how can I get all the cookies ? Thanks.

are any of those request cookies added by google? how could you get the cookies sent with the request from the response? — dm03514, Apr 28 '15 at 19:53
@dm03514, I'm not sure. I inspected the same in a safari browser (which supports yahoo search) and got back the same list of cookies. Can you elaborate on the second part that you stated ? — Aks, Apr 28 '15 at 19:59

score 5 · Accepted Answer · answered Feb 02 '16 at 00:55

The cookies being set are from other pages/resources, probably loaded by JavaScript code. You can check it making the request to the page only (without running the JS code), using tools such as wget, curl or httpie.

The only cookie this server set is csrftoken, as you can see in:

$ wget --server-response 'http://www.noobmovies.com/accounts/login/?next=/'
--2016-02-01 22:51:55--  http://www.noobmovies.com/accounts/login/?next=/
Resolving www.noobmovies.com (www.noobmovies.com)... 69.164.217.90
Connecting to www.noobmovies.com (www.noobmovies.com)|69.164.217.90|:80... connected.
HTTP request sent, awaiting response... 
  HTTP/1.1 200 OK
  Server: nginx/1.4.6 (Ubuntu)
  Date: Tue, 02 Feb 2016 00:51:58 GMT
  Content-Type: text/html; charset=utf-8
  Transfer-Encoding: chunked
  Connection: keep-alive
  Vary: Accept-Encoding
  Expires: Tue, 02 Feb 2016 00:51:58 GMT
  Vary: Cookie,Accept-Encoding
  Cache-Control: max-age=0
  Set-Cookie: csrftoken=XJ07sWhMpT1hqv4K96lXkyDWAYIFt1W5; expires=Tue, 31-Jan-2017 00:51:58 GMT; Max-Age=31449600; Path=/
  Last-Modified: Tue, 02 Feb 2016 00:51:58 GMT
Length: unspecified [text/html]
Saving to: ‘index.html?next=%2F’

index.html?next=%2F             [      <=>                                   ]  10,83K  2,93KB/s    in 3,7s    

2016-02-01 22:52:03 (2,93 KB/s) - ‘index.html?next=%2F’ saved [11085]

Note the Set-Cookie line.

not getting all cookie info using python requests module

1 Answers1

Linked