How do I load session and cookies from Selenium browser to requests library in Python?

Question

How can I load session and cookies from Selenium browser? The following code:

import requests

cookies = [{u'domain': u'academics.vit.ac.in',
            u'name': u'ASPSESSIONIDAEQDTQRB',
            u'value': u'ADGIJGJDDGLFIIOCEZJHJCGC',
            u'expiry': None, u'path': u'/',
            u'secure': True}]
response = requests.get(url2, cookies=cookies)

gives me the following exception:

Traceback (most recent call last):
  File "F:\PYTHON\python_scripts\cookies\cookies3.py", line 23, in <module>
    response = requests.get(url2, cookies=cookies)
  File "C:\Python27\lib\site-packages\requests\api.py", line 68, in get
    return request('get', url, **kwargs)<br/>
  File "C:\Python27\lib\site-packages\requests\sessions.py", line 450, in request
    prep = self.prepare_request(req)
    cookies = cookiejar_from_dict(cookies)
  File "C:\Python27\lib\site-packages\requests\cookies.py", line 439, in cookiejar_from_dict
    cookiejar.set_cookie(create_cookie(name, cookie_dict[name]))
TypeError: list indices must be integers, not dict

@That1Guy am using selenium to overcome captcha based login. then onwards i need to parse more than 1000 urls for xml info. But if i use selenium i will have to load the page using browser, but i need to do it in background only — Cibin, Apr 10 '15 at 14:02

That1Guy · Accepted Answer · 2019-12-11T18:10:44.613

92

First you have to get the cookies from your driver instance:

cookies = driver.get_cookies()

This returns cookie dictionaries for your session.

Next, set those cookies in requests:

s = requests.Session()
for cookie in cookies:
    s.cookies.set(cookie['name'], cookie['value'])

edited Dec 11 '19 at 18:10

answered Apr 10 '15 at 14:12

That1Guy

7,075
4
47
59

getting error in this line s.cookies.set(**cookie) TypeError: create_cookie() got unexpected keyword arguments: [u'expiry'] the cookie when printed is like this [{u'domain': u'acemics.it.ac.in', u'secure': True, u'value': u'PBHHJGJDHMSEEHGCDJLICARE', u'expiry': None, u'path': u'/', u'name': u'ASPSESSIONIDAEQDTQRB'}] – Cibin Apr 10 '15 at 17:26
2

error in line a = requests.get(url2, cookies=cookies) TypeError: list indices must be integers, not dict – Cibin Apr 10 '15 at 17:59
2

That1Guy, most probably the downvotes are due to the fact you don't care about extra parameters like http only, path etc. Though, it's just an educated guess, you know. – Konstantin Apr 20 '17 at 11:14
1

@MichaelCGood I exported cookies from selenium driver first. It was then onboarded to requests session object before making the next request. – Cibin Oct 01 '18 at 06:22
Can you help me with reverse of this,that is from request to selenium, I am trying to, but selenium always seems to start a fresh session :( – Pritish Dec 11 '19 at 10:42
I am not able to get this to work, e.g. if I take a website with consent cookies and I open it in selenium then pull the cookies and inject them into requests then they do not work and have not impact (I tried with Mashable as it requires consent from EU users) – no nein Jan 13 '20 at 19:37
@nonein Are you certain the cookies are being set? I'd set up some print statements so you can track which cookies are received and which are set. – That1Guy Jan 14 '20 at 14:38

score 5 · Answer 2 · answered Jan 21 '20 at 14:49

I made this solution, it's solved unexpected keyword arguments expiry

def set_cookies(cookies, s):
    for cookie in cookies:
        if 'httpOnly' in cookie:
            httpO = cookie.pop('httpOnly')
            cookie['rest'] = {'httpOnly': httpO}
        if 'expiry' in cookie:
            cookie['expires'] = cookie.pop('expiry')
        s.cookies.set(**cookie)
    return s

score 4 · Answer 3 · answered Apr 10 '20 at 13:42

4

You can use 3rd party package like selenium-requests or requestium. They provide function to share cookies between requests and selenium.

answered Apr 10 '20 at 13:42

Jing He

794
1
9
17

score 1 · Answer 4 · answered Mar 27 '22 at 08:07

I think this doesn't directly answer the question but if your goal is to make requests pretending to be the same session that selenium is, you can do this:

resp = browser.execute_script(f'''
    var xmlHttp = new XMLHttpRequest();
    xmlHttp.open( "GET", "{yourUrl}", false );
    xmlHttp.send( null );
    return xmlHttp.responseText;
'''
)

How do I load session and cookies from Selenium browser to requests library in Python?

4 Answers4

Linked