Selenium: driver.get_cookies() returns incomplete list of cookies

Question

The cookies returned are as follows:

[
    {
        "domain": "www.carid.com",
        "name": "uxatc",
        "value": "%13%18%07%13%0D%07%13J%07%14J%07%0DJ%07%13%14%07%13I%07%12%13%0F%09%0E%1E%07%13%1F%1D%1A%17%08%1E%07%14%0F%1D%1A%17%08%1E%07%08%17%0F%09%0E%1E%07%16%16%07%16%1F%07%0F%08%07%0F%16%07%10%1F%07%15%16%07%0D%08%0D%12%08%12%19%17%1E%07%0C%10%07%1F%1F%07%1F%10%07%0C%1F%07%15%0C%1D%1A%17%08%1E%07%1F%19%07%03%02%07%08%18%07%0FJJNMMINOJLJCLK%07%0FI%07",
        "path": "/",
        "httpOnly": false,
        "secure": false
    },
    {
        "domain": ".carid.com",
        "name": "xid",
        "value": "21f70e8bba820dcaf620307b1717f90c",
        "path": "/",
        "httpOnly": true,
        "secure": false
    },
    {
        "domain": ".carid.com",
        "name": "xidRes",
        "value": "21f70e8bba820dcaf620307b1717f90c",
        "expiry": 1567463776,
        "path": "/",
        "httpOnly": true,
        "secure": false
    },
    {
        "domain": "www.carid.com",
        "name": "store_language",
        "value": "US",
        "expiry": 1597790176,
        "path": "/",
        "httpOnly": false,
        "secure": true
    },
    {
        "domain": "www.carid.com",
        "name": "uxat",
        "value": "%13%18K%07%0DJ%07%13J%1D%1A%17%08%1E%07%14J%1D%1A%17%08%1E%07%13%1F%1D%1A%17%08%1E%07%14%0F%1D%1A%17%08%1E%07%08%17%0F%09%0E%1E%07%16%16%07%16%1F%07%0F%08%07%0F%16%07%10%1F%07%15%16%07%0D%08%0D%12%08%12%19%17%1E%07%0C%10%07%1F%1F%07%1F%10%07%0C%1F%07%15%0C%1D%1A%17%08%1E%07%1F%19%07%13IK%07",
        "path": "/",
        "httpOnly": false,
        "secure": false
    },
    {
        "domain": "www.carid.com",
        "name": "uxid2",
        "value": "16fc88Ve3eb83cfbccGc5",
        "path": "/",
        "httpOnly": false,
        "secure": false
    },
    {
        "domain": "www.carid.com",
        "name": "uxid",
        "value": "8a58hVe3eb83cfc13Ym8",
        "path": "/",
        "httpOnly": false,
        "secure": false
    }
]

There are 7 cookies, but the browser has 17 cookies:

Why are there missing cookies? Is there a way to get all the cookies?

I think that selenium can get only the cookies of the current domain. — Kostas Charitidis, Aug 20 '19 at 07:34
They are third party analytics cookies to trace users activity. Either the automated setup blocks them or the page detects the automation and skips them. — Florent B., Dec 12 '20 at 12:45

score 1 · Answer 1 · answered Dec 15 '20 at 05:34

Possibly, because you didn't wait for the page to fully load before getting the cookies. I checked the site and observe the cookies status. Before the page fully loads on car selecting divs, there are less cookies than after the page is fully loaded.

Even if the driver thinks that the page is fully loaded, maybe it's not. It might run some background scripts. It's best to check for it before you want to do anything.

Working example:

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
opt = webdriver.FirefoxOptions()
# opt.headless = True
driver = webdriver.Firefox(options=opt)
driver.delete_all_cookies()
driver.get("https://carid.com")
cookies_before = driver.get_cookies()
WebDriverWait(driver, 15).until(
        EC.visibility_of_element_located(
                (By.XPATH, '//div[contains(@class, "select-vehicle-button")]')
        ))
print("Loaded!")
cookies_after = driver.get_cookies()
print(bool(cookies_before == cookies_after))
print(f"Cookies before: {len(cookies_before)}.")
print(f"Cookies after: {len(cookies_after)}.")
print(cookies_after)
driver.quit()

Output when before & after are different (only sometimes):

Loaded!
False
Cookies before: 5.
Cookies after: 17.

Notes: careful if you're running this in headless. Chrome headless will not work on this site, Firefox headless will.

score 0 · Answer 2 · answered Aug 20 '19 at 07:44

0

I think you should probably use requests module to get all the cookies you want to see.

with requests.Session() as s:
      s.get('website.com')
      print(s.cookies)

if it doesn't, try

print(s.cookies._cookies)

answered Aug 20 '19 at 07:44

Resul Bulbul

45
1
10

What if you need to render javascript? – User Dec 12 '20 at 00:48
1

try driver.manage().getCookies(); more in https://stackoverflow.com/questions/24912204/selenium-webdriver-how-to-get-all-cookies-in-the-current-page/24920215 – Resul Bulbul Dec 13 '20 at 20:22

score 0 · Answer 3 · answered Dec 17 '20 at 08:15

get_cookies()

get_cookies() method returns a set of dictionaries, corresponding to cookies visible in the current session.

cookies = driver.get_cookies()

This usecase

As the AUT (application under test) is JavaScript based, while storing the cookies you need to you need to induce WebDriverWait for some of the JavaScript enabled element to turn element_to_be_clickable() and then collect the cookies.

Further, cookies contains a lot more information including domain, name, value, httpOnly, secure etc. Hence a single key/value pair may not suffice. However, you can construct your own dictionary with the name and its corresponding value of the cookies.

Demonstration

As an example:

Code Block:

# -*- coding: utf-8 -*-
from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC

driver = webdriver.Firefox(executable_path=r'C:\WebDrivers\geckodriver.exe')
driver.get("https://carid.com")
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "div.select-vehicle-button")))
cookies_list = driver.get_cookies()
print("*****")
print(f"Cookies list: {len(cookies_list)}")
print("*****")
cookies_dict = {}
for cookie in cookies_list:
    cookies_dict[cookie['name']] = cookie['value']
print(cookies_dict)

Console Output:

*****
Cookies list: 16
*****
{'xid': '7c85b0b99fb3dcf885d655c31a8fda40', 'store_language': 'US', 'uxat': '%13%18K%07%0DJ%07%13J%1D%1A%17%08%1E%07%14J%1D%1A%17%08%1E%07%13%1F%1D%1A%17%08%1E%07%14%0F%1D%1A%17%08%1E%07%08%17%0F%09%0E%1E%07%16%16%07%16%1F%07%0F%08%07%0F%16%07%10%1F%07%15%16%07%0D%08%0D%12%08%12%19%17%1E%07%0C%10%07%1F%1F%07%1F%10%07%0C%1F%07%15%0C%0F%09%0E%1E%07%1F%19%07%13IK%07', 'uxid2': 'k8e2gUea05cdacf54K17d', 'uxid': '1ecehjYea05cdad009Q12n', '_gaexp': 'GAX1.2.pKl-9WqUR9OGPHEwCLKksQ.18690.1', '_ga': 'GA1.2.2081848464.1608191540', '_gid': 'GA1.2.1590349607.1608191540', '_gat': '1', '_uetsid': 'cb51ab90403c11eba41c7dda4c599643', '_uetvid': 'cb524450403c11eb89d6b3285069b4af', '__gads': 'ID=80ccbeefc3c87d2b-22e1ddad3cc5008a:T=1608191860:RT=1608191860:S=ALNI_MYhnP8XRpcV3S7HM2qf0XPr8SRRaQ', 'RefererCookie': 'https%3A%2F%2Fwww.carid.com%2F', '_fbp': 'fb.1.1608191549984.531722481', 'xidRes': '7c85b0b99fb3dcf885d655c31a8fda40', 'uxid3': '83g8jSea05ce07dddQ117'}

Observation

This count of 16 cookies exactly matches with the count of cookies observed within the Firefox browser.

Browser Snapshot:

score 0 · Answer 4 · answered Aug 07 '23 at 15:40

For anyone still encountering this issue despite waiting for the page to load properly.

driver.get_cookies() can only get cookies from the domain currently loaded in the browsers URL-bar (and it's subdomains).

In my case I managed to get past this issue by temporarily directing my browsers to the domain belonging to the specific cookie I wanted to use.

driver.get("https://url.of.the.cookie")

driver.get_cookies()

This allowed me to get and examine those remaining, previously unlisted, cookies.

Special thanks to Titus Fortner over at Selenium for asking the relevant question.

Selenium: driver.get_cookies() returns incomplete list of cookies

4 Answers4

get_cookies()

This usecase

Demonstration

Observation