27

I've been trying to scrape some information of this real estate website with selenium. However when I access the website I need to accept cookies to continue. This only happens when the bot accesses the website, not when I do it manually. When I try to find the corresponding element either by xpath or id, as I find it when I inspect the page manually I get following error.

selenium.common.exceptions.NoSuchElementException: Message: no such element: Unable to locate element: {"method":"xpath","selector":"//*[@id="uc-btn-accept-banner"]"}

from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC



PATH = "/usr/bin/chromedriver"
driver = webdriver.Chrome(PATH)

driver.get("https://www.immoweb.be/en/search/house/for-sale?countries=BE&page=1&orderBy=relevance")
driver.find_element_by_xpath('//*[@id="uc-btn-accept-banner"]').click()

Does anyone know how to get around this? Why can't I find the element?

Below is an image of the accept cookies popup.

enter image description here

This is the HTML corresponding to the button "Keep browsing". The XPATH is as above.

<button aria-label="Keep browsing" id="uc-btn-accept-banner" class="uc-btn-new  uc-btn-accept">Keep browsing
          <span id="uc-optin-timer-display"></span></button>
Jakob Bull
  • 283
  • 1
  • 3
  • 6
  • Is the popup really an alert? Or is it elements in the DOM that are made visible and positioned withe some combination of JavaScript and CSS? – Code-Apprentice Sep 23 '20 at 16:25
  • Interesting point, I think you could be onto something, however when I look for the matching elements by id within my window I can't find them. As a result I thought they must be in a different window. How would I then approach this issue? – Jakob Bull Sep 23 '20 at 16:32
  • 1
    Can you show the popup? I opened the site and I didn't get any popups (I waited for more than 1 minute) – Ahmed I. Elsayed Sep 23 '20 at 16:34
  • My suggestion to debug this is to first be sure your selenium driver is not running headless. Then use your favorite debugger to set a breakpoint after your `driver.get()` call. Now you can inspect the page directly in Chrome to discover how to access the popup. – Code-Apprentice Sep 23 '20 at 16:53
  • You don't need to switch. What you see is a modal. Just click the button. If you remove the `alert = driver.switch_to.alert` part, your code works. – Trapli Sep 23 '20 at 19:34
  • Thanks for your feedback. I had tried this before. I will edit my question and describe the problem as it is now, would be great to get some more feedback please. – Jakob Bull Sep 24 '20 at 13:31

1 Answers1

19

You were very close!

If you open your page in a new browser you'll note the page fully loads, then, a moment later your popup appears.

The default wait strategy in selenium is just that the page is loaded. That draw delay between page loaded and display appearing is causing your scripts to fail.

You have two good synchronisation options.

1/ Include an implicit wait for your driver. This is done once per script and affects all objects. This waits 10 seconds before throwing any error, or continues when it's ready:

PATH = "/usr/bin/chromedriver"
driver = webdriver.Chrome(PATH)
driver.implicitly_wait(10)

driver.get("https://www.immoweb.be/en/search/house/for-sale?countries=BE&page=1&orderBy=relevance")
driver.find_element_by_xpath('//*[@id="uc-btn-accept-banner"]').click()

2/ Do a explicit wait on your object only:

PATH = "/usr/bin/chromedriver"
driver = webdriver.Chrome(PATH)

driver.get("https://www.immoweb.be/en/search/house/for-sale?countries=BE&page=1&orderBy=relevance")
WebDriverWait(driver, 10).until(EC.element_to_be_clickable((By.XPATH,'//*[@id="uc-btn-accept-banner"]'))).click()

More info on the wait strategies is here

RichEdwards
  • 3,423
  • 2
  • 6
  • 22