no such element: Unable to locate element using chromedriver and Selenium in production environment

Question

I have a problem with selenium chromedriver which I cannot figure out what's causing it. Some weeks ago everything was working OK, and suddenly this error started to show up. The problem is coming from the following function.

 def login_(browser):
    try:
        browser.get("some_url")
        # user credentials
        user = browser.find_element_by_xpath('//*[@id="username"]')
        user.send_keys(config('user'))
        password = browser.find_element_by_xpath('//*[@id="password"]')
        password.send_keys(config('pass'))
        login = browser.find_element_by_xpath('/html/body/div[1]/div/button')
        login.send_keys("\n")
        time.sleep(1)
        sidebar = browser.find_element_by_xpath('//*[@id="sidebar"]/ul/li[1]/a')
        sidebar.send_keys("\n")
        app_submit = browser.find_element_by_xpath('//*[@id="sidebar"]/ul/li[1]/ul/li[1]/a')
        app_submit.send_keys("\n")
    except TimeoutException or NoSuchElementException:
        raise LoginException

This function works with no problem in the development environment (macOS 10.11), but throws the following error in the production environment:

Message: no such element: Unable to locate element: {"method":"xpath","selector":"//*[@id="sidebar"]/ul/li[1]/a"}
(Session info: headless chrome=67.0.3396.79)
(Driver info: chromedriver=2.40.565383 (76257d1ab79276b2d53ee97XXX),platform=Linux 4.4.0-116-generic x86_64)

I already updated both Chrome and chromedriver (v67 & 2.40, respectively) in each environment. I also gave it more time.sleep(15). But the problem persists. My latest guess is that maybe the initialization of the webdriver is not working properly:

def initiate_webdriver():
   option = webdriver.ChromeOptions()
   option.binary_location = config('GOOGLE_CHROME_BIN')
   option.add_argument('--disable-gpu')
   option.add_argument('window-size=1600,900')
   option.add_argument('--no-sandbox')
   if not config('DEBUG', cast=bool):
       display = Display(visible=0, size=(1600, 900))
       display.start()
       option.add_argument("--headless")
   else:
       option.add_argument("--incognito")
   return webdriver.Chrome(executable_path=config('CHROMEDRIVER_PATH'), chrome_options=option)

Because, if the Display is not working, then there may not be the mentioned sidebar but some other button.

So my questions are: does anybody have had a similar issue? Is there a way to know what is the page showing at the time the driver is looking for such an element?

Is it possible that the HTML is different between the two environments? Have you verified that the same locator works in both manually? — JeffC, Jun 20 '18 at 21:26
@JeffC This locator was working alright a few weeks ago and the html code has not changed. — Melquíades Ochoa, Jun 20 '18 at 22:37
@Prany, I no longer think it is a matter of time. Adding some extra time didn't sove the problem. — Melquíades Ochoa, Jun 20 '18 at 22:39
Try checking the HTML of the website. If it was working before, but all of a sudden stop working now it's most likely that. You could also try finding the stuff by id, class name, etc. The error means that the element might have got renamed, not in there period, or got moved away from the destination you specified. — Kode, Jun 22 '18 at 18:31
provide the production URL page. We need to assert locator validity. — Manmohan_singh, Jun 25 '18 at 03:08
@MelquíadesOchoa, have you tried GovindM's answer https://stackoverflow.com/a/50994132/8329042 ? — Abhilash, Jun 25 '18 at 14:22
And this is, kids, why you should never ever use xpath like this `//*[@id="sidebar"]/ul/li[1]/a`. It doesn't tell you what you click or why it fails... It's useless! Meanwhile you have an `a` element, it will *always* have a unique property or value to refer to. And if it's not found at least you will know it's not because of "html structure" — timbre timbre, Jun 25 '18 at 17:33
Also try waiting for that element instead of finding it, may just be loading slower on another environment — timbre timbre, Jun 25 '18 at 17:34

score 4 · Accepted Answer · edited Jun 25 '18 at 05:01

It's report that the element not found error after you supplying the login , so I think the login failed and the page redirected to somewhere. You can use screenshot option to take a screenshot of the page and then see which page the driver load.

driver.save_screenshot("path to save screen.jpeg")

Also you can save the raw html code and inspect the same page.

Webdriver Screenshot

Using Selenium in Python to save a webpage on Firefox

undetected Selenium · Answer 2 · 2018-07-06T15:37:42.233

A couple of things as per the login_(browser) method:

As you have identified the Login button through:
```
login = browser.find_element_by_xpath('/html/body/div[1]/div/button')
```
I would suggest rather invoking send_keys("\n") take help of the onclick() event through login.click() to mock the clicking of Login button as follows:
```
login = browser.find_element_by_xpath('/html/body/div[1]/div/button')
login.click()
```

Next when you identify the sidebar induce WebDriverWait for the element to be clickable as follows:

WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, '//*[@id="sidebar"]/ul/li[1]/a'))).click()

As you mentioned your code code block works perfect in macOS 10.11 environment but throws the following error in the production environment (Linux) it is highly possible that different browsers renders the HTML DOM differently in different OS architecture. So instead of absolute xpath you must use relative xpath as follows:
```
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//a[@attribute='value']"))).click()
```

A couple of things as per the initiate_webdriver() method:

As per Getting Started with Headless Chrome the argument --disable-gpu is applicable only for Windows but not a valid configuration for Linux OS. So need o remove:
```
option.add_argument('--disable-gpu')
```

Note : You have to add the following imports :

from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC

This example needs the following imports in order for it to work: ```from selenium.webdriver.support.ui import Select, WebDriverWait``` ```from selenium.webdriver.support import expected_conditions as EC``` ```from selenium.webdriver.common.by import By``` — Melquíades Ochoa, Jul 04 '18 at 21:09
Thanks @MelquíadesOchoa , you were right, I have added the exports within the answer. — undetected Selenium, Jul 06 '18 at 15:38

score 2 · Answer 3 · answered Jun 25 '18 at 06:16

Whenever I encounter strange issues in Selenium like this, I prefer retrying to find the particular element which is causing intermittent troubles. One way is to wrap it around a try-except block:

try:
   sidebar = browser.find_element_by_xpath('//*[@id="sidebar"]/ul/li[1]/a')
except NoSuchElementException:
   time.sleep(10)
   print("Unable to find element in first time, trying it again")
   sidebar = browser.find_element_by_xpath('//*[@id="sidebar"]/ul/li[1]/a')

You could also put the try code in a loop with a suitable count variable to make the automation code work. (Check this). In my experience with JAVA, this idea has resolved multiple issues.

score 2 · Answer 4 · answered Jun 28 '18 at 15:39

You need to wait until the element is visible or else you will get this error. Try something like this:

from selenium.webdriver.support.wait import WebDriverWait
from selenium.webdriver.support.expected_conditions import visibility_of_element_located
from selenium.webdriver.common.by import By
TIMEOUT = 5

...
xpath = '//*[@id="sidebar"]/ul/li[1]/a'
WebDriverWait(self.selenium, TIMEOUT).until(visibility_of_element_located((By.XPATH, xpath)))
browser.find_element_by_xpath(xpath)
...

no such element: Unable to locate element using chromedriver and Selenium in production environment

4 Answers4

Linked

Related