How can I set a dynamic explicit wait using Selenium in Python?

Question

I got a little problem here with a program I built a few days ago, so I'm going to explain it very straightforward:

I'm scraping data from a page with my program, and for doing so, I set this Explicit Wait:

WebDriverWait(driver, 10).until(EC.visibility_of_element_located((By.XPATH,'/this_is_the_XPATH')))

For most of times, it works pretty well, however, there are sometimes in which the Explicit Wait above causes this error:

TimeoutException

Which makes my program stop and have to repeat a long process again.

I realized that the wait parameter set as 10 seconds is not always a good one, so I wanted to know from you guys if there's a way to set this parameter as a variable that always gets the exact load time (in seconds) for a page that is loaded completely.

Other more simple idea could be to repeat the process forcibly until the visibility of element is finally loaded, like by using try and exception blocks, but I don't know what should I type in the exception block to repeat the try one over and over until it's done.

As always, feedback is appreciated, thanks for reading in advance.

Tom McElhinney · Answer 1 · 2021-09-25T02:50:03.443

You are using the Expected Conditions function from Selenium. That initial wait of 10 seconds is simply setting the maximum time that it is allowed to wait, should the Expected Conditions function fail to get anything.

You may want to raise the 10 second max wait, if it keeps failing.

The wait will end early (ie. Not 10 seconds) if the Expected Conditions condition is met.

Thus the important part is to ensure the EC function is working as intended. I like to use element_to_be_clickable as opposed to your visibility_of_element_located, perhaps give that a try, (or others, full list of conditions here)

Try this:

from selenium.webdriver.support import expected_conditions as EC

wait = WebDriverWait(driver, 20) #set a max wait time
element = wait.until(EC.element_to_be_clickable((By.XPATH, 'some xpath'))) #set the dynamic wait, which should end the wait early (ie not 20 seconds)

Nandan A · Accepted Answer · 2021-09-25T02:27:24.160

Always write reusable functions for repetitive actions and write exception-handling for those functions.

Your code:

 WebDriverWait(driver, 10).until(EC.visibility_of_element_located((By.XPATH,'elementTxt')))

Create a reusable function:

def clickElement(elementTxt, driver, time) 
    try :

      element = WebDriverWait(driver, time).until(EC.visibility_of_element_located((By.XPATH,'/this_is_the_XPATH')))
      element.click()

     except ElementClickInterceptedException as e: 
        driver.execute_script ("arguments[0].click();",element)

     except TimeoutException as te
     element = WebDriverWait(driver, time).until(EC.element_to_be_clickable((By.XPATH,'/this_is_the_XPATH')))
     element.click()

Call the function: Now, you can call this clickElement function wherever you need to perform click action. You need to pass the xPath value, driver instance and time in seconds.

clickElement(xPathText, driver, 30)

Other more simple idea could be to repeat the process forcibly until the visibility of element is finally loaded

If we know element will not be visible then we shouldn't wait till it displays. It cause the code to stay in infinite loop but you can search for element one more time. If you see above example, in TimeoutException block we are searching for same element again if still element is not accessible then we need think other ways.

How can I set a dynamic explicit wait using Selenium in Python?

2 Answers2