2

I have a script that uses a selenium webdriver (geckodriver) and loads various webpages. The scripts works fine at the beginning, but than at a random point it stops working without raising any error (the program sorts of hangs without really doing anything).

I added some logging statement to check when it hangs, and this is caused by the WebDriverWait statement (see below). The last thing that is printed in the log is "get_records - Loaded".

The expected behavior to me would be to either print "get_records - Acquired pager", or to raise a TimeoutException after 10 seconds.

[...]
logging.info("get_records - Getting url: {}".format(url))
driver.get(url)
logging.info("get_records - Loaded")
# Get records number and result pages
elem = WebDriverWait(driver, 10).until(ec.element_to_be_clickable(
    (By.XPATH, "//td[@align='right']/span[@class='pager']"))
)
logging.info("get_records - Acquired pager")
[...]

Python version: 3.7.3 Selenium version: 3.141.0 Firefox version: 70.0.1

It seems like a similar bug happened with previous version (Selenium WebDriver (2.25) Timeout Not Working), but that bug was closed.

Is anyone having the same issue?

Update:

It seems like adding time.sleep(0.5) before elem prevents the script from freezing (either "get_records - Acquired pager" is printed, or the timeoutException is raised). Even though this is a turnaround for the issue, I would rather not put any forced wait.

d_d
  • 46
  • 7

1 Answers1

2

I actually have the exactly same experience when the script works fine at first but hangs forever after some time. The '10 seconds' timeout is that webdriver/browser tries to open a page in 10 seconds. But the timeout that python script sends request to webdriver/browser is not defined. And it's none by default meaning request will wait infinitely.

Short answer:

driver.command_executor.set_timeout(10)
driver.get(url)

Explain:

Chromedriver as example. Whenever you run a selenium script. A process named 'chromedriver' starts as well. Let's call it 'control process'. It opens the browser and controls it. And it also acts as a http server which you can get the address and port by driver.command_executor._url. It receives http request, processes it, tells the browser to do something(maybe open a url) and returns. Details here.

When you call

elem = WebDriverWait(driver, 10).until(ec.element_to_be_clickable(
    (By.XPATH, "//td[@align='right']/span[@class='pager']"))
)

you are actually sending a request to the 'control process' which is a http server and tell it to do something(find some elements in current page). The timeout '10' means that 'control process' tells browser to open a page in 10 seconds before it cancels and returns timeout status to the python script.

But what really happens here is the 'control process' is receiving request but not responding. I don't really know what's happening in the 'control process'.

Python selenium package is using urllib3.request to send request and socket._GLOBAL_DEFAULT_TIMEOUT as timeout. It is none by default that makes a request wait infinitely. So you can set it by using driver.command_executor.set_timeout(10). Now if 'control process' brokes you will get a timeout exception and maybe recreate webdriver.

zhangzhen
  • 21
  • 4