Show more using Selenium requires sleep

Question

So my code is working but I do not understand why and that annoys me.

def load_more():
  while True:
    try:
      WebDriverWait(driver, 20, ignored_exceptions = (StaleElementReferenceException)).until(EC.element_to_be_clickable((By.CSS_SELECTOR, ".more"))).click()
      time.sleep(1)
    except TimeoutException:
      return

The above function is used after I loaded the initial page to click a "show more" button until it is no longer there.

My frustration lies in the use of time.sleep(1). If I remove this the code throws a StaleElementReferenceException at a random point. Even if it's listed in the ignored_exceptions.

My question is. Why does the code require the use of the sleep() function?

, ignored_exceptions = (StaleElementReferenceException) remove this line , — PDHide, Feb 27 '21 at 12:13
please add more context line what website , what aer you trying to do etc . staleelement means reference changed after it was indetified — PDHide, Feb 27 '21 at 12:16
adding ignored exception doesn't help if the element is found and reference changes after webdriverwait returns the element , — PDHide, Feb 27 '21 at 12:17
staelement happens also if there is any asynchronous action happening in the page — PDHide, Feb 27 '21 at 12:17
@PDHide I am guessing it is the latter then as it works perfectly when I include the sleep(). What would be the correct way to deal with asynchronous actions on the page? — SomeDutchGuy, Feb 27 '21 at 12:21
when you give time.sleep() you are waiting for the background process to complete. so you have to identify the background element that is still loading and wait for that to fininsh — PDHide, Feb 27 '21 at 12:25
example a notification dialog that appears and dissappears in 3 second while you are checking for something else. — PDHide, Feb 27 '21 at 12:26
When the notification panel is displayed and when it is no lnger their the reference of entire page changes — PDHide, Feb 27 '21 at 12:26

score 1 · Accepted Answer · answered Feb 27 '21 at 12:28

1

https://www.selenium.dev/exceptions/

https://developer.mozilla.org/en-US/docs/Web/WebDriver/Errors/StaleElementReference

When an element is no longer attached to the DOM, i.e. it has been removed from the document or the document has changed, it is said to be stale

So adding ignored exception doesn't help if the element is found and reference changes after webdriverwait returns the element

when you give time.sleep() you are waiting for the background process to complete. so you have to identify the background element that is still loading and wait for that to fininsh

AN example scenario would be :

A notification dialog that appears and dissappears in 3 second while you are checking for something else

your code flow might be like:

 click something 
 do something else without thinking about the notification

But instead it should be like :

 click something
 wait for notification
 wait for notification to be removed
 do something

answered Feb 27 '21 at 12:28

PDHide

18,113
2
31
46

That clarifies. Is there an easy method to determine which element is still loading using for example a browser? I assume I should wait for the last element the page loads. – SomeDutchGuy Feb 27 '21 at 12:32
yes thats right make sure the document doesn't get modified after the element is discovered – PDHide Feb 27 '21 at 12:33
unless i know what website and what you are trying to do , its not easy to tell what it would be . you can try adding DOM break point and seeing how the DOM is getting build – PDHide Feb 27 '21 at 12:35
Yeah. I should learn to do this myself as I might need to figure it out for multiple sites. – SomeDutchGuy Feb 27 '21 at 12:36
@RuudVerhoef https://stackoverflow.com/a/65997614/6793637 see this example – PDHide Feb 27 '21 at 12:47

Show more using Selenium requires sleep

1 Answers1