0

I am scraping a website for data from a table, which is loaded via AJAX. The website is slow, and inconsistent, so sometimes I have to wait <5 sec for the table to load, while other times I have to wait 25 - 30. I am iterating through hundreds of items that filter the table, and once loaded, I go to the next item.

The functionality of the Explicit Wait / Expected Conditions does not seem to be behaving as I expect and wondered if anyone might have some insight.

I have tried numerous approaches to the problem, which I seem to have a different exception each time I run it.

This first snippet is to keep trying until it finds the element. I want to continue running until the page is fully loaded and the element is found. The problem is, the page is still loading and the element hasn't been found yet, but it still throws an exception. 

    for s in range(0,1000):
            try:
                #Other Month Value Clicked
                wait.until(EC.visibility_of_element_located((By.XPATH, "/html/body/div[9]/div/div[2]/div[" + str(mths[x]) + "]")))
            except NoSuchElementException:
                print(".", end=".", flush=True)
                time.sleep(1)
                timePeriodVal.click()
                time.sleep(1)
                timePeriodVal.click()
                continue
            finally:
                timePeriod = (driver.find_element_by_xpath("/html/body/div[9]/div/div[2]/div[" + str(mths[x]) + "]"))
                timePeriod.click()
                #print('\nTime Period clicked')
                time.sleep(1.5)
                break
Steven Johnson
  • 39
  • 1
  • 7
  • 2
    Can you provide the actual url? – G_M Oct 22 '18 at 21:03
  • 1
    Note that ExplicitWait will NOT lead to `NoSuchElementException` (you need `TimeOutException`), so `finally` block will be executed after first failed attempt. Share HTML of element you want to select, `wait` definition, what is `mths[x]` – Andersson Oct 22 '18 at 21:23
  • Thanks Andersson, So if I change to TimeOutException, is it correct to say it will throw an exception if it never finds the element, then execute the finally block once that exception is thrown? mths[x] is a variable that iterates through divs based on which month is being chosen. I would share the URL, but the site is password protected, and you won't be able to access it. The site is pretty annoying because all the html IDs change dynamically upon page reload, so it makes it annoying to hook on to any of the elements. – Steven Johnson Oct 22 '18 at 23:19
  • xpath like `html/body/div[9]/div/div[2]` could be a problem on its own. Are you sure you always need to select 9th div or 2nd div? what if something else shifts that index? etc. Instead, look for properties of elements you are choosing and skip all the irrelevant parts. Second: if you need all elements of the specific type, you may need `presence_of_all_elements_located`, or even define a custom criteria if your page is slow and inconsistent. – timbre timbre Oct 22 '18 at 23:46
  • also instead of assigning variable in finally, do this: assign `None ` to a variable timePeriod at the beginning of the loop; change it at `try`,, i.e.: `timePeriod = wait.until(...`, and then instead of code you have in finally, have a code outside of finally that goes like this: `if !(timePeriod is None) ...` – timbre timbre Oct 22 '18 at 23:46
  • This sounds like an [X-Y problem](http://xyproblem.info/). Instead of asking for help with your solution to the problem, edit your question and ask about the actual problem. What are you trying to do? – undetected Selenium Oct 23 '18 at 10:45
  • Thanks Kiril. This is very helpful. – Steven Johnson Oct 23 '18 at 12:01
  • 1
    @DebanjanB The problem was I couldn’t seem to get the code to catch the issue of not finding or attaching to an element, and would just proceed and throw an error. I’m going to try the solutions above and edit the question if those solutions don’t work. – Steven Johnson Oct 23 '18 at 12:04
  • @StevenJohnson Sure, go ahead and keep us updated about the status. Good luck !!! – undetected Selenium Oct 23 '18 at 12:05

0 Answers0