3

I've created a script using selenium to grab a text generating dynamically. It is a requirement that I use selenium, so I don't wanna go for xhr in this very case. I'm trying to use pseudo selector in selenium defining explicit wait within it. As selenium doesn't support pseudo selector, as in :contains(), I used javascript command driver.execute_script() to serve the purpose.

Now, the way I tried works inconsistently as the text I wanna grab is not within the viewport. If I scroll a little down manually while the script is running, it works. Xpath is not an option here by the way.

How can I scroll the element into view while using pseudo selector within driver.execute_script()?

I've tried like [works inconsistently]:

from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait

with webdriver.Chrome() as driver:
    wait = WebDriverWait(driver, 10)
    driver.get('https://www.nyse.com/quote/XNYS:AAN')

    item = wait.until(
        lambda driver: driver.execute_script('''return $('span:contains("AARONS")')[0];''')
    )
    print(item.text)

Expected result:

AARONS INC

PS Before marking this post as duplicate, make sure you at least read the title of the question.

MITHU
  • 113
  • 3
  • 12
  • 41
  • The issue I see is that the `span` element does not actually contain the text `AARONS INC` unless you inspect the element and attempt to expand the `span` element in devtools. – CEH Jan 20 '20 at 16:43
  • The issue is no more when you scroll a little down while running the script. The problem however is that I don't find any idea to bring the element into viewport. – MITHU Jan 20 '20 at 17:03
  • Unfortunately I am testing locally with this website and even after scrolling `AARONS INC` into the viewport (by scrolling to the `Quote` header), the `span` still does not contain the `AARONS INC` text -- I waited up to 15 seconds but the element was never found. – CEH Jan 20 '20 at 17:08
  • Please check out this ***[video link](https://filebin.net/5eb8izhjvsyyi0hu)*** to clear any confusion @Christine. – MITHU Jan 20 '20 at 17:28
  • I cannot clearly understand what you want to do. I execute your code without manually scroll and it works. – Yun Jan 21 '20 at 08:11
  • Although the above approach works inconsistently, in other cases the element may not be intractable because of being distant from the viewport. So, I would like to know how to apply `element.scrollIntoView()` or something similar (`the suitable command`) on `driver.execute_script()` specially when there is a pseudo selector defined within it @Yun. – MITHU Jan 21 '20 at 11:56

2 Answers2

5

You can use the returned WebElement to execute another JavaScript command

item = wait.until(
    lambda d: d.execute_script('''return $('span:contains("AARONS")')[0];''')
)

driver.execute_script('arguments[0].scrollIntoView();', item)

Or in a single script

item = wait.until(
    lambda d: d.execute_script(
        '''var el = $('span:contains("AARONS")')[0];
        if (typeof el !== "undefined") {
            el.scrollIntoView();
        }
        return el;''')
)

This will wait until the element exists and scroll it into view.

Guy
  • 46,488
  • 10
  • 44
  • 88
  • Thanks for your answer @Guy. My question is meant for your second answer actually. However, it throws this error `raise exception_class(message, screen, stacktrace) selenium.common.exceptions.JavascriptException: Message: javascript error: Cannot read property 'scrollIntoView' of undefined` – MITHU Jan 22 '20 at 18:27
  • This is too little of a bounty for this epic answer. I'll try in different cases to see how it behaves. Thanks a lot. The first answer I suppose will not help as it will throw error before reaching this line `driver.execute_script('arguments[0].scrollIntoView();', item)` – MITHU Jan 22 '20 at 18:38
  • @MITHU The first one will also wait for the element, just won't scroll. If you see timeout exception you can increase the time in `WebDriverWait`. – Guy Jan 22 '20 at 18:47
0

You can try also by identifying the CSS element

Create a common method with CSS parameter and can reuse it

script = "document.querySelector('#{css_selector}').scrollIntoView(true);"

driver.execute_script(script)
Kiran Reddy
  • 120
  • 1
  • 6