Why is CSS Selector not able to click on element after waiting for element to be clickable?

Question

See the code below. You can ignore pretty much everything except for the last line, I think.

from selenium import webdriver
import os
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
import xlsxwriter
from datetime import datetime
import time

chrome_driver = os.path.abspath('C:/Users/ross/Desktop/chromedriver.exe')
browser = webdriver.Chrome(chrome_driver)
browser.get('https://finra-markets.morningstar.com/BondCenter/Default.jsp')

WebDriverWait(browser, 60).until(EC.element_to_be_clickable((By.CSS_SELECTOR, '#TabContainer > div > div.rtq-tab-wrap > div.rtq-tab-menus-wrap > ul > li:nth-child(3) > a > span'))).click()
WebDriverWait(browser, 60).until(EC.element_to_be_clickable((By.CSS_SELECTOR, '#firscreener-cusip'))).send_keys("STWD")
WebDriverWait(browser, 60).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "#ms-finra-advanced-search-form > div.ms-finra-advanced-search-btn > input:nth-child(2)"))).click()
WebDriverWait(browser, 60).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "#ms-agreement > input"))).click()

WebDriverWait(browser, 60).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "#ms-finra-search-results > div > div.qs-resultData > div.qs-resultData-body > div.rtq-grid.rtq-grid-auto-h > div.rtq-grid-hd > div > div:nth-child(7) > div"))).click()
time.sleep(2)
WebDriverWait(browser, 60).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "#ms-finra-search-results > div > div.qs-resultData > div.qs-resultData-body > div.rtq-grid.rtq-grid-auto-h > div.rtq-grid-hd > div > div:nth-child(7) > div"))).click()
time.sleep(2)
whole_chart = WebDriverWait(browser, 60).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "#ms-finra-search-results > div > div.qs-resultData > div.qs-resultData-body > div.rtq-grid.rtq-grid-auto-h > div.rtq-scrollpanel > div.rtq-grid-scroll"))).text


parent = browser.find_element_by_xpath('//*[@id="ms-finra-search-results"]/div/div[3]/div[1]/div[1]/div[2]/div[2]/div')
count_divs = len(parent.find_elements_by_xpath("./div"))



for row_num in range(1):

    symbol = WebDriverWait(browser, 60).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "#ms-finra-search-results > div > div.qs-resultData > div.qs-resultData-body > div.rtq-grid.rtq-grid-auto-h > div.rtq-scrollpanel > div.rtq-grid-scroll > div > div:nth-child(" + str(row_num + 1) + ") > div:nth-child(3)"))).text

    maturity = WebDriverWait(browser, 60).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "#ms-finra-search-results > div > div.qs-resultData > div.qs-resultData-body > div.rtq-grid.rtq-grid-auto-h > div.rtq-scrollpanel > div.rtq-grid-scroll > div > div:nth-child(" + str(row_num + 1) + ") > div:nth-child(7)"))).text

    moody_rating = WebDriverWait(browser, 60).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "#ms-finra-search-results > div > div.qs-resultData > div.qs-resultData-body > div.rtq-grid.rtq-grid-auto-h > div.rtq-scrollpanel > div.rtq-grid-scroll > div > div:nth-child(" + str(row_num + 1) + ") > div:nth-child(8)"))).text

    sandp_rating = WebDriverWait(browser, 60).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "#ms-finra-search-results > div > div.qs-resultData > div.qs-resultData-body > div.rtq-grid.rtq-grid-auto-h > div.rtq-scrollpanel > div.rtq-grid-scroll > div > div:nth-child(" + str(row_num + 1) + ") > div:nth-child(9)"))).text

    bond_yield = WebDriverWait(browser, 60).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "#ms-finra-search-results > div > div.qs-resultData > div.qs-resultData-body > div.rtq-grid.rtq-grid-auto-h > div.rtq-scrollpanel > div.rtq-grid-scroll > div > div:nth-child(" + str(row_num + 1) + ") > div:nth-child(11)"))).text

    if symbol.strip() and maturity.strip() and moody_rating.strip() and sandp_rating.strip() and bond_yield.strip() and moody_rating != "WR" and sandp_rating != "NR":
        WebDriverWait(browser, 60).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "#ms-finra-search-results > div > div.qs-resultData > div.qs-resultData-body > div.rtq-grid.rtq-grid-auto-h > div.rtq-scrollpanel > div.rtq-grid-scroll > div > div:nth-child(" + str(row_num + 1) + ") > div:nth-child(2) > div > a"))).click()

        WebDriverWait(browser, 60).until(EC.element_to_be_clickable((By.CSS_SELECTOR,"#tradeHistory_link"))).click()

Basically, I get to this page: https://finra-markets.morningstar.com/BondCenter/BondDetail.jsp?ticker=C724231&symbol=STWD4571190 and I want Selenium to click on the 'Trade History' link on it so I reference the selector '#tradeHistory_link'. For some reason it isn't clicking and when I try to just get the text it won't get that either. I have also tried clicking by looking for "Trade History" on the page and then clicking. That didn't work. Lastly, I thought maybe the page hadn't fully loaded yet so I tried doing time.sleep(5) before trying to click "#tradeHistory_link" but to no avail.

What's going on here?

Note that the Terms of Use for the Finra website specifically prohibit scraping, harvesting and/or creating your own database from their website. See https://www.finra.org/terms-of-use — Robert Harvey, May 12 '21 at 16:57
@vitaliis I plan on iterating through all of the count_divs but wanted to make sure that everything was working beneath the for statement first before moving ahead — Ross Leavitt, May 12 '21 at 23:47

score 1 · Answer 1 · answered May 12 '21 at 16:59

1

First of all try using visibility_of_element_located expected condition instead of element_to_be_clickable. Element not always has the inner text finally built inside it and the moment it becomes clickable. Visibility in normally comes a bit later.
I also have to mention that your css locators looking very bad. You have to improve them.

answered May 12 '21 at 16:59

Prophet

32,350
22
54
79

Actually, I just checked out the [source code](https://github.com/SeleniumHQ/selenium/blob/trunk/py/selenium/webdriver/support/expected_conditions.py#L281) for the `element_to_be_clickable` expected condition. As you can see, it requires the element to be **both visible and enabled**, so it is actually a stronger check/potentially longer wait than `visibility_of_element_located` – C. Peck May 12 '21 at 19:37
@Prophet what do you mean that the CSS locators are looking bad? Is there a more efficient way to reference 'maturity' for example? – Ross Leavitt May 12 '21 at 23:56
1

@RossLeavitt what he means, and I agree, is that your very long CSS selectors rely on a huge part of the structure of the DOM in order to identify the elements. So there is a high chance that even small changes unrelated to your tests might cause the selector(s) to quit working. – C. Peck May 13 '21 at 03:03
@C.Peck ah gotcha. I'm sort of confused how else I would be able to get the elements though. Since this is a graph and I want to return the values of the graph I can't look for text and get it that way. – Ross Leavitt May 13 '21 at 14:05

score 1 · Answer 2 · answered May 12 '21 at 17:05

That element resides in an iframe, which is likely why you can't click it. You need to switch to that frame before Selenium can interact with elements inside it. Befor interacting with that element, try adding this line

driver.switch_to.frame(driver.find_element_by_css_selector('#ms-bond-detail-iframe'))

score 1 · Accepted Answer · answered May 12 '21 at 18:54

The Trade History link is present inside an iframe. In order to access the link you need to switch to iframe first.

Use WebDriverWait() and wait for frame_to_be_available_and_switch_to_it() and use element ID

WebDriverWait(browser,10).until(EC.frame_to_be_available_and_switch_to_it((By.ID,"ms-bond-detail-iframe")))
WebDriverWait(browser,10).until(EC.element_to_be_clickable((By.CSS_SELECTOR,"#tradeHistory_link"))).click()

To come out from iframe you need to switch to default content

browser.switch_to.default_content()

Why is CSS Selector not able to click on element after waiting for element to be clickable?

3 Answers3