0

I'm trying to scrape the table values from the link below by adjusting the calendar dates to be between two specific dates: https://markets.businessinsider.com/earnings-calendar

I use the code block below, but the last line does not work:

from selenium.webdriver.common.by import By
wd = webdriver.Chrome('chromedriver',chrome_options=chrome_options)
wd.maximize_window()
wd.get("https://markets.businessinsider.com/earnings-calendar")
wd.implicitly_wait(10)
wd.find_element(By.XPATH, "//input[@id='calendar-input-from']").click()

But I'm getting this error:

ElementNotInteractableException: Message: element not interactable
  (Session info: headless chrome=101.0.4951.64)
Stacktrace:
#0 0x55f34d8af553 <unknown>
#1 0x55f34d59a067 <unknown>
#2 0x55f34d5d1741 <unknown>
#3 0x55f34d5c659a <unknown>
#4 0x55f34d5edf82 <unknown>
#5 0x55f34d5c5e36 <unknown>
#6 0x55f34d5ee3be <unknown>
#7 0x55f34d602d6c <unknown>
#8 0x55f34d5ee363 <unknown>
#9 0x55f34d5c447c <unknown>
#10 0x55f34d5c5945 <unknown>
#11 0x55f34d91f9d0 <unknown>
#12 0x55f34d8e4a38 <unknown>
#13 0x55f34d8e475c <unknown>
#14 0x55f34d8e4fc2 <unknown>
#15 0x55f34d91d71b <unknown>
#16 0x55f34d8e5221 <unknown>
#17 0x55f34d8c85b3 <unknown>
#18 0x55f34d8ee988 <unknown>
#19 0x55f34d8eeb1a <unknown>
#20 0x55f34d907a41 <unknown>
#21 0x7eff3f2796db <unknown>

Would appreciate it if someone could help me!

Amir
  • 1

1 Answers1

0
elem=wait.until(EC.element_to_be_clickable((By.CSS_SELECTOR,'#calendar-input-from')))

You can use two methods:

driver.execute_script("arguments[0].value = '2022-06-20';", elem)

elem.send_keys("2022-06-20")

Import:

from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait 
from selenium.webdriver.support import expected_conditions as EC
Arundeep Chohan
  • 9,779
  • 5
  • 15
  • 32
  • Thank yo uArundeep! I'm new to selenium. What you suggested worked. But my code below does not fetch the links in the table. I'd appreciate it again if you take a look: – Amir Jun 04 '22 at 22:31
  • ```from selenium.webdriver.common.by import By from selenium.webdriver.support.ui import WebDriverWait from selenium.webdriver.support import expected_conditions as EC elem = WebDriverWait(wd, 10).until(EC.element_to_be_clickable((By.CSS_SELECTOR,'#calendar-input-from'))) ``` – Amir Jun 04 '22 at 22:31
  • ```wd.execute_script("arguments[0].value = '2022-06-20';", elem) elem = WebDriverWait(wd, 10).until(EC.element_to_be_clickable((By.CSS_SELECTOR,'#calendar-input-to'))) wd.execute_script("arguments[0].value = '2022-06-24';", elem) wd.find_element(By.XPATH, "//span[@id = 'search_button']").click() elems = wd.find_elements_by_xpath("//a[@href]") for elem in elems: print(elem.get_attribute("href"))``` – Amir Jun 04 '22 at 22:31
  • There is about 20208 href values so I would recommend a large wait for visibility_of_all_elements using webdriver waits. – Arundeep Chohan Jun 05 '22 at 02:28
  • larger wait time doesn't work either. I think the problem is that it still scrapes the webpage for today's date. Today the table is empty. Yesterday's table was full and I managed to get yesterday's links. So for some reason after adjusting the calendar it is still scraping today's table. – Amir Jun 05 '22 at 03:31