-1

I just need to click the load more button once to reveal a bunch more information so that I can scrape more HTML than what is loaded.

The following "should" go to github.com/topics and find the one and only button element and click it one time.

from selenium import webdriver
from selenium.webdriver.common.by import By
import time

driver = webdriver.Edge()

driver.get("https://github.com/topics")
time.sleep(5)
btn = driver.find_element(By.TAG_NAME, "button")
btn.click()
time.sleep(3)
driver.quit()

I'm told Message: element not interactable so I'm obviously doing something wrong but I'm not sure what.

TedRed
  • 7
  • 2

2 Answers2

0

There are several issues with your code:

  1. The "Load more" button is initially out of the view, so you have to scroll the page in order to click it.
  2. Your locator is bad.
  3. You need to wait for elements to appear on the page before accessing them. WebDriverWait expected_conditions explicit waits should be used for that, not hardcoded sleeps.

The following code works, it scrolls the page and clicks "Load more" 1 time.

import time

from selenium import webdriver
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC

options = Options()
options.add_argument("start-maximized")

webdriver_service = Service('C:\webdrivers\chromedriver.exe')
driver = webdriver.Chrome(options=options, service=webdriver_service)
wait = WebDriverWait(driver, 20)


url = "https://github.com/topics"
driver.get(url)

load_more = wait.until(EC.presence_of_element_located((By.XPATH, "//button[contains(.,'Load more')]")))
load_more.location_once_scrolled_into_view
time.sleep(1)
load_more.click()

UPD
You can simply modify the above code to make it clicking Load more button while it presented.
I implemented this with infinite while loop making a break if Load more button not found. This code works.

import time

from selenium import webdriver
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC

options = Options()
options.add_argument("start-maximized")

webdriver_service = Service('C:\webdrivers\chromedriver.exe')
driver = webdriver.Chrome(options=options, service=webdriver_service)
wait = WebDriverWait(driver, 5)

url = "https://github.com/topics"
driver.get(url)

while True:
    try:
        load_more = wait.until(EC.presence_of_element_located((By.XPATH, "//button[contains(.,'Load more')]")))
        load_more.location_once_scrolled_into_view
        time.sleep(1)
        load_more.click()
    except:
        break
Prophet
  • 32,350
  • 22
  • 54
  • 79
  • Thank you so much!! I tried to just loop what you did here and it works until I get down to the "S's" roughly. What's the meaning of the 20? I know I said just one but I figured looping would be pretty straightforward. I need to read more documentation for sure but let me know if you have the time!! – TedRed Nov 16 '22 at 19:55
  • Actually, that was because there was nothing else to load. I ran it 5 more times to get everything now I'm saving the html file locally using `html = driver.page_source`. Eventually I'll use beautiful soup library to save info from the html file. – TedRed Nov 16 '22 at 20:06
  • OK, I updated the answer to make it clicking `Load more` while it appears. – Prophet Nov 16 '22 at 21:59
0

use

btn = driver.findElementsByXPath("//button[contains(text(),'Load more')]");

You are not finding the right element. This is the reason why it is not "interactable"

Burak Demircan
  • 826
  • 7
  • 7