Python script to scroll down non scroll-able page

Question

I have an Airtable table I review on occasion and tried to create a Python script using selenium to scroll down a full page until it gets to the end. Here's the code but I can't get it to scroll down. I don't get any errors but it seems like it doesn't connect with the page to scroll. Any help is appreciated. Thanks

from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from time import sleep

url = 'https://airtable.com/embed/shrqYt5kSqMzHV9R5/tbl8c8kanuNB6bPYr?backgroundColor=green&viewControls=on'

driver = webdriver.Chrome()
driver.get(url)
driver.fullscreen_window()
WebDriverWait(driver, 10).until(EC.visibility_of_element_located((By.XPATH, '//html')))

scroll_pause_time = 5

# Get scroll height
last_height = driver.execute_script("return document.body.scrollHeight")

while True:
    # Scroll down to bottom
    driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")

    # Wait to load page
    sleep(scroll_pause_time)

    # Calculate new scroll height and compare with last scroll height
    new_height = driver.execute_script("return document.body.scrollHeight")
    if new_height == last_height:
        break
    last_height = new_height

score 1 · Accepted Answer · answered Sep 02 '20 at 05:56

The reason being you are not able to scroll your table is your page is anti scroll-able. You can check this by simply trying to scroll manually. Rather to load your page you have to drag your vertical scroll by clicking on it. To do so we can use drag_and_drop_by_offset method of ActionChains class as below:

# After your page is loaded
page_hight = driver.get_window_size()['height'] #Get page height
scroll_bar = driver.find_element_by_xpath("//div[contains(@class,'antiscroll-scrollbar-vertical')]")
ActionChains(driver).drag_and_drop_by_offset(scroll_bar, 0, page_hight-160).click().perform() #Substracted 160 fro page height to compensate differnec between window and screen height

Output

Python script to scroll down non scroll-able page

1 Answers1