0

I'm using RSelenium to automatically scroll down a social media website and save posts. I want to find the last instance of a web element (specifically, the post dates), but it's taking unacceptably long using my current method (using findElements() to return all post dates then extracting the last one - see code below) if I've scrolled a long way down the page.

Can anyone recommend a fast way to find the last web element (specifically, the post date) on a web page? For example, is there a way you can use findElement() (which searches for the first match) such that it starts from the bottom of the page rather than the top? Any suggestions welcome.

Here's a trivial example of my code, but it takes an unacceptably long time if I've scrolled a long way down the page.

# Load webpage of interest
library(RSelenium)
library(rvest)
rD = rsDriver(browser = "firefox")
remDr = rD[["client"]]
url = paste0("https://stocktwits.com/symbol/NZDCHF")
remDr$navigate(url) 

# Scroll down page three times, loading new content each time. 
for (i in 1:3)  { #Only scrolling 3 times for illustration
  remDr$executeScript("window.scrollTo(0,document.body.scrollHeight);")
  Sys.sleep(2) #delay by 3sec to give chance to load 
}

# Get date of last post. WORKS BUT TAKES FOREVER IF I'VE SCROLLED MANY TIMES
e = remDr$findElements("css", ".message-date")
last_date = e[[length(e)]]$getElementText()
jruf003
  • 980
  • 5
  • 19

1 Answers1

0

I found a messy workaround to solve the above. Specifically, I used the below to get the last message (which included the date), then wrote a regular expression to extract the last date.

last_child = remDriver$findElement(using = "css selector", 
                               value = ".messageli:last-child")
last_child = unlist(last_child$getElementText())
jruf003
  • 980
  • 5
  • 19