Yahoo Finance - Web scraping with R - SelectorGadget doesn't work

Question

I am trying to get the 1980 matching stocks from this Yahoo Finance Screener:

[https://finance.yahoo.com/screener/unsaved/38a77251-0996-439b-8be4-9d10ff18ff79?count=25&offset=0]

using R and rvest.

I normally use XPath but I can't get it with SelectorGadget at this website.

Could somebody help me about an alternative way to get al pages with those data.

I wanted to have code similar yo this one and that worked with Investing. Please note that the Symbol, Name, and MarketCap codes are just examples:

library(rvest)
library(dplyr)

i=0
for(z in 1:80){
  
  url_base<-paste("https://finance.yahoo.com/screener/unsaved/38a77251-0996-439b-8be4-9d10ff18ff79?count=25&offset=0")
  zpg <- read_html(url_base)
  Symbol<-zpg %>% html_nodes("table") %>% html_nodes("span") %>% html_attr("data-id" )
  Name<-zpg %>% html_nodes("table") %>% html_nodes("span") %>% html_attr("data-name" )
  MarketCap<-zpg %>% html_nodes("table") %>% html_nodes("span") %>% html_attr("data-name" )
  data<-data.frame(WebID,FullName,MarketCap)
  
  if(i==0){
    USA<-data}else{
      USA<-rbind(USA,data)
    }
  i=i+1
}

Could you add your code too? – NelsonGon Aug 08 '20 at 16:26 — NelsonGon, Aug 08 '20 at 16:26

Jakub.Novotny · Accepted Answer · 2020-08-08T17:34:05.887

1

You could try using quantmod or tidyquant.

library(tidyverse)
library(tidyquant)

# getting symbols for NASDAQ
nasdaq <- read_delim("https://nasdaqtrader.com/dynamic/SymDir/nasdaqlisted.txt", delim = "|")

# scraping the data
df <- nasdaq %>%
  head() %>% # to fetch only a few rows
  rowwise() %>%
  mutate(data = list(tq_get(Symbol, from = '2020-08-01', to = "2020-08-07", warnings = FALSE)))

# getting the data ready
df2 <- df$data %>%
  bind_rows()

edited Aug 08 '20 at 17:34

answered Aug 08 '20 at 16:40

Jakub.Novotny

2,912
2
6
21

I am not familiarized with tidyquant but it seems that with this code I would just get data from Apple. – CarlosFC Aug 08 '20 at 17:17
Correct. To get the symbols for e.g. Nasdaq, you can try, for example, this: nasdaq <- readr::read_delim("http://www.nasdaqtrader.com/dynamic/SymDir/nasdaqlisted.txt", delim = "|")$Symbol – Jakub.Novotny Aug 08 '20 at 17:18
@CarlosFC I have edited the post to show you how to download many Symbols using tidyquant at once. – Jakub.Novotny Aug 08 '20 at 17:34
Thanks a lot Jakub. I didn't know the text files from nasdaqtrader. What I actually want to get is the list of tickers and names of the shares to import them from Amibroker via Yahoo Finance. I have also found otherlisted.txt to get NYSE stocks and finally merge both files. Since it seems you're familiarized with tradinr, my question now is: I have capitalitation from NASADAQ shares (Q, G, S) but, how could I get it from NYSE? – CarlosFC Aug 09 '20 at 10:39
I am in fact not very familiar with trading. I am not sure what is meant by Q, G, S. – Jakub.Novotny Aug 10 '20 at 14:48
They respectively mean mega, big, and small capitalitation – CarlosFC Aug 10 '20 at 15:50

Yahoo Finance - Web scraping with R - SelectorGadget doesn't work

1 Answers1