{r} R rvest error in loop/map: "Error in open.connection(x, "rb") : HTTP error 404."

Question

TLDR: code is ok, gets broken in loops

Hey folks. I coded a fun little thing that takes each abbreviation for a currency (eur, usd, cad, etc..) and then shows the ratio value with other currencies.

The code runs just fine, and the scraping is good. UNTIL i loop over it in order to get all currencies at once. then i get this error:

Error in open.connection(x, "rb") : HTTP error 404.

If anyone has a nice idea as to how to solve it - please :)

Code:

library(tidyverse)
library(rvest)


# this is the function to convert currencies.

currency_converter <- function(amount=1,
                               from="eur",
                               to="usd"){
  # Start by assigning the special HTML node:
  node=".text-success"
  
  # Define web link with shortened variable names:
  f=from
  t=to
  web_link <- paste0("https://wise.com/gb/currency-converter/",f,"-to-",t,"-rate")
  
  # Extract table:
  current_value <- rvest::read_html(web_link) %>% 
    rvest::html_nodes(node) %>%
    rvest::html_text()
  current_value <- str_split(current_value,pattern = " ")[[1]] %>%
    as.numeric()
  
  # Return the corect amount:
  
  return(current_value*amount)
  
}


# this is how i get all the abbriviations for currencies

website <- "https://www.easymarkets.com/eu/learn-centre/discover-trading/currency-acronyms-and-abbreviations/"
country.node <- "td:nth-child(1)"
currency.node <- "td+ td"


country <- rvest::read_html(website) %>% 
  rvest::html_nodes(country.node) %>%
  rvest::html_text()


currency <- rvest::read_html(website) %>% 
  rvest::html_nodes(currency.node) %>%
  rvest::html_text()

countries_currency <- tibble(country,currency)
# just minor cleaning
countries_currency <- countries_currency[-which(str_detect(countries_currency$currency,"EURO")),]

head(countries_currency)

# train data before i go big time
train <- sample_n(countries_currency,5)

# the idea is to have for each currency all other currencies,
# and then apply my function and get the ratio
# filtering ratio != 0 (since it is the same currency)
# and then get a big table with all currencies for all other currencies
# this should be a huge table (nrow() ^2)


trainloop <- tibble(amount=10,
                    from=str_to_lower(rep(train$currency,5)),
                    to=str_to_lower(rep(train$currency,each=5)))
head(trainloop)

trainloop %>% 
  mutate(ratio=pmap_dbl(cur_data(),currency_converter))
# does not seem to work


keep <- c()  
for(i in 1:nrow(trainloop)){
  for(to_ in trainloop$to){
    for(from_ in trainloop$from){
      
      
      keep[i] <- currency_converter(1,from=from_,to=to_)
      
      
    }
  }  
}

  
currency_converter(from=from_,to=to_) #error

currency_converter() #not error

currency_converter(amount = 100,
                   from = "usd",
                   to = "cad") #not error

Answer 1

It's most likely caused by the assumption that Wise accepts same-to-same conversions and returns the rate of 1. A side note regarding debugging, add a message(web_link) before rvest::read_html to check which request fails.

And there seems to be another assumption - that Wise supports the same set of currencies as known by Easymarkets. Here's some convoluted JS to check if that might be true, i.e. do the numbers actually match. Easymarkets currency list, EUROs filtered out::

// JavaScript
// head over to https://www.easymarkets.com/eu/learn-centre/discover-trading/currency-acronyms-and-abbreviations/ and
// execute in dev tools console:
Array.from(document.querySelectorAll("table > tbody > tr > td:nth-child(2)")).map(td => td.innerText).filter(cur => cur.indexOf("EURO") < 0 )
// (159) ['AUD', 'GBP', 'EUR', ..., 'MNT', 'MAD', 'MZN', …]

Wise currency list:

// JavaScript
// head over to https://wise.com/gb/currency-converter/ and
// execute in dev tools console:
window.config.converter.currencies
// (142) ['EUR', 'GBP', 'USD', ..., 'BGN', 'MMK', 'MUR']

So it's 159 vs 142. Lets check which ones are missing from Wise:

# R
# copy objects from browser's dev tool to R, parse as json and 
# check diff of sets

library(jsonlite) 
c_wise <- fromJSON('[
    "EUR",
    "GBP",
    "USD",
     ...
    "WST",
    "TMT",
    "BRL"
]')

c_easym <- fromJSON('[
    "AUD",
    "GBP",
    "EUR",
     ...
    "YER",
    "ZMK",
    "ZWD"
]')
setdiff(c_easym, c_wise)
#>  [1] "AFN" "BYR" "BIF" "CDF" "CUC" "CUP" "IRR" "IQD" "LYD" "MRO" "KPW" "STD"
#> [13] "SOS" "SDG" "SYP" "TMM" "VEB" "YER" "ZMK" "ZWD"

There are also number of currencies present in Wise but missing from Easymarkets ("UZS", "MRU", "GGP", "JEP", "ZMW", "TJS", "IMP", "BYN", "TMT"). So you are generating URLs that make no sense for Wise (so it returns 404) while also missing series of known values.

BTW, R is pretty well equipped when it comes to probability & statistics, so you can use combn() or expand.grid() to generate all required combinations. Though regarding expand.grid() (and your nested loops), same-to-same conversion must be removed as you'd get E404 for those.

# R
cur <- c('EUR', 'GBP', 'USD')
cur_c <- combn(cur,2)
cur_c
#>      [,1]  [,2]  [,3] 
#> [1,] "EUR" "EUR" "GBP"
#> [2,] "GBP" "USD" "USD"
for (c in 1:ncol(cur_c)){
  conv1 <- paste0(cur_c[,c],collapse = "-to-")
  conv2 <- paste0(rev(cur_c[,c]),collapse = "-to-")
  message(conv1,";", conv2)
}
#> EUR-to-GBP;GBP-to-EUR
#> EUR-to-USD;USD-to-EUR
#> GBP-to-USD;USD-to-GBP

# or expand.grid
expand.grid(from = cur, to = cur)
#>   from  to
#> 1  EUR EUR
#> 2  GBP EUR
#> 3  USD EUR
#> 4  EUR GBP
#> 5  GBP GBP
#> 6  USD GBP
#> 7  EUR USD
#> 8  GBP USD
#> 9  USD USD