I am trying to download several csv files from this website https://www.marketinout.com/ for a series of stock backtest strategies. For some reason I am getting the above error from the rvest package when trying to navigate to the webpage with the backtest results.
My only thought is that backtest results page can take some time to load and not sure if there is some sort of setting that I need to change to allow for more time for the webpage to load? I am able to navigate to other pages on the website just fine.
I have also attempted a solution with RSelenium with no success. RSelenium will navigate to the webpage just fine, but for some reason I cannot get it to download the csv file.
Here is my code I am using. It fails when trying to navigate to https://www.marketinout.com/stock-screener/backtest/backtest_strategy.php?va=1&strategy_id=75158
library(rvest)
library(readr)
library(tidyverse)
# log in to website
link <- "https://www.marketinout.com/home/login.php"
open_session <- session(link)
pg_form <- html_form(open_session)[[1]]
filled_form <- html_form_set(pg_form,
"l" = "USERNAME")
new_session <- session_submit(open_session, filled_form)
pg_form <- html_form(new_session)[[1]]
filled_form <- html_form_set(pg_form,
"l" = "USERNAME",
"d" = "PASSWORD")
new_session <- session_submit(new_session, filled_form)
# trying to get trade history from backtest into csv
file <- new_session %>%
session_jump_to("https://www.marketinout.com/stock-screener/backtest/backtest_strategy.php?va=1&strategy_id=75158") %>% # <--- this is where the error occurs trying to navigate to this link
session_jump_to("https://www.marketinout.com/stock-screener/backtest/csv_stocks.csv")
backtest_hist <- httr::content(file$response, as="text") %>%
read_csv(col_names = T)
As I mentioned, I am able to navigate to other webpages just fine. The below code works beautifully (which navigates to a specific stock portfolio and downloads the portfolio to a csv file).
library(rvest)
library(readr)
library(tidyverse)
# log in to website
link <- "https://www.marketinout.com/home/login.php"
open_session <- session(link)
pg_form <- html_form(open_session)[[1]]
filled_form <- html_form_set(pg_form,
"l" = "USERNAME")
new_session <- session_submit(open_session, filled_form)
pg_form <- html_form(new_session)[[1]]
filled_form <- html_form_set(pg_form,
"l" = "USERNAME",
"d" = "PASSWORD")
new_session <- session_submit(new_session, filled_form)
# download stock portfolio
file <- new_session %>%
session_jump_to("https://www.marketinout.com/tools/portfolio.php?pid=10257") %>%
session_jump_to("https://www.marketinout.com/tools/csv.csv?pid=10257&view=ov")
port_hist <- httr::content(file$response, as="text") %>%
read_csv(col_names = T) # <-- WORKS PERFECTLY
Further I have attempted a solution with RSelenium but it fails at the download csv step:
library(RSelenium)
library(readr)
library(tidyverse)
# run after opening selenium image on Docker
rem_driver <- remoteDriver(remoteServerAddr = "localhost", port = 4445L, browserName = "firefox")
rem_driver$open()
rem_driver$navigate("https://www.marketinout.com/home/login.php")
# login to website
username <- rem_driver$findElement(using = "css", value = "#l")
username$sendKeysToElement(list("USERNAME"))
rem_driver$findElement("css", ".button")$clickElement()
password <- rem_driver$findElement(using = "css", value = "tr~ tr+ tr .text")
password$sendKeysToElement(list("PASSWORD"))
rem_driver$findElement("css", ".button")$clickElement()
# navigate to backtest results page and download csv file of trade history
rem_driver$navigate("https://www.marketinout.com/stock-screener/backtest/backtest_strategy.php?va=1&strategy_id=75158")
file <- rem_driver$navigate("https://www.marketinout.com/stock-screener/backtest/csv_stocks.csv") # <-- this produces an empty file with no contents
httr::content(file$response, as="text") %>% write_lines("test.csv") # <-- empty csv file
Any help would be appreciated! I've been working on this for a week with no luck.
