HTML Form authentication using R

Question

I'm trying to log into http://etportfolio.economictimes.indiatimes.com/Login.htm using R to access my stock holding within the portfolio created. Here is my script to which I get a message "Method Not Allowed"

library(RJSONIO)
library(rjson)
library(RCurl)

url <- ('http://etportfolio.economictimes.indiatimes.com/Login.htm')

agent="Firefox/23.0"
options(RCurlOptions = list(cainfo = system.file("CurlSSL", "cacert.pem", package = "RCurl")))
curl = getCurlHandle()
curlSetOpt( cookiejar = 'cookies.txt',  useragent = agent,  followlocation = TRUE ,  
autoreferer   = TRUE ,  curl = curl)


params <- list( 'ru' = "",
            'nru' = "",
            'loginPage' = "login",
            'channel' = "portfolio",
            'scriptTrackingCode' = "",
            'nonscriptTrackingCode' = "",
            'emailid' = "xxxx@hotmail.com",
            'password' = "password")

html1 = postForm(url,.params = params,curl = curl, style = "POST", verbose=TRUE)

if I try using httr package, i am able to get login success but need help on how to move forward. here is the code.

library(httr)
base_url <- ("http://etportfolio.economictimes.indiatimes.com/")

loginURL <- modify_url(
base_url,
path="/loginfrm.htm",
query = list(
title = "loginform",
action = "http://socialappsintegrator.indiatimes.com/socialsite/GenricSSOLogin?",
type = "login",
emailid = "xxx@hotmail.com",
password = "password" ) )
r <- POST(loginURL)

Can you please help identify whether I am making a mistake or is there a way by which I can authenticate using R ?

Your `url` is wrong. If you are faking a form submission, you should POST your data to the "action" of the `
` tag -- not the the page containing the HTML form. It appears that the form in question should be posting to `url <- ('http://socialappsintegrator.indiatimes.com/socialsite/GenricSSOLogin')` — MrFlick, Sep 11 '14 at 15:33
@MrFlick thank you for pointing this out. However I am a bit lost here while I donot see a return URL or a specific PID# which is the token to allow me access to extract my portfolio. Will you be able to advise ? — Apurva shah, Sep 11 '14 at 15:49
No. That's completely sure specific. In order to programmatically scrape a site you need to spend a lot of time figuring out how it works. I have neither the time nor interest in figuring out this specific site. If you are having trouble, consider asking a new question with a very specific programming question. — MrFlick, Sep 11 '14 at 16:07

score 1 · Answer 1 · answered Sep 11 '14 at 20:38

Often I find it easier depending on the use case to use selenium to drive a browser. What follows is a fairly standard script for filling a form.

library(RSelenium)

appURL <- ('http://etportfolio.economictimes.indiatimes.com/Login.htm')
myUser <- "user"
myPass <- "pass"
RSelenium::checkForServer()
RSelenium::startServer()
remDr <- remoteDriver()
remDr$open()
remDr$navigate(appURL)
webElem <- remDr$findElement("id", "emailid")
webElem$sendKeysToElement(list(myUser))
webElem <- remDr$findElement("id", "password")
webElem$sendKeysToElement(list(myPass))
# submit element here rather then click
remDr$findElement('class', 'btn-login')$submitElement()

Here is the error that I am facing........remDr$open() [1] "Connecting to remote server" Error: Summary: UnknownError Detail: An unknown server-side error occurred while processing the command. class: org.openqa.selenium.WebDriverException — Apurva shah, Sep 13 '14 at 05:01

HTML Form authentication using R

1 Answers1