web-scrape asp.net web-site with r

Question

I'd like to web-scrape the html as seen in the source code of the web-browser, for this url "https://portal.tirol.gv.at/wisPvpSrv/wisSrv/wis/wbo_wis_auszug.aspx?ATTR=Y&TREE=N&ANL_ID=T20889658R3&TYPE=0".

what I get with..

library(RCurl)
library(XML)
myurl = "https://portal.tirol.gv.at/wisPvpSrv/wisSrv/wis/wbo_wis_auszug.aspx?ATTR=Y&TREE=N&ANL_ID=T20889658R3&TYPE=0"
x = getURL(myurl, followlocation = TRUE, ssl.verifypeer = FALSE)
htmlParse(x, asText = TRUE)

..is not what I see in the browser's source code - how to circumvent this??

It looks like you need an user and password to access the site. — Luciano Selzer, Dec 02 '11 at 13:57

score 3 · Accepted Answer · edited May 23 '17 at 12:27

Here ya go:

 library(RCurl) 
 library(XML)

 cookie = 'cookiefile.txt'
 curl  =  getCurlHandle ( cookiefile = cookie ,
     useragent =  "Mozilla/5.0 (Windows; U; Windows NT 5.1; en - US; rv:1.8.1.6) Gecko/20070725 Firefox/2.0.0.6",
     header = FALSE,
     verbose = TRUE,
     netrc = TRUE,
     maxredirs = as.integer(20),
     followlocation = TRUE,
    # userpwd = "bob:duncantl", ## enter here your username:password
     ssl.verifypeer = TRUE)

 myurl = "https://portal.tirol.gv.at/wisSrvPublic/wis/wbo_wis_auszug.aspx?ANL_ID=T20889658R3&TYPE=O" 

 x = getURL(myurl, curl = curl, cainfo = "path to R/library/RCurl/CurlSSL/ca-bundle.crt")

 x2 <- gsub('\r','', gsub('\t','', gsub('\n','', x))) # remove white spaces

 htmlParse(x2, asText = TRUE)

If you can not pass the ssl verification have a look at this post : using Rcurl with HTTPs

WORKS! ..as there is public access (see comment above) I will not need the cookie or the username:pw. Just for curiosity: it is the verbose=T that does the job, right? And, what will be the effect if I don't set the useragent properly? — Kay, Dec 02 '11 at 16:11
It is the ssl verification that was needed., defined by the cainfo term, suppling the certificates — Mischa Vreeburg, Dec 03 '11 at 09:36

score 1 · Answer 2 · edited Dec 02 '11 at 14:02

If that website uses a lot of Javascript (and it seems it does) to generate content then you are pretty much stuck for starters.

If you use Firefox and get the developer toolbar then you can disable Javascript to see what the site looks like without it, and what content might be scrapable. You may hope that the site has a usable non-javascript version (this is called 'graceful degradation', where JS is only used to fancy stuff).

Otherwise use Firebug or some other JS debugger to see how the site pulls content if it's using AJAX. Then replicate those calls in R and scrape from the response.

Not that I can test any of this because if I go to that URL I get a Benutzername and Passwort prompt, and I don't have a Benutzername. If the content is behind authentication then you'll have to handle that in the RCurl process too - which might mean mucking with cookies and so on.

Good luck with that.

sry, this slightly different url would be public and usable for the same purpose : "https://portal.tirol.gv.at/wisSrvPublic/wis/wbo_wis_auszug.aspx?ANL_ID=T20889658R3&TYPE=O" — Kay, Dec 02 '11 at 14:09

web-scrape asp.net web-site with r

2 Answers2

Linked