Stack Overflow
Stack Overflow

Questions tagged [rcurl]

RCurl is an R package that provides an R-friendly wrapper for the libcurl C library.

RCurl is an R package that provides a wrapper for the libcurl C library (not the cURL command-line tool, ). RCurl greatly simplifies the process of interfacing R programs with web services and provides simple functions to perform common HTTP calls such as POST and GET. RCurl is cross-platform and installs necessary dependencies when it is added as an R package.

The core workhorse function in RCurl is curlPerform, which maps curl_easy_opt options almost verbatim. See listCurlOptions() for a list of arguments representing those options that can be passed to curlPerform (either directly or via other RCurl functions).

Users may also be interested in the package, which provides a further simplified wrapper around RCurl for common requests.

734 questions
16
votes
1 answer

R and Windows Authentication

We have a data server set up in IIS using Windows Authentication (Kerberos). Does anyone know how to call web services (GET and POST) from R authenticating against the Windows identity?
Roland Buergi
  • 1,157
  • 9
  • 23
16
votes
2 answers

RCurl: HTTP Authentication When Site Responds With HTTP 401 Code Without WWW-Authenticate

I'm implementing an R wrapper around PiCloud's REST API using the RCurl package to make HTTP(S) requests to the API server. The API uses Basic HTTP authentication to verify that users have sufficient permissions. The PiCloud documentation gives an…
bellybellybelly
  • 441
  • 1
  • 4
  • 8
15
votes
4 answers

How to get google search results

I used the following code: library(XML) library(RCurl) getGoogleURL <- function(search.term, domain = '.co.uk', quotes=TRUE) { search.term <- gsub(' ', '%20', search.term) if(quotes) search.term <- paste('%22', search.term, '%22',…
Avi
  • 2,247
  • 4
  • 30
  • 52
15
votes
2 answers

R: extracting "clean" UTF-8 text from a web page scraped with RCurl

Using R, I am trying to scrape a web page save the text, which is in Japanese, to a file. Ultimately this needs to be scaled to tackle hundreds of pages on a daily basis. I already have a workable solution in Perl, but I am trying to migrate the…
SlowLearner
  • 7,907
  • 11
  • 49
  • 80
14
votes
6 answers

devtools::install_github fails with CA cert error

When I try to call install_github, I get the following error (not just for this package, but for all github packages): > install_github('ramnathv/slidify') Downloading github repo ramnathv/slidify@master Error in curl::curl_fetch_memory(url, handle…
Jonathan Gilligan
  • 701
  • 1
  • 5
  • 21
14
votes
2 answers

How can I screenshot a website using R?

So I'm not 100% sure this is possible, but I found a good solution in Ruby and in python, so I was wondering if something similar might work in R. Basically, given a URL, I want to render that URL, take a screenshot of the rendering as a .png, and…
Zach
  • 29,791
  • 35
  • 142
  • 201
14
votes
1 answer

How to login and then download a file from aspx web pages with R

I'm trying to automate the download of the Panel Study of Income Dynamics files available on this web page using R. Clicking on any of those files takes the user through to this login/authentication page. After authentication, it's easy to…
Anthony Damico
  • 5,779
  • 7
  • 46
  • 77
13
votes
1 answer

SOAP request in R

Does anyone know how to formulate following SOAP request with R? POST /API/v201010/AdvertiserService.asmx HTTP/1.1 Host: advertising.criteo.com Content-Type: text/xml; charset=utf-8 Content-Length: length SOAPAction:…
jburkhardt
  • 675
  • 1
  • 4
  • 18
12
votes
3 answers

R: Download image using rvest

I'm attempting to download a png image from a secure site through R. To access the secure site I used Rvest which worked well. So far I've extracted the URL for the png image. How can I download the image of this link using rvest? Functions…
G. Gip
  • 337
  • 1
  • 4
  • 10
12
votes
1 answer

How to isolate a single element from a scraped web page in R

I want to use R to scrape this page: (http://www.fifa.com/worldcup/archive/germany2006/results/matches/match=97410001/report.html ) and others, to get the goal scorers and times. So far, this is what I've got: require(RCurl) require(XML) theURL…
PaulHurleyuk
  • 8,009
  • 15
  • 54
  • 78
11
votes
1 answer

Accessing FTPS using RCurl

I have an FTPS server which I can already work with using clients such as Filezilla and Syncback. I also have admin access to this FTPS server and I know that it is working properly with TLS because I see "TLS" in the Filezilla connection log. I…
Angelo
  • 2,936
  • 5
  • 29
  • 44
11
votes
1 answer

SSL verification causes RCurl and httr to break - on a website that should be legit

i'm trying to automate the login of the UK's data archive service. that website is obviously trustworthy. unfortunately, both RCurl and httr break at SSL verification. my web browser doesn't give any sort of warning. i can work around the issue…
Anthony Damico
  • 5,779
  • 7
  • 46
  • 77
11
votes
1 answer

Using RCurl/httr for Github Basic Authorization

I am trying to create an OAuth token from the command line using the instructions here. I am able to use curl from the command line, and get the correct response curl -u 'username:pwd' -d '{"scopes":["user", "gist"]}' \ …
Ramnath
  • 54,439
  • 16
  • 125
  • 152
10
votes
2 answers

R shows different HTML (when compared to web browser) for the same Google Search URL

Goal I would like to use R to download the HTML of a Google Search webpage as shown in a web browser. Problem When I download the Google Search webpage HTML in R, using the exact same URL from the web browser, I have noticed that the R downloaded…
Tony Breyal
  • 5,338
  • 3
  • 29
  • 49
10
votes
1 answer

Scraping password protected forum in r

I have a problem with logging in in my script. Despite all other good answers that I found on stackoverflow, none of the solutions worked for me. I am scraping a web forum for my PhD research, its URL is http://forum.axishistory.com. The webpage I…
Anastasia Pupynina
  • 721
  • 7
  • 13
1
2
3
…
48 49