Stack Overflow
Stack Overflow

Questions tagged [httr]

The R package 'httr' provides useful tools for working with HTTP connections. It is a simplified wrapper built on top of 'RCurl'.

The R package httr provides useful tools for working with HTTP connections. It is a simplified wrapper built on top of RCurl. It is much much less configurable but because it only attempts to encompass the most common operations it is also much much simpler.

Resources:

1019 questions
6
votes
1 answer

Issues with RCurl crawler based on concurrent requests

The following is a script to reproduce the problems i'm facing when building a crawler with RCurl that performs concurrent requests. The objective is to download the content of several thousands of web sites in order to perform statistical…
marbel
  • 7,560
  • 6
  • 49
  • 68
6
votes
2 answers

httr GET function running out of space when downloading a large file

i'm trying to download a file that's 1.1 gigabytes with httr but i'm hitting the following error: x <- GET( extract.path ) Error in curlPerform(curl = handle$handle, .opts = curl_opts$values) : cannot allocate more space: 1728053248 bytes my C…
Anthony Damico
  • 5,779
  • 7
  • 46
  • 77
6
votes
3 answers

Get response header

I would like to get response headers from GET or POST. My example is: library(httr) library(RCurl) url<-'http://www.omegahat.org/RCurl/philosophy.html' doc<-GET(url) names(doc) [1] "url" "handle" "status_code"…
Maciej
  • 3,255
  • 1
  • 28
  • 43
5
votes
1 answer

web browser authentication in GitHub Actions for Oauth token

I'm trying to use GitHub Actions to query the GitHub API using an OAuth 2.0 token through an R script. It works fine on my local machine when I run the code, where a browser window pops up indicating "Waiting for authentication in browser..." that…
fawda123
  • 479
  • 3
  • 12
5
votes
1 answer

Convert a curl OAuth2 token request to httr

I am trying to access the domain.com.au api using r and httr. I can achieve this using an API key but not with OAuth2 The docs indicate that: Obtaining an access token via Client Credentials Make a POST request to the…
t-student
  • 414
  • 2
  • 16
5
votes
1 answer

How to do a SOAP request for EUR-Lex API with R?

How would you do a SOAP request for EUR-Lex's API using R? EUR-Lex is an EU database containing many legal acts. In the manual for their web services, they describe their SOAP system but not how to use R for it. I've tried for a while now to employ…
Will M
  • 692
  • 9
  • 20
5
votes
2 answers

How to fix "invalid return_url" error when creating oauth token for Trello with httr?

I want to manage my Trello cards and boards using the trelloR package but when I try to create a token with the get_token function, I get an error message on my browser : "Invalid return_url". my_token <- get_token(key = my_key, secret =…
Samy
  • 53
  • 1
  • 4
5
votes
1 answer

how to deal with captcha when web scraping using R

I'm trying to scrape data from this website, using httr and rvest. After several times of scraping (around 90 - 100), the website will automatically transfer me to another url with captcha. this is the normal url:…
rankthefirst
  • 1,370
  • 2
  • 14
  • 26
5
votes
1 answer

Download data from Statistics Sweden. Swedish characters in query

I have managed to retrieve some data from Statistics Sweden using the web site api. The answers to this question solved most of my problems. How do I POST a JSON formatted request to GET JSON data from a URL in R into the data.frame in a less…
ChristianL
  • 140
  • 7
5
votes
1 answer

How to post within a rvest html_session?

How can i post "within" a html session? So after i opened a session via a <- rvest::html_session(url) I tried: library(httr) POST(path, add_headers(setNames(as.character(headers(a)), names(headers(a)))), …
Rentrop
  • 20,979
  • 10
  • 72
  • 100
5
votes
0 answers

How to pass ssl_verifypeer in Rvest?

I'm trying to use Rvest to scrape a table off of an internal webpage here at $JOB. I've used the methods listed here to get the xpath, etc. My code is pretty simple: library(httr) library(rvest) un = "username"; pw = "password" thexpath <-…
ClintWeathers
  • 576
  • 7
  • 22
5
votes
1 answer

R web scraper with jsessionid

I'm testing some web scrape scripts in R. I've read many tutorials, docs and tried different things but no success so far. The URL I'm trying to scrape is this one. It has public, government data, and no statements against web scrapers. It's in…
Ricardo Costa
  • 85
  • 7
5
votes
2 answers

Difference between query and body in httr::POST

I want to pass few post parameters in url, say two parameters p1 and p2. Values of p1 and p2 are xyz(string) and 1(numeric). What is difference between following commands: POST(url, body=list(p1="xyz",p2=1)) OR POST(url,…
Kanika Singhal
  • 655
  • 1
  • 9
  • 17
5
votes
2 answers

Configuring listener_endpoint in httr when using Rstudio server

I;'m struggling to connect to Google Analytics with httr oauth2.0 function oauth2.0_token(oauth_endpoints("google") , oauth_app("google", client.id, client.secret) , scope = "https://www.googleapis.com/auth/analytics.readonly") It works…
RInatM
  • 1,208
  • 1
  • 17
  • 39
5
votes
1 answer

Build an equivalent R request for a curl request

I have the following curl request: curl --request GET --header "key: value" http://urlhere How can I run the request in R?
tushaR
  • 3,083
  • 1
  • 20
  • 33
1 2 3
…
67 68