Connect to the Twitter Streaming API using R

Question

I just started playing around with the Twitter Streaming API and using the command line, redirect the raw JSON reponses to a file using the command below:

curl https://stream.twitter.com/1/statuses/sample.json -u USER:PASSWORD -o "somefile.txt"

Is it possible to stay completely within R and leverage RCurl to do the same thing? Instead of just saving the output to a file, I would like to parse each response that is returned. I have parsed twitter search results in the past, but I would like to do this as each response is received. Essentially, apply a function to each JSON response.

Thanks in advance.

EDIT: Here is the code that I have tried in R (I am on Windows, unfortunately). I need to include the reference to the .pem file to avoid the error. However, the code just "runs" and I can not seem to see what is returned. I have tried print, cat, etc.

download.file(url="http://curl.haxx.se/ca/cacert.pem", destfile="cacert.pem")
getURL("https://stream.twitter.com/1/statuses/sample.json", 
       userpwd="USER:PWD",
       cainfo = "cacert.pem")

Yes, anything you can do with command line curl you can do with `RCurl`. Why don't you include the code you've tried already? — hadley, Nov 24 '11 at 02:21
When trying the getURL function, I get a HTTP 401 error. This is I believe the result of wrong username password. — Mischa Vreeburg, Nov 26 '11 at 00:12
For the RCurl SSL issues see this post http://stackoverflow.com/questions/6736895/using-rcurl-with-https/8268667#8268667 — Mischa Vreeburg, Nov 26 '11 at 00:38

score 5 · Accepted Answer · answered Nov 29 '11 at 16:03

I was able to figure out the basics, hopefully this helps.

#==============================================================================
# Streaming twitter using RCURL
#==============================================================================
library(RCurl)
library(rjson)

# set the directory
setwd("C:\\")

#### redirects output to a file
WRITE_TO_FILE <- function(x) {

     if (nchar(x) >0 ) {
          write.table(x, file="Twitter Stream Capture.txt", append=T, row.names=F, col.names=F)
     }

}

### windows users will need to get this certificate to authenticate
download.file(url="http://curl.haxx.se/ca/cacert.pem", destfile="cacert.pem")

### write the raw JSON data from the Twitter Firehouse to a text file
getURL("https://stream.twitter.com/1/statuses/sample.json", 
       userpwd=USER:PASSWORD,
       cainfo = "cacert.pem", 
       write=WRITE_TO_FILE)

score 1 · Answer 2 · edited Nov 23 '11 at 14:58

1

Try the twitter api package for R.

install.packages('twitteR')
library(twitteR)

I think this is what you need.

edited Nov 23 '11 at 14:58

Justin

84,773
49
224
367

answered Nov 23 '11 at 14:55

Mischa Vreeburg

1,576
1
13
18

1

Thanks for this. I love the twitteR package. It's a great way to access many features of the API. At the moment, it doesn't handle access to streaming content. – Btibert3 Nov 23 '11 at 15:06

Connect to the Twitter Streaming API using R

2 Answers2

Linked