Extracting token from XHR request header with R

Question

I've been scraping data from an API using R with the httr and plyr libraries. Its pretty straight forward and works well with the following code:

library(httr)
library(plyr)

headers <- c("Accept" = "application/json, text/javascript",
         "Accept-Encoding" = "gzip, deflate, sdch",
         "Connection" = "keep-alive",
         "Referer" = "http://www.afl.com.au/stat",
         "Host" = "www.afl.com.au",
         "User-Agent" = "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.118 Safari/537.36",
         "X-Requested-With"= "XMLHttpRequest",
         "X-media-mis-token" = "f31fcfedacc75b1f1b07d5a08887f078")

query <- GET("http://www.afl.com.au/api/cfs/afl/season?seasonId=CD_S2016014", add_headers(headers))

stats <- httr::content(query)

My question is with regards to the request token required in the headers (i.e. X-media-mis-token). This is easy to get manually by inspecting the XHR elements in Chrome or Firefox, but the token is updated every 24 hrs making manual extraction a pain.

Is it possible to query the web page and extract this token automatically using R?

Can you check the answer and vote & check it? – Karthik Arumugham Feb 22 '17 at 08:24 — Karthik Arumugham, Feb 22 '17 at 08:24

score 3 · Accepted Answer · answered Feb 22 '17 at 02:38

3

You can get the X-media-mis-token token, but with a disclaimer. ;)

library(httr)
token_url <- 'http://www.afl.com.au/api/cfs/afl/WMCTok'
token <- POST(token_url, encode="json")
content(token)$token
#[1] "f31fcfedacc75b1f1b07d5a08887f078"
content(token)$disclaimer
#[1] "All content and material contained within this site is protected by copyright owned by or licensed to Telstra. Unauthorised reproduction, publishing, transmission, distribution, copying or other use is prohibited.

answered Feb 22 '17 at 02:38

Karthik Arumugham

1,300
1
11
18

The answer is correct Karthik (I thought that was clear from my previous comment?). I have voted but it doesn't count as I don't have enough reputation points. – Safarr Feb 24 '17 at 02:14
@Safarr Can pls you accept the answer by clicking the tick (check mark) on left under the vote arrows? Thanks! – Karthik Arumugham Feb 24 '17 at 02:42
Ah, sorry Karthik. This was my first post so a learning experience! – Safarr Feb 25 '17 at 03:17

Extracting token from XHR request header with R

1 Answers1