2

I'm working on a few functions to get data from StatBank Denmark, and their API. They have made a console to test JSON calls and I know the basic_request I parse to JSON in the function dst_get_data works, as I have tested it in the console.

I get a "status 400" error and an error message that says that I should "supply object when posting".

The code below should make reproducible example. It is the third function (dst_get_data) where I am stuck.

dst_meta <- function(table, ..., lang = "en"){
  require(jsonlite)
  require(httr)

  dkstat_url <- "http://api.statbank.dk/v1/tableinfo"

  params <- list("lang" = lang,
                 "table" = table,
                 "format" = "JSON")

  meta <- POST(url=dkstat_url, body=params, multipart=FALSE)
  meta <- jsonlite::fromJSON(txt=content(meta, as="text"),simplifyDataFrame=TRUE)
  #meta <- RJSONIO::fromJSON(content=content(meta),asText=TRUE, simplify=TRUE, simplifyWithNames=TRUE)

  return(meta)
}

dst_meta_parse <- function(meta, lang){

  basics_names <- c("id", "text", "description", 
                    "unit", "updated", "footnote")

  basics <- meta[names(meta) %in% basics_names]

  variables <- meta[["variables"]][,c("id", "text")]

  values <- meta[["variables"]][,"values"]
  names(values) <- variables$id

  if(lang == "en"){
    test <- grepl(pattern="Tid", names(values))
    if(sum(test) > 0){
      values$Tid$id <- sub(pattern="Q", replacement="K", x=values$Tid$id)
    }
  }

  ## Create basic_request for the data_post file
  basic_request <- vector(mode="list", length=length(variables$id))
  for(variable in 1:length(variables$id)){

    var_name <- variables$id[variable]

    if(var_name == "Tid"){
      basic_request[[variable]] <- list("code" = var_name,
                                       "values" = as.character(values[[var_name]]$id[length(values[[var_name]]$id)]))
    } else {
      basic_request[[variable]] <- list("code" = var_name,
                                       "values" = as.character(values[[var_name]]$id[1]))
    }
  }
  return(list("basics" = basics, "variables" = variables, "values" = values, "basic_request" = basic_request))
}

dst_get_data <- function(request, table,..., lang = "en", format = "CSV", value_presentation = "Default"){
  require(httr)
  require(jsonlite)

  dst_url <- "http://api.statbank.dk/v1/data"

  final_request <- list("table" = table,
                        "lang" = lang,
                        "format" = format,
                        "valuePresentation" = value_presentation,
                        "variables" = request)

  final_request <- jsonlite::toJSON(x=final_request, .escapeEscapes=TRUE, asIs=TRUE)
  print(validate(final_request))
  data <- POST(url=dst_url, body=final_request, multipart=FALSE)

  return(data)
}

test <- dst_meta(table="folk1")

test2 <- dst_meta_parse(meta = test, lang = "en")

test3 <- dst_get_data(request = test2$basic_request, table = "folk1", format="JSON")
#test3 <- dst_get_data(request = test2$basic_request, table = "folk1", format="JSON")
KERO
  • 705
  • 6
  • 15
  • Do you have any plans to make this code public as a package? – Thomas May 17 '14 at 05:45
  • @Thomas Yes, I have thought about it. But I just started yesterday, so not in its current form. It's my first package, so initially just on github though (I just need to catch up on some of the intro material). – KERO May 17 '14 at 08:03
  • Cool. Let me know if you want help putting it together (I'm on Github as leeper). I think there could be interest in such a package. – Thomas May 17 '14 at 08:07
  • Thanks! I'll keep you updated. – KERO May 17 '14 at 09:01

1 Answers1

3

When I change dst_get_data to:

dst_get_data <- function(request, table, ..., lang = "en", 
                         format = "CSV", value_presentation = "Default") {

  require(httr)
  require(jsonlite)

  dst_url <- "http://api.statbank.dk/v1/data"

  final_request <- list("table" = table,
                        "lang" = lang,
                        "format" = format,
                        "valuePresentation" = value_presentation)

  data <- POST(url=dst_url, body=final_request, multipart=FALSE)

  return(data)

}

and, the call to it to:

test3 <- dst_get_data(request = test2$basic_request, 
                      table = "folk1", format="JSONSTAT")

I get:

Response [http://api.statbank.dk/v1/data]
  Status: 200
  Content-type: text/json
{"dataset":{"dimension":{"Tid":{"label":"time","category":{"index":{"2014K2":0},"label":{"2014K2":"2014Q2"}}},"id":["Tid"],"size":[1],"role":{"time":["Tid"]}},"label":"Population at the first day of the quarter by time","source":"Statistics Denmark","updated":"2014-05-17T04:10:00Z","value":[5634437],"status":["a"]}} 

I think it was both the need for JSONSTAT and the fact that POST will do the JSON conversion for you automatically.The reason you need JSONSTAT is due to the fact that the "formats" available when you select data from the popup (in the console) are:

<select id="format" name="format"><option value="PX">PX</option>
<option selected="selected" value="CSV">CSV</option>
<option value="XLSX">XLSX</option>
<option value="HTML">HTML</option>
<option value="JSONSTAT">JSONSTAT</option>
<option value="DSTML">DSTML</option>
<option value="PNG">PNG</option>
<option value="BULK">BULK</option>
<option value="AREMOS">AREMOS</option>
<option value="SDMXCOMPACT">SDMXCOMPACT</option>
<option value="SDMXGENERIC">SDMXGENERIC</option>
</select>

Plain ol' JSON is not one of the options.

hrbrmstr
  • 77,368
  • 11
  • 139
  • 205
  • I wonder why my POST isn't working with JSON?! It's seems to me that @hadley is doing it here [https://github.com/hadley/httr/blob/master/vignettes/api-packages.Rmd](https://github.com/hadley/httr/blob/master/vignettes/api-packages.Rmd) Thank's for the pointer! – KERO May 17 '14 at 08:11
  • I updated the answer with the reason for `JSONSTAT`. The auto-convert to JSON may be a recent addition (hence the difference from Hadley's examples). – hrbrmstr May 17 '14 at 10:38
  • EDIT: deleted comment because I didn't take close look at the response. – KERO May 17 '14 at 22:18