Best way to convert a list into a JSON

Question

I have a script that looks something like this (though not exactly because I obviously can't give my API key and what not to make it fully repoducible)

library(tidyverse)
library(dplyr)
library(httr)
library(civis)
library(rjson)
library(jqr)

record <- GET(
  'https://api.secure.com/v4/data',
  accept_json(),
  add_headers(Accept = 'application/json',
              Authorization = VanAPI)
)

record then produces a list that looks something like this:

Response [https://api.secure.com/v4/data]
  Date: 2021-08-12 20:54
  Status: 200
  Content-Type: application/json; charset=utf-8
  Size: 873 B
{
  "items": [
    {
      "playerId": 12827,
      "name": "Player Tiers",
      "type": "Dynamic",
      "description": null,
      "points": 249,
      "areSubgroupsSticky": false,
      "status": "Active",
...

What is the best way to convert this to a JSON that I can then apply the jqr package to?

Right now I'm doing:

test <- content(record)

Which produces this output:

$items
$items[[1]]
$items[[1]]$playerId
[1] 12827

$items[[1]]$name
[1] "Player Tiers"

$items[[1]]$type
[1] "Dynamic"

$items[[1]]$description
NULL

$items[[1]]$points
[1] 249

$items[[1]]$areSubgroupsSticky
[1] FALSE

$items[[1]]$status
[1] "Active"

$items[[1]]$subgroups
NULL

$items[[1]]$markedSubgroup
NULL


$items[[2]]
$items[[2]]$playerId
[1] 15723

$items[[2]]$name
[1] "Team Tiers"

$items[[2]]$type
[1] "Dynamic"

$items[[2]]$description
NULL

$items[[2]]$points
[1] 35

$items[[2]]$areSubgroupsSticky
[1] FALSE

$items[[2]]$status
[1] "Active"

$items[[2]]$subgroups
NULL

$items[[2]]$markedSubgroup
NULL


$items[[3]]
$items[[3]]$playerId
[1] 16620

$items[[3]]$name
[1] "Coaches Tiers"

$items[[3]]$type
[1] "Dynamic"

$items[[3]]$description
NULL

$items[[3]]$points
[1] 12

$items[[3]]$areSubgroupsSticky
[1] FALSE

$items[[3]]$status
[1] "Active"

$items[[3]]$subgroups
NULL

$items[[3]]$markedSubgroup
NULL



$nextPageLink
NULL

$count
[1] 3

But then I try to convert it to a JSON using test2 <- rjson::toJSON(test) but that produces this:

[1] "{\"items\":[{\"playerId\":12827,\"name\":\"Player Tiers\",\"type\":\"Dynamic\",\"description\":null,\"points\":249,\"areSubgroupsSticky\":false,\"status\":\"Active\",\"subgroups\":null,\"markedSubgroup\":null},{\"playerId\":15723,\"name\":\"Team Tiers\",\"type\":\"Dynamic\",\"description\":null,\"points\":35,\"areSubgroupsSticky\":false,\"status\":\"Active\",\"subgroups\":null,\"markedSubgroup\":null},{\"playerId\":16620,\"name\":\"Coaches Tiers\",\"type\":\"Dynamic\",\"description\":null,\"points\":12,\"areSubgroupsSticky\":false,\"status\":\"Active\",\"subgroups\":null,\"markedSubgroup\":null}],\"nextPageLink\":null,\"count\":3}"

Is there a better way to get an easy, clean JSON output?

score 1 · Accepted Answer · answered Aug 12 '21 at 22:18

Reading ?content, you'll see

      as: desired type of output: 'raw', 'text' or 'parsed'. 'content'
          attempts to automatically figure out which one is most
          appropriate, based on the content-type.

Continuing with their example,

library(httr)
r <- POST("http://httpbin.org/post", body = list(a = 1, b = 2))
r
# Response [http://httpbin.org/post]
#   Date: 2021-08-12 22:15
#   Status: 200
#   Content-Type: application/json
#   Size: 586 B
# {
#   "args": {}, 
#   "data": "", 
#   "files": {}, 
#   "form": {
#     "a": "1", 
#     "b": "2"
#   }, 
#   "headers": {
#     "Accept": "application/json, text/xml, application/xml, */*", 
# ...

The "normal" use of content gives us the named list we're expecting:

str(content(r))
# List of 8
#  $ args   : Named list()
#  $ data   : chr ""
#  $ files  : Named list()
#  $ form   :List of 2
#   ..$ a: chr "1"
#   ..$ b: chr "2"
#  $ headers:List of 7
#   ..$ Accept         : chr "application/json, text/xml, application/xml, */*"
#   ..$ Accept-Encoding: chr "deflate, gzip"
#   ..$ Content-Length : chr "228"
#   ..$ Content-Type   : chr "multipart/form-data; boundary=------------------------99386898172ff715"
#   ..$ Host           : chr "httpbin.org"
#   ..$ User-Agent     : chr "libcurl/7.64.1 r-curl/4.3 httr/1.4.1"
#   ..$ X-Amzn-Trace-Id: chr "Root=1-61159d7a-55ef1d2f553f80fa4773ae03"
#  $ json   : NULL
#  $ origin : chr "172.254.236.28"
#  $ url    : chr "http://httpbin.org/post"

And we can use as="text" to get the raw json text:

content(r, as="text")
# No encoding supplied: defaulting to UTF-8.
# [1] "{\n  \"args\": {}, \n  \"data\": \"\", \n  \"files\": {}, \n  \"form\": {\n    \"a\": \"1\", \n    \"b\": \"2\"\n  }, \n  \"headers\": {\n    \"Accept\": \"application/json, text/xml, application/xml, */*\", \n    \"Accept-Encoding\": \"deflate, gzip\", \n    \"Content-Length\": \"228\", \n    \"Content-Type\": \"multipart/form-data; boundary=------------------------99386898172ff715\", \n    \"Host\": \"httpbin.org\", \n    \"User-Agent\": \"libcurl/7.64.1 r-curl/4.3 httr/1.4.1\", \n    \"X-Amzn-Trace-Id\": \"Root=1-61159d7a-55ef1d2f553f80fa4773ae03\"\n  }, \n  \"json\": null, \n  \"origin\": \"172.254.236.28\", \n  \"url\": \"http://httpbin.org/post\"\n}\n"

While this is not pretty, it is exactly what the remote server returned: from http://httpbin.org, the /post endpoint returns:

You should be able to process this json with anything json-processor, including jsonlite::fromJSON (confirmed) and likely rjson and jqr (or even jq on the command-line after saving it to a text file).

score 0 · Answer 2 · answered Aug 15 '21 at 05:20

The following is an addendum to the answer given by @r2evans.

Even though content(r) seems to be a JSON-encoded string, the jqr commands recognize it as JSON text, so that (when using jqr), there is no need to run it through jq's fromjson filter.

So for example:

> library(httr)
> library(jqr)
> r <- GET("https://raw.githubusercontent.com/postmanlabs/httpbin/master/app.json")
> jq(content(r))
{
    "name": "httpbin",
    "description": "HTTP Request & Response Service, written in Python + Flask.",
    "repository": "https://github.com/requests/httpbin",
    "website": "https://httpbin.org",
    "logo": "https://s3.amazonaws.com/f.cl.ly/items/333Y191Z2C0G2J3m3Y0b/httpbin.svg",
    "keywords": [
        "http",
        "rest",
        "API",
        "testing",
        "integration",
        "python",
        "flask"
    ],
    "addons": "sentry"
}

> jq(content(r), ".name")
"httpbin"

Best way to convert a list into a JSON

2 Answers2