How can I improve the ease of working with JSON in Haskell?

Question

Haskell has become useful as a web language (thanks Servant!), and yet JSON is still so painful for me so I must be doing something wrong (?)

I hear JSON mentioned as a pain point enough, and the responses I've heard revolve around "use PureScript", "wait for Sub/Row Typing", "use esoterica, like Vinyl", "Aeson + just deal with the explosion of boiler plate data types".

As an (unfair) reference point, I really enjoy the ease of Clojure's JSON "story" (of course, it's a dynamic language, and has it's tradeoffs for which I still prefer Haskell).

Here's an example I've been staring at for an hour.

{
    "access_token": "xxx",
    "batch": [
        {"method":"GET", "name":"oldmsg", "relative_url": "<MESSAGE-ID>?fields=from,message,id"},
        {"method":"GET", "name":"imp", "relative_url": "{result=oldmsg:$.from.id}?fields=impersonate_token"},
        {"method":"POST", "name":"newmsg", "relative_url": "<GROUP-ID>/feed?access_token={result=imp:$.impersonate_token}", "body":"message={result=oldmsg:$.message}"},
        {"method":"POST", "name":"oldcomment", "relative_url": "{result=oldmsg:$.id}/comments", "body":"message=Post moved to https://workplace.facebook.com/{result=newmsg:$.id}"},
        {"method":"POST", "name":"newcomment", "relative_url": "{result=newmsg:$.id}/comments", "body":"message=Post moved from https://workplace.facebook.com/{result=oldmsg:$.id}"},
    ]
}

I need to POST this to FB workplace, which will copy a message to a new group, and comment a link on both, linking to each other.

My first attempt looked something like:

data BatchReq = BatchReq {
  method :: Text
  , name :: Text
  , relativeUrl :: Text
  , body :: Maybe Text
  }

data BatchReqs = BatchReqs {
  accessToken :: Text
  , batch :: [BatchReq]
  }

softMove tok msgId= BatchReqs tok [
  BatchReq "GET" "oldmsg" (msgId `append` "?fields=from,message,id") Nothing
  ...
  ]

That's painfully rigid, and dealing with Maybes all over is uncomfortable. Is Nothing a JSON null? Or should the field be absent? Then I worried about deriving the Aeson instances, and had to figure out how to convert eg relativeUrl to relative_url. Then I added an endpoint, and now I have name clashes. DuplicateRecordFields! But wait, that causes so many problems elsewhere. So update the data type to use eg batchReqRelativeUrl, and peel that off when deriving instances using Typeables and Proxys. Then I needed to add endpoints, and or massage the shape of that rigid data type for which I added more datapoints, trying to not let the "tyranny of small differences" bloat my data types too much.

At this point, I was largely consuming JSON, so decided a "dynamic" thing would be to use lenses. So, to drill into a JSON field holding a group id I did:

filteredBy :: (Choice p, Applicative f) =>  (a -> Bool) -> Getting (Data.Monoid.First a) s a -> Optic' p f s s
filteredBy cond lens = filtered (\x -> maybe False cond (x ^? lens))

-- the group to which to move the message
groupId :: AsValue s => s -> AppM Text
groupId json  = maybe (error500 "couldn't find group id in json.")
                pure (json ^? l)
  where l = changeValue . key "message_tags" . values . filteredBy (== "group") (key "type") . key "id" . _String

That's rather heavy to access fields. But I also need to generate payloads, and I'm not skilled enough to see how lenses will be nice for that. Circling around to the motivating batch request, I've come up with a "dynamic" way of writing these payloads. It could be simplified with helper fns, but, I'm not even sure how much nicer it'll get with that.

softMove :: Text -> Text -> Text -> Value
softMove accessToken msgId groupId = object [
  "access_token" .= accessToken
  , "batch" .= [
        object ["method" .= String "GET", "name" .= String "oldmsg", "relative_url" .= String (msgId `append` "?fields=from,message,id")]
      , object ["method" .= String "GET", "name" .= String "imp", "relative_url" .= String "{result=oldmsg:$.from.id}?fields=impersonate_token"]
      , object ["method" .= String "POST", "name" .= String "newmsg", "relative_url" .= String (groupId `append` "/feed?access_token={result=imp:$.impersonate_token}"), "body" .= String "message={result=oldmsg:$.message}"]
      , object ["method" .= String "POST", "name" .= String "oldcomment", "relative_url" .= String "{result=oldmsg:$.id}/comments", "body" .= String "message=Post moved to https://workplace.facebook.com/{result=newmsg:$.id}"]
      , object ["method" .= String "POST", "name" .= String "newcomment", "relative_url" .= String "{result=newmsg:$.id}/comments", "body" .= String "message=Post moved from https://workplace.facebook.com/{result=oldmsg:$.id}"]
      ]
  ]

I'm considering having JSON blobs in code or reading them in as files and using Text.Printf to splice in variables...

I mean, I can do it all like this, but would sure appreciate finding an alternative. FB's API is a bit unique in that it can't be represented as a rigid data structure like a lot of REST APIs; they call it their Graph API which is quite a bit more dynamic in use, and treating it like a rigid API has been painful thus far.

(Also, thanks to all the community help getting me this far with Haskell!)

Answer 1

Update: Added some comments on the "dynamic strategy" at the bottom.

In similar situations, I've used single-character helpers to good effect:

json1 :: Value
json1 = o[ "batch" .=
           [ o[ "method" .= s"GET", "name" .= s"oldmsg",
                   "url" .= s"..." ]
           , o[ "method" .= s"POST", "name" .= s"newmsg",
                   "url" .= s"...", "body" .= s"..." ]
           ]
         ]
  where o = object
        s = String

Note that the non-standard syntax (no space between one-character helper and argument) is intentional. It's a signal to me and others reading my code that these are technical "annotations" to satisfy the type checker rather than a more usual kind of function call that's actually doing something.

While this adds a little clutter, the annotations are easy to ignore while reading the code. They're also easy to forget while writing code, but the type checker catches those, so they're easy to fix.

In your particular case, I think some more structured helpers do make sense. Something like:

softMove :: Text -> Text -> Text -> Value
softMove accessToken msgId groupId = object [
  "access_token" .= accessToken
  , "batch" .= [
        get "oldmsg" (msgId <> "?fields=from,message,id")
      , get "imp" "{result=oldmsg:$.from.id}?fields=impersonate_token"
      , post "newmsg" (groupId <> "...") "..."
      , post "oldcomment" "{result=oldmsg:$.id}/comments" "..."
      , post "newcomment" "{result=newmsg:$.id}/comments" "..."
      ]
  ]
  where get name url = object $ req "GET" name url
        post name url body = object $ req "POST" name url 
                             <> ["body" .= s body]
        req method name url = [ "method" .= s method, "name" .= s name, 
                                "relative_url" .= s url ]
        s = String

Note that you can tailor these helpers to the specific JSON you're generating in a particular case and define them locally in a where clause. You don't need to commit to some big chunk of ADT and function infrastructure that covers all JSON use-cases in your code, as you might do if the JSON was more unified in structure across the application.

Comments on the "Dynamic Strategy"

With respect to whether or not using a "dynamic strategy" is the right approach, it probably depends on more context than can realistically be shared in a Stack Overflow question. But, taking a step back, the Haskell type system is useful to the extent that it helps clearly model the problem domain. At its best, the types feel natural and assist you with writing correct code. When they stop doing this, you need to rethink your types.

The pain you encountered with a more traditional ADT-driven approach to this problem (rigidity of the types, proliferation of Maybes, and the "tyranny of small differences") suggests that these types were a bad model at least for what you were trying to do in this case. In particular, given that your problem was one of generating fairly straightforward JSON directives/commands for an external API, rather than doing lots of data manipulation on structures that also happened to allow JSON serialization/deserialization, modeling the data as Haskell ADTs was probably overkill.

My best guess is that, if you really wanted to properly model the FB Workplace API, you wouldn't want to do it at the JSON level. Instead, you'd do it at a higher level of abstraction with Message, Comment, and Group types, and you'd end up wanting to generate the JSON dynamically anyway, because your types wouldn't directly map to the JSON structures expected by the API.

It might be insightful to compare your problem to generating HTML. Consider first the lucid (blaze-based) or shakespeare templating packages. If you look at how these work, they don't try to build up HTML by generating a DOM with ADTs like data Element = ImgElement ... | BlockquoteElement ... and then serializing them to HTML. Presumably the authors decided that this abstraction wasn't really necessary, because the HTML just needs to be generated, not analyzed. Instead they use functions (lucid) or a quasiquoter (shakespeare) to build up a dynamic data structure representing an HTML document. The chosen structure is rigid enough to ensure certain sorts of validity (e.g., proper matching of opening and closing element tags) but not others (e.g., no one stops you from sticking a <p> child in the middle of your <span> element).

When you use these packages in a larger web app, you model the problem domain at a higher level of abstraction than HTML elements, and you generate the HTML in a largely dynamic fashion because there's not a clear one-to-one mapping between the types in your problem domain model and HTML elements.

On the other hand, there's a type-of-html package that does model individual elements, so it's a type error to try to nest a <tr> inside <td> and so on. Developing these types probably took a lot of work, and there's a lot of inflexibility "baked in", but the trade-off is a whole other level of type safety. On the other hand, this seems easier to do for HTML than to do for a particular finicky JSON API.