How to handle variability of JSON objects in Haskell?

Question

Some REST service has variable returning JSONs, for example some fields can appear or disappear depending on the parameters of the request, the structure itself may change, nesting, etc. So, this leads to avalanche-type growth in the number of types (along with FromJSON instances). Options are to:

try to make a lot of fields under Maybe (but this does not help very much with the variability in structure)
to introduce a lot of types
to create different phantom types (actually no big difference with prev.)

The 1. has drawback that if your call with some fixed parameters always returns good knows fields, you have to handle Nothing cases too, code becomes more complex. The 2. and 3. is tiring.

What is the most simple/convenient way to handle such variability in Haskell (if you use Aeson, sure, another option is to avoid Aeson usage)?

It is better to provide an example to make the question more clear. As I know there are two way to handle optional field of using Aeson: `.!=` and `<|>`. which is better depending on the situation. see: https://stackoverflow.com/questions/53492106/haskell-parsing-an-object-that-could-be-multiple-types-into-one-single-type — assembly.jc, Dec 06 '18 at 09:28
If to talk about variability in fields existence: there are REST calls which can have parameter like "fields=a,b,c..." which allows you to have a lot of variances. All of them become `Maybe something`. Question is how to avoid big number of types in all cases (variability in structure, fields). And `Maybe` solution is bad too (default value sometimes is not possible) — RandomB, Dec 06 '18 at 11:42

danidiaz · Answer 1 · 2018-12-06T23:11:34.200

A possible solution to the existing/non-existing fields problem using type-level computation.

Some required extensions and imports:

{-# LANGUAGE DeriveGeneric, ScopedTypeVariables, DataKinds, KindSignatures,  
             TypeApplications, TypeFamilies, TypeOperators, FlexibleContexts #-}

import Data.Aeson
import Data.Proxy
import GHC.Generics
import GHC.TypeLits

Here's a data type (to be used promoted) that indicates if some field is absent or present. Also a type family that maps absent types to ():

data Presence = Present
              | Absent

type family Encode p v :: * where
    Encode Present v = v
    Encode Absent v = ()

Now we can define a parameterized record containing all possible fields, like this:

data Foo (a :: Presence) 
         (b :: Presence) 
         (c :: Presence) = Foo { 
                                  field1 :: Encode a Int,
                                  field2 :: Encode b Bool,
                                  field3 :: Encode c Char
                               } deriving Generic

instance (FromJSON (Encode a Int),
          FromJSON (Encode b Bool),
          FromJSON (Encode c Char)) => FromJSON (Foo a b c)

One problem: writing the full type for each combination of occurrences/absences would be tedious, especially if only a few fields are present each time. But perhaps we could define an auxiliary type synonym FooWith that let us mention only those fields that are present:

type family Mentioned (ns :: [Symbol]) (n :: Symbol) :: Presence where
    Mentioned '[]       _  = Absent
    Mentioned (n ': _)  n  = Present
    Mentioned (_ ': ns) n  = Mentioned ns n

-- the field names are repeated as symbols, how to avoid this?
type FooWith (ns :: [Symbol]) = Foo (Mentioned ns "field1") 
                                    (Mentioned ns "field2") 
                                    (Mentioned ns "field3")

Example of use:

ghci> :kind! FooWith '["field2","field3"]
FooWith '["field2","field3"] :: * = Foo 'Absent 'Present 'Present

Another problem: for each request, we must repeat the list of required fields two times: one in the URL ("fields=a,b,c...") and another in the expected type. It would be better to have a single source of truth.

We can deduce the term-level list of fields to be added to the URL from the type-level list of fields, by using an auxiliary type class Demote:

class Demote (ns :: [Symbol]) where
    demote :: Proxy ns -> [String]

instance Demote '[] where
    demote _ = []

instance (KnownSymbol n, Demote ns) => Demote (n ': ns) where
    demote _ = symbolVal (Proxy @n) : demote (Proxy @ns)

For example:

ghci> demote (Proxy @["field2","field3"])
["field2","field3"]

One possible problem with this approach is that it might increase compile times. — danidiaz, Dec 06 '18 at 18:23
You can also offer `Set`-like operations to build presence sets from one or more predefined or user-defined sets. You could even try using balanced search trees to try to speed them up; you're just calculating, not proving. — dfeuer, Dec 06 '18 at 23:35

How to handle variability of JSON objects in Haskell?

1 Answers1