How to list unknown fields after json parsing

Question

Imagine we have following Go structs:

type Config struct {
    Name   string  `json:"name,omitempty"`
    Params []Param `json:"params,omitempty"`
}

type Param struct {
    Name  string `json:"name,omitempty"`
    Value string `json:"value,omitempty"`
}

and following json:

{
    "name": "parabolic",
    "subdir": "pb",
    "params": [{
        "name": "input",
        "value": "in.csv"
    }, {
        "name": "output",
        "value": "out.csv",
        "tune": "fine"
    }]
}

and we do unmarshalling:

cfg := Config{}
if err := json.Unmarshal([]byte(cfgString), &cfg); err != nil {
    log.Fatalf("Error unmarshalling json: %v", err)
}
fmt.Println(cfg)

https://play.golang.org/p/HZgo0jxbQrp

Output would be {parabolic [{input in.csv} {output out.csv}]} which makes sense - unknown fields were ignored.

Question: how to find out which fields were ignored?

I.e. getIgnoredFields(cfg, cfgString) would return ["subdir", "params[1].tune"]

(There is a DisallowUnknownFields option but it's different: this option would result Unmarshal in error while question is how to still parse json without errors and find out which fields were ignored)

there might be another way, if you encode the struct into json again, then compare the two json, you will get difference. you can use `https://github.com/yudai/gojsondiff` — alamin, Mar 06 '19 at 18:42
@Md.AlaminMahamud right, I also thought about that, but then we will have json like `{"one": "one", "two": "two"}` and it would be deserialized as struct and then back as `{"two":"two", "one": "one"}`. There would be diff, but it would be not helpful — yname, Mar 07 '19 at 06:36

score 0 · Answer 1 · answered Mar 13 '19 at 08:37

Not sure if that is the best way but what I did is:

If current-level type is map:
1. Check that all map keys are known.
  - It could be either if keys are struct field names or map keys.
  - If not known - add to list of unknown fields
2. Repeat recursively for value that corresponds to each key
If current level type is array:
1. Run recursively for each element

Code:

// ValidateUnknownFields checks that provided json
// matches provided struct. If that is not the case
// list of unknown fields is returned.
func ValidateUnknownFields(jsn []byte, strct interface{}) ([]string, error) {
    var obj interface{}
    err := json.Unmarshal(jsn, &obj)
    if err != nil {
        return nil, fmt.Errorf("error while unmarshaling json: %v", err)
    }
    return checkUnknownFields("", obj, reflect.ValueOf(strct)), nil
}

func checkUnknownFields(keyPref string, jsn interface{}, strct reflect.Value) []string {
    var uf []string
    switch concreteVal := jsn.(type) {
    case map[string]interface{}:
        // Iterate over map and check every value
        for field, val := range concreteVal {
            fullKey := fmt.Sprintf("%s.%s", keyPref, field)
            subStrct := getSubStruct(field, strct)
            if !subStrct.IsValid() {
                uf = append(uf, fullKey[1:])
            } else {
                subUf := checkUnknownFields(fullKey, val, subStrct)
                uf = append(uf, subUf...)
            }
        }
    case []interface{}:
        for i, val := range concreteVal {
            fullKey := fmt.Sprintf("%s[%v]", keyPref, i)
            subStrct := strct.Index(i)
            uf = append(uf, checkUnknownFields(fullKey, val, subStrct)...)
        }
    }
    return uf
}

Full version: https://github.com/yb172/json-unknown/blob/master/validator.go

How to list unknown fields after json parsing

1 Answers1