Unmarshal hcl to struct using viper

Question

Trying to Unmarshal a hcl config file to a struct, using viper, this error is returned: 1 error(s) decoding:\n\n* 'NATS' expected a map, got 'slice'. What is missing?

The code:

func lab() {
    var c conf

    // config file
    viper.SetConfigName("draft")
    viper.AddConfigPath(".")
    viper.SetConfigType("hcl")
    if err := viper.ReadInConfig(); err != nil {
        log.Error(err)
        return
    }

    log.Info(viper.Get("NATS")) // gives [map[port:10041 username:cl1 password:__Psw__4433__ http_port:10044]]

    if err := viper.Unmarshal(&c); err != nil {
        log.Error(err)
        return
    }

    log.Infow("got conf", "conf", c)
}

type conf struct {
    NATS struct {
        HTTPPort int
        Port     int
        Username string
        Password string
    }
}

And the config file (draft.hcl inside current directory):

NATS {
    HTTPPort = 10044
    Port     = 10041
    Username = "cl1"
    Password = "__Psw__4433__"
}

Edit

Have checked this struct with hcl package and it gets marshaled/unmarshalled correctly. Also this works correctly with yaml and viper.

There is a difference between these two where log.Info(viper.Get("NATS")) is called. While the hcl version returns a slice of maps, the yaml version returns a map: map[password:__psw__4433__ httpport:10044 port:10041 username:cl1].

@Kavesh can you share the yaml version which you are referring? — Anuruddha, Jan 13 '18 at 15:25
@Anuruddha This gist: https://gist.github.com/dc0d/7cb146f6650259d6f299223ff410ecc9 — Kaveh Shahbazian, Jan 13 '18 at 15:36

score 5 · Answer 1 · answered Jan 13 '18 at 15:05

Your conf struct is not matching the HCL. When converted to json the HCL looks like below

{
"NATS": [
    {
      "HTTPPort": 10044,
      "Password": "__Psw__4433__",
      "Port": 10041,
      "Username": "cl1"
    }
  ]
}

So the Conf Struct should look like this

type Conf struct {
    NATS []struct{
        HTTPPort int
        Port     int
        Username string
        Password string
    }
}

Modified code

package main
import (
  "log"
  "github.com/spf13/viper"
  "fmt"
)

type Conf struct {
    NATS []struct{
        HTTPPort int
        Port     int
        Username string
        Password string
    }
}

func main() {
    var c Conf
    // config file
    viper.SetConfigName("draft")
    viper.AddConfigPath(".")
    viper.SetConfigType("hcl")
    if err := viper.ReadInConfig(); err != nil {
        log.Fatal(err)
    }
    fmt.Println(viper.Get("NATS")) // gives [map[port:10041 username:cl1 password:__Psw__4433__ http_port:10044]]

    if err := viper.Unmarshal(&c); err != nil {
        log.Fatal(err)
    }
    fmt.Println(c.NATS[0].Username)
}

I've checked it and this works correctly with `hcl` package. Also it works correctly with `viper` and `yaml`. — Kaveh Shahbazian, Jan 13 '18 at 15:13
I have updated the question. Please consider reviewing the edit too. — Kaveh Shahbazian, Jan 13 '18 at 15:19

score 2 · Answer 2 · answered Jun 26 '20 at 21:29

I know this question is more than two years old now, but I came across the same issue recently.

I'm using viper to be able to load different configuration files into a Go struct, allowing configuration in JSON, YAML, TOML, HCL, just pick your favourite :)

HCL file format does wrap a map into a slice because it allows redefining a section like:

section = {
  key1 = "value"
}

section = {
  key2 = "value"
}

which is something that is not supported by the other formats.

And here's how I fixed it:

My solution implies each new block will override any previous definition of the same key, and keep all the others. You can do some merging magic but I didn't need to.

You need to make a hook to convert a slice of maps into a map:

// sliceOfMapsToMapHookFunc merges a slice of maps to a map
func sliceOfMapsToMapHookFunc() mapstructure.DecodeHookFunc {
    return func(from reflect.Type, to reflect.Type, data interface{}) (interface{}, error) {
        if from.Kind() == reflect.Slice && from.Elem().Kind() == reflect.Map && (to.Kind() == reflect.Struct || to.Kind() == reflect.Map) {
            source, ok := data.([]map[string]interface{})
            if !ok {
                return data, nil
            }
            if len(source) == 0 {
                return data, nil
            }
            if len(source) == 1 {
                return source[0], nil
            }
            // flatten the slice into one map
            convert := make(map[string]interface{})
            for _, mapItem := range source {
                for key, value := range mapItem {
                    convert[key] = value
                }
            }
            return convert, nil
        }
        return data, nil
    }
}

then you need to create a DecodeHook:

configOption := viper.DecodeHook(mapstructure.ComposeDecodeHookFunc(
        sliceOfMapsToMapHookFunc(),
        mapstructure.StringToTimeDurationHookFunc(),
        mapstructure.StringToSliceHookFunc(","),
    ))

the two other hooks are the default ones so you might want to keep them

then you pass the option to the Unmarshal method

viper.Unmarshal(&c, configOption)

With this method you don't need a slice around your structs or your maps. Also that makes it compatible with the other configuration file formats

Unmarshal hcl to struct using viper

2 Answers2