How To Pull Secrets as Config File in Golang

Question

So I am pretty new to Go and I am curious about how to import passwords and URLs dynamically without exposing them within my scripts. In python, I was able to implement this with a JSON payload and would basically import JSON, load the payload, and specify whenever I needed to pass a spec that needed secrecy.

Here is My Go Script:

package main

import (
    "io/ioutil"
    "net/http"
    "fmt"

    "gopkg.in/yaml.v2"
)
// curl -u <username>:<password> <some_url>

func main() {
    type Config struct {
    URL      string `yaml:"url"`
    Username string `yaml:"username"`
    Token    string `yaml:"token"`
    }

    func readConfig() (*Config, error) {
    config := &Config{}
    cfgFile, err := ioutil.ReadFile("config.yaml")
    if err != nil {
        return nil, err
    }
    err = yaml.Unmarshal(cfgFile, config)
    return config, err
    }

    req, err := http.NewRequest("GET", config.URL, nil)
    if err != nil {
        // handle err
    }

    req.SetBasicAuth(config.Username, config.Token)

    resp, err := http.DefaultClient.Do(req)
    if err != nil {
        // handle err
    }

    defer resp.Body.Close()
    body, _ := ioutil.ReadAll(resp.Body)
    fmt.Println(string(body))
}

As one can imagine I am not having much luck in loading my passwords, when I change the following with direct tokens exposing my passwords in the scripts it functions and produces a JSON payload in Jira.

config.URL
config.Username
config.Token

So if I have a config file in YAML as such:

config:
    URL: "<some_URL>"
    Username: "<some_username>"
    Token: "<some_token>"

How can I load the YAML file into my script? How can I load a JSON equivalent?

{
  "config": {
    "URL": "<some_URL>"
    "Username": "<some_username>",
    "Token": "<some_token>"
  }
}

Your question refers a lot to secrets/secrecy but seems to only be about loading JSON or maybe YAML? And I'm not sure what you mean by "exposing them within my script"? Can you clarify the question? Also note that Go is not a scripting language, there is no such thing as a "Go script". — Adrian, Jul 07 '21 at 14:50
You can also use environment variables and dockerize your application. You can check this repo if you like the idea https://github.com/joho/godotenv — Nurio Fernández, Jul 07 '21 at 19:48

Clark McCauley · Accepted Answer · 2021-07-07T19:58:03.550

Okay so there are a couple problems here.

You can't declare functions inside of another function unless you use variable declaration syntax

func main() {
    // This
    var myFunc = func() {
        // ...
    }
    // Not this
    func myFunc() {
        // ...
    }
}

The Config struct is telling the YAML unmarshaler to expect. Your struct should have yaml tags that match the casing and structure of the yaml file.

// It is expecting this
url: "<some_URL>"
username: "<some_username>"
token: "<some_token>"

// Your yaml looks like this
config:
  Url: "<some_URL>"
  Username: "<some_username>"
  Token: "<some_token>"

The following works for me:

package main

import (
    "fmt"
    "io/ioutil"
    "net/http"

    "gopkg.in/yaml.v2"
)

type YAMLFile struct {
    Config Config `yaml:"config"`
}
type Config struct {
    URL      string `yaml:"url"`
    Username string `yaml:"username"`
    Token    string `yaml:"token"`
}

func main() {
    config, err := readConfig()
    if err != nil {
        panic(err)
    }
    fmt.Printf("%+v", config)
    req, err := http.NewRequest("GET", config.URL, nil)
    if err != nil {
        panic(err)
    }
    req.SetBasicAuth(config.Username, config.Token)
    resp, err := http.DefaultClient.Do(req)
    if err != nil {
        panic(err)
    }
    defer resp.Body.Close()
    body, err := ioutil.ReadAll(resp.Body)
    if err != nil {
        panic(err)
    }
    fmt.Println(string(body))
}

func readConfig() (*Config, error) {
    config := &YAMLFile{}
    cfgFile, err := ioutil.ReadFile("./config.yaml")
    if err != nil {
        return nil, err
    }
    err = yaml.Unmarshal(cfgFile, config)
    return &config.Config, err
}

Thanks, Clark that really helps, I am curious though if you changed the structure at all of the config.yaml or if you kept it the same though? For me it's not working, also is it okay to have the readConfig() function after the main or is it best practice to but it before and put the func main () {} after, just curious as I am very new to Go. — R. Barrett, Jul 07 '21 at 19:38
@R.Barrett make sure your YAML keys in the file are lowercase to match the struct tags, i.e. `URL string 'yaml:"url"'` requires that the YAML contain `url`, not `Url` like your config file currently has. You can change this to whatever you'd like, it just needs to match. I don't know that there is a standard for how to structure the functions, I just put them in order of evaluation -- the main function runs first so I put it above other functions. This allows the reader to read the code in the "same" chronological order the compiler would. Again, personal preference. — Clark McCauley, Jul 07 '21 at 19:59
Thanks a bunch, Clark, you have been very helpful. I appreciate all of the help and all of the insights. — R. Barrett, Jul 08 '21 at 13:54

score 1 · Answer 2 · answered Jul 07 '21 at 23:33

Barrett,

Here is a config library I made some time ago to solve that problems: https://github.com/fulldump/goconfig.

The use is simple:

Define a struct with all the config you need:

type Config struct {
    URL      string
    Username string
    Token    string
}

Instantiate a variable with that type and fill it with the default values:

c := &Config{
    URL:     "http://default/url"
    Username: "default username"
}

Automatic fill your config variable with data from environment, command line arguments and/or json file with the folloging line:

goconfig.Read(c)

For example, in your case, you can pass a JSON file as follows ./my-binary -token "my_secret_token" -config my-config-file.json to read all config keys from a file but overwrite the token just before launching with an argument.

How To Pull Secrets as Config File in Golang

2 Answers2