How to add a new entry to yaml file via code

Question

I’ve yaml file and I need to add to it data on runtime using go code The path is like following, I mean

This is the yaml file with one entry under sif of snk_dev

spec:
  mec:
    tolerations:
    - effect: NoSchedule
      key: WorkGroup
      operator: Equal
      value: goxy
    resources:
      requests:
        cpu: 100m
        memory: 1Gi
    customConfig:
      sif:
        prom_exporter:
          type: prometheus_exporter
        snk_dev:   
          type: sk_hc_logs
          inputs:
            - tesslt
          ep: ${NT}
          dken: ${SN}
          encoding:
            codec: "json"
          index: us
          compression: gzip
          buffer:
            type: memory
  

under the following yaml path I need to add a new entry

spec->mec->customConfig->sif a new entry snd_prd

    snk_prod:   
      type: sk_hc_logs
      inputs:
        - tesslt
      ep: ${NT}
      dken: ${SN}
      encoding:
        codec: "json"
      index: us
      compression: gzip
      buffer:
        type: memory

We are using kustomize and I wonder if there is a way to do it via code, I mean prefer in advance the file that i need to add and to add it in runtime Or maybe better of using the https://github.com/go-yaml/yaml

Answer 1

Yaml codec supports decoding to map[string]any and encoding such map into yaml.

The idea is to decode both the document and the extra tree, then put the additional map under the required path and then encode back.

type YamlObject map[string]any

func main() {
    // Parse the initial document
    doc := make(YamlObject)
    yaml.Unmarshal([]byte(document), &doc)
    // Parse the additional nodes
    addon := make(YamlObject)
    yaml.Unmarshal([]byte(extra), &addon)
    // Find the node by the path
    node := findChild(doc, "spec", "mec", "customConfig", "sif")
    if node == nil {
        panic("Must not happen")
    }
    // Add the keys from the additional document
    // under the specified path
    for key, val := range addon {
        (*node)[key] = val
    }
    // Output the modified document
    outDoc, _ := yaml.Marshal(doc)
    println(string(outDoc))
}

func findChild(obj YamlObject, path ...string) *YamlObject {
    if len(path) == 0 {
        return &obj
    }
    key := path[0]
    child, ok := obj[key]
    if !ok {
        return nil
    }
    obj, ok = child.(YamlObject)
    if !ok {
        return nil
    }
    return findChild(obj, path[1:]...)
}

Full example https://go.dev/play/p/pTdXR53p0mq

Output:

spec:
    mec:
        customConfig:
            sif:
                prom_exporter:
                    type: prometheus_exporter
                snk_dev:
                    buffer:
                        type: memory
                    compression: gzip
                    dken: ${SN}
                    encoding:
                        codec: json
                    ep: ${NT}
                    index: us
                    inputs:
                        - tesslt
                    type: sk_hc_logs
                snk_prod:
                    buffer:
                        type: memory
                    compression: gzip
                    dken: ${SN}
                    encoding:
                        codec: json
                    ep: ${NT}
                    index: us
                    inputs:
                        - tesslt
                    type: sk_hc_logs
        resources:
            requests:
                cpu: 100m
                memory: 1Gi
        tolerations:
            - effect: NoSchedule
              key: WorkGroup
              operator: Equal
              value: goxy

YAML codec outouts keys in the alphabetic order

Answer 2

The key here is to generate the equivalent Go structs to model your YAML and use the Marhshal/Unmarshal functions from gopkg.in/yaml.v3 package.

You could use a tool like yaml-to-go, to autogenerate the structs needed for your YAML and then perform any more additional customisations on top of it. The answer below takes the definitions from such a tool.

Your YAML structure could be improved a bit, because snk_dev & snk_prod fields look alike. You should defining a common type for both these and define a list of YAML objects, which in-turn would have converted into a slice of structs of that particular type. But since the original YAML retains the two of them as different entities, your structs also need to be different.

Based on your comment to the answer, that the fields snk_dev & snk_prod are dynamically derived, it would make sense to define your CustomConfig to be a map[string]interface{} to allow for dynamic key names.

package main

import (
    "fmt"
    "log"

    "gopkg.in/yaml.v3"
)

type YAMLData struct {
    Spec Spec `yaml:"spec"`
}
type Tolerations struct {
    Effect   string `yaml:"effect"`
    Key      string `yaml:"key"`
    Operator string `yaml:"operator"`
    Value    string `yaml:"value"`
}
type Requests struct {
    CPU    string `yaml:"cpu"`
    Memory string `yaml:"memory"`
}
type Resources struct {
    Requests Requests `yaml:"requests"`
}
type PromExporter struct {
    Type string `yaml:"type"`
}
type Encoding struct {
    Codec string `yaml:"codec"`
}
type Buffer struct {
    Type string `yaml:"type"`
}
type SifConfig struct {
    Type        string   `yaml:"type"`
    Inputs      []string `yaml:"inputs"`
    Ep          string   `yaml:"ep"`
    Dken        string   `yaml:"dken"`
    Encoding    Encoding `yaml:"encoding"`
    Index       string   `yaml:"index"`
    Compression string   `yaml:"compression"`
    Buffer      Buffer   `yaml:"buffer"`
}
type CustomConfig struct {
    Sif map[string]interface{} `yaml:"sif"`
}
type Mec struct {
    Tolerations  []Tolerations `yaml:"tolerations"`
    Resources    Resources     `yaml:"resources"`
    CustomConfig CustomConfig  `yaml:"customConfig"`
}
type Spec struct {
    Mec Mec `yaml:"mec"`
}

var data = `spec:
  mec:
    tolerations:
    - effect: NoSchedule
      key: WorkGroup
      operator: Equal
      value: goxy
    resources:
      requests:
        cpu: 100m
        memory: 1Gi
    customConfig:
      sif:
        prom_exporter:
          type: prometheus_exporter
        snk_dev:   
          type: sk_hc_logs
          inputs:
            - tesslt
          ep: ${NT}
          dken: ${SN}
          encoding:
            codec: "json"
          index: us
          compression: gzip
          buffer:
            type: memory
`

func main() {
    t := YAMLData{}
    err := yaml.Unmarshal([]byte(data), &t)
    if err != nil {
        log.Fatalf("error: %v", err)
    }

    config := &t.Spec.Mec.CustomConfig
    config.Sif["snk_prod"] = SifConfig{
        Type:        "sk_hc_logs",
        Inputs:      []string{"tesslt"},
        Ep:          "${NT}",
        Dken:        "${SN}",
        Encoding:    Encoding{Codec: "json"},
        Index:       "us",
        Compression: "gzip",
        Buffer:      Buffer{Type: "memory"},
    }

    yamlBytes, err := yaml.Marshal(t)
    if err != nil {
        log.Fatalf("error: %v", err)
    }
    fmt.Println(string(yamlBytes))

}

The yamlBytes can be used further to be written as a separate file, which is left out of the above.

Go playground