How to update value of two struct efficiently

Question

I have the following code which parses YAML files and needs to match values from one struct external and update the internal struct's type property.

For example, this is the yaml file(translated to bin for simplicity) and content which is parsed correctly

package main

import (
    "fmt"
    "gopkg.in/yaml.v2"
    "log"
)

//internal config model for parsing

type InternalModel struct {
    models []Model2 `yaml:"models"`
}

type Model2 struct {
    Name   string `yaml:"name"`
    Type   string `yaml:"type"`
    Target string `yaml:"target"`
}

var internal_config = []byte(`
 models:
   - name: myapp
     type: app1
     target: ./

   - name: myapp2
     type: app2
     target: ./
`)

type ExternalConfig struct {
    Landscape Zone `yaml:"Landscape"`
}

type Zone struct {
    Zone   string  `yaml:"zone"`
    Models []Model `yaml:"models"`
}

type Model struct {
    AppType     string `yaml:"app-type"`
    ServiceType string `yaml:"service-type"`
}


var external_config = []byte(`
Landscape:
  zone: zone1
  models:
    - app-type: app1
      service-type: GCP
    - app-type: app2
      service-type: AMAZON
  zone: zone2
  models:
    - app-type: app3
      service-type: AZURE
    - app-type: app4Í
      service-type: HEROKU
`)

func main() {

    // This is the internal config which needs updated

    internalConfiglYaml := InternalModel{}

    err := yaml.Unmarshal([]byte(internal_config), &internalConfiglYaml)
    if err != nil {
        log.Fatalf("error in model internalConfiglYaml: %v", err)
    }
    //fmt.Printf("%+v\n", internalConfiglYaml)

    //--------------------------Second config file-----------------------//
    //This is the external config yaml

    extConfigYaml := ExternalConfig{}

    err = yaml.Unmarshal([]byte(external_config), &extConfigYaml)
    if err != nil {
        log.Fatalf("error in model extConfigYaml: %v", err)
    }
    fmt.Printf("%+v\n", extConfigYaml)

    landscape := "zone1"

    modifiedConfig := ConvertTypes(internalConfiglYaml, extConfigYaml, landscape)

    fmt.Printf("%+v\n", modifiedConfig)

}

func ConvertTypes(int_cfg InternalModel, ext_config ExternalConfig, landscape string) (out_cfg InternalModel) {

    for _, module := range int_cfg.models {

        switch module.Type {
        case "app1":
            //here I hard-coded the value "GCP" but it should come from the yaml struct after parsing
            module.Type = "GCP" // should be something like ext_config.models.service-type when the key in the struct
        case "app2":
            //here I hard-coded the value "AMAZON" but it should come from the yaml struct after parsing
            module.Type = "AMAZON"

        }
    }

    return int_cfg
}

//At the end what I need to do is to get the internal yaml file to be changed to the following struct
//The changes are when the type=app-type I need to modify the type in the internal config, here its GCP and ruby

//internal_config_after_changes := []byte(`
//
//
//models:
// - name: myapp
//   type: GCP
//   target: ./
//
// - name: myapp2
//   type: AMAZON
//   target: ./
//
//
//`)

At the end what I need to do is to get the internal yaml file to be changed to the struct above internal_config_after_changes The changes are when the type=app-type I need to modify the type value in the internal_config, here from app1 to GCP and app2 to amazon

The problem is with the second loop which I should use to iterate on the external_config and the matching values, I'm not sure how to combine them both with efficient way...

Answer 1

Golang FAQ described regarding pointers to maps and slice:

Map and slice values behave like pointers: they are descriptors that contain pointers to the underlying map or slice data. Copying a map or slice value doesn't copy the data it points to. Copying an interface value makes a copy of the thing stored in the interface value. If the interface value holds a struct, copying the interface value makes a copy of the struct. If the interface value holds a pointer, copying the interface value makes a copy of the pointer, but again not the data it points to.

On iterating through the slice of model inside ConvertType you are actually creating a copy of []Models slice whose value.Type is not changing the value of original struct due that reason.

for _, module := range int_cfg.models{}

Above code snippet is creating a copy of int_cfg.models{}.

Index the slice model to point to the exact underlying array of slice Model to change the value as:

package main

import (
    "fmt"
    "log"
    "strings"

    "gopkg.in/yaml.v2"
)

//internal config model for parsing

type InternalModel struct {
    Models []Model2 `yaml:"models"`
}

type Model2 struct {
    Name   string `yaml:"name"`
    Type   string `yaml:"type"`
    Target string `yaml:"target"`
}

var internal_config = []byte(`
  models:
    - name: myapp
      type: app1
      target: ./

    - name: myapp2
      type: app2
      target: ./
`)

type ExternalConfig struct {
    Landscape []Zone `yaml:"Landscape"`
}

type Zone struct {
    Zone   string  `yaml:"zone"`
    Models []Model `yaml:"models"`
}

type Model struct {
    AppType     string `yaml:"app-type"`
    ServiceType string `yaml:"service-type"`
}

var external_config = []byte(`
Landscape:
  - zone: zone1
    models:
     - app-type: app1
       service-type: GCP
     - app-type: app2
       service-type: AMAZON
  - zone: zone2
    models:
     - app-type: app3
       service-type: AZURE
     - app-type: app4Í
       service-type: HEROKU
`)

func main() {

    //This is the internal config which needs to be update

    internalConfiglYaml := InternalModel{}

    err := yaml.Unmarshal(internal_config, &internalConfiglYaml)
    if err != nil {
        log.Fatalf("error in model internalConfiglYaml: %v", err)
    }
    fmt.Printf("%+v\n", internalConfiglYaml)

    //--------------------------Second config file-----------------------//
    //This is the external config yaml

    extConfigYaml := ExternalConfig{}
    // var response interface{}

    err = yaml.Unmarshal(external_config, &extConfigYaml)
    if err != nil {
        log.Fatalf("error in model extConfigYaml: %v", err)
    }
    fmt.Printf("%+v\n", extConfigYaml)

    landscape := "zone1"

    modifiedConfig := ConvertTypes(&internalConfiglYaml, extConfigYaml, landscape)

    fmt.Printf("%+v\n", modifiedConfig)

}

// ConvertTypes for changing the intConfig struct types
func ConvertTypes(int_cfg *InternalModel, ext_config ExternalConfig, landscape string) (out_cfg *InternalModel) {

    for _, module := range ext_config.Landscape {
        if module.Zone == landscape {
            for i, value := range module.Models {
                switch strings.Compare(value.AppType, int_cfg.Models[i].Type) {
                case 0:
                    //here I hard-coded the value "GCP" but it should come from the yaml struct after parsing
                    int_cfg.Models[i].Type = value.ServiceType // should be something like ext_config.models.service-type when the key in the struct
                default:
                }
            }
        }
    }

    return int_cfg
}

If you check above code snippet, you will also recognize that I have changed the struct.

type InternalModel struct {
    models []Model2 `yaml:"models"`
}

to first letter uppercase to make it exportable as:

type InternalModel struct {
    Models []Model2 `yaml:"models"`
}

Because of the struct InternalModel is unexportable field model was unable to parse the provided internal_config yaml which leads to empty []slice data after unmarshal yaml.

One more thing that I have noticed is you are converting bytes into bytes again. There is no need for that.

err := yaml.Unmarshal([]byte(internal_config), &internalConfiglYaml)

So I have changed it to just:

err := yaml.Unmarshal(internal_config, &internalConfiglYaml)

Since internal_config is already declared as byte using []byte in global variable.

Answer 2

If you're looking for how to apply the app-type to service-type mapping from the ExternalConfig, I suggest converting the array into a map and using it to look up the mapping:

func ConvertTypes(intCfg *InternalModel, extCfg ExternalConfig, landscape string) {
    // Make an app-type to service-type map.
    appToSvc := make(map[string]string, len(extCfg.Landscape.Models))
    for _, m := range extCfg.Landscape.Models {
        appToSvc[m.AppType] = m.ServiceType
    }

    // For each InternalModel model, check if there's an app-type to service-type
    // mapping. If so, use the service-type instead of the app-type.
    for i, model := range intCfg.Models {
        if t, ok := appToSvc[model.Name]; ok {
            intCfg.Models[i].Type = t
        }
    }
}

I've also fixed the pass-by-pointer and range loop assignment problems as suggested by @Himanshu and @Zan Lynx.

Answer 3

The first thing to consider is that you are working with two structures that are defined at run-time, thus you will not be able to use a single loop to match them. The next thing to consider is that your InternalConfig is the one least likely to change and should thus be the first structure to parse, where after the ExternalConfig is parsed and matched with it.

From reading your question, comments and the provided answers it also seems that there are a number of assumptions on the board which few have alluded to. I will attempt to address these if possible.

So let's give it a go:

package main

import (
    // import standard library packages first (alphabetically)
    "fmt"
    "log"

    // import third party packages next (also alphabetically)
    "gopkg.in/yaml.v2"

    // finally import your own packages here
)

// InternalConfig
//   - Let's call it InternalConfig so as to ensure consistent naming standards
type InternalConfig struct {
    Models []Imodel `yaml:"models"`
    // Exported to ensure access to external packages (such as yaml)
}

// Imodel - The internal model struct
type Imodel struct {
    Name   string `yaml:"name"`
    Type   string `yaml:"type"`
    Target string `yaml:"target"`
}

var iCfgb = []byte(`
models:
   - name: myapp
     type: app1
     target: ./

   - name: myapp2
     type: app2
     target: ./
`)

type ExternalConfig struct {
    Landscape Zone `yaml:"Landscape"`
}

type Zone struct {
    Zone   string  `yaml:"zone"`
    Models []Emodel `yaml:"models"`
}

type Emodel struct {
    AppType     string `yaml:"app-type"`
    ServiceType string `yaml:"service-type"`
}


var eCfgb = []byte(`
Landscape:
  zone: zone1
  models:
    - app-type: app1
      service-type: GCP
    - app-type: app2
      service-type: AMAZON
  zone: zone2
  models:
    - app-type: app3
      service-type: AZURE
    - app-type: app4Í
      service-type: HEROKU
`)

function main() {
    iCfgYaml := InternalConfig{}
    // don't need []byte(iCfgb) as iCfgb is already a byte slice
    // We are also using a shorthand if statement as we will only use err in the
    // if scope, this is because our config is passed by reference and not value
    if err := yaml.Unmarshal(iCfgb, &iCfgYaml); err != nil {
        log.Fatalf("error in model InternalConfig: %v", err)
    }

    eCfgYaml := ExternalConfig{}
    if err := yaml.Unmarshal(eCfgb, &eCfgYaml); err != nil {
        log.Fatalf("error in model ExternalConfig: %v", err)
    }

    // From here there are clearly some assumptions made, here is my take:
    //   1. I do not see any purpose in the landscape variable, the value "zone1"
    //      can be obtained from eCfgYaml.Landscape.Zone;
    //   2. From the question it seems you want to update the InternalConfig
    //      values with values from the ExternalConfig. However, from the code you
    //      created a modified copy. I will assume this was for debugging purposes
    //      and skip the creating of another copy;

    // I have renamed ConvertTypes to UpdateInternalState to accurately reflect
    // the behavior of the function. It now returns an error instead of a copy of
    // InternalConfig (any state altering functions should most likely return an
    // error). It receives a pointer of InternalConfig and a value copy of
    // ExternalConfig. This is because we will be altering the state of
    // InternalConfig
    if err := UpdateInternalState(&iCfgYaml, eCfgConfig); err != nil {
        log.Fatalf("Error updating internal config: %v", err)
    }

    // Assume the print is for debugging, thus I denote it with DBG. Makes it easy
    // to find and remove from production code.
    fmt.Printf("DBG: %+v\n", iCfgYaml)
}

func UpdateInternalState(iCfg *InternalConfig, eCfg ExternalConfig) error {
    // Firstly, we need to track the internal config models. We will create an
    // index lookup map (ilm) for it.
    ilm := make(map[string]int)

    // Now we iterate through our internal models and update the index:
    for i, v := range iCfg.Models {
        ilm[v] = i
    }

    // Now we iterate through the external config and update the internal config
    // state accordingly:
    for _, em := range eCfg.Landscape.Models {
        // We find the index of the specified app type in the index lookup
        // We use a shorthand if statement as we will only use i and ok within
        // the scope of the if (if you need it outside you can't use shorthand)
        // If there is no match nothing happens, but if found it will update to
        // the corresponding specified value in the external config.
        if i, ok := ilm[em.AppType]; ok {
            iCfg.Models[i].Type = em.ServiceType
        }
    }

    // At any point within this function you are able to do more robust error
    // checking, such as ensuring the index lookup map must have a corresponding
    // entry, and return any errors encountered.
    return nil
}

Answer 4

This question is definitely a duplicate but I can't find one to link right now.

Your problem is that the for loop is creating a copy of each struct. Writing into element which is just a copy of xtr[i] never changes the original xtr[i]. What you do to fix it is to use the slice index instead.

Change

for _, element := range xtr {
    switch element.vType {
    case "app1":
        //user1 value is coming from the &y parsed data of the yaml
        element.vType = "user1"
    case "app2":
         //user2 value is coming from the &y parsed data
        element.vType = "user2"

    }
}

Into

for i := range xtr {
    switch xtr[i].vType {
    case "app1":
        //user1 value is coming from the &y parsed data of the yaml
        xtr[i].vType = "user1"
    case "app2":
         //user2 value is coming from the &y parsed data
        xtr[i].vType = "user2"

    }
}