json.Unmarshal and "push" to array of structs

Question

I have a task to read records from AWS MSK (in AWS lambda) and prepare some kind of payload to send it to Facebook. Records from AWS MSK are base64 encoded, but once I decode them, I get JSON string. Now I don't understand how json.Unmarshal(decodedParams) in to &jsonPayload.Data which is array of structs

type Payload struct {
    Data Data `json:"data,required"`
}

type Data []struct{
    Event             string         `json:"event_name,required"`
    EventTime         int            `json:"event_time,required"`
    EventSourceUrl    string         `json:"event_source_url,omitempty,required"`
    EventActionSource string         `json:"action_source,omitempty,required"`
    EventId           int            `json:"event_id,required"`
    UserData          UserDataType   `json:"user_data,required"`
    CustomData        CustomDataType `json:"custom_data,omitempty"`
}

type CustomDataType struct {
    SearchString     string      `json:"search_string,omitempty"`
    Value            json.Number `json:"value,omitempty"`
    Currency         string      `json:"currency,omitempty"`
}
type UserDataType struct {
    IpAddress string `json:"client_ip_address,omitempty,required"`
    UserAgent string `json:"client_user_agent,omitempty,required"`
}

// ProcessEvent function Using AWS Lambda computed event
func ProcessEvent(event events.KafkaEvent) {

    jsonPayload := Payload{}

    for _, mapper := range event.Records {

        for _, record := range mapper {

            // Base64 decode string from MSK Kafka
            decodedParams, err := base64.StdEncoding.DecodeString(record.Value)
            if err != nil {
                log.Fatal("Error decoding fb event params: ", err)
            }

            // json.Unmarshal and push to Data []structs???
            unmErr := json.Unmarshal(decodedParams, &jsonPayload.Data)
            if unmErr != nil {
                fmt.Println("UNMARSHAL ERROR")
                fmt.Println(unmErr)
            }
        }
    }
}

func main() {
    lambda.Start(ProcessEvent)
}

End result of payload should be similar to this one

{
   "data":[
      {
         "event_name":"Purchase",
         "event_time":1627975460,
         "action_source":"email",
         "user_data":{
            "em":[
               "7b17fb0bd173f625b58636fb796407c22b3d16fc78302d79f0fd30c2fc2fc068"
            ],
            "ph":[
               null
            ]
         },
         "custom_data":{
            "currency":"USD",
            "value":"142.52"
         }
      },
      {
         "event_name":"PageView",
         "event_time":1627975460,
         "action_source":"email"
      }
   ]
}

Currently I'm getting error

json: cannot unmarshal object into Go value of type main.Data

Since I'm fairly new with GO I wonder if I'm on the right path and how to push decoded json string in to Data []struct? If you need any additional informations, please let me know and I will provide. Thank you!

Well you are on the right path, it seems to me. What issues did you encounter? Did you get an error? Does the `decodedParams` json match the structure of the `Data` type? — mkopriva, Aug 03 '21 at 07:34
Oh sorry, currently i'm getting error json: cannot unmarshal object into Go value of type main.Data — Valor_, Aug 03 '21 at 07:39
Well that means that `decodeParams` contains a json object and **not a json array**. But you've defined `Data` as `[]struct`, hence json.Unmarshal expects a json array and not an object. Change `Data []struct{...` to `Data struct{...` and you should be ok, as far as unmarshaling goes. — mkopriva, Aug 03 '21 at 07:41
If `decodedParams` can sometimes contain a json object and sometime a json array, what you can do is to check the first and last byte, and if it is `{`,`}` and not `[`,`]` then you know it's an object and not an array, and you can "normalize" the json by prepending `[` and appending `]`. — mkopriva, Aug 03 '21 at 07:45
Note also that if all you need to do with `decodedParams` is to send it along to facebook then you don't need to unmarshal it, you can just send it as is using the `json.RawMessage` type. e.g. `type Payload struct { Data json.RawMessage }` and then, after decoding you would do `jsonPayload.Data = json.RawMessage(decodedParams)`. — mkopriva, Aug 03 '21 at 07:56

score 2 · Accepted Answer · answered Aug 03 '21 at 08:09

Every single iteration of event.Records is likely to overwrite whatever you have the, more of that try to imagine .Data as slice of []data

type data struct {
    Event             string         `json:"event_name,required"`
    EventTime         int            `json:"event_time,required"`
    EventSourceUrl    string         `json:"event_source_url,omitempty,required"`
    EventActionSource string         `json:"action_source,omitempty,required"`
    EventId           int            `json:"event_id,required"`
    UserData          UserDataType   `json:"user_data,required"`
    CustomData        CustomDataType `json:"custom_data,omitempty"`
}

type Data []data

So you will have code that unmarshal and push to Data slice small data struct, similar to this.

var d data
unmErr := json.Unmarshal(decodedParams, &d)
if unmErr != nil {
    fmt.Println("UNMARSHAL ERROR")
    fmt.Println(unmErr)
}

jsonPayload.Data = append(jsonPayload.Data, d)

json.Unmarshal and "push" to array of structs

1 Answers1