Merge a dynamic data structure in Go

Question

I have this incoming payload, that I cannot change.

{
"source": "some random source",
"table": "hosts_table",
"data": [
    ["address", "id", "services_with_info"],
    ["0.0.0.1", 1111, [
            ["service_3", "is very cool", 1],
            ["service_4", "is very cool", 2]
        ]
    ],
    ["0.0.0.2", 2222, [
            ["service_3", "is very cool", 3],
            ["service_4", "is very cool", 4]
        ]
    ]
]}

i need to take the first index of "data" and create a new JSON object, that looks like this...

"data": [{
"address": "0.0.0.1", 
"id": 1111, 
"services_with_info":[
    {
        "service_name": "service_1", 
        "service_message": "is very cool", 
        "service_id": 1
    }, 
    {...}
]}, 
{...}]

and then build an []Host's from it the data structure is 5k "hosts" long. I was able to map this to a struct, but need to get it into this format first. I understand how to unmarshal the JSON, but only if I can convert the payload to the above.

so data[0] will be different every time? or will those be constant values — poopoothegorilla, Jun 02 '16 at 17:03
Is there a name for the form of that incoming payload, i.e. is that a standard of some sort? If so maybe someone has already written a parser. Otherwise it looks like you'll need to write one. How constant is the real payload to the example? Always an identical format? — Snowman, Jun 02 '16 at 17:35

score 3 · Answer 1 · answered Jun 02 '16 at 17:40

You can use json.Unmarshal for this and parse data with your conditions. I'm just doing it for "data" and you can do same for "services_with_info"

b := []byte(`{
            "source": "some random source",
            "table": "hosts_table",
            "data": [
                ["address", "id", "services_with_info"],
                ["0.0.0.1", 1111, [
                        ["service_3", "is very cool", 1],
                        ["service_4", "is very cool", 2]
                    ]
                ],
                ["0.0.0.2", 2222, [
                        ["service_3", "is very cool", 3],
                        ["service_4", "is very cool", 4]
                    ]
                ]
            ]}`)
var f interface{}
err := json.Unmarshal(b, &f)
if err != nil {
    fmt.Println(err)
    return
}
m := f.(map[string]interface{})
result := make(map[string]interface{})
results := make(map[string][]map[string]interface{})
for k, v := range m {
    if k == "data" {
        s := v.([]interface{})
        header := make([]interface{}, 3)
        for i, u := range s {
            if i == 0 {
                header = u.([]interface{})
            } else {
                row := u.([]interface{})
                for j, k := range header {
                    result[k.(string)] = row[j]
                }
                results["data"] = append(results["data"], result)
            }
        }
    }
}
fmt.Println(results)

here "results" is "data" as required.

Darigaaz · Accepted Answer · 2016-06-03T01:47:09.887

I'm not sure if I understood what you wants.

May be some thing like this? Probably it needs some work, like make slice of pointers to structs instead of slice of structs to prevent allocation and copy, error handling, more custom logic to convert values, anonymize/incapsulate private structs used in the middle of conversion, add json tags to those structures etc.

I create custom Unmarshaller for Data field on IncomingPaylod: parsing expected data, converting it to []MyData and updating Data field with it.

I created custom Unmarshallers for expected_data and expected_services_with_info because we expect it as array of values (3 values: string, int and [array of string, int(?), int]), but I want to convert it to nice structs. If you dont like it, you can delete it, Unmarshal expected data to []interface{} and work with it like []interface{}{string, int, []interface{}{string, int, int} }. Easy to get it wrong, so i like structs more, its easier to read and maintain and refactor (i think there are more fields in you app).

https://play.golang.org/p/xHTvyhecra

package main

import (
    "encoding/json"
    "fmt"
    "strconv"
)

type IncomingPayload struct {
    Source string      `json:"source"`
    Table  string      `json:"table"`
    Data   MyDataSlice `json:"data"`
}
type MyDataSlice []MyData

type MyData struct {
    Address            string              `json:"address"`
    ID                 string              `json:"id"`
    Services_with_info []MyServiceWithInfo `json:"services_with_info"`
}

type MyServiceWithInfo struct {
    ServiceName    string `json:"service_name"`
    ServiceMessage string `json:"service_message"`
    ServiceID      int    `json:"service_id"`
}

type expected_data struct {
    IP   string
    ID   int
    Info []expected_services_with_info
}

type expected_services_with_info struct {
    Name string
    Desc string
    ID   int
}

func (ed *expected_data) UnmarshalJSON(buf []byte) error {
    tmp := []interface{}{&ed.IP, &ed.ID, &ed.Info}
    // converts ["address", "id", "services_with_info"] into a struct
    // will unmarshall "services_with_info" (ed.Info) with *expected_services_with_info.UnmarshalJSON
    json.Unmarshal(buf, &tmp)
    return nil
}

func (es *expected_services_with_info) UnmarshalJSON(buf []byte) error {
    tmp := []interface{}{&es.Name, &es.Desc, &es.ID}
    // converts ["service_3", "is very cool", 1] into a struct
    json.Unmarshal(buf, &tmp)
    return nil
}

func (md *MyDataSlice) UnmarshalJSON(p []byte) error {
    var incoming_data_slice []expected_data
    json.Unmarshal(p, &incoming_data_slice)
    //fmt.Println("incoming", incoming_data_slice)

    //transform incoming_data_slice to your needs using your data type
    for i := range incoming_data_slice {
        my_data := MyData{
            Address: incoming_data_slice[i].IP,               //copy
            ID:      strconv.Itoa(incoming_data_slice[i].ID), //some transformation
            //nil slice is totally fine, but if you wish you can do
            //Data: make(MyDataSlice, len(incoming_data_slice)),

        }

        //not sure what would be best: "i := range data" or "_, v := range data" (second one makes a copy? and causes allocation)
        for j := range incoming_data_slice[i].Info {
            tmp := MyServiceWithInfo{
                ServiceName: incoming_data_slice[i].Info[j].Name,
                ServiceMessage: incoming_data_slice[i].Info[j].Desc,
                ServiceID: incoming_data_slice[i].Info[j].ID,
            }
            my_data.Services_with_info = append(my_data.Services_with_info, tmp)
        }

        //and populate
        *md = append(*md, my_data)
    }

    return nil
}

func main() {

    test_json := `{
"source": "some random source",
"table": "hosts_table",
"data": [
    ["address", "id", "services_with_info"],
    ["0.0.0.1", 1111, [
            ["service_3", "is very cool", 1],
            ["service_4", "is very cool", 2]
        ]
    ],
    ["0.0.0.2", 2222, [
            ["service_3", "is very cool", 3],
            ["service_4", "is very cool", 4]
        ]
    ]
]}`

    var payload IncomingPayload
    json.Unmarshal([]byte(test_json), &payload)
    fmt.Println("payload", payload)

    buf, _ := json.MarshalIndent(payload, "", "\t")
    fmt.Println(string(buf))

}

This one looks good to me. The one optimization I would say is that you don't need the `json:` tags because the properties are formatted correctly. — Matt Williamson, Jun 05 '16 at 12:15
I also think he wants a json string output at the end to send off to another service or something. — Matt Williamson, Jun 05 '16 at 12:16
1) You need `json:` tags to out put key names in lowercase 2) That's done by `buf, _ := json.MarshalIndent(payload, "", "\t") fmt.Println(string(buf))`. — Darigaaz, Jun 05 '16 at 15:37

Merge a dynamic data structure in Go

2 Answers2

Linked