Decode very big json to array of structs

Question

I have a web application which have a REST API, get JSON as input and perform transformations of this JSON.

Here is my code:

func (a *API) getAssignments(w http.ResponseWriter, r *http.Request) {

   var document DataPacket
   err := json.NewDecoder(r.Body).Decode(&document)
   if err != nil {
       a.handleJSONParseError(err, w)
      return
   }

   // transformations

JSON which I get is a collection of structs. External application use my application and send me very big json files (300-400MB). Decode this json at the one moment of time takes a very big time and amount of memory.

Is there any way to work with this json as stream and decode structs from this collection one by one ?

Also: [Go - Decode JSON as it is still streaming in via net/http](https://stackoverflow.com/questions/44307219/go-decode-json-as-it-is-still-streaming-in-via-net-http/44308109#44308109) — icza, Jan 22 '19 at 20:55

peterSO · Accepted Answer · 2019-01-22T20:55:54.507

First, read the documentation.

Package json

import "encoding/json"

func (*Decoder) Decode
func (dec *Decoder) Decode(v interface{}) error
Decode reads the next JSON-encoded value from its input and stores it in the value pointed to by v.

Example (Stream): This example uses a Decoder to decode a streaming array of JSON objects.

Playground: https://play.golang.org/p/o6hD-UV85SZ

package main

import (
    "encoding/json"
    "fmt"
    "log"
    "strings"
)

func main() {
    const jsonStream = `
    [
        {"Name": "Ed", "Text": "Knock knock."},
        {"Name": "Sam", "Text": "Who's there?"},
        {"Name": "Ed", "Text": "Go fmt."},
        {"Name": "Sam", "Text": "Go fmt who?"},
        {"Name": "Ed", "Text": "Go fmt yourself!"}
    ]
`
    type Message struct {
        Name, Text string
    }
    dec := json.NewDecoder(strings.NewReader(jsonStream))

    // read open bracket
    t, err := dec.Token()
    if err != nil {
        log.Fatal(err)
    }
    fmt.Printf("%T: %v\n", t, t)

    // while the array contains values
    for dec.More() {
        var m Message
        // decode an array value (Message)
        err := dec.Decode(&m)
        if err != nil {
            log.Fatal(err)
        }

        fmt.Printf("%v: %v\n", m.Name, m.Text)
    }

    // read closing bracket
    t, err = dec.Token()
    if err != nil {
        log.Fatal(err)
    }
    fmt.Printf("%T: %v\n", t, t)

}

Decode very big json to array of structs

1 Answers1