I'm getting "extra data in buffer" error when trying to decode a gob in golang

Question

I'm encoding and sending multiple objects on a stream. I decode them as show in code below, keeping the connection open. I'm getting "extra data in buffer" error on the decode of all objects after the first.

func handleAggregatorConnection(conn net.Conn) {
        var connectionNumber = connectionCount
        connectionCount += 1
        log.Println("connection event: starting handle aggregator connection")


        dec := gob.NewDecoder(conn)

        var colorArrays map[string][]string
        colorArrayValue := &colorArrays

        var i P
        ai := &i


        for {
                //err := dec.Decode(colorArrayValue)
                err := dec.Decode(ai)
                if err == nil {
                        receivedColorResultFromAggregator = true
                        //log.Printf("received : %+v", colorArrayValue)
                        log.Println("received:", i)
                        aggregatorResultMap[connectionNumber] = *colorArrayValue
                        log.Println("control server: received object from aggregator ", aggregatorR\
esultMap)
                } else if err == io.EOF {
                        log.Println("reached end of stream while" +
                                "listening to aggregator")
                        delete(aggregatorResultMap, connectionNumber)
                        break
                } else {
                        log.Println("error decoding:", err)
                        break
                }
        }
        log.Println("connection event: closing aggregator connection")
        conn.Close()
}

If trying to reproduce the error, note that sending something like an int repeatetively actually works and doesn't show the error. I've tried with struct's and map's, and they will produce the error. — Rick Giuly, Aug 10 '17 at 01:51

Metal3d · Answer 1 · 2017-10-14T14:43:57.660

I don't agree your answer, I did myself a gobs reader that way:

func GetAll(db string) ([]*Record, error) {
    r := []*Record{}

    f, err := os.OpenFile(db, os.O_RDONLY, 0644)
    if err != nil {
        return nil, err
    }

    var rr error
    gdec := gob.NewDecoder(f)
    for rr != io.EOF {
        rec := Record{}
        rr = gdec.Decode(&rec)
        if rr != nil {
            continue
        }
        r = append(r, &rec)
    }

    return r, nil
}

And it works as exepected.

IMHO, you should not "break" if an error appear and continue to read data.

score -1 · Answer 2 · answered Aug 10 '17 at 01:45

Answering my own question: creating a new decoder for each decode operation makes it work.

func handleAggregatorConnection(conn net.Conn) {
        var connectionNumber = connectionCount
        connectionCount += 1
        log.Println("connection event: starting handle aggregator connection")

        for {

                dec := gob.NewDecoder(conn)

                var colorArrays map[string][]string
                colorArrayValue := &colorArrays

                var i P
                ai := &i



                //err := dec.Decode(colorArrayValue)
                err := dec.Decode(ai)
                if err == nil {
                        receivedColorResultFromAggregator = true
                        //log.Printf("received : %+v", colorArrayValue)
                        log.Println("received:", i)
                        aggregatorResultMap[connectionNumber] = *colorArrayValue
                        log.Println("control server: received object from aggregator ", aggregatorR\
esultMap)
                } else if err == io.EOF {
                        log.Println("reached end of stream while" +
                                "listening to aggregator")
                        delete(aggregatorResultMap, connectionNumber)
                        break
                } else {
                        log.Println("error decoding:", err)
                        break
                }
        }
        log.Println("connection event: closing aggregator connection")
        conn.Close()
}

I'm getting "extra data in buffer" error when trying to decode a gob in golang

2 Answers2