1

I'm decoding with gob several objects fetched from a key/value database called "bitcask". When I try to decode all of them one by one I get the "extra data in buffer" error from the gob decoder but exclusively for the first element that was added in the database and only after I've already fetched them at least once.

What am I doing wrong?

// Just the nest of all the bugs.
type Nest struct {
    buf bytes.Buffer
    dec *gob.Decoder
    enc *gob.Encoder
    db  *bitcask.Bitcask
    sync.Mutex
}

// Saves the bug into the nest.
func (n *Nest) Put(id int64, b Bug) error {
    n.Lock()
    defer n.Unlock()
    if err := n.enc.Encode(b); err != nil {
        return err
    }
    defer n.buf.Reset()
    return n.db.Put(itosl(id), n.buf.Bytes())
}

// Retrieves a bug from the nest.
func (n *Nest) Get(id int64) (Bug, error) {
    var bg Bug

    b, err := n.db.Get(itosl(id))
    if err != nil {
        return bg, err
    }
    n.Lock()
    defer n.Unlock()
    if _, err = n.buf.Write(b); err != nil {
        return bg, err
    }
    defer n.buf.Reset()
    return bg, n.dec.Decode(&bg) // error extra data in buffer 
}

Note: the "Get" function get's called for every ID in the database everytime the API endpoint is called.

The structs I'm encoding/decoding are the following:

type Comment struct {
    Date   int64  `json:"date"`
    Text   string `json:"text"`
    Author string `json:"author"`
}

type Bug struct {
    Id       int64     `json:"id"`
    Body     string    `json:"body"`
    Open     bool      `json:"open"`
    Tags     []string  `json:"tags"`
    Date     int64     `json:"date"`
    Comments []Comment `json:"comments"`
    Author   string    `json:"author"`
}

Also when I try to use a new decoder and buffer each time the "Get" function gets called (as I've seen in an answer to a similar question) the decode operation results in the following error: gob: unknown type id or corrupted data.

Refer to this link for the complete source code: https://github.com/NicoNex/ladybug

NicoNex
  • 469
  • 2
  • 5
  • 15
  • 1
    Because type specifications are encoded once per gob stream, you must create a new encoder in each invocation of Put and a new decoder in each invocation of Get. – Charlie Tumahai Oct 31 '20 at 15:42
  • 1
    But on the [gob page](https://golang.org/pkg/encoding/gob/) there's written: `The implementation compiles a custom codec for each data type in the stream and is most efficient when a single Encoder is used to transmit a stream of values, amortizing the cost of compilation.` – NicoNex Oct 31 '20 at 15:47
  • Yes, but your use case does not allow for that optimization. See the [basics](https://godoc.org/encoding/gob#hdr-Basics) section of the documentation. – Charlie Tumahai Oct 31 '20 at 16:07

0 Answers0