Using Snappy compression on Gob data in Go?

Question

I need to save a structure to disk and read it in again later, I'm trying to keep IO down to a minimum, but also not spend ages compressing and uncompressing the file, so I intend to use Snappy for compression as it's very fast and relatively efficient.

Normally I would gzip compress a gob when saving it to file, like this:

func (t *Object) Save(filename string) error {
    // Open file for writing
    fi, err := os.Create(filename)
    if err != nil {
        return err
    }
    defer fi.Close()
    // Attach gzip writer
    fz := gzip.NewWriter(fi)
    defer fz.Close()
    // Push from the gob encoder
    encoder := gob.NewEncoder(fz)
    err = encoder.Encode(t.Classifier)
    if err != nil {
        return err
    }
    return nil
}

But Snappy does not attach to these Reader/Writer interfaces that everything else seems to use. Instead it just provides the basic functionality: https://godoc.org/code.google.com/p/snappy-go/snappy

func Encode(dst, src []byte) ([]byte, error)

func Decode(dst, src []byte) ([]byte, error)

What would be the most efficient way to use this Snappy package for compressing the Gob data as its being saved to file (and also reading it back)? Ideally I don't want to use ioutil.ReadAll to just read from the gob reader into a slice of bytes and then compress that all over again as that seems to be a very heavy way of doing it which would create a lot of wasted memory.

I admit that I don't fully understand how the reader and writer interfaces work.

From looking at the package you linked, it doesn't seem to be possible without modifying that package's source. The only way to do it currently is to use ioutil.ReadAll. If you need it to work incrementally you'll have to look for another implementation or write one yourself unfortunately. — Logiraptor, Nov 12 '14 at 06:16

peterSO · Accepted Answer · 2014-11-12T06:31:46.313

package snappystream
import "github.com/mreiferson/go-snappystream"
snappystream wraps snappy-go and supplies a Reader and Writer for the snappy framed stream format.

Have you considered package snappystream? For example,

package main

import (
    "encoding/gob"
    "fmt"
    "os"

    "github.com/mreiferson/go-snappystream"
)

type Object struct {
    Classifier struct{}
}

func (t *Object) Save(filename string) error {
    // Open file for writing
    fi, err := os.Create(filename)
    if err != nil {
        return err
    }
    defer fi.Close()
    // Attach snappy writer
    fs := snappystream.NewBufferedWriter(fi)
    // Push from the gob encoder
    encoder := gob.NewEncoder(fs)
    err = encoder.Encode(t.Classifier)
    if err != nil {
        return err
    }
    err = fs.Close()
    if err != nil {
        return err
    }
    err = fi.Close()
    if err != nil {
        return err
    }
    return nil
}

func main() {
    obj := Object{}
    err := obj.Save("obj.sz")
    if err != nil {
        fmt.Println(err)
    }
}

Using Snappy compression on Gob data in Go?

1 Answers1