use gobs directly in the source code, is it possible?

Question

I would like to know if it's possible to have gob encoded data directly in the source code (e.g. in a function). The reason is to increase performance by not having to access the disk to get the gob file. I'm aware of memcached, redis and friends. i don't need TTL or any other fancy feature. Just maps in memory. The data would be encoded and dumped in the source code in a 'setup'/build process so that at runtime it would only need to 'decode' it.

The go application would basically serve as a small read-only, embedded database. I can do this using json (basically declare a var with the raw json) but I guess there would be a performance penalty so I'm wondering if it's possible with gob.

I've tried different things but I can't make make it work because basically I don't know how to define the gob var (byte, [bytes] ?? ) and the decoder seems to expect an io.Reader so before spending the whole day on this I've decided to ask you SO fellas at least if it's possible to do.

Miserable attempt:

var test string
test = "hello"

p := new(bytes.Buffer)
e := gob.NewEncoder(p)
e.Encode(test)
ers := ioutil.WriteFile("test.gob", p.Bytes(), 0600)
if ers != nil {
    panic(ers)
}

Now I would like to take test.gob and add it in a function. As I can see, the source of test.gob reads like ^H^L^@^Ehello

var test string

var b bytes.Buffer

b = byte("^H^L^@^Ehello")

de := gob.NewDecoder(b.Bytes())

er := de.Decode(&test)
if er != nil {
    fmt.Printf("cannot decode")
    panic(er)
}

fmt.Fprintf(w, test)

Maybe you'd need to encode it in something more text-safe (Base64 or similar), but it should be doable. — Intermernet, Mar 08 '14 at 13:10

JimB · Answer 1 · 2014-03-08T17:28:21.757

Store the data in a byte slice. It's raw data, and that's how you would read it in from a file.

The string in your gob file is not "^H^L^@^Ehello"! That's how your editor is displaying the non-printable characters.

b = byte("^H^L^@^Ehello")
// This isn't the string equivalent of hello, 
// and you want a []byte, not byte. 
// It should look like 

b = []byte("\b\f\x00\x05hello")
// However, you already declared b as bytes.Buffer, 
// so this assignment isn't valid anyway.


de := gob.NewDecoder(b.Bytes())
// b.Bytes() returns a []byte, you want to read the buffer itself.

Here's a working example http://play.golang.org/p/6pvt2ctwUq

func main() {
    buff := &bytes.Buffer{}
    enc := gob.NewEncoder(buff)
    enc.Encode("hello")

    fmt.Printf("Encoded: %q\n", buff.Bytes())

    // now if you wanted it directly in your source
    encoded := []byte("\b\f\x00\x05hello")
    // or []byte{0x8, 0xc, 0x0, 0x5, 0x68, 0x65, 0x6c, 0x6c, 0x6f}

    de := gob.NewDecoder(bytes.NewReader(encoded))

    var test string
    er := de.Decode(&test)
    if er != nil {
        fmt.Println("cannot decode", er)
        return
    }

    fmt.Println("Decoded:", test)
}

makes sense but it still doesn't answer the question. How do I store/save the gob in the source code ? To make it clear I was looking to encode the data and dump it in the source code in a 'setup'/build process and then so that on the runtime it should only 'decode' it. — The user with no hat, Mar 08 '14 at 17:16
Just write the []byte literal into the source code. I've used this package to automate it before: https://github.com/jteeuwen/go-bindata — JimB, Mar 08 '14 at 17:24

score 1 · Answer 2 · answered Mar 08 '14 at 14:34

For example,

package main

import (
    "bytes"
    "encoding/gob"
    "fmt"
    "io"
)

func writeGOB(data []byte) (io.Reader, error) {
    buf := new(bytes.Buffer)
    err := gob.NewEncoder(buf).Encode(data)
    return buf, err
}

func readGOB(r io.Reader) ([]byte, error) {
    var data []byte
    err := gob.NewDecoder(r).Decode(&data)
    return data, err
}

func main() {
    var in = []byte("hello")
    fmt.Println(string(in))
    r, err := writeGOB(in)
    if err != nil {
        fmt.Println(err)
        return
    }
    out, err := readGOB(r)
    if err != nil {
        fmt.Println(err)
        return
    }
    fmt.Println(string(out))
}

Output:

hello
hello

use gobs directly in the source code, is it possible?

2 Answers2