0

I am trying to store structs, or more accurately the bytes of a struct to disk. I have large arrays of byte data that I want to compress and store on disk and recover later. The data means nothing to any other application except mine.

I can encode with Gob a struct ([]bytes of) and compress it with gzip. I can also uncompress the resulting data. However it's when I try to convert that []byte data back to the struct that I fail - and the fail is a strangish one... Not an error, When I print out tmp from the function I get

tmp {A: B:0 C:0}

Yet the struct I start with is

type SomeStruct struct {
    A string
    B int64
    C float64
}
var testStruct = SomeStruct{
    A: "james bond",
    B: 24,
    C: 1.234,
}

This is my function

func GobToStruct(binaryBytes io.Reader) (SomeStruct, error) {
    s, _ := ioutil.ReadAll(binaryBytes)
    fmt.Println("read decompressed bytes ", string(s))
    decoder := gob.NewDecoder(binaryBytes)
    var tmp SomeStruct
    if err := decoder.Decode(&tmp); err != nil {
        if err != io.EOF && err != io.ErrUnexpectedEOF {
            fmt.Println("gob failed ", err)
            return tmp, err
        }
    }
    fmt.Printf("tmp %+v\r\n", tmp)
    return tmp, nil
}

What am I doing wrong so that I don't get the data back out after un-gobbing it?

EDIT: I have added the code as it runs now at this gist with a testfile that demonstrates losing the data after compressing

Jonathan Hall
  • 75,165
  • 16
  • 143
  • 189
amlwwalker
  • 3,161
  • 4
  • 26
  • 47
  • 1
    `binaryData` is at EOF after the call to `ioutil.ReadAll(binaryBytes)`. The call to Decode returns io.EOF. The application ignores io.EOF. Fix by removing the call to `ioutil.ReadAll(binaryBytes)` and by handling all errors returned from Decode as an error. – Charlie Tumahai Jan 15 '20 at 21:21
  • Hi @PaulHankin I have added a gist and linked to it at bottom of question. Thanks! – amlwwalker Jan 15 '20 at 21:22
  • Ahhh! I see @CeriseLimón Thanks!!! Updated the gist – amlwwalker Jan 15 '20 at 21:24
  • I would recommend using protocol buffers. It can do exactly this, but offer forward/backward compatibility, and built in packed serialization. – szatmary Jan 16 '20 at 00:05

0 Answers0