0

I'm trying to persist some files in MongoDB. Not interested in GridFS as files are always less than 256KB. It seems difficult to find any good example on how this typically is done.

What I do is open temporary file for reading to get an io.Reader and use bson.NewFromIOReader(io.Reader) to read bytes into bson.Raw type. However I always get an exception unexpected EOF. After some investigation it seems that this function uses io.ReadFull which may throw that exception. So I'm wondering what I do wrong. I can perform io.Copy operation on the same reader without such exception.

Any ideas?

icza
  • 389,944
  • 63
  • 907
  • 827
Michael B.
  • 31
  • 2

1 Answers1

2

If your files are only a few KBs, simplest, cleanest (and maybe even fastest) is to use ioutil.ReadFile() to read the complete file into a byte slice. Save the document where one of the property's value is this byte slice. No magic.

For example:

data, err := ioutil.ReadFile("filename.txt")
if err != nil {
    // Handle error
}

doc := bson.M{
    "data":    data,
    "created": time.Now(),
}

coll := ... // Obtain collection
if _, err := coll.InsertOne(context.Background(), doc); err != nil {
    // handle error
}

In your actual example you may want to use a struct to model your document, e.g.:

type Doc struct {
    ID      primitive.ObjectID `bson:"_id"`
    Data    []byte             `bson:"data"`
    Created time.Time          `bson:"created"`
}

And then use:

doc := &Doc{
    Data:    data,
    Created: time.Now(),
}
icza
  • 389,944
  • 63
  • 907
  • 827
  • This is what I did so far, but I consider it as a very dirty solution. Knowing that MongoDB suppose-to-be a moder database solution, doesn't provided such a simple function as reading a byte stream from io.Reader makes me a little bit sad. Yes, I don't have large files in magnitude of megabites, but think of a case where 100, maybe 1000 requests per second each having to persist 1MB file. Do you really want to want your service to allocated 1GB in memory? I don't. – Michael B. Nov 23 '20 at 16:09