What's the easiest way to store a large object in Meteor using GridFS?

Question

I'm trying to backup a Lunr Index to Mongo (daily), and because it's running around 13MB, I'm triggering MongoError: document is larger than capped size errors. I'd like to use GridFS to get around the problem, but I'm having a heck of a time getting it to click.

In the simplest terms: Within Meteor, I'd like to save a 13MB JSON object to MongoDB using GridFS, and then be able to retrieve it when necessary -- all only on the server.

I've gone through the File-Collection and CollectionFS docs, and they seem far too complicated for what I'm trying to accomplish, and don't seem to address simply storing the contents of a variable. (Or, more likely, they do, and I'm just missing it.)

Here's what I'd like to do, in pseudo-code:

Backup = new GridFSCollection('backup');

var backupdata = (serialized search index data object);
Backup.insert({name:'searchindex', data:backupdata, date:new Date().getTime()});

var retrieved = Backup.findOne({name:'searchindex'});

Any suggestions?

score 0 · Accepted Answer · answered Jan 13 '15 at 14:11

var db = MongoInternals.defaultRemoteCollectionDriver().mongo.db; 
var GridStore = MongoInternals.NpmModule.GridStore;

var largedata = new GridStore(db,'name','w'); //to write
largedata.open(function(error,gs){
    if (error) return;
    gs.write('somebigdata',function(error,done){
        if (error) return;
        largedata.close();
    })

});


var largedata = new GridStore(db,'name','r'); //to read data
largedata.open(function(error,gs){
    if (error) return;
    gs.read(function(error,result){
        if (error) return;
        //then do something with the result
        largedata.close();
    })

});

explanation

First you need the db object, which can be exposed via the MongoInternals https://github.com/meteor/meteor/tree/devel/packages/mongo

Second, you need the GridStore object, you get it from the MongoInternals as well

Then you can create a write or read object and follow the API http://mongodb.github.io/node-mongodb-native/1.4/markdown-docs/gridfs.html

The mongo used by Meteor is 1.3.x whereas the latest node mongodb native driver is 2.x.x http://mongodb.github.io/node-mongodb-native/2.0/api-docs/

So the API may change in the future. Currently, you need to do open and close and the callback are all async, you may want to wrap them with Meteor.wrapAsync (may not work on largedata.open), Future or Fiber

What's the easiest way to store a large object in Meteor using GridFS?

1 Answers1