How do I store large files using Meteor?

Question

I'm building a Meteor (meteorjs) app that needs to store and display PDF files, sometimes as large as 500Mb. GridFS doesn't seem to be integrated yet so I'm wondering if it's worth using Meteor in this case or stick to Rails.

Ideally I would not use S3 - I'd like to keep the files on my server.

UPDATE: it seems it's possible to connect outside of Meteor directly, I don't need PDFs to be automatically moved - and it likely doesn't make sense.

More specifically I'm now looking at: MongoDB -> ElasticSearch using https://github.com/richardwilly98/elasticsearch-river-mongodb

Using the instructions at https://github.com/richardwilly98/elasticsearch-river-mongodb/wiki

I would say, you would need to implement the file storage the same way you would do it in the plain node by introducing a connect handler into `WebApp.connectHandlers` — imslavko, Jul 31 '14 at 00:16
@theo3335796 Thanks - I had but I prefer not to use something with such a big warning at the top. I know Meteor is also experimental but looks a LOT more stable. NOTE: This package is under active development right now (2014-3-31). It has bugs and the API may continue to change. Please help test it and fix bugs, but don't use in production yet. imslavko - you might be right, looking into it now. — user1431782, Jul 31 '14 at 03:45
@user1431782 actually im using GridFs on 2 apps and it works pretty fine — Ethaan, Jan 13 '15 at 17:20

score 2 · Accepted Answer · answered Jan 13 '15 at 15:25

You can use GridFS inside meteor without touching any extra package

var db = MongoInternals.defaultRemoteCollectionDriver().mongo.db; //grab the database object
var GridStore = MongoInternals.NpmModule.GridStore;


WebApp.connectHandlers.use('/someurl', function(req, res) {
  var bigFile = new GridStore(db, 'bigfile.iso', 'r') //to read
  bigFile.open(function(error, result) {

    if (error) return

    bigFile.stream(); //stream the file
    bigFile.on('error', function(e) {...}) //handle error etc
    bigFile.on('end', function() {bigFile.close();}); //close the file when done

    bigFile.pipe(res); //pipe the file to res
  });
});

However, the current GridStore/mongo (v1.3.x) used by Meteor is a bit dated, the newest verion is 2.x from http://mongodb.github.io/node-mongodb-native/2.0/api-docs/ The v1.x doesnt seem to pipe well so you may need to use the newer version

The second option

var db = MongoInternals.defaultRemoteCollectionDriver().mongo.db; //grab the database object
var GridStore = Npm.require('mongodb').GridStore; //add Npm.depends({mongodb:'2.0.13'}) in your package.js

WebApp.connectHandlers.use('/someurl', function(req, res) {
    var bigFile = new GridStore(db, 'bigfile.iso', 'r').stream(true); //the new API doens't require bigFile.open() and will close automatically on end
    bigFile.on('error', function(e) {...}); //handle error etc
    bigFile.on('end', function() {...});
    bigFile.pipe(res); //pipe the file to res
});

In this example, I use the WebApp.connectHandlers, but of course you can use iron: router or something. I tried with a file of 500 MB and it pipes all well. You also need to set the res.writeHead(200) and other stuff such as content-type, etc

How do I store large files using Meteor?

1 Answers1